Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3c70b6f367 | ||
|
|
b3b555fa23 | ||
|
|
e9805c7eb6 | ||
|
|
3d5af85713 | ||
|
|
c6341f020a | ||
|
|
e157311cd8 | ||
|
|
a5f502eb9c | ||
|
|
7ed2a7d3b4 | ||
|
|
674858fe5e | ||
|
|
9a37881441 | ||
|
|
f324f04944 | ||
|
|
cdafdf6e0f | ||
|
|
b340f86a45 | ||
|
|
f450a6c84f | ||
|
|
5a5d938e8e | ||
|
|
cc86a74dc5 | ||
|
|
6e449b464c | ||
|
|
868fe28c96 | ||
|
|
6e69698343 |
@@ -78,6 +78,27 @@ These rules apply to every task across every project, regardless of harness.
|
||||
when a human reviewer outside the project is required. Document the reason in
|
||||
PROJECT.md.
|
||||
|
||||
**Exception — a GitOps repo where a branch cannot be reconciled.** In a repo whose
|
||||
deployment watches one branch (`mathias/infra`: Flux watches `main`), a branch is
|
||||
invisible to the thing that applies it. A `risk/medium` change to `alerts.yaml` or
|
||||
`litellm-config.yaml` can be linted on a branch but never *exercised* there, so a
|
||||
PR would buy a review of a state that was never deployed.
|
||||
|
||||
There, `risk/medium`+ work stays on `main` and instead requires a **recorded
|
||||
adversarial review pass before the push** — `skills:pr-gatekeeper` or equivalent,
|
||||
reviewing the staged diff. **The reviewer's verdict goes in the issue, not a PR
|
||||
thread.** No verdict recorded means the lane was not used, and saying "the gate was
|
||||
green" is not the same claim.
|
||||
|
||||
Be honest about what this is: a recorded *independent* pass, not four-eyes in the
|
||||
sense `assessor-loop` §4.1 means for a regulated organisation, because the reviewer
|
||||
is an agent. It is more than the repo had and less than the model it mirrors; do not
|
||||
describe it as the stronger property. (infra#470, decided 2026-09-29.)
|
||||
|
||||
This lane depends on the `risk/` label being right. A label that triggers a heavier
|
||||
lane is load-bearing, not cosmetic — if work that belongs in it is unlabelled, the
|
||||
lane silently does not apply.
|
||||
|
||||
6. **Close the loop — every substantive task ends with the same ritual.** Shipping
|
||||
the code is not the end of the task; capturing it is. Run this unprompted:
|
||||
- **Tag + bump SemVer** on the change (annotated tag; minor for a feature or
|
||||
@@ -106,6 +127,43 @@ These rules apply to every task across every project, regardless of harness.
|
||||
revisit Y". Those are exactly the items that need a locator. If it is worth
|
||||
saying in a summary, it is worth writing where it can be found again without you.
|
||||
|
||||
7. **Finish what's closable — only involve me for a genuine call.** If a
|
||||
tracker issue, PR, or checklist item is demonstrably resolved — the fix
|
||||
landed, the state changed, you verified it live — close/merge/check it off
|
||||
in the same turn. Don't leave it open "for the record" or park it waiting
|
||||
on me to confirm something that already happened; that's bookkeeping, not
|
||||
a decision. Reserve asking me (`AskUserQuestion`, a `DECISION NEEDED`
|
||||
comment, or otherwise pausing) for when it's genuinely mine to call:
|
||||
irreversible or prod/host-impacting actions, a real judgment call with no
|
||||
clearly-better option, ambiguous intent, or anything secret-writes/
|
||||
destructive-git already gate. Before claiming something is closable,
|
||||
verify it live (re-check the issue, re-run the command) rather than
|
||||
trusting your own earlier summary (brain: `filed-issue-not-same-as-fixed`).
|
||||
|
||||
8. **Control code needs an adversary, not a checklist.** Code whose job is to
|
||||
refuse, deny, gate or contain is not adequately verified by a green gate.
|
||||
Rule 5 makes CI the quality gate and that holds for ordinary work — but a
|
||||
passing suite only asserts what its author already thought of, and for a
|
||||
control the author is precisely the person whose imagination just failed.
|
||||
Measured on `dispatch#75`: five review rounds, five real escapes verified by
|
||||
execution, and **CI was green on every one of them** while `git rm
|
||||
././.dispatch-allow` deleted the file the control existed to protect. So
|
||||
route this work through a review with authority to block, instructed to
|
||||
verify by execution rather than by reading the implementation. Expect to be
|
||||
blocked more than once; that is the control working, not the reviewer being
|
||||
difficult. (brain: `a-green-gate-is-not-evidence-for-control-code`; the
|
||||
practices live in the `pr-gatekeeper` agent.)
|
||||
|
||||
9. **A test's name must not claim more than its assertions prove.** Name a test
|
||||
after the mechanism it observes, not the goal it serves.
|
||||
`DoesNotInheritEnv` is honest; `IsContained` is a claim the test cannot
|
||||
support. This is not pedantry about naming — when name and assertion
|
||||
diverge, the name wins in the reader's head and the gap becomes invisible,
|
||||
which is what lets the defect survive review. Found three times in one PR,
|
||||
twice in tests written to fix the previous instance of it; each passed while
|
||||
the thing it was named for was exploitable. If you cannot name the mechanism,
|
||||
you do not yet know what the test observes.
|
||||
|
||||
## CAD sprint patterns
|
||||
|
||||
Two CAD sprint archetypes (defs: brain `wiki/homelab/decisions/continuous-agentic-development-cad-concept-2026-06-16`):
|
||||
@@ -319,8 +377,20 @@ Workflow:
|
||||
and missing-untracked adapters). A drift fails the check with a
|
||||
message telling you to stage the regenerated files.
|
||||
|
||||
Behavior rules in this file and per-project rules in `PROJECT.md` apply
|
||||
unconditionally on every host, every harness.
|
||||
Behavior rules in this file and per-project rules in `PROJECT.md` apply on every
|
||||
host and every harness.
|
||||
|
||||
**Where a project rule contradicts a rule here, the project rule governs — and it
|
||||
must state its reason.** Both halves matter: the project file wins because it knows
|
||||
something this file cannot (infra's Flux constraint is real and not visible from
|
||||
here), and it must say why because a bare override is indistinguishable from drift.
|
||||
|
||||
This replaces the previous wording, which said rules here and in `PROJECT.md` both
|
||||
apply *"unconditionally"*. That was not a tiebreak but a contradiction, and it went
|
||||
unnoticed until rule 5 and infra's "never create feature branches" had to be obeyed
|
||||
at the same time (infra#470). A standard can hold two individually sensible clauses
|
||||
that are jointly unsatisfiable, and survive for as long as everyone reads it
|
||||
charitably — the same failure the Inception Sprint Oath's G6 had.
|
||||
|
||||
## Engineering Skills
|
||||
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
# Supervisor MCP server
|
||||
SUPERVISOR_PORT=3200
|
||||
SUPERVISOR_CONFIG_DIR=./config/supervisor
|
||||
SUPERVISOR_MODELS_FILE=./config/models.yaml
|
||||
|
||||
# LiteLLM gateway (iguana)
|
||||
LITELLM_BASE_URL=http://iguana:4000
|
||||
|
||||
+13
-5
@@ -81,11 +81,19 @@ jobs:
|
||||
git config user.email "cd-bot@d-ma.be"
|
||||
git config user.name "CD Bot"
|
||||
git add "k3s/apps/supervisor/ingestion-deployment.yaml"
|
||||
git commit -m "chore(deploy): ingestion → ${IMAGE_TAG}"
|
||||
GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes" \
|
||||
git push
|
||||
|
||||
echo "Infra repo updated: ingestion → ${IMAGE_TAG}"
|
||||
# Idempotent: `git commit` exits 1 on an empty index, so a run that
|
||||
# finds the bump already applied failed the whole job and rang the
|
||||
# CI-red alarm for a deploy that was fine (hyperguild#118). Two
|
||||
# sources of a no-op run: a manual re-run, and the tag-push CI run
|
||||
# that used to shadow the branch-push one for the same sha.
|
||||
if git diff --cached --quiet; then
|
||||
echo "Infra already at ${IMAGE_TAG} — nothing to commit"
|
||||
else
|
||||
git commit -m "chore(deploy): ingestion → ${IMAGE_TAG}"
|
||||
GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes" \
|
||||
git push
|
||||
echo "Infra repo updated: ingestion → ${IMAGE_TAG}"
|
||||
fi
|
||||
|
||||
- name: Trigger Flux reconcile (immediate)
|
||||
run: |
|
||||
|
||||
@@ -2,8 +2,13 @@ name: CI
|
||||
|
||||
"on":
|
||||
push:
|
||||
# Branch only. A `v*` tag here points at a commit main already tested, so
|
||||
# the tag push re-ran the whole gate and — because cd.yml keys off "CI
|
||||
# succeeded" — triggered a SECOND full image build and deploy for the same
|
||||
# sha. That second deploy then died on "nothing to commit" (hyperguild#118).
|
||||
# Nothing consumes a tag-triggered run: there is no release workflow, and
|
||||
# cd.yml reads github.sha, not the tag.
|
||||
branches: [main]
|
||||
tags: ["v*"]
|
||||
pull_request:
|
||||
branches: [main]
|
||||
|
||||
|
||||
@@ -73,6 +73,27 @@ These rules apply to every task across every project, regardless of harness.
|
||||
when a human reviewer outside the project is required. Document the reason in
|
||||
PROJECT.md.
|
||||
|
||||
**Exception — a GitOps repo where a branch cannot be reconciled.** In a repo whose
|
||||
deployment watches one branch (`mathias/infra`: Flux watches `main`), a branch is
|
||||
invisible to the thing that applies it. A `risk/medium` change to `alerts.yaml` or
|
||||
`litellm-config.yaml` can be linted on a branch but never *exercised* there, so a
|
||||
PR would buy a review of a state that was never deployed.
|
||||
|
||||
There, `risk/medium`+ work stays on `main` and instead requires a **recorded
|
||||
adversarial review pass before the push** — `skills:pr-gatekeeper` or equivalent,
|
||||
reviewing the staged diff. **The reviewer's verdict goes in the issue, not a PR
|
||||
thread.** No verdict recorded means the lane was not used, and saying "the gate was
|
||||
green" is not the same claim.
|
||||
|
||||
Be honest about what this is: a recorded *independent* pass, not four-eyes in the
|
||||
sense `assessor-loop` §4.1 means for a regulated organisation, because the reviewer
|
||||
is an agent. It is more than the repo had and less than the model it mirrors; do not
|
||||
describe it as the stronger property. (infra#470, decided 2026-09-29.)
|
||||
|
||||
This lane depends on the `risk/` label being right. A label that triggers a heavier
|
||||
lane is load-bearing, not cosmetic — if work that belongs in it is unlabelled, the
|
||||
lane silently does not apply.
|
||||
|
||||
6. **Close the loop — every substantive task ends with the same ritual.** Shipping
|
||||
the code is not the end of the task; capturing it is. Run this unprompted:
|
||||
- **Tag + bump SemVer** on the change (annotated tag; minor for a feature or
|
||||
@@ -101,6 +122,43 @@ These rules apply to every task across every project, regardless of harness.
|
||||
revisit Y". Those are exactly the items that need a locator. If it is worth
|
||||
saying in a summary, it is worth writing where it can be found again without you.
|
||||
|
||||
7. **Finish what's closable — only involve me for a genuine call.** If a
|
||||
tracker issue, PR, or checklist item is demonstrably resolved — the fix
|
||||
landed, the state changed, you verified it live — close/merge/check it off
|
||||
in the same turn. Don't leave it open "for the record" or park it waiting
|
||||
on me to confirm something that already happened; that's bookkeeping, not
|
||||
a decision. Reserve asking me (`AskUserQuestion`, a `DECISION NEEDED`
|
||||
comment, or otherwise pausing) for when it's genuinely mine to call:
|
||||
irreversible or prod/host-impacting actions, a real judgment call with no
|
||||
clearly-better option, ambiguous intent, or anything secret-writes/
|
||||
destructive-git already gate. Before claiming something is closable,
|
||||
verify it live (re-check the issue, re-run the command) rather than
|
||||
trusting your own earlier summary (brain: `filed-issue-not-same-as-fixed`).
|
||||
|
||||
8. **Control code needs an adversary, not a checklist.** Code whose job is to
|
||||
refuse, deny, gate or contain is not adequately verified by a green gate.
|
||||
Rule 5 makes CI the quality gate and that holds for ordinary work — but a
|
||||
passing suite only asserts what its author already thought of, and for a
|
||||
control the author is precisely the person whose imagination just failed.
|
||||
Measured on `dispatch#75`: five review rounds, five real escapes verified by
|
||||
execution, and **CI was green on every one of them** while `git rm
|
||||
././.dispatch-allow` deleted the file the control existed to protect. So
|
||||
route this work through a review with authority to block, instructed to
|
||||
verify by execution rather than by reading the implementation. Expect to be
|
||||
blocked more than once; that is the control working, not the reviewer being
|
||||
difficult. (brain: `a-green-gate-is-not-evidence-for-control-code`; the
|
||||
practices live in the `pr-gatekeeper` agent.)
|
||||
|
||||
9. **A test's name must not claim more than its assertions prove.** Name a test
|
||||
after the mechanism it observes, not the goal it serves.
|
||||
`DoesNotInheritEnv` is honest; `IsContained` is a claim the test cannot
|
||||
support. This is not pedantry about naming — when name and assertion
|
||||
diverge, the name wins in the reader's head and the gap becomes invisible,
|
||||
which is what lets the defect survive review. Found three times in one PR,
|
||||
twice in tests written to fix the previous instance of it; each passed while
|
||||
the thing it was named for was exploitable. If you cannot name the mechanism,
|
||||
you do not yet know what the test observes.
|
||||
|
||||
## CAD sprint patterns
|
||||
|
||||
Two CAD sprint archetypes (defs: brain `wiki/homelab/decisions/continuous-agentic-development-cad-concept-2026-06-16`):
|
||||
@@ -314,8 +372,20 @@ Workflow:
|
||||
and missing-untracked adapters). A drift fails the check with a
|
||||
message telling you to stage the regenerated files.
|
||||
|
||||
Behavior rules in this file and per-project rules in `PROJECT.md` apply
|
||||
unconditionally on every host, every harness.
|
||||
Behavior rules in this file and per-project rules in `PROJECT.md` apply on every
|
||||
host and every harness.
|
||||
|
||||
**Where a project rule contradicts a rule here, the project rule governs — and it
|
||||
must state its reason.** Both halves matter: the project file wins because it knows
|
||||
something this file cannot (infra's Flux constraint is real and not visible from
|
||||
here), and it must say why because a bare override is indistinguishable from drift.
|
||||
|
||||
This replaces the previous wording, which said rules here and in `PROJECT.md` both
|
||||
apply *"unconditionally"*. That was not a tiebreak but a contradiction, and it went
|
||||
unnoticed until rule 5 and infra's "never create feature branches" had to be obeyed
|
||||
at the same time (infra#470). A standard can hold two individually sensible clauses
|
||||
that are jointly unsatisfiable, and survive for as long as everyone reads it
|
||||
charitably — the same failure the Inception Sprint Oath's G6 had.
|
||||
|
||||
## Engineering Skills
|
||||
|
||||
|
||||
@@ -1,2 +1 @@
|
||||
ingestion: cd ingestion && INGEST_BRAIN_DIR=../brain INGEST_PORT=3300 INGEST_WATCH_INTERVAL=30 go run ./cmd/server/
|
||||
supervisor: SUPERVISOR_CONFIG_DIR=./config/supervisor SUPERVISOR_MODELS_FILE=./config/models.yaml SUPERVISOR_SESSIONS_DIR=./brain/sessions INGEST_BASE_URL=http://localhost:3300 INGEST_SVC_URL=http://localhost:3300 go run ./cmd/supervisor/
|
||||
|
||||
@@ -76,6 +76,64 @@ that prompted the question is answered by naming it here instead.
|
||||
Full reasoning, including the reversal trigger: brain
|
||||
`wiki/hyperguild/decisions/brain-mcp-stays-in-ingestion-fix-discoverability-by-name`.
|
||||
|
||||
## Ingest revises, it does not overwrite
|
||||
|
||||
Re-ingesting a source **merges into the page already on disk** rather than
|
||||
replacing it. Previously the write was unconditional, so a second ingest of the
|
||||
same source discarded everything accumulated since — including hand-written
|
||||
sections.
|
||||
|
||||
The merge is deliberately asymmetric in the existing page's favour:
|
||||
|
||||
| On the page | In the fresh extraction | Result |
|
||||
|---|---|---|
|
||||
| a section | the same section | **the on-disk version wins** |
|
||||
| — | a new section | added |
|
||||
| a bullet list (`Sources`, `Key Claims`, …) | more bullets | unioned, no duplicates |
|
||||
| an append section (`Open Questions`, `Updates`, …) | more lines | only genuinely new lines appended |
|
||||
| frontmatter | frontmatter | **the on-disk frontmatter wins** |
|
||||
|
||||
So an extraction can add to a note but never silently rewrite what a human put
|
||||
there, which is what makes it safe to run on every ingest rather than behind a
|
||||
review gate.
|
||||
|
||||
Re-ingesting unchanged content is **byte-identical** — no diff, so brain-sync
|
||||
does not commit noise that hides real edits.
|
||||
|
||||
Two fields are **generator-owned** and do get refreshed on a revise:
|
||||
`last_updated` and `source_hash`. Everything else on disk wins. The boundary is
|
||||
an explicit allowlist, so a field nobody has thought about stays human-owned.
|
||||
|
||||
### Disagreement is reported, not resolved
|
||||
|
||||
When the fresh extraction produces a *different version* of a section that
|
||||
already exists, the on-disk version still wins — but the run now says so:
|
||||
|
||||
```
|
||||
wiki/sources/shape-up.md: source now disagrees on section "Summary" — on-disk version kept
|
||||
```
|
||||
|
||||
That line lands in `log.md` with the rest of the run's warnings. Without it, "the
|
||||
source is unchanged" and "the source changed its mind and we ignored it" look
|
||||
identical from the outside.
|
||||
|
||||
This is deliberately **textual** disagreement, not semantic contradiction: cheap,
|
||||
deterministic, no model on the write path. Deciding whether a disagreement is a
|
||||
genuine contradiction stays with the Judge-gated `brain-enrich` pass, which is
|
||||
approval-gated by design. Detection here, adjudication there.
|
||||
|
||||
## Re-ingesting an unchanged source is skipped
|
||||
|
||||
Ingest stores `source_hash` — sha256 of the ingested **body**, frontmatter
|
||||
excluded — on the source page. A later ingest of the same source compares it
|
||||
first and skips extraction entirely when it matches, reporting the skip rather
|
||||
than doing it silently.
|
||||
|
||||
Body-only is deliberate: a re-crawl that stamps a new fetch date has not changed
|
||||
the source, and treating that as drift would re-extract the corpus every time a
|
||||
crawler ran. A missing or unreadable hash always means "extract" — a missed skip
|
||||
costs one extraction, a wrong skip loses an update.
|
||||
|
||||
## Phase 1 tools (available now)
|
||||
|
||||
| Tool | What it does |
|
||||
|
||||
+9
-26
@@ -21,7 +21,7 @@ tasks:
|
||||
# ── Development ────────────────────────────────────────────────────────────
|
||||
|
||||
start:
|
||||
desc: Start ingestion + supervisor (requires goreman — go install github.com/mattn/goreman@latest)
|
||||
desc: Start ingestion (requires goreman — go install github.com/mattn/goreman@latest)
|
||||
cmds:
|
||||
- goreman start
|
||||
|
||||
@@ -32,11 +32,6 @@ tasks:
|
||||
- lsof -ti:3200 | xargs kill -9 2>/dev/null || true
|
||||
- echo "hyperguild stopped"
|
||||
|
||||
supervisor:dev:
|
||||
desc: Run supervisor MCP server (development)
|
||||
cmds:
|
||||
- go run ./cmd/supervisor
|
||||
|
||||
hyperguild:dev:
|
||||
desc: Run hyperguild CLI from source (e.g. task hyperguild:dev -- tier)
|
||||
cmds:
|
||||
@@ -67,14 +62,9 @@ tasks:
|
||||
build:
|
||||
desc: Build all binaries
|
||||
cmds:
|
||||
- task: supervisor:build
|
||||
- task: hyperguild:build
|
||||
- task: ingestion:build
|
||||
|
||||
supervisor:build:
|
||||
desc: Build supervisor binary
|
||||
cmds:
|
||||
- go build -trimpath -ldflags="-s -w -X main.version={{.VERSION}}" -o bin/supervisor ./cmd/supervisor
|
||||
|
||||
ingestion:build:
|
||||
desc: Build ingestion server binary
|
||||
dir: ingestion
|
||||
@@ -84,7 +74,7 @@ tasks:
|
||||
# ── Quality ────────────────────────────────────────────────────────────────
|
||||
|
||||
check:
|
||||
desc: Run all checks (context freshness + lint + test + vet) across all modules
|
||||
desc: Run all checks (context freshness + lint + test + vet + build) across all modules
|
||||
cmds:
|
||||
# Context sync is skipped under CI, matching dma-cli, gitea-mcp, and the
|
||||
# rest of the estate. It is not a preference: context-sync.sh walks UP
|
||||
@@ -120,6 +110,12 @@ tasks:
|
||||
- task: lint
|
||||
- task: test
|
||||
- task: vet
|
||||
# `build` is in the gate deliberately (hyperguild#120). vet and test walk
|
||||
# ./... , which enumerates the packages that EXIST — so a build target
|
||||
# naming a DELETED package is invisible to both. `task build` pointed at
|
||||
# ./cmd/supervisor for months after that directory was removed, and CI
|
||||
# stayed green the whole time because nothing in check ever built.
|
||||
- task: build
|
||||
- task: security:gitleaks
|
||||
|
||||
# ── Security ─────────────────────────────────────────────
|
||||
@@ -173,19 +169,6 @@ tasks:
|
||||
- govulncheck ./... || true
|
||||
- cd ingestion && go vet ./...
|
||||
|
||||
supervisor:test:smoke:
|
||||
desc: Smoke test supervisor via MCP (requires start running)
|
||||
cmds:
|
||||
- |
|
||||
curl -s -X POST http://localhost:${SUPERVISOR_PORT:-3200}/mcp \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}' | jq .
|
||||
|
||||
smoke:routing:
|
||||
desc: Boot the routing pod against live LiteLLM + brain and verify _routing logs land
|
||||
cmds:
|
||||
- bash scripts/smoke-routing.sh
|
||||
|
||||
# ── Git / Release ──────────────────────────────────────────────────────────
|
||||
|
||||
tag:
|
||||
|
||||
@@ -55,18 +55,21 @@ Flags:
|
||||
- `--limit N` — max results (default 5)
|
||||
- `--json` — emit the raw response envelope
|
||||
|
||||
### `hyperguild brain write <type> <slug>`
|
||||
### `hyperguild brain write <wing> <hall> <slug>`
|
||||
|
||||
Reads markdown from stdin, writes a knowledge entry.
|
||||
Reads markdown from stdin, writes it to `brain/wiki/<wing>/<hall>/<slug>.md`
|
||||
with `wing`/`hall`/`created_at` frontmatter injected (there is no legacy
|
||||
`brain/knowledge/` fallback — ADR-0012 / #79, both wing and hall are
|
||||
required).
|
||||
|
||||
```bash
|
||||
$ cat <<EOF | hyperguild brain write knowledge example-lesson
|
||||
$ cat <<EOF | hyperguild brain write homelab facts example-lesson
|
||||
# Example lesson
|
||||
|
||||
## Lesson
|
||||
...
|
||||
EOF
|
||||
knowledge/example-lesson.md
|
||||
wiki/homelab/facts/example-lesson.md
|
||||
```
|
||||
|
||||
### `hyperguild brain pass-rate <skill>`
|
||||
|
||||
@@ -60,13 +60,14 @@ func runBrainWrite(ctx context.Context, args []string, stdin io.Reader, stdout,
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return fmt.Errorf("parse flags: %w", err)
|
||||
}
|
||||
if fs.NArg() < 2 {
|
||||
return errors.New("type and slug required (e.g. brain write knowledge my-slug)")
|
||||
if fs.NArg() < 3 {
|
||||
return errors.New("wing, hall and slug required (e.g. brain write homelab facts my-slug)")
|
||||
}
|
||||
kind := fs.Arg(0)
|
||||
slug := fs.Arg(1)
|
||||
wing := fs.Arg(0)
|
||||
hall := fs.Arg(1)
|
||||
slug := fs.Arg(2)
|
||||
|
||||
res, err := newBrainClient().Write(ctx, kind, slug, stdin)
|
||||
res, err := newBrainClient().Write(ctx, wing, hall, slug, stdin)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -101,7 +101,7 @@ func TestRunBrainWrite_Success(t *testing.T) {
|
||||
var out, errBuf bytes.Buffer
|
||||
err := runBrain(
|
||||
context.Background(),
|
||||
[]string{"write", "knowledge", "test-slug"},
|
||||
[]string{"write", "homelab", "facts", "test-slug"},
|
||||
strings.NewReader("# Test\n\nSome body content.\n"),
|
||||
&out, &errBuf,
|
||||
)
|
||||
@@ -111,9 +111,9 @@ func TestRunBrainWrite_Success(t *testing.T) {
|
||||
|
||||
func TestRunBrainWrite_MissingArgs(t *testing.T) {
|
||||
var out, errBuf bytes.Buffer
|
||||
err := runBrain(context.Background(), []string{"write", "knowledge"}, strings.NewReader("x"), &out, &errBuf)
|
||||
err := runBrain(context.Background(), []string{"write", "homelab"}, strings.NewReader("x"), &out, &errBuf)
|
||||
assert.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "type and slug required")
|
||||
assert.Contains(t, err.Error(), "wing, hall and slug required")
|
||||
}
|
||||
|
||||
func TestRunBrainWrite_BackendError(t *testing.T) {
|
||||
@@ -127,7 +127,7 @@ func TestRunBrainWrite_BackendError(t *testing.T) {
|
||||
var out, errBuf bytes.Buffer
|
||||
err := runBrain(
|
||||
context.Background(),
|
||||
[]string{"write", "knowledge", "bad slug"},
|
||||
[]string{"write", "homelab", "facts", "bad slug"},
|
||||
strings.NewReader("body"),
|
||||
&out, &errBuf,
|
||||
)
|
||||
@@ -150,7 +150,7 @@ func TestRunBrainWrite_EmptyStdin(t *testing.T) {
|
||||
t.Setenv("BRAIN_URL", srv.URL)
|
||||
|
||||
var out, errBuf bytes.Buffer
|
||||
err := runBrain(context.Background(), []string{"write", "knowledge", "empty"}, strings.NewReader(""), &out, &errBuf)
|
||||
err := runBrain(context.Background(), []string{"write", "homelab", "facts", "empty"}, strings.NewReader(""), &out, &errBuf)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 0, gotLen, "empty stdin should produce empty content payload")
|
||||
}
|
||||
|
||||
@@ -84,16 +84,17 @@ type WriteResult struct {
|
||||
Path string `json:"path"`
|
||||
}
|
||||
|
||||
func (c *brainClient) Write(ctx context.Context, kind, slug string, content io.Reader) (*WriteResult, error) {
|
||||
func (c *brainClient) Write(ctx context.Context, wing, hall, filename string, content io.Reader) (*WriteResult, error) {
|
||||
body, err := io.ReadAll(content)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read content: %w", err)
|
||||
}
|
||||
payload, err := json.Marshal(struct {
|
||||
Type string `json:"type"`
|
||||
Slug string `json:"slug"`
|
||||
Content string `json:"content"`
|
||||
}{Type: kind, Slug: slug, Content: string(body)})
|
||||
Wing string `json:"wing"`
|
||||
Hall string `json:"hall"`
|
||||
Filename string `json:"filename"`
|
||||
Content string `json:"content"`
|
||||
}{Wing: wing, Hall: hall, Filename: filename, Content: string(body)})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("marshal payload: %w", err)
|
||||
}
|
||||
|
||||
@@ -65,23 +65,25 @@ func TestBrainClient_Write_Success(t *testing.T) {
|
||||
assert.Equal(t, http.MethodPost, r.Method)
|
||||
body, _ := io.ReadAll(r.Body)
|
||||
var got struct {
|
||||
Type string `json:"type"`
|
||||
Slug string `json:"slug"`
|
||||
Content string `json:"content"`
|
||||
Wing string `json:"wing"`
|
||||
Hall string `json:"hall"`
|
||||
Filename string `json:"filename"`
|
||||
Content string `json:"content"`
|
||||
}
|
||||
require.NoError(t, json.Unmarshal(body, &got))
|
||||
assert.Equal(t, "knowledge", got.Type)
|
||||
assert.Equal(t, "find-h", got.Slug)
|
||||
assert.Equal(t, "homelab", got.Wing)
|
||||
assert.Equal(t, "facts", got.Hall)
|
||||
assert.Equal(t, "find-h", got.Filename)
|
||||
assert.Equal(t, "# body\n", got.Content)
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{"path":"knowledge/find-h.md"}`))
|
||||
_, _ = w.Write([]byte(`{"path":"wiki/homelab/facts/find-h.md"}`))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
c := &brainClient{baseURL: srv.URL, http: srv.Client()}
|
||||
res, err := c.Write(context.Background(), "knowledge", "find-h", strings.NewReader("# body\n"))
|
||||
res, err := c.Write(context.Background(), "homelab", "facts", "find-h", strings.NewReader("# body\n"))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "knowledge/find-h.md", res.Path)
|
||||
assert.Equal(t, "wiki/homelab/facts/find-h.md", res.Path)
|
||||
}
|
||||
|
||||
func TestBrainClient_PassRate_Success(t *testing.T) {
|
||||
|
||||
@@ -1,26 +0,0 @@
|
||||
# Model selection — first entry per skill is used.
|
||||
# Override per-call by passing model in the MCP tool args.
|
||||
# Model names come from LiteLLM /v1/models (host/name format).
|
||||
|
||||
default_chain:
|
||||
- iguana/qwen3-coder-next
|
||||
|
||||
skills:
|
||||
tdd:
|
||||
chain:
|
||||
- koala/qwen3-coder-30b
|
||||
review:
|
||||
chain:
|
||||
- iguana/devstral
|
||||
debug:
|
||||
chain:
|
||||
- iguana/deepseek-r1-14b
|
||||
spec:
|
||||
chain:
|
||||
- koala/phi4-14b
|
||||
retrospective:
|
||||
chain:
|
||||
- iguana/qwen3-coder-next
|
||||
trainer:
|
||||
chain:
|
||||
- iguana/qwen3-coder-next
|
||||
@@ -16,7 +16,7 @@ Three needs converge on a single small Go binary:
|
||||
|
||||
- [ ] `hyperguild tier` returns the same `tier.Info` that `internal/tier.Detect` produces for the same probe URLs, in < 3 s under all three tier conditions, with both human-readable and `--json` output.
|
||||
- [ ] `hyperguild brain query <topic>` returns BM25 results from the brain HTTP REST `/query` endpoint, exit 0 on success and non-zero on transport failure.
|
||||
- [ ] `hyperguild brain write <type> <slug>` reads markdown content from stdin, posts to `/write` with the type and slug, and creates `brain/knowledge/<slug>.md`. A round-trip (`hyperguild brain query <slug>` immediately after) finds the entry.
|
||||
- [ ] `hyperguild brain write <wing> <hall> <slug>` reads markdown content from stdin, posts to `/write` with wing/hall/filename, and creates `brain/wiki/<wing>/<hall>/<slug>.md`. A round-trip (`hyperguild brain query <slug>` immediately after) finds the entry. (Updated post-ADR-0012/#79: the legacy `<type> <slug>` shape and the `brain/knowledge/` fallback were removed; wing and hall are now required.)
|
||||
- [ ] `hyperguild mode <cloud|client-local|sovereign>` writes a parseable JSON file at the target path with the per-mode `mcpServers` entries; `jq -e .mcpServers` succeeds on the output.
|
||||
- [ ] All commands print usage on `--help`, exit 2 on unknown flags, exit non-zero on operational errors.
|
||||
- [ ] `task check` passes (lint + test + vet) on each task and on the merged branch.
|
||||
@@ -50,7 +50,7 @@ Three needs converge on a single small Go binary:
|
||||
|
||||
- **`brain` subcommand calls the HTTP REST API.** Two nested subcommands:
|
||||
- `brain query <topic>` issues `POST /query` with JSON body `{query, limit}` (default `--limit 5`), prints results in human-readable form by default and with `--json` for machine consumption.
|
||||
- `brain write <type> <slug>` reads stdin, posts `POST /write` with JSON body `{type, slug, content}`, prints the resulting path on success.
|
||||
- `brain write <wing> <hall> <slug>` reads stdin, posts `POST /write` with JSON body `{wing, hall, filename, content}`, prints the resulting path on success. (Updated post-ADR-0012/#79: wing and hall are required, replacing the old `<type> <slug>` shape and the `brain/knowledge/` legacy fallback.)
|
||||
Rationale: HTTP REST is simpler than MCP framing for a CLI. Per CLAUDE.md, the REST endpoints are documented as the official non-MCP interface.
|
||||
|
||||
- **`mode <name>` writes a per-mode `.mcp.json` template.** Defaults to writing `./.mcp.json` (cwd); accepts `--out <path>`. Per-mode bodies:
|
||||
|
||||
@@ -147,6 +147,37 @@ func envInt(key string, fallback int) int {
|
||||
// buildNotifier returns the ntfy notifier, or a nil interface when
|
||||
// BRAIN_NTFY_URL is unset. A nil *interface* (not a typed-nil) is required so
|
||||
// DegradingSink/Reconcile and the governance observer skip it cleanly.
|
||||
// alertingWarnings names every alert that will not fire under the current
|
||||
// configuration (#107).
|
||||
//
|
||||
// buildNotifier returns a nil interface when BRAIN_NTFY_URL is unset, and every
|
||||
// consumer skips a nil notifier silently — right for an unconfigured homelab,
|
||||
// wrong for an alert whose entire value is that someone finds out. The cost of
|
||||
// the silence is that "configured and working" and "never configured" are
|
||||
// byte-identical in behaviour: no error, no warning, only a message that never
|
||||
// arrives, which nobody notices because nobody is waiting for an alert they
|
||||
// have never seen. Two alerts were dark in production for exactly that reason
|
||||
// (#100), found by firing a real event rather than by reading anything.
|
||||
//
|
||||
// WARN rather than fatal, deliberately. The brain sibling rule
|
||||
// (`empty-credential-must-refuse-to-start-not-switch-modes`) says an empty
|
||||
// credential must be a startup error, never a silent mode switch — but that is
|
||||
// about a control being removed. Here observability is degraded, not a gate
|
||||
// bypassed, and refusing to boot would break the unconfigured-homelab case the
|
||||
// nil notifier exists to support. What the rule does demand, and what this
|
||||
// provides, is that the mode switch stops being silent.
|
||||
//
|
||||
// Pure and env-injected so the message is testable without a running server:
|
||||
// the reason this gap survived is that nothing about it was observable.
|
||||
func alertingWarnings(getenv func(string) string) []string {
|
||||
var out []string
|
||||
if getenv("BRAIN_NTFY_URL") == "" {
|
||||
out = append(out, "alerting DISABLED: BRAIN_NTFY_URL unset — "+
|
||||
"brain_edit_root governance rewrites and capture audit degradation will not notify")
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func buildNotifier() audit.Notifier {
|
||||
if n := audit.NewNtfyNotifier(os.Getenv("BRAIN_NTFY_URL"), os.Getenv("BRAIN_NTFY_TOKEN")); n != nil {
|
||||
return n
|
||||
@@ -203,6 +234,10 @@ func systemHostname() string {
|
||||
func main() {
|
||||
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
|
||||
|
||||
for _, w := range alertingWarnings(os.Getenv) {
|
||||
logger.Warn(w)
|
||||
}
|
||||
|
||||
brainDir := envOr("INGEST_BRAIN_DIR", "../brain")
|
||||
port := envOr("INGEST_PORT", "3300")
|
||||
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
@@ -36,3 +37,42 @@ func TestClaudeSink_IngestWritesToNonIndexedArchiveNotWiki(t *testing.T) {
|
||||
_, err = os.Stat(filepath.Join(dir, "wiki", "claude-sessions"))
|
||||
assert.True(t, os.IsNotExist(err), "raw transcripts must never land under wiki/ (ai-sessions#10 — BM25 pollution)")
|
||||
}
|
||||
|
||||
// --- #107: an unconfigured alert must announce itself ----------------------
|
||||
|
||||
func TestAlertingWarnings(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
env map[string]string
|
||||
want []string // substrings that must appear across the warnings
|
||||
none bool // true = expect no warnings at all
|
||||
}{
|
||||
{
|
||||
// The failure this exists to prevent: two alerts dark in
|
||||
// production, discovered only by firing a real event and watching
|
||||
// nothing arrive (#100).
|
||||
name: "notifier unconfigured names what will not fire",
|
||||
env: map[string]string{},
|
||||
want: []string{"BRAIN_NTFY_URL", "brain_edit_root", "audit"},
|
||||
},
|
||||
{
|
||||
name: "configured notifier is silent",
|
||||
env: map[string]string{"BRAIN_NTFY_URL": "http://ntfy/brain-alerts"},
|
||||
none: true,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got := alertingWarnings(func(k string) string { return tt.env[k] })
|
||||
if tt.none {
|
||||
assert.Empty(t, got)
|
||||
return
|
||||
}
|
||||
joined := strings.Join(got, " ")
|
||||
for _, w := range tt.want {
|
||||
assert.Contains(t, joined, w)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,12 @@
|
||||
package api
|
||||
|
||||
import "strings"
|
||||
import (
|
||||
"strings"
|
||||
|
||||
// Aliased: this package's own frontmatter type (below) would collide
|
||||
// with the imported package's name otherwise.
|
||||
sharedfm "github.com/mathiasbq/hyperguild/ingestion/internal/frontmatter"
|
||||
)
|
||||
|
||||
// frontmatter is an ordered, line-preserving view of a note's YAML
|
||||
// frontmatter block. It deliberately avoids a full YAML round-trip: the
|
||||
@@ -25,21 +31,37 @@ type fmLine struct {
|
||||
// frontmatter block is recognised only when the file opens with a `---`
|
||||
// fence and a closing `---` fence follows. Otherwise the whole input is
|
||||
// the body and the returned frontmatter is empty.
|
||||
//
|
||||
// Fence detection itself is frontmatter.Split (hyperguild#138) — this
|
||||
// package's own contribution is purely the ordered, line-preserving fmLine
|
||||
// shape below, which a shared parser has no business owning since it's
|
||||
// specific to how brain_update does a surgical, byte-round-trip edit.
|
||||
func parseFrontmatter(src string) (frontmatter, string) {
|
||||
var fm frontmatter
|
||||
if !strings.HasPrefix(src, "---\n") {
|
||||
blockBytes, restBytes, ok := sharedfm.Split([]byte(src))
|
||||
if !ok {
|
||||
return fm, src
|
||||
}
|
||||
rest := src[len("---\n"):]
|
||||
end := strings.Index(rest, "\n---\n")
|
||||
if end < 0 {
|
||||
// Opening fence with no closing fence — treat as bodyless content.
|
||||
return fm, src
|
||||
}
|
||||
block := rest[:end]
|
||||
body := rest[end+len("\n---\n"):]
|
||||
// sharedfm.Split's block/rest boundary sits one byte earlier than this
|
||||
// package's always has: block keeps the "\n" immediately before the
|
||||
// closing fence, and rest keeps the "\n" immediately after it (both by
|
||||
// design, for Fix's own reconstruction). This package's block/body
|
||||
// never included either — TrimSuffix/TrimPrefix restore that exact
|
||||
// boundary. Split guarantees ok=true only when "\n---\n" matched, so
|
||||
// both trims are always exactly one byte, never ambiguous.
|
||||
block := strings.TrimSuffix(string(blockBytes), "\n")
|
||||
body := strings.TrimPrefix(string(restBytes), "\n")
|
||||
|
||||
for _, line := range strings.Split(block, "\n") {
|
||||
// An indented line is a continuation (a YAML block-list item, a
|
||||
// wrapped value) never a top-level key, regardless of whether it
|
||||
// happens to contain a colon — " - 'Chapter 3: The Reckoning'" is
|
||||
// not a key named "- 'Chapter 3" (hyperguild#128). Checked before
|
||||
// Cut so a colon inside one never reaches the key/value path.
|
||||
if line != "" && (line[0] == ' ' || line[0] == '\t') {
|
||||
fm.lines = append(fm.lines, fmLine{raw: line})
|
||||
continue
|
||||
}
|
||||
key, val, ok := strings.Cut(line, ":")
|
||||
key = strings.TrimSpace(key)
|
||||
if !ok || key == "" || strings.HasPrefix(strings.TrimSpace(line), "#") {
|
||||
@@ -83,9 +105,19 @@ func (f *frontmatter) render() string {
|
||||
var b strings.Builder
|
||||
b.WriteString("---\n")
|
||||
for _, l := range f.lines {
|
||||
if l.key == "" {
|
||||
switch {
|
||||
case l.key == "":
|
||||
b.WriteString(l.raw)
|
||||
} else {
|
||||
case l.value == "":
|
||||
// A key with nothing after the colon (the YAML block-list
|
||||
// shape: "aliases:" followed by indented "- item" lines on
|
||||
// their own raw lines) must stay bare — "key: " with a
|
||||
// trailing space is a different byte sequence than the
|
||||
// original, and this type's whole purpose (see the package
|
||||
// doc) is a byte-preserving round-trip.
|
||||
b.WriteString(l.key)
|
||||
b.WriteByte(':')
|
||||
default:
|
||||
b.WriteString(l.key)
|
||||
b.WriteString(": ")
|
||||
b.WriteString(l.value)
|
||||
|
||||
@@ -59,3 +59,37 @@ func TestFrontmatterRenderRoundTrips(t *testing.T) {
|
||||
fm, _ := parseFrontmatter(src)
|
||||
assert.Equal(t, src, fm.render())
|
||||
}
|
||||
|
||||
func TestParseFrontmatterIndentedListItemWithColonIsNotMisparsedAsKey(t *testing.T) {
|
||||
// hyperguild#128: an indented YAML block-list item that happens to
|
||||
// contain a colon (e.g. a title with a subtitle) is a continuation
|
||||
// line, never a top-level key — strings.Cut doesn't know that just
|
||||
// because TrimSpace(key) happens to be non-empty. Misparsing it as a
|
||||
// key/value pair and re-rendering strips its indentation, de-indenting
|
||||
// it out of the list it belongs to. Same bug class already found and
|
||||
// fixed in the sibling internal/wiki package's parseFrontmatterBlock
|
||||
// (hyperguild#126 review round 3) — this package was the original the
|
||||
// reimplementation was modeled on and had the gap too.
|
||||
src := "---\naliases:\n - 'Chapter 3: The Reckoning'\ntitle: 'X'\n---\nbody\n"
|
||||
fm, _ := parseFrontmatter(src)
|
||||
fm.set("updated_at", "2026-09-23T00:00:00Z")
|
||||
|
||||
out := fm.render()
|
||||
assert.Contains(t, out, " - 'Chapter 3: The Reckoning'",
|
||||
"the indented list item must survive verbatim, not get re-emitted as a de-indented fake key")
|
||||
assert.NotContains(t, out, "\n- 'Chapter 3",
|
||||
"must not be split into a fake key/value pair at column 0 (as opposed to surviving indented, asserted above)")
|
||||
}
|
||||
|
||||
func TestFrontmatterRenderEmptyValueKeyStaysBareNoTrailingSpace(t *testing.T) {
|
||||
// Companion to the indented-list-item fix: "aliases:" (a key with
|
||||
// nothing after the colon, the YAML block-list header) must round-trip
|
||||
// bare, not as "aliases: " with a trailing space — a different byte
|
||||
// sequence than the original, breaking this type's whole reason for
|
||||
// being a line-preserving view rather than a YAML re-marshal.
|
||||
// render() returns only the frontmatter block (see
|
||||
// TestFrontmatterRenderRoundTrips), no trailing body.
|
||||
src := "---\naliases:\n - one\n - two\n---\n"
|
||||
fm, _ := parseFrontmatter(src)
|
||||
assert.Equal(t, src, fm.render())
|
||||
}
|
||||
|
||||
@@ -54,8 +54,6 @@ type queryRequest struct {
|
||||
type writeRequest struct {
|
||||
Content string `json:"content"`
|
||||
Filename string `json:"filename,omitempty"`
|
||||
Type string `json:"type,omitempty"`
|
||||
Domain string `json:"domain,omitempty"`
|
||||
Wing string `json:"wing,omitempty"`
|
||||
Hall string `json:"hall,omitempty"`
|
||||
SourceType string `json:"source_type,omitempty"` // "external" opts a hall=facts entry out of the internal default
|
||||
@@ -110,17 +108,13 @@ func (h *Handler) Query(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
// WriteNoteOptions configures how a brain note is written.
|
||||
//
|
||||
// When both Wing and Hall are non-empty, the note routes into the
|
||||
// structured wiki at brain/wiki/<wing>/<hall>/<slug>.md and gets
|
||||
// wing/hall/created_at injected into its YAML frontmatter.
|
||||
//
|
||||
// When either is empty, the note falls back to brain/knowledge/<filename>
|
||||
// with optional type/domain frontmatter (legacy behaviour).
|
||||
// Wing and Hall are both required: the note routes into the structured
|
||||
// wiki at brain/wiki/<wing>/<hall>/<slug>.md and gets wing/hall/created_at
|
||||
// injected into its YAML frontmatter. There is no other valid destination
|
||||
// (the legacy brain/knowledge/ fallback was removed — ADR-0012 / #79).
|
||||
type WriteNoteOptions struct {
|
||||
Content string
|
||||
Filename string
|
||||
Type string
|
||||
Domain string
|
||||
Wing string
|
||||
Hall string
|
||||
SourceType string // "internal" marks a first-party observation (e.g. claudewatcher) that needs no external citation
|
||||
@@ -132,6 +126,9 @@ func WriteNote(brainDir string, opts WriteNoteOptions) (string, error) {
|
||||
if opts.Content == "" {
|
||||
return "", fmt.Errorf("content is required")
|
||||
}
|
||||
if opts.Wing == "" || opts.Hall == "" {
|
||||
return "", fmt.Errorf("wing and hall are required")
|
||||
}
|
||||
|
||||
// In the primitive, not in a caller: POST /write reaches here directly,
|
||||
// without the BrainStore, and a gate one door wide is not a gate.
|
||||
@@ -143,13 +140,7 @@ func WriteNote(brainDir string, opts WriteNoteOptions) (string, error) {
|
||||
}
|
||||
opts.Content = clean
|
||||
|
||||
if opts.Wing != "" && opts.Hall != "" {
|
||||
return writeHallNote(brainDir, opts)
|
||||
}
|
||||
if opts.Wing != "" || opts.Hall != "" {
|
||||
return "", fmt.Errorf("wing and hall must be set together")
|
||||
}
|
||||
return writeLegacyNote(brainDir, opts)
|
||||
return writeHallNote(brainDir, opts)
|
||||
}
|
||||
|
||||
// writeHallNote routes a note into brain/wiki/<wing>/<hall>/ and injects
|
||||
@@ -207,9 +198,6 @@ func writeHallNote(brainDir string, opts WriteNoteOptions) (string, error) {
|
||||
fmt.Fprintf(&fm, "%s: %s\n", key, fallback)
|
||||
}
|
||||
}
|
||||
writeField("type", opts.Type)
|
||||
writeField("domain", opts.Domain)
|
||||
|
||||
sourceType := opts.SourceType
|
||||
if sourceType == "" && opts.Hall == "facts" {
|
||||
// Most hall=facts entries are first-party (an eval/benchmark the
|
||||
@@ -300,53 +288,7 @@ func splitFrontmatter(content string) (fields []frontmatterField, body string) {
|
||||
return fields, body
|
||||
}
|
||||
|
||||
// writeLegacyNote preserves the original brain/knowledge/ behaviour for
|
||||
// callers that have not adopted the wing/hall taxonomy.
|
||||
func writeLegacyNote(brainDir string, opts WriteNoteOptions) (string, error) {
|
||||
filename := opts.Filename
|
||||
if filename == "" {
|
||||
filename = fmt.Sprintf("%s-auto.md", time.Now().UTC().Format("2006-01-02-150405"))
|
||||
}
|
||||
|
||||
rawDir := filepath.Join(brainDir, "knowledge")
|
||||
if err := os.MkdirAll(rawDir, 0o755); err != nil {
|
||||
return "", fmt.Errorf("create raw dir: %w", err)
|
||||
}
|
||||
|
||||
finalContent := opts.Content
|
||||
if opts.Type != "" || opts.Domain != "" {
|
||||
var fm strings.Builder
|
||||
fm.WriteString("---\n")
|
||||
if opts.Type != "" {
|
||||
fmt.Fprintf(&fm, "type: %s\n", opts.Type)
|
||||
}
|
||||
if opts.Domain != "" {
|
||||
fmt.Fprintf(&fm, "domain: %s\n", opts.Domain)
|
||||
}
|
||||
fm.WriteString("---\n")
|
||||
finalContent = fm.String() + opts.Content
|
||||
}
|
||||
|
||||
if strings.ContainsAny(filename, `/\`) {
|
||||
return "", fmt.Errorf("invalid filename")
|
||||
}
|
||||
base := filepath.Base(filename)
|
||||
if base == "." || base == ".." || base == "" {
|
||||
return "", fmt.Errorf("invalid filename")
|
||||
}
|
||||
if !strings.HasSuffix(base, ".md") {
|
||||
base += ".md"
|
||||
}
|
||||
dest := filepath.Join(rawDir, base)
|
||||
if err := os.WriteFile(dest, []byte(finalContent), 0o644); err != nil {
|
||||
return "", fmt.Errorf("write: %w", err)
|
||||
}
|
||||
|
||||
rel, _ := filepath.Rel(brainDir, dest)
|
||||
return filepath.ToSlash(rel), nil
|
||||
}
|
||||
|
||||
// Write handles POST /write — write raw content to brain/knowledge/.
|
||||
// Write handles POST /write — write a note into brain/wiki/<wing>/<hall>/.
|
||||
func (h *Handler) Write(w http.ResponseWriter, r *http.Request) {
|
||||
var req writeRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
@@ -359,10 +301,10 @@ func (h *Handler) Write(w http.ResponseWriter, r *http.Request) {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
if req.Wing != "" && req.Hall != "" {
|
||||
if err := brain.BuildWingIndex(h.brainDir, req.Wing); err != nil {
|
||||
h.logger.Warn("auto-index failed", "wing", req.Wing, "err", err)
|
||||
}
|
||||
// WriteNote errors out above unless both are set, so this always runs —
|
||||
// no legacy fallback path left to guard against.
|
||||
if err := brain.BuildWingIndex(h.brainDir, req.Wing); err != nil {
|
||||
h.logger.Warn("auto-index failed", "wing", req.Wing, "err", err)
|
||||
}
|
||||
writeJSON(w, map[string]string{"path": relPath})
|
||||
}
|
||||
|
||||
@@ -65,11 +65,13 @@ func TestQuery_ReturnsResults(t *testing.T) {
|
||||
assert.NotEmpty(t, results)
|
||||
}
|
||||
|
||||
func TestWrite_CreatesKnowledgeFile(t *testing.T) {
|
||||
func TestWrite_CreatesWikiFile(t *testing.T) {
|
||||
dir, h := setup(t)
|
||||
body, _ := json.Marshal(map[string]any{
|
||||
"content": "# Test note\n\nSome content.",
|
||||
"filename": "test-note.md",
|
||||
"filename": "test-note",
|
||||
"wing": "homelab",
|
||||
"hall": "facts",
|
||||
})
|
||||
req := httptest.NewRequest(http.MethodPost, "/write", bytes.NewReader(body))
|
||||
rec := httptest.NewRecorder()
|
||||
@@ -81,11 +83,25 @@ func TestWrite_CreatesKnowledgeFile(t *testing.T) {
|
||||
require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &resp))
|
||||
assert.NotEmpty(t, resp["path"])
|
||||
|
||||
content, err := os.ReadFile(filepath.Join(dir, "knowledge", "test-note.md"))
|
||||
content, err := os.ReadFile(filepath.Join(dir, "wiki", "homelab", "facts", "test-note.md"))
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, string(content), "Some content.")
|
||||
}
|
||||
|
||||
// TestWrite_RequiresWingAndHall is the HTTP-contract counterpart of
|
||||
// TestWriteNote_RequiresWingAndHall: a caller (e.g. the MCP brain_write
|
||||
// handler) that omits wing/hall gets a 400, not a silent legacy write.
|
||||
func TestWrite_RequiresWingAndHall(t *testing.T) {
|
||||
_, h := setup(t)
|
||||
body, _ := json.Marshal(map[string]any{"content": "no destination"})
|
||||
req := httptest.NewRequest(http.MethodPost, "/write", bytes.NewReader(body))
|
||||
rec := httptest.NewRecorder()
|
||||
|
||||
h.Write(rec, req)
|
||||
|
||||
assert.Equal(t, http.StatusBadRequest, rec.Code)
|
||||
}
|
||||
|
||||
func TestQuery_RequiresQuery(t *testing.T) {
|
||||
_, h := setup(t)
|
||||
body, _ := json.Marshal(map[string]any{"limit": 5})
|
||||
@@ -97,27 +113,6 @@ func TestQuery_RequiresQuery(t *testing.T) {
|
||||
assert.Equal(t, http.StatusBadRequest, rec.Code)
|
||||
}
|
||||
|
||||
func TestWrite_IncludesFrontmatterWhenTypeProvided(t *testing.T) {
|
||||
dir, h := setup(t)
|
||||
body, _ := json.Marshal(map[string]any{
|
||||
"content": "Some learning.",
|
||||
"filename": "typed-note.md",
|
||||
"type": "concept",
|
||||
"domain": "software",
|
||||
})
|
||||
req := httptest.NewRequest(http.MethodPost, "/write", bytes.NewReader(body))
|
||||
rec := httptest.NewRecorder()
|
||||
|
||||
h.Write(rec, req)
|
||||
|
||||
assert.Equal(t, http.StatusOK, rec.Code)
|
||||
content, err := os.ReadFile(filepath.Join(dir, "knowledge", "typed-note.md"))
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, string(content), "type: concept")
|
||||
assert.Contains(t, string(content), "domain: software")
|
||||
assert.Contains(t, string(content), "Some learning.")
|
||||
}
|
||||
|
||||
func TestWriteNote_HallRouteIncludesSourceTypeWhenSet(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
|
||||
@@ -126,7 +121,6 @@ func TestWriteNote_HallRouteIncludesSourceTypeWhenSet(t *testing.T) {
|
||||
Filename: "session-koala-abc",
|
||||
Wing: "claude-sessions",
|
||||
Hall: "facts",
|
||||
Type: "source",
|
||||
SourceType: "internal",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
@@ -210,37 +204,95 @@ func TestWriteNote_HallRouteMergesExistingFrontmatterInsteadOfStacking(t *testin
|
||||
assert.Contains(t, body, "# Body")
|
||||
}
|
||||
|
||||
func TestWriteNote_HallRouteExistingTypeWinsOverOptsType(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
|
||||
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
|
||||
Content: "---\ntype: hypothesis\n---\n\nBody.\n",
|
||||
Filename: "note",
|
||||
Wing: "agentsquad",
|
||||
Hall: "decisions",
|
||||
Type: "decision", // should lose to content's own "type: hypothesis"
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, string(got), "type: hypothesis")
|
||||
assert.NotContains(t, string(got), "type: decision")
|
||||
// TestWriteNote_RequiresWingAndHall pins the post-legacy-removal contract
|
||||
// (hyperguild#79 / ADR-0012): with the knowledge/ fallback gone, wing+hall
|
||||
// is the only valid destination, so omitting either is a caller error, not
|
||||
// a silent fallback.
|
||||
func TestWriteNote_RequiresWingAndHall(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
wing string
|
||||
hall string
|
||||
}{
|
||||
{"neither set", "", ""},
|
||||
{"wing only", "homelab", ""},
|
||||
{"hall only", "", "facts"},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
_, err := api.WriteNote(dir, api.WriteNoteOptions{
|
||||
Content: "body",
|
||||
Wing: tc.wing,
|
||||
Hall: tc.hall,
|
||||
})
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "wing and hall are required")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestWrite_GeneratesFilenameIfAbsent(t *testing.T) {
|
||||
dir, h := setup(t)
|
||||
body, _ := json.Marshal(map[string]any{"content": "auto name"})
|
||||
body, _ := json.Marshal(map[string]any{
|
||||
"content": "auto name",
|
||||
"wing": "homelab",
|
||||
"hall": "facts",
|
||||
})
|
||||
req := httptest.NewRequest(http.MethodPost, "/write", bytes.NewReader(body))
|
||||
rec := httptest.NewRecorder()
|
||||
|
||||
h.Write(rec, req)
|
||||
|
||||
assert.Equal(t, http.StatusOK, rec.Code)
|
||||
entries, _ := os.ReadDir(filepath.Join(dir, "knowledge"))
|
||||
// +1 because setup already wrote tdd.md
|
||||
assert.Len(t, entries, 2)
|
||||
assert.True(t, strings.HasSuffix(entries[1].Name(), ".md"))
|
||||
entries, err := os.ReadDir(filepath.Join(dir, "wiki", "homelab", "facts"))
|
||||
require.NoError(t, err)
|
||||
require.Len(t, entries, 1)
|
||||
assert.True(t, strings.HasSuffix(entries[0].Name(), ".md"))
|
||||
}
|
||||
|
||||
// TestWrite_CLIPayloadRoundTrips exercises the real Handler.Write (not a
|
||||
// hand-rolled mock) over an actual HTTP round trip, POSTing the exact JSON
|
||||
// envelope cmd/hyperguild's brainClient.Write sends: {wing, hall, filename,
|
||||
// content}. cmd/hyperguild lives in a separate Go module from ingestion
|
||||
// (no shared go.work), so a Go-level test can't import brainClient
|
||||
// directly — this test instead pins the wire contract from the server
|
||||
// side and proves a real `hyperguild brain write <wing> <hall> <slug>`
|
||||
// invocation writes a real file with real frontmatter under wiki/<wing>/<hall>/.
|
||||
// A prior version of this contract test on the client side (in
|
||||
// cmd/hyperguild/http_test.go) used only an httptest.NewServer with a
|
||||
// hand-rolled mock body, so it stayed green through a regression where the
|
||||
// server required wing/hall but the client never sent them (hyperguild#79
|
||||
// PR review).
|
||||
func TestWrite_CLIPayloadRoundTrips(t *testing.T) {
|
||||
dir, h := setup(t)
|
||||
srv := httptest.NewServer(http.HandlerFunc(h.Write))
|
||||
defer srv.Close()
|
||||
|
||||
payload, err := json.Marshal(struct {
|
||||
Wing string `json:"wing"`
|
||||
Hall string `json:"hall"`
|
||||
Filename string `json:"filename"`
|
||||
Content string `json:"content"`
|
||||
}{Wing: "homelab", Hall: "facts", Filename: "cli-roundtrip", Content: "# CLI roundtrip\n\nReal write.\n"})
|
||||
require.NoError(t, err)
|
||||
|
||||
resp, err := http.Post(srv.URL+"/write", "application/json", bytes.NewReader(payload)) //nolint:noctx
|
||||
require.NoError(t, err)
|
||||
defer resp.Body.Close() //nolint:errcheck
|
||||
|
||||
assert.Equal(t, http.StatusOK, resp.StatusCode)
|
||||
var out struct {
|
||||
Path string `json:"path"`
|
||||
}
|
||||
require.NoError(t, json.NewDecoder(resp.Body).Decode(&out))
|
||||
assert.Equal(t, "wiki/homelab/facts/cli-roundtrip.md", out.Path)
|
||||
|
||||
written, err := os.ReadFile(filepath.Join(dir, "wiki", "homelab", "facts", "cli-roundtrip.md"))
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, string(written), "wing: homelab")
|
||||
assert.Contains(t, string(written), "hall: facts")
|
||||
assert.Contains(t, string(written), "Real write.")
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@@ -145,6 +145,68 @@ func PromoteNote(brainDir string, opts PromoteOptions) (string, error) {
|
||||
return relSlash, nil
|
||||
}
|
||||
|
||||
// RejectOptions identifies a raw note to reject and why.
|
||||
type RejectOptions struct {
|
||||
Filename string // basename in brain/raw/
|
||||
Reason string // optional
|
||||
}
|
||||
|
||||
// RejectNote records a human decision not to promote a brain/raw/ note,
|
||||
// without silently losing it: it stamps rejected_at (+ reject_reason when
|
||||
// given) into frontmatter and moves the file to brain/raw/rejected/, using
|
||||
// the same write-then-delete safety PromoteNote uses — the source survives
|
||||
// any write failure — and the same basename-only filename validation.
|
||||
//
|
||||
// A rejected note never entered the wiki, so no index rebuild or
|
||||
// auto-tunnel run: unlike PromoteNote, this is the end of the note's
|
||||
// journey, not the start of a new one. It stays out of ListPending's view
|
||||
// (raw/rejected/ is a subdirectory, and ListPending's os.ReadDir on raw/
|
||||
// does not recurse), so a rejected note is never re-surfaced as pending —
|
||||
// the human decision, once made, stays made, the same principle behind
|
||||
// this estate's "issue status is surfaced, not auto-closed, but once a
|
||||
// human decides, the decision is recorded" rule applied one level earlier.
|
||||
func RejectNote(brainDir string, opts RejectOptions) (string, error) {
|
||||
base := filepath.Base(opts.Filename)
|
||||
if base != opts.Filename || base == "." || base == ".." || strings.ContainsAny(opts.Filename, `/\`) {
|
||||
return "", fmt.Errorf("invalid filename %q", opts.Filename)
|
||||
}
|
||||
|
||||
src := filepath.Join(brainDir, "raw", base)
|
||||
raw, err := os.ReadFile(src)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return "", fmt.Errorf("pending note %q does not exist in raw/", base)
|
||||
}
|
||||
return "", fmt.Errorf("read source: %w", err)
|
||||
}
|
||||
|
||||
dest := filepath.Join(brainDir, "raw", "rejected", base)
|
||||
if _, statErr := os.Stat(dest); statErr == nil {
|
||||
rel, _ := filepath.Rel(brainDir, dest)
|
||||
return "", fmt.Errorf("a rejected note named %s already exists", filepath.ToSlash(rel))
|
||||
}
|
||||
|
||||
fm, body := parseFrontmatter(string(raw))
|
||||
fm.set("rejected_at", time.Now().UTC().Format(time.RFC3339))
|
||||
if opts.Reason != "" {
|
||||
fm.set("reject_reason", opts.Reason)
|
||||
}
|
||||
|
||||
if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
|
||||
return "", fmt.Errorf("create rejected dir: %w", err)
|
||||
}
|
||||
// Write-then-delete: the source survives any write failure.
|
||||
if err := os.WriteFile(dest, []byte(fm.render()+body), 0o644); err != nil {
|
||||
return "", fmt.Errorf("write rejected note: %w", err)
|
||||
}
|
||||
if err := os.Remove(src); err != nil {
|
||||
return "", fmt.Errorf("rejected note written but source removal failed: %w", err)
|
||||
}
|
||||
|
||||
rel, _ := filepath.Rel(brainDir, dest)
|
||||
return filepath.ToSlash(rel), nil
|
||||
}
|
||||
|
||||
// excerpt returns the first n runes of s, trimmed, single-spaced.
|
||||
func excerpt(s string, n int) string {
|
||||
s = strings.TrimSpace(s)
|
||||
|
||||
@@ -119,3 +119,85 @@ func TestPromoteRebuildsWingIndex(t *testing.T) {
|
||||
require.NoError(t, err, "wing _index regenerated")
|
||||
assert.Contains(t, string(idx), "x", "promoted note appears in the index")
|
||||
}
|
||||
|
||||
// --- RejectNote (hyperguild#81) ---------------------------------------------
|
||||
|
||||
func TestRejectHappyPath(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
writeRaw(t, dir, "2026-04-23-fed-sentiment-scrape.md",
|
||||
"---\ncreated_at: 2026-04-23T13:20:07Z\n---\n# Sentiment scrape\n\nbody.\n")
|
||||
|
||||
rel, err := RejectNote(dir, RejectOptions{
|
||||
Filename: "2026-04-23-fed-sentiment-scrape.md", Reason: "generic scrape, not homelab-relevant",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "raw/rejected/2026-04-23-fed-sentiment-scrape.md", rel)
|
||||
|
||||
// Source removed from the pending queue.
|
||||
_, statErr := os.Stat(filepath.Join(dir, "raw", "2026-04-23-fed-sentiment-scrape.md"))
|
||||
assert.True(t, os.IsNotExist(statErr), "source removed from raw/ after reject")
|
||||
|
||||
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
|
||||
require.NoError(t, err)
|
||||
s := string(got)
|
||||
assert.Contains(t, s, "created_at: 2026-04-23T13:20:07Z", "original created_at preserved")
|
||||
assert.Contains(t, s, "rejected_at:")
|
||||
assert.Contains(t, s, "reject_reason: generic scrape, not homelab-relevant")
|
||||
assert.Contains(t, s, "# Sentiment scrape", "body preserved for later reference")
|
||||
|
||||
// Rejected notes never entered the wiki, and are no longer pending.
|
||||
pending, err := ListPending(dir)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, pending)
|
||||
}
|
||||
|
||||
func TestRejectWithoutReason(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
writeRaw(t, dir, "2026-06-01-x.md", "body\n")
|
||||
rel, err := RejectNote(dir, RejectOptions{Filename: "2026-06-01-x.md"})
|
||||
require.NoError(t, err)
|
||||
got, _ := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
|
||||
assert.NotContains(t, string(got), "reject_reason:", "no reason given, none stamped")
|
||||
}
|
||||
|
||||
func TestRejectMissingSourceErrors(t *testing.T) {
|
||||
_, err := RejectNote(t.TempDir(), RejectOptions{Filename: "ghost.md"})
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
func TestRejectRejectsTraversalFilename(t *testing.T) {
|
||||
_, err := RejectNote(t.TempDir(), RejectOptions{Filename: "../escape.md"})
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
func TestRejectCollisionNoOverwrite(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
dest := filepath.Join(dir, "raw", "rejected", "x.md")
|
||||
require.NoError(t, os.MkdirAll(filepath.Dir(dest), 0o755))
|
||||
require.NoError(t, os.WriteFile(dest, []byte("EXISTING\n"), 0o644))
|
||||
writeRaw(t, dir, "x.md", "NEW\n")
|
||||
|
||||
_, err := RejectNote(dir, RejectOptions{Filename: "x.md"})
|
||||
require.Error(t, err, "collision must error, not overwrite")
|
||||
|
||||
got, _ := os.ReadFile(dest)
|
||||
assert.Equal(t, "EXISTING\n", string(got), "target not overwritten")
|
||||
_, statErr := os.Stat(filepath.Join(dir, "raw", "x.md"))
|
||||
assert.NoError(t, statErr, "source preserved on collision (atomic: no delete without write)")
|
||||
}
|
||||
|
||||
func TestRejectedNotesAreNotListedAsPendingAgain(t *testing.T) {
|
||||
// ListPending already skips tunnel-candidates-*.md; confirm raw/rejected/
|
||||
// (a subdirectory) doesn't leak into the top-level pending listing either
|
||||
// — os.ReadDir on raw/ doesn't recurse, but pin it so a future change to
|
||||
// ListPending can't silently start surfacing rejected notes as pending
|
||||
// again.
|
||||
dir := t.TempDir()
|
||||
writeRaw(t, dir, "2026-06-01-x.md", "body\n")
|
||||
_, err := RejectNote(dir, RejectOptions{Filename: "2026-06-01-x.md"})
|
||||
require.NoError(t, err)
|
||||
|
||||
pending, err := ListPending(dir)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, pending, "a rejected note must not still appear as pending")
|
||||
}
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/claim"
|
||||
)
|
||||
|
||||
// ContentHash returns the lowercase hex sha256 of b. It is the note's
|
||||
@@ -98,14 +99,60 @@ func UpdateNote(brainDir string, opts UpdateNoteOptions) (relPath, contentHash,
|
||||
}
|
||||
priorHash = ContentHash(prior)
|
||||
|
||||
// A caller may lead opts.Content with its own frontmatter fence to set or
|
||||
// change fields on supersede (e.g. the claim envelope). Split it off
|
||||
// before it reaches the body — left un-split, it would render as a
|
||||
// second, inert `---`-fenced block of literal text rather than taking
|
||||
// effect, and any claim-envelope fields inside it would reach disk
|
||||
// unvalidated (hyperguild#111: the same rule that guards note birth in
|
||||
// writeHallNote must guard a change to the same fields).
|
||||
contentFM, contentBody := parseFrontmatter(opts.Content)
|
||||
|
||||
fm, _ := parseFrontmatter(string(prior))
|
||||
for _, l := range contentFM.lines {
|
||||
// updated_at/supersedes/supersede_reason are this function's own
|
||||
// output, not something a caller's fragment sets — updated_at and
|
||||
// supersedes are already unconditionally re-set below regardless of
|
||||
// what the loop does with them, but supersede_reason is only
|
||||
// re-set when opts.Reason is non-empty, so without this skip a
|
||||
// caller could set it via its own fragment and have it survive
|
||||
// even when Reason is empty, bypassing the one intended entry
|
||||
// point for that field (hyperguild#125, found in #111's review).
|
||||
// Skipping all three here rather than relying on the unconditional
|
||||
// overwrites below to save the other two keeps the invariant
|
||||
// explicit at the one place it can be silently broken by a future
|
||||
// reordering, not implicit in call order.
|
||||
if l.key == "" || l.key == "updated_at" || l.key == "supersedes" || l.key == "supersede_reason" {
|
||||
continue
|
||||
}
|
||||
fm.set(l.key, l.value)
|
||||
}
|
||||
fm.set("updated_at", time.Now().UTC().Format(time.RFC3339))
|
||||
fm.set("supersedes", priorHash)
|
||||
if opts.Reason != "" {
|
||||
fm.set("supersede_reason", opts.Reason)
|
||||
}
|
||||
|
||||
out := []byte(fm.render() + opts.Content)
|
||||
// Validate the envelope AFTER the merge, against the full field set that
|
||||
// is about to reach disk — not just the caller's fragment. Validating the
|
||||
// fragment alone would miss a caller that supersedes claim_provenance to
|
||||
// empty while half_life/applies survive untouched from the prior note:
|
||||
// each side looks individually fine, but the merged result is an
|
||||
// unprovenanced envelope, which is the exact laundering this check
|
||||
// exists to catch. (Caught in review before merge, not filed after —
|
||||
// worth being honest about in the history: this second pass was the
|
||||
// actual fix; the first one validated the wrong document.)
|
||||
envelope := make(map[string]string, len(fm.lines))
|
||||
for _, l := range fm.lines {
|
||||
if l.key != "" {
|
||||
envelope[l.key] = l.value
|
||||
}
|
||||
}
|
||||
if err := claim.ValidateEnvelope(envelope); err != nil {
|
||||
return "", "", "", fmt.Errorf("claim envelope: %w", err)
|
||||
}
|
||||
|
||||
out := []byte(fm.render() + contentBody)
|
||||
if err := os.WriteFile(abs, out, 0o644); err != nil {
|
||||
return "", "", "", fmt.Errorf("write: %w", err)
|
||||
}
|
||||
|
||||
@@ -52,6 +52,114 @@ func TestUpdateNoteSupersedesAndStamps(t *testing.T) {
|
||||
assert.Contains(t, s, "supersede_reason: facts changed")
|
||||
}
|
||||
|
||||
func TestUpdateNoteRejectsInvalidEnvelopeAndWritesNothing(t *testing.T) {
|
||||
// hyperguild#111: WriteNote validates the claim envelope before create;
|
||||
// UpdateNote didn't before supersede, so the same rule must hold on
|
||||
// change as on birth, fail-closed, nothing written on rejection.
|
||||
brainDir := t.TempDir()
|
||||
original := "---\nwing: jepa-fx\nhall: facts\ncreated_at: 2026-01-01T00:00:00Z\n---\n# Old\n\nold body\n"
|
||||
rel := seedNote(t, brainDir, "wiki/jepa-fx/facts/val-vol.md", original)
|
||||
|
||||
_, _, _, err := UpdateNote(brainDir, UpdateNoteOptions{
|
||||
Path: rel,
|
||||
Content: "---\nhalf_life: forever\n---\n# New\n\nnew body\n",
|
||||
})
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "claim envelope")
|
||||
|
||||
got, err := os.ReadFile(filepath.Join(brainDir, filepath.FromSlash(rel)))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, original, string(got), "rejected update must leave the note byte-identical")
|
||||
}
|
||||
|
||||
func TestUpdateNoteRejectsAppliesWithoutProvenance(t *testing.T) {
|
||||
brainDir := t.TempDir()
|
||||
original := "---\nwing: jepa-fx\nhall: facts\n---\nold\n"
|
||||
rel := seedNote(t, brainDir, "wiki/jepa-fx/facts/val-vol.md", original)
|
||||
|
||||
_, _, _, err := UpdateNote(brainDir, UpdateNoteOptions{
|
||||
Path: rel,
|
||||
Content: "---\napplies: universal\n---\nnew\n",
|
||||
})
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "claim envelope")
|
||||
|
||||
got, err := os.ReadFile(filepath.Join(brainDir, filepath.FromSlash(rel)))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, original, string(got))
|
||||
}
|
||||
|
||||
func TestUpdateNoteAcceptsValidEnvelopeAndMergesIntoFrontmatter(t *testing.T) {
|
||||
brainDir := t.TempDir()
|
||||
rel := seedNote(t, brainDir, "wiki/jepa-fx/facts/val-vol.md",
|
||||
"---\nwing: jepa-fx\nhall: facts\ncreated_at: 2026-01-01T00:00:00Z\n---\nold\n")
|
||||
|
||||
_, _, _, err := UpdateNote(brainDir, UpdateNoteOptions{
|
||||
Path: rel,
|
||||
Content: "---\nhalf_life: durable\napplies: universal\nclaim_provenance: asserted\n---\nnew body\n",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
got, err := os.ReadFile(filepath.Join(brainDir, filepath.FromSlash(rel)))
|
||||
require.NoError(t, err)
|
||||
s := string(got)
|
||||
assert.Contains(t, s, "half_life: durable")
|
||||
assert.Contains(t, s, "applies: universal")
|
||||
assert.Contains(t, s, "claim_provenance: asserted")
|
||||
assert.Contains(t, s, "wing: jepa-fx", "prior fields survive")
|
||||
assert.Contains(t, s, "new body")
|
||||
assert.Equal(t, 2, strings.Count(s, "---\n"),
|
||||
"exactly one open+close fence pair — the caller's own fence must not leak in as a second, inert block")
|
||||
}
|
||||
|
||||
func TestUpdateNoteRejectsClearingProvenanceWhileEnvelopeSurvives(t *testing.T) {
|
||||
// Found in review, not written speculatively: validating only the
|
||||
// caller's fragment (rather than the merged result) let a caller blank
|
||||
// out claim_provenance alone while half_life/applies survived from the
|
||||
// prior note untouched — each side looks fine on its own, the merged
|
||||
// envelope on disk is the unprovenanced one the whole check exists to
|
||||
// catch.
|
||||
brainDir := t.TempDir()
|
||||
original := "---\nwing: jepa-fx\nhall: facts\nhalf_life: durable\napplies: universal\nclaim_provenance: asserted\n---\nold\n"
|
||||
rel := seedNote(t, brainDir, "wiki/jepa-fx/facts/val-vol.md", original)
|
||||
|
||||
_, _, _, err := UpdateNote(brainDir, UpdateNoteOptions{
|
||||
Path: rel,
|
||||
Content: "---\nclaim_provenance: \n---\nnew body\n",
|
||||
})
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "claim envelope")
|
||||
|
||||
got, err := os.ReadFile(filepath.Join(brainDir, filepath.FromSlash(rel)))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, original, string(got), "rejected update must leave the note byte-identical")
|
||||
}
|
||||
|
||||
func TestUpdateNoteWithoutEnvelopeFieldsIsUnchangedFromToday(t *testing.T) {
|
||||
// Regression pin: plain fence-less content (the common case) must behave
|
||||
// exactly as it did before the envelope-merge fix.
|
||||
brainDir := t.TempDir()
|
||||
rel := seedNote(t, brainDir, "wiki/jepa-fx/facts/val-vol.md",
|
||||
"---\nwing: jepa-fx\nhall: facts\ncreated_at: 2026-01-01T00:00:00Z\ncustom: keep-me\n---\n# Old\n\nold body\n")
|
||||
|
||||
relPath, hash, priorHash, err := UpdateNote(brainDir, UpdateNoteOptions{
|
||||
Path: rel,
|
||||
Content: "# New\n\nnew body\n",
|
||||
Reason: "facts changed",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, rel, relPath)
|
||||
assert.NotEmpty(t, hash)
|
||||
assert.NotEqual(t, hash, priorHash)
|
||||
|
||||
got, err := os.ReadFile(filepath.Join(brainDir, filepath.FromSlash(rel)))
|
||||
require.NoError(t, err)
|
||||
s := string(got)
|
||||
assert.Contains(t, s, "# New")
|
||||
assert.Contains(t, s, "custom: keep-me")
|
||||
assert.Contains(t, s, "supersedes: "+priorHash)
|
||||
}
|
||||
|
||||
func TestUpdateNoteResolvesByWingHallSlug(t *testing.T) {
|
||||
brainDir := t.TempDir()
|
||||
seedNote(t, brainDir, "wiki/jepa-fx/facts/val-vol.md",
|
||||
@@ -123,6 +231,45 @@ func TestUpdateThenReadRoundTripsHash(t *testing.T) {
|
||||
assert.Equal(t, hash, readHash, "update content_hash must round-trip through ReadNote")
|
||||
}
|
||||
|
||||
func TestUpdateNoteSupersedeReasonNotSettableViaCallerFragmentWhenReasonOptionEmpty(t *testing.T) {
|
||||
// hyperguild#125, found in #111's review: the merge loop applied every
|
||||
// field from the caller's leading frontmatter fragment, including
|
||||
// supersede_reason -- which is only re-set unconditionally when
|
||||
// opts.Reason is non-empty, so a caller could set it via its own
|
||||
// fragment and have it survive even with Reason left empty, bypassing
|
||||
// the one intended entry point for that field.
|
||||
brainDir := t.TempDir()
|
||||
rel := seedNote(t, brainDir, "wiki/a/facts/n.md", "---\nwing: a\nhall: facts\n---\nold\n")
|
||||
|
||||
_, _, _, err := UpdateNote(brainDir, UpdateNoteOptions{
|
||||
Path: rel,
|
||||
Content: "---\nsupersede_reason: snuck in via the fragment\n---\nnew\n",
|
||||
// Reason deliberately left empty.
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
got, err := os.ReadFile(filepath.Join(brainDir, filepath.FromSlash(rel)))
|
||||
require.NoError(t, err)
|
||||
assert.NotContains(t, string(got), "supersede_reason",
|
||||
"supersede_reason must not appear at all when opts.Reason is empty, regardless of what the fragment tried to set")
|
||||
}
|
||||
|
||||
func TestUpdateNoteSupersedeReasonFromOptionStillWorks(t *testing.T) {
|
||||
// Regression pin: the legitimate entry point (opts.Reason) must be
|
||||
// completely unaffected by #125's fix.
|
||||
brainDir := t.TempDir()
|
||||
rel := seedNote(t, brainDir, "wiki/a/facts/n.md", "---\nwing: a\nhall: facts\n---\nold\n")
|
||||
|
||||
_, _, _, err := UpdateNote(brainDir, UpdateNoteOptions{
|
||||
Path: rel, Content: "new\n", Reason: "facts changed",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
got, err := os.ReadFile(filepath.Join(brainDir, filepath.FromSlash(rel)))
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, string(got), "supersede_reason: facts changed")
|
||||
}
|
||||
|
||||
func TestContentHashStable(t *testing.T) {
|
||||
assert.Equal(t, ContentHash([]byte("abc")), ContentHash([]byte("abc")))
|
||||
assert.NotEqual(t, ContentHash([]byte("abc")), ContentHash([]byte("abd")))
|
||||
|
||||
@@ -54,11 +54,24 @@ func DetectTunnels(brainDir, content string) ([]TunnelCandidate, error) {
|
||||
return nil
|
||||
}
|
||||
title, _ := readTitleAndCreated(path, strings.TrimSuffix(d.Name(), ".md"))
|
||||
needle := strings.ToLower(strings.TrimSpace(title))
|
||||
if needle == "" {
|
||||
title = strings.TrimSpace(title)
|
||||
if title == "" {
|
||||
return nil
|
||||
}
|
||||
idx := strings.Index(lowerContent, needle)
|
||||
|
||||
// A one-word title has to match the prose case-exactly. Matched
|
||||
// case-insensitively, it links every entry that happens to use the
|
||||
// word: "WRONG" (wiki/telos/decisions/wrong.md) collected 185 inbound
|
||||
// See-also links and "Status" 136, making the two most-linked nodes in
|
||||
// the graph artifacts of the English language rather than references.
|
||||
// Multi-word titles keep the case-insensitive match — "risk tier
|
||||
// middleware" in prose is a real mention however it is capitalised,
|
||||
// and a three-word collision is not an accident.
|
||||
haystack, needle := lowerContent, strings.ToLower(title)
|
||||
if isSingleWord(title) {
|
||||
haystack, needle = content, title
|
||||
}
|
||||
idx := strings.Index(haystack, needle)
|
||||
if idx == -1 {
|
||||
return nil
|
||||
}
|
||||
@@ -69,7 +82,7 @@ func DetectTunnels(brainDir, content string) ([]TunnelCandidate, error) {
|
||||
out = append(out, TunnelCandidate{
|
||||
TargetPath: filepath.ToSlash(rel),
|
||||
MatchedTerm: title,
|
||||
Exact: isWholeWord(lowerContent, idx, len(needle)),
|
||||
Exact: isWholeWord(haystack, idx, len(needle)),
|
||||
})
|
||||
return nil
|
||||
})
|
||||
@@ -79,6 +92,11 @@ func DetectTunnels(brainDir, content string) ([]TunnelCandidate, error) {
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// isSingleWord reports whether title is one whitespace-free token.
|
||||
func isSingleWord(title string) bool {
|
||||
return !strings.ContainsAny(title, " \t")
|
||||
}
|
||||
|
||||
// isWholeWord reports whether the substring at [idx, idx+n) in s is
|
||||
// bounded by non-alphanumeric characters (or string edges).
|
||||
func isWholeWord(s string, idx, n int) bool {
|
||||
|
||||
@@ -102,16 +102,29 @@ func TestDetectTunnels_ExactTitleMatch(t *testing.T) {
|
||||
|
||||
func TestDetectTunnels_FuzzyMatch(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
seedNote(t, dir, "wiki/x/facts/routing.md",
|
||||
"---\ntitle: Routing\n---\nbody.\n")
|
||||
seedNote(t, dir, "wiki/x/facts/routing-table.md",
|
||||
"---\ntitle: 'Routing Table'\n---\nbody.\n")
|
||||
|
||||
// Substring of title appears in content, but not as a whole word.
|
||||
candidates, err := brain.DetectTunnels(dir, "rerouting handles failover")
|
||||
candidates, err := brain.DetectTunnels(dir, "the routing tables were rebuilt")
|
||||
require.NoError(t, err)
|
||||
require.Len(t, candidates, 1)
|
||||
assert.False(t, candidates[0].Exact, "substring-only match should be fuzzy")
|
||||
}
|
||||
|
||||
// A one-word title whose only appearance is lowercase inside another word is
|
||||
// not even a candidate: it would fill the review file with English rather than
|
||||
// with references.
|
||||
func TestDetectTunnels_SingleWordSubstringIsNotACandidate(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
seedNote(t, dir, "wiki/x/facts/routing.md",
|
||||
"---\ntitle: Routing\n---\nbody.\n")
|
||||
|
||||
candidates, err := brain.DetectTunnels(dir, "rerouting handles failover")
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, candidates)
|
||||
}
|
||||
|
||||
func TestDetectTunnels_NoFrontmatterFallsBackToSlug(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
seedNote(t, dir, "wiki/x/facts/widget-flags.md", "# widget flags\n\nbody.\n")
|
||||
@@ -126,20 +139,21 @@ func TestDetectTunnels_NoFrontmatterFallsBackToSlug(t *testing.T) {
|
||||
|
||||
func TestAutoTunnel_FuzzyGoesToCandidatesFile(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
// Existing note in a different wing whose title is "Routing".
|
||||
seedNote(t, dir, "wiki/other/facts/routing.md",
|
||||
"---\nwing: other\nhall: facts\ntitle: Routing\n---\nbody.\n")
|
||||
// Source note in another wing whose body mentions "rerouting" (substring match only).
|
||||
// Existing note in a different wing whose title is "Routing Table".
|
||||
seedNote(t, dir, "wiki/other/facts/routing-table.md",
|
||||
"---\nwing: other\nhall: facts\ntitle: 'Routing Table'\n---\nbody.\n")
|
||||
// Source note in another wing whose body mentions "routing tables"
|
||||
// (substring match only — the title is not there as a whole word).
|
||||
seedNote(t, dir, "wiki/jepa-fx/facts/new.md",
|
||||
"---\nwing: jepa-fx\nhall: facts\n---\nrerouting traffic\n")
|
||||
"---\nwing: jepa-fx\nhall: facts\n---\nrouting tables rebuilt\n")
|
||||
|
||||
require.NoError(t, brain.AutoTunnel(dir,
|
||||
"wiki/jepa-fx/facts/new.md", "rerouting traffic"))
|
||||
"wiki/jepa-fx/facts/new.md", "routing tables rebuilt"))
|
||||
|
||||
// Source must not get auto-linked (fuzzy).
|
||||
got, err := os.ReadFile(filepath.Join(dir, "wiki/jepa-fx/facts/new.md"))
|
||||
require.NoError(t, err)
|
||||
assert.NotContains(t, string(got), "[[other/facts/routing]]")
|
||||
assert.NotContains(t, string(got), "[[other/facts/routing-table]]")
|
||||
|
||||
// Candidates file must list the pair.
|
||||
matches, err := filepath.Glob(filepath.Join(dir, "raw", "tunnel-candidates-*.md"))
|
||||
@@ -148,7 +162,7 @@ func TestAutoTunnel_FuzzyGoesToCandidatesFile(t *testing.T) {
|
||||
body, err := os.ReadFile(matches[0])
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, string(body), "wiki/jepa-fx/facts/new.md")
|
||||
assert.Contains(t, string(body), "wiki/other/facts/routing.md")
|
||||
assert.Contains(t, string(body), "wiki/other/facts/routing-table.md")
|
||||
assert.Contains(t, string(body), "Routing")
|
||||
}
|
||||
|
||||
@@ -175,3 +189,35 @@ func TestWriteTunnel_AppendsToExistingSeeAlso(t *testing.T) {
|
||||
assert.Contains(t, s, "[[a/facts/old]]")
|
||||
assert.Contains(t, s, "[[b/facts/y]]")
|
||||
}
|
||||
|
||||
// A one-word title matched case-insensitively links every entry that happens
|
||||
// to use that word. Live cost before this rule: 185 entries auto-linked to
|
||||
// wiki/telos/decisions/wrong.md (title "WRONG") and 136 to .../status.md,
|
||||
// making the two most-linked nodes in the graph pure artifacts.
|
||||
func TestDetectTunnels_SingleWordTitleNeedsAnExactCaseMatch(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
seedNote(t, dir, "wiki/telos/decisions/wrong.md",
|
||||
"---\nwing: telos\nhall: decisions\ntitle: WRONG\n---\n# WRONG\n\nRetired beliefs.\n")
|
||||
|
||||
got, err := brain.DetectTunnels(dir, "The obvious fix was to reject the write. Wrong here.")
|
||||
require.NoError(t, err)
|
||||
for _, c := range got {
|
||||
assert.False(t, c.Exact, "a lowercase prose word must not auto-link a one-word title: %+v", c)
|
||||
}
|
||||
|
||||
got, err = brain.DetectTunnels(dir, "The beliefs listed under WRONG were retired.")
|
||||
require.NoError(t, err)
|
||||
require.Len(t, got, 1)
|
||||
assert.True(t, got[0].Exact, "the title's own casing appearing in prose is a real reference")
|
||||
}
|
||||
|
||||
func TestDetectTunnels_MultiWordTitleStaysCaseInsensitive(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
seedNote(t, dir, "wiki/homelab/facts/risk-tier.md",
|
||||
"---\nwing: homelab\nhall: facts\ntitle: 'Risk Tier Middleware'\n---\nbody\n")
|
||||
|
||||
got, err := brain.DetectTunnels(dir, "We rebuilt the risk tier middleware last week.")
|
||||
require.NoError(t, err)
|
||||
require.Len(t, got, 1)
|
||||
assert.True(t, got[0].Exact)
|
||||
}
|
||||
|
||||
@@ -51,8 +51,6 @@ func (s *Store) Write(ctx context.Context, n capture.Note) (capture.Ref, error)
|
||||
relPath, err := api.WriteNote(s.brainDir, api.WriteNoteOptions{
|
||||
Content: n.Content,
|
||||
Filename: n.Filename,
|
||||
Type: n.Type,
|
||||
Domain: n.Domain,
|
||||
Wing: n.Wing,
|
||||
Hall: n.Hall,
|
||||
})
|
||||
|
||||
@@ -31,17 +31,14 @@ type StoredNote struct {
|
||||
Body string
|
||||
}
|
||||
|
||||
// Note is the brain-write payload. It carries both the wing/hall taxonomy
|
||||
// and the legacy type/domain fields so a single BrainStore serves both
|
||||
// capture insights and the existing MCP brain_write surface. Reason is
|
||||
// Note is the brain-write payload: wing+hall are the only routing fields
|
||||
// (the legacy type/domain params were removed — ADR-0012 / #79). Reason is
|
||||
// the supersede rationale, used only by Update.
|
||||
type Note struct {
|
||||
Content string
|
||||
Filename string
|
||||
Wing string
|
||||
Hall string
|
||||
Type string
|
||||
Domain string
|
||||
Reason string
|
||||
}
|
||||
|
||||
|
||||
@@ -43,6 +43,26 @@ var DefaultRules = []Rule{
|
||||
// JSON, so both segments begin with "eyJ".
|
||||
{Name: "jwt", RE: regexp.MustCompile(`eyJ[A-Za-z0-9_\-]{8,}\.eyJ[A-Za-z0-9_\-]{8,}\.[A-Za-z0-9_\-]{8,}`)},
|
||||
{Name: "postgres-uri-with-password", RE: regexp.MustCompile(`postgres(?:ql)?://[^:\s/]+:[^@\s/]+@`)},
|
||||
// Any credential embedded in a URL's userinfo, which is what a git credential
|
||||
// store holds: `https://<user>:<token>@host`. The postgres rule above is the
|
||||
// same shape narrowed to one scheme, and is kept ahead of this one only so a
|
||||
// DSN reports the more specific name -- same ordering convention as
|
||||
// authorization-header before bearer-token.
|
||||
//
|
||||
// Added after this shape passed ALL rules and was therefore ingested rather
|
||||
// than dropped (hyperguild#142). It reached ~/.claude/projects -- this
|
||||
// watcher's own input -- because a test suite in mathias/infra executed a
|
||||
// script that configures git while inheriting the operator's real $HOME, read
|
||||
// koala's live credential store, and printed it. Two credentials were rotated
|
||||
// (infra#466). The scrubber is the control that decides whether that class of
|
||||
// mistake is survivable, and for this shape it was failing open.
|
||||
//
|
||||
// The {8,} floor on the secret half is what keeps `https://host:443/path` and
|
||||
// `https://user:@host` out; the `[^:\s/@]+` user half stops a host:port pair
|
||||
// being read as userinfo. Both directions are covered by
|
||||
// TestScrub_URLUserinfoCredential, because a rule that dropped every URL with
|
||||
// a colon in it would discard most of the signal this watcher exists to keep.
|
||||
{Name: "url-userinfo-credential", RE: regexp.MustCompile(`[a-zA-Z][a-zA-Z0-9+.\-]*://[^:\s/@]+:[^@\s/]{8,}@`)},
|
||||
{Name: "private-key", RE: regexp.MustCompile(`-----BEGIN[^-]*PRIVATE KEY-----`)},
|
||||
{Name: "ssh-key", RE: regexp.MustCompile(`ssh-(?:rsa|ed25519|ecdsa)\s+[A-Za-z0-9+/=]{40,}`)},
|
||||
{Name: "github-pat", RE: regexp.MustCompile(`\b(?:ghp|gho|ghu|ghr|gha)_[A-Za-z0-9]{30,}\b`)},
|
||||
|
||||
@@ -46,6 +46,64 @@ func TestScrub_PoisonedFixtures(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// A credential embedded in a URL's userinfo is the shape a git credential store
|
||||
// holds -- `https://<user>:<token>@host` -- and it walked past every rule until
|
||||
// hyperguild#142. There was a rule for `postgres://user:pass@` and nothing for the
|
||||
// general form, so `~/.git-credentials` content was ingested rather than dropped.
|
||||
//
|
||||
// Found the hard way: a test suite in mathias/infra executed a script that
|
||||
// configures git while inheriting the operator's real $HOME, read koala's live
|
||||
// credential store, and printed two live credentials into a session transcript.
|
||||
// ~/.claude/projects is this watcher's own input, so the scrubber was the last
|
||||
// line of defence for that mistake and it failed open. Two credentials were
|
||||
// rotated (infra#466).
|
||||
//
|
||||
// The lesson is not "do not read the credential file" -- that mistake will recur.
|
||||
// It is that this shape must fail closed, because the scrubber is what decides
|
||||
// whether such a mistake is survivable.
|
||||
func TestScrub_URLUserinfoCredential(t *testing.T) {
|
||||
// Every secret below is INVENTED. None is or was a real credential.
|
||||
mustScrub := []struct {
|
||||
name string
|
||||
content string
|
||||
}{
|
||||
{"git-credential-store-user", "https://mathias:0123456789abcdef0123456789abcdef01234567@git.d-ma.be"},
|
||||
{"git-credential-store-oauth2", "https://oauth2:0123456789abcdef0123456789abcdef01234567@git.d-ma.be"},
|
||||
{"in-context-payload-line", `payload line: https://mathias:0123456789abcdef0123456789abcdef01234567@git.d-ma.be`},
|
||||
{"clone-url-in-argv", "git clone https://oauth2:abcdefghijklmnopqrstuvwx@git.d-ma.be/mathias/infra.git"},
|
||||
{"http-not-https", "http://admin:supersecretvalue123@10.0.1.20/api"},
|
||||
{"git-plus-ssh-scheme", "git+ssh://deploy:abcdefghijklmnop@example.com/repo"},
|
||||
{"redis-scheme", "redis://default:abcdefghijklmnopqrst@10.0.1.20:6379/0"},
|
||||
}
|
||||
for _, tc := range mustScrub {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
assert.NotEmpty(t, Scrub(tc.content),
|
||||
"a credential in URL userinfo must be dropped, not ingested")
|
||||
})
|
||||
}
|
||||
|
||||
// The other direction matters as much: a rule that scrubs every URL with a
|
||||
// colon in it would drop most of the signal this watcher exists to keep.
|
||||
mustNotScrub := []struct {
|
||||
name string
|
||||
content string
|
||||
}{
|
||||
{"host-with-port", "cloning https://git.d-ma.be:443/mathias/infra.git"},
|
||||
{"postgres-no-password", "postgres://localhost:5432/brain"},
|
||||
{"url-with-colon-in-query", "GET https://example.com/path?filter=a:b&x=1"},
|
||||
{"plain-url", "see https://git.d-ma.be/mathias/infra/issues/466"},
|
||||
{"userinfo-no-secret", "https://user:@example.com"},
|
||||
{"format-string-placeholder", `printf 'https://oauth2:%s@%s\n' "$TOKEN" "$HOST"`},
|
||||
{"scheme-relative-time", "duration https://example.com took 00:03:22"},
|
||||
}
|
||||
for _, tc := range mustNotScrub {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
assert.Empty(t, Scrub(tc.content),
|
||||
"this is not a credential and must pass through")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestScrub_CleanContentPassesThrough(t *testing.T) {
|
||||
cases := []string{
|
||||
"",
|
||||
|
||||
@@ -66,7 +66,7 @@ func Fix(doc []byte) ([]byte, bool, error) {
|
||||
return doc, false, fmt.Errorf("frontmatter: quoting did not repair the block: %w", err)
|
||||
}
|
||||
|
||||
_, rest, _ := split(doc)
|
||||
_, rest, _ := Split(doc)
|
||||
out := []byte(fence + "\n")
|
||||
out = append(out, fixed...)
|
||||
out = append(out, []byte(fence)...)
|
||||
@@ -122,14 +122,25 @@ func needsQuoting(value string) bool {
|
||||
|
||||
// block returns the raw YAML between the opening and closing fences.
|
||||
func block(doc []byte) ([]byte, bool) {
|
||||
b, _, ok := split(doc)
|
||||
b, _, ok := Split(doc)
|
||||
return b, ok
|
||||
}
|
||||
|
||||
// split returns the frontmatter block and everything from the closing fence
|
||||
// onward (the closing fence itself excluded, its trailing newline included).
|
||||
func split(doc []byte) (block, rest []byte, ok bool) {
|
||||
s := string(doc)
|
||||
// Split returns doc's frontmatter block and everything from the closing
|
||||
// fence onward (the closing fence itself excluded, its trailing newline
|
||||
// included), or ok=false if doc has no properly-fenced frontmatter.
|
||||
//
|
||||
// The canonical fence detector for this codebase (hyperguild#138): three
|
||||
// independent reimplementations of this same logic each grew their own gap
|
||||
// through separate bug-fixing before being consolidated here —
|
||||
// internal/mcp's needed a CRLF/BOM guard added after both silently defeated
|
||||
// its own HasPrefix check, and internal/api's closing-fence match had no
|
||||
// trailing-newline requirement, so a document ending in "\n---extra" with no
|
||||
// real newline after the fence was treated as validly closed. Both classes
|
||||
// are closed here, once, for every caller.
|
||||
func Split(doc []byte) (block, rest []byte, ok bool) {
|
||||
s := strings.TrimPrefix(string(doc), "\xef\xbb\xbf")
|
||||
s = strings.ReplaceAll(s, "\r\n", "\n")
|
||||
if !strings.HasPrefix(s, fence+"\n") {
|
||||
return nil, nil, false
|
||||
}
|
||||
@@ -139,5 +150,18 @@ func split(doc []byte) (block, rest []byte, ok bool) {
|
||||
if idx < 0 {
|
||||
return nil, nil, false
|
||||
}
|
||||
return []byte(body[:idx+1]), []byte(body[idx+1+len(fence):]), true
|
||||
// What follows the closing fence must be either a newline (a real body
|
||||
// follows) or nothing at all (the fence is the literal end of the
|
||||
// document — no body, not a bypass attempt). Anything else, e.g.
|
||||
// "\n---extra", is not a valid close: an earlier version of this check
|
||||
// accepted it, which is exactly the looser match hyperguild#138 closes.
|
||||
after := idx + 1 + len(fence)
|
||||
switch {
|
||||
case after == len(body):
|
||||
return []byte(body[:idx+1]), nil, true
|
||||
case body[after] == '\n':
|
||||
return []byte(body[:idx+1]), []byte(body[after:]), true
|
||||
default:
|
||||
return nil, nil, false
|
||||
}
|
||||
}
|
||||
|
||||
@@ -168,9 +168,9 @@ func TestFix_PreservesBodyExactly(t *testing.T) {
|
||||
require.True(t, changed)
|
||||
assert.True(t, len(got) > 0)
|
||||
|
||||
_, rest, ok := split([]byte(doc))
|
||||
_, rest, ok := Split([]byte(doc))
|
||||
require.True(t, ok)
|
||||
_, gotRest, ok := split(got)
|
||||
_, gotRest, ok := Split(got)
|
||||
require.True(t, ok)
|
||||
assert.Equal(t, string(rest), string(gotRest), "the body must not be touched")
|
||||
}
|
||||
@@ -186,3 +186,55 @@ func TestFix_RefusesToReturnAnUnparseableResult(t *testing.T) {
|
||||
assert.False(t, changed)
|
||||
assert.Equal(t, doc, string(got), "an unfixable document must come back byte-identical")
|
||||
}
|
||||
|
||||
// TestSplit_NormalizesCRLFBeforeFenceDetection is hyperguild#138: a note
|
||||
// whose fence lines use CRLF ("---\r\n") failed strings.HasPrefix(doc,
|
||||
// "---\n") on the precondition read in internal/mcp's independent
|
||||
// reimplementation -- encoding alone, no adversarial input, silently
|
||||
// defeated fence detection there before this consolidation. Split must
|
||||
// never repeat that.
|
||||
func TestSplit_NormalizesCRLFBeforeFenceDetection(t *testing.T) {
|
||||
doc := "---\r\ntitle: 'n'\r\nhalf_life: durable\r\n---\r\n\r\nbody\r\n"
|
||||
block, rest, ok := Split([]byte(doc))
|
||||
require.True(t, ok)
|
||||
assert.Contains(t, string(block), "half_life: durable")
|
||||
assert.Contains(t, string(rest), "body")
|
||||
}
|
||||
|
||||
// TestSplit_StripsALeadingUTF8BOM is the same mechanism one layer earlier:
|
||||
// a leading BOM (Notepad's default "UTF-8" save, PowerShell's pre-v6
|
||||
// Out-File/Set-Content) defeats the same HasPrefix check before CRLF
|
||||
// normalization even applies.
|
||||
func TestSplit_StripsALeadingUTF8BOM(t *testing.T) {
|
||||
doc := "\xef\xbb\xbf---\ntitle: 'n'\n---\n\nbody\n"
|
||||
block, rest, ok := Split([]byte(doc))
|
||||
require.True(t, ok)
|
||||
assert.Contains(t, string(block), "title: 'n'")
|
||||
assert.Contains(t, string(rest), "body")
|
||||
}
|
||||
|
||||
// TestSplit_RequiresANewlineAfterTheClosingFence closes the looser match
|
||||
// internal/api's independent reimplementation carried (strings.Index(body,
|
||||
// "\n"+fence) with no trailing-newline requirement): a document ending in
|
||||
// "\n---extra" with no newline separating the fence from what follows it is
|
||||
// not a valid closing fence and must not be treated as one.
|
||||
func TestSplit_RequiresANewlineAfterTheClosingFence(t *testing.T) {
|
||||
doc := "---\ntitle: 'n'\n---extra-not-a-real-close"
|
||||
_, _, ok := Split([]byte(doc))
|
||||
assert.False(t, ok, "a fence line not followed by a newline is not a valid closing fence")
|
||||
}
|
||||
|
||||
// TestSplit_AcceptsAClosingFenceThatIsTheLastBytesOfTheDocument is a
|
||||
// regression pin found in review of hyperguild#138: the fix for the
|
||||
// "\n---extra" bypass above must not also reject the legitimate case of a
|
||||
// closing fence with nothing after it at all (no trailing newline because
|
||||
// there's no body, not because a bypass is being attempted) -- this
|
||||
// package's own private split() accepted that shape before the
|
||||
// consolidation, and cmd/frontmatterlint's Lint/Fix relied on it.
|
||||
func TestSplit_AcceptsAClosingFenceThatIsTheLastBytesOfTheDocument(t *testing.T) {
|
||||
doc := "---\ntitle: 'n'\n---"
|
||||
block, rest, ok := Split([]byte(doc))
|
||||
require.True(t, ok, "a closing fence at literal end-of-file, with nothing after it, is valid -- there is no body to require a newline before")
|
||||
assert.Contains(t, string(block), "title: 'n'")
|
||||
assert.Equal(t, "", string(rest))
|
||||
}
|
||||
|
||||
@@ -53,14 +53,12 @@ func (s *Server) tools() []map[string]any {
|
||||
},
|
||||
{
|
||||
"name": "brain_write",
|
||||
"description": "Write a markdown note to the brain. With wing+hall set, routes to brain/wiki/<wing>/<hall>/ with wing/hall/created_at frontmatter; otherwise writes to brain/knowledge/ (legacy).",
|
||||
"inputSchema": schema([]string{"content"}, map[string]any{
|
||||
"description": "Write a markdown note to brain/wiki/<wing>/<hall>/ with wing/hall/created_at frontmatter. Both wing and hall are required.",
|
||||
"inputSchema": schema([]string{"content", "wing", "hall"}, map[string]any{
|
||||
"content": str("markdown content"),
|
||||
"filename": str("optional filename or slug"),
|
||||
"type": str("optional frontmatter type (legacy)"),
|
||||
"domain": str("optional frontmatter domain (legacy)"),
|
||||
"wing": str("optional topic domain, e.g. jepa-fx"),
|
||||
"hall": enum("optional memory type (requires wing)", halls...),
|
||||
"wing": str("topic domain, e.g. jepa-fx"),
|
||||
"hall": enum("memory type", halls...),
|
||||
}),
|
||||
},
|
||||
{
|
||||
@@ -98,6 +96,14 @@ func (s *Server) tools() []map[string]any {
|
||||
"slug": str("optional target slug; defaults to filename minus date prefix"),
|
||||
}),
|
||||
},
|
||||
{
|
||||
"name": "brain_reject",
|
||||
"description": "Reject a brain/raw/ note: stamps rejected_at (+ reject_reason if given) into frontmatter and moves it to brain/raw/rejected/ — never deleted outright, and never re-surfaced by brain_pending. The third option alongside promote/leave-pending; a human decision, never automatic. Returns {path}.",
|
||||
"inputSchema": schema([]string{"filename"}, map[string]any{
|
||||
"filename": str("basename in brain/raw/, e.g. 2026-04-23-fed-sentiment-scrape.md"),
|
||||
"reason": str("optional; why this isn't worth promoting"),
|
||||
}),
|
||||
},
|
||||
noteToolDescriptors()[0],
|
||||
noteToolDescriptors()[1],
|
||||
{
|
||||
@@ -227,8 +233,6 @@ func (s *Server) brainQuery(ctx context.Context, args json.RawMessage) (json.Raw
|
||||
type brainWriteArgs struct {
|
||||
Content string `json:"content"`
|
||||
Filename string `json:"filename,omitempty"`
|
||||
Type string `json:"type,omitempty"`
|
||||
Domain string `json:"domain,omitempty"`
|
||||
Wing string `json:"wing,omitempty"`
|
||||
Hall string `json:"hall,omitempty"`
|
||||
}
|
||||
@@ -254,8 +258,6 @@ func (s *Server) brainWrite(ctx context.Context, args json.RawMessage) (json.Raw
|
||||
ref, err := s.store.Write(ctx, capture.Note{
|
||||
Content: a.Content,
|
||||
Filename: a.Filename,
|
||||
Type: a.Type,
|
||||
Domain: a.Domain,
|
||||
Wing: a.Wing,
|
||||
Hall: a.Hall,
|
||||
})
|
||||
@@ -395,6 +397,26 @@ func (s *Server) brainPromote(ctx context.Context, args json.RawMessage) (json.R
|
||||
return json.Marshal(map[string]string{"path": relPath})
|
||||
}
|
||||
|
||||
type brainRejectArgs struct {
|
||||
Filename string `json:"filename"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
}
|
||||
|
||||
// brainReject records a decision not to promote a raw/ note (api.RejectNote:
|
||||
// frontmatter-stamp + move to raw/rejected/). No graph indexing — a rejected
|
||||
// note never entered the wiki, so there's nothing to index.
|
||||
func (s *Server) brainReject(_ context.Context, args json.RawMessage) (json.RawMessage, error) {
|
||||
var a brainRejectArgs
|
||||
if err := json.Unmarshal(args, &a); err != nil {
|
||||
return nil, fmt.Errorf("parse args: %w", err)
|
||||
}
|
||||
relPath, err := api.RejectNote(s.brainDir, api.RejectOptions{Filename: a.Filename, Reason: a.Reason})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return json.Marshal(map[string]string{"path": relPath})
|
||||
}
|
||||
|
||||
// indexInGraph is a best-effort wrapper around graphsync.IndexDoc that
|
||||
// logs failures but never propagates them — the underlying write/ingest
|
||||
// has already succeeded and the graph is an augmentation, not a
|
||||
|
||||
@@ -51,25 +51,50 @@ func TestBrainQueryReturnsResults(t *testing.T) {
|
||||
assert.Contains(t, text, "tdd.md")
|
||||
}
|
||||
|
||||
func TestBrainWriteCreatesFile(t *testing.T) {
|
||||
// TestBrainWriteIgnoresLegacyTypeAndDomainParams pins the post-#79 contract:
|
||||
// domain/type were removed from the schema and the handler's arg struct, so
|
||||
// a caller that still sends them (the old MCP surface) gets a normal wing+
|
||||
// hall write with no error and no effect from the stale fields — silently
|
||||
// dropped by json.Unmarshal, not rejected. Only content in its OWN
|
||||
// frontmatter (unrelated to these params) can set type/domain, per
|
||||
// TestWriteNote_HallRouteMergesExistingFrontmatterInsteadOfStacking in the
|
||||
// api package.
|
||||
func TestBrainWriteIgnoresLegacyTypeAndDomainParams(t *testing.T) {
|
||||
brainDir := t.TempDir()
|
||||
srv := mcp.NewServer(brainDir, nil, nil, nil)
|
||||
|
||||
resp := toolCall(t, srv, "brain_write", map[string]any{
|
||||
"content": "# Test\n\nbody",
|
||||
"filename": "test.md",
|
||||
"filename": "test",
|
||||
"wing": "homelab",
|
||||
"hall": "facts",
|
||||
"type": "note",
|
||||
"domain": "personal",
|
||||
})
|
||||
require.Nil(t, resp["error"])
|
||||
|
||||
got, err := os.ReadFile(filepath.Join(brainDir, "knowledge", "test.md"))
|
||||
got, err := os.ReadFile(filepath.Join(brainDir, "wiki", "homelab", "facts", "test.md"))
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, string(got), "type: note")
|
||||
assert.Contains(t, string(got), "domain: personal")
|
||||
assert.NotContains(t, string(got), "type: note")
|
||||
assert.NotContains(t, string(got), "domain: personal")
|
||||
assert.Contains(t, string(got), "# Test")
|
||||
}
|
||||
|
||||
// TestBrainWriteRequiresWingAndHall is the MCP-tool-call counterpart of
|
||||
// api.TestWriteNote_RequiresWingAndHall: the legacy brain/knowledge/
|
||||
// fallback is gone, so a caller omitting either param gets a tool error,
|
||||
// not a silent legacy write.
|
||||
func TestBrainWriteRequiresWingAndHall(t *testing.T) {
|
||||
brainDir := t.TempDir()
|
||||
srv := mcp.NewServer(brainDir, nil, nil, nil)
|
||||
|
||||
resp := toolCall(t, srv, "brain_write", map[string]any{
|
||||
"content": "x",
|
||||
"filename": "no-destination",
|
||||
})
|
||||
require.NotNil(t, resp["error"])
|
||||
}
|
||||
|
||||
func TestBrainWriteWingHallRoutesToWiki(t *testing.T) {
|
||||
brainDir := t.TempDir()
|
||||
srv := mcp.NewServer(brainDir, nil, nil, nil)
|
||||
@@ -208,30 +233,11 @@ func TestBrainTunnelRejectsMissing(t *testing.T) {
|
||||
require.NotNil(t, resp["error"])
|
||||
}
|
||||
|
||||
func TestBrainWriteRejectsTraversal(t *testing.T) {
|
||||
brainDir := t.TempDir()
|
||||
srv := mcp.NewServer(brainDir, nil, nil, nil)
|
||||
|
||||
resp := toolCall(t, srv, "brain_write", map[string]any{
|
||||
"content": "x",
|
||||
"filename": "../escape.md",
|
||||
})
|
||||
require.NotNil(t, resp["error"])
|
||||
}
|
||||
|
||||
func TestBrainWriteAcceptsDoubleDotInName(t *testing.T) {
|
||||
brainDir := t.TempDir()
|
||||
srv := mcp.NewServer(brainDir, nil, nil, nil)
|
||||
|
||||
resp := toolCall(t, srv, "brain_write", map[string]any{
|
||||
"content": "x",
|
||||
"filename": "notes..draft.md",
|
||||
})
|
||||
require.Nil(t, resp["error"])
|
||||
|
||||
_, err := os.Stat(filepath.Join(brainDir, "knowledge", "notes..draft.md"))
|
||||
require.NoError(t, err, "filename with embedded .. should be allowed")
|
||||
}
|
||||
// Filename traversal/sanitisation on the wiki route is covered directly
|
||||
// against brain.NotePath/Sanitise in internal/brain/path_test.go
|
||||
// (TestNotePath_SanitisesWingAndSlug, TestSanitise) — those tests exercised
|
||||
// the now-deleted brain/knowledge/ legacy path's own separate filename
|
||||
// validation, which had no equivalent once that path was removed (#79).
|
||||
|
||||
func TestBrainIngestRawDryRun(t *testing.T) {
|
||||
brainDir := t.TempDir()
|
||||
@@ -390,3 +396,38 @@ func TestBrainPromoteInvalidHallErrors(t *testing.T) {
|
||||
_, srcErr := os.Stat(filepath.Join(raw, "x.md"))
|
||||
assert.NoError(t, srcErr, "source untouched on validation error")
|
||||
}
|
||||
|
||||
func TestBrainRejectMovesToRejectedSubdir(t *testing.T) {
|
||||
brainDir := t.TempDir()
|
||||
raw := filepath.Join(brainDir, "raw")
|
||||
require.NoError(t, os.MkdirAll(raw, 0o755))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(raw, "2026-04-23-scrape.md"),
|
||||
[]byte("---\ncreated_at: 2026-04-23T13:20:07Z\n---\n# Scrape\n\nbody\n"), 0o644))
|
||||
srv := mcp.NewServer(brainDir, nil, nil, nil)
|
||||
|
||||
resp := toolCall(t, srv, "brain_reject", map[string]any{
|
||||
"filename": "2026-04-23-scrape.md", "reason": "generic scrape, not homelab-relevant",
|
||||
})
|
||||
require.Nil(t, resp["error"], "got: %v", resp["error"])
|
||||
text := resp["result"].(map[string]any)["content"].([]any)[0].(map[string]any)["text"].(string)
|
||||
assert.Contains(t, text, "raw/rejected/2026-04-23-scrape.md")
|
||||
|
||||
got, err := os.ReadFile(filepath.Join(brainDir, "raw/rejected/2026-04-23-scrape.md"))
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, string(got), "reject_reason: generic scrape, not homelab-relevant")
|
||||
|
||||
_, srcErr := os.Stat(filepath.Join(raw, "2026-04-23-scrape.md"))
|
||||
assert.True(t, os.IsNotExist(srcErr), "source removed from raw/")
|
||||
|
||||
// A rejected note must not still show up as pending.
|
||||
pendingResp := toolCall(t, srv, "brain_pending", map[string]any{})
|
||||
pendingText := pendingResp["result"].(map[string]any)["content"].([]any)[0].(map[string]any)["text"].(string)
|
||||
assert.Contains(t, pendingText, `"pending":[]`)
|
||||
}
|
||||
|
||||
func TestBrainRejectMissingSourceErrors(t *testing.T) {
|
||||
brainDir := t.TempDir()
|
||||
srv := mcp.NewServer(brainDir, nil, nil, nil)
|
||||
resp := toolCall(t, srv, "brain_reject", map[string]any{"filename": "ghost.md"})
|
||||
require.NotNil(t, resp["error"])
|
||||
}
|
||||
|
||||
@@ -275,6 +275,8 @@ func (s *Server) handleCall(ctx context.Context, name string, args json.RawMessa
|
||||
return s.brainPending(ctx, args)
|
||||
case "brain_promote":
|
||||
return s.brainPromote(ctx, args)
|
||||
case "brain_reject":
|
||||
return s.brainReject(ctx, args)
|
||||
case "capture":
|
||||
return s.brainCapture(ctx, args)
|
||||
case "brain_read":
|
||||
|
||||
@@ -58,7 +58,7 @@ func TestServerToolsList(t *testing.T) {
|
||||
}
|
||||
assert.ElementsMatch(t, []string{
|
||||
"brain_query", "brain_write", "brain_update", "brain_get",
|
||||
"brain_pending", "brain_promote",
|
||||
"brain_pending", "brain_promote", "brain_reject",
|
||||
"brain_read", "brain_edit",
|
||||
"brain_tunnel",
|
||||
"brain_ingest_raw", "brain_ingest",
|
||||
|
||||
@@ -3,6 +3,7 @@ package mcp
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"strings"
|
||||
@@ -10,7 +11,9 @@ import (
|
||||
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/claim"
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/classification"
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/frontmatter"
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/notes"
|
||||
)
|
||||
|
||||
@@ -178,8 +181,29 @@ func (s *Server) editWithScope(args json.RawMessage, scope notes.Scope) (notes.R
|
||||
if err := json.Unmarshal(args, &a); err != nil {
|
||||
return notes.Result{}, nil, fmt.Errorf("parse args: %w", err)
|
||||
}
|
||||
|
||||
// hyperguild#123: brain_write/brain_update both validate the claim
|
||||
// envelope before writing (hyperguild#111: "same rules when a note
|
||||
// changes as when it's born"); brain_edit didn't — a byte-level string
|
||||
// replace has no frontmatter awareness at all, so it could rewrite an
|
||||
// existing half_life:/applies:/claim_provenance: line straight past the
|
||||
// validator entirely. Scoped to notes only (governance files have no
|
||||
// claim-envelope concept).
|
||||
if scope == notes.ScopeNotes {
|
||||
if verr := checkClaimEnvelope(s.brainDir, a.Path, a.OldStr, a.NewStr); verr != nil {
|
||||
return notes.Result{}, nil, fmt.Errorf("claim envelope: %w", verr)
|
||||
}
|
||||
}
|
||||
|
||||
res, err := notes.Edit(s.brainDir, a.Path, a.OldStr, a.NewStr, scope)
|
||||
if err != nil {
|
||||
if errors.Is(err, notes.ErrDegenerate) && a.OldStr == "" {
|
||||
// hyperguild#115: "old_str is empty" alone doesn't say whether the
|
||||
// caller sent it empty or a hop between the caller and here lost
|
||||
// it. Naming the raw argument byte count makes that distinguishable
|
||||
// without a second investigation.
|
||||
return notes.Result{}, nil, fmt.Errorf("%w (raw arguments were %d bytes)", err, len(args))
|
||||
}
|
||||
return notes.Result{}, nil, err
|
||||
}
|
||||
|
||||
@@ -220,3 +244,156 @@ func touchesFrontmatter(oldStr, newStr string) bool {
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// checkClaimEnvelope previews what old_str->new_str would produce — the
|
||||
// exact strings.Replace(before, oldStr, newStr, 1) formula notes.Edit
|
||||
// itself uses internally — and validates the resulting frontmatter's claim
|
||||
// envelope before the real edit is allowed to run. Fail-closed: an invalid
|
||||
// result blocks the edit entirely, the same posture writeHallNote/UpdateNote
|
||||
// already apply on create/supersede.
|
||||
//
|
||||
// Gated on old_str appearing within the note's frontmatter BLOCK, not on
|
||||
// old_str/new_str mentioning a claim-envelope field NAME — found in review:
|
||||
// a field-name-substring gate misses a value-only edit like
|
||||
// "asserted"->"guessed" on a claim_provenance line, which never mentions
|
||||
// the field name at all. That is exactly the shape of edit a minimal,
|
||||
// natural diff produces, so a field-name gate was closing the issue in
|
||||
// name only. Checking block containment instead means any edit that
|
||||
// couldn't possibly touch the frontmatter (its target text only exists in
|
||||
// the body) is still skipped cheaply, while nothing inside the fence can
|
||||
// slip through unvalidated regardless of which token it targets.
|
||||
//
|
||||
// This does not replicate notes.Edit's uniqueness/CRLF/conflict-marker
|
||||
// checks — if old_str is absent or ambiguous, this preview may not match
|
||||
// what Edit ultimately does, but Edit independently and correctly rejects
|
||||
// those cases at write time regardless of what this function decides, so
|
||||
// the only thing riding on this preview's accuracy is the envelope check
|
||||
// itself, not write safety.
|
||||
// checkClaimEnvelope previews what old_str->new_str would produce and
|
||||
// validates the resulting claim envelope before the real edit runs
|
||||
// (hyperguild#123). Seven review rounds closed every plausible bypass in the
|
||||
// fence-detection family (field-name substring, block-containment,
|
||||
// fence-stripping, CRLF, UTF-8 BOM) and the first instance of a
|
||||
// value-extraction gap (a folded YAML block scalar hiding a value behind an
|
||||
// empty inline colon).
|
||||
//
|
||||
// Known, accepted v1 gap: frontmatterScalars's block-scalar fold is a
|
||||
// best-effort approximation, not a YAML parser — a blank line interrupting a
|
||||
// folded continuation, or other contrived multi-line YAML shapes not yet
|
||||
// probed (block-scalar indicators `|`/`>`, inline comments on a key line,
|
||||
// quoted multi-line values), can still hide an invalid value from
|
||||
// ValidateEnvelope. Accepted deliberately rather than chased indefinitely:
|
||||
// every bypass found so far requires the caller to already hold brain_edit
|
||||
// write access and to deliberately construct a specific malformed-YAML shape
|
||||
// with knowledge of this exact gap — a narrow, low-likelihood threat next to
|
||||
// the diminishing returns of another incremental patch round. The real fix
|
||||
// (a shared, tested, encoding-normalized frontmatter parser used by all
|
||||
// three independent implementations in this codebase) is tracked separately
|
||||
// as hyperguild#138, not bolted on here piece by piece.
|
||||
func checkClaimEnvelope(brainDir, path, oldStr, newStr string) error {
|
||||
if oldStr == "" {
|
||||
// hyperguild#115: strings.Replace treats "" as matching position 0,
|
||||
// which would make an empty old_str look like an edit that inserts
|
||||
// text before the fence — a degenerate-input case that belongs to
|
||||
// notes.Edit's own "old_str is empty" refusal, not to the envelope
|
||||
// check.
|
||||
return nil
|
||||
}
|
||||
before, err := notes.Read(brainDir, path)
|
||||
if err != nil {
|
||||
// Let notes.Edit itself produce the real error (missing file, etc.)
|
||||
// — this function's job is the envelope, not read failures.
|
||||
return nil
|
||||
}
|
||||
// No gate on where old_str falls relative to the frontmatter fence:
|
||||
// round 2 found that requiring old_str to be fully contained in the
|
||||
// extracted block was itself bypassable by an old_str disambiguated
|
||||
// across the closing "---" into the body. Simpler and bypass-proof:
|
||||
// always compute what the edit would produce and validate that
|
||||
// resulting state's envelope. A body-only edit leaves the frontmatter
|
||||
// (and thus the validation outcome) unchanged, so this is a no-op for
|
||||
// the common case and fail-closed for every case that touches the
|
||||
// envelope, regardless of how old_str is phrased.
|
||||
_, hadFence := frontmatterBlock(string(before))
|
||||
after := strings.Replace(string(before), oldStr, newStr, 1)
|
||||
_, hasFence := frontmatterBlock(after)
|
||||
if hadFence && !hasFence {
|
||||
// Round 3: an edit that drops the closing fence degrades `after`
|
||||
// to "unfenced" — frontmatterScalars then sees an empty map and
|
||||
// ValidateEnvelope({}) is nil by design (absent fields are valid),
|
||||
// so an invalid value would otherwise land with zero error while
|
||||
// also corrupting the note's structure. A previously-fenced note
|
||||
// must stay fenced, full stop.
|
||||
return fmt.Errorf("edit would remove the frontmatter fence")
|
||||
}
|
||||
return claim.ValidateEnvelope(frontmatterScalars(after))
|
||||
}
|
||||
|
||||
// frontmatterBlock returns the raw text between doc's opening and closing
|
||||
// `---` fences, or ok=false if doc has no (properly closed) fence.
|
||||
func frontmatterBlock(doc string) (block string, ok bool) {
|
||||
// hyperguild#138: fence detection itself is frontmatter.Split, shared
|
||||
// with internal/api and internal/frontmatter — the CRLF/BOM guards that
|
||||
// used to live here (added piecemeal across rounds 4 and 5 of #123's
|
||||
// review) are now the shared parser's job, so every caller inherits a
|
||||
// fix instead of needing bug-for-bug parity edits.
|
||||
b, _, ok := frontmatter.Split([]byte(doc))
|
||||
if !ok {
|
||||
return "", false
|
||||
}
|
||||
// Split's block keeps the "\n" immediately before the closing fence
|
||||
// (by design — see its own doc comment); this function never included
|
||||
// it. Split guarantees ok=true only when "\n---\n" matched, so this is
|
||||
// always exactly one byte, never ambiguous.
|
||||
return strings.TrimSuffix(string(b), "\n"), true
|
||||
}
|
||||
|
||||
// frontmatterScalars extracts top-level "key: value" scalar pairs from
|
||||
// doc's leading fence block, for the claim-envelope check only — not a
|
||||
// general frontmatter parser. Indented lines (YAML block-list items,
|
||||
// wrapped values) are skipped rather than misparsed as keys, same fix as
|
||||
// hyperguild#126/#128 in the sibling packages this mirrors.
|
||||
func frontmatterScalars(doc string) map[string]string {
|
||||
out := map[string]string{}
|
||||
block, ok := frontmatterBlock(doc)
|
||||
if !ok {
|
||||
return out
|
||||
}
|
||||
lines := strings.Split(block, "\n")
|
||||
for i := 0; i < len(lines); i++ {
|
||||
line := lines[i]
|
||||
if line != "" && (line[0] == ' ' || line[0] == '\t') {
|
||||
continue
|
||||
}
|
||||
key, val, ok := strings.Cut(line, ":")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
val = strings.TrimSpace(val)
|
||||
if val == "" {
|
||||
// Round 6: a colon with nothing after it is ambiguous — it can
|
||||
// be a genuinely empty/omitted value, or the start of a YAML
|
||||
// block scalar whose real value lives on the indented
|
||||
// continuation line(s) that follow (e.g. "half_life:\n
|
||||
// durable"). Treating the latter as absent let an edit smuggle
|
||||
// an invalid value straight past ValidateEnvelope: the field
|
||||
// read as unset while the disk actually carried a value. Fold
|
||||
// the continuation in as a best-effort join so oneOf() still
|
||||
// sees it. A genuinely empty value (no indented line follows)
|
||||
// still folds to "", preserving present()'s existing
|
||||
// placeholder-not-assertion contract.
|
||||
var cont []string
|
||||
for i+1 < len(lines) {
|
||||
next := lines[i+1]
|
||||
if next == "" || (next[0] != ' ' && next[0] != '\t') {
|
||||
break
|
||||
}
|
||||
cont = append(cont, strings.TrimSpace(next))
|
||||
i++
|
||||
}
|
||||
val = strings.Join(cont, " ")
|
||||
}
|
||||
out[strings.TrimSpace(key)] = val
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
@@ -70,6 +70,266 @@ func TestBrainReadAndEdit_RoundTripThroughJSONRPC(t *testing.T) {
|
||||
"only the matched bytes may change")
|
||||
}
|
||||
|
||||
func TestBrainEdit_EmptyOldStrErrorNamesRawArgumentBytes(t *testing.T) {
|
||||
// hyperguild#115: a session reported "old_str is empty" with an old_str
|
||||
// that, on their end, was never empty. The server-side path was
|
||||
// unreachable to reproduce (verified separately: a fresh brain_edit call
|
||||
// with real content round-trips cleanly), which points at something
|
||||
// between a specific client and this call losing the field. The fix this
|
||||
// issue actually asked for isn't a behavior change — old_str=="" must
|
||||
// still refuse — it's that the refusal should say how many raw argument
|
||||
// bytes it received, so "client sent empty" is distinguishable from
|
||||
// "server received empty after some hop lost it" without a second
|
||||
// session's worth of guessing.
|
||||
dir := newBrainRepo(t)
|
||||
srv := mcp.NewServer(dir, nil, nil, nil)
|
||||
|
||||
resp := call(t, srv, "brain_edit", map[string]any{
|
||||
"path": "wiki/homelab/facts/n.md", "old_str": "", "new_str": "BETA",
|
||||
})
|
||||
errMsg, ok := resp["error"].(map[string]any)["message"].(string)
|
||||
require.True(t, ok, "%v", resp)
|
||||
assert.Contains(t, errMsg, "old_str is empty")
|
||||
assert.Contains(t, errMsg, "raw arguments were", "%v", errMsg)
|
||||
}
|
||||
|
||||
func TestBrainEdit_RejectsAnInvalidClaimEnvelopeWrittenViaStringReplace(t *testing.T) {
|
||||
// hyperguild#123: brain_write/brain_update both validate the claim
|
||||
// envelope (half_life/applies/claim_provenance) before writing —
|
||||
// brain_edit didn't, even though it's the one live, model-facing path
|
||||
// that can rewrite an existing envelope line directly via raw string
|
||||
// replace, no parsing involved. "same rules when a note changes as when
|
||||
// it's born" (hyperguild#111's own framing) applies here too.
|
||||
dir := newBrainRepo(t)
|
||||
note := filepath.Join(dir, "wiki/homelab/facts/n.md")
|
||||
require.NoError(t, os.WriteFile(note,
|
||||
[]byte("---\ntitle: 'n'\nhalf_life: durable\napplies: universal\nclaim_provenance: asserted\n---\n\nbody\n"), 0o644))
|
||||
srv := mcp.NewServer(dir, nil, nil, nil)
|
||||
|
||||
resp := call(t, srv, "brain_edit", map[string]any{
|
||||
"path": "wiki/homelab/facts/n.md", "old_str": "claim_provenance: asserted", "new_str": "claim_provenance: guessed",
|
||||
})
|
||||
require.Contains(t, resp, "error", "%v", resp)
|
||||
|
||||
got, err := os.ReadFile(note)
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, string(got), "claim_provenance: asserted", "rejected edit must leave the note byte-identical")
|
||||
}
|
||||
|
||||
func TestBrainEdit_RejectsAValueOnlyEditThatBreaksTheEnvelope(t *testing.T) {
|
||||
// Found in review: the original gate (touchesClaimEnvelope) checked
|
||||
// whether old_str/new_str mention the FIELD NAME — but a minimal diff
|
||||
// (the natural shape of any real edit, model-generated or human) can
|
||||
// target just the VALUE token and never mention the field name at all.
|
||||
// "asserted" -> "guessed" on a claim_provenance line walked straight
|
||||
// past the field-name substring check and wrote an invalid enum value
|
||||
// with no error — the exact bypass #123 was supposed to close, closed
|
||||
// in name only.
|
||||
dir := newBrainRepo(t)
|
||||
note := filepath.Join(dir, "wiki/homelab/facts/n.md")
|
||||
require.NoError(t, os.WriteFile(note,
|
||||
[]byte("---\ntitle: 'n'\nhalf_life: durable\napplies: universal\nclaim_provenance: asserted\n---\n\nbody\n"), 0o644))
|
||||
srv := mcp.NewServer(dir, nil, nil, nil)
|
||||
|
||||
resp := call(t, srv, "brain_edit", map[string]any{
|
||||
"path": "wiki/homelab/facts/n.md", "old_str": "asserted", "new_str": "guessed",
|
||||
})
|
||||
require.Contains(t, resp, "error", "%v", resp)
|
||||
|
||||
got, err := os.ReadFile(note)
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, string(got), "claim_provenance: asserted", "rejected edit must leave the note byte-identical")
|
||||
}
|
||||
|
||||
func TestBrainEdit_RejectsAnOldStrThatCrossesTheClosingFence(t *testing.T) {
|
||||
// Found in round-2 review: the position-based gate required old_str to
|
||||
// be fully CONTAINED in the extracted frontmatter block (which excludes
|
||||
// both "---" fence lines). An old_str disambiguated by extending past
|
||||
// the closing fence into the body — a natural choice, since notes.Edit
|
||||
// requires old_str to be unique in the file — fails that containment
|
||||
// test, so checkClaimEnvelope skips validation entirely while
|
||||
// notes.Edit still applies the replace verbatim. Same invalid-enum
|
||||
// outcome as the original two bypasses, reached a third way.
|
||||
dir := newBrainRepo(t)
|
||||
note := filepath.Join(dir, "wiki/homelab/facts/n.md")
|
||||
require.NoError(t, os.WriteFile(note,
|
||||
[]byte("---\ntitle: 'n'\nhalf_life: durable\napplies: universal\nclaim_provenance: asserted\n---\n\nbody\n"), 0o644))
|
||||
srv := mcp.NewServer(dir, nil, nil, nil)
|
||||
|
||||
resp := call(t, srv, "brain_edit", map[string]any{
|
||||
"path": "wiki/homelab/facts/n.md",
|
||||
"old_str": "asserted\n---\n\nbody",
|
||||
"new_str": "guessed\n---\n\nbody",
|
||||
})
|
||||
require.Contains(t, resp, "error", "%v", resp)
|
||||
|
||||
got, err := os.ReadFile(note)
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, string(got), "claim_provenance: asserted", "rejected edit must leave the note byte-identical")
|
||||
}
|
||||
|
||||
func TestBrainEdit_RejectsAnEditThatRemovesTheClosingFence(t *testing.T) {
|
||||
// Found in round-3 review: checkClaimEnvelope always validates the
|
||||
// resulting frontmatter of `after` — but an edit whose new_str drops
|
||||
// the closing "---" degrades `after` to "unfenced" in the same call.
|
||||
// frontmatterBlock(after) then returns ok=false, frontmatterScalars
|
||||
// returns an empty map, and claim.ValidateEnvelope({}) is nil by
|
||||
// design (absent fields are valid) — so an invalid enum value lands on
|
||||
// disk with zero error, and the note is left permanently unfenced.
|
||||
// Structural fix: a note that had a valid fence before the edit must
|
||||
// still have one after, or the edit is rejected outright.
|
||||
dir := newBrainRepo(t)
|
||||
note := filepath.Join(dir, "wiki/homelab/facts/n.md")
|
||||
require.NoError(t, os.WriteFile(note,
|
||||
[]byte("---\ntitle: 'n'\nhalf_life: durable\napplies: universal\nclaim_provenance: asserted\n---\n\nbody\n"), 0o644))
|
||||
srv := mcp.NewServer(dir, nil, nil, nil)
|
||||
|
||||
resp := call(t, srv, "brain_edit", map[string]any{
|
||||
"path": "wiki/homelab/facts/n.md",
|
||||
"old_str": "claim_provenance: asserted\n---\n",
|
||||
"new_str": "claim_provenance: not-a-real-value\n",
|
||||
})
|
||||
require.Contains(t, resp, "error", "%v", resp)
|
||||
|
||||
got, err := os.ReadFile(note)
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, string(got), "claim_provenance: asserted\n---\n", "rejected edit must leave the note byte-identical, fence intact")
|
||||
}
|
||||
|
||||
func TestBrainEdit_RejectsAnInvalidEnvelopeOnACRLFFencedNote(t *testing.T) {
|
||||
// Found in round-4 review: frontmatterBlock's fence detection matches
|
||||
// the literal byte sequence "---\n" / "\n---\n". A note whose fence
|
||||
// lines use CRLF ("---\r\n") fails strings.HasPrefix(doc, "---\n") on
|
||||
// the BEFORE read, so hadFence is false — not because the fence is
|
||||
// missing, but because \r sits in front of every \n. hadFence&&!hasFence
|
||||
// can then never fire, frontmatterScalars falls back to an empty map,
|
||||
// and claim.ValidateEnvelope({}) is nil by design. No adversarial
|
||||
// old_str/new_str crafting needed: encoding alone silently disables the
|
||||
// whole envelope check for this note. Zero CRLF-fenced notes exist in
|
||||
// the live corpus today, but nothing in the code prevents one.
|
||||
dir := newBrainRepo(t)
|
||||
note := filepath.Join(dir, "wiki/homelab/facts/n.md")
|
||||
require.NoError(t, os.WriteFile(note,
|
||||
[]byte("---\r\ntitle: 'n'\r\nhalf_life: durable\r\napplies: universal\r\nclaim_provenance: asserted\r\n---\r\n\r\nbody\r\n"), 0o644))
|
||||
srv := mcp.NewServer(dir, nil, nil, nil)
|
||||
|
||||
resp := call(t, srv, "brain_edit", map[string]any{
|
||||
"path": "wiki/homelab/facts/n.md", "old_str": "asserted", "new_str": "guessed",
|
||||
})
|
||||
require.Contains(t, resp, "error", "%v", resp)
|
||||
|
||||
got, err := os.ReadFile(note)
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, string(got), "claim_provenance: asserted", "rejected edit must leave the note byte-identical")
|
||||
}
|
||||
|
||||
func TestBrainEdit_RejectsAnInvalidEnvelopeOnABOMPrefixedNote(t *testing.T) {
|
||||
// Found in round-5 review: the same "encoding property empties
|
||||
// frontmatterScalars's map" mechanism as the CRLF bypass, one layer
|
||||
// earlier — a leading UTF-8 BOM (EF BB BF) defeats
|
||||
// strings.HasPrefix(doc, "---\n") exactly like CRLF did, before that
|
||||
// prefix is even reached. Notepad's default "UTF-8" save and
|
||||
// PowerShell's pre-v6 Out-File/Set-Content both emit a BOM, so this is
|
||||
// a live, plausible authoring path, not an exotic one — same severity
|
||||
// and same one-line cost as the CRLF fix, fixed the same way rather
|
||||
// than deferred to hyperguild#138.
|
||||
dir := newBrainRepo(t)
|
||||
note := filepath.Join(dir, "wiki/homelab/facts/n.md")
|
||||
require.NoError(t, os.WriteFile(note,
|
||||
[]byte("\xef\xbb\xbf---\ntitle: 'n'\nhalf_life: durable\napplies: universal\nclaim_provenance: asserted\n---\n\nbody\n"), 0o644))
|
||||
srv := mcp.NewServer(dir, nil, nil, nil)
|
||||
|
||||
resp := call(t, srv, "brain_edit", map[string]any{
|
||||
"path": "wiki/homelab/facts/n.md", "old_str": "asserted", "new_str": "guessed",
|
||||
})
|
||||
require.Contains(t, resp, "error", "%v", resp)
|
||||
|
||||
got, err := os.ReadFile(note)
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, string(got), "claim_provenance: asserted", "rejected edit must leave the note byte-identical")
|
||||
}
|
||||
|
||||
func TestBrainEdit_RejectsAFoldedScalarThatHidesAnInvalidValue(t *testing.T) {
|
||||
// Found in round-6 review: a different mechanism from the encoding
|
||||
// family (rounds 1-6 were all about fence detection; this is the field
|
||||
// extractor). old_str="half_life: durable", new_str="half_life:\n
|
||||
// garbage-value" turns the field into a YAML block-scalar shape: the
|
||||
// top-level line is now "half_life:" with an empty inline value, and
|
||||
// the real value lives on the indented continuation line that follows.
|
||||
// frontmatterScalars's indentation guard skips that continuation line
|
||||
// outright, so the field reads as present-with-empty-value —
|
||||
// present()'s own contract then treats that as absent (a placeholder,
|
||||
// not an assertion) — and claim.ValidateEnvelope never sees the
|
||||
// invalid value at all. No encoding quirk needed: one ordinary
|
||||
// brain_edit call smuggles an out-of-enum value straight past the
|
||||
// check the field was supposed to be validated by.
|
||||
dir := newBrainRepo(t)
|
||||
note := filepath.Join(dir, "wiki/homelab/facts/n.md")
|
||||
require.NoError(t, os.WriteFile(note,
|
||||
[]byte("---\ntitle: 'n'\nhalf_life: durable\napplies: universal\nclaim_provenance: asserted\n---\n\nbody\n"), 0o644))
|
||||
srv := mcp.NewServer(dir, nil, nil, nil)
|
||||
|
||||
resp := call(t, srv, "brain_edit", map[string]any{
|
||||
"path": "wiki/homelab/facts/n.md",
|
||||
"old_str": "half_life: durable",
|
||||
"new_str": "half_life:\n garbage-value",
|
||||
})
|
||||
require.Contains(t, resp, "error", "%v", resp)
|
||||
|
||||
got, err := os.ReadFile(note)
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, string(got), "half_life: durable", "rejected edit must leave the note byte-identical")
|
||||
}
|
||||
|
||||
func TestBrainEdit_AllowsAValidClaimEnvelopeEdit(t *testing.T) {
|
||||
dir := newBrainRepo(t)
|
||||
note := filepath.Join(dir, "wiki/homelab/facts/n.md")
|
||||
require.NoError(t, os.WriteFile(note,
|
||||
[]byte("---\ntitle: 'n'\nhalf_life: durable\napplies: universal\nclaim_provenance: asserted\n---\n\nbody\n"), 0o644))
|
||||
srv := mcp.NewServer(dir, nil, nil, nil)
|
||||
|
||||
resp := call(t, srv, "brain_edit", map[string]any{
|
||||
"path": "wiki/homelab/facts/n.md", "old_str": "half_life: durable", "new_str": "half_life: seasonal",
|
||||
})
|
||||
require.NotContains(t, resp, "error", "%v", resp)
|
||||
|
||||
got, err := os.ReadFile(note)
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, string(got), "half_life: seasonal")
|
||||
}
|
||||
|
||||
func TestBrainEdit_UnrelatedEditsSkipTheEnvelopeCheckEntirely(t *testing.T) {
|
||||
// The gate only engages when old_str/new_str actually mention an
|
||||
// envelope field name — an ordinary body edit on a note that happens to
|
||||
// carry a valid envelope must not pay for (or be affected by) a check
|
||||
// that has nothing to do with it.
|
||||
dir := newBrainRepo(t)
|
||||
note := filepath.Join(dir, "wiki/homelab/facts/n.md")
|
||||
require.NoError(t, os.WriteFile(note,
|
||||
[]byte("---\ntitle: 'n'\nhalf_life: durable\napplies: universal\nclaim_provenance: asserted\n---\n\nold body\n"), 0o644))
|
||||
srv := mcp.NewServer(dir, nil, nil, nil)
|
||||
|
||||
resp := call(t, srv, "brain_edit", map[string]any{
|
||||
"path": "wiki/homelab/facts/n.md", "old_str": "old body", "new_str": "new body",
|
||||
})
|
||||
require.NotContains(t, resp, "error", "%v", resp)
|
||||
|
||||
got, err := os.ReadFile(note)
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, string(got), "new body")
|
||||
}
|
||||
|
||||
func TestBrainEditRoot_GovernanceFilesAreNeverEnvelopeChecked(t *testing.T) {
|
||||
// CLAUDE.md has no claim-envelope concept at all — the check must be
|
||||
// scoped to notes (ScopeNotes), never applied on the governance path.
|
||||
srv, _, _, _ := govServer(t)
|
||||
|
||||
resp := authCall(t, srv, "Bearer "+capStaticTok, "brain_edit_root", map[string]any{
|
||||
"path": "CLAUDE.md", "old_str": "# rules", "new_str": "half_life: durable",
|
||||
})
|
||||
require.NotContains(t, resp, "error", "%v", resp)
|
||||
}
|
||||
|
||||
func TestBrainEdit_RefusesGovernanceFilesFromTheModelFacingTool(t *testing.T) {
|
||||
dir := newBrainRepo(t)
|
||||
srv := mcp.NewServer(dir, nil, nil, nil)
|
||||
|
||||
@@ -76,6 +76,9 @@ func buildFrontmatter(rp RawPage, date string) string {
|
||||
}
|
||||
fmt.Fprintf(&sb, "date_ingested: %s\n", date)
|
||||
fmt.Fprintf(&sb, "last_updated: %s\n", date)
|
||||
if rp.SourceHash != "" {
|
||||
fmt.Fprintf(&sb, "source_hash: %s\n", rp.SourceHash)
|
||||
}
|
||||
if rp.Source != "" {
|
||||
fmt.Fprintf(&sb, "source: %s\n", yamlScalar(rp.Source))
|
||||
}
|
||||
|
||||
@@ -20,9 +20,13 @@ type RawPage struct {
|
||||
// ingested content's own frontmatter (see parseContentFrontmatter) — never
|
||||
// set by the LLM. json:"-" keeps them immune to same-named keys the LLM
|
||||
// might emit. Only meaningful for Type == "source".
|
||||
Source string `json:"-"`
|
||||
Author string `json:"-"`
|
||||
Published string `json:"-"`
|
||||
// SourceHash is sha256 of the ingested BODY (frontmatter excluded), stamped
|
||||
// so the next ingest of the same source can tell "unchanged" from "edited"
|
||||
// without re-running extraction (#105).
|
||||
SourceHash string `json:"-"`
|
||||
Source string `json:"-"`
|
||||
Author string `json:"-"`
|
||||
Published string `json:"-"`
|
||||
}
|
||||
|
||||
// ParseRawPages parses LLM output as a JSON array of RawPage objects.
|
||||
@@ -112,6 +116,11 @@ type sourceMeta struct {
|
||||
Source string
|
||||
Author string
|
||||
Published string
|
||||
// BodyHash is sha256 of the content with any leading frontmatter removed.
|
||||
// Body-only on purpose: a re-crawl that stamps a new fetch date has not
|
||||
// changed the source, and treating that as drift would re-extract the
|
||||
// whole corpus every time a crawler ran (#105).
|
||||
BodyHash string
|
||||
}
|
||||
|
||||
// parseContentFrontmatter extracts source/author/published from a leading
|
||||
@@ -154,6 +163,7 @@ func applySourceMeta(pages []RawPage, meta sourceMeta) {
|
||||
if pages[i].Type != "source" {
|
||||
continue
|
||||
}
|
||||
pages[i].SourceHash = meta.BodyHash
|
||||
pages[i].Source = meta.Source
|
||||
pages[i].Author = meta.Author
|
||||
pages[i].Published = meta.Published
|
||||
|
||||
@@ -3,6 +3,8 @@ package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -26,6 +28,11 @@ type Config struct {
|
||||
type Result struct {
|
||||
Pages []string // relative paths written (or would-be written in dry-run)
|
||||
Warnings []string
|
||||
// Skipped reports that the source was byte-identical to the last ingest,
|
||||
// so no extraction ran and nothing was written. Reported rather than
|
||||
// silent: a run that does nothing must say so, or "the pipeline is fine"
|
||||
// and "the pipeline stopped working" look the same (#105, #107).
|
||||
Skipped bool
|
||||
}
|
||||
|
||||
// Run ingests content and writes structured wiki pages to brainDir/wiki/.
|
||||
@@ -43,13 +50,32 @@ func Run(ctx context.Context, cfg Config, brainDir, content, source string, dryR
|
||||
|
||||
sourceSlug := wiki.Slug(source)
|
||||
date := time.Now().UTC().Format("2006-01-02")
|
||||
|
||||
// Drift check BEFORE any extraction: an unchanged source costs a full LLM
|
||||
// pass over every chunk to produce a result that merges to nothing (#105).
|
||||
// The merge is already idempotent (#103), so this is cost and signal, not
|
||||
// correctness — which is why it is safe for it to be an early return.
|
||||
bodyHash := hashBody(content)
|
||||
if !dryRun && bodyHash != "" && bodyHash == storedSourceHash(brainDir, sourceSlug) {
|
||||
return Result{Skipped: true, Warnings: []string{
|
||||
fmt.Sprintf("source %q unchanged since last ingest (hash %s) — extraction skipped", source, bodyHash[:12]),
|
||||
}}, nil
|
||||
}
|
||||
|
||||
// The source's own page path is deterministic (sourceSlug, computed above)
|
||||
// even before extraction runs, unlike a concept/entity page's path, which
|
||||
// only exists once the model has produced a title. So page-aware
|
||||
// extraction (#106) is scoped to the source page: it's the one page every
|
||||
// chunk's prompt can be given in advance.
|
||||
existingPage := readSourcePage(brainDir, sourceSlug)
|
||||
|
||||
chunks := Chunk(content, cfg.ChunkSize)
|
||||
|
||||
var allRaw []RawPage
|
||||
var allWarnings []string
|
||||
|
||||
for _, chunk := range chunks {
|
||||
userPrompt := BuildPrompt(schema, source, chunk, inventory)
|
||||
userPrompt := BuildPrompt(schema, source, chunk, inventory, existingPage)
|
||||
output, err := cfg.Complete(ctx, systemPrompt, userPrompt)
|
||||
if err != nil {
|
||||
return Result{}, fmt.Errorf("LLM call: %w", err)
|
||||
@@ -59,7 +85,9 @@ func Run(ctx context.Context, cfg Config, brainDir, content, source string, dryR
|
||||
allWarnings = append(allWarnings, warnings...)
|
||||
}
|
||||
|
||||
applySourceMeta(allRaw, parseContentFrontmatter(content))
|
||||
meta := parseContentFrontmatter(content)
|
||||
meta.BodyHash = bodyHash
|
||||
applySourceMeta(allRaw, meta)
|
||||
|
||||
return buildAndWrite(allRaw, sourceSlug, date, brainDir, source, inventory, allWarnings, dryRun)
|
||||
}
|
||||
@@ -96,7 +124,19 @@ func buildAndWrite(rawPages []RawPage, sourceSlug, date, brainDir, source string
|
||||
if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
|
||||
return Result{}, fmt.Errorf("mkdir for %s: %w", page.Path, err)
|
||||
}
|
||||
if err := os.WriteFile(dest, []byte(page.Content), 0o644); err != nil {
|
||||
content, conflicts, err := reviseOrCreate(dest, page)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
for _, heading := range conflicts {
|
||||
// Reported, not resolved. The on-disk version stands; whether
|
||||
// the disagreement is a genuine contradiction is the
|
||||
// Judge-gated enrich pass's call, not the write path's.
|
||||
warnings = append(warnings, fmt.Sprintf(
|
||||
"%s: source now disagrees on section %q — on-disk version kept",
|
||||
page.Path, heading))
|
||||
}
|
||||
if err := os.WriteFile(dest, []byte(content), 0o644); err != nil {
|
||||
return Result{}, fmt.Errorf("write %s: %w", page.Path, err)
|
||||
}
|
||||
}
|
||||
@@ -115,6 +155,82 @@ func buildAndWrite(rawPages []RawPage, sourceSlug, date, brainDir, source string
|
||||
return Result{Pages: written, Warnings: warnings}, nil
|
||||
}
|
||||
|
||||
// hashBody returns sha256 of content with any leading frontmatter block
|
||||
// removed, or "" for empty content.
|
||||
//
|
||||
// Body-only on purpose: a re-crawl that stamps a new fetch date has not changed
|
||||
// the source, and treating that as drift would re-extract the corpus every time
|
||||
// a crawler ran (#105).
|
||||
func hashBody(content string) string {
|
||||
body := content
|
||||
if strings.HasPrefix(body, "---\n") {
|
||||
if _, rest, found := strings.Cut(body[4:], "\n---"); found {
|
||||
body = strings.TrimPrefix(rest, "\n")
|
||||
}
|
||||
}
|
||||
if strings.TrimSpace(body) == "" {
|
||||
return ""
|
||||
}
|
||||
sum := sha256.Sum256([]byte(body))
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
// readSourcePage returns the full on-disk content of the source page at
|
||||
// wiki/sources/<sourceSlug>.md, or "" when it doesn't exist yet.
|
||||
func readSourcePage(brainDir, sourceSlug string) string {
|
||||
raw, err := os.ReadFile(filepath.Join(brainDir, "wiki", "sources", sourceSlug+".md"))
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return string(raw)
|
||||
}
|
||||
|
||||
// storedSourceHash reads source_hash from the source page a previous ingest
|
||||
// wrote, or "" when there is no such page or no such field.
|
||||
//
|
||||
// Absence always means "extract". That is the fail-safe direction: a missed
|
||||
// skip costs one extraction, while a wrong skip loses an update silently — and
|
||||
// silence is the failure mode this repo keeps paying for.
|
||||
func storedSourceHash(brainDir, sourceSlug string) string {
|
||||
for _, line := range strings.Split(readSourcePage(brainDir, sourceSlug), "\n") {
|
||||
if key, value, found := strings.Cut(line, ":"); found && strings.TrimSpace(key) == "source_hash" {
|
||||
return strings.TrimSpace(value)
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// reviseOrCreate returns the bytes to write for one extracted page: merged into
|
||||
// whatever is already on disk, or the page as-is when the path is new.
|
||||
//
|
||||
// This is the compile half of compile-on-ingest. Before it, the write was an
|
||||
// unconditional os.WriteFile, so re-ingesting a source replaced its page
|
||||
// wholesale and discarded everything accumulated since — including human edits
|
||||
// (hyperguild#43: "re-ingest overwrites wholesale").
|
||||
//
|
||||
// The merge is deliberately asymmetric in the existing page's favour:
|
||||
// wiki.Merge keeps the on-disk frontmatter and the on-disk version of any
|
||||
// section both sides carry, adds sections only the extraction produced, and
|
||||
// unions bullet lists. So a freshly extracted page can add to a note but never
|
||||
// silently rewrite what a human put there — which is what makes it safe to run
|
||||
// this on every ingest rather than behind a review gate.
|
||||
//
|
||||
// Known limitation: because the on-disk frontmatter wins, `last_updated` keeps
|
||||
// its first-ingest date on a revised page. Left deliberately — the alternative
|
||||
// is this function editing frontmatter it does not own, and no lint reads that
|
||||
// field today.
|
||||
func reviseOrCreate(dest string, page wiki.Page) (string, []string, error) {
|
||||
existing, err := os.ReadFile(dest)
|
||||
switch {
|
||||
case os.IsNotExist(err):
|
||||
return page.Content, nil, nil
|
||||
case err != nil:
|
||||
return "", nil, fmt.Errorf("read existing %s: %w", page.Path, err)
|
||||
}
|
||||
merged, conflicts := wiki.MergeReport(wiki.Page{Path: page.Path, Content: string(existing)}, page)
|
||||
return merged.Content, conflicts, nil
|
||||
}
|
||||
|
||||
// mergeAll deduplicates pages by path, merging content from later occurrences.
|
||||
func mergeAll(pages []wiki.Page) []wiki.Page {
|
||||
order := make([]string, 0, len(pages))
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -171,3 +172,254 @@ func mustJSON(v any) string {
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// --- B1: compile-on-ingest — revise the existing page, never clobber it -----
|
||||
|
||||
func TestRunRaw_MergesIntoAnExistingPageInsteadOfOverwritingIt(t *testing.T) {
|
||||
// Before this, buildAndWrite did an unconditional os.WriteFile: re-ingesting
|
||||
// a source replaced its page wholesale, discarding anything accumulated
|
||||
// since — including human edits. hyperguild#43 recorded the symptom as
|
||||
// "re-ingest overwrites wholesale".
|
||||
dir := t.TempDir()
|
||||
page := filepath.Join(dir, "wiki", "sources", "shape-up.md")
|
||||
require.NoError(t, os.MkdirAll(filepath.Dir(page), 0o755))
|
||||
require.NoError(t, os.WriteFile(page, []byte(`---
|
||||
title: 'Shape Up'
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
A book about shaping product work.
|
||||
|
||||
## My Take
|
||||
|
||||
Read this before the next planning cycle.
|
||||
`), 0o644))
|
||||
|
||||
_, err := RunRaw(dir, "shape-up", []RawPage{{
|
||||
Title: "Shape Up", Type: "source", Subtype: "book", Domain: "product-strategy",
|
||||
Content: "## Summary\n\nA book about shaping product work.\n\n## Key Claims\n\n- Appetite beats estimation.\n",
|
||||
}}, false)
|
||||
require.NoError(t, err)
|
||||
|
||||
got, err := os.ReadFile(page)
|
||||
require.NoError(t, err)
|
||||
s := string(got)
|
||||
|
||||
assert.Contains(t, s, "## My Take", "a section the extraction never saw must survive the merge")
|
||||
assert.Contains(t, s, "Read this before the next planning cycle.")
|
||||
assert.Contains(t, s, "Appetite beats estimation.", "newly extracted material must still land")
|
||||
}
|
||||
|
||||
func TestRunRaw_ReIngestingUnchangedContentIsByteIdentical(t *testing.T) {
|
||||
// Compile-on-ingest runs on every ingest, so "nothing changed" must produce
|
||||
// no diff at all. Otherwise every re-ingest dirties the tree, brain-sync
|
||||
// commits it, and the git history fills with noise that hides real edits.
|
||||
dir := t.TempDir()
|
||||
raw := []RawPage{{
|
||||
Title: "Shape Up", Type: "source", Subtype: "book", Domain: "product-strategy",
|
||||
Content: "## Summary\n\nA book about shaping product work.\n\n## Open Questions\n\n- Does appetite scale?\n",
|
||||
}}
|
||||
|
||||
_, err := RunRaw(dir, "shape-up", raw, false)
|
||||
require.NoError(t, err)
|
||||
first, err := os.ReadFile(filepath.Join(dir, "wiki", "sources", "shape-up.md"))
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = RunRaw(dir, "shape-up", raw, false)
|
||||
require.NoError(t, err)
|
||||
second, err := os.ReadFile(filepath.Join(dir, "wiki", "sources", "shape-up.md"))
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Equal(t, string(first), string(second),
|
||||
"re-ingesting unchanged content must leave the page byte-identical")
|
||||
}
|
||||
|
||||
// --- #105: source drift — an unchanged re-ingest must not re-extract --------
|
||||
|
||||
func TestRun_SkipsExtractionWhenTheSourceIsUnchanged(t *testing.T) {
|
||||
// #103 made a re-ingest harmless (byte-identical merge). It is still not
|
||||
// free: the full LLM extraction runs over every chunk to produce a result
|
||||
// that merges to nothing. This is the cost half.
|
||||
dir := t.TempDir()
|
||||
var calls int
|
||||
cfg := Config{Complete: func(_ context.Context, _, _ string) (string, error) {
|
||||
calls++
|
||||
return `[{"title":"Shape Up","type":"source","subtype":"book","domain":"product","content":"## Summary\n\nA book.\n"}]`, nil
|
||||
}}
|
||||
|
||||
content := "---\nsource: https://example.com/shape-up\n---\n\nThe body of the article.\n"
|
||||
|
||||
first, err := Run(context.Background(), cfg, dir, content, "shape-up", false)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, 1, calls, "first ingest must extract")
|
||||
assert.False(t, first.Skipped)
|
||||
|
||||
second, err := Run(context.Background(), cfg, dir, content, "shape-up", false)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 1, calls, "unchanged content must not be extracted again")
|
||||
assert.True(t, second.Skipped, "the skip must be reported, not silent")
|
||||
}
|
||||
|
||||
func TestRun_ReExtractsWhenTheSourceBodyChanged(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
var calls int
|
||||
cfg := Config{Complete: func(_ context.Context, _, _ string) (string, error) {
|
||||
calls++
|
||||
return `[{"title":"Shape Up","type":"source","subtype":"book","domain":"product","content":"## Summary\n\nA book.\n"}]`, nil
|
||||
}}
|
||||
|
||||
_, err := Run(context.Background(), cfg, dir, "---\nsource: x\n---\n\nversion one\n", "shape-up", false)
|
||||
require.NoError(t, err)
|
||||
_, err = Run(context.Background(), cfg, dir, "---\nsource: x\n---\n\nversion two — the source was edited\n", "shape-up", false)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Equal(t, 2, calls, "changed content must be re-extracted")
|
||||
}
|
||||
|
||||
// --- #106: page-aware extraction — the model sees the page it is revising --
|
||||
|
||||
func TestRun_ExistingSourcePageIsIncludedInTheExtractionPrompt(t *testing.T) {
|
||||
// #103 made the merge asymmetric (on-disk wins); #106 is the other half —
|
||||
// the extraction itself gets to see what's already there instead of
|
||||
// producing a page blind and letting the merge reconcile it afterwards.
|
||||
// This only proves the existing page reaches the prompt; the byte-identical
|
||||
// guarantee for an echoed-back revision is a separate test below.
|
||||
dir := t.TempDir()
|
||||
for _, sub := range []string{"wiki/concepts", "wiki/entities", "wiki/sources"} {
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(dir, sub), 0o755))
|
||||
}
|
||||
|
||||
seeded := "---\ntitle: 'Shape Up'\n---\n\n## Summary\n\nA book about shaping product work.\n\n## Key Claims\n\n- Appetite beats estimation.\n"
|
||||
page := filepath.Join(dir, "wiki", "sources", "shape-up.md")
|
||||
require.NoError(t, os.WriteFile(page, []byte(seeded), 0o644))
|
||||
|
||||
var gotPrompt string
|
||||
cfg := Config{Complete: func(_ context.Context, _, user string) (string, error) {
|
||||
gotPrompt = user
|
||||
return mustJSON([]RawPage{{
|
||||
Title: "Shape Up", Type: "source", Subtype: "book",
|
||||
Content: "## Summary\n\nA book about shaping product work.\n",
|
||||
}}), nil
|
||||
}}
|
||||
|
||||
// Deliberately does not mention "Appetite beats estimation." — that phrase
|
||||
// can only reach the prompt via the existing on-disk page, not the source
|
||||
// content, so its presence proves the page was threaded through.
|
||||
content := "A short editorial update to the book's listing. No new claims here."
|
||||
_, err := Run(context.Background(), cfg, dir, content, "shape-up", false)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Contains(t, gotPrompt, "Appetite beats estimation.",
|
||||
"the existing page's content must be visible to the extraction prompt")
|
||||
assert.Contains(t, gotPrompt, "revis", "the prompt must ask for a revision, not a fresh page")
|
||||
}
|
||||
|
||||
func TestRun_NewSourceHasNoExistingPageInThePrompt(t *testing.T) {
|
||||
// The other side of the same behavior: a source with no page on disk yet
|
||||
// must not get an "existing page" section injected into its prompt at all.
|
||||
dir := t.TempDir()
|
||||
for _, sub := range []string{"wiki/concepts", "wiki/entities", "wiki/sources"} {
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(dir, sub), 0o755))
|
||||
}
|
||||
|
||||
var gotPrompt string
|
||||
cfg := Config{Complete: func(_ context.Context, _, user string) (string, error) {
|
||||
gotPrompt = user
|
||||
return mustJSON([]RawPage{{
|
||||
Title: "New Thing", Type: "source", Subtype: "article",
|
||||
Content: "## Summary\n\nBrand new.\n",
|
||||
}}), nil
|
||||
}}
|
||||
|
||||
_, err := Run(context.Background(), cfg, dir, "Brand new content.", "new-thing", false)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.NotContains(t, gotPrompt, "Current page",
|
||||
"a source with no page on disk yet must not get a revision section in its prompt")
|
||||
}
|
||||
|
||||
func TestRun_RevisionEchoingExistingPageStaysByteIdentical(t *testing.T) {
|
||||
// The issue's explicit risk: a model given the current page will sometimes
|
||||
// echo it back verbatim. That must not duplicate content or otherwise
|
||||
// perturb the page — the deterministic merge is still the backstop.
|
||||
dir := t.TempDir()
|
||||
for _, sub := range []string{"wiki/concepts", "wiki/entities", "wiki/sources"} {
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(dir, sub), 0o755))
|
||||
}
|
||||
|
||||
seeded := "---\ntitle: 'Shape Up'\n---\n\n## Summary\n\nA book about shaping product work.\n\n## Key Claims\n\n- Appetite beats estimation.\n"
|
||||
page := filepath.Join(dir, "wiki", "sources", "shape-up.md")
|
||||
require.NoError(t, os.WriteFile(page, []byte(seeded), 0o644))
|
||||
|
||||
cfg := Config{Complete: func(_ context.Context, _, _ string) (string, error) {
|
||||
return mustJSON([]RawPage{{
|
||||
Title: "Shape Up", Type: "source", Subtype: "book",
|
||||
Content: "## Summary\n\nA book about shaping product work.\n\n## Key Claims\n\n- Appetite beats estimation.\n",
|
||||
}}), nil
|
||||
}}
|
||||
|
||||
content := "A short editorial update to the book's listing. No new claims here."
|
||||
result, err := Run(context.Background(), cfg, dir, content, "shape-up", false)
|
||||
require.NoError(t, err)
|
||||
require.False(t, result.Skipped)
|
||||
|
||||
got, err := os.ReadFile(page)
|
||||
require.NoError(t, err)
|
||||
|
||||
want := "---\ntitle: 'Shape Up'\nlast_updated: " + time.Now().UTC().Format("2006-01-02") +
|
||||
"\nsource_hash: " + hashBody(content) + "\n---\n\n" +
|
||||
"## Summary\n\nA book about shaping product work.\n\n" +
|
||||
"## Key Claims\n\n- Appetite beats estimation.\n"
|
||||
assert.Equal(t, want, string(got),
|
||||
"a revision pass that echoes the existing page back must not duplicate or alter its content")
|
||||
}
|
||||
|
||||
func TestRun_FrontmatterOnlyChangeIsNotDrift(t *testing.T) {
|
||||
// The hash covers the body, so re-fetching a page whose metadata moved
|
||||
// (a re-crawl stamping a new date) is not treated as the source changing.
|
||||
dir := t.TempDir()
|
||||
var calls int
|
||||
cfg := Config{Complete: func(_ context.Context, _, _ string) (string, error) {
|
||||
calls++
|
||||
return `[{"title":"Shape Up","type":"source","subtype":"book","domain":"product","content":"## Summary\n\nA book.\n"}]`, nil
|
||||
}}
|
||||
|
||||
_, err := Run(context.Background(), cfg, dir, "---\nsource: x\nfetched: monday\n---\n\nsame body\n", "shape-up", false)
|
||||
require.NoError(t, err)
|
||||
_, err = Run(context.Background(), cfg, dir, "---\nsource: x\nfetched: tuesday\n---\n\nsame body\n", "shape-up", false)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Equal(t, 1, calls, "only the body counts as the source")
|
||||
}
|
||||
|
||||
func TestRunRaw_ReportsWhenTheSourceDisagreesWithWhatIsOnDisk(t *testing.T) {
|
||||
// The on-disk version still wins — that is what makes unattended ingest
|
||||
// safe. But the run must say that the source now claims something else,
|
||||
// or a changed source is indistinguishable from an unchanged one.
|
||||
dir := t.TempDir()
|
||||
page := filepath.Join(dir, "wiki", "sources", "shape-up.md")
|
||||
require.NoError(t, os.MkdirAll(filepath.Dir(page), 0o755))
|
||||
require.NoError(t, os.WriteFile(page, []byte(`---
|
||||
title: 'Shape Up'
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Appetite is fixed, scope is variable.
|
||||
`), 0o644))
|
||||
|
||||
res, err := RunRaw(dir, "shape-up", []RawPage{{
|
||||
Title: "Shape Up", Type: "source", Subtype: "book", Domain: "product",
|
||||
Content: "## Summary\n\nAppetite is negotiable and scope is fixed.\n",
|
||||
}}, false)
|
||||
require.NoError(t, err)
|
||||
|
||||
got, err := os.ReadFile(page)
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, string(got), "Appetite is fixed", "on-disk version wins")
|
||||
|
||||
joined := strings.Join(res.Warnings, " ")
|
||||
assert.Contains(t, joined, "Summary", "the disagreeing section must be named")
|
||||
assert.Contains(t, joined, "shape-up", "and the page it happened on")
|
||||
}
|
||||
|
||||
@@ -22,8 +22,18 @@ Each element must have exactly these fields:
|
||||
Wikilinks in content: [[Display Name]] — just the display name, no slug, no pipe separator.
|
||||
Only link to pages listed in the inventory or pages you are creating in this response.`
|
||||
|
||||
// BuildPrompt constructs the user prompt for a single chunk.
|
||||
func BuildPrompt(schema, source, content string, inventory map[wiki.PageType][]wiki.Entry) string {
|
||||
// BuildPrompt constructs the user prompt for a single chunk. existingPage is
|
||||
// the full current content (frontmatter + body) of the source's own wiki page
|
||||
// when one already exists on disk, or "" for a source being ingested for the
|
||||
// first time. When non-empty, the model is asked to revise that page rather
|
||||
// than write a fresh one — see hyperguild#106. The deterministic merge in
|
||||
// wiki.Merge remains the backstop regardless of what the model returns: an
|
||||
// on-disk section the model doesn't touch survives untouched, and a section
|
||||
// it does touch only takes effect where Merge's strategy allows the fresh
|
||||
// side to change anything (bullet union, append) — never for an ordinary
|
||||
// section, so the model proposing a revision is never how a page gets
|
||||
// overwritten.
|
||||
func BuildPrompt(schema, source, content string, inventory map[wiki.PageType][]wiki.Entry, existingPage string) string {
|
||||
var sb strings.Builder
|
||||
|
||||
fmt.Fprintf(&sb, "Today's date is %s.\n\n", time.Now().UTC().Format("2006-01-02"))
|
||||
@@ -49,6 +59,18 @@ func BuildPrompt(schema, source, content string, inventory map[wiki.PageType][]w
|
||||
sb.WriteString("\n")
|
||||
}
|
||||
|
||||
if existingPage != "" {
|
||||
sb.WriteString("## Current page for this source — revise it, don't restart\n\n")
|
||||
sb.WriteString("This source has already been ingested and has the wiki page shown below.\n")
|
||||
sb.WriteString("You are revising that page, not writing a fresh one:\n")
|
||||
sb.WriteString(" - Add sections or claims the source below supports that this page doesn't have yet.\n")
|
||||
sb.WriteString(" - Strengthen a claim where the source gives more detail, evidence, or nuance than what's already there.\n")
|
||||
sb.WriteString(" - Leave unchanged whatever the source doesn't affect — reuse the existing wording verbatim rather than paraphrasing it, so an unchanged claim is recognized as unchanged.\n")
|
||||
sb.WriteString(" - A merge step keeps this page's on-disk content for any section you don't return, so it's fine to omit a section you aren't revising.\n\n")
|
||||
sb.WriteString(existingPage)
|
||||
sb.WriteString("\n\n")
|
||||
}
|
||||
|
||||
sb.WriteString("## Non-negotiable rules\n\n")
|
||||
sb.WriteString("1. Output ONLY a valid JSON array — no prose, no fences.\n")
|
||||
sb.WriteString("2. Fields: title, type, subtype (if applicable), domain (if applicable), content.\n")
|
||||
|
||||
@@ -59,10 +59,16 @@ func processDir(ctx context.Context, cfg Config, date string) []error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Skip processed/ and failed/ subdirectories entirely.
|
||||
// Skip processed/, failed/, and rejected/ subdirectories entirely.
|
||||
// rejected/ (hyperguild#81, brain_reject) holds notes a human already
|
||||
// looked at and declined — walking into it would run the extraction
|
||||
// pipeline over a note and publish it to the wiki, the exact opposite
|
||||
// of the human's decision. The whole reason RejectNote moves a note
|
||||
// there instead of leaving it in raw/ was to keep it out of every
|
||||
// path that treats raw/ as "awaiting processing" — this is that path.
|
||||
if d.IsDir() {
|
||||
name := d.Name()
|
||||
if name == "processed" || name == "failed" {
|
||||
if name == "processed" || name == "failed" || name == "rejected" {
|
||||
return filepath.SkipDir
|
||||
}
|
||||
return nil
|
||||
|
||||
@@ -176,15 +176,20 @@ func TestDeriveSource(t *testing.T) {
|
||||
func TestProcessDir_SkipsSubdirs(t *testing.T) {
|
||||
brainDir := setupBrainDir(t)
|
||||
|
||||
// Create processed/ and failed/ subdirs with files inside.
|
||||
for _, sub := range []string{"processed/2026-04-22", "failed"} {
|
||||
// Create processed/, failed/, and rejected/ subdirs with files inside.
|
||||
for _, sub := range []string{"processed/2026-04-22", "failed", "rejected"} {
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(brainDir, "raw", sub), 0o755))
|
||||
}
|
||||
|
||||
processedFile := filepath.Join(brainDir, "raw", "processed", "2026-04-22", "old-file.md")
|
||||
failedFile := filepath.Join(brainDir, "raw", "failed", "broken-file.md")
|
||||
// hyperguild#81/#133: a human already declined this note via brain_reject.
|
||||
// If the watcher walks into rejected/, it re-ingests and publishes it to
|
||||
// the wiki — the exact opposite of the human's decision.
|
||||
rejectedFile := filepath.Join(brainDir, "raw", "rejected", "declined-file.md")
|
||||
require.NoError(t, os.WriteFile(processedFile, []byte("old"), 0o644))
|
||||
require.NoError(t, os.WriteFile(failedFile, []byte("broken"), 0o644))
|
||||
require.NoError(t, os.WriteFile(rejectedFile, []byte("---\nrejected_at: 2026-09-23T00:00:00Z\n---\ndeclined"), 0o644))
|
||||
|
||||
// Also place a valid file in raw/ root that should be processed.
|
||||
validFile := filepath.Join(brainDir, "raw", "valid.md")
|
||||
@@ -223,11 +228,13 @@ func TestProcessDir_SkipsSubdirs(t *testing.T) {
|
||||
// Complete should have been called exactly once (for valid.md, not for files in subdirs).
|
||||
assert.Len(t, processedSources, 1, "only the file in raw/ root should be processed")
|
||||
|
||||
// Files in processed/ and failed/ must remain untouched.
|
||||
// Files in processed/, failed/, and rejected/ must remain untouched.
|
||||
_, err := os.Stat(processedFile)
|
||||
assert.NoError(t, err, "processed subdir file should be untouched")
|
||||
_, err = os.Stat(failedFile)
|
||||
assert.NoError(t, err, "failed subdir file should be untouched")
|
||||
_, err = os.Stat(rejectedFile)
|
||||
assert.NoError(t, err, "rejected subdir file should be untouched, never re-ingested")
|
||||
}
|
||||
|
||||
// TestProcessDir_SkipsTunnelCandidateFiles guards against hyperguild#88:
|
||||
|
||||
@@ -3,6 +3,7 @@ package wiki
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
@@ -29,38 +30,279 @@ type section struct {
|
||||
}
|
||||
|
||||
// Merge combines two Page values with the same path.
|
||||
// Frontmatter is taken from a. Sections are merged by strategy:
|
||||
// bullet sections union unique lines, append sections concatenate,
|
||||
// all others keep a's version. Sections in b not present in a are appended.
|
||||
// Frontmatter is taken from a. The preamble (content before the first ##
|
||||
// heading) is taken from a — same "on-disk wins" default every unlisted
|
||||
// named section gets, kept as a separate field rather than a section with
|
||||
// an empty heading so it can never collide with one (see parseSections).
|
||||
// Sections are merged by strategy: bullet sections union unique lines,
|
||||
// append sections concatenate, all others keep a's version. Sections in b
|
||||
// not present in a are appended.
|
||||
// Merge discards any disagreement silently; prefer MergeReport where the
|
||||
// caller can act on it.
|
||||
func Merge(a, b Page) Page {
|
||||
fmA, secsA := parseSections(a.Content)
|
||||
_, secsB := parseSections(b.Content)
|
||||
page, _ := MergeReport(a, b)
|
||||
return page
|
||||
}
|
||||
|
||||
idx := make(map[string]int, len(secsA))
|
||||
// MergeReport is Merge, plus the headings of sections where b carried a
|
||||
// different version of a section a already had — and lost.
|
||||
//
|
||||
// That loss is the property which makes ingest safe to run unattended (#103):
|
||||
// a regenerating writer can add to a note but never rewrite it. But the
|
||||
// discarded text is also a signal, and until now it went in the bin: the source
|
||||
// now says something different about a section we already hold. Nobody could
|
||||
// tell "the source is unchanged" from "the source changed its mind and we
|
||||
// ignored it".
|
||||
//
|
||||
// Deliberately NOT a semantic contradiction check. This reports textual
|
||||
// disagreement, which is cheap, deterministic, and needs no model on the write
|
||||
// path. Deciding whether a disagreement is a genuine contradiction stays with
|
||||
// the Judge-gated enrich pass that already exists (brain-gardener#175), which
|
||||
// is approval-gated by design — detection here, adjudication there.
|
||||
func MergeReport(a, b Page) (Page, []string) {
|
||||
fmA, preambleA, secsA := parseSections(a.Content)
|
||||
fmB, _, secsB := parseSections(b.Content)
|
||||
|
||||
var conflicts []string
|
||||
|
||||
// Sections in secsA sharing a normalized heading (pre-existing corpus
|
||||
// drift — "## Notes" and "## NOTES" both already on disk, the exact
|
||||
// pattern this normalization exists to catch going forward, found in
|
||||
// review of PR #130/#132) are matched to b's occurrences of that same
|
||||
// heading POSITIONALLY: b's 1st occurrence of the key pairs with a's
|
||||
// 1st, b's 2nd with a's 2nd, and so on. A single "last write wins" (or
|
||||
// even "first write wins") map from key to one index would instead
|
||||
// route every one of b's occurrences to the SAME on-disk section —
|
||||
// which either freezes every duplicate but one (never receiving
|
||||
// updates again) or, worse, compares mismatched pairs and reports a
|
||||
// disagreement between two sections that were never supposed to be
|
||||
// compared, false-positiving on the plain self-merge case a revision
|
||||
// pass produces when the model echoes a page back unchanged (found in
|
||||
// review of PR #132).
|
||||
byKey := make(map[string][]int, len(secsA))
|
||||
for i, s := range secsA {
|
||||
idx[s.heading] = i
|
||||
key := normalizeHeading(s.heading)
|
||||
byKey[key] = append(byKey[key], i)
|
||||
}
|
||||
cursor := make(map[string]int, len(byKey))
|
||||
|
||||
for _, sB := range secsB {
|
||||
i, exists := idx[sB.heading]
|
||||
if !exists {
|
||||
idx[sB.heading] = len(secsA)
|
||||
key := normalizeHeading(sB.heading)
|
||||
indices := byKey[key]
|
||||
pos := cursor[key]
|
||||
cursor[key] = pos + 1
|
||||
|
||||
if pos >= len(indices) {
|
||||
// b has more occurrences of this heading than a does (including
|
||||
// the case where a has none at all) — the extra is new content,
|
||||
// appended as its own section and registered so a THIRD
|
||||
// occurrence of the same heading in b pairs with this one
|
||||
// rather than appending yet another duplicate.
|
||||
byKey[key] = append(byKey[key], len(secsA))
|
||||
secsA = append(secsA, sB)
|
||||
continue
|
||||
}
|
||||
i := indices[pos]
|
||||
sA := secsA[i]
|
||||
switch {
|
||||
case bulletSections[sB.heading]:
|
||||
case bulletSectionsNorm[key]:
|
||||
secsA[i].content = mergeBullets(sA.content, sB.content)
|
||||
case appendSections[sB.heading]:
|
||||
secsA[i].content = strings.TrimRight(sA.content, "\n") + "\n\n" + strings.TrimLeft(sB.content, "\n")
|
||||
case appendSectionsNorm[key]:
|
||||
secsA[i].content = appendNew(sA.content, sB.content)
|
||||
default:
|
||||
// a wins — but say that b disagreed, rather than dropping it mute.
|
||||
if canonicalBlock(sA.content) != canonicalBlock(sB.content) {
|
||||
conflicts = append(conflicts, sB.heading)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return Page{Path: a.Path, Content: rebuildContent(fmA, secsA)}
|
||||
return Page{Path: a.Path, Content: rebuildContent(refreshOwned(fmA, fmB), preambleA, secsA)}, conflicts
|
||||
}
|
||||
|
||||
func parseSections(markdown string) (frontmatter string, sections []section) {
|
||||
// normalizeHeading returns the key Merge uses to decide "is this the same
|
||||
// section" — trimmed and case-folded, so "Key Claims" and "Key claims" match
|
||||
// (hyperguild#106: heading casing drift must not fork a section). It is used
|
||||
// only for that comparison; the on-disk section's original heading text
|
||||
// (secsA[i].heading, untouched here) is what actually gets written out, so a
|
||||
// human's heading casing is never silently rewritten by a merge.
|
||||
func normalizeHeading(h string) string {
|
||||
return strings.ToLower(strings.TrimSpace(h))
|
||||
}
|
||||
|
||||
// bulletSectionsNorm and appendSectionsNorm are bulletSections/appendSections
|
||||
// keyed by normalizeHeading instead of exact text, so a section's merge
|
||||
// strategy (union bullets, append) is chosen by the same normalized identity
|
||||
// Merge uses to match it — otherwise a heading that matched an on-disk
|
||||
// section (case-insensitively) could still fail the exact-string strategy
|
||||
// lookup and silently fall through to "keep a's version," dropping b's
|
||||
// content instead of unioning or appending it.
|
||||
var bulletSectionsNorm = normalizeSet(bulletSections)
|
||||
var appendSectionsNorm = normalizeSet(appendSections)
|
||||
|
||||
func normalizeSet(names map[string]bool) map[string]bool {
|
||||
out := make(map[string]bool, len(names))
|
||||
for k := range names {
|
||||
out[normalizeHeading(k)] = true
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// generatorOwned names the frontmatter fields the ingest pipeline may refresh
|
||||
// on a page it is revising. Everything else on disk wins.
|
||||
//
|
||||
// #103 made the on-disk frontmatter win wholesale, which is what stops a
|
||||
// regenerating writer from overwriting human-set metadata — a title someone
|
||||
// fixed, or the claim-property envelope (brain#21). The cost was that fields
|
||||
// the generator legitimately owns froze at their first-ingest values: when it
|
||||
// last ran, and what it last saw.
|
||||
//
|
||||
// An explicit allowlist rather than a carve-out: the ownership boundary is
|
||||
// nameable, and a field nobody has thought about stays human-owned by default,
|
||||
// which is the safe direction.
|
||||
var generatorOwned = map[string]bool{
|
||||
"last_updated": true,
|
||||
"source_hash": true,
|
||||
}
|
||||
|
||||
// fmLine is one line of a frontmatter block: a parsed key/value pair, or a
|
||||
// raw line (blank, comment, or anything else that isn't a simple "key:
|
||||
// value") kept verbatim. Same shape as api.frontmatter's fmLine — this
|
||||
// package doesn't import that one (it's unexported there, and a's/b's
|
||||
// frontmatter here always arrives as a string, not a parsed struct), so the
|
||||
// small, already-proven set()-appends-if-absent behavior is reproduced
|
||||
// locally rather than string-index surgery reinvented per call site.
|
||||
type fmLine struct {
|
||||
key, value, raw string
|
||||
}
|
||||
|
||||
// parseFrontmatterBlock parses a "---\n...\n---\n" fenced block (the shape
|
||||
// parseSections produces) into an ordered line list. A block with no
|
||||
// opening/closing fence — including the empty string, a page with no
|
||||
// frontmatter at all — returns nil, ok=false: there is nothing to update in
|
||||
// place, only something to potentially create.
|
||||
func parseFrontmatterBlock(block string) (lines []fmLine, ok bool) {
|
||||
if !strings.HasPrefix(block, "---\n") {
|
||||
return nil, false
|
||||
}
|
||||
rest := block[len("---\n"):]
|
||||
end := strings.Index(rest, "\n---\n")
|
||||
if end < 0 {
|
||||
return nil, false
|
||||
}
|
||||
for _, line := range strings.Split(rest[:end], "\n") {
|
||||
// An indented line is a continuation (a YAML block-list item, a
|
||||
// wrapped value) never a top-level key, regardless of whether it
|
||||
// happens to contain a colon — " - 'Chapter 3: The Reckoning'"
|
||||
// is not a key named "- 'Chapter 3". Checked before Cut so a
|
||||
// colon inside one never reaches the key/value path at all.
|
||||
if line != "" && (line[0] == ' ' || line[0] == '\t') {
|
||||
lines = append(lines, fmLine{raw: line})
|
||||
continue
|
||||
}
|
||||
key, val, found := strings.Cut(line, ":")
|
||||
key = strings.TrimSpace(key)
|
||||
if !found || key == "" || strings.HasPrefix(strings.TrimSpace(line), "#") {
|
||||
lines = append(lines, fmLine{raw: line})
|
||||
continue
|
||||
}
|
||||
lines = append(lines, fmLine{key: key, value: strings.TrimSpace(val)})
|
||||
}
|
||||
return lines, true
|
||||
}
|
||||
|
||||
// setLine updates key's value in place if lines already has it, or appends
|
||||
// a new entry if it doesn't — the one behavior the original refreshOwned
|
||||
// got wrong (PR #126 review, round 2: it only ever updated in place).
|
||||
func setLine(lines []fmLine, key, value string) []fmLine {
|
||||
for i := range lines {
|
||||
if lines[i].key == key {
|
||||
lines[i].value = value
|
||||
return lines
|
||||
}
|
||||
}
|
||||
return append(lines, fmLine{key: key, value: value})
|
||||
}
|
||||
|
||||
func renderFrontmatterBlock(lines []fmLine) string {
|
||||
if len(lines) == 0 {
|
||||
return ""
|
||||
}
|
||||
var sb strings.Builder
|
||||
sb.WriteString("---\n")
|
||||
for _, l := range lines {
|
||||
switch {
|
||||
case l.key == "":
|
||||
sb.WriteString(l.raw)
|
||||
case l.value == "":
|
||||
// A key with nothing after the colon (the YAML block-list shape:
|
||||
// "aliases:" followed by indented "- item" lines on their own
|
||||
// raw lines) must stay bare — "key: " with a trailing space
|
||||
// is a different, if invisible, byte sequence, and this file's
|
||||
// whole purpose is byte-stability across an unrelated merge.
|
||||
sb.WriteString(l.key)
|
||||
sb.WriteByte(':')
|
||||
default:
|
||||
sb.WriteString(l.key)
|
||||
sb.WriteString(": ")
|
||||
sb.WriteString(l.value)
|
||||
}
|
||||
sb.WriteByte('\n')
|
||||
}
|
||||
sb.WriteString("---\n")
|
||||
return sb.String()
|
||||
}
|
||||
|
||||
// refreshOwned returns a's frontmatter with the generator-owned fields taken
|
||||
// from b where b supplies them. Fields absent from b keep a's value, so a
|
||||
// generator that stops emitting a field cannot blank it.
|
||||
//
|
||||
// A field b supplies that a doesn't have yet — every page ingested before
|
||||
// this mechanism existed has no source_hash line at all — is appended, not
|
||||
// dropped (round 1 of review: dropping it silently meant source_hash never
|
||||
// populated for the pre-existing corpus). If a has no frontmatter fence at
|
||||
// all, one is created to hold the new fields rather than splicing raw
|
||||
// "key: value" text into the body (round 2: that corrupted the page and the
|
||||
// fields would duplicate on every subsequent merge, since unfenced text
|
||||
// isn't recognized as frontmatter by parseSections on the next pass).
|
||||
func refreshOwned(a, b string) string {
|
||||
bLines, _ := parseFrontmatterBlock(b)
|
||||
bVals := make(map[string]string)
|
||||
for _, l := range bLines {
|
||||
if l.key != "" && generatorOwned[l.key] {
|
||||
bVals[l.key] = l.value
|
||||
}
|
||||
}
|
||||
if len(bVals) == 0 {
|
||||
return a
|
||||
}
|
||||
|
||||
aLines, _ := parseFrontmatterBlock(a) // ok==false (no fence) -> aLines nil, same as "start fresh"
|
||||
|
||||
// Sorted for a deterministic, byte-stable result when more than one
|
||||
// field is appended in the same call — map iteration order isn't.
|
||||
keys := make([]string, 0, len(bVals))
|
||||
for k := range bVals {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, k := range keys {
|
||||
aLines = setLine(aLines, k, bVals[k])
|
||||
}
|
||||
return renderFrontmatterBlock(aLines)
|
||||
}
|
||||
|
||||
// parseSections splits markdown into its frontmatter, its preamble (content
|
||||
// before the first `## ` heading), and its `## `-headed sections.
|
||||
//
|
||||
// The preamble is returned separately rather than as a section with an
|
||||
// empty heading — a section keyed by heading=="" would collide with a
|
||||
// literal degenerate `## ` (marker, no title text; strings.TrimPrefix also
|
||||
// yields ""), which is a distinct, addressable section a page can genuinely
|
||||
// have, not the same thing as "no heading yet." Two entries sharing one key
|
||||
// in Merge's idx map means one of them silently loses lookup visibility —
|
||||
// found in review as real, reproducible data loss, not a hypothetical.
|
||||
func parseSections(markdown string) (frontmatter, preamble string, sections []section) {
|
||||
lines := strings.Split(markdown, "\n")
|
||||
i := 0
|
||||
|
||||
@@ -78,6 +320,22 @@ func parseSections(markdown string) (frontmatter string, sections []section) {
|
||||
frontmatter = fmt.Sprintf("---\n%s\n---\n", strings.Join(fmLines, "\n"))
|
||||
}
|
||||
|
||||
// Everything before the first `## ` heading — an intro paragraph, or, if
|
||||
// the page has no headings at all, the entire body — is ordinary content,
|
||||
// not something this parser is entitled to drop (hyperguild#103 review:
|
||||
// dropping it meant Merge, one caller up, would silently replace the
|
||||
// whole on-disk body with the fresh extraction whenever a page didn't
|
||||
// follow the ## convention — exactly the class of bug this file exists
|
||||
// to prevent).
|
||||
var preambleLines []string
|
||||
for ; i < len(lines); i++ {
|
||||
if strings.HasPrefix(lines[i], "## ") {
|
||||
break
|
||||
}
|
||||
preambleLines = append(preambleLines, lines[i])
|
||||
}
|
||||
preamble = strings.Join(preambleLines, "\n")
|
||||
|
||||
var cur *section
|
||||
for ; i < len(lines); i++ {
|
||||
line := lines[i]
|
||||
@@ -96,24 +354,100 @@ func parseSections(markdown string) (frontmatter string, sections []section) {
|
||||
return
|
||||
}
|
||||
|
||||
func rebuildContent(frontmatter string, sections []section) string {
|
||||
// rebuildContent serialises sections back into a page.
|
||||
//
|
||||
// Each section body is normalised to exactly one trailing newline and no
|
||||
// leading blank line. Without that, a section keeps whatever blank lines it was
|
||||
// parsed with and this function adds its own, so every merge pass grows the page
|
||||
// by a blank line — invisible in a diff viewer, fatal to byte-stability. Compile-on-ingest
|
||||
// merges on every run, so "merging changes nothing when nothing changed" has to
|
||||
// hold at the byte level, not just the content level.
|
||||
func rebuildContent(frontmatter, preamble string, sections []section) string {
|
||||
var sb strings.Builder
|
||||
sb.WriteString(frontmatter)
|
||||
if body := strings.Trim(preamble, "\n"); body != "" {
|
||||
fmt.Fprintf(&sb, "\n%s\n", body)
|
||||
}
|
||||
for _, sec := range sections {
|
||||
fmt.Fprintf(&sb, "\n## %s\n\n%s", sec.heading, sec.content)
|
||||
body := strings.Trim(sec.content, "\n")
|
||||
fmt.Fprintf(&sb, "\n## %s\n\n%s\n", sec.heading, body)
|
||||
}
|
||||
return sb.String()
|
||||
}
|
||||
|
||||
// canonicalLine collapses a line to its comparison key: internal whitespace
|
||||
// runs (a re-extraction's bullet spacing, tabs vs spaces, trailing spaces)
|
||||
// must not make the same logical line look new. strings.Fields already
|
||||
// trims and splits on any whitespace run, so joining it back with single
|
||||
// spaces is exactly "same words, same order, whitespace-insensitive."
|
||||
func canonicalLine(line string) string {
|
||||
return strings.Join(strings.Fields(line), " ")
|
||||
}
|
||||
|
||||
// canonicalBlock applies canonicalLine per line of a multi-line section body
|
||||
// — the same whitespace-insensitivity appendNew/mergeBullets already give a
|
||||
// single line, extended to a whole section for MergeReport's disagreement
|
||||
// check (found in review, PR #129: comparing raw strings.TrimSpace only
|
||||
// tolerated leading/trailing whitespace, not a re-extraction's mid-line
|
||||
// spacing, which every other comparison in this file already tolerates).
|
||||
func canonicalBlock(block string) string {
|
||||
lines := strings.Split(block, "\n")
|
||||
out := make([]string, len(lines))
|
||||
for i, l := range lines {
|
||||
out[i] = canonicalLine(l)
|
||||
}
|
||||
return strings.Join(out, "\n")
|
||||
}
|
||||
|
||||
// appendNew concatenates the lines of b that a does not already carry.
|
||||
//
|
||||
// Append sections used to concatenate wholesale, which is correct exactly once.
|
||||
// Compile-on-ingest merges a freshly extracted page into the one already on
|
||||
// disk, so an unchanged source re-ingested would grow "Open Questions" and
|
||||
// "Updates" on every run until the page is unreadable — and the drift guard
|
||||
// that would otherwise catch it (hashing the source body) does not exist yet,
|
||||
// so idempotency has to hold here rather than upstream.
|
||||
//
|
||||
// Line-level rather than block-level: the common case is a re-ingest carrying
|
||||
// one genuinely new line alongside several it already has. A block-level
|
||||
// "skip if identical" would duplicate the old lines every time that happened.
|
||||
//
|
||||
// The cost is that a line repeated verbatim on purpose collapses to one. For
|
||||
// these sections — questions, updates, evolving notes — a repeat is noise
|
||||
// rather than signal, and the failure is conservative: the page stops growing
|
||||
// instead of growing wrongly.
|
||||
func appendNew(a, b string) string {
|
||||
seen := make(map[string]bool)
|
||||
for _, line := range strings.Split(a, "\n") {
|
||||
if c := canonicalLine(line); c != "" {
|
||||
seen[c] = true
|
||||
}
|
||||
}
|
||||
|
||||
var add []string
|
||||
for _, line := range strings.Split(b, "\n") {
|
||||
c := canonicalLine(line)
|
||||
if c == "" || seen[c] {
|
||||
continue
|
||||
}
|
||||
seen[c] = true
|
||||
add = append(add, line)
|
||||
}
|
||||
if len(add) == 0 {
|
||||
return a // nothing new — leave the section byte-identical
|
||||
}
|
||||
return strings.TrimRight(a, "\n") + "\n\n" + strings.Join(add, "\n") + "\n"
|
||||
}
|
||||
|
||||
func mergeBullets(a, b string) string {
|
||||
seen := make(map[string]bool)
|
||||
var lines []string
|
||||
for _, line := range strings.Split(a+b, "\n") {
|
||||
trimmed := strings.TrimSpace(line)
|
||||
if trimmed == "" || seen[trimmed] {
|
||||
c := canonicalLine(line)
|
||||
if c == "" || seen[c] {
|
||||
continue
|
||||
}
|
||||
seen[trimmed] = true
|
||||
seen[c] = true
|
||||
lines = append(lines, line)
|
||||
}
|
||||
return strings.Join(lines, "\n") + "\n"
|
||||
|
||||
@@ -53,3 +53,456 @@ func TestMerge_KeepsFrontmatterFromA(t *testing.T) {
|
||||
assert.Contains(t, got.Content, "title: A")
|
||||
assert.NotContains(t, got.Content, "title: B")
|
||||
}
|
||||
|
||||
// --- hyperguild#103 review: a page with content outside any `## ` section ---
|
||||
// must never lose that content on merge. parseSections used to only capture
|
||||
// text between `## ` headings — anything before the first heading (a common,
|
||||
// ordinary shape for a hand-written note: an intro paragraph before the first
|
||||
// section) or, worse, an entire page with no `## ` headings at all (a legacy
|
||||
// note, a stub, any hand-edit that doesn't follow the section convention)
|
||||
// vanished silently. Merge would then fill the page entirely from the fresh
|
||||
// extraction — the opposite of "never silently rewrite what a human put
|
||||
// there," the guarantee this whole file exists to provide.
|
||||
|
||||
func TestMerge_PreservesOnDiskContentWithNoSectionHeadings(t *testing.T) {
|
||||
a := Page{Path: "wiki/x.md", Content: "---\ntitle: X\n---\n\nJust some free text a human wrote. No section headings at all.\n"}
|
||||
b := Page{Path: "wiki/x.md", Content: "---\ntitle: X\n---\n\n## Definition\n\nFresh extraction's take.\n"}
|
||||
|
||||
got := Merge(a, b)
|
||||
assert.Contains(t, got.Content, "Just some free text a human wrote",
|
||||
"on-disk content with no ## headings must survive the merge, not be replaced wholesale by the fresh extraction")
|
||||
}
|
||||
|
||||
func TestMerge_DegenerateEmptyHeadingDoesNotCollideWithPreamble(t *testing.T) {
|
||||
// Second review pass on #103: representing the preamble as a section with
|
||||
// heading=="" collides with a literal `## ` (marker, no title text) —
|
||||
// strings.TrimPrefix("## ", "## ") is also "". `a` has only a preamble
|
||||
// (no degenerate heading of its own); `b` introduces a genuinely new
|
||||
// degenerate `## ` section. Because both key to "", Merge's idx thinks
|
||||
// b's section "already exists" in a (it's really just a's unrelated
|
||||
// preamble) and never adds it — b's section vanishes from the output
|
||||
// entirely, not merged, not kept, not appended.
|
||||
a := Page{Path: "p.md", Content: "---\ntitle: X\n---\n\nAn intro paragraph.\n\n## Definition\n\nA thing.\n"}
|
||||
b := Page{Path: "p.md", Content: "---\ntitle: X\n---\n\n## \n\nA genuinely new degenerate section's body.\n\n## Definition\n\nA thing.\n"}
|
||||
|
||||
got := Merge(a, b)
|
||||
assert.Contains(t, got.Content, "An intro paragraph.", "a's own preamble must survive")
|
||||
assert.Contains(t, got.Content, "A genuinely new degenerate section's body.",
|
||||
"b's degenerate section is new content, not a collision with a's unrelated preamble — it must not be silently dropped")
|
||||
}
|
||||
|
||||
func TestMerge_PreservesPreambleBeforeFirstHeading(t *testing.T) {
|
||||
a := Page{Path: "wiki/x.md", Content: "---\ntitle: X\n---\n\nAn intro paragraph before the first heading.\n\n## Definition\n\nA thing.\n"}
|
||||
b := Page{Path: "wiki/x.md", Content: "---\ntitle: X\n---\n\n## Definition\n\nA thing.\n"}
|
||||
|
||||
got := Merge(a, b)
|
||||
assert.Contains(t, got.Content, "An intro paragraph before the first heading",
|
||||
"text before the first ## heading is ordinary page content, not something the parser is entitled to drop")
|
||||
}
|
||||
|
||||
// --- hyperguild#103 review: dedup must be whitespace-insensitive -----------
|
||||
// Page content is produced by an LLM per run, so byte-identical re-extraction
|
||||
// of "the same" logical line is not guaranteed — different bullet spacing or
|
||||
// trailing whitespace on an otherwise-identical line must still dedup, or
|
||||
// idempotency only holds when the extractor happens to reproduce whitespace
|
||||
// exactly, which is not a real guarantee.
|
||||
|
||||
func TestMerge_AppendSectionDedupsAcrossWhitespaceVariants(t *testing.T) {
|
||||
a := Page{Path: "p.md", Content: "---\ntitle: X\n---\n\n## Open Questions\n\n- Does this converge?\n"}
|
||||
b := Page{Path: "p.md", Content: "---\ntitle: X\n---\n\n## Open Questions\n\n- Does this converge? \n"}
|
||||
|
||||
got := Merge(a, b)
|
||||
assert.Equal(t, 1, strings.Count(got.Content, "Does this converge?"),
|
||||
"same logical line with different whitespace must dedup, not duplicate")
|
||||
}
|
||||
|
||||
func TestMerge_BulletSectionDedupsAcrossWhitespaceVariants(t *testing.T) {
|
||||
a := Page{Path: "p.md", Content: "---\ntitle: X\n---\n\n## Related Concepts\n\n- [[bar|Bar]]\n"}
|
||||
b := Page{Path: "p.md", Content: "---\ntitle: X\n---\n\n## Related Concepts\n\n- [[bar|Bar]]\n"}
|
||||
|
||||
got := Merge(a, b)
|
||||
assert.Equal(t, 1, strings.Count(got.Content, "[[bar|Bar]]"),
|
||||
"same logical bullet with different whitespace must dedup, not duplicate")
|
||||
}
|
||||
|
||||
// --- brain#21/B1: merging must be safe to run again -------------------------
|
||||
|
||||
func TestMerge_AppendSectionDoesNotDuplicateContentAlreadyPresent(t *testing.T) {
|
||||
// Compile-on-ingest merges a freshly extracted page into the one already
|
||||
// on disk. Bullet sections union, so they are naturally idempotent — but
|
||||
// append sections concatenate, so re-ingesting an unchanged source would
|
||||
// grow "Open Questions" on every run until the page is unreadable.
|
||||
page := Page{Path: "wiki/sources/x.md", Content: `---
|
||||
title: X
|
||||
---
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Does this converge?
|
||||
`}
|
||||
|
||||
once := Merge(page, page)
|
||||
twice := Merge(once, page)
|
||||
|
||||
assert.Equal(t, 1, strings.Count(once.Content, "Does this converge?"),
|
||||
"merging a page with itself must not duplicate an append section")
|
||||
assert.Equal(t, once.Content, twice.Content,
|
||||
"merge must be idempotent — a third pass may not keep growing the page")
|
||||
}
|
||||
|
||||
func TestMerge_AppendSectionStillAccumulatesGenuinelyNewContent(t *testing.T) {
|
||||
// The dedup must not turn append sections into keep-a-only sections:
|
||||
// new material is exactly what compile-on-ingest exists to add.
|
||||
existing := Page{Path: "wiki/sources/x.md", Content: `---
|
||||
title: X
|
||||
---
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Does this converge?
|
||||
`}
|
||||
fresh := Page{Path: "wiki/sources/x.md", Content: `---
|
||||
title: X
|
||||
---
|
||||
|
||||
## Open Questions
|
||||
|
||||
- What happens under load?
|
||||
`}
|
||||
|
||||
got := Merge(existing, fresh).Content
|
||||
assert.Contains(t, got, "Does this converge?", "existing question must survive")
|
||||
assert.Contains(t, got, "What happens under load?", "new question must be added")
|
||||
}
|
||||
|
||||
func TestMerge_GeneratorOwnedFrontmatterFieldsAreRefreshed(t *testing.T) {
|
||||
// #103 made on-disk frontmatter win wholesale, which is what protects
|
||||
// human-set fields from a regenerating writer. The cost was that fields the
|
||||
// generator legitimately owns — when it last ran, what it last saw — froze
|
||||
// at their first-ingest values. An explicit owned-set is the boundary:
|
||||
// named fields the generator may refresh, everything else human-owned.
|
||||
existing := Page{Path: "wiki/sources/x.md", Content: `---
|
||||
title: 'X'
|
||||
last_updated: 2026-01-01
|
||||
source_hash: aaaa
|
||||
half_life: durable
|
||||
claim_provenance: asserted
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Original.
|
||||
`}
|
||||
fresh := Page{Path: "wiki/sources/x.md", Content: `---
|
||||
title: 'Renamed by the model'
|
||||
last_updated: 2026-08-22
|
||||
source_hash: bbbb
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Rewritten by the model.
|
||||
`}
|
||||
|
||||
got := Merge(existing, fresh).Content
|
||||
|
||||
assert.Contains(t, got, "last_updated: 2026-08-22", "generator owns last_updated")
|
||||
assert.Contains(t, got, "source_hash: bbbb", "generator owns source_hash")
|
||||
assert.Contains(t, got, "title: 'X'", "a human-set title must survive regeneration")
|
||||
assert.Contains(t, got, "half_life: durable", "the claim envelope must survive regeneration")
|
||||
assert.Contains(t, got, "claim_provenance: asserted")
|
||||
assert.Contains(t, got, "Original.", "an existing section still wins over the model's rewrite")
|
||||
assert.NotContains(t, got, "Renamed by the model")
|
||||
}
|
||||
|
||||
func TestMerge_GeneratorOwnedFieldsAreAppendedWhenAPageNeverHadThemBefore(t *testing.T) {
|
||||
// Found in review of PR #126: refreshOwned only replaced a generator-owned
|
||||
// line that already existed in a's frontmatter — it never appended one a
|
||||
// didn't have. Every page ingested before this feature shipped has no
|
||||
// source_hash line at all, so on its first re-ingest after deploy, the
|
||||
// fresh hash was silently dropped and the skip-on-unchanged feature this
|
||||
// PR exists to build would never engage for the entire pre-existing
|
||||
// corpus, permanently, with nothing telling anyone.
|
||||
existing := Page{Path: "wiki/sources/x.md", Content: `---
|
||||
title: 'X'
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Original.
|
||||
`}
|
||||
fresh := Page{Path: "wiki/sources/x.md", Content: `---
|
||||
title: 'Renamed by the model'
|
||||
last_updated: 2026-08-22
|
||||
source_hash: bbbb
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Rewritten by the model.
|
||||
`}
|
||||
|
||||
got := Merge(existing, fresh).Content
|
||||
|
||||
// Exact frontmatter block, not Contains: the second review round found
|
||||
// that a substring check doesn't prove the fields landed inside a valid
|
||||
// fence, in the right order, or that the fence itself wasn't corrupted —
|
||||
// two independent mutations (closeIdx targeting the wrong fence, reversed
|
||||
// sort) both survived a Contains-only version of this test.
|
||||
want := "---\ntitle: 'X'\nlast_updated: 2026-08-22\nsource_hash: bbbb\n---\n\n## Summary\n\nOriginal.\n"
|
||||
assert.Equal(t, want, got)
|
||||
}
|
||||
|
||||
func TestMerge_GeneratorOwnedFieldsCreateAFrontmatterBlockWhenThePageHadNoneAtAll(t *testing.T) {
|
||||
// Second review round on PR #126: when a has no frontmatter fence at all,
|
||||
// the append fallback spliced raw, unfenced "key: value" lines straight
|
||||
// into the body -- worse than the original bug, because the next
|
||||
// parseSections call wouldn't recognize them as frontmatter (they don't
|
||||
// start with `---`), so they'd be absorbed as ordinary preamble text and
|
||||
// re-duplicate on every subsequent merge instead of updating in place.
|
||||
existing := Page{Path: "wiki/sources/x.md", Content: "## Summary\n\nOriginal.\n"}
|
||||
fresh := Page{Path: "wiki/sources/x.md", Content: `---
|
||||
last_updated: 2026-08-22
|
||||
source_hash: bbbb
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Rewritten by the model.
|
||||
`}
|
||||
|
||||
got := Merge(existing, fresh).Content
|
||||
|
||||
want := "---\nlast_updated: 2026-08-22\nsource_hash: bbbb\n---\n\n## Summary\n\nOriginal.\n"
|
||||
assert.Equal(t, want, got)
|
||||
}
|
||||
|
||||
func TestMerge_GeneratorOwnedFieldsMixedReplaceAndAppend(t *testing.T) {
|
||||
// a already has last_updated (replace in place) but not source_hash
|
||||
// (append) -- the two code paths in the same call, together.
|
||||
existing := Page{Path: "wiki/sources/x.md", Content: `---
|
||||
title: 'X'
|
||||
last_updated: 2026-01-01
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Original.
|
||||
`}
|
||||
fresh := Page{Path: "wiki/sources/x.md", Content: `---
|
||||
last_updated: 2026-08-22
|
||||
source_hash: bbbb
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Rewritten by the model.
|
||||
`}
|
||||
|
||||
got := Merge(existing, fresh).Content
|
||||
|
||||
want := "---\ntitle: 'X'\nlast_updated: 2026-08-22\nsource_hash: bbbb\n---\n\n## Summary\n\nOriginal.\n"
|
||||
assert.Equal(t, want, got)
|
||||
}
|
||||
|
||||
// --- hyperguild#106: heading casing drift must not fork a section ----------
|
||||
// Page-aware extraction asks the model to revise the page it's given, but
|
||||
// nothing guarantees the model reproduces a heading's exact on-disk casing
|
||||
// ("Key Claims" vs "Key claims"). Without normalizing the match key, that
|
||||
// drift silently creates a near-duplicate section instead of updating the
|
||||
// original — the failure mode #106 exists to close.
|
||||
|
||||
func TestMerge_HeadingCaseDriftMatchesSameSection(t *testing.T) {
|
||||
a := Page{Path: "p.md", Content: "---\ntitle: X\n---\n\n## Key Claims\n\n- Appetite beats estimation.\n"}
|
||||
b := Page{Path: "p.md", Content: "---\ntitle: X\n---\n\n## Key claims\n\n- New pattern holds under load.\n"}
|
||||
|
||||
got := Merge(a, b).Content
|
||||
|
||||
assert.Equal(t, 1, strings.Count(got, "## Key"),
|
||||
"casing drift in the heading must not create a second section")
|
||||
assert.Contains(t, got, "Appetite beats estimation.")
|
||||
assert.Contains(t, got, "New pattern holds under load.",
|
||||
"the differently-cased section's content must still be unioned in")
|
||||
assert.Contains(t, got, "## Key Claims",
|
||||
"the on-disk heading's original casing must be preserved in the output")
|
||||
assert.NotContains(t, got, "## Key claims",
|
||||
"the fresh extraction's heading casing must not overwrite the on-disk display text")
|
||||
}
|
||||
|
||||
func TestMerge_HeadingWhitespaceDriftMatchesSameSection(t *testing.T) {
|
||||
a := Page{Path: "p.md", Content: "---\ntitle: X\n---\n\n## Open Questions \n\n- Does appetite scale?\n"}
|
||||
b := Page{Path: "p.md", Content: "---\ntitle: X\n---\n\n## Open Questions\n\n- What about remote teams?\n"}
|
||||
|
||||
got := Merge(a, b).Content
|
||||
|
||||
assert.Equal(t, 1, strings.Count(got, "## Open Questions"),
|
||||
"surrounding whitespace drift in the heading must not create a second section")
|
||||
assert.Contains(t, got, "Does appetite scale?")
|
||||
assert.Contains(t, got, "What about remote teams?")
|
||||
}
|
||||
|
||||
func TestMerge_PreExistingCaseDuplicateHeadingsRouteToTheFirstOccurrence(t *testing.T) {
|
||||
// Found in review of PR #130: normalizeHeading's idx map is built by
|
||||
// iterating secsA with idx[normalizeHeading(s.heading)] = i -- last
|
||||
// write wins. Before heading normalization existed, two on-disk
|
||||
// sections differing only by case (real, pre-existing corpus drift --
|
||||
// the exact pattern #106 itself cites) never collided, since idx keyed
|
||||
// on the exact string. After normalization they collide, and every
|
||||
// fresh update silently, deterministically routes to whichever
|
||||
// duplicate happens to appear LAST in the file -- the first one is
|
||||
// frozen forever, never receiving updates again, with nothing to say
|
||||
// so. A Contains-only check doesn't catch this: nothing is literally
|
||||
// deleted, a bullet just lands under the wrong of two still-present
|
||||
// headings. Assert it lands under the FIRST occurrence instead —
|
||||
// deterministic and matches "earliest is canonical," same reasoning
|
||||
// as every other on-disk-wins default in this file.
|
||||
a := Page{Path: "p.md", Content: "---\ntitle: X\n---\n\n## Key Claims\n\n- claim one\n\n## KEY CLAIMS\n\n- claim two\n"}
|
||||
b := Page{Path: "p.md", Content: "---\ntitle: X\n---\n\n## Key Claims\n\n- claim one\n- fresh claim\n"}
|
||||
|
||||
got := Merge(a, b).Content
|
||||
|
||||
assert.Contains(t, got, "## Key Claims\n\n- claim one\n- fresh claim\n",
|
||||
"the fresh bullet must land in the first-occurrence section, not silently route to the last duplicate")
|
||||
assert.Contains(t, got, "## KEY CLAIMS\n\n- claim two\n",
|
||||
"the second duplicate is untouched by this update, not merged into or overwritten")
|
||||
}
|
||||
|
||||
func TestMergeReport_SelfMergeOfPreExistingDuplicateHeadingsReportsNoConflict(t *testing.T) {
|
||||
// Found in review of PR #132: first-occurrence-wins routes ALL of b's
|
||||
// occurrences of a normalized heading to a's FIRST occurrence, not to
|
||||
// their own positional counterpart. So merging a page against itself
|
||||
// (or an echoed-back copy, exactly what #106's revision prompt can
|
||||
// produce) pairs b's second "## NOTES" against a's first "## Notes" —
|
||||
// different content, same normalized key — and MergeReport wrongly
|
||||
// reports a disagreement where none exists. Fixed by pairing
|
||||
// positionally: b's Nth occurrence of a normalized heading matches a's
|
||||
// Nth occurrence, not always the first.
|
||||
page := Page{Path: "p.md", Content: "---\ntitle: X\n---\n\n## Notes\n\nFirst note.\n\n## NOTES\n\nSecond note.\n"}
|
||||
|
||||
_, conflicts := MergeReport(page, page)
|
||||
assert.Empty(t, conflicts, "merging a page with pre-existing duplicate headings against itself must report no conflicts")
|
||||
}
|
||||
|
||||
func TestMergeReport_DuplicateHeadingsPairPositionallyNotAllToFirst(t *testing.T) {
|
||||
// Companion case: b has genuinely different content in its second
|
||||
// occurrence of the normalized heading — that must compare against a's
|
||||
// second occurrence (and report if they disagree), not against a's
|
||||
// first (which would either false-positive if a's first differs, or
|
||||
// false-negative-mask a real disagreement in the second pair).
|
||||
a := Page{Path: "p.md", Content: "---\ntitle: X\n---\n\n## Notes\n\nFirst note.\n\n## NOTES\n\nSecond note.\n"}
|
||||
b := Page{Path: "p.md", Content: "---\ntitle: X\n---\n\n## Notes\n\nFirst note.\n\n## NOTES\n\nSecond note, revised.\n"}
|
||||
|
||||
_, conflicts := MergeReport(a, b)
|
||||
assert.Equal(t, []string{"NOTES"}, conflicts,
|
||||
"only the second occurrence, which actually disagrees, is reported")
|
||||
}
|
||||
|
||||
func TestRefreshOwned_CommentLinesSurviveUnmangled(t *testing.T) {
|
||||
// Third review round on PR #126: parseFrontmatterBlock dropped the
|
||||
// sibling api package's comment guard (strings.HasPrefix(trimmed, "#")).
|
||||
// Without it, a comment line containing a colon gets cut at the colon
|
||||
// and re-rendered with collapsed whitespace instead of surviving
|
||||
// verbatim.
|
||||
a := "---\n#see also: https://x/y\ntitle: 'X'\n---\n"
|
||||
b := "---\nsource_hash: bbbb\n---\n"
|
||||
|
||||
got := refreshOwned(a, b)
|
||||
assert.Equal(t, "---\n#see also: https://x/y\ntitle: 'X'\nsource_hash: bbbb\n---\n", got)
|
||||
}
|
||||
|
||||
func TestRefreshOwned_IndentedListItemsWithColonsAreNotMisparsedAsKeys(t *testing.T) {
|
||||
// Same round: an indented YAML block-list item that happens to contain a
|
||||
// colon (e.g. a title with a subtitle) is a continuation line, never a
|
||||
// top-level key, regardless of what strings.Cut finds in it. Misparsing
|
||||
// it as a key/value pair and re-rendering strips its indentation,
|
||||
// de-indenting it out of the list it belongs to -- and refreshOwned
|
||||
// re-renders this block on essentially every non-skipped ingest, so this
|
||||
// is the normal path, not an edge case.
|
||||
a := "---\naliases:\n - 'Chapter 3: The Reckoning'\ntitle: 'X'\n---\n"
|
||||
b := "---\nsource_hash: bbbb\n---\n"
|
||||
|
||||
got := refreshOwned(a, b)
|
||||
assert.Equal(t, "---\naliases:\n - 'Chapter 3: The Reckoning'\ntitle: 'X'\nsource_hash: bbbb\n---\n", got)
|
||||
}
|
||||
|
||||
func TestMerge_ReportsSectionsWhereTheFreshVersionWasDiscarded(t *testing.T) {
|
||||
// The asymmetry that makes ingest safe (on-disk wins) also throws away a
|
||||
// real signal: the source now says something different about a section we
|
||||
// already have. Today that disagreement is silently discarded, so nobody
|
||||
// can tell "the source is unchanged" from "the source changed its mind and
|
||||
// we ignored it".
|
||||
existing := Page{Path: "wiki/sources/x.md", Content: `---
|
||||
title: 'X'
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Appetite is fixed, scope is variable.
|
||||
|
||||
## Method
|
||||
|
||||
Six-week cycles.
|
||||
`}
|
||||
fresh := Page{Path: "wiki/sources/x.md", Content: `---
|
||||
title: 'X'
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Appetite is negotiable and scope is fixed.
|
||||
|
||||
## Method
|
||||
|
||||
Six-week cycles.
|
||||
`}
|
||||
|
||||
got, conflicts := MergeReport(existing, fresh)
|
||||
|
||||
assert.Contains(t, got.Content, "Appetite is fixed", "on-disk version still wins")
|
||||
assert.NotContains(t, got.Content, "negotiable")
|
||||
assert.Equal(t, []string{"Summary"}, conflicts,
|
||||
"only the section that actually disagreed is reported")
|
||||
}
|
||||
|
||||
func TestMerge_ReportsNothingWhenTheFreshVersionAgrees(t *testing.T) {
|
||||
page := Page{Path: "wiki/sources/x.md", Content: `---
|
||||
title: 'X'
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Appetite is fixed, scope is variable.
|
||||
`}
|
||||
|
||||
_, conflicts := MergeReport(page, page)
|
||||
assert.Empty(t, conflicts, "identical content is not a disagreement")
|
||||
}
|
||||
|
||||
func TestMerge_ReportsNothingForInternalWhitespaceVariantsOnly(t *testing.T) {
|
||||
// Found in review of PR #129: the conflict comparison used raw
|
||||
// strings.TrimSpace, which only tolerates LEADING/TRAILING whitespace —
|
||||
// unlike appendNew/mergeBullets, which already canonicalize internal
|
||||
// whitespace runs too (#103: a re-extraction's spacing isn't guaranteed
|
||||
// byte-stable). A mid-line double-space from re-extraction would
|
||||
// false-positive as a "disagreement" here even though every other
|
||||
// whitespace-sensitive comparison in this file already tolerates it.
|
||||
existing := Page{Path: "wiki/sources/x.md", Content: `---
|
||||
title: 'X'
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Appetite is fixed, scope is variable.
|
||||
`}
|
||||
fresh := Page{Path: "wiki/sources/x.md", Content: `---
|
||||
title: 'X'
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Appetite is fixed, scope is variable.
|
||||
`}
|
||||
|
||||
_, conflicts := MergeReport(existing, fresh)
|
||||
assert.Empty(t, conflicts, "internal whitespace differences alone are not a disagreement")
|
||||
}
|
||||
|
||||
@@ -7,7 +7,6 @@ type Config struct {
|
||||
LiteLLMBaseURL string // LITELLM_BASE_URL, default http://iguana:4000
|
||||
LiteLLMAPIKey string // LITELLM_API_KEY
|
||||
ConfigDir string // SUPERVISOR_CONFIG_DIR, default ./config/supervisor
|
||||
ModelsFile string // SUPERVISOR_MODELS_FILE, default <ConfigDir>/../models.yaml
|
||||
IngestBaseURL string // INGEST_BASE_URL, default http://localhost:3300
|
||||
IngestSvcURL string // INGEST_SVC_URL — base URL for brain_ingest (/ingest, /ingest-path)
|
||||
KBRetrievalURL string // KB_RETRIEVAL_URL — base URL for brain_search
|
||||
@@ -23,7 +22,6 @@ func Load() (Config, error) {
|
||||
LiteLLMAPIKey: os.Getenv("LITELLM_API_KEY"),
|
||||
ConfigDir: envOr("SUPERVISOR_CONFIG_DIR", "./config/supervisor"),
|
||||
}
|
||||
cfg.ModelsFile = envOr("SUPERVISOR_MODELS_FILE", cfg.ConfigDir+"/../models.yaml")
|
||||
cfg.IngestBaseURL = envOr("INGEST_BASE_URL", "http://localhost:3300")
|
||||
cfg.IngestSvcURL = envOr("INGEST_SVC_URL", "")
|
||||
cfg.KBRetrievalURL = envOr("KB_RETRIEVAL_URL", "")
|
||||
|
||||
@@ -1,49 +0,0 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
type skillChain struct {
|
||||
Chain []string `yaml:"chain"`
|
||||
}
|
||||
|
||||
type modelsFile struct {
|
||||
DefaultChain []string `yaml:"default_chain"`
|
||||
Skills map[string]skillChain `yaml:"skills"`
|
||||
}
|
||||
|
||||
type Models struct {
|
||||
data modelsFile
|
||||
}
|
||||
|
||||
func LoadModels(path string) (Models, error) {
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return Models{}, fmt.Errorf("load models: %w", err)
|
||||
}
|
||||
var f modelsFile
|
||||
if err := yaml.Unmarshal(raw, &f); err != nil {
|
||||
return Models{}, fmt.Errorf("parse models: %w", err)
|
||||
}
|
||||
return Models{data: f}, nil
|
||||
}
|
||||
|
||||
// ModelFor returns the primary model to use for a skill.
|
||||
// If override is non-empty, it is returned directly.
|
||||
// Falls back to default_chain[0] when the skill has no explicit entry.
|
||||
func (m Models) ModelFor(skill, override string) string {
|
||||
if override != "" {
|
||||
return override
|
||||
}
|
||||
if sc, ok := m.data.Skills[skill]; ok && len(sc.Chain) > 0 {
|
||||
return sc.Chain[0]
|
||||
}
|
||||
if len(m.data.DefaultChain) > 0 {
|
||||
return m.data.DefaultChain[0]
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -1,53 +0,0 @@
|
||||
package config_test
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"git.d-ma.be/mathias/hyperguild/internal/config"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
const testYAML = `
|
||||
default_chain:
|
||||
- ollama/qwen3-coder-30b-tuned
|
||||
- claude-sonnet-4-6
|
||||
|
||||
skills:
|
||||
review:
|
||||
chain:
|
||||
- ollama/devstral-tuned
|
||||
- ollama/gemma4
|
||||
- claude-sonnet-4-6
|
||||
spec:
|
||||
chain:
|
||||
- ollama/phi4
|
||||
- claude-opus-4-6
|
||||
`
|
||||
|
||||
func writeModels(t *testing.T, content string) string {
|
||||
t.Helper()
|
||||
f := filepath.Join(t.TempDir(), "models.yaml")
|
||||
require.NoError(t, os.WriteFile(f, []byte(content), 0o644))
|
||||
return f
|
||||
}
|
||||
|
||||
func TestModelsModelForSkillWithEntry(t *testing.T) {
|
||||
m, err := config.LoadModels(writeModels(t, testYAML))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "ollama/devstral-tuned", m.ModelFor("review", ""))
|
||||
}
|
||||
|
||||
func TestModelsModelForDefaultFallback(t *testing.T) {
|
||||
m, err := config.LoadModels(writeModels(t, testYAML))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "ollama/qwen3-coder-30b-tuned", m.ModelFor("trainer", ""))
|
||||
}
|
||||
|
||||
func TestModelsModelForCallerOverride(t *testing.T) {
|
||||
m, err := config.LoadModels(writeModels(t, testYAML))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "claude-opus-4-6", m.ModelFor("review", "claude-opus-4-6"))
|
||||
}
|
||||
@@ -1,182 +0,0 @@
|
||||
// internal/skills/brain/handlers.go
|
||||
package brain
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
)
|
||||
|
||||
// Handle dispatches brain tool calls.
|
||||
func (s *Skill) Handle(ctx context.Context, tool string, args json.RawMessage) (json.RawMessage, error) {
|
||||
switch tool {
|
||||
case "brain_query":
|
||||
return s.query(ctx, args)
|
||||
case "brain_write":
|
||||
return s.write(ctx, args)
|
||||
case "brain_ingest_raw":
|
||||
return s.ingestRaw(ctx, args)
|
||||
case "brain_ingest":
|
||||
return s.ingest(ctx, args)
|
||||
case "brain_search":
|
||||
return s.search(ctx, args)
|
||||
default:
|
||||
return nil, fmt.Errorf("unknown brain tool: %s", tool)
|
||||
}
|
||||
}
|
||||
|
||||
type queryArgs struct {
|
||||
Query string `json:"query"`
|
||||
Limit int `json:"limit,omitempty"`
|
||||
}
|
||||
|
||||
func (s *Skill) query(ctx context.Context, args json.RawMessage) (json.RawMessage, error) {
|
||||
var a queryArgs
|
||||
if err := json.Unmarshal(args, &a); err != nil {
|
||||
return nil, fmt.Errorf("parse args: %w", err)
|
||||
}
|
||||
if a.Query == "" {
|
||||
return nil, fmt.Errorf("query is required")
|
||||
}
|
||||
if a.Limit == 0 {
|
||||
a.Limit = 5
|
||||
}
|
||||
return s.post(ctx, "/query", a)
|
||||
}
|
||||
|
||||
type writeArgs struct {
|
||||
Content string `json:"content"`
|
||||
Type string `json:"type,omitempty"`
|
||||
Domain string `json:"domain,omitempty"`
|
||||
Filename string `json:"filename,omitempty"`
|
||||
}
|
||||
|
||||
func (s *Skill) write(ctx context.Context, args json.RawMessage) (json.RawMessage, error) {
|
||||
var a writeArgs
|
||||
if err := json.Unmarshal(args, &a); err != nil {
|
||||
return nil, fmt.Errorf("parse args: %w", err)
|
||||
}
|
||||
if a.Content == "" {
|
||||
return nil, fmt.Errorf("content is required")
|
||||
}
|
||||
return s.post(ctx, "/write", a)
|
||||
}
|
||||
|
||||
type ingestArgs struct {
|
||||
Content string `json:"content,omitempty"`
|
||||
Source string `json:"source,omitempty"`
|
||||
Path string `json:"path,omitempty"`
|
||||
DryRun bool `json:"dry_run,omitempty"`
|
||||
}
|
||||
|
||||
func (s *Skill) ingest(ctx context.Context, args json.RawMessage) (json.RawMessage, error) {
|
||||
var a ingestArgs
|
||||
if err := json.Unmarshal(args, &a); err != nil {
|
||||
return nil, fmt.Errorf("parse args: %w", err)
|
||||
}
|
||||
if s.cfg.IngestSvcURL == "" {
|
||||
return nil, fmt.Errorf("brain_ingest: INGEST_SVC_URL not configured")
|
||||
}
|
||||
if a.Path != "" && a.Content != "" {
|
||||
return nil, fmt.Errorf("path and content+source are mutually exclusive: provide one or the other")
|
||||
}
|
||||
if a.Path != "" {
|
||||
return s.postTo(ctx, s.cfg.IngestSvcURL+"/ingest-path", map[string]any{
|
||||
"path": a.Path,
|
||||
"source": a.Source,
|
||||
"dry_run": a.DryRun,
|
||||
})
|
||||
}
|
||||
if a.Content != "" && a.Source != "" {
|
||||
return s.postTo(ctx, s.cfg.IngestSvcURL+"/ingest", map[string]any{
|
||||
"content": a.Content,
|
||||
"source": a.Source,
|
||||
"dry_run": a.DryRun,
|
||||
})
|
||||
}
|
||||
return nil, fmt.Errorf("either content+source or path is required")
|
||||
}
|
||||
|
||||
type ingestRawArgs struct {
|
||||
Source string `json:"source"`
|
||||
Pages []any `json:"pages"`
|
||||
DryRun bool `json:"dry_run,omitempty"`
|
||||
}
|
||||
|
||||
func (s *Skill) ingestRaw(ctx context.Context, args json.RawMessage) (json.RawMessage, error) {
|
||||
var a ingestRawArgs
|
||||
if err := json.Unmarshal(args, &a); err != nil {
|
||||
return nil, fmt.Errorf("parse args: %w", err)
|
||||
}
|
||||
if s.cfg.IngestSvcURL == "" {
|
||||
return nil, fmt.Errorf("brain_ingest_raw: INGEST_SVC_URL not configured")
|
||||
}
|
||||
if a.Source == "" {
|
||||
return nil, fmt.Errorf("source is required")
|
||||
}
|
||||
if len(a.Pages) == 0 {
|
||||
return nil, fmt.Errorf("pages is required and must be non-empty")
|
||||
}
|
||||
return s.postTo(ctx, s.cfg.IngestSvcURL+"/ingest-raw", map[string]any{
|
||||
"source": a.Source,
|
||||
"pages": a.Pages,
|
||||
"dry_run": a.DryRun,
|
||||
})
|
||||
}
|
||||
|
||||
type searchArgs struct {
|
||||
Query string `json:"query"`
|
||||
Collection string `json:"collection,omitempty"`
|
||||
Limit int `json:"limit,omitempty"`
|
||||
}
|
||||
|
||||
func (s *Skill) search(ctx context.Context, args json.RawMessage) (json.RawMessage, error) {
|
||||
var a searchArgs
|
||||
if err := json.Unmarshal(args, &a); err != nil {
|
||||
return nil, fmt.Errorf("parse args: %w", err)
|
||||
}
|
||||
if a.Query == "" {
|
||||
return nil, fmt.Errorf("query is required")
|
||||
}
|
||||
if a.Limit == 0 {
|
||||
a.Limit = 5
|
||||
}
|
||||
if s.cfg.KBRetrievalURL == "" {
|
||||
return nil, fmt.Errorf("brain_search: KB_RETRIEVAL_URL not configured")
|
||||
}
|
||||
return s.postTo(ctx, s.cfg.KBRetrievalURL+"/api/v1/search", a)
|
||||
}
|
||||
|
||||
func (s *Skill) post(ctx context.Context, path string, body any) (json.RawMessage, error) {
|
||||
return s.postTo(ctx, s.cfg.IngestBaseURL+path, body)
|
||||
}
|
||||
|
||||
func (s *Skill) postTo(ctx context.Context, url string, body any) (json.RawMessage, error) {
|
||||
b, err := json.Marshal(body)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("marshal request: %w", err)
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(b))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("build request: %w", err)
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("call ingestion server: %w", err)
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
|
||||
out, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read response: %w", err)
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil, fmt.Errorf("ingestion server returned %d: %s", resp.StatusCode, out)
|
||||
}
|
||||
return json.RawMessage(out), nil
|
||||
}
|
||||
@@ -1,122 +0,0 @@
|
||||
// internal/skills/brain/handlers_test.go
|
||||
package brain_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"git.d-ma.be/mathias/hyperguild/internal/skills/brain"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestHandle_BrainQuery_CallsIngestServer(t *testing.T) {
|
||||
called := false
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
assert.Equal(t, "/query", r.URL.Path)
|
||||
called = true
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{
|
||||
"results": []map[string]any{
|
||||
{"path": "wiki/concepts/tdd.md", "title": "TDD", "excerpt": "Test-driven development.", "score": 3},
|
||||
},
|
||||
})
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
s := brain.New(brain.Config{IngestBaseURL: srv.URL})
|
||||
args, _ := json.Marshal(map[string]string{"query": "test driven development"})
|
||||
out, err := s.Handle(context.Background(), "brain_query", args)
|
||||
require.NoError(t, err)
|
||||
assert.True(t, called)
|
||||
|
||||
var result map[string]any
|
||||
require.NoError(t, json.Unmarshal(out, &result))
|
||||
results := result["results"].([]any)
|
||||
assert.Len(t, results, 1)
|
||||
}
|
||||
|
||||
func TestHandle_BrainWrite_CallsIngestServer(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
assert.Equal(t, "/write", r.URL.Path)
|
||||
var body map[string]string
|
||||
require.NoError(t, json.NewDecoder(r.Body).Decode(&body))
|
||||
assert.Equal(t, "concept", body["type"])
|
||||
assert.Equal(t, "# Test\n\nSome learning.", body["content"])
|
||||
_ = json.NewEncoder(w).Encode(map[string]string{"path": "raw/test.md"})
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
s := brain.New(brain.Config{IngestBaseURL: srv.URL})
|
||||
args, _ := json.Marshal(map[string]string{"content": "# Test\n\nSome learning.", "type": "concept"})
|
||||
out, err := s.Handle(context.Background(), "brain_write", args)
|
||||
require.NoError(t, err)
|
||||
var result map[string]string
|
||||
require.NoError(t, json.Unmarshal(out, &result))
|
||||
assert.Equal(t, "raw/test.md", result["path"])
|
||||
}
|
||||
|
||||
func TestHandle_UnknownTool_ReturnsError(t *testing.T) {
|
||||
s := brain.New(brain.Config{IngestBaseURL: "http://localhost:3300"})
|
||||
_, err := s.Handle(context.Background(), "brain_unknown", nil)
|
||||
assert.Error(t, err)
|
||||
}
|
||||
|
||||
func TestIngest_RoutesToIngestPath(t *testing.T) {
|
||||
var capturedPath string
|
||||
var capturedBody map[string]any
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
capturedPath = r.URL.Path
|
||||
require.NoError(t, json.NewDecoder(r.Body).Decode(&capturedBody))
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{"pages": []string{"wiki/foo.md"}})
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
s := brain.New(brain.Config{IngestSvcURL: srv.URL})
|
||||
args, _ := json.Marshal(map[string]any{"path": "/tmp/some-file.md"})
|
||||
out, err := s.Handle(context.Background(), "brain_ingest", args)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Equal(t, "/ingest-path", capturedPath)
|
||||
assert.Equal(t, "/tmp/some-file.md", capturedBody["path"])
|
||||
|
||||
var result map[string]any
|
||||
require.NoError(t, json.Unmarshal(out, &result))
|
||||
pages := result["pages"].([]any)
|
||||
assert.Len(t, pages, 1)
|
||||
}
|
||||
|
||||
func TestIngest_RoutesToIngest(t *testing.T) {
|
||||
var capturedPath string
|
||||
var capturedBody map[string]any
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
capturedPath = r.URL.Path
|
||||
require.NoError(t, json.NewDecoder(r.Body).Decode(&capturedBody))
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{"pages": []string{"wiki/bar.md"}})
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
s := brain.New(brain.Config{IngestSvcURL: srv.URL})
|
||||
args, _ := json.Marshal(map[string]any{"content": "some content", "source": "my-source.md"})
|
||||
out, err := s.Handle(context.Background(), "brain_ingest", args)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Equal(t, "/ingest", capturedPath)
|
||||
assert.Equal(t, "some content", capturedBody["content"])
|
||||
assert.Equal(t, "my-source.md", capturedBody["source"])
|
||||
|
||||
var result map[string]any
|
||||
require.NoError(t, json.Unmarshal(out, &result))
|
||||
pages := result["pages"].([]any)
|
||||
assert.Len(t, pages, 1)
|
||||
}
|
||||
|
||||
func TestIngest_MissingRequiredFields(t *testing.T) {
|
||||
s := brain.New(brain.Config{IngestSvcURL: "http://localhost:3300"})
|
||||
args, _ := json.Marshal(map[string]any{})
|
||||
_, err := s.Handle(context.Background(), "brain_ingest", args)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "either content+source or path is required")
|
||||
}
|
||||
@@ -1,111 +0,0 @@
|
||||
// internal/skills/brain/skill.go
|
||||
package brain
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
|
||||
"git.d-ma.be/mathias/hyperguild/internal/registry"
|
||||
)
|
||||
|
||||
// Config holds brain skill configuration.
|
||||
type Config struct {
|
||||
IngestBaseURL string // base URL of the ingestion HTTP server (brain_query, brain_write)
|
||||
IngestSvcURL string // base URL of the ingestion-svc HTTP server (brain_ingest)
|
||||
KBRetrievalURL string // base URL of the kb-retrieval server (brain_search)
|
||||
}
|
||||
|
||||
// Skill implements registry.Skill for brain_query and brain_write.
|
||||
type Skill struct {
|
||||
cfg Config
|
||||
}
|
||||
|
||||
// New constructs a brain Skill.
|
||||
func New(cfg Config) *Skill { return &Skill{cfg: cfg} }
|
||||
|
||||
// Name returns the skill name used for routing.
|
||||
func (s *Skill) Name() string { return "brain" }
|
||||
|
||||
// Tools returns the MCP tool definitions for brain_query and brain_write.
|
||||
func (s *Skill) Tools() []registry.ToolDef {
|
||||
schema := func(required []string, props map[string]any) json.RawMessage {
|
||||
b, _ := json.Marshal(map[string]any{"type": "object", "required": required, "properties": props})
|
||||
return b
|
||||
}
|
||||
str := map[string]any{"type": "string"}
|
||||
num := map[string]any{"type": "integer"}
|
||||
|
||||
tools := []registry.ToolDef{
|
||||
{
|
||||
Name: "brain_query",
|
||||
Description: "BM25 full-text search across brain/knowledge/ and brain/wiki/ markdown files. Fast, no embeddings needed. Call before any significant task.",
|
||||
InputSchema: schema([]string{"query"}, map[string]any{
|
||||
"query": str,
|
||||
"limit": num,
|
||||
}),
|
||||
},
|
||||
{
|
||||
Name: "brain_write",
|
||||
Description: "Write a raw knowledge note to brain/knowledge/ for later ingestion.",
|
||||
InputSchema: schema([]string{"content"}, map[string]any{
|
||||
"content": str,
|
||||
"type": str,
|
||||
"domain": str,
|
||||
"filename": str,
|
||||
}),
|
||||
},
|
||||
}
|
||||
if s.cfg.IngestSvcURL != "" {
|
||||
tools = append(tools, registry.ToolDef{
|
||||
Name: "brain_ingest_raw",
|
||||
Description: "Ingest pre-structured pages into the brain wiki, bypassing the LLM extraction step. " +
|
||||
"Use when you (the calling agent) have already extracted entities, concepts, and content from a source. " +
|
||||
"Provide source (human-readable name) and pages (array of {title, type, subtype, domain, content} objects). " +
|
||||
"The pipeline computes slugs, paths, frontmatter, wikilink canonicalization, and source back-references. " +
|
||||
"Returns the list of wiki pages written.",
|
||||
InputSchema: schema([]string{"source", "pages"}, map[string]any{
|
||||
"source": map[string]any{"type": "string", "description": "human-readable name for the source, e.g. 'shape-up-book'"},
|
||||
"pages": map[string]any{
|
||||
"type": "array",
|
||||
"items": map[string]any{
|
||||
"type": "object",
|
||||
"required": []string{"title", "type", "content"},
|
||||
"properties": map[string]any{
|
||||
"title": map[string]any{"type": "string", "description": "page title, e.g. 'Hash Encoding'"},
|
||||
"type": map[string]any{"type": "string", "enum": []string{"source", "concept", "entity"}, "description": "page type"},
|
||||
"subtype": map[string]any{"type": "string", "description": "entity: person|company|tool|model|framework|technology; source: article|pdf|book|video|note|project"},
|
||||
"domain": map[string]any{"type": "string", "description": "knowledge domain, e.g. 'Machine Learning'"},
|
||||
"content": map[string]any{"type": "string", "description": "markdown body — no frontmatter, use [[Display Name]] for wikilinks"},
|
||||
},
|
||||
},
|
||||
},
|
||||
"dry_run": map[string]any{"type": "boolean"},
|
||||
}),
|
||||
})
|
||||
tools = append(tools, registry.ToolDef{
|
||||
Name: "brain_ingest",
|
||||
Description: "Ingest content into the brain wiki (brain/wiki/). Calls an LLM to produce structured wiki pages. " +
|
||||
"Use for substantial documents, articles, or knowledge worth structuring. " +
|
||||
"Provide EITHER (a) path — absolute path to a file or directory, " +
|
||||
"OR (b) content + source — raw text and a human-readable name. " +
|
||||
"Providing both is an error. Returns the list of wiki pages written.",
|
||||
InputSchema: schema([]string{}, map[string]any{
|
||||
"content": map[string]any{"type": "string", "description": "raw text to ingest; required when path is not set"},
|
||||
"source": map[string]any{"type": "string", "description": "human-readable name for the content, e.g. 'shape-up-book'; required when path is not set"},
|
||||
"path": map[string]any{"type": "string", "description": "absolute path to a file or directory to ingest; mutually exclusive with content+source"},
|
||||
"dry_run": map[string]any{"type": "boolean"},
|
||||
}),
|
||||
})
|
||||
}
|
||||
if s.cfg.KBRetrievalURL != "" {
|
||||
tools = append(tools, registry.ToolDef{
|
||||
Name: "brain_search",
|
||||
Description: "Semantic vector search across the brain wiki using embeddings. Use when brain_query returns no results or you need conceptually-related results rather than keyword matches.",
|
||||
InputSchema: schema([]string{"query"}, map[string]any{
|
||||
"query": str,
|
||||
"collection": str,
|
||||
"limit": num,
|
||||
}),
|
||||
})
|
||||
}
|
||||
return tools
|
||||
}
|
||||
@@ -58,7 +58,7 @@ if [ "$FORCE" = false ]; then
|
||||
while IFS= read -r _match; do
|
||||
_lineno="${_match%%:*}"
|
||||
_content="${_match#*:}"
|
||||
_token=$(printf '%s' "$_content" | grep -o '{{[^}]*}}' | head -1)
|
||||
_token=$(head -1 < <(grep -o '{{[^}]*}}' <<<"$_content"))
|
||||
echo " $PROJECT_FILE:$_lineno: unfilled placeholder $_token" >&2
|
||||
done <<< "$_placeholders"
|
||||
echo "" >&2
|
||||
|
||||
@@ -1,64 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Boot the routing binary and exercise its four tools against live deps.
|
||||
# Skipped when LITELLM_BASE_URL or BRAIN_URL is unreachable.
|
||||
|
||||
LITELLM_BASE_URL="${LITELLM_BASE_URL:-https://llm-api.d-ma.be}"
|
||||
BRAIN_URL="${BRAIN_URL:-http://koala:30330}"
|
||||
|
||||
if ! curl -sS --max-time 2 "${LITELLM_BASE_URL}/v1/models" >/dev/null 2>&1; then
|
||||
echo "SKIP: LITELLM at ${LITELLM_BASE_URL} unreachable"
|
||||
exit 0
|
||||
fi
|
||||
if ! curl -sS --max-time 2 "${BRAIN_URL}/query" -X POST -d '{"query":"x","k":1}' -H 'Content-Type: application/json' >/dev/null 2>&1; then
|
||||
echo "SKIP: BRAIN at ${BRAIN_URL} unreachable"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
PORT=33310
|
||||
BIN=$(mktemp)
|
||||
trap 'rm -f $BIN; pkill -P $$ -f "$BIN" 2>/dev/null || true' EXIT
|
||||
|
||||
go build -o "$BIN" ./cmd/routing
|
||||
|
||||
LITELLM_BASE_URL="$LITELLM_BASE_URL" BRAIN_URL="$BRAIN_URL" \
|
||||
ROUTING_PORT="$PORT" SUPERVISOR_CONFIG_DIR="$(pwd)/config/supervisor" \
|
||||
"$BIN" &
|
||||
|
||||
# Wait for the binary to bind.
|
||||
for _ in $(seq 1 50); do
|
||||
curl -sS "http://127.0.0.1:${PORT}/healthz" >/dev/null 2>&1 && break
|
||||
sleep 0.1
|
||||
done
|
||||
|
||||
call_tool() {
|
||||
local tool="$1"
|
||||
local args="$2"
|
||||
curl -sS -X POST "http://127.0.0.1:${PORT}/mcp" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"${tool}\",\"arguments\":${args}}}" \
|
||||
| jq -e '.result // .error' > /dev/null
|
||||
}
|
||||
|
||||
echo "calling tools/list..."
|
||||
curl -sS -X POST "http://127.0.0.1:${PORT}/mcp" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' \
|
||||
| jq -r '.result.tools | map(.name) | sort | .[]'
|
||||
|
||||
echo "calling each tool..."
|
||||
call_tool review '{"project_root":"/tmp","files":["README.md"],"session_id":"smoke-1"}'
|
||||
call_tool debug '{"project_root":"/tmp","error":"smoke test","session_id":"smoke-1"}'
|
||||
call_tool retrospective '{"session_id":"smoke-1"}'
|
||||
call_tool trainer '{"session_id":"smoke-1"}'
|
||||
|
||||
echo "checking brain has _routing entries..."
|
||||
sleep 2
|
||||
COUNT=$(curl -sS "${BRAIN_URL}/pass-rate?skill=_routing&window=1h" | jq -r '.total // 0')
|
||||
if [ "${COUNT}" -lt 4 ]; then
|
||||
echo "FAIL: expected >=4 _routing entries in last 1h, got ${COUNT}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "PASS: smoke:routing"
|
||||
@@ -49,12 +49,13 @@ Persist the session via a **single `capture` call** (the `brain:capture` MCP too
|
||||
**Assemble one payload:**
|
||||
|
||||
- **`insights[]`** — the generalizable learnings from Phase 1 (decisions/failures worth re-reading). Each: `{text, wing, hall}`; add `supersede_slug` to revise a prior note in place instead of creating a duplicate. `hall` ∈ facts/decisions/failures/hypotheses/sources.
|
||||
Optional claim envelope, per insight: `half_life` ∈ volatile/seasonal/durable, `applies` ∈ local/homelab/universal, `claim_provenance` ∈ asserted/inferred — required together whenever either `half_life` or `applies` is set (an envelope with no provenance is rejected). A close-session insight is usually exactly the durable, generalizable, human-decided kind this describes: `half_life: durable, applies: universal, claim_provenance: asserted` is the common case, not a rare opt-in.
|
||||
- **`tickets[]`** — the issue actions from Phase 3: `{repo, action, ...}` where action ∈ create/close/comment. Owner is always `mathias` (server-forced).
|
||||
- **`context`** — `{harness: "claudeai-chat", session_ref: <chatid8-or-slug>, fidelity: "live-capture", actor: "mathias", classification: <see gate below>}`.
|
||||
|
||||
**THE CLASSIFICATION GATE (read before calling — this is where capture refuses).**
|
||||
Capture computes an **effective classification = the strictest across EVERY target it touches** (each insight's `wing`, each ticket's `repo`), then refuses if that effective level is `confidential` and the origin is us-nexus (claude.ai is us-nexus). Levels come from `classification.yaml` at the brain root (source of truth, #67), with the code defaults as the floor: `hyperguild`/`homelab` → internal; `client-*` → confidential; **anything untagged → confidential (fail-safe)**.
|
||||
- **Tagged `internal` today** (safe through claude.ai): wings `hyperguild`, `homelab`; repos `brain`, `ai-sessions`, `infra`, `hyperguild`, `homelab`, `tapir`, `agentsquad`, `jepa-fx-risk`, `swedsl`. Treat `classification.yaml` as authoritative — this list is a hint, not gospel.
|
||||
- **The rule, not a list**: untagged → confidential (fail-safe). `classification.yaml` at the brain root is the sole authority on what's tagged `internal` today — it grew from 10 to ~39 repos in one sweep (hyperguild#91) and will keep moving, so read the file rather than trust an enumeration frozen at whatever moment this skill was last edited.
|
||||
- Declare `context.classification: "internal"` for normal homelab work.
|
||||
- Insight `wing`s and ticket `repo`s are classification INPUTS, not free-form metadata — every target must resolve `internal` or the whole capture escalates to `confidential` and the gate refuses via claude.ai. Listing the central homelab repos (incl. `brain`/`ai-sessions`) is now fine; they're tagged.
|
||||
- If a session genuinely touched **`client-*` or otherwise-untagged** material, it cannot be captured through claude.ai — note that in the verdict rather than trying to force it.
|
||||
|
||||
Reference in New Issue
Block a user