Compare commits

...
Author SHA1 Message Date
mathiasandClaude Opus 4.8 f04b03e07e chore(context): re-sync derived adapters after root rule-0 update
CI / Lint / Test / Vet (pull_request) Successful in 13s
CI / Mirror to GitHub (pull_request) Has been skipped
context-sync regenerated the adapters from the updated root AGENT.md
(rule 0 pre-task ritual + TDD constraint). The committed adapters had
drifted; this is the documented `task check` remedy, not a content
change in this repo.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 08:25:16 +02:00
mathiasandClaude Opus 4.8 6c61f93146 feat(mcp): register brain_update + brain_get, extend brain_write handle
Wires the #45 verbs into the MCP surface (all three sites: tools()
descriptors, handleCall dispatch, package doc comment).

- brain_update: supersede-by-slug or full path; rebuilds wing _index and
  re-tunnels cross-wing matches against the new body (idempotent,
  best-effort), re-indexes the graph, returns {id, path, content_hash,
  superseded}.
- brain_get: fetch by id or path (both are the brain-relative handle);
  returns {id, path, content_hash, frontmatter, body}.
- brain_write: return contract extended from {path} to {id, path,
  content_hash} — path kept for backward compat — so the create path
  also yields a stable handle.

id == relPath; content_hash == sha256 of the file bytes. Tests cover the
supersede happy path, missing-target error + no-create, get by id/path,
write handle, and an end-to-end re-embed test that drives the real
vectorstore.Sync re-index after an update.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 08:25:10 +02:00
mathiasandClaude Opus 4.8 95a69fc2c1 feat(brain): add UpdateNote/ReadNote supersede primitives + frontmatter editor
Implements the api-layer half of #45. UpdateNote supersedes a note in
place (whole-note body replace, frontmatter re-stamp: updated_at,
supersedes=prior content hash, supersede_reason), preserving created_at,
wing, hall, and any custom fields. Never creates — a missing target is
an error so callers fall back to brain_write. ReadNote is the read-after-
write primitive (frontmatter + body + content_hash). ContentHash is the
sha256 handle that round-trips write/update → get.

Frontmatter is edited via a line-preserving ordered editor rather than a
yaml.v3 round-trip, which would reorder keys and strip comments — the
brain writes flat key:value frontmatter by hand.

Embeddings are not refreshed here: the rewritten file's mtime advances,
which the out-of-band vectorstore.Sync ticker uses to re-embed it — the
same mechanism brain_write relies on.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 08:25:00 +02:00
mathiasandClaude Opus 4.8 bb8bc0478c chore(cd): use git.d-ma.be for SSH alias + INFRA_REPO (post-rename consistency)
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 4s
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 00:19:42 +02:00
mathiasandClaude Opus 4.8 a961a3c064 fix(vectorstore): hard-split oversized heading-less chunks
A doc with no headings and no blank-line paragraphs (JSON-lines, e.g.
wiki/telos/decisions/human-intent-column.md) survived both chunk passes whole
and was sent to nomic-embed over its context window → 'input length exceeds
the context length' (400, the steady embed errors=1). Add a final hard-split
pass (line then UTF-8 rune boundaries) so no chunk exceeds maxBytes. TDD.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 00:19:42 +02:00
mathiasandClaude Opus 4.8 bec28f9014 fix(cd): point registry/patch/verify refs at git.d-ma.be after host rename
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 3s
The gitea.d-ma.be→git.d-ma.be rename moved the infra deployment manifests, but
cd.yml still sed-patched 'gitea.d-ma.be/mathias/ingestion:' — which no longer
matches, so the patch produced no change and 'git commit' failed under set -e,
wedging all deploys (image stuck at e8dbcf6). Repoint the registry image refs,
the infra-patch sed patterns, and the rollout-verify EXPECTED values to
git.d-ma.be (same registry backend). SSH alias + INFRA_REPO left as-is.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 23:22:07 +02:00
mathiasandClaude Opus 4.8 b62ac57382 fix(brain_answer): reranker is a filter, not a gate — fall back to BM25 when it keeps nothing
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 3s
The Qwen3-Reranker is a web-search cross-encoder. Against conversational /
personal-intent queries (e.g. 'what am I optimizing toward?') it scores every
candidate as 'no', so brain_answer collapsed to 'No relevant content found'
even though BM25 had retrieved on-topic notes (incl. the telos wing). Treat
the reranker as a filter: when it keeps zero results, fall back to the
BM25/vector ordering (capped to the no-reranker depth of 10). Refs brain#11.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 22:53:02 +02:00
mathiasandClaude Opus 4.8 aa918388b9 docs(brain): add two-column intent merge scaffold
CI / Lint / Test / Vet (push) Successful in 14s
CI / Mirror to GitHub (push) Successful in 4s
Drop-in unified findings doc for the brain-MCP intent study. Locks the
shared row schema + closed intent vocab both columns must conform to.
Agent column filled from agent-intent-column.jsonl (46 acts, 37%
mismatch); human column left as PENDING cells + <<SYNTH>> blocks so the
Claude.ai-history analysis merges in without re-deriving structure.

Pre-seeds the cross-consumer divergence questions: agent mismatch is
write-side-heavy (supersede + verify-landed); hypothesis is human
mismatch is read-side-heavy (semantic + answer) — interface may fail the
two consumers at opposite ends.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 20:45:24 +02:00
mathiasandClaude Opus 4.8 e8dbcf6eef docs(brain): add agent-consumer brain-MCP intent analysis
CI / Lint / Test / Vet (push) Successful in 14s
CI / Mirror to GitHub (push) Successful in 4s
Agent-consumer column of the two-part brain intent↔interface study.
Reconstructs the knowledge-act behind every brain MCP call in the
Claude Code agent transcripts on koala (the only corpus with brain
calls; brain/sessions and agentsquad eval logs carry none).

46 distinct knowledge-acts, 37% interface mismatch, 0% intent_unclear.
Headline gap: no update/supersede verb → agents blind re-write same
slug (5x); no read-after-write → lexical re-query of own note (4x);
lexical-only reads → semantic-as-keyword-stuffing chains (3x);
brain_answer hedged with parallel brain_query.

Canonical schema brain-intent-extraction.md absent on host; vocab
reconstructed, every row tagged schema_source=reconstructed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 19:44:59 +02:00
mathiasandClaude Opus 4.8 0eeb1df4a2 fix(claudewatcher): scrub bare 1Password service-account tokens (ops_)
CI / Lint / Test / Vet (push) Successful in 17s
CI / Mirror to GitHub (push) Successful in 4s
A ~/.zshrc read surfaced OP_SERVICE_ACCOUNT_TOKEN into a transcript. The
_TOKEN= form was already caught by homelab-env-token, but a bare ops_<b64>
value was not. Add an op-service-account rule (ordered early). Tests cover
both env-assigned and bare forms.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 00:28:47 +02:00
mathiasandClaude Opus 4.8 9febb1bba1 fix(claudewatcher): harden secret scrubber against \b evasion + add JWT
CI / Lint / Test / Vet (push) Successful in 17s
CI / Mirror to GitHub (push) Successful in 4s
A shell mangle that glued a key to a preceding word ('yes'+'sk-...') had
no word boundary, so the leading \b in the openai-sk rule failed to match
and a LiteLLM master key leaked past the scrubber into ingest (2026-06-11).

- openai-sk: drop leading \b, match sk- shape anywhere ({32,} floor keeps
  short task-/disk- words clean).
- add jwt rule for bare header.payload.sig tokens (no Bearer prefix).
- regression tests: the exact yessk- evasion, standalone sk-, bare JWT,
  plus clean-content guards for task-/disk-.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 08:55:33 +02:00
mathias 5dc247b994 fix(ci): quote "on" key so Gitea parses workflow triggers
CI / Lint / Test / Vet (push) Successful in 17s
CI / Mirror to GitHub (push) Successful in 4s
Bare on: parses as YAML boolean true (Norway problem); Gitea then ignores the triggers and silently skips jobs. Quoting forces the string key.
2026-06-03 08:38:46 +02:00
mathias 2125558196 docs: extend harness boundary decision to cover Crush as third harness
CI / Mirror to GitHub (push) Successful in 4s
CI / Lint / Test / Vet (push) Successful in 12s
2026-05-28 11:38:36 +00:00
mathias 2beaac2feb docs: add 2026-05-28 decisions — harness boundary + field benchmark definition
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 3s
2026-05-28 10:31:41 +00:00
mathias 525811bc1a docs: add hypothesis statement and harness boundary clarification
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 4s
2026-05-28 10:30:39 +00:00
mathias bad0581623 merge: client-name scrubber rule (refs hyperguild#27)
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 4s
2026-05-26 07:10:05 +02:00
mathias a94b860c2e feat(claudewatcher): client-name guard via RegisterRule + env
Pre-rollout guard. Source code stays clean — client identities come
from CLAUDE_INGEST_CLIENT_BLOCK env (sourced from a SOPS-encrypted k8s
secret in infra repo). Env value is a regex alternation; main wraps
it with `(?i)\b(...)\b` so word-boundary matching avoids false hits
inside longer identifiers (e.g. "Sebastian" doesn't trigger on "SEB").

DefaultRules (credential shapes) still take precedence so any leak
that's BOTH a client mention AND a credential shape logs as the
credential — strictly more dangerous, points triage at the right
thing. Tests cover precedence + case variations + word-boundary
respect + invalid-pattern rejection.

Refs: infra#73 Track E.1 pre-rollout grill (option B).

Bump-Type: minor
2026-05-26 07:10:05 +02:00
mathias f8cf27e5de merge: claudewatcher (closes hyperguild#27, refs infra#73)
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 4s
2026-05-25 19:59:13 +02:00
mathias 49b188e9c9 feat(server): wire claudewatcher behind CLAUDE_SESSIONS_DIR
Opt-in by setting CLAUDE_SESSIONS_DIR to the ~/.claude/projects path.
When set, the server starts claudewatcher.Watch in a goroutine that
ticks every CLAUDE_INGEST_INTERVAL seconds (default 60). Requires
BRAIN_PG_DSN for the cursor table — fail-fast if missing.

Each Batch becomes one wiki note at:
  brain/wiki/claude-sessions/facts/session-<host>-<session_id>.md

with frontmatter type=source + domain=<project basename>. Per-turn
content capped at 2000 chars (full transcripts stay in
~/.claude/projects already); the brain entry is a digest, not a
mirror.

CLAUDE_INGEST_HOST overrides the os.Hostname()-derived host label,
useful when multiple ingestion pods consume the same DSN from
different machines.

Closes hyperguild#27.

Bump-Type: minor
2026-05-25 19:59:07 +02:00
mathias bc011cc1f0 feat(claudewatcher): ingest Claude Code session transcripts into brain
New package internal/claudewatcher. The volume gate (24 turns/week of
agentsquad logs vs 500/week gate) exposed that the real signal lives
in daily Claude Code usage at ~/.claude/projects/*/<uuid>.jsonl, not
in agentsquad output. This package captures that signal. See infra#73
Track E + hyperguild#27 for the full reframe.

Components:
- parser: tolerant JSONL parser over the observed Claude Code session
  schema (user / assistant / attachment / system + bookkeeping types).
  Skip-flag fast-paths queue-operation, last-prompt, permission-mode,
  ai-title, bridge-session, file-history-snapshot.
- scrubber: 11-rule fail-closed regex set for credential shapes
  (bearer, postgres URIs, PEM, ssh-key, ghp_/sk-/sk-ant-/AKIA, homelab
  env tokens, SOPS markers). Drop turn + log on match.
- cursor: postgres-backed claude_session_cursors table, keyed by
  (host, file_path) with byte_offset. Resumable across pod restarts.
- watcher: poll loop. Walks SessionsDir, processes each .jsonl from
  its cursor offset, runs scrubber, emits a Batch per file to a
  Sink interface, advances cursor on successful Ingest.

No classifier integration in this commit — every kept turn is emitted
in a per-session batch. The cmd/server wiring (next commit) routes
batches to brain/wiki/claude-sessions/facts/. Classifier-driven hall
routing (decisions / failures / hypotheses) is a follow-up.

19 unit tests across parser + scrubber + watcher. task check green.

Refs: infra#73, hyperguild#27
2026-05-25 19:58:58 +02:00
31 changed files with 2938 additions and 101 deletions
+54 -8
View File
@@ -27,6 +27,14 @@ and climate/sustainability tech.
These rules apply to every task across every project, regardless of harness.
0. **Pre-task ritual — before ANY implementation (non-negotiable).** Run this before writing a single line:
- **Query the brain** (`brain_query`) for the domain + symptom. If the result changes your approach, surface it before acting. 5 seconds beats 5 hours.
- **Load the relevant skill** — see trigger table in *Engineering Skills* below.
- **Write the failing test first.** Name the test before the function. If the target is untestable (e.g. `main()` wiring), extract the logic into a testable function first. No implementation without a red test.
- **State the observable success criterion** — what specific behavior, output, or passing test proves this is done?
**TDD is non-negotiable.** "Tests pass" is not proof of correctness — only proof the tests ran. Write tests that would catch the bug before writing code that fixes it.
1. **No assumptions.** Don't hide confusion — surface it. Surface tradeoffs explicitly.
Think before coding; if the problem is unclear, ask or state assumptions before acting.
2. **Minimum viable code.** Solve with the smallest change that works. Nothing
@@ -49,6 +57,22 @@ These rules apply to every task across every project, regardless of harness.
PR flow only when a human reviewer outside the project is required. Document
the reason in PROJECT.md.
6. **Close the loop — every substantive task ends with the same ritual.** Shipping
the code is not the end of the task; capturing it is. Run this unprompted:
- **Tag + bump SemVer** on the change (annotated tag; minor for a feature or
new/changed ADR, patch for a fix; docs in the same commit). Check the repo's
actual last tag — stated versions in docs drift stale.
- **Push** main and the tag (CI is the gate).
- **Persist generalizable learnings to the brain** (`brain_write`, wing/hall) —
the reusable patterns and the footguns that would bite anyone again, never
project status. See *Knowledge base — when to write* below.
- **File discovered-but-deferred work as tracker issues** on the project's own
repo — token-budget gaps, recorded ADR limitations, v2 follow-ups. Don't let
"out of scope, recorded" rot in a commit message; make it a ticket with a
source pointer.
- Surface the brain entries and issue numbers in the closing summary so the
trail is auditable.
## Default stack
| Layer | Default | Fallback | Last resort |
@@ -78,6 +102,26 @@ Exploratory: Rust, Zig — I'll tell you when I want these.
- **Security**: no secrets in code, govulncheck before adding deps, SOPS for encrypted config
- **Dependencies**: prefer stdlib. testify, slog, templ, sqlc, google.golang.org/adk (agent projects only) are pre-approved; anything else needs justification in the commit message
## Secret handling (every harness, every command)
Tool output is persisted: terminal → `~/.claude/projects` transcripts →
claudewatcher → brain/wiki → gitea history. A secret printed once is
searchable forever, and clearing it means rotating the key. So:
1. **Never print, echo, log, or transform a secret to inspect it.** No
`base64`/`xxd`/`cat` of a key, and never pipe a secret through a transform
to defeat `op run`'s output masking (it masks raw values; base64 hides them
from the mask — that exact trick leaked a key on 2026-06-11).
2. **Secrets stay in the subprocess.** Reference them only as env vars consumed
*inside* `op run --env-file ~/.op-env -- <cmd>`. Never place a literal secret
in a command's argv (it lands in the tool call and the transcript).
3. **Existence check without revealing the value:** `[ -n "$X" ] && echo set`
never `${X:-...}` (returns the value when set) and never echo a substring of it.
4. **Cross-host secrets:** run the secret-consuming command on the host that has
the secret; do not forward a raw key over ssh argv/stdout.
5. If a secret does leak into output, say so immediately and flag it for rotation —
don't bury it.
## Infrastructure
Three machines on Tailscale:
@@ -157,7 +201,7 @@ entries that age well are about *why*, *how to avoid*, and *what to do when*.
| **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool |
| **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same |
| **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` |
| **Browser / human inspection** | `https://gitea.d-ma.be/mathias/hyperguild``knowledge/` and `wiki/` markdown files |
| **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild``knowledge/` and `wiki/` markdown files |
- **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`.
- **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as
@@ -219,15 +263,17 @@ unconditionally on every host, every harness.
## Engineering Skills
Shared engineering skills are available in `~/dev/.skills/`. Load on demand via the index.
Shared engineering skills are available in `~/dev/.skills/`. Load at task start — not "on demand" but on schedule, before writing code. See `~/dev/.skills/SKILLS_INDEX.md` for the full list.
See `~/dev/.skills/SKILLS_INDEX.md` for the full list with descriptions and "use when" triggers.
**Skill trigger table — load before starting, not after getting stuck:**
Key skills:
- **TDD**: always write tests first — load `tdd` skill
- **Code Review**: load `code-review` skill before any review
- **SOLID/Clean Code**: load `solid` or `clean-code` skill for design work
- **Problem first**: load `problem-analysis` skill before coding non-trivial features
| Task type | Load |
|-----------|------|
| Any feature or bug fix | `tdd` |
| Refactor or design | `clean-code` or `solid` |
| Debug | `problem-analysis` |
| Review code or PRs | `code-review` |
| Frame a problem before coding | `problem-analysis` |
---
+54 -8
View File
@@ -32,6 +32,14 @@ and climate/sustainability tech.
These rules apply to every task across every project, regardless of harness.
0. **Pre-task ritual — before ANY implementation (non-negotiable).** Run this before writing a single line:
- **Query the brain** (`brain_query`) for the domain + symptom. If the result changes your approach, surface it before acting. 5 seconds beats 5 hours.
- **Load the relevant skill** — see trigger table in *Engineering Skills* below.
- **Write the failing test first.** Name the test before the function. If the target is untestable (e.g. `main()` wiring), extract the logic into a testable function first. No implementation without a red test.
- **State the observable success criterion** — what specific behavior, output, or passing test proves this is done?
**TDD is non-negotiable.** "Tests pass" is not proof of correctness — only proof the tests ran. Write tests that would catch the bug before writing code that fixes it.
1. **No assumptions.** Don't hide confusion — surface it. Surface tradeoffs explicitly.
Think before coding; if the problem is unclear, ask or state assumptions before acting.
2. **Minimum viable code.** Solve with the smallest change that works. Nothing
@@ -54,6 +62,22 @@ These rules apply to every task across every project, regardless of harness.
PR flow only when a human reviewer outside the project is required. Document
the reason in PROJECT.md.
6. **Close the loop — every substantive task ends with the same ritual.** Shipping
the code is not the end of the task; capturing it is. Run this unprompted:
- **Tag + bump SemVer** on the change (annotated tag; minor for a feature or
new/changed ADR, patch for a fix; docs in the same commit). Check the repo's
actual last tag — stated versions in docs drift stale.
- **Push** main and the tag (CI is the gate).
- **Persist generalizable learnings to the brain** (`brain_write`, wing/hall) —
the reusable patterns and the footguns that would bite anyone again, never
project status. See *Knowledge base — when to write* below.
- **File discovered-but-deferred work as tracker issues** on the project's own
repo — token-budget gaps, recorded ADR limitations, v2 follow-ups. Don't let
"out of scope, recorded" rot in a commit message; make it a ticket with a
source pointer.
- Surface the brain entries and issue numbers in the closing summary so the
trail is auditable.
## Default stack
| Layer | Default | Fallback | Last resort |
@@ -83,6 +107,26 @@ Exploratory: Rust, Zig — I'll tell you when I want these.
- **Security**: no secrets in code, govulncheck before adding deps, SOPS for encrypted config
- **Dependencies**: prefer stdlib. testify, slog, templ, sqlc, google.golang.org/adk (agent projects only) are pre-approved; anything else needs justification in the commit message
## Secret handling (every harness, every command)
Tool output is persisted: terminal → `~/.claude/projects` transcripts →
claudewatcher → brain/wiki → gitea history. A secret printed once is
searchable forever, and clearing it means rotating the key. So:
1. **Never print, echo, log, or transform a secret to inspect it.** No
`base64`/`xxd`/`cat` of a key, and never pipe a secret through a transform
to defeat `op run`'s output masking (it masks raw values; base64 hides them
from the mask — that exact trick leaked a key on 2026-06-11).
2. **Secrets stay in the subprocess.** Reference them only as env vars consumed
*inside* `op run --env-file ~/.op-env -- <cmd>`. Never place a literal secret
in a command's argv (it lands in the tool call and the transcript).
3. **Existence check without revealing the value:** `[ -n "$X" ] && echo set` —
never `${X:-...}` (returns the value when set) and never echo a substring of it.
4. **Cross-host secrets:** run the secret-consuming command on the host that has
the secret; do not forward a raw key over ssh argv/stdout.
5. If a secret does leak into output, say so immediately and flag it for rotation —
don't bury it.
## Infrastructure
Three machines on Tailscale:
@@ -162,7 +206,7 @@ entries that age well are about *why*, *how to avoid*, and *what to do when*.
| **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool |
| **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same |
| **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` |
| **Browser / human inspection** | `https://gitea.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files |
| **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files |
- **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`.
- **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as
@@ -224,15 +268,17 @@ unconditionally on every host, every harness.
## Engineering Skills
Shared engineering skills are available in `~/dev/.skills/`. Load on demand via the index.
Shared engineering skills are available in `~/dev/.skills/`. Load at task start — not "on demand" but on schedule, before writing code. See `~/dev/.skills/SKILLS_INDEX.md` for the full list.
See `~/dev/.skills/SKILLS_INDEX.md` for the full list with descriptions and "use when" triggers.
**Skill trigger table — load before starting, not after getting stuck:**
Key skills:
- **TDD**: always write tests first — load `tdd` skill
- **Code Review**: load `code-review` skill before any review
- **SOLID/Clean Code**: load `solid` or `clean-code` skill for design work
- **Problem first**: load `problem-analysis` skill before coding non-trivial features
| Task type | Load |
|-----------|------|
| Any feature or bug fix | `tdd` |
| Refactor or design | `clean-code` or `solid` |
| Debug | `problem-analysis` |
| Review code or PRs | `code-review` |
| Frame a problem before coding | `problem-analysis` |
---
+54 -8
View File
@@ -30,6 +30,14 @@ and climate/sustainability tech.
These rules apply to every task across every project, regardless of harness.
0. **Pre-task ritual — before ANY implementation (non-negotiable).** Run this before writing a single line:
- **Query the brain** (`brain_query`) for the domain + symptom. If the result changes your approach, surface it before acting. 5 seconds beats 5 hours.
- **Load the relevant skill** — see trigger table in *Engineering Skills* below.
- **Write the failing test first.** Name the test before the function. If the target is untestable (e.g. `main()` wiring), extract the logic into a testable function first. No implementation without a red test.
- **State the observable success criterion** — what specific behavior, output, or passing test proves this is done?
**TDD is non-negotiable.** "Tests pass" is not proof of correctness — only proof the tests ran. Write tests that would catch the bug before writing code that fixes it.
1. **No assumptions.** Don't hide confusion — surface it. Surface tradeoffs explicitly.
Think before coding; if the problem is unclear, ask or state assumptions before acting.
2. **Minimum viable code.** Solve with the smallest change that works. Nothing
@@ -52,6 +60,22 @@ These rules apply to every task across every project, regardless of harness.
PR flow only when a human reviewer outside the project is required. Document
the reason in PROJECT.md.
6. **Close the loop — every substantive task ends with the same ritual.** Shipping
the code is not the end of the task; capturing it is. Run this unprompted:
- **Tag + bump SemVer** on the change (annotated tag; minor for a feature or
new/changed ADR, patch for a fix; docs in the same commit). Check the repo's
actual last tag — stated versions in docs drift stale.
- **Push** main and the tag (CI is the gate).
- **Persist generalizable learnings to the brain** (`brain_write`, wing/hall) —
the reusable patterns and the footguns that would bite anyone again, never
project status. See *Knowledge base — when to write* below.
- **File discovered-but-deferred work as tracker issues** on the project's own
repo — token-budget gaps, recorded ADR limitations, v2 follow-ups. Don't let
"out of scope, recorded" rot in a commit message; make it a ticket with a
source pointer.
- Surface the brain entries and issue numbers in the closing summary so the
trail is auditable.
## Default stack
| Layer | Default | Fallback | Last resort |
@@ -81,6 +105,26 @@ Exploratory: Rust, Zig — I'll tell you when I want these.
- **Security**: no secrets in code, govulncheck before adding deps, SOPS for encrypted config
- **Dependencies**: prefer stdlib. testify, slog, templ, sqlc, google.golang.org/adk (agent projects only) are pre-approved; anything else needs justification in the commit message
## Secret handling (every harness, every command)
Tool output is persisted: terminal → `~/.claude/projects` transcripts →
claudewatcher → brain/wiki → gitea history. A secret printed once is
searchable forever, and clearing it means rotating the key. So:
1. **Never print, echo, log, or transform a secret to inspect it.** No
`base64`/`xxd`/`cat` of a key, and never pipe a secret through a transform
to defeat `op run`'s output masking (it masks raw values; base64 hides them
from the mask — that exact trick leaked a key on 2026-06-11).
2. **Secrets stay in the subprocess.** Reference them only as env vars consumed
*inside* `op run --env-file ~/.op-env -- <cmd>`. Never place a literal secret
in a command's argv (it lands in the tool call and the transcript).
3. **Existence check without revealing the value:** `[ -n "$X" ] && echo set` —
never `${X:-...}` (returns the value when set) and never echo a substring of it.
4. **Cross-host secrets:** run the secret-consuming command on the host that has
the secret; do not forward a raw key over ssh argv/stdout.
5. If a secret does leak into output, say so immediately and flag it for rotation —
don't bury it.
## Infrastructure
Three machines on Tailscale:
@@ -160,7 +204,7 @@ entries that age well are about *why*, *how to avoid*, and *what to do when*.
| **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool |
| **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same |
| **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` |
| **Browser / human inspection** | `https://gitea.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files |
| **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files |
- **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`.
- **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as
@@ -222,15 +266,17 @@ unconditionally on every host, every harness.
## Engineering Skills
Shared engineering skills are available in `~/dev/.skills/`. Load on demand via the index.
Shared engineering skills are available in `~/dev/.skills/`. Load at task start — not "on demand" but on schedule, before writing code. See `~/dev/.skills/SKILLS_INDEX.md` for the full list.
See `~/dev/.skills/SKILLS_INDEX.md` for the full list with descriptions and "use when" triggers.
**Skill trigger table — load before starting, not after getting stuck:**
Key skills:
- **TDD**: always write tests first — load `tdd` skill
- **Code Review**: load `code-review` skill before any review
- **SOLID/Clean Code**: load `solid` or `clean-code` skill for design work
- **Problem first**: load `problem-analysis` skill before coding non-trivial features
| Task type | Load |
|-----------|------|
| Any feature or bug fix | `tdd` |
| Refactor or design | `clean-code` or `solid` |
| Debug | `problem-analysis` |
| Review code or PRs | `code-review` |
| Frame a problem before coding | `problem-analysis` |
---
+9 -9
View File
@@ -1,6 +1,6 @@
name: cd
on:
"on":
workflow_run:
workflows: ["CI"]
types: [completed]
@@ -13,9 +13,9 @@ jobs:
if: ${{ github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.event == 'push' }}
environment: staging
env:
INGESTION_IMAGE: gitea.d-ma.be/mathias/ingestion
ROUTING_IMAGE: gitea.d-ma.be/mathias/routing
INFRA_REPO: git@gitea.d-ma.be:mathias/infra.git
INGESTION_IMAGE: git.d-ma.be/mathias/ingestion
ROUTING_IMAGE: git.d-ma.be/mathias/routing
INFRA_REPO: git@git.d-ma.be:mathias/infra.git
BUILDKIT_HOST: unix:///run/buildkit/buildkitd.sock
steps:
- name: Checkout
@@ -71,17 +71,17 @@ jobs:
mkdir -p ~/.ssh
echo "${{ secrets.INFRA_DEPLOY_KEY }}" > ~/.ssh/infra_deploy_key
chmod 600 ~/.ssh/infra_deploy_key
printf 'Host gitea.d-ma.be\n HostName 127.0.0.1\n Port 30022\n StrictHostKeyChecking no\n' >> ~/.ssh/config
printf 'Host git.d-ma.be\n HostName 127.0.0.1\n Port 30022\n StrictHostKeyChecking no\n' >> ~/.ssh/config
GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes" \
git clone "${INFRA_REPO}" /tmp/infra-update
cd /tmp/infra-update
sed -i "s|gitea.d-ma.be/mathias/ingestion:.*|gitea.d-ma.be/mathias/ingestion:${IMAGE_TAG}|" \
sed -i "s|git.d-ma.be/mathias/ingestion:.*|git.d-ma.be/mathias/ingestion:${IMAGE_TAG}|" \
"k3s/apps/supervisor/ingestion-deployment.yaml"
sed -i "s|gitea.d-ma.be/mathias/routing:.*|gitea.d-ma.be/mathias/routing:${IMAGE_TAG}|" \
sed -i "s|git.d-ma.be/mathias/routing:.*|git.d-ma.be/mathias/routing:${IMAGE_TAG}|" \
"k3s/apps/routing/deployment.yaml"
git config user.email "cd-bot@d-ma.be"
@@ -103,7 +103,7 @@ jobs:
- name: Wait for Flux to apply new ingestion image
run: |
EXPECTED="gitea.d-ma.be/mathias/ingestion:${{ github.sha }}"
EXPECTED="git.d-ma.be/mathias/ingestion:${{ github.sha }}"
for i in $(seq 1 60); do
CURRENT=$(kubectl get deploy ingestion -n supervisor \
-o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null || echo "")
@@ -135,7 +135,7 @@ jobs:
- name: Wait for Flux to apply new routing image
run: |
EXPECTED="gitea.d-ma.be/mathias/routing:${{ github.sha }}"
EXPECTED="git.d-ma.be/mathias/routing:${{ github.sha }}"
for i in $(seq 1 60); do
CURRENT=$(kubectl get deploy routing -n routing \
-o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null || echo "")
+1 -1
View File
@@ -1,6 +1,6 @@
name: CI
on:
"on":
push:
branches: [main]
tags: ["v*"]
+54 -8
View File
@@ -27,6 +27,14 @@ and climate/sustainability tech.
These rules apply to every task across every project, regardless of harness.
0. **Pre-task ritual — before ANY implementation (non-negotiable).** Run this before writing a single line:
- **Query the brain** (`brain_query`) for the domain + symptom. If the result changes your approach, surface it before acting. 5 seconds beats 5 hours.
- **Load the relevant skill** — see trigger table in *Engineering Skills* below.
- **Write the failing test first.** Name the test before the function. If the target is untestable (e.g. `main()` wiring), extract the logic into a testable function first. No implementation without a red test.
- **State the observable success criterion** — what specific behavior, output, or passing test proves this is done?
**TDD is non-negotiable.** "Tests pass" is not proof of correctness — only proof the tests ran. Write tests that would catch the bug before writing code that fixes it.
1. **No assumptions.** Don't hide confusion — surface it. Surface tradeoffs explicitly.
Think before coding; if the problem is unclear, ask or state assumptions before acting.
2. **Minimum viable code.** Solve with the smallest change that works. Nothing
@@ -49,6 +57,22 @@ These rules apply to every task across every project, regardless of harness.
PR flow only when a human reviewer outside the project is required. Document
the reason in PROJECT.md.
6. **Close the loop — every substantive task ends with the same ritual.** Shipping
the code is not the end of the task; capturing it is. Run this unprompted:
- **Tag + bump SemVer** on the change (annotated tag; minor for a feature or
new/changed ADR, patch for a fix; docs in the same commit). Check the repo's
actual last tag — stated versions in docs drift stale.
- **Push** main and the tag (CI is the gate).
- **Persist generalizable learnings to the brain** (`brain_write`, wing/hall) —
the reusable patterns and the footguns that would bite anyone again, never
project status. See *Knowledge base — when to write* below.
- **File discovered-but-deferred work as tracker issues** on the project's own
repo — token-budget gaps, recorded ADR limitations, v2 follow-ups. Don't let
"out of scope, recorded" rot in a commit message; make it a ticket with a
source pointer.
- Surface the brain entries and issue numbers in the closing summary so the
trail is auditable.
## Default stack
| Layer | Default | Fallback | Last resort |
@@ -78,6 +102,26 @@ Exploratory: Rust, Zig — I'll tell you when I want these.
- **Security**: no secrets in code, govulncheck before adding deps, SOPS for encrypted config
- **Dependencies**: prefer stdlib. testify, slog, templ, sqlc, google.golang.org/adk (agent projects only) are pre-approved; anything else needs justification in the commit message
## Secret handling (every harness, every command)
Tool output is persisted: terminal → `~/.claude/projects` transcripts →
claudewatcher → brain/wiki → gitea history. A secret printed once is
searchable forever, and clearing it means rotating the key. So:
1. **Never print, echo, log, or transform a secret to inspect it.** No
`base64`/`xxd`/`cat` of a key, and never pipe a secret through a transform
to defeat `op run`'s output masking (it masks raw values; base64 hides them
from the mask — that exact trick leaked a key on 2026-06-11).
2. **Secrets stay in the subprocess.** Reference them only as env vars consumed
*inside* `op run --env-file ~/.op-env -- <cmd>`. Never place a literal secret
in a command's argv (it lands in the tool call and the transcript).
3. **Existence check without revealing the value:** `[ -n "$X" ] && echo set`
never `${X:-...}` (returns the value when set) and never echo a substring of it.
4. **Cross-host secrets:** run the secret-consuming command on the host that has
the secret; do not forward a raw key over ssh argv/stdout.
5. If a secret does leak into output, say so immediately and flag it for rotation —
don't bury it.
## Infrastructure
Three machines on Tailscale:
@@ -157,7 +201,7 @@ entries that age well are about *why*, *how to avoid*, and *what to do when*.
| **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool |
| **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same |
| **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` |
| **Browser / human inspection** | `https://gitea.d-ma.be/mathias/hyperguild``knowledge/` and `wiki/` markdown files |
| **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild``knowledge/` and `wiki/` markdown files |
- **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`.
- **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as
@@ -219,15 +263,17 @@ unconditionally on every host, every harness.
## Engineering Skills
Shared engineering skills are available in `~/dev/.skills/`. Load on demand via the index.
Shared engineering skills are available in `~/dev/.skills/`. Load at task start — not "on demand" but on schedule, before writing code. See `~/dev/.skills/SKILLS_INDEX.md` for the full list.
See `~/dev/.skills/SKILLS_INDEX.md` for the full list with descriptions and "use when" triggers.
**Skill trigger table — load before starting, not after getting stuck:**
Key skills:
- **TDD**: always write tests first — load `tdd` skill
- **Code Review**: load `code-review` skill before any review
- **SOLID/Clean Code**: load `solid` or `clean-code` skill for design work
- **Problem first**: load `problem-analysis` skill before coding non-trivial features
| Task type | Load |
|-----------|------|
| Any feature or bug fix | `tdd` |
| Refactor or design | `clean-code` or `solid` |
| Debug | `problem-analysis` |
| Review code or PRs | `code-review` |
| Frame a problem before coding | `problem-analysis` |
---
+68
View File
@@ -4,6 +4,74 @@ Record *why* things are the way they are. Future-you will thank present-you.
---
## 2026-05-28 — three active harnesses: hyperguild, agentsquad, Crush (extends earlier boundary decision)
**Context:** After wiring Crush to LiteLLM in May 2026, there are now three active harnesses.
The earlier boundary decision only covered hyperguild vs agentsquad. Crush's role was undefined.
**Decision:** Three harnesses, three distinct roles, shared skills layer.
| Harness | Engine | Primary use | Brain MCP? | Routing pod? | Skills? |
|---------|--------|-------------|------------|--------------|---------|
| **hyperguild** | Claude Code + MCP | Disciplined solo coding sessions, TDD/review/debug workflows | Yes | Yes | Yes (SKILL.md) |
| **agentsquad** | OpenCode + LiteLLM | Multi-agent task execution, executor/reviewer pipelines | No | No (own routing) | Yes (SKILL.md) |
| **Crush** | Charmbracelet TUI + LiteLLM | Interactive local coding, quick iterations on flamingo | No (not yet) | No (direct LiteLLM) | Yes (SKILL.md) |
**Crush specifics (as of 2026-05-28):**
- Config: `~/.config/crush/crush.json` on flamingo (see brain: `homelab/facts/crush-litellm-wiring-2026-05`)
- Connects directly to LiteLLM at `http://koala:4000/v1/` using `sk-local-123`
- Auth type: `openai-compat` (not `openai`)
- Does NOT go through the routing pod — model selection is manual in the Crush UI
- Brain MCP not wired — Crush has no MCP client capability today; revisit if Crush adds MCP support
**Shared across all three:**
- `mathias/skills` — any SKILL.md file works in all three harnesses
- LiteLLM proxy on koala (`http://koala:4000/v1/`) — Crush and agentsquad both route through it; hyperguild does too for local model calls
**Consequences:** No consolidation needed. crush.json must be kept in sync when litellm_config.yaml model names change. The `crush.json` canonical location is `~/.config/crush/crush.json` on flamingo — not yet tracked in a dotfiles repo (track as tech debt).
---
## 2026-05-28 — "field benchmark" for local models = pass-rate at scale (supersedes GOTTH eval suite)
**Context:** The GOTTH eval suite (45 offline prompts across 5 categories) was replaced by
a "field benchmark" in May 2026, but the replacement was never defined concretely.
**Decision:** The field benchmark is per-skill pass rate over real routing pod usage,
collected automatically by `internal/routing/passrate.go` and exposed at:
```
GET /pass-rate?skill=<name>&window=<duration>
```
No separate eval suite. No synthetic prompts. The benchmark runs itself once the routing
pod receives real traffic. Target: 30-day rolling window per skill, reviewed monthly.
**Bootstrap note:** With no session history, `passrate.go` returns `nil` and the router
defaults to the thinking model for every call. The fast-model path activates only after
real pass-rate data accumulates. Seed with real usage — do not pre-populate.
**Consequences:** Zero maintenance overhead for the benchmark. The tradeoff is that results
are only meaningful after ~2 weeks of real usage, and skills that are rarely invoked will
have statistically thin pass-rate data. Revisit if a skill has fewer than 20 calls in 30 days.
---
## 2026-05-28 — brain injection in skill handlers: review is done, others unverified
**Context:** The April 2026 scope reset listed "brain_query injection into skill handlers"
as the top priority. As of 2026-05-28, `internal/skills/review/handlers.go` calls
`brain.Query(ctx, ...)` before dispatching to the LLM — confirmed in code review.
Status of debug, retrospective, and trainer handlers is unverified.
**Decision:** Treat review as the reference implementation. Verify debug, retrospective,
trainer against the same pattern before shipping new skill work. Tracked in issue #32.
**Consequences:** The April concern may be stale for review. A one-pass audit of the other
three skill handlers closes this fully.
---
## 2026-04-08 — AGENTS.md as cross-tool standard, not CLAUDE.md
**Context**: Multiple tools (Crush, Pi, Antigravity) read `AGENTS.md` natively. Claude Code reads `CLAUDE.md`. Building on `CLAUDE.md` as the primary format locks into one vendor.
+74 -51
View File
@@ -5,14 +5,31 @@ Instead of letting Claude Code do whatever it wants, hyperguild enforces structu
workflows (TDD red/green/refactor), logs every session, and accumulates learnings
into a searchable brain.
## Hypothesis
> We believe routing skill tasks through local models, backed by brain context,
> produces measurably better outcomes than raw Claude Code alone —
> measurable by per-skill pass rate over rolling 30-day windows
> (available at `GET /pass-rate?skill=<name>&window=30d` on the brain pod).
This is the falsifiable claim the routing pod and pass-rate infrastructure exist to test.
If per-skill pass rates don't improve over baseline (all-cloud) after 30 days of real
usage, the fast-model routing path should be reconsidered.
## Harness
**hyperguild = Claude Code + MCP.** This is a supervisor for Claude Code sessions specifically.
For multi-agent orchestration (OpenCode + LiteLLM, executor/reviewer pipelines), see
[agentsquad](http://gitea.d-ma.be/mathias/agentsquad) — a separate harness for a different
orchestration model. Skills (mathias/skills) are shared between both.
## How it works
```
Your Claude Code session (in any project)
│ MCP over HTTP (Tailscale)
├──▶ supervisor :3200 (NodePort 30320 on koala) — skill workers: tdd, debug, spec, …
├──▶ routing :3210 (NodePort 30310 on koala) — Mode 2 only: review, debug, retrospective, trainer
├──▶ routing :3210 (NodePort 30310 on koala) — review, debug, retrospective, trainer
└──▶ brain :3300 (NodePort 30330 on koala) — brain_query, brain_write, brain_ingest, session_log
└─ also serves the legacy REST endpoints (/query, /write, /ingest, …)
@@ -20,34 +37,28 @@ Your Claude Code session (in any project)
brain/
├── sessions/ — JSONL log, one file per session_id
├── wiki/ — searchable knowledge (full-text)
│ ├── concepts/
│ ├── entities/
│ └── sources/
── raw/ — retrospective output, staged for review
└── training-data/ — SFT/DPO/RL data (Phase 2)
├── wiki/ — searchable knowledge (wing/hall layout)
│ ├── homelab/
│ ├── claude-sessions/
│ └── ...
── knowledge/ — legacy flat notes (migration pending: hyperguild#22)
```
## Phase 1 tools (available now)
| Tool | What it does |
|------|-------------|
| `tdd_red` | Writes a failing test for a spec, verifies it fails |
| `tdd_green` | Writes the minimal implementation to make tests pass |
| `tdd_refactor` | Cleans up implementation while keeping tests green |
| `session_log` | Appends a structured entry to the session JSONL log |
| `retrospective` | Reads the session log, identifies novel learnings, writes to brain/raw/ |
| `retrospective` | Reads the session log, identifies novel learnings, writes to brain |
| `review` | Structured code review via local model, brain-context injected |
| `debug` | Hypothesis-driven debugging via local model |
| `brain_query` | Full-text search over brain/wiki/ |
| `brain_write` | Writes a note to brain/raw/ (with optional YAML frontmatter) |
| `brain_write` | Writes a note to brain (with wing/hall routing) |
| `brain_answer` | BM25 + LLM synthesis — Q&A over brain corpus |
| `tier` | Returns the current connectivity tier (1=cloud, 2=LAN, 3=offline) |
## Start the servers
```bash
# Requires goreman: go install github.com/mattn/goreman@latest
task start # starts ingestion (:3300) + supervisor (:3200) via goreman
task stop # kills both by port
```
> **Note:** `tdd_red/green/refactor` and `spec` were retired in Plan 7 (2026-05-12).
> They are now SKILL.md files in [mathias/skills](http://gitea.d-ma.be/mathias/skills).
## Connect a project
@@ -56,9 +67,9 @@ Create `.mcp.json` in your project root:
```json
{
"mcpServers": {
"supervisor": {
"routing": {
"type": "http",
"url": "http://koala:30320/mcp"
"url": "http://koala:30310/mcp"
},
"brain": {
"type": "http",
@@ -68,65 +79,77 @@ Create `.mcp.json` in your project root:
}
```
Two MCP servers are exposed today, both reachable over Tailscale:
Two MCP servers are exposed, both reachable over Tailscale:
- **`supervisor`** at `koala:30320` — skill workers (`tdd_red/green/refactor`,
`review`, `debug`, `spec`, `retrospective`, `trainer`, `tier`).
- **`routing`** at `koala:30310` — skill workers (`review`, `debug`, `retrospective`, `trainer`).
Routes each call to fast local model or thinking model based on per-skill pass rate.
- **`brain`** at `koala:30330` — knowledge access (`brain_query`, `brain_write`,
`brain_ingest`, `brain_ingest_raw`) and `session_log`. Hosted by the ingestion
service directly, no separate pod.
`brain_ingest`, `brain_ingest_raw`, `brain_answer`, `brain_classify`) and `session_log`.
No local binary or stdio shim is required — Claude Code talks to both via HTTP.
Open Claude Code in your project — run `/mcp` to confirm both servers are listed.
## A typical TDD session
## A typical session
```
1. Call tdd_red → spec in, failing test file out
2. Call tdd_green → test path in, implementation out
3. Call tdd_refactor → impl + test in, cleaned code out
4. Call session_log → log each phase result
5. Call retrospective → extracts learnings → brain/raw/
6. Review brain/raw/, move worthy notes to brain/wiki/concepts/
7. Future sessions: call brain_query to retrieve relevant context
1. Call review → brain context injected + local model review → findings
2. Call session_log → log each phase result
3. Call retrospective → extracts learnings → brain
4. Future sessions: call brain_query / brain_answer to retrieve relevant context
```
## Tier detection
The supervisor probes connectivity at call time:
The routing pod probes connectivity at call time:
| Tier | Label | Condition |
|------|-------|-----------|
|------|-------|-----------|
| 1 | full-online | Can reach api.anthropic.com |
| 2 | lan-only | Can reach LiteLLM but not Anthropic |
| 3 | airplane | No external connectivity |
## Model routing
The routing pod selects models per skill call based on historical pass rate:
| Pass rate | Decision |
|-----------|----------|
| ≥ 0.90 (FLOOR) | Fast model (`HYPERGUILD_FAST_MODEL`) |
| ≤ 0.70 (CEIL) | Thinking model (`HYPERGUILD_THINKING_MODEL`) |
| between CEIL and FLOOR | Sample band — probabilistic routing |
| nil (no history yet) | Defaults to thinking model |
> **Bootstrap note:** With no session history, all calls route to the thinking model.
> The fast-model path activates only after real pass-rate data accumulates at `/pass-rate`.
> Seed with real usage — don't try to pre-populate.
## Key env vars
| Variable | Default | Purpose |
|----------|---------|---------|
|----------|---------|---------|
| `INGEST_BRAIN_DIR` | `../brain` | Brain directory for ingestion server |
| `INGEST_PORT` | `3300` | Ingestion server port |
| `SUPERVISOR_CONFIG_DIR` | `./config/supervisor` | Skill discipline files |
| `SUPERVISOR_SESSIONS_DIR` | `./brain/sessions` | JSONL session logs |
| `INGEST_BASE_URL` | `http://localhost:3300` | Supervisor → ingestion |
| `INGEST_BASE_URL` | `http://localhost:3300` | Routing pod → brain |
| `LITELLM_BASE_URL` | — | LiteLLM proxy for Tier 2 model routing |
| `SUPERVISOR_MCP_TOKEN` | — | Optional bearer token for the supervisor MCP HTTP endpoint; when empty, no auth is enforced |
| `ROUTING_PORT` | `3210` | Routing pod's listen port |
| `ROUTING_MCP_TOKEN` | — | Optional bearer token for the routing MCP HTTP endpoint |
| `ROUTING_MCP_TOKEN` | — | Optional bearer token; when empty, no auth enforced |
| `BRAIN_URL` | `http://ingestion.supervisor:3300` | Routing pod → brain (in-cluster) |
| `HYPERGUILD_FAST_MODEL` | `koala/qwen35-9b-fast` | Fast model for high-pass-rate skill calls |
| `HYPERGUILD_THINKING_MODEL` | `iguana/gemma4-26b` | Thinking model for low-pass-rate skill calls |
| `HYPERGUILD_ROUTE_LOCAL_FLOOR` | `0.90` | At/above pass rate, route to fast model |
| `HYPERGUILD_ROUTE_LOCAL_CEIL` | `0.70` | Below pass rate, route to thinking model. Between CEIL and FLOOR is the sample band. |
| `HYPERGUILD_ROUTE_LOCAL_FLOOR` | `0.90` | Fast model threshold |
| `HYPERGUILD_ROUTE_LOCAL_CEIL` | `0.70` | Thinking model threshold |
| `HYPERGUILD_PASS_RATE_TTL_SECONDS` | `60` | Per-skill pass-rate cache TTL |
> **Operator note:** LiteLLM at `LITELLM_BASE_URL` must register both `HYPERGUILD_FAST_MODEL` and `HYPERGUILD_THINKING_MODEL` for routing to do useful work. If a model is missing, LiteLLM returns 4xx, the routing pod's fast route fails, the fail-open retry on the thinking model likely also fails (since both are missing), and the only signal is `final_status: "fail"` on `_routing` entries in the brain.
> **Operator note:** LiteLLM at `LITELLM_BASE_URL` must register both `HYPERGUILD_FAST_MODEL`
> and `HYPERGUILD_THINKING_MODEL`. If a model is missing, the fail-open retry also fails and
> the only signal is `final_status: "fail"` on `_routing` entries in the brain.
## Phase 2 (planned)
## Open issues
- `review` skill — structured code review with iron law enforcement
- `debug` skill — hypothesis-driven debugging sessions
- `spec` skill — generates specs from conversations
- `trainer` — extracts SFT/DPO pairs from session logs for fine-tuning
See [issues](http://gitea.d-ma.be/mathias/hyperguild/issues) — key open items:
- **#25** — skills platform overhaul (audit first, then lazy loading + brain feedback loop)
- **#24** — reduce context burn from skill listing
- **#22** — migrate legacy brain notes to wing/hall layout (one-shot script, low risk)
- **#31** — connect routing-mcp to claude.ai as custom connector
@@ -0,0 +1,48 @@
{"_meta":true,"note":"Agent-consumer column of brain-MCP intent analysis. consumer_type fixed=autonomous_agent. CAVEAT: canonical schema file brain-intent-extraction.md is NOT present on this host (koala) — only this session's own task prompt references it. The closed intent vocabulary below was RECONSTRUCTED from the task prompt's framing + brain/schema.md. Re-map intent labels if the canonical vocab differs. schema_source=reconstructed on every row.","closed_intent_vocab":["semantic_retrieval","lexical_lookup","check_prior_art","synthesized_answer","store_new_knowledge","update_or_supersede","ingest_raw_source","verify_write_landed","discover_capability","intent_unclear"],"intent_tool_match_values":["match","mismatch","partial"],"corpus":"~/.claude/projects/*/*.jsonl (Claude Code agent transcripts on koala). brain/sessions/*.jsonl empty. agentsquad docs/eval/*.jsonl are code-review eval results, NOT brain calls. No separate Crush logs found. Zero brain calls appear under any mcp__ name with a human typing the call — all brain acts are agent-initiated (CLAUDE.md reflex), so all qualify as autonomous_agent."}
{"id":"a01","session":"tapir-c","ts":"2026-06-?T15:01:51","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"single pre-task query 'YouTube Data API captions download ownership limitation timedtext adapter Go' — named-entity lexical lookup, fit BM25 well, no reformulation."}
{"id":"a02","session":"tapir","ts":"2026-06-05T21:44:10","tool":"brain_ingest","intent":"ingest_raw_source","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_ingest 14s earlier — tool not ambient, had to be discovered/loaded first.","evidence":"source=tapir-scheduled-discovery-session-2026-06-05, a session learnings dump."}
{"id":"a03","session":"tapir","ts":"2026-06-?T14:00:59","tool":"brain_ingest","intent":"ingest_raw_source","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"source=tapir-rls-identity-bootstrapping, first write of RLS lesson."}
{"id":"a04","session":"tapir","ts":"2026-06-?T14:02:08","tool":"brain_ingest","intent":"update_or_supersede","intent_tool_match":"mismatch","workaround":"RE-INGESTED same source name 'tapir-rls-identity-bootstrapping' 69s later with edited/condensed body. No update/patch/supersede verb exists, so the agent overwrote-by-re-ingest. Whether this dedups or creates a v2 duplicate is opaque to the agent.","observed_friction":"agent revised content within 70s of first write — classic edit-after-write with no edit primitive.","evidence":"two brain_ingest, identical source string, divergent content."}
{"id":"a05","session":"tapir","ts":"2026-06-?T14:56:27","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"postgres-cascade-skips-tables-without-fk.md — distinct new lesson."}
{"id":"a06","session":"tapir","ts":"2026-06-?T21:08:57","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_query — discovery tax again.","evidence":"'Dex passwords.dex.coreos.com CRD ...' keyword-rich, single shot."}
{"id":"a07","session":"AI-infra","ts":"2026-05-?T15:38:03","tool":"brain_query(HTTP-curl)","intent":"discover_capability","intent_tool_match":"mismatch","workaround":"raw `curl -X POST` to brain-mcp endpoint instead of MCP tool. Preceded by two ToolSearch ('brain knowledge memory' then 'brain') that did not yield a usable loaded tool, so agent fell back to HTTP.","observed_friction":"3-step ladder: ToolSearch 'brain knowledge memory' -> ToolSearch 'brain' -> curl. Agent did not know which act maps to which tool name.","evidence":"curl -s -o /tmp/brain-init.txt -w code:%{http_code} -X POST ..."}
{"id":"a08","session":"AI-infra","ts":"2026-05-?T04:46:13","tool":"brain_query(HTTP-curl)","intent":"discover_capability","intent_tool_match":"mismatch","workaround":"hand-set TOKEN=... then curl brain-test endpoint — probing whether the HTTP brain path is reachable/authed at all. MCP path not used.","observed_friction":"agent testing connectivity by hand; MCP auth/availability not trusted.","evidence":"TOKEN=...; curl -s -o /tmp/brain-test ..."}
{"id":"a09","session":"AI-infra","ts":"2026-05-?T05:23:35","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_query (after an earlier 05:03 ToolSearch 'brain ingestion knowledge wiki' that explored layers).","evidence":"'koala machine state RTX 5070 llama-swap' — named-entity recall, fits lexical."}
{"id":"a10","session":"AI-infra","ts":"2026-05-?T05:27:36","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"batch of 3 in ~1s (k3s/flux gitops; llama-swap ai-stack GPU; MCP Dex OAuth claude.ai) — parallel prior-art sweep, all named-entity."}
{"id":"a11","session":"AI-infra","ts":"2026-05-?T07:13:50","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"batch of 4 in ~2s before a debugging session (flux healthCheck; exit 255 restart loop; NVML mismatch; mirror rebase). Named symptoms, lexical fit OK on first pass."}
{"id":"a12","session":"AI-infra","ts":"2026-05-?T07:14:26","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"4 writes in ~35s (flux-healthcheck-stale; exit-255-unknown-reason-not-oom; nvidia-nvml-mismatch; mcp-static-bearer) — answers to the 4 queries just run, captured as lessons. Healthy query->fix->write loop."}
{"id":"a13","session":"AI-infra","ts":"2026-05-?T07:15:25","tool":"brain_query","intent":"verify_write_landed","intent_tool_match":"mismatch","workaround":"25s after writing exit-255-unknown-reason-not-oom.md, re-queried 'exit 255 unknown SIGKILL containerd' — reformulated terms (SIGKILL/containerd not in original query 'exit 255 unknown reason restart loop diagnosis'). Either confirming the fresh write is retrievable or re-searching because first lexical query missed. No read-after-write / get-by-id act exists.","observed_friction":"reformulation chain: 'exit 255 unknown reason restart loop diagnosis' -> 'exit 255 unknown SIGKILL containerd'. Same need, different keywords.","evidence":"query at 07:13:51 vs 07:15:25 bracketing the 07:14:37 write."}
{"id":"a14","session":"AI-infra","ts":"2026-05-?T07:22:55","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"batch of 5 in ~20s before a homelab security audit (piguard/iguana tailscale; unifi UCG firewall; SOPS age; ingress TLS cert-manager; koala UFW iptables). Named-entity sweep."}
{"id":"a15","session":"AI-infra","ts":"2026-05-?T09:27:53","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"audit-shortcut-tls-blocks-zero; policy-audit-mode-blocks-nothing — distinct new audit lessons."}
{"id":"a16","session":"AI-infra","ts":"2026-05-?T09:28:22","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"homelab-security-chains-not-bugs.md FIRST write (worked example: koala 2026-05-13)."}
{"id":"a17","session":"AI-infra","ts":"2026-05-?T10:32:49","tool":"brain_write","intent":"update_or_supersede","intent_tool_match":"mismatch","workaround":"RE-WROTE homelab-security-chains-not-bugs.md ~64min later with a different/expanded worked example (host-user dotfile, over-broad ClusterRole). Same filename, additive revision, no patch/append/supersede verb — agent overwrites and hopes the index replaces rather than duplicates.","observed_friction":"the in-between hour of audit work produced a better example; only way to fold it in was a full re-write of the same slug.","evidence":"two brain_write same filename at 09:28:22 and 10:32:49, divergent worked examples."}
{"id":"a18","session":"AI-infra","ts":"2026-05-?T10:18:25","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"homelab-document-accepted-risk-to-break-audit-cycle.md — distinct."}
{"id":"a19","session":"AI-infra","ts":"2026-05-?T10:32:55","tool":"brain_query","intent":"verify_write_landed","intent_tool_match":"mismatch","workaround":"6s after the homelab-chains re-write, queried 'RBAC MCP cluster pods log chain' — checking the chain reasoning is retrievable / finding the related entry. Read-after-write done via lexical search.","observed_friction":null,"evidence":"query immediately follows the 10:32:49 write."}
{"id":"a20","session":"AI-infra","ts":"2026-05-?T18:57:34","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_write,brain_query — re-discovered tools this session.","evidence":"'extension build pinned version major version upgrade postgres pgvector'."}
{"id":"a21","session":"AI-infra","ts":"2026-05-?T18:57:58","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"extension-version-lags-platform-major-upgrade.md."}
{"id":"a22","session":"AI-infra","ts":"2026-05-?T18:58:06","tool":"brain_query","intent":"verify_write_landed","intent_tool_match":"mismatch","workaround":"8s after writing extension-version-lags, re-queried 'pgvector postgres extension version compile error bump' — reformulated from the 18:57:34 query ('extension build pinned version...'). Lexical re-search to confirm the just-written lesson is findable, with different keyword guess.","observed_friction":"reformulation: 'extension build pinned version major version upgrade postgres pgvector' -> 'pgvector postgres extension version compile error bump'.","evidence":"write at 18:57:58 bracketed by queries 18:57:34 and 18:58:06."}
{"id":"a23","session":"AI-infra","ts":"2026-05-?T18:31:23","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"webfetch-readme-when-image-or-flag-uncertain.md FIRST write."}
{"id":"a24","session":"AI-infra","ts":"2026-05-?T18:34:23","tool":"brain_write","intent":"update_or_supersede","intent_tool_match":"mismatch","workaround":"RE-WROTE webfetch-readme-when-image-or-flag-uncertain.md 3min later, near-identical body. Looks like a retry/overwrite (uncertain the first landed, or minor edit). No idempotent upsert with confirmation, so agent re-fires the write.","observed_friction":"followed 7s later by a brain_query on the same topic ('OSS tool image registry CLI flag webhook path schema drift README pre-flight') — write-write-query, i.e. overwrite then verify-by-search.","evidence":"two brain_write same filename 18:31:23 / 18:34:23, then query 18:34:31."}
{"id":"a25","session":"AI-infra","ts":"2026-05-?T18:34:31","tool":"brain_query","intent":"verify_write_landed","intent_tool_match":"mismatch","workaround":"keyword-stuffed lexical query 'OSS tool image registry CLI flag webhook path schema drift README pre-flight' fired right after the webfetch-readme write — agent dumps every concept token hoping BM25 surfaces its own fresh note. This is semantic intent (find that conceptual lesson) coerced into a bag-of-keywords.","observed_friction":"query is a concatenation of the note's section headings — a tell that the agent is groping lexically for content it knows by meaning.","evidence":"query text mirrors the just-written note's bullet topics."}
{"id":"a26","session":"dev","ts":"2026-06-?T21:18:26","tool":"brain_answer","intent":"synthesized_answer","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_answer.","evidence":"'tapir transcript persistence shared cross-user dedup table RLS isolation ADR-021 ...' -> 22min later a brain_write (acted on the answer). Answer consumed, not re-queried. Healthy."}
{"id":"a27","session":"dev","ts":"2026-06-?T21:40:20","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"tapir-migration-and-rls-test-infra-gotchas, with wing/hall absent here (flat) — see schema-confusion note a40."}
{"id":"a28","session":"dev","ts":"2026-06-?T05:35:04","tool":"brain_answer","intent":"synthesized_answer","intent_tool_match":"partial","workaround":"asked 'gitea MCP not working workaround file issue via API which token ... how to authenticate gitea API' — a how-do-I question. Next brain act (05:39 query) is a different topic (tapir transcript), so the answer was apparently sufficient OR abandoned; ambiguous.","observed_friction":null,"evidence":"brain_answer then unrelated brain_query 4min later."}
{"id":"a29","session":"dev","ts":"2026-06-?T05:39:54","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"'tapir transcript persistence ADR-021 shared non-RLS'."}
{"id":"a30","session":"dev","ts":"2026-06-?T05:57:37","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"gitea-mcp-per-repo-tools-404-and-rest-fallback FIRST write."}
{"id":"a31","session":"dev","ts":"2026-06-?T05:57:55","tool":"brain_write","intent":"update_or_supersede","intent_tool_match":"mismatch","workaround":"RE-WROTE gitea-mcp-per-repo-tools-404-and-rest-fallback 18s later — overwrite/retry of same slug, no upsert confirmation.","observed_friction":"sub-20s gap = almost certainly a content tweak the agent could not express as an edit.","evidence":"two brain_write same filename 05:57:37 / 05:57:55."}
{"id":"a32","session":"dev","ts":"2026-05-?T11:51:47","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"infra-litellm-absorption-2026-05-16.md."}
{"id":"a33","session":"dev","ts":"2026-05-?T12:07:04","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"batch of 3 (litellm rebuild time piguard; docker compose orphaned volumes; prometheus_client ModuleNotFoundError) — lexical, error-string driven."}
{"id":"a34","session":"dev","ts":"2026-05-?T15:08:15","tool":"brain_answer","intent":"synthesized_answer","intent_tool_match":"mismatch","workaround":"THREE brain_answer at 15:08 (moved compose volumes? / pi rebuild time? / litellm ModuleNotFound prometheus) — the SAME three topics queried lexically an hour earlier (12:07) — were IMMEDIATELY followed at 15:09 by THREE brain_query on the same three topics. The agent asked the synthesizer, was unsatisfied, and fell straight back to raw lexical search. Strongest answer->query fallback in the corpus.","observed_friction":"answer/query duplication across one intent: agent hedges by firing both interfaces, trusting neither.","evidence":"15:08 answers vs 15:09 queries, topic-for-topic aligned."}
{"id":"a35","session":"dev","ts":"2026-05-?T15:09:16","tool":"brain_query","intent":"semantic_retrieval","intent_tool_match":"mismatch","workaround":"after the 3 brain_answer calls failed to satisfy, re-issued as lexical brain_query ('moved compose stack to new directory volumes disappeared empty'; 'raspberry pi docker build time arm slow'; 'how to enable prometheus metrics on litellm proxy callback'). The want is meaning-based ('did my volumes move?') but the only retrieval that 'worked' was keyword search — and these are full natural-language sentences crammed into a BM25 box.","observed_friction":"natural-language questions ('how to enable...', 'moved ... disappeared') passed to a lexical query tool — semantic intent, lexical interface.","evidence":"3 queries at 15:09 mirror the 3 answers at 15:08."}
{"id":"a36","session":"dev","ts":"2026-05-?T20:31:50","tool":"brain_answer","intent":"synthesized_answer","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"'What happened with the litellm migration on 2026-05-16?' — episodic recall question, answer fit; no re-query followed."}
{"id":"a37","session":"dev","ts":"2026-05-?T21:07:17","tool":"brain_query","intent":"semantic_retrieval","intent_tool_match":"mismatch","workaround":"FOUR-step reformulation chain over one Go bug: 'bytes.Buffer Bytes Reset aliasing slice sharing' -> 'go buffer reuse map backing array bug' -> [write go-bytes-buffer-bytes-reset-aliasing-trap.md] -> 'go map values all show same content after loop' -> 'bytes.Buffer Bytes returns same data every iteration'. The agent knows the SYMPTOM (all map values identical) and the CAUSE (Bytes() aliasing) but cannot phrase a single lexical query that bridges them — it wants concept retrieval and is forced to brute-force keyword variants.","observed_friction":"4 distinct phrasings of the same bug, two before and two after writing the lesson — also doubles as verify_write_landed on the trailing queries.","evidence":"21:07:17, 21:07:17, (write 21:08:01), 21:08:10, 21:08:19."}
{"id":"a38","session":"dev","ts":"2026-05-?T21:08:01","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"go-bytes-buffer-bytes-reset-aliasing-trap.md."}
{"id":"a39","session":"dev","ts":"2026-05-?T07:54:26","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"mcp-tool-design-get-needs-list-partner.md — a design principle."}
{"id":"a40","session":"dev","ts":"2026-06-?T06:25:00","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"'Dex to Authentik migration auth.d-ma.be issuer cutover OIDC subject ...'."}
{"id":"a41","session":"dev","ts":"2026-06-?T06:25:08","tool":"brain_answer","intent":"synthesized_answer","intent_tool_match":"mismatch","workaround":"brain_query (a40) and brain_answer (a41) fired ~8s apart on the SAME intent (Dex->Authentik subject-keyed token orphan). Agent runs lexical search AND synthesized answer in parallel for one question rather than choosing — it cannot predict which interface will return usable knowledge, so it pays both.","observed_friction":"query+answer doublet on one need.","evidence":"06:25:00 query then 06:25:08 answer, same topic."}
{"id":"a42","session":"dev","ts":"2026-06-?T13:48:49","tool":"brain_query","intent":"verify_write_landed","intent_tool_match":"mismatch","workaround":"'authentik cutover validation probe' then 6min later 'authentik cutover post-flip validation' — reformulated pair, likely searching for the agent's own earlier cutover notes / confirming validation steps are recorded. Lexical re-search standing in for recall-my-recent-context.","observed_friction":"reformulation: 'validation probe' -> 'post-flip validation'.","evidence":"13:48:49 and 13:54:56."}
{"id":"a43","session":"dev","ts":"2026-06-?T13:59:17","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_write.","evidence":"oidc-issuer-host-change-vs-idp-swap-subject."}
{"id":"a44","session":"dev","ts":"2026-06-?T13:59:50","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"cannot-move-ingress-host-across-namespaces-flux-dryrun FIRST write."}
{"id":"a45","session":"dev","ts":"2026-06-?T14:00:13","tool":"brain_write","intent":"update_or_supersede","intent_tool_match":"mismatch","workaround":"RE-WROTE cannot-move-ingress-host-across-namespaces-flux-dryrun 23s later — overwrite of same slug, no edit/upsert primitive.","observed_friction":"sub-30s gap = content correction expressed as a full re-write.","evidence":"two brain_write same filename 13:59:50 / 14:00:13."}
{"id":"a46","session":"dev","ts":"2026-06-?T13:53:54","tool":"brain_write(HTTP-staged)","intent":"store_new_knowledge","intent_tool_match":"mismatch","workaround":"after `ToolSearch select:mcp__claude_ai_brain__authenticate` (MCP auth flow), the agent staged the entry as `cat > /tmp/brain_entry.json` ({filename:'postgres-force-rls-cross-u...', content}) for a curl write rather than calling brain_write directly — MCP write path was not usable (auth/loading), so it dropped to the HTTP bodge.","observed_friction":"reached for an 'authenticate' tool, then abandoned MCP for hand-built JSON + curl. Matches known pattern: brain/op MCP auth lapses often.","evidence":"ToolSearch authenticate 13:53:27 -> cat /tmp/brain_entry.json 13:53:54."}
{"id":"a47","session":"template-go-agent","ts":"2026-05-?T18:46:26","tool":"BASH(not-a-brain-act)","intent":"intent_unclear","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"'brain_' substring was inside a git commit message body ('agent boundaries, network policy, agent s...'), NOT a brain call. Excluded from knowledge-act analysis; logged for audit completeness."}
@@ -0,0 +1,148 @@
# Agent-Consumer Brain Intent Analysis — koala column
**Consumer:** `autonomous_agent` (all rows). **Host:** koala. **Date:** 2026-06-15.
**Raw rows:** `agent-intent-column.jsonl` (46 real knowledge-acts + 1 excluded false-positive).
## Caveat — canonical schema not on this host
The shared closed-vocabulary file `brain-intent-extraction.md` **does not exist on
koala** — the only reference to it is inside *this task's own prompt*. The intent
vocabulary below was **reconstructed** from the prompt's framing + `brain/schema.md`.
Every row carries `schema_source: reconstructed`. If the canonical vocab differs,
re-map the `intent` field; the `intent_tool_match` / `workaround` / `observed_friction`
evidence stands regardless of label names.
**Reconstructed closed vocab:** `semantic_retrieval`, `lexical_lookup`,
`check_prior_art`, `synthesized_answer`, `store_new_knowledge`, `update_or_supersede`,
`ingest_raw_source`, `verify_write_landed`, `discover_capability`, `intent_unclear`.
## Corpus
- `~/.claude/projects/*/*.jsonl` — Claude Code agent transcripts (98 files). **The only
source with brain calls.**
- `brain/sessions/*.jsonl` — empty (only `.gitkeep`).
- `agentsquad docs/eval/*.jsonl` — code-review eval results, **not** brain calls.
- No separate Crush session logs on this host.
- **Zero** brain calls were human-typed. Every brain act is agent-initiated (the
CLAUDE.md "query as reflex / close-the-loop write" behaviour), so all qualify as
`autonomous_agent`. The human gave the top-level task; the agent chose every brain act.
## 1. Intent histogram, split by `intent_tool_match`
| intent | match | mismatch | partial | total |
|---|---|---|---|---|
| check_prior_art | 10 | 0 | 0 | 10 |
| store_new_knowledge | 14 | 1 | 0 | 15 |
| update_or_supersede | 0 | 5 | 0 | 5 |
| synthesized_answer | 2 | 2 | 1 | 5 |
| verify_write_landed | 0 | 4 | 0 | 4 |
| semantic_retrieval | 0 | 3 | 0 | 3 |
| ingest_raw_source | 2 | 0 | 0 | 2 |
| discover_capability | 0 | 2 | 0 | 2 |
| **total** | **28** | **17** | **1** | **46** |
> Batch note: several rows collapse a same-second fan-out of identical-intent calls
> (a10=3, a11=4, a14=5, a33=3, a34=3, a35=3). Call-level the corpus is ~62 brain calls;
> the table counts the 46 distinct knowledge-acts. Frequency is deliberately *not* the
> point — the mismatch column is.
**37% of agent knowledge-acts (17/46) are interface mismatches.** Every mismatch falls
into one of four intents: `update_or_supersede`, `verify_write_landed`,
`semantic_retrieval`, `discover_capability` — plus one `store` that had to use HTTP.
## 2. Mismatch list, grouped by intent (primary deliverable)
### update_or_supersede → re-write same slug (5/5 mismatch) — HIGHEST VALUE
There is **no update / patch / append / supersede verb**. When an agent improves a note
it already wrote, the only move is to call `brain_write`/`brain_ingest` **again with the
same filename/source** and hope the index replaces rather than duplicates. Observed:
| slug | 1st write | 2nd write | gap | what changed |
|---|---|---|---|---|
| `tapir-rls-identity-bootstrapping` (ingest) | 14:00:59 | 14:02:08 | 69s | condensed body |
| `homelab-security-chains-not-bugs.md` | 09:28:22 | 10:32:49 | 64m | new worked example |
| `webfetch-readme-when-image-or-flag-uncertain.md` | 18:31:23 | 18:34:23 | 3m | near-identical (retry) |
| `gitea-mcp-per-repo-tools-404-and-rest-fallback` | 05:57:37 | 05:57:55 | 18s | content tweak |
| `cannot-move-ingress-host-across-namespaces-flux-dryrun` | 13:59:50 | 14:00:13 | 23s | content tweak |
Sub-30s gaps (3 of 5) read as "I wanted to edit but can only overwrite." The agent has
no way to know whether the second write deduped or created a contradictory v2 — opacity
the brain's own design principle (`mcp-tool-design-get-needs-list-partner.md`, written
*by one of these very agents*) would flag: every `_write` needs a `_get`/`_update` partner.
### verify_write_landed → lexical re-query (4/4 mismatch)
No read-after-write / get-by-id confirmation. After every substantive write, agents
re-query lexically to check the note is retrievable — and *reformulate the keywords*
because they can't predict what BM25 indexed:
- `exit-255` lesson: query `exit 255 unknown reason restart loop diagnosis` → write →
query `exit 255 unknown SIGKILL containerd`.
- `extension-version-lags`: query `extension build pinned version...pgvector` → write →
query `pgvector postgres extension version compile error bump`.
- `webfetch-readme`: write → write → query stuffed with the note's own section headings.
### semantic_retrieval → BM25 keyword-stuffing (3/3 mismatch)
Agent knows the *meaning* but not the *indexed words*, so it brute-forces phrasings of
one need against a lexical tool:
- **4-step chain on one Go bug:** `bytes.Buffer Bytes Reset aliasing slice sharing`
`go buffer reuse map backing array bug` → (write) → `go map values all show same
content after loop``bytes.Buffer Bytes returns same data every iteration`. Symptom
and cause both known; no single lexical query bridges them.
- Natural-language questions (`how to enable prometheus metrics on litellm proxy
callback`, `moved compose stack to new directory volumes disappeared empty`) shoved
into `brain_query`.
### synthesized_answer → fall back to / hedge with brain_query (2 mismatch + 1 partial)
`brain_answer` is frequently **not trusted as terminal**:
- **Strongest signal:** 3× `brain_answer` at 15:08 (compose volumes / pi rebuild time /
litellm ModuleNotFound) → 3× `brain_query` at 15:09 on the *same three topics*. The
agent asked the synthesizer, was unsatisfied, and immediately re-ran raw search.
- Dex→Authentik: `brain_query` and `brain_answer` fired **8s apart on one question** —
the agent pays both interfaces because it can't predict which returns usable knowledge.
- (Counter-examples exist: `brain_answer` for episodic recall — "what happened with the
litellm migration on 2026-05-16?" — was consumed and not re-queried. So `answer`
works for *episodic/temporal* recall, fails for *how-do-I / does-X-hold* reasoning.)
### discover_capability + store-via-HTTP (3 mismatch)
brain tools are **not ambient** — they are deferred and must be `ToolSearch`-loaded each
session. Agents fumble the discovery (`ToolSearch 'brain knowledge memory'` →
`'brain'` → `'brain ingestion knowledge wiki'`) and, when MCP load/auth fails, drop to
**raw `curl` against `brain-mcp` / hand-built `/tmp/brain_entry.json`**. One agent even
`ToolSearch`-ed an `authenticate` tool, then abandoned MCP for the HTTP bodge — matching
the known "brain/op MCP auth lapses too often" footgun.
### Write-interface / layer schema confusion (cross-cutting)
`brain_write` was called with **three different param shapes** in the same corpus:
`{filename, type:"lesson", content}`, `{filename, content}` (no type), and
`{wing:"tapir", hall:"failures", filename, content}` — plus `brain_ingest {source,
content}`. Agents are unsure which verb and which layer (flat slug vs `wing`/`hall`
knowledge routing vs raw ingest) a given knowledge-act maps to. This is the
`knowledge/ vs wiki/` confusion expressed at the parameter level.
## 3. `intent_unclear` rate
**0 / 46 genuine brain acts (0%).** Agent intent is unusually legible because these are
Claude Code transcripts: the surrounding task, the query/filename strings, and the
write content all disambiguate. One row (`a47`) was tagged `intent_unclear` and
**excluded** — its `brain_` substring was inside a git commit message, not a brain call.
Example of the only ambiguity that arose: a `brain_answer` on "gitea MCP not working...
how to authenticate" followed by an unrelated query — can't tell if the answer satisfied
or was abandoned (`partial`, row a28).
## 4. The single biggest intent↔interface gap
**The brain offers one write verb and one lexical read verb, but autonomous agents
perform four distinct knowledge-acts against them — and three of the four have no fitting
interface.** The deepest gap is the **missing update/supersede path**: agents close every
task by writing a lesson (the CLAUDE.md ritual), routinely improve it minutes-to-an-hour
later, and — having no edit primitive — re-write the same slug blind, unable to tell
whether they corrected the entry or forked a contradiction into the index. This compounds
with the lexical-only read side: because there is no `get-by-id` or semantic retrieval,
agents can't even reliably *find their own just-written note* to check it, so they
keyword-stuff reformulated queries and hedge `brain_answer` with parallel `brain_query`.
The interface is built for *append-and-keyword-search*; the agents are trying to
*curate a living, deduplicated knowledge base*, and the seam between those two shows up
as the 5 blind re-writes, 4 read-after-write re-queries, and 3 semantic-as-lexical chains
that dominate the mismatch column.
---
*Evidence-only per task scope — no redesign proposed.*
@@ -0,0 +1,140 @@
# Brain-MCP Intent↔Interface Findings — Unified (two-column merge)
**Status — 2026-06-16**
-**Agent column** filled from `agent-intent-column.jsonl` (46 acts, koala).
-**Human column** = `PENDING`. Drop the Claude.ai-history analysis into
`human-intent-column.jsonl` (same dir, schema below), then fill the `PENDING`
cells and the synthesis blocks marked `<<SYNTH>>`.
- ⚠️ Canonical `brain-intent-extraction.md` still absent on koala. Vocab below is
the **reconstructed** lock both columns must share. If the real file surfaces,
re-map `intent` labels in *both* columns identically before merging.
---
## Shared schema (LOCKED — both columns conform)
Per-call row, JSONL:
| field | values / form | notes |
|---|---|---|
| `id` | `a01..` (agent) / `h01..` (human) | column prefix kept distinct |
| `session` | string | source session/conversation id |
| `ts` | ISO-8601 | best-effort |
| `tool` | brain tool name (+ `(HTTP-curl)` / `(HTTP-staged)` suffix for bodges) | |
| `intent` | closed vocab ↓ | the knowledge-act WANTED |
| `intent_tool_match` | `match` \| `mismatch` \| `partial` | does the called tool fit the want |
| `consumer_type` | `autonomous_agent` \| `human_interactive` | fixed per column |
| `workaround` | string \| null | the bodge when mismatch — **primary signal** |
| `observed_friction` | string \| null | reformulation chains, discovery tax, hedging |
| `evidence` | string | excerpt anchoring the classification |
| `schema_source` | `reconstructed` | flip to `canonical` if real vocab lands |
### Closed intent vocab (LOCKED)
`semantic_retrieval`, `lexical_lookup`, `check_prior_art`, `synthesized_answer`,
`store_new_knowledge`, `update_or_supersede`, `ingest_raw_source`,
`verify_write_landed`, `discover_capability`, `intent_unclear`.
---
## Master comparison — by intent
| intent | agent acts | agent mismatch | human acts | human mismatch | shared gap |
|---|---|---|---|---|---|
| check_prior_art | 10 | 0% | `PENDING` | `PENDING` | — |
| store_new_knowledge | 15 | 7% (1/15) | `PENDING` | `PENDING` | `<<SYNTH>>` |
| update_or_supersede | 5 | **100%** (5/5) | `PENDING` | `PENDING` | `<<SYNTH>>` no edit verb |
| synthesized_answer | 5 | 40% (2/5)+1 partial | `PENDING` | `PENDING` | `<<SYNTH>>` |
| verify_write_landed | 4 | **100%** (4/4) | `PENDING` | `PENDING` | `<<SYNTH>>` no read-after-write |
| semantic_retrieval | 3 | **100%** (3/3) | `PENDING` | `PENDING` | `<<SYNTH>>` lexical-only read |
| ingest_raw_source | 2 | 0% | `PENDING` | `PENDING` | — |
| discover_capability | 2 | **100%** (2/2) | `PENDING` | `PENDING` | agent-specific (ToolSearch/auth)? |
| intent_unclear | 0 | — | `PENDING` | `PENDING` | divergence expected ↓ |
| **TOTAL** | **46** | **37% (17)** | `PENDING` | `PENDING` | |
---
## Per-intent merged findings
### update_or_supersede — agent: 5/5 mismatch (highest value)
**Agent:** no edit/patch/append verb. Agents re-write same slug blind:
`homelab-security-chains-not-bugs.md` (+64m), `tapir-rls-identity-bootstrapping`,
`webfetch-readme...`, `gitea-mcp-per-repo-tools-404...`,
`cannot-move-ingress-host...` — 3 of 5 sub-30s ("wanted edit, got overwrite").
Cannot tell if write deduped or forked a contradiction.
**Human:** `PENDING` — *look for: user editing a prior note, asking "update what I
saved about X", or expressing frustration that an old fact is stale/duplicated.*
**<<SYNTH>>** shared verdict once both filled.
### verify_write_landed — agent: 4/4 mismatch
**Agent:** no `get-by-id`/read-after-write. Agents lexically re-query their own
fresh note with reformulated keywords (`exit 255 unknown reason``...SIGKILL
containerd`; `extension build pinned...``pgvector ...compile error bump`).
**Human:** `PENDING` — *humans may not exhibit this (they trust the write UI
confirmation). If absent in human column, it's an agent-specific gap → flag.*
**<<SYNTH>>**.
### semantic_retrieval — agent: 3/3 mismatch
**Agent:** meaning known, indexed words unknown → BM25 keyword-stuffing. 4-step
chain on one Go `bytes.Buffer` bug; NL questions shoved into `brain_query`.
**Human:** `PENDING` — *humans likely hit this HARDER (they phrase conversationally).
Compare reformulation-chain length agent vs human.*
**<<SYNTH>>** — likely the strongest cross-consumer overlap.
### synthesized_answer — agent: 2 mismatch + 1 partial
**Agent:** `brain_answer` not trusted terminal — 3 answers → 3 same-topic queries
1min later; query+answer fired 8s apart hedging one need. Works for *episodic*
recall, fails for *how-do-I / does-X-hold*.
**Human:** `PENDING` — *humans may prefer `brain_answer` as primary (chat-native).
If human match-rate >> agent, the tool fits humans not agents → key divergence.*
**<<SYNTH>>**.
### store_new_knowledge — agent: 14/15 match
**Agent:** healthy, except 1 HTTP-staged bodge when MCP auth lapsed. Also surfaced
write-schema confusion: 3 param shapes (`{filename,type}` / `{filename}` /
`{wing,hall,filename}`) + `ingest{source}`.
**Human:** `PENDING`*humans rarely write directly; expect low volume.*
**<<SYNTH>>**.
### check_prior_art / ingest_raw_source — agent: 0% mismatch
Lexical fits named-entity recall and raw-source capture. **Human:** `PENDING`.
### discover_capability — agent: 2/2 mismatch (agent-specific)
Brain tools deferred → `ToolSearch`-load each session; auth lapse → `curl` bodge.
**Likely has NO human analog** (humans get ambient connectors). Candidate for
"agent-only gap" bucket. **Human:** `PENDING` to confirm absent.
---
## Cross-consumer divergence — questions to resolve at merge
1. **intent_unclear rate.** Agent = 0% (transcripts self-document). Human expected
higher (conversational, implicit). Big delta = the columns measure legibility
differently, not just intent.
2. **Where does each consumer's mismatch concentrate?** Agent mismatch is
write-side-heavy (supersede + verify-landed = 9/17). Hypothesis: human mismatch
is read-side-heavy (semantic + answer). If true → **the interface fails the two
consumers at opposite ends.**
3. **Agent-only gaps** (`discover_capability`, `verify_write_landed`) vs
**shared gaps** (`semantic_retrieval`, `update_or_supersede`). Shared gaps =
highest-priority evidence; agent-only = harness/auth issues.
---
## Combined headline — `<<SYNTH>>` (fill when human column lands)
> Agent-side draft (to be reconciled with human-side):
> Brain = append + keyword-search; agents want a curated, dedup'd, self-verifying KB.
> Missing update/supersede path + lexical-only reads are the seam. **Open question
> for the merge: do humans hit the same read-side wall, making semantic-retrieval the
> universal gap — or do agents uniquely suffer the write-side (supersede / verify)
> wall that humans sidestep via the chat UI?**
---
## Drop-in checklist (when human column arrives)
1. Place `human-intent-column.jsonl` in this dir; conform to LOCKED schema.
2. Fill every `PENDING` cell in master table + per-intent blocks.
3. Resolve the 3 divergence questions with evidence.
4. Replace each `<<SYNTH>>` with the reconciled verdict; write the combined headline.
5. If canonical vocab surfaced: re-map both columns' `intent`, flip `schema_source`.
6. Commit as `docs(brain): merge human+agent intent columns`.
+105
View File
@@ -15,6 +15,7 @@ import (
chassisauth "gitea.d-ma.be/mathias/mcp-chassis/auth"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
"github.com/mathiasbq/hyperguild/ingestion/internal/claudewatcher"
"github.com/mathiasbq/hyperguild/ingestion/internal/embed"
"github.com/mathiasbq/hyperguild/ingestion/internal/graphstore"
"github.com/mathiasbq/hyperguild/ingestion/internal/graphsync"
@@ -29,6 +30,50 @@ import (
"github.com/mathiasbq/hyperguild/ingestion/internal/watcher"
)
// claudeSink converts each claudewatcher.Batch into one wiki note under
// brain/wiki/claude-sessions/facts/. v1 emits one note per session
// keyed by host + session id; classifier-driven hall routing is a
// follow-up (hyperguild#27 v2).
type claudeSink struct {
brainDir string
logger *slog.Logger
}
func (s *claudeSink) Ingest(ctx context.Context, b claudewatcher.Batch) error {
if len(b.Turns) == 0 {
return nil
}
var sb strings.Builder
fmt.Fprintf(&sb, "# Claude session %s (%s)\n\n", b.SessionID, b.Host)
fmt.Fprintf(&sb, "_Project: `%s`. File: `%s`. Turns: %d._\n\n", b.ProjectID, b.FilePath, len(b.Turns))
for _, t := range b.Turns {
fmt.Fprintf(&sb, "## %s — %s\n\n", t.Type, t.Timestamp.UTC().Format(time.RFC3339))
if t.ToolName != "" {
fmt.Fprintf(&sb, "_tool: `%s`_\n\n", t.ToolName)
}
// Cap per-turn excerpt to keep page size bounded; the full
// transcript lives on disk under ~/.claude/projects/ already.
content := t.Content
if len(content) > 2000 {
content = content[:2000] + "…"
}
sb.WriteString(content)
sb.WriteString("\n\n")
}
slug := "session-" + b.Host + "-" + b.SessionID
if _, err := api.WriteNote(s.brainDir, api.WriteNoteOptions{
Filename: slug,
Wing: "claude-sessions",
Hall: "facts",
Type: "source",
Domain: b.ProjectID,
Content: sb.String(),
}); err != nil {
return fmt.Errorf("write claude session note: %w", err)
}
return nil
}
// redactDSN parses a Postgres URL and replaces its password with `***`
// for safe inclusion in logs. Falls back to a non-leaking placeholder
// if parsing fails — we never log a raw DSN.
@@ -73,6 +118,16 @@ func envInt(key string, fallback int) int {
return fallback
}
// systemHostname returns os.Hostname() with a "unknown" fallback so the
// caller never has to handle the rare error path.
func systemHostname() string {
h, err := os.Hostname()
if err != nil || h == "" {
return "unknown"
}
return h
}
func main() {
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
@@ -191,6 +246,56 @@ func main() {
Pipeline: pipelineCfg,
})
}
// Claude Code session ingestion (hyperguild#27 / infra#73 Track E.1).
// Off by default — explicitly opt in by setting CLAUDE_SESSIONS_DIR
// to the ~/.claude/projects path. Requires BRAIN_PG_DSN for the
// cursor table (resumable offsets across restarts).
if claudeDir := os.Getenv("CLAUDE_SESSIONS_DIR"); claudeDir != "" {
if pgDSN == "" {
logger.Error("CLAUDE_SESSIONS_DIR set but BRAIN_PG_DSN missing — claudewatcher needs the cursor table")
os.Exit(1)
}
// Client-name guard. The env value is a regex alternation
// (e.g. "SEB|Mastercard"); we wrap it with word boundaries
// and case-insensitive flag so substrings inside longer
// identifiers don't false-match. Sourced from a SOPS secret
// so client identities never live in source.
if clientBlock := os.Getenv("CLAUDE_INGEST_CLIENT_BLOCK"); clientBlock != "" {
pattern := `(?i)\b(` + clientBlock + `)\b`
if err := claudewatcher.RegisterRule("client-name", pattern); err != nil {
logger.Error("claudewatcher client-block rule invalid", "err", err)
os.Exit(1)
}
logger.Info("claudewatcher client-block guard registered")
}
cursorStore, cerr := claudewatcher.NewCursorStore(ctx, pgDSN)
if cerr != nil {
logger.Error("claudewatcher cursor init", "err", cerr)
os.Exit(1)
}
if cerr := cursorStore.Init(ctx); cerr != nil {
logger.Error("claudewatcher cursor migrate", "err", cerr)
os.Exit(1)
}
host := envOr("CLAUDE_INGEST_HOST", systemHostname())
interval := time.Duration(envInt("CLAUDE_INGEST_INTERVAL", 60)) * time.Second
sink := &claudeSink{brainDir: brainDir, logger: logger}
go func() {
if err := claudewatcher.Watch(ctx, claudewatcher.Config{
SessionsDir: claudeDir,
Host: host,
Interval: interval,
Sink: sink,
Cursors: cursorStore,
Logger: logger,
}); err != nil && err != context.Canceled {
logger.Error("claudewatcher exited", "err", err)
}
}()
logger.Info("claudewatcher started",
"sessions_dir", claudeDir, "host", host, "interval", interval)
}
if vectorStore != nil {
embedSyncInterval := envInt("BRAIN_EMBED_SYNC_INTERVAL", 300)
vectorstore.StartSync(ctx, brainDir, vectorStore,
+97
View File
@@ -0,0 +1,97 @@
package api
import "strings"
// frontmatter is an ordered, line-preserving view of a note's YAML
// frontmatter block. It deliberately avoids a full YAML round-trip: the
// brain writes flat `key: value` frontmatter by hand, and a yaml.v3
// re-marshal would reorder keys and strip comments. Preserving the
// original lines verbatim keeps brain_update a surgical edit — only the
// keys it manages (updated_at, supersedes, supersede_reason) change.
type frontmatter struct {
lines []fmLine
}
// fmLine is one frontmatter line. For `key: value` lines, key and value
// are populated; for blank lines, comments, or anything that isn't a
// simple scalar pair, key is empty and raw holds the line verbatim.
type fmLine struct {
key string
value string
raw string
}
// parseFrontmatter splits src into its frontmatter block and body. A
// frontmatter block is recognised only when the file opens with a `---`
// fence and a closing `---` fence follows. Otherwise the whole input is
// the body and the returned frontmatter is empty.
func parseFrontmatter(src string) (frontmatter, string) {
var fm frontmatter
if !strings.HasPrefix(src, "---\n") {
return fm, src
}
rest := src[len("---\n"):]
end := strings.Index(rest, "\n---\n")
if end < 0 {
// Opening fence with no closing fence — treat as bodyless content.
return fm, src
}
block := rest[:end]
body := rest[end+len("\n---\n"):]
for _, line := range strings.Split(block, "\n") {
key, val, ok := strings.Cut(line, ":")
key = strings.TrimSpace(key)
if !ok || key == "" || strings.HasPrefix(strings.TrimSpace(line), "#") {
fm.lines = append(fm.lines, fmLine{raw: line})
continue
}
fm.lines = append(fm.lines, fmLine{key: key, value: strings.TrimSpace(val)})
}
return fm, body
}
// get returns the value for key, or "" if absent.
func (f *frontmatter) get(key string) string {
for _, l := range f.lines {
if l.key == key {
return l.value
}
}
return ""
}
// set overrides the value for an existing key in place, or appends a new
// `key: value` line when the key is absent.
func (f *frontmatter) set(key, value string) {
for i := range f.lines {
if f.lines[i].key == key {
f.lines[i].value = value
return
}
}
f.lines = append(f.lines, fmLine{key: key, value: value})
}
// render serialises the frontmatter back into a `---`-fenced block. An
// empty frontmatter renders to the empty string so bodies without a
// header stay header-less.
func (f *frontmatter) render() string {
if len(f.lines) == 0 {
return ""
}
var b strings.Builder
b.WriteString("---\n")
for _, l := range f.lines {
if l.key == "" {
b.WriteString(l.raw)
} else {
b.WriteString(l.key)
b.WriteString(": ")
b.WriteString(l.value)
}
b.WriteByte('\n')
}
b.WriteString("---\n")
return b.String()
}
@@ -0,0 +1,61 @@
package api
import (
"strings"
"testing"
"github.com/stretchr/testify/assert"
)
func TestParseFrontmatterSplitsHeaderAndBody(t *testing.T) {
src := "---\nwing: jepa-fx\nhall: facts\ncreated_at: 2026-01-01T00:00:00Z\n---\n# Title\n\nbody text\n"
fm, body := parseFrontmatter(src)
assert.Equal(t, "jepa-fx", fm.get("wing"))
assert.Equal(t, "facts", fm.get("hall"))
assert.Equal(t, "2026-01-01T00:00:00Z", fm.get("created_at"))
assert.Equal(t, "# Title\n\nbody text\n", body)
}
func TestParseFrontmatterNoHeader(t *testing.T) {
src := "# Just a body\n\nno frontmatter here\n"
fm, body := parseFrontmatter(src)
assert.Empty(t, fm.lines)
assert.Equal(t, src, body)
}
func TestFrontmatterSetOverridesExistingKey(t *testing.T) {
fm, _ := parseFrontmatter("---\nwing: a\nupdated_at: old\n---\nbody\n")
fm.set("updated_at", "new")
assert.Equal(t, "new", fm.get("updated_at"))
// No duplicate key.
assert.Equal(t, 1, strings.Count(fm.render(), "updated_at:"))
}
func TestFrontmatterSetAppendsNewKey(t *testing.T) {
fm, _ := parseFrontmatter("---\nwing: a\n---\nbody\n")
fm.set("supersedes", "abc123")
out := fm.render()
assert.Contains(t, out, "wing: a")
assert.Contains(t, out, "supersedes: abc123")
}
func TestFrontmatterRenderPreservesCustomFields(t *testing.T) {
src := "---\nwing: a\nhall: facts\ncustom_field: keep-me\ntags: [x, y]\n---\nbody\n"
fm, _ := parseFrontmatter(src)
fm.set("updated_at", "2026-06-22T00:00:00Z")
out := fm.render()
assert.Contains(t, out, "custom_field: keep-me")
assert.Contains(t, out, "tags: [x, y]")
assert.Contains(t, out, "updated_at: 2026-06-22T00:00:00Z")
}
func TestFrontmatterRenderRoundTrips(t *testing.T) {
src := "---\nwing: a\nhall: facts\n---\n"
fm, _ := parseFrontmatter(src)
assert.Equal(t, src, fm.render())
}
+131
View File
@@ -0,0 +1,131 @@
package api
import (
"crypto/sha256"
"encoding/hex"
"fmt"
"os"
"path/filepath"
"strings"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
)
// ContentHash returns the lowercase hex sha256 of b. It is the note's
// content_hash handle: brain_write / brain_update return it, brain_get
// recomputes it from the file on disk, and brain_update stamps the prior
// note's hash into the new note's `supersedes` frontmatter.
func ContentHash(b []byte) string {
sum := sha256.Sum256(b)
return hex.EncodeToString(sum[:])
}
// resolveWithin maps a brainDir-relative path to an absolute path and
// guarantees it does not escape brainDir. Returns the cleaned relPath
// (forward-slashed) and the absolute path.
func resolveWithin(brainDir, relPath string) (rel, abs string, err error) {
clean := filepath.Clean("/" + filepath.ToSlash(relPath))
rel = strings.TrimPrefix(clean, "/")
abs = filepath.Join(brainDir, filepath.FromSlash(rel))
check, err := filepath.Rel(brainDir, abs)
if err != nil || check == ".." || strings.HasPrefix(check, ".."+string(filepath.Separator)) {
return "", "", fmt.Errorf("path %q escapes brain dir", relPath)
}
return rel, abs, nil
}
// UpdateNoteOptions identifies the note to supersede and supplies its new
// body. Path takes precedence; otherwise the target is resolved from
// Wing/Hall/Slug via brain.NotePath.
type UpdateNoteOptions struct {
Path string // brainDir-relative path; takes precedence over wing/hall/slug
Wing string
Hall string
Slug string
Content string // new full body (whole-note replace)
Reason string // optional; stamped as supersede_reason
}
// UpdateNote supersedes an existing note in place. It replaces the body
// with opts.Content, preserves the existing frontmatter (created_at,
// wing, hall, and any custom fields), and stamps updated_at, supersedes
// (the prior content hash), and supersede_reason (when given).
//
// It never creates: if the target does not exist, it returns an error so
// the caller can fall back to brain_write. Returns the note's relPath,
// the new content hash, and the prior content hash.
//
// Embeddings are NOT refreshed here. The rewritten file's mtime advances,
// which the mtime-driven vectorstore.Sync ticker uses to re-embed it on
// its next pass — the same out-of-band mechanism brain_write relies on.
func UpdateNote(brainDir string, opts UpdateNoteOptions) (relPath, contentHash, priorHash string, err error) {
if opts.Content == "" {
return "", "", "", fmt.Errorf("content is required")
}
var rel string
if opts.Path != "" {
rel = opts.Path
} else {
full, perr := brain.NotePath(brainDir, opts.Wing, opts.Hall, opts.Slug)
if perr != nil {
return "", "", "", perr
}
rel, _ = filepath.Rel(brainDir, full)
rel = filepath.ToSlash(rel)
}
rel, abs, err := resolveWithin(brainDir, rel)
if err != nil {
return "", "", "", err
}
prior, err := os.ReadFile(abs)
if err != nil {
if os.IsNotExist(err) {
return "", "", "", fmt.Errorf("note %q does not exist: use brain_write to create", rel)
}
return "", "", "", fmt.Errorf("read target: %w", err)
}
priorHash = ContentHash(prior)
fm, _ := parseFrontmatter(string(prior))
fm.set("updated_at", time.Now().UTC().Format(time.RFC3339))
fm.set("supersedes", priorHash)
if opts.Reason != "" {
fm.set("supersede_reason", opts.Reason)
}
out := []byte(fm.render() + opts.Content)
if err := os.WriteFile(abs, out, 0o644); err != nil {
return "", "", "", fmt.Errorf("write: %w", err)
}
return rel, ContentHash(out), priorHash, nil
}
// ReadNote reads the note at the brainDir-relative relPath and returns
// its parsed frontmatter, body, and content hash. It is the read-after-
// write primitive behind brain_get: the hash it returns equals the hash
// brain_write / brain_update returned for the same bytes.
func ReadNote(brainDir, relPath string) (fm map[string]string, body, contentHash string, err error) {
_, abs, err := resolveWithin(brainDir, relPath)
if err != nil {
return nil, "", "", err
}
raw, err := os.ReadFile(abs)
if err != nil {
if os.IsNotExist(err) {
return nil, "", "", fmt.Errorf("note %q does not exist", relPath)
}
return nil, "", "", fmt.Errorf("read note: %w", err)
}
parsed, body := parseFrontmatter(string(raw))
fm = make(map[string]string, len(parsed.lines))
for _, l := range parsed.lines {
if l.key != "" {
fm[l.key] = l.value
}
}
return fm, body, ContentHash(raw), nil
}
+130
View File
@@ -0,0 +1,130 @@
package api
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// seedNote writes a note directly to disk and returns its relPath.
func seedNote(t *testing.T, brainDir, rel, content string) string {
t.Helper()
full := filepath.Join(brainDir, filepath.FromSlash(rel))
require.NoError(t, os.MkdirAll(filepath.Dir(full), 0o755))
require.NoError(t, os.WriteFile(full, []byte(content), 0o644))
return rel
}
func TestUpdateNoteSupersedesAndStamps(t *testing.T) {
brainDir := t.TempDir()
rel := seedNote(t, brainDir, "wiki/jepa-fx/facts/val-vol.md",
"---\nwing: jepa-fx\nhall: facts\ncreated_at: 2026-01-01T00:00:00Z\ncustom: keep-me\n---\n# Old\n\nold body\n")
relPath, hash, priorHash, err := UpdateNote(brainDir, UpdateNoteOptions{
Path: rel,
Content: "# New\n\nnew body\n",
Reason: "facts changed",
})
require.NoError(t, err)
assert.Equal(t, rel, relPath)
assert.NotEmpty(t, hash)
assert.NotEmpty(t, priorHash)
assert.NotEqual(t, hash, priorHash)
got, err := os.ReadFile(filepath.Join(brainDir, filepath.FromSlash(rel)))
require.NoError(t, err)
s := string(got)
// Body replaced.
assert.Contains(t, s, "# New")
assert.NotContains(t, s, "old body")
// Prior fields preserved.
assert.Contains(t, s, "wing: jepa-fx")
assert.Contains(t, s, "hall: facts")
assert.Contains(t, s, "created_at: 2026-01-01T00:00:00Z")
assert.Contains(t, s, "custom: keep-me")
// Supersession stamped.
assert.Contains(t, s, "updated_at:")
assert.Contains(t, s, "supersedes: "+priorHash)
assert.Contains(t, s, "supersede_reason: facts changed")
}
func TestUpdateNoteResolvesByWingHallSlug(t *testing.T) {
brainDir := t.TempDir()
seedNote(t, brainDir, "wiki/jepa-fx/facts/val-vol.md",
"---\nwing: jepa-fx\nhall: facts\n---\nold\n")
relPath, _, _, err := UpdateNote(brainDir, UpdateNoteOptions{
Wing: "jepa-fx", Hall: "facts", Slug: "val-vol",
Content: "new\n",
})
require.NoError(t, err)
assert.Equal(t, "wiki/jepa-fx/facts/val-vol.md", relPath)
}
func TestUpdateNoteErrorsOnMissingAndDoesNotCreate(t *testing.T) {
brainDir := t.TempDir()
_, _, _, err := UpdateNote(brainDir, UpdateNoteOptions{
Wing: "jepa-fx", Hall: "facts", Slug: "ghost",
Content: "x\n",
})
require.Error(t, err)
assert.Contains(t, err.Error(), "does not exist")
// No file created.
_, statErr := os.Stat(filepath.Join(brainDir, "wiki/jepa-fx/facts/ghost.md"))
assert.True(t, os.IsNotExist(statErr), "missing-target update must not create a note")
}
func TestUpdateNoteRejectsTraversal(t *testing.T) {
brainDir := t.TempDir()
_, _, _, err := UpdateNote(brainDir, UpdateNoteOptions{
Path: "../escape.md",
Content: "x\n",
})
require.Error(t, err)
}
func TestReadNoteReturnsFrontmatterBodyHash(t *testing.T) {
brainDir := t.TempDir()
rel := seedNote(t, brainDir, "wiki/jepa-fx/facts/n.md",
"---\nwing: jepa-fx\nhall: facts\n---\n# Body\n\ntext\n")
fm, body, hash, err := ReadNote(brainDir, rel)
require.NoError(t, err)
assert.Equal(t, "jepa-fx", fm["wing"])
assert.Equal(t, "facts", fm["hall"])
assert.Equal(t, "# Body\n\ntext\n", body)
// Hash matches ContentHash of the raw bytes on disk (round-trip).
raw, _ := os.ReadFile(filepath.Join(brainDir, filepath.FromSlash(rel)))
assert.Equal(t, ContentHash(raw), hash)
}
func TestReadNoteRejectsTraversal(t *testing.T) {
brainDir := t.TempDir()
_, _, _, err := ReadNote(brainDir, "../../etc/passwd")
require.Error(t, err)
}
func TestUpdateThenReadRoundTripsHash(t *testing.T) {
brainDir := t.TempDir()
rel := seedNote(t, brainDir, "wiki/a/facts/n.md", "---\nwing: a\nhall: facts\n---\nold\n")
_, hash, _, err := UpdateNote(brainDir, UpdateNoteOptions{Path: rel, Content: "new\n"})
require.NoError(t, err)
_, _, readHash, err := ReadNote(brainDir, rel)
require.NoError(t, err)
assert.Equal(t, hash, readHash, "update content_hash must round-trip through ReadNote")
}
func TestContentHashStable(t *testing.T) {
assert.Equal(t, ContentHash([]byte("abc")), ContentHash([]byte("abc")))
assert.NotEqual(t, ContentHash([]byte("abc")), ContentHash([]byte("abd")))
assert.True(t, strings.HasPrefix(ContentHash([]byte("")), "")) // hex, non-panicking
}
+110
View File
@@ -0,0 +1,110 @@
package claudewatcher
import (
"context"
"errors"
"fmt"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
)
// CursorStore tracks how far the watcher has ingested into each
// session JSONL file. Keyed by (host, file_path) so the same `~/.claude`
// path on different hosts doesn't collide and resumability survives
// pod restarts. Idempotent Init lives alongside the rest of the
// claudewatcher schema; no separate migration framework.
type CursorStore struct {
pool *pgxpool.Pool
}
// NewCursorStore opens a pool against dsn. Caller closes the store.
func NewCursorStore(ctx context.Context, dsn string) (*CursorStore, error) {
pool, err := pgxpool.New(ctx, dsn)
if err != nil {
return nil, fmt.Errorf("pgxpool: %w", err)
}
if err := pool.Ping(ctx); err != nil {
pool.Close()
return nil, fmt.Errorf("ping: %w", err)
}
return &CursorStore{pool: pool}, nil
}
// NewCursorStoreFromPool wraps an existing pool (so the watcher can
// share the brain DSN pool with vectorstore/graphstore without a
// second connection set). Caller must NOT close the wrapped pool via
// the store — close the pool directly.
func NewCursorStoreFromPool(pool *pgxpool.Pool) *CursorStore {
return &CursorStore{pool: pool}
}
// Close releases the underlying connection pool when this store owns
// it. No-op when the pool was injected via NewCursorStoreFromPool —
// pgxpool.Close is idempotent so we lean on that.
func (s *CursorStore) Close() {
if s.pool != nil {
s.pool.Close()
}
}
// Init creates the claude_session_cursors table when missing.
func (s *CursorStore) Init(ctx context.Context) error {
const ddl = `
CREATE TABLE IF NOT EXISTS claude_session_cursors (
host TEXT NOT NULL,
file_path TEXT NOT NULL,
byte_offset BIGINT NOT NULL DEFAULT 0,
last_seen_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (host, file_path)
);
CREATE INDEX IF NOT EXISTS claude_session_cursors_host_idx
ON claude_session_cursors (host);
`
_, err := s.pool.Exec(ctx, ddl)
return err
}
// GetOffset returns the last recorded byte offset for (host, filePath).
// Missing rows are reported as offset=0, ok=false so the caller can
// distinguish "never ingested" from "ingested at the start of the
// file" (both produce identical behaviour but the metric is useful).
func (s *CursorStore) GetOffset(ctx context.Context, host, filePath string) (int64, bool, error) {
if host == "" || filePath == "" {
return 0, false, errors.New("host and file_path are required")
}
var offset int64
err := s.pool.QueryRow(ctx, `
SELECT byte_offset FROM claude_session_cursors WHERE host = $1 AND file_path = $2
`, host, filePath).Scan(&offset)
if errors.Is(err, pgx.ErrNoRows) {
return 0, false, nil
}
if err != nil {
return 0, false, fmt.Errorf("query: %w", err)
}
return offset, true, nil
}
// SetOffset writes the new offset for (host, filePath). Used after
// every successful parse + ingest batch so a crash mid-file rewinds
// only to the last committed checkpoint.
func (s *CursorStore) SetOffset(ctx context.Context, host, filePath string, offset int64) error {
if host == "" || filePath == "" {
return errors.New("host and file_path are required")
}
if offset < 0 {
return errors.New("offset must be >= 0")
}
_, err := s.pool.Exec(ctx, `
INSERT INTO claude_session_cursors (host, file_path, byte_offset, last_seen_at)
VALUES ($1, $2, $3, now())
ON CONFLICT (host, file_path) DO UPDATE
SET byte_offset = EXCLUDED.byte_offset,
last_seen_at = now()
`, host, filePath, offset)
if err != nil {
return fmt.Errorf("upsert offset: %w", err)
}
return nil
}
+305
View File
@@ -0,0 +1,305 @@
// Package claudewatcher ingests Claude Code session transcripts
// (`~/.claude/projects/*/<uuid>.jsonl`) into the brain corpus.
//
// Schema (observed 2026-05-25 across ~30 session files on koala):
//
// type=user — user prompts + tool results
// type=assistant — model turns; tool_use blocks live in message.content
// type=attachment — hook outputs, ingested files
// type=system — turn-boundary metadata
// type=file-history-snapshot — git-style snapshot of edited files
// type=queue-operation, last-prompt, permission-mode, ai-title,
// bridge-session — internal bookkeeping, ignored
//
// The parser is intentionally tolerant: malformed lines are skipped
// (caller logs and advances), missing optional fields default to "",
// and unknown `type` values are returned as Turn entries with
// `Skip=true` so callers can filter cheaply.
package claudewatcher
import (
"bufio"
"encoding/json"
"errors"
"fmt"
"io"
"strings"
"time"
)
// Turn is one parsed JSONL entry from a Claude Code session log.
//
// Skip is true for entry types we never want to ingest (queue
// bookkeeping, snapshots, etc.). Callers fast-path these without
// running the scrubber or classifier.
type Turn struct {
SessionID string
Type string
ParentUUID string
Timestamp time.Time
Cwd string
GitBranch string
Content string // plain-text projection of the entry, ready for the scrubber/classifier
ToolName string // populated when an assistant turn invokes a tool
OffsetAfter int64 // byte offset in the file just past this entry
Skip bool
ParseWarning string // non-empty when the entry parsed but had a sub-field we couldn't normalise
}
// ParseStream reads JSONL lines from r starting at startOffset and
// invokes emit for each parsed entry. emit may return ErrStop to
// terminate the scan cleanly. Other emit errors propagate.
//
// startOffset is informational — the caller is expected to have already
// seeked the underlying reader to that offset. ParseStream adds the
// number of bytes consumed per line to it to compute Turn.OffsetAfter.
//
// Lines that fail to unmarshal are logged via warnf and skipped; they
// do NOT advance OffsetAfter past the malformed line by themselves,
// but the next valid line resumes correctly because bufio.Scanner
// preserves stream position.
func ParseStream(
r io.Reader,
startOffset int64,
warnf func(format string, args ...any),
emit func(Turn) error,
) (int64, error) {
scanner := bufio.NewScanner(r)
scanner.Buffer(make([]byte, 0, 64*1024), 8*1024*1024) // some lines are big (tool outputs)
offset := startOffset
for scanner.Scan() {
raw := scanner.Bytes()
lineLen := int64(len(raw)) + 1 // +1 for the newline
t, err := parseTurn(raw)
if err != nil {
if warnf != nil {
warnf("parse: %v (%d bytes)", err, len(raw))
}
offset += lineLen
continue
}
t.OffsetAfter = offset + lineLen
if err := emit(t); err != nil {
if errors.Is(err, ErrStop) {
return t.OffsetAfter, nil
}
return offset, fmt.Errorf("emit: %w", err)
}
offset = t.OffsetAfter
}
if err := scanner.Err(); err != nil {
return offset, fmt.Errorf("scan: %w", err)
}
return offset, nil
}
// ErrStop terminates a ParseStream loop without surfacing an error.
var ErrStop = errors.New("claudewatcher: stop")
// rawEntry is a permissive shape that covers every type observed in
// the JSONL files. Fields we don't care about are intentionally
// omitted to keep the unmarshal cheap.
type rawEntry struct {
Type string `json:"type"`
SessionID string `json:"sessionId"`
ParentUUID string `json:"parentUuid"`
Timestamp string `json:"timestamp"`
Cwd string `json:"cwd"`
GitBranch string `json:"gitBranch"`
Message json.RawMessage `json:"message"`
Attachment json.RawMessage `json:"attachment"`
Content string `json:"content"` // queue-operation
LastPrompt string `json:"lastPrompt"` // last-prompt
Subtype string `json:"subtype"` // system
}
// skipTypes lists every entry type we want to never ingest. Marked Skip
// at parse time so the caller's filter is a single boolean check.
var skipTypes = map[string]struct{}{
"queue-operation": {},
"last-prompt": {},
"permission-mode": {},
"ai-title": {},
"bridge-session": {},
"file-history-snapshot": {},
}
func parseTurn(raw []byte) (Turn, error) {
var e rawEntry
if err := json.Unmarshal(raw, &e); err != nil {
return Turn{}, fmt.Errorf("unmarshal: %w", err)
}
t := Turn{
Type: e.Type,
SessionID: e.SessionID,
ParentUUID: e.ParentUUID,
Cwd: e.Cwd,
GitBranch: e.GitBranch,
}
if _, skip := skipTypes[e.Type]; skip {
t.Skip = true
return t, nil
}
if e.Timestamp != "" {
if ts, err := time.Parse(time.RFC3339Nano, e.Timestamp); err == nil {
t.Timestamp = ts
} else {
t.ParseWarning = "timestamp"
}
}
switch e.Type {
case "user":
t.Content = extractMessageText(e.Message)
case "assistant":
t.Content, t.ToolName = extractAssistantTurn(e.Message)
case "attachment":
t.Content = extractAttachmentText(e.Attachment)
case "system":
t.Content = "[system " + e.Subtype + "]"
default:
// Unknown type — keep the row but mark Skip so callers ignore.
t.Skip = true
}
return t, nil
}
// extractMessageText pulls the textual projection out of a user/assistant
// message field. The shape is the Anthropic Messages API content-block
// array (an array of {type, text|tool_use|tool_result, ...}). We
// concatenate every text-bearing block and ignore the rest.
func extractMessageText(raw json.RawMessage) string {
if len(raw) == 0 {
return ""
}
var msg struct {
Role string `json:"role"`
Content json.RawMessage `json:"content"`
Stop string `json:"stop_reason"`
Model string `json:"model"`
Usage map[string]any `json:"usage"`
Meta map[string]string `json:"meta"`
}
if err := json.Unmarshal(raw, &msg); err != nil {
// Some user turns have message as plain string.
var s string
if err2 := json.Unmarshal(raw, &s); err2 == nil {
return s
}
return ""
}
// Content can be a string OR an array.
var asString string
if err := json.Unmarshal(msg.Content, &asString); err == nil {
return asString
}
var blocks []struct {
Type string `json:"type"`
Text string `json:"text"`
Content json.RawMessage `json:"content"`
}
if err := json.Unmarshal(msg.Content, &blocks); err != nil {
return ""
}
var sb strings.Builder
for _, b := range blocks {
switch b.Type {
case "text":
sb.WriteString(b.Text)
sb.WriteByte('\n')
case "tool_result":
// Tool result content may itself be a string or array of blocks.
var s string
if err := json.Unmarshal(b.Content, &s); err == nil {
sb.WriteString("[tool_result] ")
sb.WriteString(s)
sb.WriteByte('\n')
continue
}
var sub []struct {
Type string `json:"type"`
Text string `json:"text"`
}
if err := json.Unmarshal(b.Content, &sub); err == nil {
for _, s := range sub {
if s.Type == "text" {
sb.WriteString("[tool_result] ")
sb.WriteString(s.Text)
sb.WriteByte('\n')
}
}
}
}
}
return strings.TrimRight(sb.String(), "\n")
}
// extractAssistantTurn pulls text + the first tool name (if any) from
// an assistant content-block array. Multi-tool turns lose the second
// name; the goal is signal for classification, not perfect fidelity.
func extractAssistantTurn(raw json.RawMessage) (string, string) {
if len(raw) == 0 {
return "", ""
}
var msg struct {
Content json.RawMessage `json:"content"`
}
if err := json.Unmarshal(raw, &msg); err != nil {
return "", ""
}
var blocks []struct {
Type string `json:"type"`
Text string `json:"text"`
Name string `json:"name"`
Tool json.RawMessage `json:"input"`
}
if err := json.Unmarshal(msg.Content, &blocks); err != nil {
return "", ""
}
var sb strings.Builder
var firstTool string
for _, b := range blocks {
switch b.Type {
case "text":
sb.WriteString(b.Text)
sb.WriteByte('\n')
case "tool_use":
if firstTool == "" {
firstTool = b.Name
}
sb.WriteString("[tool_use:")
sb.WriteString(b.Name)
sb.WriteString("]\n")
}
}
return strings.TrimRight(sb.String(), "\n"), firstTool
}
// extractAttachmentText pulls text content from an attachment payload,
// or returns a short tag when the attachment is a hook event.
func extractAttachmentText(raw json.RawMessage) string {
if len(raw) == 0 {
return ""
}
var a struct {
Type string `json:"type"`
HookName string `json:"hookName"`
HookEvent string `json:"hookEvent"`
Content string `json:"content"`
Text string `json:"text"`
}
if err := json.Unmarshal(raw, &a); err != nil {
return ""
}
if a.Content != "" {
return a.Content
}
if a.Text != "" {
return a.Text
}
if a.HookName != "" {
return "[hook " + a.HookEvent + ":" + a.HookName + "]"
}
return ""
}
@@ -0,0 +1,157 @@
package claudewatcher
import (
"errors"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func collect(t *testing.T, body string) ([]Turn, int64, error) {
t.Helper()
var out []Turn
end, err := ParseStream(strings.NewReader(body), 0, nil, func(tr Turn) error {
out = append(out, tr)
return nil
})
return out, end, err
}
func TestParseStream_UserTurnStringContent(t *testing.T) {
body := `{"type":"user","sessionId":"S","timestamp":"2026-05-25T07:00:00Z","message":"hello world"}
`
turns, end, err := collect(t, body)
require.NoError(t, err)
require.Len(t, turns, 1)
assert.Equal(t, "user", turns[0].Type)
assert.Equal(t, "S", turns[0].SessionID)
assert.Equal(t, "hello world", turns[0].Content)
assert.False(t, turns[0].Skip)
assert.Equal(t, int64(len(body)), end)
}
func TestParseStream_UserTurnContentBlocks(t *testing.T) {
body := `{"type":"user","sessionId":"S","timestamp":"2026-05-25T07:00:00Z","message":{"role":"user","content":[{"type":"text","text":"line 1"},{"type":"text","text":"line 2"}]}}
`
turns, _, err := collect(t, body)
require.NoError(t, err)
require.Len(t, turns, 1)
assert.Equal(t, "line 1\nline 2", turns[0].Content)
}
func TestParseStream_AssistantToolUse(t *testing.T) {
body := `{"type":"assistant","sessionId":"S","timestamp":"2026-05-25T07:00:00Z","message":{"content":[{"type":"text","text":"calling now"},{"type":"tool_use","name":"Edit","input":{}}]}}
`
turns, _, err := collect(t, body)
require.NoError(t, err)
require.Len(t, turns, 1)
assert.Equal(t, "Edit", turns[0].ToolName)
assert.Contains(t, turns[0].Content, "calling now")
assert.Contains(t, turns[0].Content, "[tool_use:Edit]")
}
func TestParseStream_AssistantToolResult(t *testing.T) {
body := `{"type":"user","sessionId":"S","timestamp":"2026-05-25T07:00:00Z","message":{"content":[{"type":"tool_result","content":"output of cmd"}]}}
`
turns, _, err := collect(t, body)
require.NoError(t, err)
require.Len(t, turns, 1)
assert.Contains(t, turns[0].Content, "[tool_result] output of cmd")
}
func TestParseStream_SkipsBookkeepingTypes(t *testing.T) {
body := strings.Join([]string{
`{"type":"queue-operation","sessionId":"S","content":"x"}`,
`{"type":"last-prompt","sessionId":"S","lastPrompt":"y"}`,
`{"type":"permission-mode","sessionId":"S","permissionMode":"auto"}`,
`{"type":"ai-title","sessionId":"S","aiTitle":"My session"}`,
`{"type":"file-history-snapshot","messageId":"abc"}`,
}, "\n") + "\n"
turns, _, err := collect(t, body)
require.NoError(t, err)
require.Len(t, turns, 5)
for _, tr := range turns {
assert.True(t, tr.Skip, "expected Skip=true for %q", tr.Type)
}
}
func TestParseStream_UnknownTypeIsSkip(t *testing.T) {
body := `{"type":"future-thing","sessionId":"S"}` + "\n"
turns, _, err := collect(t, body)
require.NoError(t, err)
require.Len(t, turns, 1)
assert.True(t, turns[0].Skip)
}
func TestParseStream_MalformedLineIsSkippedNotFatal(t *testing.T) {
body := strings.Join([]string{
`{"type":"user","sessionId":"S","message":"first"}`,
`{not valid json`,
`{"type":"user","sessionId":"S","message":"third"}`,
}, "\n") + "\n"
var warnings int
var turns []Turn
_, err := ParseStream(strings.NewReader(body), 0, func(format string, args ...any) {
warnings++
}, func(tr Turn) error {
turns = append(turns, tr)
return nil
})
require.NoError(t, err)
require.Len(t, turns, 2, "first + third should make it through")
assert.Equal(t, 1, warnings)
}
func TestParseStream_EmitErrStopHaltsCleanly(t *testing.T) {
body := strings.Join([]string{
`{"type":"user","sessionId":"S","message":"a"}`,
`{"type":"user","sessionId":"S","message":"b"}`,
`{"type":"user","sessionId":"S","message":"c"}`,
}, "\n") + "\n"
count := 0
end, err := ParseStream(strings.NewReader(body), 0, nil, func(tr Turn) error {
count++
if count == 2 {
return ErrStop
}
return nil
})
require.NoError(t, err)
assert.Equal(t, 2, count)
assert.Greater(t, end, int64(0))
}
func TestParseStream_EmitOtherErrorPropagates(t *testing.T) {
body := `{"type":"user","sessionId":"S","message":"a"}` + "\n"
want := errors.New("boom")
_, err := ParseStream(strings.NewReader(body), 0, nil, func(tr Turn) error {
return want
})
require.Error(t, err)
assert.Contains(t, err.Error(), "boom")
}
func TestParseStream_AttachmentHookEvent(t *testing.T) {
body := `{"type":"attachment","sessionId":"S","timestamp":"2026-05-25T07:00:00Z","attachment":{"type":"hook_success","hookName":"SessionStart:startup","hookEvent":"SessionStart","content":"hook body"}}
`
turns, _, err := collect(t, body)
require.NoError(t, err)
require.Len(t, turns, 1)
assert.Equal(t, "hook body", turns[0].Content)
}
func TestParseStream_OffsetAdvances(t *testing.T) {
body := `{"type":"user","sessionId":"S","message":"a"}` + "\n" +
`{"type":"user","sessionId":"S","message":"b"}` + "\n"
var offsets []int64
_, err := ParseStream(strings.NewReader(body), 100, nil, func(tr Turn) error {
offsets = append(offsets, tr.OffsetAfter)
return nil
})
require.NoError(t, err)
require.Len(t, offsets, 2)
assert.Greater(t, offsets[0], int64(100))
assert.Greater(t, offsets[1], offsets[0])
}
@@ -0,0 +1,126 @@
package claudewatcher
import (
"fmt"
"regexp"
"sync"
)
// Scrubber drops any turn whose content matches a known-bad pattern.
// Fail-closed by design: we'd rather lose signal than ingest credentials
// into a public-readable brain. The caller logs the drop reason.
//
// Rules cover the credential shapes most common to leak through Claude
// Code sessions: bearer tokens, postgres URIs with embedded auth, OAuth
// secret values, SOPS-encrypted secret blobs (we don't want the
// ciphertext either — it's a marker that the original message contained
// secret state), PEM-encoded private keys, and the explicit env-var
// naming conventions used in the homelab.
//
// Pattern philosophy: match by shape, not by content. A 40-char hex
// string in isolation is fine; the same string after `Authorization:
// Bearer ` is not. Tuned to catch known leak vectors from prior
// secret-hygiene incidents (POSTGRES_PASSWORD via kubectl exec env,
// INFRA_MCP_TOKEN via sops -d output) without dropping every Edit on a
// config file.
// Rule is a single named regex with a redact hint shown in the warn log.
type Rule struct {
Name string
RE *regexp.Regexp
}
// DefaultRules is the regex set applied by Scrub. Mutable for tests but
// callers should treat it as read-only at runtime.
var DefaultRules = []Rule{
// authorization-header is checked before the bare bearer rule so
// contextual hits ("Authorization: Bearer X") report the more
// specific match name in logs.
{Name: "authorization-header", RE: regexp.MustCompile(`(?i)Authorization\s*:\s*[A-Za-z]+\s+\S{8,}`)},
{Name: "bearer-token", RE: regexp.MustCompile(`(?i)Bearer\s+[A-Za-z0-9._\-]{16,}`)},
// JWT (header.payload.sig), e.g. a Dex/OAuth token dumped to stdout
// without a "Bearer " prefix. Both header and payload base64url-encode
// JSON, so both segments begin with "eyJ".
{Name: "jwt", RE: regexp.MustCompile(`eyJ[A-Za-z0-9_\-]{8,}\.eyJ[A-Za-z0-9_\-]{8,}\.[A-Za-z0-9_\-]{8,}`)},
{Name: "postgres-uri-with-password", RE: regexp.MustCompile(`postgres(?:ql)?://[^:\s/]+:[^@\s/]+@`)},
{Name: "private-key", RE: regexp.MustCompile(`-----BEGIN[^-]*PRIVATE KEY-----`)},
{Name: "ssh-key", RE: regexp.MustCompile(`ssh-(?:rsa|ed25519|ecdsa)\s+[A-Za-z0-9+/=]{40,}`)},
{Name: "github-pat", RE: regexp.MustCompile(`\b(?:ghp|gho|ghu|ghr|gha)_[A-Za-z0-9]{30,}\b`)},
// 1Password service-account token (ops_<base64url>). Long, high-value root
// credential; guard the bare value (the _TOKEN= form also hits homelab-env-token).
{Name: "op-service-account", RE: regexp.MustCompile(`\bops_[A-Za-z0-9_\-]{40,}`)},
// No leading \b: a shell mangle can glue the key to a preceding word
// ("yes"+"sk-...") which has no word boundary, and that exact case
// leaked a LiteLLM master key past this rule (2026-06-11). Match the
// sk- shape wherever it appears; the {32,} length floor keeps short
// "task-"/"disk-" words from tripping it.
{Name: "openai-sk", RE: regexp.MustCompile(`sk-(?:proj-)?[A-Za-z0-9]{32,}`)},
{Name: "anthropic-sk", RE: regexp.MustCompile(`\bsk-ant-[A-Za-z0-9_\-]{32,}\b`)},
{Name: "aws-access-key", RE: regexp.MustCompile(`\bAKIA[0-9A-Z]{16}\b`)},
{Name: "homelab-env-token", RE: regexp.MustCompile(`(?i)(?:_TOKEN|_PASSWORD|_API_KEY|_SECRET)\s*[:=]\s*['"]?[A-Za-z0-9._/+\-]{12,}`)},
{Name: "sops-encrypted-marker", RE: regexp.MustCompile(`ENC\[AES256_GCM,data:[A-Za-z0-9+/=]{8,}`)},
}
// extraRules is appended to DefaultRules at process startup via
// RegisterRule. The mutex guards concurrent RegisterRule calls (rare)
// against concurrent Scrub reads (hot path). Scrub takes a read lock
// only when extraRules is non-empty, so steady-state cost is zero
// when no client-name guard is configured.
var (
extraRulesMu sync.RWMutex
extraRules []Rule
)
// RegisterRule appends a runtime-configured regex to the scrubber's
// rule set. Used by main to inject client-name guards from
// CLAUDE_INGEST_CLIENT_BLOCK env var (or equivalent SOPS-encrypted
// secret) without baking client identities into source code.
//
// pattern is compiled as-is — callers wrap with `\b...\b` and case
// flags as needed. Duplicate names are accepted (rules are positional);
// the second registration just fires after the first.
func RegisterRule(name, pattern string) error {
re, err := regexp.Compile(pattern)
if err != nil {
return fmt.Errorf("compile rule %q: %w", name, err)
}
extraRulesMu.Lock()
extraRules = append(extraRules, Rule{Name: name, RE: re})
extraRulesMu.Unlock()
return nil
}
// ResetExtraRules clears every RegisterRule-added rule. Test-only.
func ResetExtraRules() {
extraRulesMu.Lock()
extraRules = nil
extraRulesMu.Unlock()
}
// Scrub reports the first matching rule, or empty when content is clean.
// Empty string is treated as clean. Caller decides what to do on a hit;
// the convention in claudewatcher is to drop the turn entirely and emit
// a slog.Warn naming the rule.
//
// Rule order: DefaultRules first (credential shapes), then runtime
// RegisterRule additions (client-name guards). Credential leaks
// outrank client-name hits in the log because they're strictly more
// dangerous.
func Scrub(content string) string {
if content == "" {
return ""
}
for _, r := range DefaultRules {
if r.RE.MatchString(content) {
return r.Name
}
}
extraRulesMu.RLock()
defer extraRulesMu.RUnlock()
for _, r := range extraRules {
if r.RE.MatchString(content) {
return r.Name
}
}
return ""
}
@@ -0,0 +1,132 @@
package claudewatcher
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestScrub_PoisonedFixtures(t *testing.T) {
// One representative bad-string per rule. If a rule fires for the
// wrong content shape later, this table localises the regression.
cases := []struct {
name string
content string
want string
}{
{"bearer-token", "curl -H 'Authorization: Bearer abcdef1234567890ghijklmnop'", "authorization-header"},
{"bearer-no-header", "header = Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "bearer-token"},
{"postgres-uri", "DATABASE_URL=postgres://user:s3cret@10.0.1.20:5432/brain", "postgres-uri-with-password"},
{"private-key", "-----BEGIN OPENSSH PRIVATE KEY-----\nb3BlbnNzaC1rZXktdjEAAAAA", "private-key"},
{"ssh-public", "deploy: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK1234567890abcdefghij user@host", "ssh-key"},
{"github-pat-classic", "GH_TOKEN=ghp_aBcD1234EfGh5678IjKl9012MnOp3456QrSt", "github-pat"},
{"openai-key", "OPENAI_API_KEY=sk-proj-AAAABBBBCCCCDDDDEEEEFFFFGGGGHHHHIIII", "openai-sk"},
{"anthropic-key", "ANTHROPIC_API_KEY=sk-ant-api03-aaaaBBBBccccDDDDeeeeFFFFggggHHHHiiiiJJJJkkkk", "anthropic-sk"},
{"aws-access-key", "AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE", "aws-access-key"},
{"homelab-env", "POSTGRES_PASSWORD=hunter2supersecretvalue", "homelab-env-token"},
{"sops-marker", "value: ENC[AES256_GCM,data:abc123def456,iv:zzz]", "sops-encrypted-marker"},
// Regression: a shell mangle glued the key to a preceding word
// ("yes"+"sk-..."), defeating the leading \b in the sk- rule and
// leaking a LiteLLM master key past the scrubber (2026-06-11).
{"sk-glued-to-word", "master key resolved: yessk-7181ca984603239d8c4819361bf33b94b9c3c07018791868", "openai-sk"},
{"sk-standalone-hex", "sk-7181ca984603239d8c4819361bf33b94b9c3c07018791868", "openai-sk"},
// Bare JWT not preceded by "Bearer" (e.g. a Dex token dumped to stdout).
{"jwt-bare", "token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0.dQw4w9WgXcQabcdef", "jwt"},
// 1Password service-account token (ops_<base64url>), env-assigned and bare.
// Both hit the dedicated op-service-account rule (ordered before the
// generic homelab-env-token). Guards ~/.zshrc reads etc. (2026-06-14).
{"op-sa-env", "export OP_SERVICE_ACCOUNT_TOKEN=ops_eyJzaWduSW5BZGRyZXNzIjoibXkuMXBhc3N3b3JkLmNvbSJ9", "op-service-account"},
{"op-sa-bare", "ops_eyJzaWduSW5BZGRyZXNzIjoibXkuMXBhc3N3b3JkLmNvbSIsInVzZXJBdXRoIjp7fX0aGVsbG8", "op-service-account"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got := Scrub(tc.content)
assert.Equal(t, tc.want, got)
})
}
}
func TestScrub_CleanContentPassesThrough(t *testing.T) {
cases := []string{
"",
"plain text with no credentials",
"a 40 char hex string aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa is fine in isolation",
"`Bearer` token mentioned in docs without an actual value",
"file at ~/.ssh/id_ed25519",
"the function Authorization() takes no args",
"comment: see API key in 1Password",
// loosened sk- rule must not trip on short "task-"/"disk-" words
"run task-build then task-test in the pipeline",
"mounted /dev/disk-by-id/wwn-0x5000",
}
for _, c := range cases {
assert.Empty(t, Scrub(c), "expected clean for %q", c)
}
}
func TestScrub_FirstMatchWins(t *testing.T) {
// Content matching multiple rules: report the first rule order in
// DefaultRules. Stability matters for log triage.
content := "Authorization: Bearer ghp_aBcD1234EfGh5678IjKl9012MnOp3456QrSt"
assert.Equal(t, "authorization-header", Scrub(content))
}
func TestRegisterRule_ClientNameGuard(t *testing.T) {
t.Cleanup(ResetExtraRules)
require := func(err error) {
if err != nil {
t.Fatalf("unexpected err: %v", err)
}
}
require(RegisterRule("client-name", `(?i)\b(SEB|Mastercard)\b`))
// Hits — case variations + word-boundary respect.
for _, hit := range []string{
"mentioned SEB in this commit",
"the Mastercard project deadline",
"working on mastercard scope",
"SEB internal review",
} {
assert.Equal(t, "client-name", Scrub(hit), "should match %q", hit)
}
// Misses — substring within a longer word should NOT match
// thanks to \b. "Sebastian" contains "seb" but \b prevents hit.
for _, miss := range []string{
"Sebastian wrote the docs",
"unrelated text",
"researcher",
"https://example.com/search?seb=1", // 'seb' bounded by ?=, still matches \b
} {
got := Scrub(miss)
if miss == "https://example.com/search?seb=1" {
// `seb=` has word-boundary at '='; this DOES match \bseb\b.
// Accept either outcome; document the tradeoff.
assert.Contains(t, []string{"", "client-name"}, got)
continue
}
assert.Empty(t, got, "should NOT match %q", miss)
}
}
func TestRegisterRule_CredentialsTakePrecedence(t *testing.T) {
t.Cleanup(ResetExtraRules)
require := func(err error) {
if err != nil {
t.Fatalf("unexpected err: %v", err)
}
}
require(RegisterRule("client-name", `\b(SEB)\b`))
// Content matches both a credential rule AND a client rule —
// credential rule wins by ordering, so log triage points at the
// strictly more dangerous leak.
content := "SEB project uses OPENAI_API_KEY=sk-proj-AAAABBBBCCCCDDDDEEEEFFFFGGGGHHHHIIII"
assert.Equal(t, "openai-sk", Scrub(content))
}
func TestRegisterRule_RejectsInvalidPattern(t *testing.T) {
t.Cleanup(ResetExtraRules)
err := RegisterRule("bad", "[unclosed")
assert.Error(t, err)
}
+234
View File
@@ -0,0 +1,234 @@
package claudewatcher
import (
"context"
"fmt"
"log/slog"
"os"
"path/filepath"
"strings"
"time"
)
// Sink consumes batches of ingest-ready turns from the watcher. The
// production implementation builds wiki pages and calls pipeline.RunRaw
// against the brain. Tests substitute a counter.
//
// A Batch represents the turns ingested from one session file between
// two cursor checkpoints. Implementations must be idempotent — the
// watcher only advances the cursor on a nil return.
type Sink interface {
Ingest(ctx context.Context, b Batch) error
}
// Batch is a per-file slice of turns plus identifying metadata.
type Batch struct {
Host string // origin host, e.g. "koala"
FilePath string // absolute path to the source .jsonl file
SessionID string // first session_id seen in the batch
ProjectID string // basename of the parent dir, e.g. "-home-mathias-dev"
Turns []Turn // never empty; caller filters Skip + scrubber matches
}
// Config drives one Watch loop. SessionsDir is the absolute path to the
// Claude Code projects directory (~/.claude/projects). Host is the
// label written into cursors and ingested page frontmatter. Interval
// is the poll cadence; a zero or negative value disables the loop.
//
// Sink is required. Cursors is optional — when nil the watcher
// re-reads from byte 0 on every tick (useful for first-run testing
// without a postgres dependency).
type Config struct {
SessionsDir string
Host string
Interval time.Duration
Sink Sink
Cursors *CursorStore
Logger *slog.Logger
}
// Watch runs the polling loop until ctx is cancelled. Returns ctx.Err()
// on shutdown. Each tick walks SessionsDir for *.jsonl files, advances
// each file's cursor, and emits one Batch per file with new turns.
// Errors during a single file's parse or ingest are logged but do not
// abort the loop — a single bad file shouldn't block the others.
func Watch(ctx context.Context, cfg Config) error {
if cfg.SessionsDir == "" {
return fmt.Errorf("sessions dir is required")
}
if cfg.Sink == nil {
return fmt.Errorf("sink is required")
}
if cfg.Interval <= 0 {
return fmt.Errorf("interval must be positive")
}
if cfg.Host == "" {
cfg.Host = "unknown"
}
if cfg.Logger == nil {
cfg.Logger = slog.Default()
}
cfg.Logger.Info("claudewatcher: started",
"sessions_dir", cfg.SessionsDir,
"host", cfg.Host,
"interval", cfg.Interval)
ticker := time.NewTicker(cfg.Interval)
defer ticker.Stop()
// Run an immediate first sweep so first-launch users don't wait one
// tick before anything happens.
runTick(ctx, cfg)
for {
select {
case <-ctx.Done():
return ctx.Err()
case <-ticker.C:
runTick(ctx, cfg)
}
}
}
// runTick is one polling pass. Exposed (lowercase) for tests via
// TickOnce.
func runTick(ctx context.Context, cfg Config) {
files, err := listSessionFiles(cfg.SessionsDir)
if err != nil {
cfg.Logger.Warn("claudewatcher: list session files", "err", err)
return
}
for _, f := range files {
if ctx.Err() != nil {
return
}
if err := processFile(ctx, cfg, f); err != nil {
cfg.Logger.Warn("claudewatcher: file failed",
"path", f, "err", err)
}
}
}
// TickOnce runs one sweep synchronously and returns. Used by tests +
// by ad-hoc CLI invocations.
func TickOnce(ctx context.Context, cfg Config) error {
if cfg.SessionsDir == "" || cfg.Sink == nil {
return fmt.Errorf("config invalid")
}
if cfg.Host == "" {
cfg.Host = "unknown"
}
if cfg.Logger == nil {
cfg.Logger = slog.Default()
}
runTick(ctx, cfg)
return nil
}
func listSessionFiles(root string) ([]string, error) {
var out []string
err := filepath.WalkDir(root, func(path string, d os.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if d.IsDir() {
return nil
}
if !strings.HasSuffix(path, ".jsonl") {
return nil
}
out = append(out, path)
return nil
})
if err != nil {
return nil, fmt.Errorf("walk %s: %w", root, err)
}
return out, nil
}
func processFile(ctx context.Context, cfg Config, path string) error {
startOffset := int64(0)
if cfg.Cursors != nil {
off, _, err := cfg.Cursors.GetOffset(ctx, cfg.Host, path)
if err != nil {
return fmt.Errorf("get cursor: %w", err)
}
startOffset = off
}
stat, err := os.Stat(path)
if err != nil {
return fmt.Errorf("stat: %w", err)
}
if stat.Size() <= startOffset {
return nil // nothing new
}
f, err := os.Open(path)
if err != nil {
return fmt.Errorf("open: %w", err)
}
defer func() { _ = f.Close() }()
if _, err := f.Seek(startOffset, 0); err != nil {
return fmt.Errorf("seek: %w", err)
}
var keep []Turn
var sessionID string
var droppedScrub int
endOffset, err := ParseStream(f, startOffset,
func(format string, args ...any) {
cfg.Logger.Warn(fmt.Sprintf("claudewatcher: parse: "+format, args...))
},
func(t Turn) error {
if t.Skip || t.Content == "" {
return nil
}
if rule := Scrub(t.Content); rule != "" {
droppedScrub++
cfg.Logger.Warn("claudewatcher: turn dropped by scrubber",
"rule", rule, "path", path, "session_id", t.SessionID)
return nil
}
if sessionID == "" {
sessionID = t.SessionID
}
keep = append(keep, t)
return nil
})
if err != nil {
return fmt.Errorf("parse stream: %w", err)
}
if len(keep) == 0 {
if cfg.Cursors != nil {
if err := cfg.Cursors.SetOffset(ctx, cfg.Host, path, endOffset); err != nil {
return fmt.Errorf("advance cursor (no-turns): %w", err)
}
}
if droppedScrub > 0 {
cfg.Logger.Info("claudewatcher: only scrubbed turns this tick",
"path", path, "dropped", droppedScrub)
}
return nil
}
batch := Batch{
Host: cfg.Host,
FilePath: path,
SessionID: sessionID,
ProjectID: filepath.Base(filepath.Dir(path)),
Turns: keep,
}
if err := cfg.Sink.Ingest(ctx, batch); err != nil {
return fmt.Errorf("sink ingest: %w", err)
}
if cfg.Cursors != nil {
if err := cfg.Cursors.SetOffset(ctx, cfg.Host, path, endOffset); err != nil {
return fmt.Errorf("advance cursor: %w", err)
}
}
cfg.Logger.Info("claudewatcher: ingested batch",
"path", path, "session_id", sessionID,
"turns_kept", len(keep), "dropped_scrub", droppedScrub,
"new_offset", endOffset)
return nil
}
@@ -0,0 +1,174 @@
package claudewatcher
import (
"context"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// memSink captures batches without touching postgres. Thread-safe so
// TickOnce can run from any goroutine in concurrent tests.
type memSink struct {
mu sync.Mutex
batches []Batch
failOn string // file basename to error on
}
func (m *memSink) Ingest(_ context.Context, b Batch) error {
m.mu.Lock()
defer m.mu.Unlock()
if m.failOn != "" && strings.Contains(b.FilePath, m.failOn) {
return assert.AnError
}
m.batches = append(m.batches, b)
return nil
}
func writeSession(t *testing.T, dir, sessionID string, lines []string) string {
t.Helper()
path := filepath.Join(dir, sessionID+".jsonl")
body := strings.Join(lines, "\n") + "\n"
require.NoError(t, os.WriteFile(path, []byte(body), 0o644))
return path
}
func TestTickOnce_NoCursorReingestsEverythingEveryTick(t *testing.T) {
tmp := t.TempDir()
projectDir := filepath.Join(tmp, "-home-mathias-dev")
require.NoError(t, os.MkdirAll(projectDir, 0o755))
writeSession(t, projectDir, "sess1", []string{
`{"type":"user","sessionId":"sess1","message":"first prompt"}`,
`{"type":"assistant","sessionId":"sess1","message":{"content":[{"type":"text","text":"first answer"}]}}`,
})
sink := &memSink{}
cfg := Config{
SessionsDir: tmp,
Host: "koala",
Sink: sink,
}
require.NoError(t, TickOnce(context.Background(), cfg))
require.NoError(t, TickOnce(context.Background(), cfg))
require.Len(t, sink.batches, 2, "no cursor => re-emits same batch every tick")
assert.Equal(t, "sess1", sink.batches[0].SessionID)
assert.Equal(t, "koala", sink.batches[0].Host)
assert.Equal(t, "-home-mathias-dev", sink.batches[0].ProjectID)
assert.Len(t, sink.batches[0].Turns, 2)
}
func TestTickOnce_FiltersSkipTurnsAndScrubberMatches(t *testing.T) {
tmp := t.TempDir()
proj := filepath.Join(tmp, "-home-mathias-dev")
require.NoError(t, os.MkdirAll(proj, 0o755))
writeSession(t, proj, "sess-scrub", []string{
`{"type":"queue-operation","sessionId":"sess-scrub","content":"x"}`, // Skip
`{"type":"user","sessionId":"sess-scrub","message":"normal prompt"}`,
`{"type":"assistant","sessionId":"sess-scrub","message":{"content":[{"type":"text","text":"value POSTGRES_PASSWORD=hunter2supersecretvalue"}]}}`, // scrubbed
})
sink := &memSink{}
require.NoError(t, TickOnce(context.Background(), Config{
SessionsDir: tmp, Host: "koala", Sink: sink,
}))
require.Len(t, sink.batches, 1)
turns := sink.batches[0].Turns
require.Len(t, turns, 1, "skip + scrubbed turns must not reach the sink")
assert.Equal(t, "user", turns[0].Type)
}
func TestTickOnce_AllScrubbedNoBatchEmitted(t *testing.T) {
tmp := t.TempDir()
proj := filepath.Join(tmp, "-home-mathias-dev")
require.NoError(t, os.MkdirAll(proj, 0o755))
writeSession(t, proj, "all-bad", []string{
`{"type":"user","sessionId":"all-bad","message":"Authorization: Bearer abcdef1234567890ghijklmnop"}`,
})
sink := &memSink{}
require.NoError(t, TickOnce(context.Background(), Config{
SessionsDir: tmp, Host: "koala", Sink: sink,
}))
assert.Empty(t, sink.batches, "no usable turns => no batch")
}
func TestTickOnce_IgnoresNonJsonlFiles(t *testing.T) {
tmp := t.TempDir()
proj := filepath.Join(tmp, "-home-mathias-dev")
require.NoError(t, os.MkdirAll(proj, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(proj, "README.md"), []byte("ignore me"), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(proj, "config.json"), []byte("{}"), 0o644))
sink := &memSink{}
require.NoError(t, TickOnce(context.Background(), Config{
SessionsDir: tmp, Host: "koala", Sink: sink,
}))
assert.Empty(t, sink.batches)
}
func TestTickOnce_HandlesMultipleProjectsAndSessions(t *testing.T) {
tmp := t.TempDir()
projA := filepath.Join(tmp, "-home-mathias-dev")
projB := filepath.Join(tmp, "-home-mathias-AI-infra")
require.NoError(t, os.MkdirAll(projA, 0o755))
require.NoError(t, os.MkdirAll(projB, 0o755))
writeSession(t, projA, "a1", []string{`{"type":"user","sessionId":"a1","message":"q1"}`})
writeSession(t, projA, "a2", []string{`{"type":"user","sessionId":"a2","message":"q2"}`})
writeSession(t, projB, "b1", []string{`{"type":"user","sessionId":"b1","message":"q3"}`})
sink := &memSink{}
require.NoError(t, TickOnce(context.Background(), Config{
SessionsDir: tmp, Host: "koala", Sink: sink,
}))
require.Len(t, sink.batches, 3)
projects := map[string]int{}
for _, b := range sink.batches {
projects[b.ProjectID]++
}
assert.Equal(t, 2, projects["-home-mathias-dev"])
assert.Equal(t, 1, projects["-home-mathias-AI-infra"])
}
func TestTickOnce_SinkErrorDoesNotKillOtherFiles(t *testing.T) {
tmp := t.TempDir()
proj := filepath.Join(tmp, "-home-mathias-dev")
require.NoError(t, os.MkdirAll(proj, 0o755))
writeSession(t, proj, "good", []string{`{"type":"user","sessionId":"good","message":"q"}`})
writeSession(t, proj, "bad-session", []string{`{"type":"user","sessionId":"bad-session","message":"q"}`})
sink := &memSink{failOn: "bad-session"}
require.NoError(t, TickOnce(context.Background(), Config{
SessionsDir: tmp, Host: "koala", Sink: sink,
}))
require.Len(t, sink.batches, 1, "good session still ingested")
assert.Equal(t, "good", sink.batches[0].SessionID)
}
func TestWatch_RespectsContextCancel(t *testing.T) {
tmp := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(tmp, "-home-mathias-dev"), 0o755))
sink := &memSink{}
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() {
done <- Watch(ctx, Config{
SessionsDir: tmp,
Host: "koala",
Interval: 10 * time.Millisecond,
Sink: sink,
})
}()
time.Sleep(50 * time.Millisecond)
cancel()
select {
case err := <-done:
assert.ErrorIs(t, err, context.Canceled)
case <-time.After(2 * time.Second):
t.Fatal("Watch did not return after cancel")
}
}
+204
View File
@@ -0,0 +1,204 @@
package mcp_test
import (
"context"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/mcp"
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// callResult parses the JSON text payload of a successful tool call.
func callResult(t *testing.T, resp map[string]any) map[string]any {
t.Helper()
require.Nil(t, resp["error"], "tool returned error: %v", resp["error"])
text := resp["result"].(map[string]any)["content"].([]any)[0].(map[string]any)["text"].(string)
var out map[string]any
require.NoError(t, json.Unmarshal([]byte(text), &out))
return out
}
func TestBrainUpdateSupersedesExisting(t *testing.T) {
brainDir := t.TempDir()
srv := mcp.NewServer(brainDir, nil, nil, nil)
// Seed via brain_write so the note carries real frontmatter.
callResult(t, toolCall(t, srv, "brain_write", map[string]any{
"content": "# Old\n\nold body\n", "filename": "val-vol",
"wing": "jepa-fx", "hall": "facts",
}))
out := callResult(t, toolCall(t, srv, "brain_update", map[string]any{
"wing": "jepa-fx", "hall": "facts", "slug": "val-vol",
"content": "# New\n\nnew body\n", "reason": "facts changed",
}))
assert.Equal(t, "wiki/jepa-fx/facts/val-vol.md", out["path"])
assert.Equal(t, out["path"], out["id"])
assert.NotEmpty(t, out["content_hash"])
assert.Equal(t, true, out["superseded"])
got, err := os.ReadFile(filepath.Join(brainDir, "wiki/jepa-fx/facts/val-vol.md"))
require.NoError(t, err)
s := string(got)
assert.Contains(t, s, "# New")
assert.NotContains(t, s, "old body")
assert.Contains(t, s, "wing: jepa-fx")
assert.Contains(t, s, "supersede_reason: facts changed")
assert.Contains(t, s, "supersedes:")
}
func TestBrainUpdateMissingTargetErrorsNoCreate(t *testing.T) {
brainDir := t.TempDir()
srv := mcp.NewServer(brainDir, nil, nil, nil)
resp := toolCall(t, srv, "brain_update", map[string]any{
"wing": "jepa-fx", "hall": "facts", "slug": "ghost",
"content": "x\n",
})
require.NotNil(t, resp["error"])
assert.Contains(t, resp["error"].(map[string]any)["message"].(string), "does not exist")
_, statErr := os.Stat(filepath.Join(brainDir, "wiki/jepa-fx/facts/ghost.md"))
assert.True(t, os.IsNotExist(statErr))
}
func TestBrainUpdateByFullPath(t *testing.T) {
brainDir := t.TempDir()
srv := mcp.NewServer(brainDir, nil, nil, nil)
callResult(t, toolCall(t, srv, "brain_write", map[string]any{
"content": "old\n", "filename": "n", "wing": "a", "hall": "facts",
}))
out := callResult(t, toolCall(t, srv, "brain_update", map[string]any{
"slug": "wiki/a/facts/n.md", "content": "fresh\n",
}))
assert.Equal(t, "wiki/a/facts/n.md", out["path"])
}
func TestBrainGetByIDAndPath(t *testing.T) {
brainDir := t.TempDir()
srv := mcp.NewServer(brainDir, nil, nil, nil)
w := callResult(t, toolCall(t, srv, "brain_write", map[string]any{
"content": "# Body\n\ntext\n", "filename": "n", "wing": "a", "hall": "facts",
}))
id := w["id"].(string)
hash := w["content_hash"].(string)
require.NotEmpty(t, id)
require.NotEmpty(t, hash)
// by id
g1 := callResult(t, toolCall(t, srv, "brain_get", map[string]any{"id": id}))
assert.Equal(t, id, g1["path"])
assert.Equal(t, hash, g1["content_hash"], "content_hash must round-trip write→get")
assert.Contains(t, g1["body"].(string), "# Body")
fm := g1["frontmatter"].(map[string]any)
assert.Equal(t, "a", fm["wing"])
// by path
g2 := callResult(t, toolCall(t, srv, "brain_get", map[string]any{"path": id}))
assert.Equal(t, hash, g2["content_hash"])
}
func TestBrainGetMissingArgsErrors(t *testing.T) {
srv := mcp.NewServer(t.TempDir(), nil, nil, nil)
resp := toolCall(t, srv, "brain_get", map[string]any{})
require.NotNil(t, resp["error"])
}
func TestBrainWriteReturnsHandle(t *testing.T) {
brainDir := t.TempDir()
srv := mcp.NewServer(brainDir, nil, nil, nil)
out := callResult(t, toolCall(t, srv, "brain_write", map[string]any{
"content": "# X\n\nbody\n", "filename": "x", "wing": "a", "hall": "facts",
}))
assert.Equal(t, "wiki/a/facts/x.md", out["path"])
assert.Equal(t, out["path"], out["id"])
assert.NotEmpty(t, out["content_hash"])
}
// --- retrieval-reflects-new-content: exercises the real mtime-driven Sync ---
type fakeVecStore struct {
chunks map[string][]float32
deleted []string
}
func (f *fakeVecStore) KnownPathsWithTime(_ context.Context) (map[string]time.Time, error) {
m := make(map[string]time.Time, len(f.chunks))
for p := range f.chunks {
m[p] = time.Unix(0, 0) // always stale → mtime(now) is always newer
}
return m, nil
}
func (f *fakeVecStore) Upsert(_ context.Context, path string, vec []float32) error {
f.chunks[path] = vec
return nil
}
func (f *fakeVecStore) Delete(_ context.Context, path string) error {
delete(f.chunks, path)
f.deleted = append(f.deleted, path)
return nil
}
type fakeEmbedder struct{ seen []string }
func (e *fakeEmbedder) Embed(_ context.Context, text string) ([]float32, error) {
e.seen = append(e.seen, text)
return []float32{1, 0, 0}, nil
}
// TestBrainUpdateReembedsNewContent proves the supersede contract end to
// end against the actual embedding mechanism: brain_update rewrites the
// file, advancing its mtime, and the next vectorstore.Sync pass re-embeds
// the NEW body and drops the stale chunk. No stub of the re-index path.
func TestBrainUpdateReembedsNewContent(t *testing.T) {
brainDir := t.TempDir()
srv := mcp.NewServer(brainDir, nil, nil, nil)
ctx := context.Background()
callResult(t, toolCall(t, srv, "brain_write", map[string]any{
"content": "# Note\n\nthe OLD distinctive payload\n",
"filename": "n", "wing": "a", "hall": "facts",
}))
store := &fakeVecStore{chunks: map[string][]float32{}}
emb := &fakeEmbedder{}
// First sync embeds the original content.
_, err := vectorstore.Sync(ctx, brainDir, store, emb)
require.NoError(t, err)
require.NotEmpty(t, store.chunks)
require.True(t, anyContains(emb.seen, "OLD distinctive payload"))
callResult(t, toolCall(t, srv, "brain_update", map[string]any{
"wing": "a", "hall": "facts", "slug": "n",
"content": "# Note\n\nthe NEW distinctive payload\n",
}))
emb.seen = nil // only watch what the second pass embeds
_, err = vectorstore.Sync(ctx, brainDir, store, emb)
require.NoError(t, err)
assert.True(t, anyContains(emb.seen, "NEW distinctive payload"),
"Sync must re-embed the superseded body; saw %v", emb.seen)
assert.False(t, anyContains(emb.seen, "OLD distinctive payload"),
"the old body must not be re-embedded")
assert.NotEmpty(t, store.deleted, "stale chunks must be deleted before re-embed")
}
func anyContains(ss []string, sub string) bool {
for _, s := range ss {
if strings.Contains(s, sub) {
return true
}
}
return false
}
+125 -1
View File
@@ -61,6 +61,26 @@ func (s *Server) tools() []map[string]any {
"hall": enum("optional memory type (requires wing)", halls...),
}),
},
{
"name": "brain_update",
"description": "Supersede an existing brain note in place: whole-note body replace + frontmatter re-stamp (updated_at, supersedes=prior content hash, supersede_reason). Errors if the target does not exist — use brain_write to create. Returns {id, path, content_hash, superseded}. Prior version recoverable from git.",
"inputSchema": schema([]string{"content"}, map[string]any{
"content": str("new full body (whole-note replace)"),
"slug": str("target note slug within wing/hall, OR a full brain-relative path (e.g. wiki/jepa-fx/facts/x.md)"),
"wing": str("wing of the target (required unless slug/path is a full path)"),
"hall": enum("hall of the target (required unless slug/path is a full path)", halls...),
"path": str("full brain-relative path to the target; takes precedence over slug/wing/hall"),
"reason": str("optional short note on why superseded — stamped into frontmatter"),
}),
},
{
"name": "brain_get",
"description": "Fetch a single brain note by id or path (both are the brain-relative path — the note handle). Returns {id, path, content_hash, frontmatter, body}. Read-after-write confirmation without a lexical re-query.",
"inputSchema": schema([]string{}, map[string]any{
"id": str("note id (brain-relative path) as returned by brain_write/brain_update"),
"path": str("brain-relative path to the note; equivalent to id"),
}),
},
{
"name": "brain_tunnel",
"description": "Create an explicit bidirectional [[wikilink]] between two notes in different wings. Idempotent.",
@@ -222,7 +242,111 @@ func (s *Server) brainWrite(ctx context.Context, args json.RawMessage) (json.Raw
}
}
s.indexInGraph(ctx, "brain_write", relPath)
return json.Marshal(map[string]string{"path": relPath})
// Read-after-write handle: id == relPath, content_hash == sha256 of
// the bytes just written. path is kept for backward compatibility.
_, _, hash, _ := api.ReadNote(s.brainDir, relPath)
return json.Marshal(map[string]string{"id": relPath, "path": relPath, "content_hash": hash})
}
type brainUpdateArgs struct {
Slug string `json:"slug,omitempty"`
Wing string `json:"wing,omitempty"`
Hall string `json:"hall,omitempty"`
Path string `json:"path,omitempty"`
Content string `json:"content"`
Reason string `json:"reason,omitempty"`
}
// brainUpdate supersedes an existing note in place: whole-note body
// replace, frontmatter re-stamp (updated_at/supersedes/supersede_reason),
// graph re-index, and wing _index rebuild. It never creates — a missing
// target is an error so the caller can fall back to brain_write.
//
// Embedding re-sync is delegated to the out-of-band vectorstore.Sync
// ticker: the rewritten file's mtime advances, so the next pass re-embeds
// it. This mirrors brain_write, which likewise does not embed in-handler.
func (s *Server) brainUpdate(ctx context.Context, args json.RawMessage) (json.RawMessage, error) {
var a brainUpdateArgs
if err := json.Unmarshal(args, &a); err != nil {
return nil, fmt.Errorf("parse args: %w", err)
}
if a.Content == "" {
return nil, fmt.Errorf("content is required")
}
opts := api.UpdateNoteOptions{Content: a.Content, Reason: a.Reason}
switch {
case a.Path != "":
opts.Path = a.Path
case strings.Contains(a.Slug, "/"):
// slug carries a full path (issue #45: "slug ... OR full path").
opts.Path = a.Slug
default:
opts.Wing, opts.Hall, opts.Slug = a.Wing, a.Hall, a.Slug
}
relPath, hash, _, err := api.UpdateNote(s.brainDir, opts)
if err != nil {
return nil, err
}
// Best-effort wiki upkeep, mirroring brain_write: rebuild the wing
// _index and re-tunnel cross-wing matches against the new body. Both
// are idempotent and never block — the note is already superseded.
if wing := wingFromRelPath(relPath); wing != "" {
if err := brain.BuildWingIndex(s.brainDir, wing); err != nil {
slog.Warn("brain_update: auto-index failed", "wing", wing, "err", err)
}
if err := brain.AutoTunnel(s.brainDir, relPath, a.Content); err != nil {
slog.Warn("brain_update: auto-tunnel failed", "src", relPath, "err", err)
}
}
s.indexInGraph(ctx, "brain_update", relPath)
return json.Marshal(map[string]any{
"id": relPath, "path": relPath, "content_hash": hash, "superseded": true,
})
}
// wingFromRelPath extracts the wing segment from a structured wiki path
// (wiki/<wing>/<hall>/<slug>.md). Returns "" for legacy/non-wiki paths.
func wingFromRelPath(relPath string) string {
parts := strings.Split(relPath, "/")
if len(parts) >= 4 && parts[0] == "wiki" {
return parts[1]
}
return ""
}
type brainGetArgs struct {
ID string `json:"id,omitempty"`
Path string `json:"path,omitempty"`
}
// brainGet fetches a note by id or path (both are the brainDir-relative
// path — the de-facto handle). Read-only; the create-path read-after-
// write primitive that lets callers confirm a write landed without a
// lexical re-query.
func (s *Server) brainGet(_ context.Context, args json.RawMessage) (json.RawMessage, error) {
var a brainGetArgs
if err := json.Unmarshal(args, &a); err != nil {
return nil, fmt.Errorf("parse args: %w", err)
}
target := a.Path
if target == "" {
target = a.ID
}
if target == "" {
return nil, fmt.Errorf("id or path is required")
}
fm, body, hash, err := api.ReadNote(s.brainDir, target)
if err != nil {
return nil, err
}
return json.Marshal(map[string]any{
"id": target, "path": target, "content_hash": hash,
"frontmatter": fm, "body": body,
})
}
// indexInGraph is a best-effort wrapper around graphsync.IndexDoc that
+7 -3
View File
@@ -1,7 +1,7 @@
// Package mcp implements an MCP HTTP handler for the ingestion service.
// Exposed tools: brain_query, brain_write, brain_index, brain_tunnel,
// brain_ingest, brain_ingest_raw, brain_answer, brain_classify,
// brain_graph, brain_context, session_log.
// Exposed tools: brain_query, brain_write, brain_update, brain_get,
// brain_index, brain_tunnel, brain_ingest, brain_ingest_raw,
// brain_answer, brain_classify, brain_graph, brain_context, session_log.
package mcp
import (
@@ -177,6 +177,10 @@ func (s *Server) handleCall(ctx context.Context, name string, args json.RawMessa
return s.brainQuery(ctx, args)
case "brain_write":
return s.brainWrite(ctx, args)
case "brain_update":
return s.brainUpdate(ctx, args)
case "brain_get":
return s.brainGet(ctx, args)
case "brain_index":
return s.brainIndex(ctx, args)
case "brain_tunnel":
+2 -1
View File
@@ -55,7 +55,8 @@ func TestServerToolsList(t *testing.T) {
names = append(names, t.(map[string]any)["name"].(string))
}
assert.ElementsMatch(t, []string{
"brain_query", "brain_write", "brain_index", "brain_tunnel",
"brain_query", "brain_write", "brain_update", "brain_get",
"brain_index", "brain_tunnel",
"brain_ingest_raw", "brain_ingest",
"brain_answer", "brain_classify", "brain_graph", "brain_context",
"session_log",
+16 -3
View File
@@ -77,9 +77,22 @@ func (s *Server) brainAnswer(ctx context.Context, args json.RawMessage) (json.Ra
return nil, fmt.Errorf("search: %w", err)
}
if s.reranker != nil && len(results) > 0 {
results, err = rerankResults(ctx, s.reranker, a.Query, results, 5)
if err != nil {
return nil, fmt.Errorf("rerank: %w", err)
reranked, rerr := rerankResults(ctx, s.reranker, a.Query, results, 5)
if rerr != nil {
return nil, fmt.Errorf("rerank: %w", rerr)
}
// The reranker is a filter, not a gate. The Qwen3-Reranker is a
// web-search cross-encoder: against a conversational / personal-
// intent query ("what am I optimizing toward?") it scores even
// on-topic notes as "no", which would collapse the whole answer to
// "no relevant content" despite BM25 having retrieved relevant
// content. When the reranker keeps nothing, fall back to the
// BM25/vector ordering (capped to the no-reranker depth) rather
// than returning an empty answer.
if len(reranked) > 0 {
results = reranked
} else if len(results) > 10 {
results = results[:10]
}
}
if len(results) == 0 {
@@ -98,6 +98,42 @@ func TestBrainAnswer_RerankerFiltersBeforeLLM(t *testing.T) {
assert.NotContains(t, sawSources, "noise.md")
}
func TestBrainAnswer_RerankerKeepsNone_FallsBackToBM25(t *testing.T) {
brainDir := brainDirWithContent(t) // test.md BM25-matches "pass-rate logging"
// Reranker rejects every candidate ("no" to all) — models a
// web-search cross-encoder facing a conversational / personal-intent
// query, which is exactly when it wrongly scores on-topic notes as
// irrelevant. The answer must still synthesize from the BM25 hits, not
// collapse to "no relevant content".
rrSrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_ = json.NewEncoder(w).Encode(map[string]any{"response": "no", "done": true})
}))
defer rrSrv.Close()
var sawSources string
llm := func(_ context.Context, _, user string) (string, error) {
sawSources = user
return "fallback answer", nil
}
srv := mcp.NewServer(brainDir, nil, nil, llm).
WithReranker(reranker.New(rrSrv.URL, "qwen3"))
ts := httptest.NewServer(srv)
defer ts.Close()
rpc := callTool(t, ts, "brain_answer", map[string]any{"query": "pass-rate logging"})
require.Nil(t, rpc["error"])
content := rpc["result"].(map[string]any)["content"].([]any)[0].(map[string]any)["text"].(string)
var result map[string]any
require.NoError(t, json.Unmarshal([]byte(content), &result))
assert.Equal(t, "fallback answer", result["answer"])
assert.NotEmpty(t, result["sources"], "reranker keeping nothing must fall back to BM25, not empty")
assert.Contains(t, sawSources, "test.md")
}
func TestBrainAnswer_NoLLM(t *testing.T) {
srv := mcp.NewServer(t.TempDir(), nil, nil, nil)
ts := httptest.NewServer(srv)
+65
View File
@@ -3,6 +3,7 @@ package vectorstore
import (
"fmt"
"strings"
"unicode/utf8"
)
// NumberedChunk pairs a chunk's body with the storage path it will use
@@ -66,6 +67,70 @@ func ChunkMarkdown(content string, maxBytes int) []string {
}
out = append(out, splitAtParagraphs(s, maxBytes)...)
}
// Final guarantee: no chunk exceeds maxBytes. A single heading-less,
// paragraph-less block (JSON-lines, minified content) survives the two
// passes above whole — splitAtParagraphs emits an over-budget paragraph
// rather than truncating prose. Hard-split any such chunk at line/rune
// boundaries so the embedder never rejects an over-context chunk.
final := make([]string, 0, len(out))
for _, c := range out {
if len(c) <= maxBytes {
final = append(final, c)
continue
}
final = append(final, hardSplit(c, maxBytes)...)
}
return final
}
// hardSplit slices s into pieces no larger than maxBytes, breaking at line
// boundaries where possible and otherwise mid-line at a UTF-8 rune boundary.
// Last resort for content that has neither headings nor blank-line paragraphs.
func hardSplit(s string, maxBytes int) []string {
var out []string
var cur strings.Builder
flush := func() {
if cur.Len() > 0 {
out = append(out, cur.String())
cur.Reset()
}
}
for _, line := range strings.SplitAfter(s, "\n") {
if line == "" {
continue
}
if len(line) > maxBytes {
flush()
out = append(out, runeSplit(line, maxBytes)...)
continue
}
if cur.Len() > 0 && cur.Len()+len(line) > maxBytes {
flush()
}
cur.WriteString(line)
}
flush()
return out
}
// runeSplit slices s into <=maxBytes pieces without splitting a UTF-8 rune.
func runeSplit(s string, maxBytes int) []string {
var out []string
for len(s) > maxBytes {
cut := maxBytes
for cut > 0 && !utf8.RuneStart(s[cut]) {
cut--
}
if cut == 0 { // single rune wider than the budget; emit it whole
cut = maxBytes
}
out = append(out, s[:cut])
s = s[cut:]
}
if len(s) > 0 {
out = append(out, s)
}
return out
}
@@ -30,6 +30,23 @@ func TestChunkMarkdown_SplitsAtHeadings(t *testing.T) {
}
}
func TestChunkMarkdown_HardSplitsHeadinglessOversizedBlock(t *testing.T) {
// A document with no headings and no blank-line paragraph breaks (e.g.
// JSON-lines like wiki/telos/decisions/human-intent-column.md). The old
// chunker emitted it as one over-budget chunk → nomic-embed returned
// "input length exceeds the context length" (400). Every chunk must now
// fit the budget, with no content lost.
maxBytes := 200
src := strings.Repeat("x", 1000) // one 1000-byte blob, no headings, no \n\n
out := vectorstore.ChunkMarkdown(src, maxBytes)
require.Greater(t, len(out), 1, "oversized blob must be split")
for i, c := range out {
assert.LessOrEqual(t, len(c), maxBytes, "chunk %d over budget: %d bytes", i, len(c))
}
assert.Equal(t, 1000, strings.Count(strings.Join(out, ""), "x"), "no content lost")
}
func TestChunkMarkdown_FurtherSplitsOversizedSection(t *testing.T) {
// One H2 section with 4 paragraphs of ~80 chars each, limit 100.
src := "## big\n\n" +