Compare commits

...
76 Commits
Author SHA1 Message Date
mathiasandClaude Sonnet 5 f0055483a3 fix(watcher): never re-ingest AutoTunnel's own tunnel-candidates log (#88)
CI / Lint / Test / Vet (push) Failing after 1s
CI / Mirror to GitHub (push) Has been skipped
Root cause was neither of the two mechanisms the issue named
(CanonicalizeLinks over-canonicalizing, or a schema/prompt bias) —
CanonicalizeLinks correctly resolved [[Mission]] because
wiki/concepts/mission.md genuinely exists; the extraction prompt has
no "mission" bias either. The real bug: watcher.processDir walks
brain/raw/ and feeds every .md/.txt/.pdf file to pipeline.Run as
source material to extract, with no exclusion for
tunnel-candidates-*.md — AutoTunnel's own fuzzy-match human-review
queue (logFuzzyCandidates writes it into that same raw/ directory).
The watcher's next poll re-ingested the raw candidate log itself, and
the LLM naturally surfaced "mission" as a wikilink because the log
literally lists `(term: "mission")` entries in its content — a
generic-word title match DetectTunnels correctly flagged as fuzzy
(not auto-tunneled) but which still leaked downstream once treated as
real source material.

api.ListPending already excludes tunnel-candidates-* when listing
raw/ for promotion; processDir gets the same exclusion.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 14:37:50 +02:00
mathiasandClaude Sonnet 5 6d014c1d0f fix(ingest): CanonicalizeLinks strips wing:/wiki/ prefix and repairs path-style links (#87)
CI / Mirror to GitHub (push) Has been skipped
CI / Lint / Test / Vet (push) Failing after 2s
plainLinkRE matches any [[...]] without a pipe, including path-prefixed
forms like [[wing:homelab/failures/foo]] or [[wiki/agentsquad/decisions/bar]]
that the LLM extraction step occasionally emits. Title-lookup against
titleToSlug always fails for these (they're paths, not titles), so they
were silently left broken.

Before falling to the unknown-wikilink warning, strip a known wing:/wiki/
root prefix and emit the clean wing/hall/slug path directly — matching
the brain-graph path-style link convention.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 14:32:31 +02:00
mathiasandClaude Sonnet 5 1001acfb44 fix(ingest): merge existing frontmatter in writeHallNote instead of stacking (#86)
CI / Lint / Test / Vet (push) Failing after 1s
CI / Mirror to GitHub (push) Has been skipped
opts.Content can already carry its own "---\n...\n---" block (e.g. from
an upstream extraction/promotion step). writeHallNote used to prepend a
second block unconditionally, making a standard frontmatter parser blind
to everything the first block declared (title, tags, ...).

splitFrontmatter now pulls the existing block's fields out first;
wing/hall/created_at are always fresh-injected, and type/domain/
source_type prefer the note's own existing value over the opts
fallback. Remaining existing fields are carried through verbatim into
the single merged block.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-26 23:45:36 +02:00
mathiasandClaude Sonnet 5 6ad275b505 fix(ingest): thread source/author/published frontmatter through extraction (#85)
CI / Lint / Test / Vet (push) Failing after 1s
CI / Mirror to GitHub (push) Has been skipped
pipeline.Run parses source/author/published from the raw content's own
frontmatter and applies them deterministically to source-type RawPages
after LLM extraction — never relies on the LLM to copy them through.
buildFrontmatter now emits all three (source pages only) when present.

Backfill of the 46 already-affected wiki/sources/*.md notes is a
separate concern per the issue, not done here.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-26 23:42:33 +02:00
mathiasandClaude Sonnet 5 b600cc986c chore(module): rename module github.com/mathiasbq/supervisor -> git.d-ma.be/mathias/hyperguild (#42)
CI / Lint / Test / Vet (push) Failing after 5s
CI / Mirror to GitHub (push) Has been skipped
Aligns module path with canonical Gitea source (infra ADR-0004) and
repo name. Binary only, no downstream consumers.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-26 23:37:11 +02:00
mathiasandClaude Opus 4.8 9bdab1c48c fix(ingest): parse docmark's SSE-framed response, not bare JSON
CI / Lint / Test / Vet (push) Successful in 16s
CI / Mirror to GitHub (push) Has been skipped
Found via a real live-cluster end-to-end test (POST /ingest-path with a real
.docx against the deployed ingestion + docmark): 'decode docmark response:
invalid character e looking for beginning of value'. docmark's Streamable-HTTP
transport frames every response as SSE (event: message\r\ndata: {...}\r\n\r\n,
Content-Type: text/event-stream) -- stateless_http=True removes the need for
an initialize handshake/session ID, it does NOT change the wire framing to
bare JSON. My original client + its own test mock both assumed bare JSON,
so the unit tests passed while the real call failed.

sseDataPayload() extracts the data: line before JSON-unmarshaling; falls back
to the raw body if no SSE framing is present (forward-compatible). Test mock
(docmark_test.go) now emits the SAME framing the live server actually sends,
confirmed by directly probing docmark's live response before writing the fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 08:53:24 +02:00
mathiasandClaude Opus 4.8 6520c2fc4b feat(ingest): route DOCX/XLSX/PPTX/images through docmark (ADR-0013)
CI / Lint / Test / Vet (push) Successful in 16s
CI / Mirror to GitHub (push) Successful in 4s
Wires the sovereign docmark MCP server (git.d-ma.be/mathias/docmark) as the
converter for formats ingestion previously couldn't handle at all -- DOCX,
XLSX, PPTX, PNG/JPG. A single self-contained tools/call POST (docmark runs
stateless_http, no session handshake needed) does the conversion.

- internal/extract/docmark.go: extractViaDocmark(path) -- reads DOCMARK_URL +
  DOCMARK_BEARER_TOKEN from env, base64-encodes the file, calls docmark's
  convert_to_markdown tool, surfaces tool/HTTP errors with context.
- internal/extract/extract.go: routes .docx/.xlsx/.pptx/.png/.jpg/.jpeg to it.
- internal/api/handler.go: isSupportedExtension() replaces the static
  supportedExtensions map -- the new extensions are gated on DOCMARK_URL being
  set, checked per-request. When DOCMARK_URL is unset (e.g. this repo's
  current deployed state), behavior is byte-for-byte unchanged from today:
  same 400/silent-skip as before. Zero risk until the env var is actually
  wired in infra.

TDD throughout: docmark.go tested via httptest mock (success, not-configured,
tool-error, HTTP-error, all 6 new extensions route correctly); handler-level
test proves the DOCMARK_URL gate (docx 400 when unset, 200 when a working
mock is configured). Full task check green (fmt/vet/govulncheck/all tests).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 08:46:03 +02:00
mathiasandClaude Opus 4.8 fcbd1072b6 fix(lint): check fmt.Fprintf errors in webhook handler (infra#190)
CI / Lint / Test / Vet (push) Successful in 16s
CI / Mirror to GitHub (push) Has been skipped
errcheck flagged two unchecked fmt.Fprintf return values in
internal/webhook/webhook.go, failing CI (run 310/311) for the whole
'trigger brain-sync on Gitea push' feature -- so no new ingestion image was
ever built, and the deployed image predates this feature entirely even
though infra's manifest (secret, RBAC, env wiring) was already live.

Both writes are best-effort informational text after WriteHeader has already
committed the status code -- a failed write here only happens on client
disconnect and nothing depends on it succeeding, so explicitly discard
(_, _ =) rather than log-and-continue noise.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 08:29:39 +02:00
mathias 3b7706b358 chore(context): re-sync derived adapters (skills-source paragraph) 2026-07-09 08:28:54 +02:00
mathias 394a227877 feat(webhook): trigger brain-sync on Gitea push instead of 15-min poll
CI / Mirror to GitHub (push) Has been skipped
CI / Lint / Test / Vet (push) Failing after 8s
Adds POST /webhooks/brain-sync: verifies Gitea's HMAC-SHA256 signature,
checks the push is to mathias/brain main, then creates a one-off Job from
the existing brain-sync CronJob's template (same script the 15-min poll
already runs, just triggered on-demand). Off by default -- opt in via
GITEA_WEBHOOK_SECRET, since it needs Job-create RBAC in the "brain"
namespace a fresh deploy won't have.

10 new tests (internal/webhook), including a fake-clientset reactor to
simulate server-side GenerateName expansion, which the plain fake tracker
doesn't do on its own.

Needs (follow-up, infra repo): RBAC granting ingestion's ServiceAccount
get on cronjobs/brain-sync + create on jobs in the brain namespace, the
GITEA_WEBHOOK_SECRET env, and the actual Gitea webhook registration.
2026-07-07 22:00:56 +02:00
mathias 0f84ab5eda fix(ingest): route raw claude-session dumps to non-indexed archive, not wiki
CI / Lint / Test / Vet (push) Successful in 11s
CI / Mirror to GitHub (push) Has been skipped
claudeSink wrote every raw transcript into brain/wiki/claude-sessions/facts/
(BM25-indexed), duplicating what ai-sessions' curated summaries already
cover and out-ranking them in search (177,818 vs 45,335 BM25 weight on the
same query, per ai-sessions#10 investigation). Raw dumps now land in
brain/archive/claude-sessions/<host>/ — kept for deep lookups, never
indexed, never deleted.

Refs: ai-sessions#10
2026-07-07 11:36:01 +02:00
mathias 5b57843346 fix(lint): revert redundant Write conversion (staticcheck S1016)
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 3s
writeRequest and WriteNoteOptions have matching fields again now that
both carry SourceType, so the explicit field-by-field literal added
in ee1204d is flagged as a redundant conversion. Back to the plain
type conversion. Caught by task check (golangci-lint), not run
locally before the previous push -- plain go test passed, task check
(which adds -race and golangci-lint) didn't.
2026-07-04 14:13:16 +02:00
mathias ee1204d76b fix(api): default hall=facts source_type to internal (brain-gardener#7)
CI / Lint / Test / Vet (push) Failing after 5s
CI / Mirror to GitHub (push) Has been skipped
The unsourced-check fix (source_type: internal) only covered
claudewatcher's writes. 41 residual findings on the audit's next run
were manually-authored hall=facts entries written via brain_write/
capture -- also first-party observations, just a different write path,
with no way to mark them internal short of hand-editing every entry.

writeHallNote now defaults source_type to internal whenever hall ==
"facts" and the caller didn't set it explicitly. An explicit
source_type: external (now threaded through the /write HTTP endpoint)
opts a genuinely citation-needing entry back out.
2026-07-04 14:09:20 +02:00
mathias 6d58336ce2 fix(claudewatcher): tag session-dump notes source_type: internal
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 3s
brain-gardener#5: the audit's unsourced check was flagging 59 raw
claudewatcher session dumps as high-severity hallucination risk,
because it can't distinguish a first-party observation (a session
transcript) from an external claim needing citation.

Adds WriteNoteOptions.SourceType, emitted as source_type: <value> in
the wing/hall frontmatter route. claudeSink (the claudewatcher sink)
now always sets source_type: internal on the notes it writes.
2026-07-04 13:53:08 +02:00
mathias 0785f14220 Merge pull request 'Consolidate to single harness: icebox cmd/routing, keep cmd/hyperguild + ingestion (#75)' (#76) from feat/consolidate-single-harness into main
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 4s
2026-07-01 16:27:06 +00:00
mathiasandClaude Opus 4.8 a7db0dd00d feat(mode): brain+gitea in all modes; drop iceboxed routing entry (#75)
CI / Lint / Test / Vet (pull_request) Successful in 12s
CI / Mirror to GitHub (pull_request) Has been skipped
Step 7 of the consolidation: reflect gitea-mcp as an available connection
(Gitea-as-audit-trail is an invariant going forward) and stop emitting the
now-iceboxed routing pod.

- cmd/hyperguild mode.go: every mode (cloud/client-local/sovereign) now lists
  brain + gitea; removed the client-local `routing` entry (koala:30310, the
  iceboxed pod). Tests updated to the new contract.
- .mcp.json: add gitea-mcp (git-mcp.d-ma.be, Bearer ${GITEA_MCP_TOKEN}).
- cmd/hyperguild/README: modes doc updated (brain+gitea, no routing).

Refs #75.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 13:27:31 +02:00
mathiasandClaude Opus 4.8 fb59c390e2 refactor(harness): icebox cmd/routing path — consolidate to single harness (#75)
Consolidate to the hyperguild-alone harness that ran the infra#170 loop-1
experiment (cmd/hyperguild + brain-mcp; routing/injection machinery off).
Verified cmd/hyperguild has zero transitive dep on internal/routing or
cmd/routing's packages (imports only internal/tier). Removed the routing
path, which is dormant from the live session (.mcp.json wires only brain-mcp)
and entangled with the survivorship-biased pass-rate router (infra#174).

Iceboxed (recoverable at tag icebox/cmd-routing-2026-07-01 / branch
icebox/cmd-routing, commit 00e5f62):
- cmd/routing/, internal/routing/
- internal/skills/{review,debug,retrospective,trainer,project}/
  (each imported solely by cmd/routing — verified)
- Dockerfile.routing
- cd.yml: routing image build + infra bump + Flux-wait/rollout-verify
  (ingestion build/deploy unchanged)

Kept: cmd/hyperguild, ingestion/, internal/tier (used by cmd/hyperguild +
skills/org), internal/skills/{brain,org,sessionlog} (per #75; already orphaned).

Regression: root module 21→14 pkgs (−7 = exactly the removed set, no other
breakage); ingestion 24/24 unchanged. See ICEBOX.md. Refs #75, infra#170, #174.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 13:22:26 +02:00
mathiasandClaude Opus 4.8 00e5f62c8e docs(runbook): correct flywheel (nil→local) + instrumentation-fixed note
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 4s
Two corrections after the #73 investigation: (1) the router routes cold
(nil pass-rate) calls to the LOCAL fast tier, not cloud — the earlier
"pay in on cloud" description was wrong (per policy.go). (2) Note that
pass-rate logging was broken until v0.11.1 (#73) and is now verified;
reset the window to 2026-06-30→07-14. Refs #73, #35.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 22:16:03 +02:00
mathiasandClaude Opus 4.8 b9d03316fd fix(ingestion): migrate mcp-chassis import to git.d-ma.be path (#74)
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Has been skipped
Unblocks CD: the build's go mod download failed because mcp-chassis was
imported via the old gitea.d-ma.be module path, which the renamed server no
longer serves a matching go-import meta tag for. Point at the renamed module
git.d-ma.be/mathias/mcp-chassis v0.2.0 (mcp-chassis 69d389d) + go mod tidy.
Fresh download now resolves; ingestion builds + tests green.

This restores the normal CI path so the routing pass-rate fix (da9bdc4,
#73) builds and deploys through gitea Actions. Refs #74, #73, #35.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 21:39:41 +02:00
mathiasandClaude Opus 4.8 da9bdc4cbb fix(routing): repair pass-rate instrumentation (3 bugs) — #73, #35
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 4s
The #35 data gate could never fill: a real review call routed cleanly to
qwen36 but /pass-rate stayed total:0 under every key. Root cause was three
independent defects in the session_log path, each alone fatal:

- A: a successful routed call logged final_status "skip", never "pass".
  /pass-rate computes pass/(pass+fail) and skips count as neither, so the
  >=0.90 gate was mathematically unreachable. Success now logs "pass".
- B: every record was written under skill "_routing", so /pass-rate?skill=
  review|debug (what #35 measures) always read zero. Now uses the real
  e.Skill; routing decisions stay groupable via session_id "_routing".
- C: the session_log POST to the bearer-gated ingestion /mcp carried no
  Authorization header → silent 401, swallowed by best-effort logging
  (the documented mcpclient-empty-token-silent-401 footgun). Logger now
  takes a token (BRAIN_MCP_TOKEN) and sets the bearer when non-empty.

Tests rewritten to assert correct behavior (they had encoded the bugs:
"skip" on success, "_routing" skill). New test covers the auth header and
the empty-token path.

Infra (BRAIN_MCP_TOKEN ExternalSecret + env on the routing deployment) and
redeploy follow separately. Refs #73, #35.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 08:36:18 +02:00
mathiasandClaude Opus 4.8 dcb9ff4a56 docs(runbook): how to exercise review/debug traffic for the pass-rate gate
CI / Lint / Test / Vet (push) Successful in 20s
CI / Mirror to GitHub (push) Successful in 3s
The #35 data gate stays at zero because pass-rate only accrues from review/
debug calls *through the routing pod* — Crush, cloud chat, and the local
.skills all bypass it. Document the connect → route → flywheel steps, the
endpoints (koala:30310/mcp + routing-mcp.d-ma.be), the cold-start behavior
(nil pass-rate routes cloud until passes accrue past the 0.90 floor, then
local qwen36 activates), the 50-invocation / 2026-07-10 target, and the
Berget fallback. Refs #35.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 23:41:07 +02:00
mathias 0454527b83 Merge pull request 'feat: brain_pending + brain_promote — close the raw→wiki curation loop (#38)' (#70) from feat/brain-pending-promote into main
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 3s
2026-06-26 14:49:27 +00:00
mathiasandClaude Opus 4.8 ef11864121 feat(mcp): brain_pending + brain_promote tools + REST routes (#38)
CI / Lint / Test / Vet (pull_request) Successful in 13s
CI / Mirror to GitHub (pull_request) Has been skipped
Wires the curation primitives into the MCP surface (all three sites:
tools() descriptors, handleCall dispatch, package doc) and registers the
GET /pending + POST /promote REST routes. brain_promote additionally
re-indexes the promoted note into the graph (best-effort), matching the
other write paths. brain_pending is the human-review-queue complement to
the agent-facing brain_write.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 16:34:15 +02:00
mathiasandClaude Opus 4.8 14b04a25cb feat(brain): ListPending + PromoteNote — raw→wiki curation primitives (#38)
Closes the human curation loop: list brain/raw/ notes awaiting review
(oldest-first, with excerpt) and promote one into brain/wiki/<wing>/<hall>/.

PromoteNote rewrites frontmatter (sets wing/hall/promoted_at, preserves
created_at + custom fields via the existing frontmatter editor), rebuilds
the wing _index, and runs auto-tunnel. Atomic from the caller's view:
hall/wing/slug/collision validation happens before any fs change, and the
source is deleted only after the destination write succeeds (write-then-
delete, never move) — a collision or write failure leaves raw/ intact.
Default slug = filename minus the YYYY-MM-DD- prefix.

Also exposes GET /pending + POST /promote for shell scripts (bad
hall/collision/missing-source → 400, not 500).

Scope note: operates on raw/ (the retrospective-skill review queue) per
this issue's spec. The knowledge/ legacy pile is #22's bulk-migration
concern, not this ongoing queue.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 16:34:15 +02:00
mathias 66a9b8e725 Merge pull request 'docs(close-session): refresh classification gate post-#67 (#62)' (#69) from fix/close-session-classification-refresh into main
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 3s
2026-06-26 14:28:48 +00:00
mathiasandClaude Opus 4.8 9f8fb9c138 docs(close-session): refresh classification gate post-#67 (#62)
CI / Lint / Test / Vet (pull_request) Successful in 13s
CI / Mirror to GitHub (pull_request) Has been skipped
The capture-routing in 723dab5 (#62 minimal) carried pre-#67 prose: it
warned "no populated classification.yaml yet" and steered repos_touched
away from brain/ai-sessions as if they'd escalate to confidential. #67
tagged the homelab repos internal, so that guidance is now wrong and
over-restrictive — listing the central repos is fine, and the summary's
fixed ai-sessions target no longer escalates the call. Point at
classification.yaml as the source of truth; reserve the refusal warning
for client-*/untagged material. Completes #62's veneer.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 16:25:57 +02:00
mathias d39a18dd69 Merge pull request 'fix: wire ai-sessions SummaryWriter into the capture relay (#66)' (#68) from fix/capture-summarywriter into main
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 4s
2026-06-23 15:22:42 +00:00
mathiasandClaude Opus 4.8 5288554338 fix(gitea): WriteFile creates via POST, updates via PUT (real contents API)
CI / Lint / Test / Vet (pull_request) Successful in 13s
CI / Mirror to GitHub (pull_request) Has been skipped
A live token-scope probe against mathias/ai-sessions revealed gitea's
contents API uses POST to create and PUT (sha required) to update — the
first impl always PUT'd, so creating a new summary 422'd "[SHA]: Required".
The httptest mock had the same wrong assumption. Pick the method by
whether the file exists (GET sha). Token confirmed contents:write
(push:true) by the probe.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 17:22:37 +02:00
mathiasandClaude Opus 4.8 2368564523 fix(capture): wire ai-sessions SummaryWriter into the relay (#66)
CI / Lint / Test / Vet (pull_request) Successful in 13s
CI / Mirror to GitHub (pull_request) Has been skipped
The deployed CaptureService was constructed with a nil SummaryWriter, so
a capture carrying a summary block returned the partial-failure
"no summary writer configured" (surfaced in the 2026-06-23 claude.ai
dogfood). The Gitea client already reaches mathias/* over BRAIN_GITEA_TOKEN
and now implements SummaryWriter, so inject it (type-asserted from the
tracker) — no new credential, no manifest change. Session summaries now
write to mathias/ai-sessions at summaries/<harness>/<YYYY-MM>/...

Reuses the existing token deliberately; assumes it carries contents:write
scope on ai-sessions (it already does issue writes for the tracker). If
the token is issue-scoped only, the live write 422s — a token-scope widen,
not a code fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 10:47:00 +02:00
mathiasandClaude Opus 4.8 0e28b2125b feat(gitea): WriteFile contents-API upsert — satisfies SummaryWriter (#66)
Adds Client.WriteFile (Gitea contents API) so the Gitea client also
implements capture.SummaryWriter. Upserts: a GET resolves the current
blob sha so an existing file is updated (the richer-fidelity-supersedes
rule for re-captured sessions) rather than 422'd. Owner stays the fixed
const; token only in the Authorization header (no leak — regression
tested). Refactors the HTTP path into a shared request() helper so the
contents flow can branch on 404 without it being an error.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 10:47:00 +02:00
mathias 723dab51ae feat(close-session): route closeout through the capture tool (#62 minimal)
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 3s
Collapse Phases 4 (summary commit) + 5 (brain note) into a single
capture-driven Phase 4. The skill now assembles one brain:capture payload
(insights + tickets + summary + context) and dry-runs-then-executes;
capture owns the brain/gitea/ai-sessions writes, the I1 gate, the I5
audit record, and the supersession/read-after-write discipline server-side.

Adds the classification-gate lesson learned dogfooding capture this
session: effective classification = strictest across ALL targets
(wings, ticket repos, repos_touched); untagged repos (brain, ai-sessions)
fail-safe to confidential and get refused via claude.ai, so repos_touched
must stay to internal-default repos. Legacy inline path kept only as the
capture-unreachable fallback. Staleness prose removed (server owns it now).

Partial of #62; harness-token + harvest-adapter + fidelity-supersession
work remain.
2026-06-23 06:27:57 +00:00
mathiasandClaude Opus 4.8 06e21c019e docs(capture): as-built implementation report for the #49 epic
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 4s
Records what shipped (v0.11.0): architecture, sub-issue→PR map, I1–I5
compliance, the operational env reference, test coverage, and the
deferred follow-ups. Companion to specs/capture-bdd-spec.md (the design
contract) for onboarding + future audit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 07:41:24 +02:00
mathias 76514215f4 Merge pull request 'feat: capture relay — MCP capture tool for non-library harnesses (#55, capture 49f)' (#61) from feat/capture-mcp-relay into main
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Has been skipped
2026-06-23 05:26:55 +00:00
mathiasandClaude Opus 4.8 7cf5bc221d feat(mcp): capture relay tool — MCP door for non-library harnesses (#55)
CI / Lint / Test / Vet (pull_request) Successful in 13s
CI / Mirror to GitHub (pull_request) Has been skipped
Adds the `capture` MCP tool: the #55 relay for harnesses that cannot run
the use-case in-process (claude.ai Chat/Cowork/Design, Crush, Pi, LLM
Council). They reach it through the existing /mcp OAuth connector.

- Thin: forwards to the SAME CaptureService as POST /capture; holds no
  state and retains nothing beyond the I5 audit record. The containment
  properties accepted in infra security-baseline (I2 ledger) hold by
  construction.
- Per-principal: ServeHTTP re-derives the caller's principal from the
  Bearer header (the chassis middleware gates but discards it) and stashes
  it in context; the tool resolves the trust-zone origin from it. A
  caller-asserted harness/origin in the body is ignored — origin is
  server-derived, so the I1 confidential refusal still fires for us-nexus
  callers (claude.ai), and sovereign-allowlisted JWT principals pass.
- Registered only when WithCapture is wired (all three sites: tools(),
  handleCall, package doc); main wires REST + MCP from the same service,
  resolver, and credentials.

Tests: listed-only-when-wired, forwards-via-static-principal,
confidential-via-us-nexus-refused, confidential-via-sovereign-allowed,
unauthenticated-rejected, caller-cannot-forge-origin.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 00:21:11 +02:00
mathiasandClaude Opus 4.8 f78a5474a5 refactor(capturehttp): export Authenticate + DecodeRequest for reuse (#55)
Lifts the Bearer principal-derivation and the request→CaptureInput decode
out of the REST handler into exported package funcs, so the MCP capture
tool (#55 relay) reuses the exact same auth precedence and wire shape —
one implementation, not two. No behaviour change to POST /capture.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 00:21:11 +02:00
mathias c307b72bd5 Merge pull request 'feat: I5 audit path + classification-aware degradation (#54, capture 49e)' (#60) from feat/capture-audit-degradation into main
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 4s
2026-06-22 21:57:52 +00:00
mathiasandClaude Opus 4.8 38a2e91002 feat(capturehttp): 503 on audit-unavailable; wire degrading sink (#54)
CI / Lint / Test / Vet (pull_request) Successful in 12s
CI / Mirror to GitHub (pull_request) Has been skipped
- Map capture.ErrAuditUnavailable → HTTP 503 (audit substrate down /
  confidential unauditable / floor).
- main: buildAuditSink selects the DegradingSink (loki central + durable
  file buffer under brainDir + optional ntfy) when BRAIN_LOKI_URL is set
  and starts the reconcile loop; else the plain slog sink. Notifier kept
  as a nil interface (not typed-nil) when unconfigured so the sink and
  reconcile skip it cleanly.

Env: BRAIN_LOKI_URL, BRAIN_NTFY_URL, BRAIN_NTFY_TOKEN,
BRAIN_AUDIT_RECONCILE_INTERVAL (default 60s). Buffer at
<brain>/.audit-buffer/capture.jsonl.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 23:54:24 +02:00
mathiasandClaude Opus 4.8 77f5e06d6b feat(audit): DegradingSink + durable buffer + loki/ntfy + reconcile (#54)
The classification-aware I5 audit path (§4.4):

- DegradingSink.Reserve: central up → AuditCentral; central down +
  confidential → refuse (no buffer); central down + internal/public +
  buffer writable → AuditBuffered; central down + buffer unwritable →
  floor refuse. Record executes the reserved outcome and, when buffered,
  fires an ntfy alert.
- FileBuffer: durable JSONL buffer that survives process restart; Confirm
  rewrites the file without a record, so a buffered record is cleared ONLY
  after its central write is confirmed.
- LokiCentral: /ready probe + /loki/api/v1/push (full audit entry as the
  structured line). NtfyNotifier: degraded-state alerts; token only in the
  auth header, never logged (regression-tested).
- Reconcile + StartReconcile: replay buffered records to central on
  recovery, confirm-then-clear per record; a failed push keeps the record
  buffered (no loss). SlogSink updated to the two-phase shape (always
  central, never fails) — the default when no loki endpoint is set.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 23:54:24 +02:00
mathiasandClaude Opus 4.8 202212e8d5 feat(capture): two-phase classification-aware AuditSink port (#54)
Splits the audit port into Reserve (before any write) + Record (after),
so "confidential + sink-down → refuse before any write" is literally true
even though the audit record — which lists what landed — can only be
written afterwards.

- AuditSink.Reserve(ctx, level) → AuditOutcome | error. The error path
  refuses the capture before writing: confidential + central sink down,
  or the all-tiers floor (nothing can record).
- AuditSink.Record(ctx, entry, outcome) persists per the reserved outcome.
- Service: I5 gate runs after the I1 gate and after the dry-run
  short-circuit (dry-run never probes the sink). AuditBuffered surfaces on
  the receipt. New ErrAuditUnavailable sentinel (→ HTTP 503).

The tier→behaviour decision lives in the sink impl (#54's DegradingSink),
not the service — the service just honours Reserve's verdict.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 23:54:24 +02:00
mathias b7938d4636 Merge pull request 'feat: POST /capture REST adapter + OAuth2 + I1 sovereignty gate (#53, capture 49d)' (#59) from feat/capture-rest-i1-gate into main
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 4s
2026-06-22 21:44:14 +00:00
mathiasandClaude Opus 4.8 a1997838b0 feat(capturehttp): POST /capture REST adapter + OAuth2 + origin resolver (#53)
CI / Lint / Test / Vet (pull_request) Successful in 12s
CI / Mirror to GitHub (pull_request) Has been skipped
The HTTP door for the capture capability. Thin: authenticate → derive
trust-zone origin → decode → capture.Service → map receipt to status.

- Auth mirrors the chassis Bearer precedence (static token wins, then Dex
  JWT) but returns the resolved principal + auth path, which the chassis
  middleware hides — capture needs the principal to derive the origin.
  Depends on a small Validator interface (the chassis *JWTValidator
  satisfies it) so the JWT/origin path is testable without a live JWKS.
- OriginResolver maps principal → trust zone: static-token caller and
  allowlisted JWT subjects → sovereign; every other principal → us-nexus
  (fail safe, so the I1 gate refuses confidential by default). Principal
  and origin are server-set on the input, overwriting any body the caller
  sent.
- HTTP status: 200 all-ok / dry-run, 207 partial, 502 all-failed, 403 on
  the I1 refusal, 400 on fail-closed validation.
- Wired in main behind the same static+JWT credentials as /mcp, reusing
  the MCP server's graph-wired brain store (one implementation) and the
  classification tags (#50). Mounts only when a Gitea tracker is
  configured. Sovereign JWT principals via BRAIN_CAPTURE_SOVEREIGN_PRINCIPALS.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 23:42:18 +02:00
mathiasandClaude Opus 4.8 77680c7445 feat(audit): minimal slog AuditSink for capture I5 (#53)
Emits the request-level audit record to structured logs (scraped by the
existing alloy/loki substrate) and surfaces security events at warn
level. Placeholder behind the AuditSink interface — the classification-
aware loki+buffer+reconcile sink (confidential fails closed, internal
degrades) lands in #54 and replaces this without touching callers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 23:42:18 +02:00
mathiasandClaude Opus 4.8 d7a842f356 feat(capture): I1 sovereignty gate + server-derived origin (#53)
Adds the trust-zone Origin to CaptureContext and the I1 gate to the
use-case: a confidential effective classification through a us-nexus
origin is refused before ANY write (ErrSovereigntyRefused), and the
refusal is itself audited. A caller-asserted harness label that names a
different zone than the server-derived origin is logged as a security
event — context.Harness is descriptive-only, never a gate input.

The gate triggers only on an explicit ZoneUSNexus, so the unset default
(ZoneUnknown) can never make it fire on caller-controllable input; the
REST adapter always sets a concrete zone.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 23:42:18 +02:00
mathias aad90f2dfe Merge pull request 'feat: Gitea IssueTracker client + inject into brain server (#52, capture 49c)' (#58) from feat/capture-gitea-tracker into main
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 4s
2026-06-22 21:32:02 +00:00
mathias 07fca9ee73 Merge pull request 'feat: CaptureService use-case + BrainStore extraction (#51, capture 49b)' (#57) from feat/capture-service into main
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Has been cancelled
2026-06-22 21:31:47 +00:00
mathias f6bf9b5f57 Merge pull request 'feat: capture classification taxonomy + per-wing/repo tags (#50, capture 49a)' (#56) from feat/capture-classification into main
CI / Lint / Test / Vet (push) Has been cancelled
CI / Mirror to GitHub (push) Has been cancelled
2026-06-22 21:31:18 +00:00
mathiasandClaude Opus 4.8 6606b38a76 feat(gitea): IssueTracker client + inject into brain server (#52)
Implements the IssueTracker port as a real Gitea REST client (#49c) — the
new outbound dependency the brain server gains for capture.

- CreateIssue / CommentIssue / CloseIssue(+optional closing comment) over
  the Gitea API. Owner is the const "mathias", never caller-supplied, so
  a caller cannot redirect a write to another owner's repo.
- Token read once at construction (BRAIN_GITEA_TOKEN), held in the struct,
  travels only in the Authorization header — never logged or in argv.
  Error messages carry status + truncated body, never the token
  (regression-tested). gitea.New returns nil when URL or token is unset,
  so missing config = tracker disabled via one nil check.
- Injected into the MCP server behind the capture.IssueTracker interface
  via WithIssueTracker (constructor injection, swappable/testable); main
  wires it from BRAIN_GITEA_URL (default https://git.d-ma.be) +
  BRAIN_GITEA_TOKEN. Consumed by the capture use-case in #53.

Tests use httptest transports: create (owner+auth header asserted),
comment, close with/without comment, error path that proves the token
never leaks into an error string.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 23:27:29 +02:00
mathiasandClaude Opus 4.8 4cfc98de56 refactor(capture): CloseIssue carries a closing comment (#52)
#52's IssueTracker spec is CloseIssue(repo, number, comment). Refine the
#51 port signature to match and have the service pass the ticket body as
the closing comment (empty ⇒ close only). Keeps the close-with-comment
flow first-class rather than forcing two separate ticket items.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 23:27:29 +02:00
mathiasandClaude Opus 4.8 0ac165cca3 feat(brainstore): shared BrainStore impl; re-point MCP handlers (#51)
Extracts the #45 write/update/get logic + the wiki upkeep that must
accompany a write (wing _index rebuild, cross-wing auto-tunnel, graph
re-index) into a single concrete brainstore.Store implementing
capture.BrainStore. The MCP brain_write/brain_update/brain_get handlers
are re-pointed at it, so there is one implementation, not two — the DRY
payoff #51 is named for. capture and MCP now share the exact same brain
write path and read-after-write contract.

The Server gains a *brainstore.Store, constructed in NewServer and given
the graph store in WithGraph. Embedding refresh stays out-of-band
(mtime-driven vectorstore.Sync), unchanged. Existing MCP brain_update/
brain_get/brain_write tests pass unmodified — behaviour and the
{id, path, content_hash} response contract are preserved.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 23:21:43 +02:00
mathiasandClaude Opus 4.8 43f92e3102 feat(capture): CaptureService use-case + ports + entities (#51)
The Clean-Architecture core of the capture capability (#49b). Pure
orchestration over ports — no HTTP, no live Gitea, no audit I/O — fully
unit-tested against fakes before any adapter exists.

- Ports: BrainStore (#45 write/update/get), IssueTracker, SummaryWriter,
  ClassificationPolicy (satisfied by #50's classification.Config),
  AuditSink. Entities: Insight, Ticket, Summary, CaptureContext,
  CaptureInput, CaptureReceipt.
- CaptureService.Capture: validate-before-write (fail-closed), resolve
  effective classification (stricter of declared vs target-derived;
  under-declaration logged as a security event), orchestrate insights
  (write/supersede) → tickets → summary best-effort, emit a request-level
  audit record of exactly what landed, return a partial-aware receipt.
- dry_run short-circuits after validation, writes nothing (not even audit).

Out of scope here, layered on later: the I1 origin sovereignty gate (#53,
needs the server-derived principal) and the classification-aware audit
degradation/refusal (#54). "Effective" is folded into the service as
classification.Stricter rather than a port method — the stricter-wins
rule is use-case policy.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 23:21:43 +02:00
mathiasandClaude Opus 4.8 98cfae595c feat(classification): data-sensitivity taxonomy + per-wing/repo tags (#50)
CI / Lint / Test / Vet (pull_request) Successful in 13s
CI / Mirror to GitHub (pull_request) Has been skipped
Implements the I1-prerequisite from #49/#50: the classification taxonomy
and the per-wing / per-repo tagging the capture server reads to derive a
target's sensitivity.

- Levels public < internal < confidential, ordered so "stricter wins"
  (spec §4.1 model C) is a plain max via Stricter().
- Tags read from an optional classification.yaml at the brain root
  (wings:/repos: maps). Absent file → defaults-only, not an error.
- Defaulting: client-* → confidential; hyperguild/homelab → internal;
  everything else → confidential. Fail-safe-to-strictest is the
  load-bearing property: a missing tag never silently downgrades.
- Config.Derive(Target) is the function the use-case calls; Wing/Repo
  are the per-kind helpers. ParseLevel rejects unknown tokens; a bad
  level in the config file is a hard load error.

Central classification.yaml (not _index.md frontmatter, not gitea repo
topics): classifying a repo needs no live Gitea client, so #50 has no
dependency on the tracker work (#52); it's auditable in one place; and
it avoids BuildWingIndex clobbering a wing's regenerated _index.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 22:42:05 +02:00
mathias 2a595b5a92 docs(capture): make Q4 audit-sink-down posture classification-aware (#49)
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 3s
Reconsidered Q4: instead of one global degrade-and-warn, the posture now
inherits from effective classification (Q1):
- confidential + audit-sink-down -> hard-refuse (no buffer; removes the
  buffer-integrity question for confidential data)
- internal/public + audit-sink-down -> degrade-and-warn + durable local
  buffer + ntfy + reconcile-on-recovery
- floor (all tiers): refuse if nothing can record the audit
Updated the I5 Gherkin scenarios + obligations row to match. Couples Q4
to the Q1 classification spine -> one coherent sensitivity model.
2026-06-22 20:20:40 +00:00
mathias b7a2cc5fdf docs(capture): resolve §4 open questions into binding decisions (#49)
CI / Lint / Test / Vet (push) Successful in 14s
CI / Mirror to GitHub (push) Successful in 4s
Q1 classification trust: model (C) — caller declares, server cross-checks
  target tag, stricter wins, mismatch logged. Needs a classification
  taxonomy + per-wing/repo tags (prerequisite, sub-task of #49).
Q2 harness origin: server-derived from authenticated principal;
  context.harness is descriptive-only, never a gate input.
Q3 central relay: ships in v1 (needed for claude.ai/Crush/Pi/LLM Council)
  + I2 security-baseline ledger entry is v1 work.
Q4 audit-sink-down: degrade-and-warn + durable local buffer + reconcile
  on recovery; refuse only if NOTHING can record the audit.

Updated the I1 sovereignty + I5 auditability Gherkin scenarios to match;
added classification-mismatch and origin-spoofing scenarios.
2026-06-22 20:17:07 +00:00
mathias db638cca11 docs(capture): add use-case + BDD spec for the capture capability (#49)
CI / Lint / Test / Vet (push) Successful in 14s
CI / Mirror to GitHub (push) Successful in 3s
Pre-build spec for the uniform cross-harness capture path. Grounds the
design in the homelab invariants (I1-I5, now canonical on infra main):
- I1 sovereignty gate: refuse confidential capture via us-nexus harness
- I2: distributed-library form (no high-degree observer node) is
  admissible; central relay needs a security-baseline ledger entry
- I5: capture is a privileged write path, must emit audit records
Gherkin scenarios cover the happy path, the sovereignty refusal,
supersession + staleness discipline (#45/#47), fail-closed validation,
best-effort partial-failure receipts, dry-run, and fidelity supersession.
Four open questions flagged for review (classification trust is the
highest-risk one).
2026-06-22 17:25:03 +00:00
mathias 38579598e0 feat(skills): add close-session workflow skill
CI / Lint / Test / Vet (push) Successful in 14s
CI / Mirror to GitHub (push) Successful in 4s
Disciplined end-of-session closeout for Claude.ai chats: harvest →
ground-truth gitea → confirm issue actions → commit session summary →
brain orientation note → safe-to-archive verdict.

Finalized against current infra (git.d-ma.be, koala:30401 LiteLLM,
Authentik) and the brain_update/brain_get verbs. Phase 5 uses
supersede-by-slug + brain_get read-after-write with the batch
no-semantic-query-after-supersede discipline. session_log/brain_tunnel
inlined (now callable from claude.ai). Summary frontmatter is the
reduced live-capture schema with fidelity:live-capture to distinguish
from batch-export summaries.
2026-06-22 15:57:02 +00:00
mathias d6fa92b176 Merge pull request 'chore(context): drop unsupported cursor + aider adapters' (#48) from chore/drop-cursor-aider-adapters into main
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 4s
2026-06-22 09:03:59 +00:00
mathias 9173f9058d chore(context): remove generated .aider.conventions.md
CI / Lint / Test / Vet (pull_request) Successful in 13s
CI / Mirror to GitHub (pull_request) Has been skipped
Aider is not a supported harness; the generator no longer emits this.
Note: generator no longer writes .aider.conf.yml either.
2026-06-22 09:01:39 +00:00
mathias 3e84a41fed chore(context): remove generated .cursorrules
Cursor is not a supported harness; the generator no longer emits this.
2026-06-22 09:01:29 +00:00
mathias 0e0571c7da chore(context): stop policing cursor/aider adapters in task check
Drop the context:sync:cursor task and remove .cursorrules +
.aider.conventions.md from the drift-guard file list in `check`, now
that the generator no longer emits them.
2026-06-22 09:01:19 +00:00
mathias 7a27cf71a2 chore(context): drop cursor + aider adapters from generator
Cursor and Aider are not supported harnesses. Remove generate_cursor
and generate_aider, their no-arg calls, and their case arms. The active
adapters are claude (CLAUDE.md), agents (AGENTS.md), and system-prompt
(.context/system-prompt.txt).
2026-06-22 09:00:55 +00:00
mathias 63df6d3283 Merge pull request 'feat: brain_update supersede verb + brain_get / brain_write read-after-write handle (#45)' (#46) from feat/brain-update-supersede into main
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 4s
2026-06-22 08:59:28 +00:00
mathiasandClaude Opus 4.8 f04b03e07e chore(context): re-sync derived adapters after root rule-0 update
CI / Lint / Test / Vet (pull_request) Successful in 13s
CI / Mirror to GitHub (pull_request) Has been skipped
context-sync regenerated the adapters from the updated root AGENT.md
(rule 0 pre-task ritual + TDD constraint). The committed adapters had
drifted; this is the documented `task check` remedy, not a content
change in this repo.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 08:25:16 +02:00
mathiasandClaude Opus 4.8 6c61f93146 feat(mcp): register brain_update + brain_get, extend brain_write handle
Wires the #45 verbs into the MCP surface (all three sites: tools()
descriptors, handleCall dispatch, package doc comment).

- brain_update: supersede-by-slug or full path; rebuilds wing _index and
  re-tunnels cross-wing matches against the new body (idempotent,
  best-effort), re-indexes the graph, returns {id, path, content_hash,
  superseded}.
- brain_get: fetch by id or path (both are the brain-relative handle);
  returns {id, path, content_hash, frontmatter, body}.
- brain_write: return contract extended from {path} to {id, path,
  content_hash} — path kept for backward compat — so the create path
  also yields a stable handle.

id == relPath; content_hash == sha256 of the file bytes. Tests cover the
supersede happy path, missing-target error + no-create, get by id/path,
write handle, and an end-to-end re-embed test that drives the real
vectorstore.Sync re-index after an update.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 08:25:10 +02:00
mathiasandClaude Opus 4.8 95a69fc2c1 feat(brain): add UpdateNote/ReadNote supersede primitives + frontmatter editor
Implements the api-layer half of #45. UpdateNote supersedes a note in
place (whole-note body replace, frontmatter re-stamp: updated_at,
supersedes=prior content hash, supersede_reason), preserving created_at,
wing, hall, and any custom fields. Never creates — a missing target is
an error so callers fall back to brain_write. ReadNote is the read-after-
write primitive (frontmatter + body + content_hash). ContentHash is the
sha256 handle that round-trips write/update → get.

Frontmatter is edited via a line-preserving ordered editor rather than a
yaml.v3 round-trip, which would reorder keys and strip comments — the
brain writes flat key:value frontmatter by hand.

Embeddings are not refreshed here: the rewritten file's mtime advances,
which the out-of-band vectorstore.Sync ticker uses to re-embed it — the
same mechanism brain_write relies on.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 08:25:00 +02:00
mathiasandClaude Opus 4.8 bb8bc0478c chore(cd): use git.d-ma.be for SSH alias + INFRA_REPO (post-rename consistency)
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 4s
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 00:19:42 +02:00
mathiasandClaude Opus 4.8 a961a3c064 fix(vectorstore): hard-split oversized heading-less chunks
A doc with no headings and no blank-line paragraphs (JSON-lines, e.g.
wiki/telos/decisions/human-intent-column.md) survived both chunk passes whole
and was sent to nomic-embed over its context window → 'input length exceeds
the context length' (400, the steady embed errors=1). Add a final hard-split
pass (line then UTF-8 rune boundaries) so no chunk exceeds maxBytes. TDD.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 00:19:42 +02:00
mathiasandClaude Opus 4.8 bec28f9014 fix(cd): point registry/patch/verify refs at git.d-ma.be after host rename
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 3s
The gitea.d-ma.be→git.d-ma.be rename moved the infra deployment manifests, but
cd.yml still sed-patched 'gitea.d-ma.be/mathias/ingestion:' — which no longer
matches, so the patch produced no change and 'git commit' failed under set -e,
wedging all deploys (image stuck at e8dbcf6). Repoint the registry image refs,
the infra-patch sed patterns, and the rollout-verify EXPECTED values to
git.d-ma.be (same registry backend). SSH alias + INFRA_REPO left as-is.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 23:22:07 +02:00
mathiasandClaude Opus 4.8 b62ac57382 fix(brain_answer): reranker is a filter, not a gate — fall back to BM25 when it keeps nothing
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 3s
The Qwen3-Reranker is a web-search cross-encoder. Against conversational /
personal-intent queries (e.g. 'what am I optimizing toward?') it scores every
candidate as 'no', so brain_answer collapsed to 'No relevant content found'
even though BM25 had retrieved on-topic notes (incl. the telos wing). Treat
the reranker as a filter: when it keeps zero results, fall back to the
BM25/vector ordering (capped to the no-reranker depth of 10). Refs brain#11.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 22:53:02 +02:00
mathiasandClaude Opus 4.8 aa918388b9 docs(brain): add two-column intent merge scaffold
CI / Lint / Test / Vet (push) Successful in 14s
CI / Mirror to GitHub (push) Successful in 4s
Drop-in unified findings doc for the brain-MCP intent study. Locks the
shared row schema + closed intent vocab both columns must conform to.
Agent column filled from agent-intent-column.jsonl (46 acts, 37%
mismatch); human column left as PENDING cells + <<SYNTH>> blocks so the
Claude.ai-history analysis merges in without re-deriving structure.

Pre-seeds the cross-consumer divergence questions: agent mismatch is
write-side-heavy (supersede + verify-landed); hypothesis is human
mismatch is read-side-heavy (semantic + answer) — interface may fail the
two consumers at opposite ends.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 20:45:24 +02:00
mathiasandClaude Opus 4.8 e8dbcf6eef docs(brain): add agent-consumer brain-MCP intent analysis
CI / Lint / Test / Vet (push) Successful in 14s
CI / Mirror to GitHub (push) Successful in 4s
Agent-consumer column of the two-part brain intent↔interface study.
Reconstructs the knowledge-act behind every brain MCP call in the
Claude Code agent transcripts on koala (the only corpus with brain
calls; brain/sessions and agentsquad eval logs carry none).

46 distinct knowledge-acts, 37% interface mismatch, 0% intent_unclear.
Headline gap: no update/supersede verb → agents blind re-write same
slug (5x); no read-after-write → lexical re-query of own note (4x);
lexical-only reads → semantic-as-keyword-stuffing chains (3x);
brain_answer hedged with parallel brain_query.

Canonical schema brain-intent-extraction.md absent on host; vocab
reconstructed, every row tagged schema_source=reconstructed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 19:44:59 +02:00
mathiasandClaude Opus 4.8 0eeb1df4a2 fix(claudewatcher): scrub bare 1Password service-account tokens (ops_)
CI / Lint / Test / Vet (push) Successful in 17s
CI / Mirror to GitHub (push) Successful in 4s
A ~/.zshrc read surfaced OP_SERVICE_ACCOUNT_TOKEN into a transcript. The
_TOKEN= form was already caught by homelab-env-token, but a bare ops_<b64>
value was not. Add an op-service-account rule (ordered early). Tests cover
both env-assigned and bare forms.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 00:28:47 +02:00
mathiasandClaude Opus 4.8 9febb1bba1 fix(claudewatcher): harden secret scrubber against \b evasion + add JWT
CI / Lint / Test / Vet (push) Successful in 17s
CI / Mirror to GitHub (push) Successful in 4s
A shell mangle that glued a key to a preceding word ('yes'+'sk-...') had
no word boundary, so the leading \b in the openai-sk rule failed to match
and a LiteLLM master key leaked past the scrubber into ingest (2026-06-11).

- openai-sk: drop leading \b, match sk- shape anywhere ({32,} floor keeps
  short task-/disk- words clean).
- add jwt rule for bare header.payload.sig tokens (no Bearer prefix).
- regression tests: the exact yessk- evasion, standalone sk-, bare JWT,
  plus clean-content guards for task-/disk-.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 08:55:33 +02:00
mathias 5dc247b994 fix(ci): quote "on" key so Gitea parses workflow triggers
CI / Lint / Test / Vet (push) Successful in 17s
CI / Mirror to GitHub (push) Successful in 4s
Bare on: parses as YAML boolean true (Norway problem); Gitea then ignores the triggers and silently skips jobs. Quoting forces the string key.
2026-06-03 08:38:46 +02:00
mathias 2125558196 docs: extend harness boundary decision to cover Crush as third harness
CI / Mirror to GitHub (push) Successful in 4s
CI / Lint / Test / Vet (push) Successful in 12s
2026-05-28 11:38:36 +00:00
mathias 2beaac2feb docs: add 2026-05-28 decisions — harness boundary + field benchmark definition
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 3s
2026-05-28 10:31:41 +00:00
mathias 525811bc1a docs: add hypothesis statement and harness boundary clarification
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 4s
2026-05-28 10:30:39 +00:00
139 changed files with 7949 additions and 3761 deletions
-315
View File
@@ -1,315 +0,0 @@
# Agent context — Mathias workspace
<!-- Canonical root context for all AI coding agents.
Lives at: ~/dev/.context/AGENT.md
Applies to every project under ~/dev/ unless overridden.
Run `task context:sync` from ~/dev/ to regenerate harness-specific files.
Project-level context in .context/PROJECT.md layers on top of this. -->
## Who I am
I'm Mathias, a digital product manager and technology consultant based in Sweden.
I build software, research emerging tech, and deliver consulting engagements
for clients under NDA. I work across AI/ML, financial automation, web applications,
and climate/sustainability tech.
## How I work with agents
- I think like a product manager — I care about *why* before *how*
- I want agents to be opinionated and push back, not just execute blindly
- I prefer concise responses; skip ceremony and get to the point
- When I say "build this", I mean production-quality with tests, not a demo
- Ask me before making irreversible changes or adding heavy dependencies
- I work with confidential client data — never send it to cloud APIs unless I explicitly say it's OK
## Behavior rules
These rules apply to every task across every project, regardless of harness.
1. **No assumptions.** Don't hide confusion — surface it. Surface tradeoffs explicitly.
Think before coding; if the problem is unclear, ask or state assumptions before acting.
2. **Minimum viable code.** Solve with the smallest change that works. Nothing
speculative, no "while we're here" cleanups, no premature abstractions. Simplicity first.
3. **Surgical changes.** Touch only what the task requires. Leave unrelated code,
files, and formatting alone. Diffs should be small and reviewable.
4. **Goal-driven execution.** Define clear success criteria up front for every task.
Loop — implement, verify, refine — until those criteria are met. Don't claim
completion without evidence (tests pass, command output, observed behavior).
5. **Trunk-Based Development — commit directly to main.** Every commit is one
logical change (one tool, one fix, one test) with passing tests. Main is always
deployable. Never create long-lived feature branches.
**Exception — parallel agents on same repo:** If another agent is known to be
actively working on the same repo simultaneously, create a short-lived branch
(`agent/<description>`), finish the task, and merge to main within the same
session. Do not leave agent branches open between sessions.
**Exception — external contributor or client four-eyes requirement:** Use
PR flow only when a human reviewer outside the project is required. Document
the reason in PROJECT.md.
## Default stack
| Layer | Default | Fallback | Last resort |
|-------|---------|----------|-------------|
| Language | Go | Python | TypeScript, Java, C |
| UI | HTMX + Templ | Server-rendered HTML | React (only if SPA is justified) |
| Build | Task (taskfile.dev) | Make | — |
| Containers | Docker Compose (dev), k3s (prod) | — | — |
| DB | PostgreSQL + sqlc | SQLite | — |
| Search | pgvector (vector), BM25 | Qdrant (when >1M vectors or hybrid retrieval) | — |
| Logging | slog (structured) | — | — |
| Testing | Table-driven, testify | — | — |
| Agents (Go) | google.golang.org/adk + pkg/litellm adapter | — | — |
Exploratory: Rust, Zig — I'll tell you when I want these.
## Code conventions
- **Go style**: golines, gofumpt, golangci-lint
- **Errors**: `fmt.Errorf("operation: %w", err)` — never naked, never log-and-return
- **Naming**: stdlib conventions, no stuttering
- **Architecture**: prefer stdlib over frameworks, constructor injection, env-var config parsed into typed structs
- **Git**: conventional commits (`feat:`, `fix:`, `chore:`), commit directly to main,
one logical change per commit, CI is the quality gate
- **Never**: long-lived feature branches, PRs for solo work, direct push without
passing `task check` locally first
- **Security**: no secrets in code, govulncheck before adding deps, SOPS for encrypted config
- **Dependencies**: prefer stdlib. testify, slog, templ, sqlc, google.golang.org/adk (agent projects only) are pre-approved; anything else needs justification in the commit message
## Infrastructure
Three machines on Tailscale:
| Machine | Role | Key specs |
|---------|------|-----------|
| koala | GPU inference, heavy compute | RTX 5070, runs k3s + llama-swap + shared postgres18/pgvector |
| iguana | Services, builds | M2 Ultra Mac |
| flamingo | Daily driver, edge | Mac mini, ~/dev is here |
- **Model routing**: LiteLLM in front of llama-swap (local) + cloud APIs (when permitted)
- **Orchestration**: k3s cluster across all three machines
- **Networking**: Tailscale mesh
## Project landscape
All development repos live at `~/dev/` (softlink from `~/Documents/local-dev/`).
Organized in thematic folders:
| Folder | Focus | Count |
|--------|-------|-------|
| `GO/` | Go web frameworks, API integrations, learning projects | ~10 |
| `AI/` | ML research, AI frameworks (FinRL, DSPy, crawl4ai) | ~6 |
| `AGENTS/` | Autonomous agents, coding agents, MCP servers, infra | ~15 |
| `QKX/` | Invoice processing, financial automation, payment systems | ~13 |
| `XT/` | Climate data, sustainability (Klimatkollen, Garbo) | ~2 |
See `~/dev/PROJECT_SUMMARY.md` for detailed descriptions of each project.
### Key active projects
- **super-koala** (`AGENTS/`) — multi-component agent stack with LangGraph, DSPy, MCP
- **azure-tiger** (`QKX/`) — invoice extraction → ISO 20022 payment instructions
- **gocrwl** (`AGENTS/`) — Go web crawler with containerized deployment
- **koala-ai-stack** (`AGENTS/`) — local AI server infrastructure management
- **klimatkollen** (`XT/`) — Swedish municipal climate data platform
## Knowledge base — actively use it
A persistent brain (BM25 search + LLM-synthesised Q&A) survives across sessions,
hosts, and harnesses. It holds 100+ hard-won entries: infra incident postmortems,
Go pitfalls, framework gotchas, design principles, ADRs. **It is not optional
reference material — query it actively, not just when explicitly told.**
### When to query (treat as a reflex)
- **Before** starting a non-trivial task — search for prior art with the symptom
AND the system component ("how did we solve X in Y?"). 5 seconds beats 5 hours.
- **When debugging** — search for the error string, the stack frame, the affected
service. Past you may have already paid this tax.
- **Before adopting** a pattern, library, framework, or model name — check if it
was tried and rejected, or what the integration footguns are.
- **When making architectural decisions** — search for the domain + "ADR" or
"decision" to find prior reasoning before re-deriving it.
- **When a recommendation feels novel** — challenge yourself: "has this been
documented?" The brain often has it.
### When to write
After you discover something that **future-you would forget** and that **isn't
recoverable from the code, git log, or PR description alone**:
- Bugs whose root cause is non-obvious and generalisable beyond this project.
- Framework / library / model-name quirks that bit you and would bite anyone.
- Design principles validated under fire (e.g. "every `_get` needs a `_list`").
- Postmortems for incidents: what broke, why, how diagnosed, what to do next time.
DON'T write project status, sprint progress, PR summaries, or "what I did this
session" — those rot fast and the originals are in git/gitea anyway. Brain
entries that age well are about *why*, *how to avoid*, and *what to do when*.
### How to access (per harness)
| Harness | Query | Write |
|---------|-------|-------|
| **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool |
| **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same |
| **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` |
| **Browser / human inspection** | `https://gitea.d-ma.be/mathias/hyperguild``knowledge/` and `wiki/` markdown files |
- **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`.
- **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as
fallback. Both are configurable in the `supervisor/ingestion-deployment.yaml`
on the koala k3s cluster; don't hardcode local-only model names into the
berget URL (see knowledge entry on namespace mismatches).
### Quick reflex checks
If you find yourself about to say any of these out loud, you owe yourself a brain query first:
- "I think the issue might be..."
- "Let me try X and see..."
- "I'll just write a script to..."
- "This is probably a new bug..."
- "Has anyone done this before?" — *yes, probably, go check.*
## Client work rules
When working on a project tagged with a client name:
1. Never send code, data, or context to cloud APIs — use local models only
2. Never reference other client projects or their data
3. Keep all artifacts within the client's git org / directory
4. Treat everything as confidential unless told otherwise
## Harness-agnostic principles
This context is designed to work with any AI coding tool:
- Claude Code, Cursor, Aider, Open WebUI, Charmbracelet Mods/Crush
- Pi Coding Agent, Mistral Vibe, Antigravity
- Any tool that accepts a system prompt or reads a markdown context file
The canonical source is always `.context/AGENT.md` (root) and `.context/PROJECT.md` (per-project).
Derived files are committed (see *How context propagates* below) so a `git pull` on any host yields full agent context with no setup.
## How context propagates
Canonical sources of truth:
- Universal: `~/dev/.context/AGENT.md` (this file)
- Project: `<repo>/.context/PROJECT.md` (per-repo)
Derived files (committed, regenerated by `task context:sync`):
- `CLAUDE.md`, `AGENTS.md`, `.cursorrules`, `.aider.conventions.md`,
`.context/system-prompt.txt`
Workflow:
1. Edit a canonical file. Run `task context:sync`. Commit canonical and
derived together. Push.
2. On any other host, `git pull` brings both. Claude Code (tree-walking)
uses `CLAUDE.md`; Crush / Pi / Antigravity (cwd-only) use `AGENTS.md`;
Cursor uses `.cursorrules`; Aider uses `.aider.conventions.md`.
3. `task check` runs `context:sync` then asserts `git status --porcelain`
is empty over the derived files (catches both modified-tracked drift
and missing-untracked adapters). A drift fails the check with a
message telling you to stage the regenerated files.
Behavior rules in this file and per-project rules in `PROJECT.md` apply
unconditionally on every host, every harness.
## Engineering Skills
Shared engineering skills are available in `~/dev/.skills/`. Load on demand via the index.
See `~/dev/.skills/SKILLS_INDEX.md` for the full list with descriptions and "use when" triggers.
Key skills:
- **TDD**: always write tests first — load `tdd` skill
- **Code Review**: load `code-review` skill before any review
- **SOLID/Clean Code**: load `solid` or `clean-code` skill for design work
- **Problem first**: load `problem-analysis` skill before coding non-trivial features
---
# Project context
<!-- Canonical project context. Edit this, run `task context:sync`.
Root agent context from ~/dev/.context/AGENT.md is automatically
prepended for harnesses that don't walk the directory tree. -->
## Identity
- **Name**: supervisor
- **Owner**: Mathias
- **Client**: personal
- **Repo**:
- **Status**: active
## Stack
- **Primary language**: Go
- **UI layer**: HTMX + Templ (when applicable)
- **Fallback languages**: Python, TypeScript (justify in PR if used)
- **Build**: Task (taskfile.dev), not Make
- **Containers**: Docker (compose for dev, k3s for deploy)
- **Target infra**: koala (GPU workloads), iguana (services), flamingo (edge)
## Conventions
### Code style
- Go: follow `golines`, `gofumpt`, `golangci-lint` with project config
- Tests: table-driven, in `_test.go` next to source, `testify` for assertions
- Errors: wrap with `fmt.Errorf("operation: %w", err)`, no naked returns
- Naming: stdlib conventions, no stuttering (`http.Client` not `http.HTTPClient`)
### Architecture preferences
- Prefer standard library over frameworks (net/http over gin/echo)
- Dependency injection via constructor functions, not containers
- Configuration via environment variables, parsed at startup into a typed struct
- Structured logging via `slog`
### Git
- Conventional commits: `feat:`, `fix:`, `chore:`, `docs:`, `refactor:`
- Branch naming: `feat/short-description`, `fix/short-description`
- PRs: one concern per PR, description explains *why* not *what*
### Security
- No secrets in code, ever — use env vars or SOPS-encrypted files
- Client data never leaves local network unless explicitly cleared
- Dependencies: audit with `govulncheck` before adding
## MCP endpoints
Two MCP servers are live, both reachable over Tailscale and via HTTPS domain:
- **`brain`** at `https://brain-mcp.d-ma.be/mcp` (NodePort `koala:30330`) —
`brain_query`, `brain_write`, `brain_ingest`, `brain_ingest_raw`,
`brain_answer`, `brain_classify`, `session_log`. Hosted by the ingestion
service. Auth: Dex JWT (claude.ai OAuth) or static `BRAIN_MCP_TOKEN`.
- **`routing`** at `http://koala:30310/mcp` — Mode 2 routing pod. Advertises
`review`, `debug`, `retrospective`, `trainer`; per-call routes to local model
or Claude based on brain `/pass-rate`. Bearer auth via `ROUTING_MCP_TOKEN`
(opt-in). Only `mode client-local` registers this endpoint.
The supervisor MCP (`koala:30320`) was retired in Plan 7 (2026-05-12). Its
skill workers (`tdd`, `spec`) are now SKILL.md files; routed skills moved to
the routing pod; brain tools moved to the brain MCP.
The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`,
`/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300
for shell scripts and non-MCP clients.
`brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai
gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title,
and tags. Both require `BRAIN_LLM_PRIMARY_URL` to be set in the ingestion pod.
## Agent instructions
When acting as a coding agent on this project:
1. Read this file and all `SKILL.md` files in `.skills/` before starting work
2. Run `task check` before committing (lint + test + vet)
3. If unsure about a convention, check `DECISIONS.md` or ask
4. Never modify files outside the project root without explicit permission
5. When adding a dependency, explain why in the commit message
6. For client projects: never send code or context to cloud APIs — use local models via LiteLLM
+54 -8
View File
@@ -32,6 +32,14 @@ and climate/sustainability tech.
These rules apply to every task across every project, regardless of harness. These rules apply to every task across every project, regardless of harness.
0. **Pre-task ritual — before ANY implementation (non-negotiable).** Run this before writing a single line:
- **Query the brain** (`brain_query`) for the domain + symptom. If the result changes your approach, surface it before acting. 5 seconds beats 5 hours.
- **Load the relevant skill** — see trigger table in *Engineering Skills* below.
- **Write the failing test first.** Name the test before the function. If the target is untestable (e.g. `main()` wiring), extract the logic into a testable function first. No implementation without a red test.
- **State the observable success criterion** — what specific behavior, output, or passing test proves this is done?
**TDD is non-negotiable.** "Tests pass" is not proof of correctness — only proof the tests ran. Write tests that would catch the bug before writing code that fixes it.
1. **No assumptions.** Don't hide confusion — surface it. Surface tradeoffs explicitly. 1. **No assumptions.** Don't hide confusion — surface it. Surface tradeoffs explicitly.
Think before coding; if the problem is unclear, ask or state assumptions before acting. Think before coding; if the problem is unclear, ask or state assumptions before acting.
2. **Minimum viable code.** Solve with the smallest change that works. Nothing 2. **Minimum viable code.** Solve with the smallest change that works. Nothing
@@ -54,6 +62,22 @@ These rules apply to every task across every project, regardless of harness.
PR flow only when a human reviewer outside the project is required. Document PR flow only when a human reviewer outside the project is required. Document
the reason in PROJECT.md. the reason in PROJECT.md.
6. **Close the loop — every substantive task ends with the same ritual.** Shipping
the code is not the end of the task; capturing it is. Run this unprompted:
- **Tag + bump SemVer** on the change (annotated tag; minor for a feature or
new/changed ADR, patch for a fix; docs in the same commit). Check the repo's
actual last tag — stated versions in docs drift stale.
- **Push** main and the tag (CI is the gate).
- **Persist generalizable learnings to the brain** (`brain_write`, wing/hall) —
the reusable patterns and the footguns that would bite anyone again, never
project status. See *Knowledge base — when to write* below.
- **File discovered-but-deferred work as tracker issues** on the project's own
repo — token-budget gaps, recorded ADR limitations, v2 follow-ups. Don't let
"out of scope, recorded" rot in a commit message; make it a ticket with a
source pointer.
- Surface the brain entries and issue numbers in the closing summary so the
trail is auditable.
## Default stack ## Default stack
| Layer | Default | Fallback | Last resort | | Layer | Default | Fallback | Last resort |
@@ -83,6 +107,26 @@ Exploratory: Rust, Zig — I'll tell you when I want these.
- **Security**: no secrets in code, govulncheck before adding deps, SOPS for encrypted config - **Security**: no secrets in code, govulncheck before adding deps, SOPS for encrypted config
- **Dependencies**: prefer stdlib. testify, slog, templ, sqlc, google.golang.org/adk (agent projects only) are pre-approved; anything else needs justification in the commit message - **Dependencies**: prefer stdlib. testify, slog, templ, sqlc, google.golang.org/adk (agent projects only) are pre-approved; anything else needs justification in the commit message
## Secret handling (every harness, every command)
Tool output is persisted: terminal → `~/.claude/projects` transcripts →
claudewatcher → brain/wiki → gitea history. A secret printed once is
searchable forever, and clearing it means rotating the key. So:
1. **Never print, echo, log, or transform a secret to inspect it.** No
`base64`/`xxd`/`cat` of a key, and never pipe a secret through a transform
to defeat `op run`'s output masking (it masks raw values; base64 hides them
from the mask — that exact trick leaked a key on 2026-06-11).
2. **Secrets stay in the subprocess.** Reference them only as env vars consumed
*inside* `op run --env-file ~/.op-env -- <cmd>`. Never place a literal secret
in a command's argv (it lands in the tool call and the transcript).
3. **Existence check without revealing the value:** `[ -n "$X" ] && echo set` —
never `${X:-...}` (returns the value when set) and never echo a substring of it.
4. **Cross-host secrets:** run the secret-consuming command on the host that has
the secret; do not forward a raw key over ssh argv/stdout.
5. If a secret does leak into output, say so immediately and flag it for rotation —
don't bury it.
## Infrastructure ## Infrastructure
Three machines on Tailscale: Three machines on Tailscale:
@@ -162,7 +206,7 @@ entries that age well are about *why*, *how to avoid*, and *what to do when*.
| **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool | | **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool |
| **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same | | **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same |
| **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` | | **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` |
| **Browser / human inspection** | `https://gitea.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files | | **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files |
- **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`. - **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`.
- **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as - **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as
@@ -224,15 +268,17 @@ unconditionally on every host, every harness.
## Engineering Skills ## Engineering Skills
Shared engineering skills are available in `~/dev/.skills/`. Load on demand via the index. Shared engineering skills live in the **`mathias/skills`** repo (`git.d-ma.be/mathias/skills`). Clone it to `~/dev/skills/` and run `SKILLS_CHECKOUT_DIR="$PWD" bash install.sh` there to wire every skill into your harnesses (Claude Code, Crush, Antigravity, Mistral Vibe) as native, on-demand skills. (Use `install.sh`, not `task install` — the latter is currently broken, skills#7.) Load at task start — not "on demand" but on schedule, before writing code. Browse `~/dev/skills/SKILLS_INDEX.md` for the full list.
See `~/dev/.skills/SKILLS_INDEX.md` for the full list with descriptions and "use when" triggers. **Skill trigger table — load before starting, not after getting stuck:**
Key skills: | Task type | Load |
- **TDD**: always write tests first — load `tdd` skill |-----------|------|
- **Code Review**: load `code-review` skill before any review | Any feature or bug fix | `tdd` |
- **SOLID/Clean Code**: load `solid` or `clean-code` skill for design work | Refactor or design | `clean-code` or `solid` |
- **Problem first**: load `problem-analysis` skill before coding non-trivial features | Debug | `problem-analysis` |
| Review code or PRs | `code-review` |
| Frame a problem before coding | `problem-analysis` |
--- ---
-318
View File
@@ -1,318 +0,0 @@
# Cursor rules — auto-generated
# Do not edit. Run: task context:sync
# Agent context — Mathias workspace
<!-- Canonical root context for all AI coding agents.
Lives at: ~/dev/.context/AGENT.md
Applies to every project under ~/dev/ unless overridden.
Run `task context:sync` from ~/dev/ to regenerate harness-specific files.
Project-level context in .context/PROJECT.md layers on top of this. -->
## Who I am
I'm Mathias, a digital product manager and technology consultant based in Sweden.
I build software, research emerging tech, and deliver consulting engagements
for clients under NDA. I work across AI/ML, financial automation, web applications,
and climate/sustainability tech.
## How I work with agents
- I think like a product manager — I care about *why* before *how*
- I want agents to be opinionated and push back, not just execute blindly
- I prefer concise responses; skip ceremony and get to the point
- When I say "build this", I mean production-quality with tests, not a demo
- Ask me before making irreversible changes or adding heavy dependencies
- I work with confidential client data — never send it to cloud APIs unless I explicitly say it's OK
## Behavior rules
These rules apply to every task across every project, regardless of harness.
1. **No assumptions.** Don't hide confusion — surface it. Surface tradeoffs explicitly.
Think before coding; if the problem is unclear, ask or state assumptions before acting.
2. **Minimum viable code.** Solve with the smallest change that works. Nothing
speculative, no "while we're here" cleanups, no premature abstractions. Simplicity first.
3. **Surgical changes.** Touch only what the task requires. Leave unrelated code,
files, and formatting alone. Diffs should be small and reviewable.
4. **Goal-driven execution.** Define clear success criteria up front for every task.
Loop — implement, verify, refine — until those criteria are met. Don't claim
completion without evidence (tests pass, command output, observed behavior).
5. **Trunk-Based Development — commit directly to main.** Every commit is one
logical change (one tool, one fix, one test) with passing tests. Main is always
deployable. Never create long-lived feature branches.
**Exception — parallel agents on same repo:** If another agent is known to be
actively working on the same repo simultaneously, create a short-lived branch
(`agent/<description>`), finish the task, and merge to main within the same
session. Do not leave agent branches open between sessions.
**Exception — external contributor or client four-eyes requirement:** Use
PR flow only when a human reviewer outside the project is required. Document
the reason in PROJECT.md.
## Default stack
| Layer | Default | Fallback | Last resort |
|-------|---------|----------|-------------|
| Language | Go | Python | TypeScript, Java, C |
| UI | HTMX + Templ | Server-rendered HTML | React (only if SPA is justified) |
| Build | Task (taskfile.dev) | Make | — |
| Containers | Docker Compose (dev), k3s (prod) | — | — |
| DB | PostgreSQL + sqlc | SQLite | — |
| Search | pgvector (vector), BM25 | Qdrant (when >1M vectors or hybrid retrieval) | — |
| Logging | slog (structured) | — | — |
| Testing | Table-driven, testify | — | — |
| Agents (Go) | google.golang.org/adk + pkg/litellm adapter | — | — |
Exploratory: Rust, Zig — I'll tell you when I want these.
## Code conventions
- **Go style**: golines, gofumpt, golangci-lint
- **Errors**: `fmt.Errorf("operation: %w", err)` — never naked, never log-and-return
- **Naming**: stdlib conventions, no stuttering
- **Architecture**: prefer stdlib over frameworks, constructor injection, env-var config parsed into typed structs
- **Git**: conventional commits (`feat:`, `fix:`, `chore:`), commit directly to main,
one logical change per commit, CI is the quality gate
- **Never**: long-lived feature branches, PRs for solo work, direct push without
passing `task check` locally first
- **Security**: no secrets in code, govulncheck before adding deps, SOPS for encrypted config
- **Dependencies**: prefer stdlib. testify, slog, templ, sqlc, google.golang.org/adk (agent projects only) are pre-approved; anything else needs justification in the commit message
## Infrastructure
Three machines on Tailscale:
| Machine | Role | Key specs |
|---------|------|-----------|
| koala | GPU inference, heavy compute | RTX 5070, runs k3s + llama-swap + shared postgres18/pgvector |
| iguana | Services, builds | M2 Ultra Mac |
| flamingo | Daily driver, edge | Mac mini, ~/dev is here |
- **Model routing**: LiteLLM in front of llama-swap (local) + cloud APIs (when permitted)
- **Orchestration**: k3s cluster across all three machines
- **Networking**: Tailscale mesh
## Project landscape
All development repos live at `~/dev/` (softlink from `~/Documents/local-dev/`).
Organized in thematic folders:
| Folder | Focus | Count |
|--------|-------|-------|
| `GO/` | Go web frameworks, API integrations, learning projects | ~10 |
| `AI/` | ML research, AI frameworks (FinRL, DSPy, crawl4ai) | ~6 |
| `AGENTS/` | Autonomous agents, coding agents, MCP servers, infra | ~15 |
| `QKX/` | Invoice processing, financial automation, payment systems | ~13 |
| `XT/` | Climate data, sustainability (Klimatkollen, Garbo) | ~2 |
See `~/dev/PROJECT_SUMMARY.md` for detailed descriptions of each project.
### Key active projects
- **super-koala** (`AGENTS/`) — multi-component agent stack with LangGraph, DSPy, MCP
- **azure-tiger** (`QKX/`) — invoice extraction → ISO 20022 payment instructions
- **gocrwl** (`AGENTS/`) — Go web crawler with containerized deployment
- **koala-ai-stack** (`AGENTS/`) — local AI server infrastructure management
- **klimatkollen** (`XT/`) — Swedish municipal climate data platform
## Knowledge base — actively use it
A persistent brain (BM25 search + LLM-synthesised Q&A) survives across sessions,
hosts, and harnesses. It holds 100+ hard-won entries: infra incident postmortems,
Go pitfalls, framework gotchas, design principles, ADRs. **It is not optional
reference material — query it actively, not just when explicitly told.**
### When to query (treat as a reflex)
- **Before** starting a non-trivial task — search for prior art with the symptom
AND the system component ("how did we solve X in Y?"). 5 seconds beats 5 hours.
- **When debugging** — search for the error string, the stack frame, the affected
service. Past you may have already paid this tax.
- **Before adopting** a pattern, library, framework, or model name — check if it
was tried and rejected, or what the integration footguns are.
- **When making architectural decisions** — search for the domain + "ADR" or
"decision" to find prior reasoning before re-deriving it.
- **When a recommendation feels novel** — challenge yourself: "has this been
documented?" The brain often has it.
### When to write
After you discover something that **future-you would forget** and that **isn't
recoverable from the code, git log, or PR description alone**:
- Bugs whose root cause is non-obvious and generalisable beyond this project.
- Framework / library / model-name quirks that bit you and would bite anyone.
- Design principles validated under fire (e.g. "every `_get` needs a `_list`").
- Postmortems for incidents: what broke, why, how diagnosed, what to do next time.
DON'T write project status, sprint progress, PR summaries, or "what I did this
session" — those rot fast and the originals are in git/gitea anyway. Brain
entries that age well are about *why*, *how to avoid*, and *what to do when*.
### How to access (per harness)
| Harness | Query | Write |
|---------|-------|-------|
| **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool |
| **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same |
| **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` |
| **Browser / human inspection** | `https://gitea.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files |
- **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`.
- **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as
fallback. Both are configurable in the `supervisor/ingestion-deployment.yaml`
on the koala k3s cluster; don't hardcode local-only model names into the
berget URL (see knowledge entry on namespace mismatches).
### Quick reflex checks
If you find yourself about to say any of these out loud, you owe yourself a brain query first:
- "I think the issue might be..."
- "Let me try X and see..."
- "I'll just write a script to..."
- "This is probably a new bug..."
- "Has anyone done this before?" — *yes, probably, go check.*
## Client work rules
When working on a project tagged with a client name:
1. Never send code, data, or context to cloud APIs — use local models only
2. Never reference other client projects or their data
3. Keep all artifacts within the client's git org / directory
4. Treat everything as confidential unless told otherwise
## Harness-agnostic principles
This context is designed to work with any AI coding tool:
- Claude Code, Cursor, Aider, Open WebUI, Charmbracelet Mods/Crush
- Pi Coding Agent, Mistral Vibe, Antigravity
- Any tool that accepts a system prompt or reads a markdown context file
The canonical source is always `.context/AGENT.md` (root) and `.context/PROJECT.md` (per-project).
Derived files are committed (see *How context propagates* below) so a `git pull` on any host yields full agent context with no setup.
## How context propagates
Canonical sources of truth:
- Universal: `~/dev/.context/AGENT.md` (this file)
- Project: `<repo>/.context/PROJECT.md` (per-repo)
Derived files (committed, regenerated by `task context:sync`):
- `CLAUDE.md`, `AGENTS.md`, `.cursorrules`, `.aider.conventions.md`,
`.context/system-prompt.txt`
Workflow:
1. Edit a canonical file. Run `task context:sync`. Commit canonical and
derived together. Push.
2. On any other host, `git pull` brings both. Claude Code (tree-walking)
uses `CLAUDE.md`; Crush / Pi / Antigravity (cwd-only) use `AGENTS.md`;
Cursor uses `.cursorrules`; Aider uses `.aider.conventions.md`.
3. `task check` runs `context:sync` then asserts `git status --porcelain`
is empty over the derived files (catches both modified-tracked drift
and missing-untracked adapters). A drift fails the check with a
message telling you to stage the regenerated files.
Behavior rules in this file and per-project rules in `PROJECT.md` apply
unconditionally on every host, every harness.
## Engineering Skills
Shared engineering skills are available in `~/dev/.skills/`. Load on demand via the index.
See `~/dev/.skills/SKILLS_INDEX.md` for the full list with descriptions and "use when" triggers.
Key skills:
- **TDD**: always write tests first — load `tdd` skill
- **Code Review**: load `code-review` skill before any review
- **SOLID/Clean Code**: load `solid` or `clean-code` skill for design work
- **Problem first**: load `problem-analysis` skill before coding non-trivial features
---
# Project context
<!-- Canonical project context. Edit this, run `task context:sync`.
Root agent context from ~/dev/.context/AGENT.md is automatically
prepended for harnesses that don't walk the directory tree. -->
## Identity
- **Name**: supervisor
- **Owner**: Mathias
- **Client**: personal
- **Repo**:
- **Status**: active
## Stack
- **Primary language**: Go
- **UI layer**: HTMX + Templ (when applicable)
- **Fallback languages**: Python, TypeScript (justify in PR if used)
- **Build**: Task (taskfile.dev), not Make
- **Containers**: Docker (compose for dev, k3s for deploy)
- **Target infra**: koala (GPU workloads), iguana (services), flamingo (edge)
## Conventions
### Code style
- Go: follow `golines`, `gofumpt`, `golangci-lint` with project config
- Tests: table-driven, in `_test.go` next to source, `testify` for assertions
- Errors: wrap with `fmt.Errorf("operation: %w", err)`, no naked returns
- Naming: stdlib conventions, no stuttering (`http.Client` not `http.HTTPClient`)
### Architecture preferences
- Prefer standard library over frameworks (net/http over gin/echo)
- Dependency injection via constructor functions, not containers
- Configuration via environment variables, parsed at startup into a typed struct
- Structured logging via `slog`
### Git
- Conventional commits: `feat:`, `fix:`, `chore:`, `docs:`, `refactor:`
- Branch naming: `feat/short-description`, `fix/short-description`
- PRs: one concern per PR, description explains *why* not *what*
### Security
- No secrets in code, ever — use env vars or SOPS-encrypted files
- Client data never leaves local network unless explicitly cleared
- Dependencies: audit with `govulncheck` before adding
## MCP endpoints
Two MCP servers are live, both reachable over Tailscale and via HTTPS domain:
- **`brain`** at `https://brain-mcp.d-ma.be/mcp` (NodePort `koala:30330`) —
`brain_query`, `brain_write`, `brain_ingest`, `brain_ingest_raw`,
`brain_answer`, `brain_classify`, `session_log`. Hosted by the ingestion
service. Auth: Dex JWT (claude.ai OAuth) or static `BRAIN_MCP_TOKEN`.
- **`routing`** at `http://koala:30310/mcp` — Mode 2 routing pod. Advertises
`review`, `debug`, `retrospective`, `trainer`; per-call routes to local model
or Claude based on brain `/pass-rate`. Bearer auth via `ROUTING_MCP_TOKEN`
(opt-in). Only `mode client-local` registers this endpoint.
The supervisor MCP (`koala:30320`) was retired in Plan 7 (2026-05-12). Its
skill workers (`tdd`, `spec`) are now SKILL.md files; routed skills moved to
the routing pod; brain tools moved to the brain MCP.
The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`,
`/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300
for shell scripts and non-MCP clients.
`brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai
gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title,
and tags. Both require `BRAIN_LLM_PRIMARY_URL` to be set in the ingestion pod.
## Agent instructions
When acting as a coding agent on this project:
1. Read this file and all `SKILL.md` files in `.skills/` before starting work
2. Run `task check` before committing (lint + test + vet)
3. If unsure about a convention, check `DECISIONS.md` or ask
4. Never modify files outside the project root without explicit permission
5. When adding a dependency, explain why in the commit message
6. For client projects: never send code or context to cloud APIs — use local models via LiteLLM
+9 -68
View File
@@ -1,6 +1,6 @@
name: cd name: cd
on: "on":
workflow_run: workflow_run:
workflows: ["CI"] workflows: ["CI"]
types: [completed] types: [completed]
@@ -13,9 +13,8 @@ jobs:
if: ${{ github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.event == 'push' }} if: ${{ github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.event == 'push' }}
environment: staging environment: staging
env: env:
INGESTION_IMAGE: gitea.d-ma.be/mathias/ingestion INGESTION_IMAGE: git.d-ma.be/mathias/ingestion
ROUTING_IMAGE: gitea.d-ma.be/mathias/routing INFRA_REPO: git@git.d-ma.be:mathias/infra.git
INFRA_REPO: git@gitea.d-ma.be:mathias/infra.git
BUILDKIT_HOST: unix:///run/buildkit/buildkitd.sock BUILDKIT_HOST: unix:///run/buildkit/buildkitd.sock
steps: steps:
- name: Checkout - name: Checkout
@@ -41,28 +40,6 @@ jobs:
echo "Built and pushed ${INGESTION_IMAGE}:${IMAGE_TAG}" echo "Built and pushed ${INGESTION_IMAGE}:${IMAGE_TAG}"
- name: Build and push routing image
run: |
set -e
trap 'rm -f /tmp/routing-image.tar' EXIT
IMAGE_TAG="${{ github.sha }}"
echo "Building ${ROUTING_IMAGE}:${IMAGE_TAG}"
buildctl --addr "${BUILDKIT_HOST}" build \
--frontend dockerfile.v0 \
--local context=. \
--local dockerfile=. \
--opt filename=Dockerfile.routing \
--opt build-arg:VERSION="${IMAGE_TAG}" \
--output type=oci,dest=/tmp/routing-image.tar
skopeo copy \
oci-archive:/tmp/routing-image.tar \
docker://${ROUTING_IMAGE}:${IMAGE_TAG} \
--dest-creds "${{ secrets.REGISTRY_CREDS }}"
echo "Built and pushed ${ROUTING_IMAGE}:${IMAGE_TAG}"
- name: Update infra repo - name: Update infra repo
run: | run: |
set -e set -e
@@ -71,28 +48,24 @@ jobs:
mkdir -p ~/.ssh mkdir -p ~/.ssh
echo "${{ secrets.INFRA_DEPLOY_KEY }}" > ~/.ssh/infra_deploy_key echo "${{ secrets.INFRA_DEPLOY_KEY }}" > ~/.ssh/infra_deploy_key
chmod 600 ~/.ssh/infra_deploy_key chmod 600 ~/.ssh/infra_deploy_key
printf 'Host gitea.d-ma.be\n HostName 127.0.0.1\n Port 30022\n StrictHostKeyChecking no\n' >> ~/.ssh/config printf 'Host git.d-ma.be\n HostName 127.0.0.1\n Port 30022\n StrictHostKeyChecking no\n' >> ~/.ssh/config
GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes" \ GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes" \
git clone "${INFRA_REPO}" /tmp/infra-update git clone "${INFRA_REPO}" /tmp/infra-update
cd /tmp/infra-update cd /tmp/infra-update
sed -i "s|gitea.d-ma.be/mathias/ingestion:.*|gitea.d-ma.be/mathias/ingestion:${IMAGE_TAG}|" \ sed -i "s|git.d-ma.be/mathias/ingestion:.*|git.d-ma.be/mathias/ingestion:${IMAGE_TAG}|" \
"k3s/apps/supervisor/ingestion-deployment.yaml" "k3s/apps/supervisor/ingestion-deployment.yaml"
sed -i "s|gitea.d-ma.be/mathias/routing:.*|gitea.d-ma.be/mathias/routing:${IMAGE_TAG}|" \
"k3s/apps/routing/deployment.yaml"
git config user.email "cd-bot@d-ma.be" git config user.email "cd-bot@d-ma.be"
git config user.name "CD Bot" git config user.name "CD Bot"
git add "k3s/apps/supervisor/ingestion-deployment.yaml" \ git add "k3s/apps/supervisor/ingestion-deployment.yaml"
"k3s/apps/routing/deployment.yaml" git commit -m "chore(deploy): ingestion → ${IMAGE_TAG}"
git commit -m "chore(deploy): ingestion+routing → ${IMAGE_TAG}"
GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes" \ GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes" \
git push git push
echo "Infra repo updated: ingestion+routing → ${IMAGE_TAG}" echo "Infra repo updated: ingestion → ${IMAGE_TAG}"
- name: Trigger Flux reconcile (immediate) - name: Trigger Flux reconcile (immediate)
run: | run: |
@@ -103,7 +76,7 @@ jobs:
- name: Wait for Flux to apply new ingestion image - name: Wait for Flux to apply new ingestion image
run: | run: |
EXPECTED="gitea.d-ma.be/mathias/ingestion:${{ github.sha }}" EXPECTED="git.d-ma.be/mathias/ingestion:${{ github.sha }}"
for i in $(seq 1 60); do for i in $(seq 1 60); do
CURRENT=$(kubectl get deploy ingestion -n supervisor \ CURRENT=$(kubectl get deploy ingestion -n supervisor \
-o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null || echo "") -o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null || echo "")
@@ -132,35 +105,3 @@ jobs:
kubectl describe pods -n supervisor -l app=ingestion | tail -40 kubectl describe pods -n supervisor -l app=ingestion | tail -40
exit 1 exit 1
} }
- name: Wait for Flux to apply new routing image
run: |
EXPECTED="gitea.d-ma.be/mathias/routing:${{ github.sha }}"
for i in $(seq 1 60); do
CURRENT=$(kubectl get deploy routing -n routing \
-o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null || echo "")
if [ "$CURRENT" = "$EXPECTED" ]; then
echo "✓ Flux applied routing image after ${i}s"
break
fi
sleep 1
done
kubectl get deploy routing -n routing \
-o jsonpath='{.spec.template.spec.containers[0].image}' \
| grep -qx "$EXPECTED" \
|| { echo "✗ Flux did not apply routing image within 60s"; exit 1; }
- name: Verify routing rollout
run: |
kubectl rollout status deployment/routing \
--namespace routing \
--timeout=120s \
|| {
echo "── pod status ──"
kubectl get pods -n routing -o wide
echo "── events ──"
kubectl get events -n routing --sort-by='.lastTimestamp' | tail -20
echo "── describe ──"
kubectl describe pods -n routing -l app=routing | tail -40
exit 1
}
+1 -1
View File
@@ -1,6 +1,6 @@
name: CI name: CI
on: "on":
push: push:
branches: [main] branches: [main]
tags: ["v*"] tags: ["v*"]
+7
View File
@@ -6,6 +6,13 @@
"headers": { "headers": {
"Authorization": "Bearer ${BRAIN_MCP_TOKEN}" "Authorization": "Bearer ${BRAIN_MCP_TOKEN}"
} }
},
"gitea": {
"type": "http",
"url": "https://git-mcp.d-ma.be/mcp",
"headers": {
"Authorization": "Bearer ${GITEA_MCP_TOKEN}"
}
} }
} }
} }
+54 -8
View File
@@ -27,6 +27,14 @@ and climate/sustainability tech.
These rules apply to every task across every project, regardless of harness. These rules apply to every task across every project, regardless of harness.
0. **Pre-task ritual — before ANY implementation (non-negotiable).** Run this before writing a single line:
- **Query the brain** (`brain_query`) for the domain + symptom. If the result changes your approach, surface it before acting. 5 seconds beats 5 hours.
- **Load the relevant skill** — see trigger table in *Engineering Skills* below.
- **Write the failing test first.** Name the test before the function. If the target is untestable (e.g. `main()` wiring), extract the logic into a testable function first. No implementation without a red test.
- **State the observable success criterion** — what specific behavior, output, or passing test proves this is done?
**TDD is non-negotiable.** "Tests pass" is not proof of correctness — only proof the tests ran. Write tests that would catch the bug before writing code that fixes it.
1. **No assumptions.** Don't hide confusion — surface it. Surface tradeoffs explicitly. 1. **No assumptions.** Don't hide confusion — surface it. Surface tradeoffs explicitly.
Think before coding; if the problem is unclear, ask or state assumptions before acting. Think before coding; if the problem is unclear, ask or state assumptions before acting.
2. **Minimum viable code.** Solve with the smallest change that works. Nothing 2. **Minimum viable code.** Solve with the smallest change that works. Nothing
@@ -49,6 +57,22 @@ These rules apply to every task across every project, regardless of harness.
PR flow only when a human reviewer outside the project is required. Document PR flow only when a human reviewer outside the project is required. Document
the reason in PROJECT.md. the reason in PROJECT.md.
6. **Close the loop — every substantive task ends with the same ritual.** Shipping
the code is not the end of the task; capturing it is. Run this unprompted:
- **Tag + bump SemVer** on the change (annotated tag; minor for a feature or
new/changed ADR, patch for a fix; docs in the same commit). Check the repo's
actual last tag — stated versions in docs drift stale.
- **Push** main and the tag (CI is the gate).
- **Persist generalizable learnings to the brain** (`brain_write`, wing/hall) —
the reusable patterns and the footguns that would bite anyone again, never
project status. See *Knowledge base — when to write* below.
- **File discovered-but-deferred work as tracker issues** on the project's own
repo — token-budget gaps, recorded ADR limitations, v2 follow-ups. Don't let
"out of scope, recorded" rot in a commit message; make it a ticket with a
source pointer.
- Surface the brain entries and issue numbers in the closing summary so the
trail is auditable.
## Default stack ## Default stack
| Layer | Default | Fallback | Last resort | | Layer | Default | Fallback | Last resort |
@@ -78,6 +102,26 @@ Exploratory: Rust, Zig — I'll tell you when I want these.
- **Security**: no secrets in code, govulncheck before adding deps, SOPS for encrypted config - **Security**: no secrets in code, govulncheck before adding deps, SOPS for encrypted config
- **Dependencies**: prefer stdlib. testify, slog, templ, sqlc, google.golang.org/adk (agent projects only) are pre-approved; anything else needs justification in the commit message - **Dependencies**: prefer stdlib. testify, slog, templ, sqlc, google.golang.org/adk (agent projects only) are pre-approved; anything else needs justification in the commit message
## Secret handling (every harness, every command)
Tool output is persisted: terminal → `~/.claude/projects` transcripts →
claudewatcher → brain/wiki → gitea history. A secret printed once is
searchable forever, and clearing it means rotating the key. So:
1. **Never print, echo, log, or transform a secret to inspect it.** No
`base64`/`xxd`/`cat` of a key, and never pipe a secret through a transform
to defeat `op run`'s output masking (it masks raw values; base64 hides them
from the mask — that exact trick leaked a key on 2026-06-11).
2. **Secrets stay in the subprocess.** Reference them only as env vars consumed
*inside* `op run --env-file ~/.op-env -- <cmd>`. Never place a literal secret
in a command's argv (it lands in the tool call and the transcript).
3. **Existence check without revealing the value:** `[ -n "$X" ] && echo set`
never `${X:-...}` (returns the value when set) and never echo a substring of it.
4. **Cross-host secrets:** run the secret-consuming command on the host that has
the secret; do not forward a raw key over ssh argv/stdout.
5. If a secret does leak into output, say so immediately and flag it for rotation —
don't bury it.
## Infrastructure ## Infrastructure
Three machines on Tailscale: Three machines on Tailscale:
@@ -157,7 +201,7 @@ entries that age well are about *why*, *how to avoid*, and *what to do when*.
| **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool | | **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool |
| **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same | | **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same |
| **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` | | **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` |
| **Browser / human inspection** | `https://gitea.d-ma.be/mathias/hyperguild``knowledge/` and `wiki/` markdown files | | **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild``knowledge/` and `wiki/` markdown files |
- **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`. - **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`.
- **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as - **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as
@@ -219,15 +263,17 @@ unconditionally on every host, every harness.
## Engineering Skills ## Engineering Skills
Shared engineering skills are available in `~/dev/.skills/`. Load on demand via the index. Shared engineering skills live in the **`mathias/skills`** repo (`git.d-ma.be/mathias/skills`). Clone it to `~/dev/skills/` and run `SKILLS_CHECKOUT_DIR="$PWD" bash install.sh` there to wire every skill into your harnesses (Claude Code, Crush, Antigravity, Mistral Vibe) as native, on-demand skills. (Use `install.sh`, not `task install` — the latter is currently broken, skills#7.) Load at task start — not "on demand" but on schedule, before writing code. Browse `~/dev/skills/SKILLS_INDEX.md` for the full list.
See `~/dev/.skills/SKILLS_INDEX.md` for the full list with descriptions and "use when" triggers. **Skill trigger table — load before starting, not after getting stuck:**
Key skills: | Task type | Load |
- **TDD**: always write tests first — load `tdd` skill |-----------|------|
- **Code Review**: load `code-review` skill before any review | Any feature or bug fix | `tdd` |
- **SOLID/Clean Code**: load `solid` or `clean-code` skill for design work | Refactor or design | `clean-code` or `solid` |
- **Problem first**: load `problem-analysis` skill before coding non-trivial features | Debug | `problem-analysis` |
| Review code or PRs | `code-review` |
| Frame a problem before coding | `problem-analysis` |
--- ---
+68
View File
@@ -4,6 +4,74 @@ Record *why* things are the way they are. Future-you will thank present-you.
--- ---
## 2026-05-28 — three active harnesses: hyperguild, agentsquad, Crush (extends earlier boundary decision)
**Context:** After wiring Crush to LiteLLM in May 2026, there are now three active harnesses.
The earlier boundary decision only covered hyperguild vs agentsquad. Crush's role was undefined.
**Decision:** Three harnesses, three distinct roles, shared skills layer.
| Harness | Engine | Primary use | Brain MCP? | Routing pod? | Skills? |
|---------|--------|-------------|------------|--------------|---------|
| **hyperguild** | Claude Code + MCP | Disciplined solo coding sessions, TDD/review/debug workflows | Yes | Yes | Yes (SKILL.md) |
| **agentsquad** | OpenCode + LiteLLM | Multi-agent task execution, executor/reviewer pipelines | No | No (own routing) | Yes (SKILL.md) |
| **Crush** | Charmbracelet TUI + LiteLLM | Interactive local coding, quick iterations on flamingo | No (not yet) | No (direct LiteLLM) | Yes (SKILL.md) |
**Crush specifics (as of 2026-05-28):**
- Config: `~/.config/crush/crush.json` on flamingo (see brain: `homelab/facts/crush-litellm-wiring-2026-05`)
- Connects directly to LiteLLM at `http://koala:4000/v1/` using `sk-local-123`
- Auth type: `openai-compat` (not `openai`)
- Does NOT go through the routing pod — model selection is manual in the Crush UI
- Brain MCP not wired — Crush has no MCP client capability today; revisit if Crush adds MCP support
**Shared across all three:**
- `mathias/skills` — any SKILL.md file works in all three harnesses
- LiteLLM proxy on koala (`http://koala:4000/v1/`) — Crush and agentsquad both route through it; hyperguild does too for local model calls
**Consequences:** No consolidation needed. crush.json must be kept in sync when litellm_config.yaml model names change. The `crush.json` canonical location is `~/.config/crush/crush.json` on flamingo — not yet tracked in a dotfiles repo (track as tech debt).
---
## 2026-05-28 — "field benchmark" for local models = pass-rate at scale (supersedes GOTTH eval suite)
**Context:** The GOTTH eval suite (45 offline prompts across 5 categories) was replaced by
a "field benchmark" in May 2026, but the replacement was never defined concretely.
**Decision:** The field benchmark is per-skill pass rate over real routing pod usage,
collected automatically by `internal/routing/passrate.go` and exposed at:
```
GET /pass-rate?skill=<name>&window=<duration>
```
No separate eval suite. No synthetic prompts. The benchmark runs itself once the routing
pod receives real traffic. Target: 30-day rolling window per skill, reviewed monthly.
**Bootstrap note:** With no session history, `passrate.go` returns `nil` and the router
defaults to the thinking model for every call. The fast-model path activates only after
real pass-rate data accumulates. Seed with real usage — do not pre-populate.
**Consequences:** Zero maintenance overhead for the benchmark. The tradeoff is that results
are only meaningful after ~2 weeks of real usage, and skills that are rarely invoked will
have statistically thin pass-rate data. Revisit if a skill has fewer than 20 calls in 30 days.
---
## 2026-05-28 — brain injection in skill handlers: review is done, others unverified
**Context:** The April 2026 scope reset listed "brain_query injection into skill handlers"
as the top priority. As of 2026-05-28, `internal/skills/review/handlers.go` calls
`brain.Query(ctx, ...)` before dispatching to the LLM — confirmed in code review.
Status of debug, retrospective, and trainer handlers is unverified.
**Decision:** Treat review as the reference implementation. Verify debug, retrospective,
trainer against the same pattern before shipping new skill work. Tracked in issue #32.
**Consequences:** The April concern may be stale for review. A one-pass audit of the other
three skill handlers closes this fully.
---
## 2026-04-08 — AGENTS.md as cross-tool standard, not CLAUDE.md ## 2026-04-08 — AGENTS.md as cross-tool standard, not CLAUDE.md
**Context**: Multiple tools (Crush, Pi, Antigravity) read `AGENTS.md` natively. Claude Code reads `CLAUDE.md`. Building on `CLAUDE.md` as the primary format locks into one vendor. **Context**: Multiple tools (Crush, Pi, Antigravity) read `AGENTS.md` natively. Claude Code reads `CLAUDE.md`. Building on `CLAUDE.md` as the primary format locks into one vendor.
-30
View File
@@ -1,30 +0,0 @@
# syntax=docker/dockerfile:1
# ── Build stage ───────────────────────────────────────────────────────────────
FROM golang:1.26-bookworm AS builder
ARG VERSION=dev
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
go build -trimpath -ldflags="-s -w -X main.version=${VERSION}" \
-o /out/routing ./cmd/routing
# ── Runtime stage ─────────────────────────────────────────────────────────────
FROM gcr.io/distroless/base-debian12
COPY --from=builder /out/routing /usr/local/bin/routing
COPY config/ /app/config/
ENV SUPERVISOR_CONFIG_DIR=/app/config/supervisor
ENV ROUTING_PORT=3210
EXPOSE 3210
USER 65532:65532
ENTRYPOINT ["/usr/local/bin/routing"]
+49
View File
@@ -0,0 +1,49 @@
# Icebox — retired code (recoverable, not destroyed)
Per issue #75 (consolidate to a single harness), the routing-pod path was removed
from the live tree. It is **preserved and recoverable**, not deleted without trace.
## What was iceboxed (2026-07-01, issue #75)
| Path | Why |
|------|-----|
| `cmd/routing/` | The routing MCP-server binary; every skill call was wrapped through the broken pass-rate router (`wrap(skillName)`). |
| `internal/routing/` | Router / Fetcher / Policy / pass-rate. Signal is survivorship-biased and unusable as-is (infra#174). |
| `internal/skills/{review,debug,retrospective,trainer,project}/` | Skill handlers usable **only** through `cmd/routing` (verified: each imported solely by `cmd/routing`). |
| `Dockerfile.routing` | Built `cmd/routing` exclusively. |
| `.gitea/workflows/cd.yml` (routing steps only) | Removed the routing image build + infra image-bump + Flux-wait/rollout-verify for routing; **ingestion build/deploy is unchanged**. |
## Why
The live minimal harness is `cmd/hyperguild` + `brain-mcp` (+ `gitea-mcp` available) —
that is what ran the infra#170 loop-1 experiment (routing/injection machinery off) and
closed it twice. `cmd/hyperguild` has **zero** transitive dependency on `internal/routing`
or `cmd/routing`'s packages (it imports only `internal/tier`). The routing pass-rate signal
is being retired, not resurrected (fresh start, per infra#174).
## How to recover
Everything above is preserved at commit `00e5f62` under:
- **tag** `icebox/cmd-routing-2026-07-01`
- **branch** `icebox/cmd-routing`
```bash
# inspect
git checkout icebox/cmd-routing-2026-07-01
# restore specific packages onto a branch
git checkout icebox/cmd-routing-2026-07-01 -- cmd/routing internal/routing \
internal/skills/review internal/skills/debug internal/skills/retrospective \
internal/skills/trainer internal/skills/project Dockerfile.routing
```
## Deliberately NOT touched here (separate scope)
- **Live k8s routing deployment** (`infra` repo, `k3s/apps/routing/`) still runs its last
image; CD no longer rebuilds/redeploys it. Tearing down that deployment is a separate
infra-repo task.
- **`internal/skills/{brain,org,sessionlog}/`** — kept per #75; already had no importer
(orphaned before this cut), harmless, compile + test green.
- **`config/supervisor/{review,debug,retrospective,trainer-*}.md`** — routing skill prompts,
now orphaned data; left in place (not code, no build impact).
+72 -49
View File
@@ -5,14 +5,31 @@ Instead of letting Claude Code do whatever it wants, hyperguild enforces structu
workflows (TDD red/green/refactor), logs every session, and accumulates learnings workflows (TDD red/green/refactor), logs every session, and accumulates learnings
into a searchable brain. into a searchable brain.
## Hypothesis
> We believe routing skill tasks through local models, backed by brain context,
> produces measurably better outcomes than raw Claude Code alone —
> measurable by per-skill pass rate over rolling 30-day windows
> (available at `GET /pass-rate?skill=<name>&window=30d` on the brain pod).
This is the falsifiable claim the routing pod and pass-rate infrastructure exist to test.
If per-skill pass rates don't improve over baseline (all-cloud) after 30 days of real
usage, the fast-model routing path should be reconsidered.
## Harness
**hyperguild = Claude Code + MCP.** This is a supervisor for Claude Code sessions specifically.
For multi-agent orchestration (OpenCode + LiteLLM, executor/reviewer pipelines), see
[agentsquad](http://gitea.d-ma.be/mathias/agentsquad) — a separate harness for a different
orchestration model. Skills (mathias/skills) are shared between both.
## How it works ## How it works
``` ```
Your Claude Code session (in any project) Your Claude Code session (in any project)
│ MCP over HTTP (Tailscale) │ MCP over HTTP (Tailscale)
├──▶ supervisor :3200 (NodePort 30320 on koala) — skill workers: tdd, debug, spec, … ├──▶ routing :3210 (NodePort 30310 on koala) — review, debug, retrospective, trainer
├──▶ routing :3210 (NodePort 30310 on koala) — Mode 2 only: review, debug, retrospective, trainer
└──▶ brain :3300 (NodePort 30330 on koala) — brain_query, brain_write, brain_ingest, session_log └──▶ brain :3300 (NodePort 30330 on koala) — brain_query, brain_write, brain_ingest, session_log
└─ also serves the legacy REST endpoints (/query, /write, /ingest, …) └─ also serves the legacy REST endpoints (/query, /write, /ingest, …)
@@ -20,34 +37,28 @@ Your Claude Code session (in any project)
brain/ brain/
├── sessions/ — JSONL log, one file per session_id ├── sessions/ — JSONL log, one file per session_id
├── wiki/ — searchable knowledge (full-text) ├── wiki/ — searchable knowledge (wing/hall layout)
│ ├── concepts/ │ ├── homelab/
│ ├── entities/ │ ├── claude-sessions/
│ └── sources/ │ └── ...
── raw/ — retrospective output, staged for review ── knowledge/ — legacy flat notes (migration pending: hyperguild#22)
└── training-data/ — SFT/DPO/RL data (Phase 2)
``` ```
## Phase 1 tools (available now) ## Phase 1 tools (available now)
| Tool | What it does | | Tool | What it does |
|------|-------------| |------|-------------|
| `tdd_red` | Writes a failing test for a spec, verifies it fails |
| `tdd_green` | Writes the minimal implementation to make tests pass |
| `tdd_refactor` | Cleans up implementation while keeping tests green |
| `session_log` | Appends a structured entry to the session JSONL log | | `session_log` | Appends a structured entry to the session JSONL log |
| `retrospective` | Reads the session log, identifies novel learnings, writes to brain/raw/ | | `retrospective` | Reads the session log, identifies novel learnings, writes to brain |
| `review` | Structured code review via local model, brain-context injected |
| `debug` | Hypothesis-driven debugging via local model |
| `brain_query` | Full-text search over brain/wiki/ | | `brain_query` | Full-text search over brain/wiki/ |
| `brain_write` | Writes a note to brain/raw/ (with optional YAML frontmatter) | | `brain_write` | Writes a note to brain (with wing/hall routing) |
| `brain_answer` | BM25 + LLM synthesis — Q&A over brain corpus |
| `tier` | Returns the current connectivity tier (1=cloud, 2=LAN, 3=offline) | | `tier` | Returns the current connectivity tier (1=cloud, 2=LAN, 3=offline) |
## Start the servers > **Note:** `tdd_red/green/refactor` and `spec` were retired in Plan 7 (2026-05-12).
> They are now SKILL.md files in [mathias/skills](http://gitea.d-ma.be/mathias/skills).
```bash
# Requires goreman: go install github.com/mattn/goreman@latest
task start # starts ingestion (:3300) + supervisor (:3200) via goreman
task stop # kills both by port
```
## Connect a project ## Connect a project
@@ -56,9 +67,9 @@ Create `.mcp.json` in your project root:
```json ```json
{ {
"mcpServers": { "mcpServers": {
"supervisor": { "routing": {
"type": "http", "type": "http",
"url": "http://koala:30320/mcp" "url": "http://koala:30310/mcp"
}, },
"brain": { "brain": {
"type": "http", "type": "http",
@@ -68,33 +79,29 @@ Create `.mcp.json` in your project root:
} }
``` ```
Two MCP servers are exposed today, both reachable over Tailscale: Two MCP servers are exposed, both reachable over Tailscale:
- **`supervisor`** at `koala:30320` — skill workers (`tdd_red/green/refactor`, - **`routing`** at `koala:30310` — skill workers (`review`, `debug`, `retrospective`, `trainer`).
`review`, `debug`, `spec`, `retrospective`, `trainer`, `tier`). Routes each call to fast local model or thinking model based on per-skill pass rate.
- **`brain`** at `koala:30330` — knowledge access (`brain_query`, `brain_write`, - **`brain`** at `koala:30330` — knowledge access (`brain_query`, `brain_write`,
`brain_ingest`, `brain_ingest_raw`) and `session_log`. Hosted by the ingestion `brain_ingest`, `brain_ingest_raw`, `brain_answer`, `brain_classify`) and `session_log`.
service directly, no separate pod.
No local binary or stdio shim is required — Claude Code talks to both via HTTP. No local binary or stdio shim is required — Claude Code talks to both via HTTP.
Open Claude Code in your project — run `/mcp` to confirm both servers are listed. Open Claude Code in your project — run `/mcp` to confirm both servers are listed.
## A typical TDD session ## A typical session
``` ```
1. Call tdd_red → spec in, failing test file out 1. Call review → brain context injected + local model review → findings
2. Call tdd_green → test path in, implementation out 2. Call session_log → log each phase result
3. Call tdd_refactor → impl + test in, cleaned code out 3. Call retrospective → extracts learnings → brain
4. Call session_log → log each phase result 4. Future sessions: call brain_query / brain_answer to retrieve relevant context
5. Call retrospective → extracts learnings → brain/raw/
6. Review brain/raw/, move worthy notes to brain/wiki/concepts/
7. Future sessions: call brain_query to retrieve relevant context
``` ```
## Tier detection ## Tier detection
The supervisor probes connectivity at call time: The routing pod probes connectivity at call time:
| Tier | Label | Condition | | Tier | Label | Condition |
|------|-------|-----------| |------|-------|-----------|
@@ -102,31 +109,47 @@ The supervisor probes connectivity at call time:
| 2 | lan-only | Can reach LiteLLM but not Anthropic | | 2 | lan-only | Can reach LiteLLM but not Anthropic |
| 3 | airplane | No external connectivity | | 3 | airplane | No external connectivity |
## Model routing
The routing pod selects models per skill call based on historical pass rate:
| Pass rate | Decision |
|-----------|----------|
| ≥ 0.90 (FLOOR) | Fast model (`HYPERGUILD_FAST_MODEL`) |
| ≤ 0.70 (CEIL) | Thinking model (`HYPERGUILD_THINKING_MODEL`) |
| between CEIL and FLOOR | Sample band — probabilistic routing |
| nil (no history yet) | Defaults to thinking model |
> **Bootstrap note:** With no session history, all calls route to the thinking model.
> The fast-model path activates only after real pass-rate data accumulates at `/pass-rate`.
> Seed with real usage — don't try to pre-populate.
## Key env vars ## Key env vars
| Variable | Default | Purpose | | Variable | Default | Purpose |
|----------|---------|---------| |----------|---------|---------|
| `INGEST_BRAIN_DIR` | `../brain` | Brain directory for ingestion server | | `INGEST_BRAIN_DIR` | `../brain` | Brain directory for ingestion server |
| `INGEST_PORT` | `3300` | Ingestion server port | | `INGEST_PORT` | `3300` | Ingestion server port |
| `SUPERVISOR_CONFIG_DIR` | `./config/supervisor` | Skill discipline files | | `INGEST_BASE_URL` | `http://localhost:3300` | Routing pod → brain |
| `SUPERVISOR_SESSIONS_DIR` | `./brain/sessions` | JSONL session logs |
| `INGEST_BASE_URL` | `http://localhost:3300` | Supervisor → ingestion |
| `LITELLM_BASE_URL` | — | LiteLLM proxy for Tier 2 model routing | | `LITELLM_BASE_URL` | — | LiteLLM proxy for Tier 2 model routing |
| `SUPERVISOR_MCP_TOKEN` | — | Optional bearer token for the supervisor MCP HTTP endpoint; when empty, no auth is enforced |
| `ROUTING_PORT` | `3210` | Routing pod's listen port | | `ROUTING_PORT` | `3210` | Routing pod's listen port |
| `ROUTING_MCP_TOKEN` | — | Optional bearer token for the routing MCP HTTP endpoint | | `ROUTING_MCP_TOKEN` | — | Optional bearer token; when empty, no auth enforced |
| `BRAIN_URL` | `http://ingestion.supervisor:3300` | Routing pod → brain (in-cluster) | | `BRAIN_URL` | `http://ingestion.supervisor:3300` | Routing pod → brain (in-cluster) |
| `HYPERGUILD_FAST_MODEL` | `koala/qwen35-9b-fast` | Fast model for high-pass-rate skill calls | | `HYPERGUILD_FAST_MODEL` | `koala/qwen35-9b-fast` | Fast model for high-pass-rate skill calls |
| `HYPERGUILD_THINKING_MODEL` | `iguana/gemma4-26b` | Thinking model for low-pass-rate skill calls | | `HYPERGUILD_THINKING_MODEL` | `iguana/gemma4-26b` | Thinking model for low-pass-rate skill calls |
| `HYPERGUILD_ROUTE_LOCAL_FLOOR` | `0.90` | At/above pass rate, route to fast model | | `HYPERGUILD_ROUTE_LOCAL_FLOOR` | `0.90` | Fast model threshold |
| `HYPERGUILD_ROUTE_LOCAL_CEIL` | `0.70` | Below pass rate, route to thinking model. Between CEIL and FLOOR is the sample band. | | `HYPERGUILD_ROUTE_LOCAL_CEIL` | `0.70` | Thinking model threshold |
| `HYPERGUILD_PASS_RATE_TTL_SECONDS` | `60` | Per-skill pass-rate cache TTL | | `HYPERGUILD_PASS_RATE_TTL_SECONDS` | `60` | Per-skill pass-rate cache TTL |
> **Operator note:** LiteLLM at `LITELLM_BASE_URL` must register both `HYPERGUILD_FAST_MODEL` and `HYPERGUILD_THINKING_MODEL` for routing to do useful work. If a model is missing, LiteLLM returns 4xx, the routing pod's fast route fails, the fail-open retry on the thinking model likely also fails (since both are missing), and the only signal is `final_status: "fail"` on `_routing` entries in the brain. > **Operator note:** LiteLLM at `LITELLM_BASE_URL` must register both `HYPERGUILD_FAST_MODEL`
> and `HYPERGUILD_THINKING_MODEL`. If a model is missing, the fail-open retry also fails and
> the only signal is `final_status: "fail"` on `_routing` entries in the brain.
## Phase 2 (planned) ## Open issues
- `review` skill — structured code review with iron law enforcement See [issues](http://gitea.d-ma.be/mathias/hyperguild/issues) — key open items:
- `debug` skill — hypothesis-driven debugging sessions
- `spec` skill — generates specs from conversations - **#25** — skills platform overhaul (audit first, then lazy loading + brain feedback loop)
- `trainer` — extracts SFT/DPO pairs from session logs for fine-tuning - **#24** — reduce context burn from skill listing
- **#22** — migrate legacy brain notes to wing/hall layout (one-shot script, low risk)
- **#31** — connect routing-mcp to claude.ai as custom connector
+2 -4
View File
@@ -17,8 +17,6 @@ tasks:
cmds: [bash scripts/context-sync.sh claude] cmds: [bash scripts/context-sync.sh claude]
context:sync:agents: context:sync:agents:
cmds: [bash scripts/context-sync.sh agents] cmds: [bash scripts/context-sync.sh agents]
context:sync:cursor:
cmds: [bash scripts/context-sync.sh cursor]
# ── Development ──────────────────────────────────────────────────────────── # ── Development ────────────────────────────────────────────────────────────
@@ -90,12 +88,12 @@ tasks:
cmds: cmds:
- task: context:sync - task: context:sync
- cmd: | - cmd: |
drift=$(git status --porcelain -- AGENTS.md CLAUDE.md .cursorrules .aider.conventions.md .context/system-prompt.txt 2>/dev/null) drift=$(git status --porcelain -- AGENTS.md CLAUDE.md .context/system-prompt.txt 2>/dev/null)
if [ -n "$drift" ]; then if [ -n "$drift" ]; then
echo "ERROR: derived adapters drifted from canonical context." >&2 echo "ERROR: derived adapters drifted from canonical context." >&2
echo "$drift" >&2 echo "$drift" >&2
echo "" >&2 echo "" >&2
echo "Run: git add AGENTS.md CLAUDE.md .cursorrules .aider.conventions.md .context/system-prompt.txt" >&2 echo "Run: git add AGENTS.md CLAUDE.md .context/system-prompt.txt" >&2
echo " git commit -m 'chore: re-sync context adapters'" >&2 echo " git commit -m 'chore: re-sync context adapters'" >&2
exit 1 exit 1
fi fi
@@ -0,0 +1,48 @@
{"_meta":true,"note":"Agent-consumer column of brain-MCP intent analysis. consumer_type fixed=autonomous_agent. CAVEAT: canonical schema file brain-intent-extraction.md is NOT present on this host (koala) — only this session's own task prompt references it. The closed intent vocabulary below was RECONSTRUCTED from the task prompt's framing + brain/schema.md. Re-map intent labels if the canonical vocab differs. schema_source=reconstructed on every row.","closed_intent_vocab":["semantic_retrieval","lexical_lookup","check_prior_art","synthesized_answer","store_new_knowledge","update_or_supersede","ingest_raw_source","verify_write_landed","discover_capability","intent_unclear"],"intent_tool_match_values":["match","mismatch","partial"],"corpus":"~/.claude/projects/*/*.jsonl (Claude Code agent transcripts on koala). brain/sessions/*.jsonl empty. agentsquad docs/eval/*.jsonl are code-review eval results, NOT brain calls. No separate Crush logs found. Zero brain calls appear under any mcp__ name with a human typing the call — all brain acts are agent-initiated (CLAUDE.md reflex), so all qualify as autonomous_agent."}
{"id":"a01","session":"tapir-c","ts":"2026-06-?T15:01:51","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"single pre-task query 'YouTube Data API captions download ownership limitation timedtext adapter Go' — named-entity lexical lookup, fit BM25 well, no reformulation."}
{"id":"a02","session":"tapir","ts":"2026-06-05T21:44:10","tool":"brain_ingest","intent":"ingest_raw_source","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_ingest 14s earlier — tool not ambient, had to be discovered/loaded first.","evidence":"source=tapir-scheduled-discovery-session-2026-06-05, a session learnings dump."}
{"id":"a03","session":"tapir","ts":"2026-06-?T14:00:59","tool":"brain_ingest","intent":"ingest_raw_source","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"source=tapir-rls-identity-bootstrapping, first write of RLS lesson."}
{"id":"a04","session":"tapir","ts":"2026-06-?T14:02:08","tool":"brain_ingest","intent":"update_or_supersede","intent_tool_match":"mismatch","workaround":"RE-INGESTED same source name 'tapir-rls-identity-bootstrapping' 69s later with edited/condensed body. No update/patch/supersede verb exists, so the agent overwrote-by-re-ingest. Whether this dedups or creates a v2 duplicate is opaque to the agent.","observed_friction":"agent revised content within 70s of first write — classic edit-after-write with no edit primitive.","evidence":"two brain_ingest, identical source string, divergent content."}
{"id":"a05","session":"tapir","ts":"2026-06-?T14:56:27","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"postgres-cascade-skips-tables-without-fk.md — distinct new lesson."}
{"id":"a06","session":"tapir","ts":"2026-06-?T21:08:57","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_query — discovery tax again.","evidence":"'Dex passwords.dex.coreos.com CRD ...' keyword-rich, single shot."}
{"id":"a07","session":"AI-infra","ts":"2026-05-?T15:38:03","tool":"brain_query(HTTP-curl)","intent":"discover_capability","intent_tool_match":"mismatch","workaround":"raw `curl -X POST` to brain-mcp endpoint instead of MCP tool. Preceded by two ToolSearch ('brain knowledge memory' then 'brain') that did not yield a usable loaded tool, so agent fell back to HTTP.","observed_friction":"3-step ladder: ToolSearch 'brain knowledge memory' -> ToolSearch 'brain' -> curl. Agent did not know which act maps to which tool name.","evidence":"curl -s -o /tmp/brain-init.txt -w code:%{http_code} -X POST ..."}
{"id":"a08","session":"AI-infra","ts":"2026-05-?T04:46:13","tool":"brain_query(HTTP-curl)","intent":"discover_capability","intent_tool_match":"mismatch","workaround":"hand-set TOKEN=... then curl brain-test endpoint — probing whether the HTTP brain path is reachable/authed at all. MCP path not used.","observed_friction":"agent testing connectivity by hand; MCP auth/availability not trusted.","evidence":"TOKEN=...; curl -s -o /tmp/brain-test ..."}
{"id":"a09","session":"AI-infra","ts":"2026-05-?T05:23:35","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_query (after an earlier 05:03 ToolSearch 'brain ingestion knowledge wiki' that explored layers).","evidence":"'koala machine state RTX 5070 llama-swap' — named-entity recall, fits lexical."}
{"id":"a10","session":"AI-infra","ts":"2026-05-?T05:27:36","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"batch of 3 in ~1s (k3s/flux gitops; llama-swap ai-stack GPU; MCP Dex OAuth claude.ai) — parallel prior-art sweep, all named-entity."}
{"id":"a11","session":"AI-infra","ts":"2026-05-?T07:13:50","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"batch of 4 in ~2s before a debugging session (flux healthCheck; exit 255 restart loop; NVML mismatch; mirror rebase). Named symptoms, lexical fit OK on first pass."}
{"id":"a12","session":"AI-infra","ts":"2026-05-?T07:14:26","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"4 writes in ~35s (flux-healthcheck-stale; exit-255-unknown-reason-not-oom; nvidia-nvml-mismatch; mcp-static-bearer) — answers to the 4 queries just run, captured as lessons. Healthy query->fix->write loop."}
{"id":"a13","session":"AI-infra","ts":"2026-05-?T07:15:25","tool":"brain_query","intent":"verify_write_landed","intent_tool_match":"mismatch","workaround":"25s after writing exit-255-unknown-reason-not-oom.md, re-queried 'exit 255 unknown SIGKILL containerd' — reformulated terms (SIGKILL/containerd not in original query 'exit 255 unknown reason restart loop diagnosis'). Either confirming the fresh write is retrievable or re-searching because first lexical query missed. No read-after-write / get-by-id act exists.","observed_friction":"reformulation chain: 'exit 255 unknown reason restart loop diagnosis' -> 'exit 255 unknown SIGKILL containerd'. Same need, different keywords.","evidence":"query at 07:13:51 vs 07:15:25 bracketing the 07:14:37 write."}
{"id":"a14","session":"AI-infra","ts":"2026-05-?T07:22:55","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"batch of 5 in ~20s before a homelab security audit (piguard/iguana tailscale; unifi UCG firewall; SOPS age; ingress TLS cert-manager; koala UFW iptables). Named-entity sweep."}
{"id":"a15","session":"AI-infra","ts":"2026-05-?T09:27:53","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"audit-shortcut-tls-blocks-zero; policy-audit-mode-blocks-nothing — distinct new audit lessons."}
{"id":"a16","session":"AI-infra","ts":"2026-05-?T09:28:22","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"homelab-security-chains-not-bugs.md FIRST write (worked example: koala 2026-05-13)."}
{"id":"a17","session":"AI-infra","ts":"2026-05-?T10:32:49","tool":"brain_write","intent":"update_or_supersede","intent_tool_match":"mismatch","workaround":"RE-WROTE homelab-security-chains-not-bugs.md ~64min later with a different/expanded worked example (host-user dotfile, over-broad ClusterRole). Same filename, additive revision, no patch/append/supersede verb — agent overwrites and hopes the index replaces rather than duplicates.","observed_friction":"the in-between hour of audit work produced a better example; only way to fold it in was a full re-write of the same slug.","evidence":"two brain_write same filename at 09:28:22 and 10:32:49, divergent worked examples."}
{"id":"a18","session":"AI-infra","ts":"2026-05-?T10:18:25","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"homelab-document-accepted-risk-to-break-audit-cycle.md — distinct."}
{"id":"a19","session":"AI-infra","ts":"2026-05-?T10:32:55","tool":"brain_query","intent":"verify_write_landed","intent_tool_match":"mismatch","workaround":"6s after the homelab-chains re-write, queried 'RBAC MCP cluster pods log chain' — checking the chain reasoning is retrievable / finding the related entry. Read-after-write done via lexical search.","observed_friction":null,"evidence":"query immediately follows the 10:32:49 write."}
{"id":"a20","session":"AI-infra","ts":"2026-05-?T18:57:34","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_write,brain_query — re-discovered tools this session.","evidence":"'extension build pinned version major version upgrade postgres pgvector'."}
{"id":"a21","session":"AI-infra","ts":"2026-05-?T18:57:58","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"extension-version-lags-platform-major-upgrade.md."}
{"id":"a22","session":"AI-infra","ts":"2026-05-?T18:58:06","tool":"brain_query","intent":"verify_write_landed","intent_tool_match":"mismatch","workaround":"8s after writing extension-version-lags, re-queried 'pgvector postgres extension version compile error bump' — reformulated from the 18:57:34 query ('extension build pinned version...'). Lexical re-search to confirm the just-written lesson is findable, with different keyword guess.","observed_friction":"reformulation: 'extension build pinned version major version upgrade postgres pgvector' -> 'pgvector postgres extension version compile error bump'.","evidence":"write at 18:57:58 bracketed by queries 18:57:34 and 18:58:06."}
{"id":"a23","session":"AI-infra","ts":"2026-05-?T18:31:23","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"webfetch-readme-when-image-or-flag-uncertain.md FIRST write."}
{"id":"a24","session":"AI-infra","ts":"2026-05-?T18:34:23","tool":"brain_write","intent":"update_or_supersede","intent_tool_match":"mismatch","workaround":"RE-WROTE webfetch-readme-when-image-or-flag-uncertain.md 3min later, near-identical body. Looks like a retry/overwrite (uncertain the first landed, or minor edit). No idempotent upsert with confirmation, so agent re-fires the write.","observed_friction":"followed 7s later by a brain_query on the same topic ('OSS tool image registry CLI flag webhook path schema drift README pre-flight') — write-write-query, i.e. overwrite then verify-by-search.","evidence":"two brain_write same filename 18:31:23 / 18:34:23, then query 18:34:31."}
{"id":"a25","session":"AI-infra","ts":"2026-05-?T18:34:31","tool":"brain_query","intent":"verify_write_landed","intent_tool_match":"mismatch","workaround":"keyword-stuffed lexical query 'OSS tool image registry CLI flag webhook path schema drift README pre-flight' fired right after the webfetch-readme write — agent dumps every concept token hoping BM25 surfaces its own fresh note. This is semantic intent (find that conceptual lesson) coerced into a bag-of-keywords.","observed_friction":"query is a concatenation of the note's section headings — a tell that the agent is groping lexically for content it knows by meaning.","evidence":"query text mirrors the just-written note's bullet topics."}
{"id":"a26","session":"dev","ts":"2026-06-?T21:18:26","tool":"brain_answer","intent":"synthesized_answer","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_answer.","evidence":"'tapir transcript persistence shared cross-user dedup table RLS isolation ADR-021 ...' -> 22min later a brain_write (acted on the answer). Answer consumed, not re-queried. Healthy."}
{"id":"a27","session":"dev","ts":"2026-06-?T21:40:20","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"tapir-migration-and-rls-test-infra-gotchas, with wing/hall absent here (flat) — see schema-confusion note a40."}
{"id":"a28","session":"dev","ts":"2026-06-?T05:35:04","tool":"brain_answer","intent":"synthesized_answer","intent_tool_match":"partial","workaround":"asked 'gitea MCP not working workaround file issue via API which token ... how to authenticate gitea API' — a how-do-I question. Next brain act (05:39 query) is a different topic (tapir transcript), so the answer was apparently sufficient OR abandoned; ambiguous.","observed_friction":null,"evidence":"brain_answer then unrelated brain_query 4min later."}
{"id":"a29","session":"dev","ts":"2026-06-?T05:39:54","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"'tapir transcript persistence ADR-021 shared non-RLS'."}
{"id":"a30","session":"dev","ts":"2026-06-?T05:57:37","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"gitea-mcp-per-repo-tools-404-and-rest-fallback FIRST write."}
{"id":"a31","session":"dev","ts":"2026-06-?T05:57:55","tool":"brain_write","intent":"update_or_supersede","intent_tool_match":"mismatch","workaround":"RE-WROTE gitea-mcp-per-repo-tools-404-and-rest-fallback 18s later — overwrite/retry of same slug, no upsert confirmation.","observed_friction":"sub-20s gap = almost certainly a content tweak the agent could not express as an edit.","evidence":"two brain_write same filename 05:57:37 / 05:57:55."}
{"id":"a32","session":"dev","ts":"2026-05-?T11:51:47","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"infra-litellm-absorption-2026-05-16.md."}
{"id":"a33","session":"dev","ts":"2026-05-?T12:07:04","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"batch of 3 (litellm rebuild time piguard; docker compose orphaned volumes; prometheus_client ModuleNotFoundError) — lexical, error-string driven."}
{"id":"a34","session":"dev","ts":"2026-05-?T15:08:15","tool":"brain_answer","intent":"synthesized_answer","intent_tool_match":"mismatch","workaround":"THREE brain_answer at 15:08 (moved compose volumes? / pi rebuild time? / litellm ModuleNotFound prometheus) — the SAME three topics queried lexically an hour earlier (12:07) — were IMMEDIATELY followed at 15:09 by THREE brain_query on the same three topics. The agent asked the synthesizer, was unsatisfied, and fell straight back to raw lexical search. Strongest answer->query fallback in the corpus.","observed_friction":"answer/query duplication across one intent: agent hedges by firing both interfaces, trusting neither.","evidence":"15:08 answers vs 15:09 queries, topic-for-topic aligned."}
{"id":"a35","session":"dev","ts":"2026-05-?T15:09:16","tool":"brain_query","intent":"semantic_retrieval","intent_tool_match":"mismatch","workaround":"after the 3 brain_answer calls failed to satisfy, re-issued as lexical brain_query ('moved compose stack to new directory volumes disappeared empty'; 'raspberry pi docker build time arm slow'; 'how to enable prometheus metrics on litellm proxy callback'). The want is meaning-based ('did my volumes move?') but the only retrieval that 'worked' was keyword search — and these are full natural-language sentences crammed into a BM25 box.","observed_friction":"natural-language questions ('how to enable...', 'moved ... disappeared') passed to a lexical query tool — semantic intent, lexical interface.","evidence":"3 queries at 15:09 mirror the 3 answers at 15:08."}
{"id":"a36","session":"dev","ts":"2026-05-?T20:31:50","tool":"brain_answer","intent":"synthesized_answer","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"'What happened with the litellm migration on 2026-05-16?' — episodic recall question, answer fit; no re-query followed."}
{"id":"a37","session":"dev","ts":"2026-05-?T21:07:17","tool":"brain_query","intent":"semantic_retrieval","intent_tool_match":"mismatch","workaround":"FOUR-step reformulation chain over one Go bug: 'bytes.Buffer Bytes Reset aliasing slice sharing' -> 'go buffer reuse map backing array bug' -> [write go-bytes-buffer-bytes-reset-aliasing-trap.md] -> 'go map values all show same content after loop' -> 'bytes.Buffer Bytes returns same data every iteration'. The agent knows the SYMPTOM (all map values identical) and the CAUSE (Bytes() aliasing) but cannot phrase a single lexical query that bridges them — it wants concept retrieval and is forced to brute-force keyword variants.","observed_friction":"4 distinct phrasings of the same bug, two before and two after writing the lesson — also doubles as verify_write_landed on the trailing queries.","evidence":"21:07:17, 21:07:17, (write 21:08:01), 21:08:10, 21:08:19."}
{"id":"a38","session":"dev","ts":"2026-05-?T21:08:01","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"go-bytes-buffer-bytes-reset-aliasing-trap.md."}
{"id":"a39","session":"dev","ts":"2026-05-?T07:54:26","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"mcp-tool-design-get-needs-list-partner.md — a design principle."}
{"id":"a40","session":"dev","ts":"2026-06-?T06:25:00","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"'Dex to Authentik migration auth.d-ma.be issuer cutover OIDC subject ...'."}
{"id":"a41","session":"dev","ts":"2026-06-?T06:25:08","tool":"brain_answer","intent":"synthesized_answer","intent_tool_match":"mismatch","workaround":"brain_query (a40) and brain_answer (a41) fired ~8s apart on the SAME intent (Dex->Authentik subject-keyed token orphan). Agent runs lexical search AND synthesized answer in parallel for one question rather than choosing — it cannot predict which interface will return usable knowledge, so it pays both.","observed_friction":"query+answer doublet on one need.","evidence":"06:25:00 query then 06:25:08 answer, same topic."}
{"id":"a42","session":"dev","ts":"2026-06-?T13:48:49","tool":"brain_query","intent":"verify_write_landed","intent_tool_match":"mismatch","workaround":"'authentik cutover validation probe' then 6min later 'authentik cutover post-flip validation' — reformulated pair, likely searching for the agent's own earlier cutover notes / confirming validation steps are recorded. Lexical re-search standing in for recall-my-recent-context.","observed_friction":"reformulation: 'validation probe' -> 'post-flip validation'.","evidence":"13:48:49 and 13:54:56."}
{"id":"a43","session":"dev","ts":"2026-06-?T13:59:17","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_write.","evidence":"oidc-issuer-host-change-vs-idp-swap-subject."}
{"id":"a44","session":"dev","ts":"2026-06-?T13:59:50","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"cannot-move-ingress-host-across-namespaces-flux-dryrun FIRST write."}
{"id":"a45","session":"dev","ts":"2026-06-?T14:00:13","tool":"brain_write","intent":"update_or_supersede","intent_tool_match":"mismatch","workaround":"RE-WROTE cannot-move-ingress-host-across-namespaces-flux-dryrun 23s later — overwrite of same slug, no edit/upsert primitive.","observed_friction":"sub-30s gap = content correction expressed as a full re-write.","evidence":"two brain_write same filename 13:59:50 / 14:00:13."}
{"id":"a46","session":"dev","ts":"2026-06-?T13:53:54","tool":"brain_write(HTTP-staged)","intent":"store_new_knowledge","intent_tool_match":"mismatch","workaround":"after `ToolSearch select:mcp__claude_ai_brain__authenticate` (MCP auth flow), the agent staged the entry as `cat > /tmp/brain_entry.json` ({filename:'postgres-force-rls-cross-u...', content}) for a curl write rather than calling brain_write directly — MCP write path was not usable (auth/loading), so it dropped to the HTTP bodge.","observed_friction":"reached for an 'authenticate' tool, then abandoned MCP for hand-built JSON + curl. Matches known pattern: brain/op MCP auth lapses often.","evidence":"ToolSearch authenticate 13:53:27 -> cat /tmp/brain_entry.json 13:53:54."}
{"id":"a47","session":"template-go-agent","ts":"2026-05-?T18:46:26","tool":"BASH(not-a-brain-act)","intent":"intent_unclear","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"'brain_' substring was inside a git commit message body ('agent boundaries, network policy, agent s...'), NOT a brain call. Excluded from knowledge-act analysis; logged for audit completeness."}
@@ -0,0 +1,148 @@
# Agent-Consumer Brain Intent Analysis — koala column
**Consumer:** `autonomous_agent` (all rows). **Host:** koala. **Date:** 2026-06-15.
**Raw rows:** `agent-intent-column.jsonl` (46 real knowledge-acts + 1 excluded false-positive).
## Caveat — canonical schema not on this host
The shared closed-vocabulary file `brain-intent-extraction.md` **does not exist on
koala** — the only reference to it is inside *this task's own prompt*. The intent
vocabulary below was **reconstructed** from the prompt's framing + `brain/schema.md`.
Every row carries `schema_source: reconstructed`. If the canonical vocab differs,
re-map the `intent` field; the `intent_tool_match` / `workaround` / `observed_friction`
evidence stands regardless of label names.
**Reconstructed closed vocab:** `semantic_retrieval`, `lexical_lookup`,
`check_prior_art`, `synthesized_answer`, `store_new_knowledge`, `update_or_supersede`,
`ingest_raw_source`, `verify_write_landed`, `discover_capability`, `intent_unclear`.
## Corpus
- `~/.claude/projects/*/*.jsonl` — Claude Code agent transcripts (98 files). **The only
source with brain calls.**
- `brain/sessions/*.jsonl` — empty (only `.gitkeep`).
- `agentsquad docs/eval/*.jsonl` — code-review eval results, **not** brain calls.
- No separate Crush session logs on this host.
- **Zero** brain calls were human-typed. Every brain act is agent-initiated (the
CLAUDE.md "query as reflex / close-the-loop write" behaviour), so all qualify as
`autonomous_agent`. The human gave the top-level task; the agent chose every brain act.
## 1. Intent histogram, split by `intent_tool_match`
| intent | match | mismatch | partial | total |
|---|---|---|---|---|
| check_prior_art | 10 | 0 | 0 | 10 |
| store_new_knowledge | 14 | 1 | 0 | 15 |
| update_or_supersede | 0 | 5 | 0 | 5 |
| synthesized_answer | 2 | 2 | 1 | 5 |
| verify_write_landed | 0 | 4 | 0 | 4 |
| semantic_retrieval | 0 | 3 | 0 | 3 |
| ingest_raw_source | 2 | 0 | 0 | 2 |
| discover_capability | 0 | 2 | 0 | 2 |
| **total** | **28** | **17** | **1** | **46** |
> Batch note: several rows collapse a same-second fan-out of identical-intent calls
> (a10=3, a11=4, a14=5, a33=3, a34=3, a35=3). Call-level the corpus is ~62 brain calls;
> the table counts the 46 distinct knowledge-acts. Frequency is deliberately *not* the
> point — the mismatch column is.
**37% of agent knowledge-acts (17/46) are interface mismatches.** Every mismatch falls
into one of four intents: `update_or_supersede`, `verify_write_landed`,
`semantic_retrieval`, `discover_capability` — plus one `store` that had to use HTTP.
## 2. Mismatch list, grouped by intent (primary deliverable)
### update_or_supersede → re-write same slug (5/5 mismatch) — HIGHEST VALUE
There is **no update / patch / append / supersede verb**. When an agent improves a note
it already wrote, the only move is to call `brain_write`/`brain_ingest` **again with the
same filename/source** and hope the index replaces rather than duplicates. Observed:
| slug | 1st write | 2nd write | gap | what changed |
|---|---|---|---|---|
| `tapir-rls-identity-bootstrapping` (ingest) | 14:00:59 | 14:02:08 | 69s | condensed body |
| `homelab-security-chains-not-bugs.md` | 09:28:22 | 10:32:49 | 64m | new worked example |
| `webfetch-readme-when-image-or-flag-uncertain.md` | 18:31:23 | 18:34:23 | 3m | near-identical (retry) |
| `gitea-mcp-per-repo-tools-404-and-rest-fallback` | 05:57:37 | 05:57:55 | 18s | content tweak |
| `cannot-move-ingress-host-across-namespaces-flux-dryrun` | 13:59:50 | 14:00:13 | 23s | content tweak |
Sub-30s gaps (3 of 5) read as "I wanted to edit but can only overwrite." The agent has
no way to know whether the second write deduped or created a contradictory v2 — opacity
the brain's own design principle (`mcp-tool-design-get-needs-list-partner.md`, written
*by one of these very agents*) would flag: every `_write` needs a `_get`/`_update` partner.
### verify_write_landed → lexical re-query (4/4 mismatch)
No read-after-write / get-by-id confirmation. After every substantive write, agents
re-query lexically to check the note is retrievable — and *reformulate the keywords*
because they can't predict what BM25 indexed:
- `exit-255` lesson: query `exit 255 unknown reason restart loop diagnosis` → write →
query `exit 255 unknown SIGKILL containerd`.
- `extension-version-lags`: query `extension build pinned version...pgvector` → write →
query `pgvector postgres extension version compile error bump`.
- `webfetch-readme`: write → write → query stuffed with the note's own section headings.
### semantic_retrieval → BM25 keyword-stuffing (3/3 mismatch)
Agent knows the *meaning* but not the *indexed words*, so it brute-forces phrasings of
one need against a lexical tool:
- **4-step chain on one Go bug:** `bytes.Buffer Bytes Reset aliasing slice sharing`
`go buffer reuse map backing array bug` → (write) → `go map values all show same
content after loop``bytes.Buffer Bytes returns same data every iteration`. Symptom
and cause both known; no single lexical query bridges them.
- Natural-language questions (`how to enable prometheus metrics on litellm proxy
callback`, `moved compose stack to new directory volumes disappeared empty`) shoved
into `brain_query`.
### synthesized_answer → fall back to / hedge with brain_query (2 mismatch + 1 partial)
`brain_answer` is frequently **not trusted as terminal**:
- **Strongest signal:** 3× `brain_answer` at 15:08 (compose volumes / pi rebuild time /
litellm ModuleNotFound) → 3× `brain_query` at 15:09 on the *same three topics*. The
agent asked the synthesizer, was unsatisfied, and immediately re-ran raw search.
- Dex→Authentik: `brain_query` and `brain_answer` fired **8s apart on one question** —
the agent pays both interfaces because it can't predict which returns usable knowledge.
- (Counter-examples exist: `brain_answer` for episodic recall — "what happened with the
litellm migration on 2026-05-16?" — was consumed and not re-queried. So `answer`
works for *episodic/temporal* recall, fails for *how-do-I / does-X-hold* reasoning.)
### discover_capability + store-via-HTTP (3 mismatch)
brain tools are **not ambient** — they are deferred and must be `ToolSearch`-loaded each
session. Agents fumble the discovery (`ToolSearch 'brain knowledge memory'` →
`'brain'` → `'brain ingestion knowledge wiki'`) and, when MCP load/auth fails, drop to
**raw `curl` against `brain-mcp` / hand-built `/tmp/brain_entry.json`**. One agent even
`ToolSearch`-ed an `authenticate` tool, then abandoned MCP for the HTTP bodge — matching
the known "brain/op MCP auth lapses too often" footgun.
### Write-interface / layer schema confusion (cross-cutting)
`brain_write` was called with **three different param shapes** in the same corpus:
`{filename, type:"lesson", content}`, `{filename, content}` (no type), and
`{wing:"tapir", hall:"failures", filename, content}` — plus `brain_ingest {source,
content}`. Agents are unsure which verb and which layer (flat slug vs `wing`/`hall`
knowledge routing vs raw ingest) a given knowledge-act maps to. This is the
`knowledge/ vs wiki/` confusion expressed at the parameter level.
## 3. `intent_unclear` rate
**0 / 46 genuine brain acts (0%).** Agent intent is unusually legible because these are
Claude Code transcripts: the surrounding task, the query/filename strings, and the
write content all disambiguate. One row (`a47`) was tagged `intent_unclear` and
**excluded** — its `brain_` substring was inside a git commit message, not a brain call.
Example of the only ambiguity that arose: a `brain_answer` on "gitea MCP not working...
how to authenticate" followed by an unrelated query — can't tell if the answer satisfied
or was abandoned (`partial`, row a28).
## 4. The single biggest intent↔interface gap
**The brain offers one write verb and one lexical read verb, but autonomous agents
perform four distinct knowledge-acts against them — and three of the four have no fitting
interface.** The deepest gap is the **missing update/supersede path**: agents close every
task by writing a lesson (the CLAUDE.md ritual), routinely improve it minutes-to-an-hour
later, and — having no edit primitive — re-write the same slug blind, unable to tell
whether they corrected the entry or forked a contradiction into the index. This compounds
with the lexical-only read side: because there is no `get-by-id` or semantic retrieval,
agents can't even reliably *find their own just-written note* to check it, so they
keyword-stuff reformulated queries and hedge `brain_answer` with parallel `brain_query`.
The interface is built for *append-and-keyword-search*; the agents are trying to
*curate a living, deduplicated knowledge base*, and the seam between those two shows up
as the 5 blind re-writes, 4 read-after-write re-queries, and 3 semantic-as-lexical chains
that dominate the mismatch column.
---
*Evidence-only per task scope — no redesign proposed.*
@@ -0,0 +1,140 @@
# Brain-MCP Intent↔Interface Findings — Unified (two-column merge)
**Status — 2026-06-16**
-**Agent column** filled from `agent-intent-column.jsonl` (46 acts, koala).
-**Human column** = `PENDING`. Drop the Claude.ai-history analysis into
`human-intent-column.jsonl` (same dir, schema below), then fill the `PENDING`
cells and the synthesis blocks marked `<<SYNTH>>`.
- ⚠️ Canonical `brain-intent-extraction.md` still absent on koala. Vocab below is
the **reconstructed** lock both columns must share. If the real file surfaces,
re-map `intent` labels in *both* columns identically before merging.
---
## Shared schema (LOCKED — both columns conform)
Per-call row, JSONL:
| field | values / form | notes |
|---|---|---|
| `id` | `a01..` (agent) / `h01..` (human) | column prefix kept distinct |
| `session` | string | source session/conversation id |
| `ts` | ISO-8601 | best-effort |
| `tool` | brain tool name (+ `(HTTP-curl)` / `(HTTP-staged)` suffix for bodges) | |
| `intent` | closed vocab ↓ | the knowledge-act WANTED |
| `intent_tool_match` | `match` \| `mismatch` \| `partial` | does the called tool fit the want |
| `consumer_type` | `autonomous_agent` \| `human_interactive` | fixed per column |
| `workaround` | string \| null | the bodge when mismatch — **primary signal** |
| `observed_friction` | string \| null | reformulation chains, discovery tax, hedging |
| `evidence` | string | excerpt anchoring the classification |
| `schema_source` | `reconstructed` | flip to `canonical` if real vocab lands |
### Closed intent vocab (LOCKED)
`semantic_retrieval`, `lexical_lookup`, `check_prior_art`, `synthesized_answer`,
`store_new_knowledge`, `update_or_supersede`, `ingest_raw_source`,
`verify_write_landed`, `discover_capability`, `intent_unclear`.
---
## Master comparison — by intent
| intent | agent acts | agent mismatch | human acts | human mismatch | shared gap |
|---|---|---|---|---|---|
| check_prior_art | 10 | 0% | `PENDING` | `PENDING` | — |
| store_new_knowledge | 15 | 7% (1/15) | `PENDING` | `PENDING` | `<<SYNTH>>` |
| update_or_supersede | 5 | **100%** (5/5) | `PENDING` | `PENDING` | `<<SYNTH>>` no edit verb |
| synthesized_answer | 5 | 40% (2/5)+1 partial | `PENDING` | `PENDING` | `<<SYNTH>>` |
| verify_write_landed | 4 | **100%** (4/4) | `PENDING` | `PENDING` | `<<SYNTH>>` no read-after-write |
| semantic_retrieval | 3 | **100%** (3/3) | `PENDING` | `PENDING` | `<<SYNTH>>` lexical-only read |
| ingest_raw_source | 2 | 0% | `PENDING` | `PENDING` | — |
| discover_capability | 2 | **100%** (2/2) | `PENDING` | `PENDING` | agent-specific (ToolSearch/auth)? |
| intent_unclear | 0 | — | `PENDING` | `PENDING` | divergence expected ↓ |
| **TOTAL** | **46** | **37% (17)** | `PENDING` | `PENDING` | |
---
## Per-intent merged findings
### update_or_supersede — agent: 5/5 mismatch (highest value)
**Agent:** no edit/patch/append verb. Agents re-write same slug blind:
`homelab-security-chains-not-bugs.md` (+64m), `tapir-rls-identity-bootstrapping`,
`webfetch-readme...`, `gitea-mcp-per-repo-tools-404...`,
`cannot-move-ingress-host...` — 3 of 5 sub-30s ("wanted edit, got overwrite").
Cannot tell if write deduped or forked a contradiction.
**Human:** `PENDING` — *look for: user editing a prior note, asking "update what I
saved about X", or expressing frustration that an old fact is stale/duplicated.*
**<<SYNTH>>** shared verdict once both filled.
### verify_write_landed — agent: 4/4 mismatch
**Agent:** no `get-by-id`/read-after-write. Agents lexically re-query their own
fresh note with reformulated keywords (`exit 255 unknown reason``...SIGKILL
containerd`; `extension build pinned...``pgvector ...compile error bump`).
**Human:** `PENDING` — *humans may not exhibit this (they trust the write UI
confirmation). If absent in human column, it's an agent-specific gap → flag.*
**<<SYNTH>>**.
### semantic_retrieval — agent: 3/3 mismatch
**Agent:** meaning known, indexed words unknown → BM25 keyword-stuffing. 4-step
chain on one Go `bytes.Buffer` bug; NL questions shoved into `brain_query`.
**Human:** `PENDING` — *humans likely hit this HARDER (they phrase conversationally).
Compare reformulation-chain length agent vs human.*
**<<SYNTH>>** — likely the strongest cross-consumer overlap.
### synthesized_answer — agent: 2 mismatch + 1 partial
**Agent:** `brain_answer` not trusted terminal — 3 answers → 3 same-topic queries
1min later; query+answer fired 8s apart hedging one need. Works for *episodic*
recall, fails for *how-do-I / does-X-hold*.
**Human:** `PENDING` — *humans may prefer `brain_answer` as primary (chat-native).
If human match-rate >> agent, the tool fits humans not agents → key divergence.*
**<<SYNTH>>**.
### store_new_knowledge — agent: 14/15 match
**Agent:** healthy, except 1 HTTP-staged bodge when MCP auth lapsed. Also surfaced
write-schema confusion: 3 param shapes (`{filename,type}` / `{filename}` /
`{wing,hall,filename}`) + `ingest{source}`.
**Human:** `PENDING`*humans rarely write directly; expect low volume.*
**<<SYNTH>>**.
### check_prior_art / ingest_raw_source — agent: 0% mismatch
Lexical fits named-entity recall and raw-source capture. **Human:** `PENDING`.
### discover_capability — agent: 2/2 mismatch (agent-specific)
Brain tools deferred → `ToolSearch`-load each session; auth lapse → `curl` bodge.
**Likely has NO human analog** (humans get ambient connectors). Candidate for
"agent-only gap" bucket. **Human:** `PENDING` to confirm absent.
---
## Cross-consumer divergence — questions to resolve at merge
1. **intent_unclear rate.** Agent = 0% (transcripts self-document). Human expected
higher (conversational, implicit). Big delta = the columns measure legibility
differently, not just intent.
2. **Where does each consumer's mismatch concentrate?** Agent mismatch is
write-side-heavy (supersede + verify-landed = 9/17). Hypothesis: human mismatch
is read-side-heavy (semantic + answer). If true → **the interface fails the two
consumers at opposite ends.**
3. **Agent-only gaps** (`discover_capability`, `verify_write_landed`) vs
**shared gaps** (`semantic_retrieval`, `update_or_supersede`). Shared gaps =
highest-priority evidence; agent-only = harness/auth issues.
---
## Combined headline — `<<SYNTH>>` (fill when human column lands)
> Agent-side draft (to be reconciled with human-side):
> Brain = append + keyword-search; agents want a curated, dedup'd, self-verifying KB.
> Missing update/supersede path + lexical-only reads are the seam. **Open question
> for the merge: do humans hit the same read-side wall, making semantic-retrieval the
> universal gap — or do agents uniquely suffer the write-side (supersede / verify)
> wall that humans sidestep via the chat UI?**
---
## Drop-in checklist (when human column arrives)
1. Place `human-intent-column.jsonl` in this dir; conform to LOCKED schema.
2. Fill every `PENDING` cell in master table + per-intent blocks.
3. Resolve the 3 divergence questions with evidence.
4. Replace each `<<SYNTH>>` with the reconciled verdict; write the combined headline.
5. If canonical vocab surfaced: re-map both columns' `intent`, flip `schema_source`.
6. Commit as `docs(brain): merge human+agent intent columns`.
+8 -9
View File
@@ -112,16 +112,15 @@ Flags:
- `--out PATH` — output file (default `./.mcp.json`) - `--out PATH` — output file (default `./.mcp.json`)
- `--force` — overwrite an existing file - `--force` — overwrite an existing file
Modes: Modes (all list **brain + gitea** — Gitea is the audit-trail invariant of the
consolidated single harness, #75):
- **cloud** — brain MCP only. Claude Code with no routing. - **cloud** — brain + gitea MCP.
- **client-local** — brain + routing pod. The `routing` entry points at - **client-local** — brain + gitea MCP. (The former `routing` entry pointing at
`koala:30310/mcp` (the routing pod, deployed in Plan 6). The `koala:30310/mcp` was removed — the routing pod is iceboxed, #75.)
`X-Hyperguild-Mode: client-local` header is forward-compat for future - **sovereign** — brain + gitea, with a `_mode_note` explaining that this mode
modes; the pod treats absent or unknown values as `client-local`. primarily uses Crush + LiteLLM and the `.mcp.json` is a Claude Code fallback
- **sovereign** — brain only, with a `_mode_note` explaining that this for emergency offline use.
mode primarily uses Crush + LiteLLM and the `.mcp.json` is a Claude
Code fallback for emergency offline use.
## Environment ## Environment
+26 -19
View File
@@ -59,31 +59,40 @@ func runMode(ctx context.Context, args []string, _ io.Reader, stdout, stderr io.
return nil return nil
} }
// giteaMCPURL is the public Gitea MCP endpoint (OAuth via Dex/Authentik). Gitea
// is the audit-trail invariant of the consolidated single harness (#75), so every
// mode lists it as an available connection.
const giteaMCPURL = "https://git-mcp.d-ma.be/mcp"
func brainEntry(brainURL string) map[string]any {
return map[string]any{
"url": brainURL + "/mcp",
"description": "Brain MCP — knowledge query, write, ingestion, session log",
}
}
func giteaEntry() map[string]any {
return map[string]any{
"url": giteaMCPURL,
"description": "Gitea MCP — issues/PRs/repo ops (audit-trail invariant)",
}
}
func modeCloud(brainURL string) map[string]any { func modeCloud(brainURL string) map[string]any {
return map[string]any{ return map[string]any{
"mcpServers": map[string]any{ "mcpServers": map[string]any{
"brain": map[string]any{ "brain": brainEntry(brainURL),
"url": brainURL + "/mcp", "gitea": giteaEntry(),
"description": "Brain MCP — knowledge query, write, ingestion, session log",
},
}, },
} }
} }
func modeClientLocal(brainURL string) map[string]any { func modeClientLocal(brainURL string) map[string]any {
// The routing pod is iceboxed (#75); the consolidated harness is brain + gitea.
return map[string]any{ return map[string]any{
"mcpServers": map[string]any{ "mcpServers": map[string]any{
"brain": map[string]any{ "brain": brainEntry(brainURL),
"url": brainURL + "/mcp", "gitea": giteaEntry(),
"description": "Brain MCP — knowledge query, write, ingestion, session log",
},
"routing": map[string]any{
"url": "http://koala:30310/mcp",
"description": "Mode 2 routing pod — routes skill calls to LiteLLM/local",
"headers": map[string]any{
"X-Hyperguild-Mode": "client-local",
},
},
}, },
} }
} }
@@ -92,10 +101,8 @@ func modeSovereign(brainURL string) map[string]any {
return map[string]any{ return map[string]any{
"_mode_note": "Sovereign mode primarily uses Crush + LiteLLM. This .mcp.json is provided as Claude Code fallback (e.g. emergency offline editing).", "_mode_note": "Sovereign mode primarily uses Crush + LiteLLM. This .mcp.json is provided as Claude Code fallback (e.g. emergency offline editing).",
"mcpServers": map[string]any{ "mcpServers": map[string]any{
"brain": map[string]any{ "brain": brainEntry(brainURL),
"url": brainURL + "/mcp", "gitea": giteaEntry(),
"description": "Brain MCP — knowledge query, write, ingestion, session log",
},
}, },
} }
} }
+11 -17
View File
@@ -35,11 +35,13 @@ func TestRunMode_Cloud_Default(t *testing.T) {
servers, ok := got["mcpServers"].(map[string]any) servers, ok := got["mcpServers"].(map[string]any)
require.True(t, ok, "mcpServers must be a JSON object") require.True(t, ok, "mcpServers must be a JSON object")
assert.Contains(t, servers, "brain") assert.Contains(t, servers, "brain")
assert.Contains(t, servers, "gitea")
assert.NotContains(t, servers, "routing") assert.NotContains(t, servers, "routing")
assert.NotContains(t, got, "_mode_note") assert.NotContains(t, got, "_mode_note")
} }
func TestRunMode_ClientLocal_HasRoutingEntry(t *testing.T) { func TestRunMode_ClientLocal_NoRouting_HasGitea(t *testing.T) {
// #75: the routing pod is iceboxed; the consolidated harness is brain + gitea.
dir := t.TempDir() dir := t.TempDir()
outPath := filepath.Join(dir, ".mcp.json") outPath := filepath.Join(dir, ".mcp.json")
t.Setenv("BRAIN_URL", "http://koala:30330") t.Setenv("BRAIN_URL", "http://koala:30330")
@@ -51,17 +53,11 @@ func TestRunMode_ClientLocal_HasRoutingEntry(t *testing.T) {
got := readJSON(t, outPath) got := readJSON(t, outPath)
servers := got["mcpServers"].(map[string]any) servers := got["mcpServers"].(map[string]any)
require.Contains(t, servers, "brain") require.Contains(t, servers, "brain")
require.Contains(t, servers, "routing") require.Contains(t, servers, "gitea")
assert.NotContains(t, servers, "routing", "routing pod iceboxed (#75)")
routing := servers["routing"].(map[string]any)
assert.NotContains(t, routing, "_routing_pending", "placeholder should be removed once Plan 6 ships")
headers, ok := routing["headers"].(map[string]any)
require.True(t, ok, "routing entry should have headers block")
assert.Equal(t, "client-local", headers["X-Hyperguild-Mode"])
} }
func TestModeClientLocalHasRoutingHeader(t *testing.T) { func TestModeGiteaEntryPresentAndWellFormed(t *testing.T) {
tmp := t.TempDir() + "/mcp.json" tmp := t.TempDir() + "/mcp.json"
out := &bytes.Buffer{} out := &bytes.Buffer{}
stderr := &bytes.Buffer{} stderr := &bytes.Buffer{}
@@ -73,13 +69,10 @@ func TestModeClientLocalHasRoutingHeader(t *testing.T) {
require.NoError(t, json.Unmarshal(body, &doc)) require.NoError(t, json.Unmarshal(body, &doc))
servers := doc["mcpServers"].(map[string]any) servers := doc["mcpServers"].(map[string]any)
routing := servers["routing"].(map[string]any) require.NotContains(t, servers, "routing")
assert.Equal(t, "http://koala:30310/mcp", routing["url"]) gitea, ok := servers["gitea"].(map[string]any)
assert.NotContains(t, routing, "_routing_pending", "placeholder should be removed once Plan 6 ships") require.True(t, ok, "gitea entry must be present (audit-trail invariant)")
assert.Equal(t, "https://git-mcp.d-ma.be/mcp", gitea["url"])
headers, ok := routing["headers"].(map[string]any)
require.True(t, ok, "routing entry should have headers block")
assert.Equal(t, "client-local", headers["X-Hyperguild-Mode"])
} }
func TestRunMode_Sovereign_HasModeNote(t *testing.T) { func TestRunMode_Sovereign_HasModeNote(t *testing.T) {
@@ -94,6 +87,7 @@ func TestRunMode_Sovereign_HasModeNote(t *testing.T) {
assert.Contains(t, got, "_mode_note") assert.Contains(t, got, "_mode_note")
servers := got["mcpServers"].(map[string]any) servers := got["mcpServers"].(map[string]any)
assert.Contains(t, servers, "brain") assert.Contains(t, servers, "brain")
assert.Contains(t, servers, "gitea")
assert.NotContains(t, servers, "routing") assert.NotContains(t, servers, "routing")
} }
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"io" "io"
"os" "os"
"github.com/mathiasbq/supervisor/internal/tier" "git.d-ma.be/mathias/hyperguild/internal/tier"
) )
const defaultAnthropicProbe = "https://api.anthropic.com" const defaultAnthropicProbe = "https://api.anthropic.com"
-170
View File
@@ -1,170 +0,0 @@
package main
// The internal/skills/{debug,retrospective,review,trainer} packages imported
// below are also imported by cmd/supervisor. Plan 7 (supervisor retirement)
// MUST NOT delete these four packages — the routing pod is their second
// consumer. Plan 7 deletes only internal/skills/{tdd,spec,tier} (the skills
// that don't route to local), the supervisor binary, and supervisor manifests.
// See docs/superpowers/specs/2026-05-04-mode-2-routing-pod-design.md (Constraints).
import (
"context"
"log/slog"
"net/http"
"os"
"time"
"github.com/mathiasbq/supervisor/internal/auth"
"github.com/mathiasbq/supervisor/internal/config"
iexec "github.com/mathiasbq/supervisor/internal/exec"
"github.com/mathiasbq/supervisor/internal/githubclient"
"github.com/mathiasbq/supervisor/internal/mcp"
"github.com/mathiasbq/supervisor/internal/mcpclient"
"github.com/mathiasbq/supervisor/internal/registry"
"github.com/mathiasbq/supervisor/internal/routing"
"github.com/mathiasbq/supervisor/internal/skills/debug"
"github.com/mathiasbq/supervisor/internal/skills/project"
"github.com/mathiasbq/supervisor/internal/skills/retrospective"
"github.com/mathiasbq/supervisor/internal/skills/review"
"github.com/mathiasbq/supervisor/internal/skills/trainer"
)
func main() {
logger := slog.New(slog.NewTextHandler(os.Stderr, nil))
slog.SetDefault(logger)
cfg, err := config.LoadRouting()
if err != nil {
logger.Error("config load failed", "err", err)
os.Exit(1)
}
configDir := envOr("SUPERVISOR_CONFIG_DIR", "/app/config/supervisor")
mustRead := func(path string) string {
b, err := os.ReadFile(configDir + "/" + path)
if err != nil {
logger.Error("read prompt failed", "path", path, "err", err)
os.Exit(1)
}
return string(b)
}
llm := iexec.NewLiteLLM(cfg.LiteLLMBaseURL, cfg.LiteLLMAPIKey, 0)
router := &routing.Router{
Fetcher: routing.NewFetcher(cfg.BrainURL, "7d", time.Duration(cfg.PassRateTTLSeconds)*time.Second),
Logger: routing.NewLogger(cfg.BrainURL),
Policy: routing.Policy{Floor: cfg.RouteLocalFloor, Ceil: cfg.RouteLocalCeil},
FastModel: cfg.FastModel,
ThinkingModel: cfg.ThinkingModel,
Complete: llm.Complete,
}
// Skill packages call CompleteFunc(ctx, model, system, user) — no session_id
// or project_root in the signature. Rather than modifying every skill's API
// (and inflating Plan 6's blast radius), the routing pod logs every decision
// under a fixed session_id "_routing". Operators query
// `GET /pass-rate?skill=_routing&window=...` to inspect routing health.
const routingSessionID = "_routing"
wrap := func(skillName string) routing.CompleteFunc {
return func(ctx context.Context, _, system, user string) (string, int64, error) {
// The model param is ignored: the router picks the model based on policy.
return router.Run(ctx, routing.RunInput{
Skill: skillName,
System: system,
User: user,
SessionID: routingSessionID,
ProjectRoot: "",
})
}
}
reg := registry.New()
reg.Register(review.New(review.Config{
SkillPrompt: mustRead("review.md"),
DefaultModel: cfg.FastModel,
CompleteFunc: review.CompleteFunc(wrap("review")),
}))
reg.Register(debug.New(debug.Config{
SkillPrompt: mustRead("debug.md"),
DefaultModel: cfg.FastModel,
CompleteFunc: debug.CompleteFunc(wrap("debug")),
}))
reg.Register(retrospective.New(retrospective.Config{
SkillPrompt: mustRead("retrospective.md"),
DefaultModel: cfg.FastModel,
CompleteFunc: retrospective.CompleteFunc(wrap("retrospective")),
}))
reg.Register(trainer.New(trainer.Config{
ReaderPrompt: mustRead("trainer-reader.md"),
WriterPrompt: mustRead("trainer-writer.md"),
DefaultModel: cfg.FastModel,
CompleteFunc: trainer.CompleteFunc(wrap("trainer")),
}))
if cfg.GiteaMCPURL != "" {
mcpC, err := mcpclient.New(cfg.GiteaMCPURL, cfg.GiteaMCPToken)
if err != nil {
logger.Error("mcpclient init for project_create — GITEA_MCP_URL is set but GITEA_MCP_TOKEN is empty (check routing-secrets)", "err", err)
os.Exit(1)
}
var ghClient *githubclient.Client
if cfg.GitHubPAT != "" {
ghClient = githubclient.New(cfg.GitHubPAT)
}
reg.Register(project.New(project.Config{
Client: mcpC,
GitHub: ghClient,
GiteaOwner: cfg.GiteaOwner,
GitHubOwner: cfg.GitHubOwner,
GitHubPAT: cfg.GitHubPAT,
InfraRepo: cfg.InfraRepo,
}))
logger.Info("project_create registered", "gitea_mcp_url", cfg.GiteaMCPURL,
"gitea_owner", cfg.GiteaOwner, "github_owner", cfg.GitHubOwner,
"infra_repo", cfg.InfraRepo, "github_pat_set", cfg.GitHubPAT != "")
} else {
logger.Info("project_create skipped — GITEA_MCP_URL not set")
}
var validator *auth.Validator
if dexURL := os.Getenv("DEX_ISSUER_URL"); dexURL != "" {
audience := os.Getenv("MCP_AUDIENCE")
v, err := auth.NewValidator(dexURL, audience)
if err != nil {
logger.Error("build jwt validator", "err", err)
os.Exit(1)
}
validator = v
logger.Info("jwt auth enabled", "issuer", dexURL)
}
srv := mcp.NewServer(reg, cfg.MCPAuthToken, validator)
mux := http.NewServeMux()
mux.Handle("/mcp", srv)
mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
})
if dexURL := os.Getenv("DEX_ISSUER_URL"); dexURL != "" {
resourceURL := os.Getenv("MCP_RESOURCE_URL")
mux.HandleFunc("GET /.well-known/oauth-protected-resource",
auth.ProtectedResourceHandler(resourceURL, dexURL))
}
addr := ":" + cfg.Port
logger.Info("routing pod starting", "addr", addr,
"fast", cfg.FastModel, "thinking", cfg.ThinkingModel,
"floor", cfg.RouteLocalFloor, "ceil", cfg.RouteLocalCeil)
if err := http.ListenAndServe(addr, mux); err != nil { //nolint:gosec
logger.Error("server stopped", "err", err)
os.Exit(1)
}
}
func envOr(key, def string) string {
if v := os.Getenv(key); v != "" {
return v
}
return def
}
-135
View File
@@ -1,135 +0,0 @@
package main_test
import (
"context"
"encoding/json"
"io"
"net"
"net/http"
"net/http/httptest"
"os"
"os/exec"
"strconv"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// TestRoutingPodEndToEnd boots the binary against fake LiteLLM + brain servers,
// calls tools/list and one tools/call, and verifies the brain saw a session_log POST.
func TestRoutingPodEndToEnd(t *testing.T) {
if testing.Short() {
t.Skip("end-to-end binary boot")
}
var brainHits int
llm := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_ = json.NewEncoder(w).Encode(map[string]any{
"choices": []map[string]any{{"message": map[string]any{"role": "assistant", "content": "stub"}}},
})
}))
defer llm.Close()
brain := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/pass-rate":
brainHits++
_ = json.NewEncoder(w).Encode(map[string]any{"pass_rate": 0.95})
case "/mcp":
brainHits++
_ = json.NewEncoder(w).Encode(map[string]any{"jsonrpc": "2.0", "id": 1, "result": map[string]any{}})
}
}))
defer brain.Close()
port := freePort(t)
addr := "127.0.0.1:" + port
baseURL := "http://" + addr
bin := buildRouting(t)
cmd := exec.Command(bin)
cmd.Env = []string{
"ROUTING_PORT=" + port,
"LITELLM_BASE_URL=" + llm.URL,
"LITELLM_API_KEY=stub",
"BRAIN_URL=" + brain.URL,
"SUPERVISOR_CONFIG_DIR=../../config/supervisor",
"PATH=" + os.Getenv("PATH"),
"HOME=" + os.Getenv("HOME"),
}
require.NoError(t, cmd.Start())
t.Cleanup(func() { _ = cmd.Process.Kill() })
require.NoError(t, waitForPort(t, addr, 30*time.Second))
resp := mcpCall(t, baseURL+"/mcp", `{"jsonrpc":"2.0","id":1,"method":"tools/list"}`)
assert.Contains(t, resp, `"review"`)
assert.Contains(t, resp, `"debug"`)
assert.Contains(t, resp, `"retrospective"`)
assert.Contains(t, resp, `"trainer"`)
resp = mcpCall(t, baseURL+"/mcp", `{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"review","arguments":{"project_root":"/tmp","files":["README.md"]}}}`)
_ = resp // shape varies by skill; we only need a 200
// Wait briefly for the async session_log to land.
deadline := time.Now().Add(2 * time.Second)
for time.Now().Before(deadline) && brainHits < 2 {
time.Sleep(50 * time.Millisecond)
}
assert.GreaterOrEqual(t, brainHits, 2, "expected at least one /pass-rate hit and one /mcp session_log hit")
}
func buildRouting(t *testing.T) string {
t.Helper()
bin := t.TempDir() + "/routing"
out, err := exec.Command("go", "build", "-o", bin, "github.com/mathiasbq/supervisor/cmd/routing").CombinedOutput()
require.NoError(t, err, "build failed: %s", out)
return bin
}
func waitForPort(_ *testing.T, addr string, dur time.Duration) error {
deadline := time.Now().Add(dur)
for time.Now().Before(deadline) {
c, err := http.Get("http://" + addr + "/healthz") //nolint:noctx
if err == nil {
_ = c.Body.Close()
return nil
}
conn, err := http.NewRequest(http.MethodPost, "http://"+addr+"/mcp", strings.NewReader(`{}`))
if err == nil {
r, err := http.DefaultClient.Do(conn)
if err == nil {
_ = r.Body.Close()
return nil
}
}
time.Sleep(50 * time.Millisecond)
}
return context.DeadlineExceeded
}
func mcpCall(t *testing.T, url, body string) string {
t.Helper()
r, err := http.Post(url, "application/json", strings.NewReader(body)) //nolint:noctx
require.NoError(t, err)
defer func() { _ = r.Body.Close() }()
raw, err := io.ReadAll(r.Body)
require.NoError(t, err)
return string(raw)
}
// freePort grabs an OS-assigned TCP port and releases it. There is a small
// race window before the subprocess re-binds it, but it is acceptable for
// test isolation against a hardcoded port colliding with another test or
// stray process.
func freePort(t *testing.T) string {
t.Helper()
l, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err)
port := l.Addr().(*net.TCPAddr).Port
require.NoError(t, l.Close())
return strconv.Itoa(port)
}
@@ -0,0 +1,104 @@
# Runbook: exercising review/debug traffic to fill the pass-rate dataset
**Why this exists:** the routing pod's local-vs-cloud decision is gated on a
pass-rate history that only accrues from real `review`/`debug` invocations
**through the pod**. Until the dataset has data, the fast (local) path never
activates and the core hypothesis (hyperguild #35) can't be validated. This
runbook is how you spin that flywheel.
## The one trap
Pass-rate accrues **only** when a skill tool is called via the routing pod's MCP
endpoint. These look like they should count but **do not**:
- **Crush** — talks to LiteLLM directly, bypasses the pod. No log.
- **claude.ai web / Claude Desktop without the connector** — no log.
- **Running the local `code-review` / `debug` skills** (`~/dev/.skills`) inline in
a Claude Code session — those are local skills, not the pod's MCP tools. No log.
Only a `tools/call` to the routing pod records a pass/fail.
## Endpoints
| Purpose | URL | Auth |
|---------|-----|------|
| Routing MCP (local, Tailscale) | `http://koala:30310/mcp` | Bearer `ROUTING_MCP_TOKEN` |
| Routing MCP (remote) | `https://routing-mcp.d-ma.be/mcp` | OAuth via `auth.d-ma.be` (audience `claude-ai`) |
| Pass-rate readout | `http://koala:30330/pass-rate?skill=<name>` | none (read-only) |
Tools advertised: **`review`**, **`debug`** (the two the #35 gate measures),
plus `session_log`, `retrospective`, `trainer`.
## Step 1 — connect the routing pod as an MCP server
**Local** (needs the bearer token; keep it out of argv via 1Password):
```bash
op run --env-file ~/.op-env -- \
claude mcp add routing --transport http http://koala:30310/mcp \
--header "Authorization: Bearer $ROUTING_MCP_TOKEN"
```
**Remote** (claude.ai / Claude Desktop): add a custom connector pointing at
`https://routing-mcp.d-ma.be/mcp`; it completes OAuth against `auth.d-ma.be`,
no static token.
Verify: a `tools/list` should return `review`, `debug`, `session_log`,
`retrospective`, `trainer`.
## Step 2 — route real work through it
In normal sessions, invoke the pod's tools instead of reviewing/debugging inline:
- *"Use the **routing** `review` tool on this diff."*
- *"**debug** this failure through the routing pod."*
Each call logs an outcome to ingestion → `/pass-rate` ticks up.
## Step 3 — how routing actually picks the model
Per `internal/routing/policy.go`:
1. pass-rate `nil` (cold) → **local** fast tier. The router defaults to local
from invocation #1, not to cloud — so the fast tier is exercised immediately.
2. pass-rate `>= 0.90` (floor) → **local**; `< 0.70` (ceil) → **cloud/thinking**;
in the `[0.70, 0.90)` band a request-hash bit samples 50/50.
3. On a local execution error the router falls open to the thinking model for
that one call (logged `thinking_fallback`).
So you are not "paying in on cloud" — cold calls already run on the (validated)
local fast tier **`koala/qwen36-35b-a3b`** (Qwen3.6-35B-A3B MTP, promoted
2026-06-29, infra `c66a195`, `HYPERGUILD_FAST_MODEL`). Accumulating passes just
keeps it there once real pass-rate is computed.
> **Instrumentation note (#73, fixed 2026-06-30):** until v0.11.1 the pod logged
> successes as `"skip"` (not `"pass"`), under `skill:"_routing"`, via an
> unauthenticated POST that silently 401'd — so `/pass-rate` stayed at zero no
> matter how much you used it. That's fixed and verified (a real review call now
> moves `/pass-rate?skill=review` 0→1). If you see traffic not registering,
> re-check #73's three failure modes first.
## Target & verification
- **50 logged invocations** across `review` + `debug` within the 14-day window.
The clock restarts **2026-06-30** (the day instrumentation was verified working;
the original 2026-06-26→07-10 window measured broken plumbing) → **kill-date
2026-07-14**, ~4 calls/day (1 already logged from the #73 smoke test).
- Check progress anytime:
```bash
curl -s "http://koala:30330/pass-rate?skill=review"
curl -s "http://koala:30330/pass-rate?skill=debug"
```
- If ~45/day isn't realistic alongside Crush, that is **not** a failure — per
#35 deliverable #1 it's the signal hyperguild isn't on the work critical path,
and the pre-decided **Berget fallback** (`gpt-oss-120b` / `qwen3-32b`) carries
the fast tier instead.
## Refs
- hyperguild #35 — the validation issue (data gate = deliverable #1)
- `docs/multi-model-routing.md` — routing policy
- brain: `wiki/homelab/hypotheses/qwen36-35b-a3b-fast-model-experiment-2026-05-28.md`
- infra `c66a195` — qwen36 promotion; `models.yml` / `llama-swap-configmap.yaml`
+1 -1
View File
@@ -1,4 +1,4 @@
module github.com/mathiasbq/supervisor module git.d-ma.be/mathias/hyperguild
go 1.26.1 go 1.26.1
+152 -14
View File
@@ -8,15 +8,21 @@ import (
"net/http" "net/http"
"net/url" "net/url"
"os" "os"
"path/filepath"
"strconv" "strconv"
"strings" "strings"
"time" "time"
chassisauth "gitea.d-ma.be/mathias/mcp-chassis/auth" chassisauth "git.d-ma.be/mathias/mcp-chassis/auth"
"github.com/mathiasbq/hyperguild/ingestion/internal/api" "github.com/mathiasbq/hyperguild/ingestion/internal/api"
"github.com/mathiasbq/hyperguild/ingestion/internal/audit"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
"github.com/mathiasbq/hyperguild/ingestion/internal/capturehttp"
"github.com/mathiasbq/hyperguild/ingestion/internal/classification"
"github.com/mathiasbq/hyperguild/ingestion/internal/claudewatcher" "github.com/mathiasbq/hyperguild/ingestion/internal/claudewatcher"
"github.com/mathiasbq/hyperguild/ingestion/internal/embed" "github.com/mathiasbq/hyperguild/ingestion/internal/embed"
"github.com/mathiasbq/hyperguild/ingestion/internal/gitea"
"github.com/mathiasbq/hyperguild/ingestion/internal/graphstore" "github.com/mathiasbq/hyperguild/ingestion/internal/graphstore"
"github.com/mathiasbq/hyperguild/ingestion/internal/graphsync" "github.com/mathiasbq/hyperguild/ingestion/internal/graphsync"
"github.com/mathiasbq/hyperguild/ingestion/internal/llm" "github.com/mathiasbq/hyperguild/ingestion/internal/llm"
@@ -28,12 +34,33 @@ import (
"github.com/mathiasbq/hyperguild/ingestion/internal/search" "github.com/mathiasbq/hyperguild/ingestion/internal/search"
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore" "github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
"github.com/mathiasbq/hyperguild/ingestion/internal/watcher" "github.com/mathiasbq/hyperguild/ingestion/internal/watcher"
"github.com/mathiasbq/hyperguild/ingestion/internal/webhook"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/rest"
"k8s.io/client-go/tools/clientcmd"
) )
// claudeSink converts each claudewatcher.Batch into one wiki note under // kubeClient builds an in-cluster Kubernetes client (falls back to
// brain/wiki/claude-sessions/facts/. v1 emits one note per session // $KUBECONFIG for local dev/testing against a real cluster).
// keyed by host + session id; classifier-driven hall routing is a func kubeClient() (kubernetes.Interface, error) {
// follow-up (hyperguild#27 v2). if cfg, err := rest.InClusterConfig(); err == nil {
return kubernetes.NewForConfig(cfg)
}
rules := clientcmd.NewDefaultClientConfigLoadingRules()
cc := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(rules, &clientcmd.ConfigOverrides{})
cfg, err := cc.ClientConfig()
if err != nil {
return nil, fmt.Errorf("kube config (no in-cluster, no kubeconfig): %w", err)
}
return kubernetes.NewForConfig(cfg)
}
// claudeSink converts each claudewatcher.Batch into a raw session dump
// under brain/archive/claude-sessions/<host>/. Deliberately NOT a wiki
// note (api.WriteNote / brain/wiki/) — raw full transcripts out-ranked
// curated ai-sessions summaries in BM25 (177,818 vs 45,335 on the same
// query) and duplicated content already summarized elsewhere. Kept for
// deep lookups, never indexed. See ai-sessions#10.
type claudeSink struct { type claudeSink struct {
brainDir string brainDir string
logger *slog.Logger logger *slog.Logger
@@ -61,16 +88,14 @@ func (s *claudeSink) Ingest(ctx context.Context, b claudewatcher.Batch) error {
sb.WriteString("\n\n") sb.WriteString("\n\n")
} }
slug := "session-" + b.Host + "-" + b.SessionID slug := "session-" + b.Host + "-" + b.SessionID
if _, err := api.WriteNote(s.brainDir, api.WriteNoteOptions{ dest := filepath.Join(s.brainDir, "archive", "claude-sessions", b.Host, slug+".md")
Filename: slug, if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
Wing: "claude-sessions", return fmt.Errorf("create claude-sessions archive dir: %w", err)
Hall: "facts",
Type: "source",
Domain: b.ProjectID,
Content: sb.String(),
}); err != nil {
return fmt.Errorf("write claude session note: %w", err)
} }
if err := os.WriteFile(dest, []byte(sb.String()), 0o644); err != nil {
return fmt.Errorf("write claude session archive: %w", err)
}
s.logger.Debug("claude session archived (non-indexed)", "path", dest)
return nil return nil
} }
@@ -118,6 +143,47 @@ func envInt(key string, fallback int) int {
return fallback return fallback
} }
// buildAuditSink selects the capture audit sink. When BRAIN_LOKI_URL is
// set it builds the classification-aware DegradingSink (loki central +
// durable file buffer + optional ntfy) and starts the reconcile loop;
// otherwise it falls back to a plain slog sink. The buffer lives under the
// brain dir so it survives process restarts.
func buildAuditSink(ctx context.Context, brainDir string, logger *slog.Logger) capture.AuditSink {
lokiURL := os.Getenv("BRAIN_LOKI_URL")
central := audit.NewLokiCentral(lokiURL)
if central == nil {
logger.Info("capture audit: slog sink (BRAIN_LOKI_URL unset)")
return audit.NewSlogSink(logger)
}
buffer, err := audit.NewFileBuffer(filepath.Join(brainDir, ".audit-buffer", "capture.jsonl"))
if err != nil {
logger.Error("capture audit buffer init", "err", err)
os.Exit(1)
}
// Keep notifier as a nil interface (not a typed-nil) when unconfigured
// so DegradingSink/Reconcile skip it cleanly.
var notifier audit.Notifier
if n := audit.NewNtfyNotifier(os.Getenv("BRAIN_NTFY_URL"), os.Getenv("BRAIN_NTFY_TOKEN")); n != nil {
notifier = n
}
reconcileInterval := time.Duration(envInt("BRAIN_AUDIT_RECONCILE_INTERVAL", 60)) * time.Second
audit.StartReconcile(ctx, central, buffer, notifier, reconcileInterval)
logger.Info("capture audit: loki+buffer sink", "loki", lokiURL, "reconcile_s", int(reconcileInterval.Seconds()))
return audit.NewDegradingSink(central, buffer, notifier)
}
// splitList parses a comma-separated env value into a trimmed,
// empty-free slice. Used for the capture sovereign-principal allowlist.
func splitList(v string) []string {
var out []string
for _, p := range strings.Split(v, ",") {
if p = strings.TrimSpace(p); p != "" {
out = append(out, p)
}
}
return out
}
// systemHostname returns os.Hostname() with a "unknown" fallback so the // systemHostname returns os.Hostname() with a "unknown" fallback so the
// caller never has to handle the rare error path. // caller never has to handle the rare error path.
func systemHostname() string { func systemHostname() string {
@@ -175,6 +241,15 @@ func main() {
logger.Info("brain reranker configured", "url", rerankURL, "model", rerankModel) logger.Info("brain reranker configured", "url", rerankURL, "model", rerankModel)
} }
// Gitea ticket tracker for the capture capability (#52). Token via env
// only — never logged or in argv. Both vars must be set to enable it;
// gitea.New returns nil otherwise, leaving ticket integration off.
giteaURL := envOr("BRAIN_GITEA_URL", "https://git.d-ma.be")
if tracker := gitea.New(giteaURL, os.Getenv("BRAIN_GITEA_TOKEN")); tracker != nil {
mcpSrv = mcpSrv.WithIssueTracker(tracker)
logger.Info("brain gitea tracker configured", "url", giteaURL)
}
// Hybrid retrieval (pgvector + nomic-embed-text). Both env vars must // Hybrid retrieval (pgvector + nomic-embed-text). Both env vars must
// be set together for the path to wire on; otherwise BM25-only. // be set together for the path to wire on; otherwise BM25-only.
var vectorStore *vectorstore.PGStore var vectorStore *vectorstore.PGStore
@@ -296,6 +371,29 @@ func main() {
logger.Info("claudewatcher started", logger.Info("claudewatcher started",
"sessions_dir", claudeDir, "host", host, "interval", interval) "sessions_dir", claudeDir, "host", host, "interval", interval)
} }
// Gitea push webhook -> on-demand brain-sync Job, instead of waiting up
// to 15 minutes for the next CronJob poll. Off by default (opt in via
// GITEA_WEBHOOK_SECRET) since it needs Job-create RBAC in the "brain"
// namespace that a fresh deploy won't have granted yet.
var webhookHandler *webhook.Handler
if webhookSecret := os.Getenv("GITEA_WEBHOOK_SECRET"); webhookSecret != "" {
kc, err := kubeClient()
if err != nil {
logger.Error("brain-sync webhook: kube client", "err", err)
os.Exit(1)
}
webhookHandler = &webhook.Handler{
Secret: webhookSecret,
Clientset: kc,
Namespace: envOr("BRAIN_SYNC_NAMESPACE", "brain"),
CronJobName: envOr("BRAIN_SYNC_CRONJOB", "brain-sync"),
WatchRepo: envOr("BRAIN_SYNC_WATCH_REPO", "mathias/brain"),
Logger: logger,
}
logger.Info("brain-sync webhook enabled", "namespace", webhookHandler.Namespace, "cronjob", webhookHandler.CronJobName)
}
if vectorStore != nil { if vectorStore != nil {
embedSyncInterval := envInt("BRAIN_EMBED_SYNC_INTERVAL", 300) embedSyncInterval := envInt("BRAIN_EMBED_SYNC_INTERVAL", 300)
vectorstore.StartSync(ctx, brainDir, vectorStore, vectorstore.StartSync(ctx, brainDir, vectorStore,
@@ -313,8 +411,13 @@ func main() {
mux.HandleFunc("POST /ingest-path", h.IngestPath) mux.HandleFunc("POST /ingest-path", h.IngestPath)
mux.HandleFunc("POST /ingest-raw", h.IngestRaw) mux.HandleFunc("POST /ingest-raw", h.IngestRaw)
mux.HandleFunc("POST /backfill-refs", h.BackfillRefs) mux.HandleFunc("POST /backfill-refs", h.BackfillRefs)
mux.HandleFunc("GET /pending", h.Pending)
mux.HandleFunc("POST /promote", h.Promote)
mux.HandleFunc("POST /backfill-embeddings", h.BackfillEmbeddings) mux.HandleFunc("POST /backfill-embeddings", h.BackfillEmbeddings)
mux.HandleFunc("GET /pass-rate", h.PassRate) mux.HandleFunc("GET /pass-rate", h.PassRate)
if webhookHandler != nil {
mux.Handle("POST /webhooks/brain-sync", webhookHandler)
}
jwtValidator, err := chassisauth.NewJWTValidator(ctx, os.Getenv("DEX_ISSUER_URL"), os.Getenv("MCP_AUDIENCE")) jwtValidator, err := chassisauth.NewJWTValidator(ctx, os.Getenv("DEX_ISSUER_URL"), os.Getenv("MCP_AUDIENCE"))
if err != nil { if err != nil {
logger.Error("build jwt validator", "err", err) logger.Error("build jwt validator", "err", err)
@@ -339,6 +442,41 @@ func main() {
mux.Handle("/mcp", chassisauth.BearerMiddleware(mcpToken, jwtValidator, "brain", resourceMetadataURL, mcpSrv)) mux.Handle("/mcp", chassisauth.BearerMiddleware(mcpToken, jwtValidator, "brain", resourceMetadataURL, mcpSrv))
// POST /capture (#53/#54): the uniform capture REST door. Needs a ticket
// tracker to file action items, so it only mounts when Gitea is
// configured. It reuses the MCP server's graph-wired brain store (one
// implementation), the classification tags for the I1 gate, and a
// classification-aware audit sink (loki + durable buffer + ntfy when
// BRAIN_LOKI_URL is set, else a plain slog sink). The handler does its
// own auth (static + JWT) because it needs the principal to derive the
// trust-zone origin — the chassis middleware hides it.
if tracker := mcpSrv.IssueTracker(); tracker != nil {
classCfg, cerr := classification.Load(brainDir)
if cerr != nil {
logger.Error("load classification config", "err", cerr)
os.Exit(1)
}
auditSink := buildAuditSink(ctx, brainDir, logger)
// The Gitea client also satisfies SummaryWriter (#66): session
// summaries are written to mathias/ai-sessions over the same API
// token. nil only if a future tracker impl lacks file writes.
summaryWriter, _ := tracker.(capture.SummaryWriter)
captureSvc := capture.NewService(
mcpSrv.BrainStore(), tracker, summaryWriter, classCfg, auditSink)
sovereign := splitList(os.Getenv("BRAIN_CAPTURE_SOVEREIGN_PRINCIPALS"))
resolver := capturehttp.NewOriginResolver(sovereign)
captureH := capturehttp.New(captureSvc, jwtValidator, mcpToken, "local-cli", resolver)
mux.Handle("POST /capture", captureH)
// Same use-case behind the MCP `capture` tool (#55 relay) so MCP-native
// harnesses (claude.ai, Crush, Pi, LLM Council) reach capture through
// the existing /mcp OAuth connector. mcpSrv is already wrapped above;
// WithCapture mutates the same instance, so the tool appears live.
mcpSrv.WithCapture(captureSvc, jwtValidator, mcpToken, "local-cli", resolver)
logger.Info("capture enabled (REST + MCP tool)", "sovereign_principals", len(sovereign))
} else {
logger.Info("capture endpoint disabled (BRAIN_GITEA_TOKEN unset)")
}
// Opt-in OAuth 2.0 client_credentials flow for claude.ai's custom-MCP // Opt-in OAuth 2.0 client_credentials flow for claude.ai's custom-MCP
// integration UI, which has no static-Bearer field. Setting both // integration UI, which has no static-Bearer field. Setting both
// OAUTH_CLIENT_ID and OAUTH_CLIENT_SECRET enables the token exchange; // OAUTH_CLIENT_ID and OAUTH_CLIENT_SECRET enables the token exchange;
+38
View File
@@ -0,0 +1,38 @@
// ingestion/cmd/server/main_test.go
package main
import (
"context"
"log/slog"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/claudewatcher"
)
func TestClaudeSink_IngestWritesToNonIndexedArchiveNotWiki(t *testing.T) {
dir := t.TempDir()
sink := &claudeSink{brainDir: dir, logger: slog.New(slog.NewTextHandler(os.Stderr, nil))}
err := sink.Ingest(context.Background(), claudewatcher.Batch{
Host: "koala",
FilePath: "/host-home-claude/projects/-home-mathias-dev/abc.jsonl",
SessionID: "abc",
ProjectID: "-home-mathias-dev",
Turns: []claudewatcher.Turn{
{Type: "assistant", Content: "did a thing"},
},
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(dir, "archive", "claude-sessions", "koala", "session-koala-abc.md"))
require.NoError(t, err, "raw session dump must land in the non-indexed archive")
assert.Contains(t, string(got), "did a thing")
_, err = os.Stat(filepath.Join(dir, "wiki", "claude-sessions"))
assert.True(t, os.IsNotExist(err), "raw transcripts must never land under wiki/ (ai-sessions#10 — BM25 pollution)")
}
+49 -6
View File
@@ -3,29 +3,72 @@ module github.com/mathiasbq/hyperguild/ingestion
go 1.26.1 go 1.26.1
require ( require (
github.com/lestrrat-go/jwx/v2 v2.1.6
github.com/stretchr/testify v1.11.1 github.com/stretchr/testify v1.11.1
k8s.io/api v0.31.3
k8s.io/apimachinery v0.31.3
k8s.io/client-go v0.31.3
) )
require ( require (
gitea.d-ma.be/mathias/mcp-chassis v0.1.0 // indirect github.com/emicklei/go-restful/v3 v3.11.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect github.com/fxamacker/cbor/v2 v2.7.0 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-openapi/jsonpointer v0.19.6 // indirect
github.com/go-openapi/jsonreference v0.20.2 // indirect
github.com/go-openapi/swag v0.22.4 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/google/gnostic-models v0.6.8 // indirect
github.com/google/go-cmp v0.6.0 // indirect
github.com/google/gofuzz v1.2.0 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/imdario/mergo v0.3.6 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/lestrrat-go/jwx/v2 v2.1.6 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/rogpeppe/go-internal v1.15.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/x448/float16 v0.8.4 // indirect
golang.org/x/net v0.26.0 // indirect
golang.org/x/oauth2 v0.21.0 // indirect
golang.org/x/term v0.28.0 // indirect
golang.org/x/time v0.3.0 // indirect
google.golang.org/protobuf v1.34.2 // indirect
gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
k8s.io/klog/v2 v2.130.1 // indirect
k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 // indirect
k8s.io/utils v0.0.0-20240711033017-18e509b52bc8 // indirect
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect
sigs.k8s.io/structured-merge-diff/v4 v4.4.1 // indirect
sigs.k8s.io/yaml v1.4.0 // indirect
)
require (
git.d-ma.be/mathias/mcp-chassis v0.2.0
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0 // indirect github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0 // indirect
github.com/goccy/go-json v0.10.3 // indirect github.com/goccy/go-json v0.10.3 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/pgx/v5 v5.9.2 // indirect github.com/jackc/pgx/v5 v5.9.2
github.com/jackc/puddle/v2 v2.2.2 // indirect github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/lestrrat-go/blackmagic v1.0.3 // indirect github.com/lestrrat-go/blackmagic v1.0.3 // indirect
github.com/lestrrat-go/httpcc v1.0.1 // indirect github.com/lestrrat-go/httpcc v1.0.1 // indirect
github.com/lestrrat-go/httprc v1.0.6 // indirect github.com/lestrrat-go/httprc v1.0.6 // indirect
github.com/lestrrat-go/iter v1.0.2 // indirect github.com/lestrrat-go/iter v1.0.2 // indirect
github.com/lestrrat-go/option v1.0.1 // indirect github.com/lestrrat-go/option v1.0.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/segmentio/asm v1.2.0 // indirect github.com/segmentio/asm v1.2.0 // indirect
golang.org/x/crypto v0.32.0 // indirect golang.org/x/crypto v0.32.0 // indirect
golang.org/x/sync v0.17.0 // indirect golang.org/x/sync v0.17.0 // indirect
golang.org/x/sys v0.31.0 // indirect golang.org/x/sys v0.31.0 // indirect
golang.org/x/text v0.29.0 // indirect golang.org/x/text v0.29.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect gopkg.in/yaml.v3 v3.0.1
) )
+143 -5
View File
@@ -1,12 +1,47 @@
gitea.d-ma.be/mathias/mcp-chassis v0.1.0 h1:8RXO34+n7Vu8HnUMagars6fc4oemqRpMu7MVtjaj4qY= git.d-ma.be/mathias/mcp-chassis v0.2.0 h1:6fLmb7xqRa2nNVWsHaUbbfbArgDXJw/gDhb09clBIjo=
gitea.d-ma.be/mathias/mcp-chassis v0.1.0/go.mod h1:ajbLlwr2L7FAN3TBU39KucZkKJM02wTbKbDKDEW2YvE= git.d-ma.be/mathias/mcp-chassis v0.2.0/go.mod h1:Ks7EK2UnGAN0H3rJjKUxUagX8/ZBdtLrOlcUbv0RwH8=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0 h1:NMZiJj8QnKe1LgsbDayM4UoHwbvwDRwnI3hwNaAHRnc= github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0 h1:NMZiJj8QnKe1LgsbDayM4UoHwbvwDRwnI3hwNaAHRnc=
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0/go.mod h1:ZXNYxsqcloTdSy/rNShjYzMhyjf0LaoftYK0p+A3h40= github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0/go.mod h1:ZXNYxsqcloTdSy/rNShjYzMhyjf0LaoftYK0p+A3h40=
github.com/emicklei/go-restful/v3 v3.11.0 h1:rAQeMHw1c7zTmncogyy8VvRZwtkmkZ4FxERmMY4rD+g=
github.com/emicklei/go-restful/v3 v3.11.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc=
github.com/fxamacker/cbor/v2 v2.7.0 h1:iM5WgngdRBanHcxugY4JySA0nk1wZorNOpTgCMedv5E=
github.com/fxamacker/cbor/v2 v2.7.0/go.mod h1:pxXPTn3joSm21Gbwsv0w9OSA2y1HFR9qXEeXQVeNoDQ=
github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-openapi/jsonpointer v0.19.6 h1:eCs3fxoIi3Wh6vtgmLTOjdhSpiqphQ+DaPn38N2ZdrE=
github.com/go-openapi/jsonpointer v0.19.6/go.mod h1:osyAmYz/mB/C3I+WsTTSgw1ONzaLJoLCyoi6/zppojs=
github.com/go-openapi/jsonreference v0.20.2 h1:3sVjiK66+uXK/6oQ8xgcRKcFgQ5KXa2KvnJRumpMGbE=
github.com/go-openapi/jsonreference v0.20.2/go.mod h1:Bl1zwGIM8/wsvqjsOQLJ/SH+En5Ap4rVB5KVcIDZG2k=
github.com/go-openapi/swag v0.22.3/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+VcZ0yl14=
github.com/go-openapi/swag v0.22.4 h1:QLMzNJnMGPRNDCbySlcj1x01tzU8/9LTTL9hZZZogBU=
github.com/go-openapi/swag v0.22.4/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+VcZ0yl14=
github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI=
github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8=
github.com/goccy/go-json v0.10.3 h1:KZ5WoDbxAIgm2HNbYckL0se1fHD6rz5j4ywS6ebzDqA= github.com/goccy/go-json v0.10.3 h1:KZ5WoDbxAIgm2HNbYckL0se1fHD6rz5j4ywS6ebzDqA=
github.com/goccy/go-json v0.10.3/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M= github.com/goccy/go-json v0.10.3/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
github.com/google/gnostic-models v0.6.8 h1:yo/ABAfM5IMRsS1VnXjTBvUb61tFIHozhlYvRgGre9I=
github.com/google/gnostic-models v0.6.8/go.mod h1:5n7qKqH0f5wFt+aWF8CW6pZLLNOfYuF5OpfBSENuI8U=
github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0=
github.com/google/gofuzz v1.2.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/pprof v0.0.0-20240525223248-4bfdf5a9a2af h1:kmjWCqn2qkEml422C2Rrd27c3VGxi6a/6HNq8QmHRKM=
github.com/google/pprof v0.0.0-20240525223248-4bfdf5a9a2af/go.mod h1:K1liHPHnj73Fdn/EKuT8nrFqBihUSKXoLYU0BuatOYo=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/imdario/mergo v0.3.6 h1:xTNEAn+kxVO7dTZGu0CegyqKZmoWFI0rF8UxjlB2d28=
github.com/imdario/mergo v0.3.6/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM= github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg= github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo= github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
@@ -15,6 +50,19 @@ github.com/jackc/pgx/v5 v5.9.2 h1:3ZhOzMWnR4yJ+RW1XImIPsD1aNSz4T4fyP7zlQb56hw=
github.com/jackc/pgx/v5 v5.9.2/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4= github.com/jackc/pgx/v5 v5.9.2/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo= github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4= github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/lestrrat-go/blackmagic v1.0.3 h1:94HXkVLxkZO9vJI/w2u1T0DAoprShFd13xtnSINtDWs= github.com/lestrrat-go/blackmagic v1.0.3 h1:94HXkVLxkZO9vJI/w2u1T0DAoprShFd13xtnSINtDWs=
github.com/lestrrat-go/blackmagic v1.0.3/go.mod h1:6AWFyKNNj0zEXQYfTMPfZrAXUWUfTIZ5ECEUEJaijtw= github.com/lestrrat-go/blackmagic v1.0.3/go.mod h1:6AWFyKNNj0zEXQYfTMPfZrAXUWUfTIZ5ECEUEJaijtw=
github.com/lestrrat-go/httpcc v1.0.1 h1:ydWCStUeJLkpYyjLDHihupbn2tYmZ7m22BGkcvZZrIE= github.com/lestrrat-go/httpcc v1.0.1 h1:ydWCStUeJLkpYyjLDHihupbn2tYmZ7m22BGkcvZZrIE=
@@ -27,28 +75,118 @@ github.com/lestrrat-go/jwx/v2 v2.1.6 h1:hxM1gfDILk/l5ylers6BX/Eq1m/pnxe9NBwW6lVf
github.com/lestrrat-go/jwx/v2 v2.1.6/go.mod h1:Y722kU5r/8mV7fYDifjug0r8FK8mZdw0K0GpJw/l8pU= github.com/lestrrat-go/jwx/v2 v2.1.6/go.mod h1:Y722kU5r/8mV7fYDifjug0r8FK8mZdw0K0GpJw/l8pU=
github.com/lestrrat-go/option v1.0.1 h1:oAzP2fvZGQKWkvHa1/SAcFolBEca1oN+mQ7eooNBEYU= github.com/lestrrat-go/option v1.0.1 h1:oAzP2fvZGQKWkvHa1/SAcFolBEca1oN+mQ7eooNBEYU=
github.com/lestrrat-go/option v1.0.1/go.mod h1:5ZHFbivi4xwXxhxY9XHDe2FHo6/Z7WWmtT7T5nBBp3I= github.com/lestrrat-go/option v1.0.1/go.mod h1:5ZHFbivi4xwXxhxY9XHDe2FHo6/Z7WWmtT7T5nBBp3I=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0=
github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/onsi/ginkgo/v2 v2.19.0 h1:9Cnnf7UHo57Hy3k6/m5k3dRfGTMXGvxhHFvkDTCTpvA=
github.com/onsi/ginkgo/v2 v2.19.0/go.mod h1:rlwLi9PilAFJ8jCg9UE1QP6VBpd6/xj3SRC0d6TU0To=
github.com/onsi/gomega v1.19.0 h1:4ieX6qQjPP/BfC3mpsAtIGGlxTWPeA3Inl/7DtXw1tw=
github.com/onsi/gomega v1.19.0/go.mod h1:LY+I3pBVzYsTBU1AnDwOSxaYi9WoWiqgwooUqq9yPro=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/rogpeppe/go-internal v1.15.0 h1:D0RCU5rMAp+SpgkiNdrjfJ+LX4J1M32V2NeCY7EJ6hc=
github.com/rogpeppe/go-internal v1.15.0/go.mod h1:DrUVZyrJU+txYW5/1kwtXQSMFio52ZOxX7yM1VHvnxs=
github.com/segmentio/asm v1.2.0 h1:9BQrFxC+YOHJlTlHGkTrFWf59nbL3XnCoFLTwDCI7ys= github.com/segmentio/asm v1.2.0 h1:9BQrFxC+YOHJlTlHGkTrFWf59nbL3XnCoFLTwDCI7ys=
github.com/segmentio/asm v1.2.0/go.mod h1:BqMnlJP91P8d+4ibuonYZw9mfnzI9HfxselHZr5aAcs= github.com/segmentio/asm v1.2.0/go.mod h1:BqMnlJP91P8d+4ibuonYZw9mfnzI9HfxselHZr5aAcs=
github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM=
github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.32.0 h1:euUpcYgM8WcP71gNpTqQCn6rC2t6ULUPiOzfWaXVVfc= golang.org/x/crypto v0.32.0 h1:euUpcYgM8WcP71gNpTqQCn6rC2t6ULUPiOzfWaXVVfc=
golang.org/x/crypto v0.32.0/go.mod h1:ZnnJkOaASj8g0AjIduWNlq2NRxL0PlBrbKVyZ6V/Ugc= golang.org/x/crypto v0.32.0/go.mod h1:ZnnJkOaASj8g0AjIduWNlq2NRxL0PlBrbKVyZ6V/Ugc=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.26.0 h1:soB7SVo0PWrY4vPW/+ay0jKDNScG2X9wFeYlXIvJsOQ=
golang.org/x/net v0.26.0/go.mod h1:5YKkiSynbBIh3p6iOc/vibscux0x38BZDkn8sCUPxHE=
golang.org/x/oauth2 v0.21.0 h1:tsimM75w1tF/uws5rbeHzIWxEqElMehnc+iW793zsZs=
golang.org/x/oauth2 v0.21.0/go.mod h1:XYTD2NtWslqkgxebSiOHnXEap4TF09sJSc7H1sXbhtI=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.17.0 h1:l60nONMj9l5drqw6jlhIELNv9I0A4OFgRsG9k2oT9Ug= golang.org/x/sync v0.17.0 h1:l60nONMj9l5drqw6jlhIELNv9I0A4OFgRsG9k2oT9Ug=
golang.org/x/sync v0.17.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI= golang.org/x/sync v0.17.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik= golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik=
golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k= golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/term v0.28.0 h1:/Ts8HFuMR2E6IP/jlo7QVLZHggjKQbhu/7H0LJFr3Gg=
golang.org/x/term v0.28.0/go.mod h1:Sw/lC2IAUZ92udQNf3WodGtn4k/XoLyZoh8v/8uiwek=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.29.0 h1:1neNs90w9YzJ9BocxfsQNHKuAT4pkghyXc4nhZ6sJvk= golang.org/x/text v0.29.0 h1:1neNs90w9YzJ9BocxfsQNHKuAT4pkghyXc4nhZ6sJvk=
golang.org/x/text v0.29.0/go.mod h1:7MhJOA9CD2qZyOKYazxdYMF85OwPdEr9jTtBpO7ydH4= golang.org/x/text v0.29.0/go.mod h1:7MhJOA9CD2qZyOKYazxdYMF85OwPdEr9jTtBpO7ydH4=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4=
golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.36.0 h1:kWS0uv/zsvHEle1LbV5LE8QujrxB3wfQyxHfhOk0Qkg=
golang.org/x/tools v0.36.0/go.mod h1:WBDiHKJK8YgLHlcQPYQzNCkUxUypCaa5ZegCVutKm+s=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/protobuf v1.34.2 h1:6xV6lTsCfpGD21XK49h7MhtcApnLqkfYgPcdHftf6hg=
google.golang.org/protobuf v1.34.2/go.mod h1:qYOHts0dSfpeUzUFpOMr/WGzszTmLH+DiWniOlNbLDw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/evanphx/json-patch.v4 v4.12.0 h1:n6jtcsulIzXPJaxegRbvFNNrZDjbij7ny3gmSPG+6V4=
gopkg.in/evanphx/json-patch.v4 v4.12.0/go.mod h1:p8EYWUEYMpynmqDbY58zCKCFZw8pRWMG4EsWvDvM72M=
gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc=
gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
k8s.io/api v0.31.3 h1:umzm5o8lFbdN/hIXbrK9oRpOproJO62CV1zqxXrLgk8=
k8s.io/api v0.31.3/go.mod h1:UJrkIp9pnMOI9K2nlL6vwpxRzzEX5sWgn8kGQe92kCE=
k8s.io/apimachinery v0.31.3 h1:6l0WhcYgasZ/wk9ktLq5vLaoXJJr5ts6lkaQzgeYPq4=
k8s.io/apimachinery v0.31.3/go.mod h1:rsPdaZJfTfLsNJSQzNHQvYoTmxhoOEofxtOsF3rtsMo=
k8s.io/client-go v0.31.3 h1:CAlZuM+PH2cm+86LOBemaJI/lQ5linJ6UFxKX/SoG+4=
k8s.io/client-go v0.31.3/go.mod h1:2CgjPUTpv3fE5dNygAr2NcM8nhHzXvxB8KL5gYc3kJs=
k8s.io/klog/v2 v2.130.1 h1:n9Xl7H1Xvksem4KFG4PYbdQCQxqc/tTUyrgXaOhHSzk=
k8s.io/klog/v2 v2.130.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE=
k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 h1:BZqlfIlq5YbRMFko6/PM7FjZpUb45WallggurYhKGag=
k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340/go.mod h1:yD4MZYeKMBwQKVht279WycxKyM84kkAx2DPrTXaeb98=
k8s.io/utils v0.0.0-20240711033017-18e509b52bc8 h1:pUdcCO1Lk/tbT5ztQWOBi5HBgbBP1J8+AsQnQCKsi8A=
k8s.io/utils v0.0.0-20240711033017-18e509b52bc8/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0=
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd h1:EDPBXCAspyGV4jQlpZSudPeMmr1bNJefnuqLsRAsHZo=
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd/go.mod h1:B8JuhiUyNFVKdsE8h686QcCxMaH6HrOAZj4vswFpcB0=
sigs.k8s.io/structured-merge-diff/v4 v4.4.1 h1:150L+0vs/8DA78h1u02ooW1/fFq/Lwr+sGiqlzvrtq4=
sigs.k8s.io/structured-merge-diff/v4 v4.4.1/go.mod h1:N8hJocpFajUSSeSJ9bOZ77VzejKZaXsTtZo4/u7Io08=
sigs.k8s.io/yaml v1.4.0 h1:Mk1wCc2gy/F0THH0TAp1QYyJNzRm2KCLy3o5ASXVI5E=
sigs.k8s.io/yaml v1.4.0/go.mod h1:Ejl7/uTz7PSA4eKMyQCUTnhZYNmLIl+5c2lQPGR2BPY=
+97
View File
@@ -0,0 +1,97 @@
package api
import "strings"
// frontmatter is an ordered, line-preserving view of a note's YAML
// frontmatter block. It deliberately avoids a full YAML round-trip: the
// brain writes flat `key: value` frontmatter by hand, and a yaml.v3
// re-marshal would reorder keys and strip comments. Preserving the
// original lines verbatim keeps brain_update a surgical edit — only the
// keys it manages (updated_at, supersedes, supersede_reason) change.
type frontmatter struct {
lines []fmLine
}
// fmLine is one frontmatter line. For `key: value` lines, key and value
// are populated; for blank lines, comments, or anything that isn't a
// simple scalar pair, key is empty and raw holds the line verbatim.
type fmLine struct {
key string
value string
raw string
}
// parseFrontmatter splits src into its frontmatter block and body. A
// frontmatter block is recognised only when the file opens with a `---`
// fence and a closing `---` fence follows. Otherwise the whole input is
// the body and the returned frontmatter is empty.
func parseFrontmatter(src string) (frontmatter, string) {
var fm frontmatter
if !strings.HasPrefix(src, "---\n") {
return fm, src
}
rest := src[len("---\n"):]
end := strings.Index(rest, "\n---\n")
if end < 0 {
// Opening fence with no closing fence — treat as bodyless content.
return fm, src
}
block := rest[:end]
body := rest[end+len("\n---\n"):]
for _, line := range strings.Split(block, "\n") {
key, val, ok := strings.Cut(line, ":")
key = strings.TrimSpace(key)
if !ok || key == "" || strings.HasPrefix(strings.TrimSpace(line), "#") {
fm.lines = append(fm.lines, fmLine{raw: line})
continue
}
fm.lines = append(fm.lines, fmLine{key: key, value: strings.TrimSpace(val)})
}
return fm, body
}
// get returns the value for key, or "" if absent.
func (f *frontmatter) get(key string) string {
for _, l := range f.lines {
if l.key == key {
return l.value
}
}
return ""
}
// set overrides the value for an existing key in place, or appends a new
// `key: value` line when the key is absent.
func (f *frontmatter) set(key, value string) {
for i := range f.lines {
if f.lines[i].key == key {
f.lines[i].value = value
return
}
}
f.lines = append(f.lines, fmLine{key: key, value: value})
}
// render serialises the frontmatter back into a `---`-fenced block. An
// empty frontmatter renders to the empty string so bodies without a
// header stay header-less.
func (f *frontmatter) render() string {
if len(f.lines) == 0 {
return ""
}
var b strings.Builder
b.WriteString("---\n")
for _, l := range f.lines {
if l.key == "" {
b.WriteString(l.raw)
} else {
b.WriteString(l.key)
b.WriteString(": ")
b.WriteString(l.value)
}
b.WriteByte('\n')
}
b.WriteString("---\n")
return b.String()
}
@@ -0,0 +1,61 @@
package api
import (
"strings"
"testing"
"github.com/stretchr/testify/assert"
)
func TestParseFrontmatterSplitsHeaderAndBody(t *testing.T) {
src := "---\nwing: jepa-fx\nhall: facts\ncreated_at: 2026-01-01T00:00:00Z\n---\n# Title\n\nbody text\n"
fm, body := parseFrontmatter(src)
assert.Equal(t, "jepa-fx", fm.get("wing"))
assert.Equal(t, "facts", fm.get("hall"))
assert.Equal(t, "2026-01-01T00:00:00Z", fm.get("created_at"))
assert.Equal(t, "# Title\n\nbody text\n", body)
}
func TestParseFrontmatterNoHeader(t *testing.T) {
src := "# Just a body\n\nno frontmatter here\n"
fm, body := parseFrontmatter(src)
assert.Empty(t, fm.lines)
assert.Equal(t, src, body)
}
func TestFrontmatterSetOverridesExistingKey(t *testing.T) {
fm, _ := parseFrontmatter("---\nwing: a\nupdated_at: old\n---\nbody\n")
fm.set("updated_at", "new")
assert.Equal(t, "new", fm.get("updated_at"))
// No duplicate key.
assert.Equal(t, 1, strings.Count(fm.render(), "updated_at:"))
}
func TestFrontmatterSetAppendsNewKey(t *testing.T) {
fm, _ := parseFrontmatter("---\nwing: a\n---\nbody\n")
fm.set("supersedes", "abc123")
out := fm.render()
assert.Contains(t, out, "wing: a")
assert.Contains(t, out, "supersedes: abc123")
}
func TestFrontmatterRenderPreservesCustomFields(t *testing.T) {
src := "---\nwing: a\nhall: facts\ncustom_field: keep-me\ntags: [x, y]\n---\nbody\n"
fm, _ := parseFrontmatter(src)
fm.set("updated_at", "2026-06-22T00:00:00Z")
out := fm.render()
assert.Contains(t, out, "custom_field: keep-me")
assert.Contains(t, out, "tags: [x, y]")
assert.Contains(t, out, "updated_at: 2026-06-22T00:00:00Z")
}
func TestFrontmatterRenderRoundTrips(t *testing.T) {
src := "---\nwing: a\nhall: facts\n---\n"
fm, _ := parseFrontmatter(src)
assert.Equal(t, src, fm.render())
}
+153 -24
View File
@@ -51,12 +51,13 @@ type queryRequest struct {
} }
type writeRequest struct { type writeRequest struct {
Content string `json:"content"` Content string `json:"content"`
Filename string `json:"filename,omitempty"` Filename string `json:"filename,omitempty"`
Type string `json:"type,omitempty"` Type string `json:"type,omitempty"`
Domain string `json:"domain,omitempty"` Domain string `json:"domain,omitempty"`
Wing string `json:"wing,omitempty"` Wing string `json:"wing,omitempty"`
Hall string `json:"hall,omitempty"` Hall string `json:"hall,omitempty"`
SourceType string `json:"source_type,omitempty"` // "external" opts a hall=facts entry out of the internal default
} }
type ingestRequest struct { type ingestRequest struct {
@@ -115,12 +116,13 @@ func (h *Handler) Query(w http.ResponseWriter, r *http.Request) {
// When either is empty, the note falls back to brain/knowledge/<filename> // When either is empty, the note falls back to brain/knowledge/<filename>
// with optional type/domain frontmatter (legacy behaviour). // with optional type/domain frontmatter (legacy behaviour).
type WriteNoteOptions struct { type WriteNoteOptions struct {
Content string Content string
Filename string Filename string
Type string Type string
Domain string Domain string
Wing string Wing string
Hall string Hall string
SourceType string // "internal" marks a first-party observation (e.g. claudewatcher) that needs no external citation
} }
// WriteNote writes a markdown note into the brain. Returns the path // WriteNote writes a markdown note into the brain. Returns the path
@@ -154,26 +156,111 @@ func writeHallNote(brainDir string, opts WriteNoteOptions) (string, error) {
return "", fmt.Errorf("create hall dir: %w", err) return "", fmt.Errorf("create hall dir: %w", err)
} }
existingFields, body := splitFrontmatter(opts.Content)
existingByKey := make(map[string]frontmatterField, len(existingFields))
for _, f := range existingFields {
existingByKey[f.key] = f
}
emitted := make(map[string]bool, 6)
var fm strings.Builder var fm strings.Builder
fm.WriteString("---\n") fm.WriteString("---\n")
fmt.Fprintf(&fm, "wing: %s\n", brain.Sanitise(opts.Wing)) fmt.Fprintf(&fm, "wing: %s\n", brain.Sanitise(opts.Wing))
fmt.Fprintf(&fm, "hall: %s\n", opts.Hall) fmt.Fprintf(&fm, "hall: %s\n", opts.Hall)
fmt.Fprintf(&fm, "created_at: %s\n", time.Now().UTC().Format(time.RFC3339)) fmt.Fprintf(&fm, "created_at: %s\n", time.Now().UTC().Format(time.RFC3339))
if opts.Type != "" { emitted["wing"], emitted["hall"], emitted["created_at"] = true, true, true
fmt.Fprintf(&fm, "type: %s\n", opts.Type)
// writeField merges one key: opts.Content's own value (if the note already
// carries this field in its own frontmatter) always wins over the fallback,
// so promotion/extraction-step metadata survives verbatim instead of being
// shadowed by a second, stacked frontmatter block (#86).
writeField := func(key, fallback string) {
emitted[key] = true
if f, ok := existingByKey[key]; ok {
for _, line := range f.lines {
fm.WriteString(line)
fm.WriteString("\n")
}
return
}
if fallback != "" {
fmt.Fprintf(&fm, "%s: %s\n", key, fallback)
}
} }
if opts.Domain != "" { writeField("type", opts.Type)
fmt.Fprintf(&fm, "domain: %s\n", opts.Domain) writeField("domain", opts.Domain)
sourceType := opts.SourceType
if sourceType == "" && opts.Hall == "facts" {
// Most hall=facts entries are first-party (an eval/benchmark the
// writer ran itself), not external claims — default to internal and
// require an explicit source_type: external opt-out for the rare
// citation-needing entry (brain-gardener#7).
sourceType = "internal"
}
writeField("source_type", sourceType)
for _, f := range existingFields {
if emitted[f.key] {
continue
}
for _, line := range f.lines {
fm.WriteString(line)
fm.WriteString("\n")
}
} }
fm.WriteString("---\n") fm.WriteString("---\n")
if err := os.WriteFile(dest, []byte(fm.String()+opts.Content), 0o644); err != nil { if err := os.WriteFile(dest, []byte(fm.String()+body), 0o644); err != nil {
return "", fmt.Errorf("write: %w", err) return "", fmt.Errorf("write: %w", err)
} }
rel, _ := filepath.Rel(brainDir, dest) rel, _ := filepath.Rel(brainDir, dest)
return filepath.ToSlash(rel), nil return filepath.ToSlash(rel), nil
} }
// frontmatterField is one top-level YAML key from a frontmatter block,
// along with its raw line and any indented continuation lines (e.g. a
// bulleted list value spanning multiple lines).
type frontmatterField struct {
key string
lines []string
}
// splitFrontmatter splits a leading "---\n...\n---\n" YAML block out of
// content, returning its top-level fields in original order and the
// remaining body. If content has no leading frontmatter block, fields is
// nil and body is content unchanged.
func splitFrontmatter(content string) (fields []frontmatterField, body string) {
if !strings.HasPrefix(content, "---\n") {
return nil, content
}
lines := strings.Split(content, "\n")
i := 1
var cur *frontmatterField
for ; i < len(lines); i++ {
line := lines[i]
if strings.TrimSpace(line) == "---" {
i++
break
}
if line != "" && !strings.HasPrefix(line, " ") && !strings.HasPrefix(line, "\t") {
if cur != nil {
fields = append(fields, *cur)
}
key, _, _ := strings.Cut(line, ":")
cur = &frontmatterField{key: strings.TrimSpace(key), lines: []string{line}}
} else if cur != nil {
cur.lines = append(cur.lines, line)
}
}
if cur != nil {
fields = append(fields, *cur)
}
body = strings.Join(lines[i:], "\n")
return fields, body
}
// writeLegacyNote preserves the original brain/knowledge/ behaviour for // writeLegacyNote preserves the original brain/knowledge/ behaviour for
// callers that have not adopted the wing/hall taxonomy. // callers that have not adopted the wing/hall taxonomy.
func writeLegacyNote(brainDir string, opts WriteNoteOptions) (string, error) { func writeLegacyNote(brainDir string, opts WriteNoteOptions) (string, error) {
@@ -332,11 +419,19 @@ func (h *Handler) Ingest(w http.ResponseWriter, r *http.Request) {
writeJSON(w, ingestResponse{Pages: pages, Warnings: warnings}) writeJSON(w, ingestResponse{Pages: pages, Warnings: warnings})
} }
// supportedExtensions lists file extensions that IngestPath will process. // isSupportedExtension reports whether IngestPath will process ext.
var supportedExtensions = map[string]bool{ // .docx/.xlsx/.pptx/.png/.jpg/.jpeg require docmark (ADR-0013) and are only
".md": true, // supported when DOCMARK_URL is configured — checked per-call (not cached at
".txt": true, // package init) so it reflects the environment at request time.
".pdf": true, func isSupportedExtension(ext string) bool {
switch ext {
case ".md", ".txt", ".pdf":
return true
case ".docx", ".xlsx", ".pptx", ".png", ".jpg", ".jpeg":
return os.Getenv("DOCMARK_URL") != ""
default:
return false
}
} }
// IngestPath handles POST /ingest-path — ingest a file or directory. // IngestPath handles POST /ingest-path — ingest a file or directory.
@@ -369,7 +464,7 @@ func (h *Handler) IngestPath(w http.ResponseWriter, r *http.Request) {
return nil return nil
} }
ext := strings.ToLower(filepath.Ext(path)) ext := strings.ToLower(filepath.Ext(path))
if !supportedExtensions[ext] { if !isSupportedExtension(ext) {
return nil return nil
} }
content, readErr := extract.Text(path) content, readErr := extract.Text(path)
@@ -397,7 +492,7 @@ func (h *Handler) IngestPath(w http.ResponseWriter, r *http.Request) {
} }
} else { } else {
ext := strings.ToLower(filepath.Ext(req.Path)) ext := strings.ToLower(filepath.Ext(req.Path))
if !supportedExtensions[ext] { if !isSupportedExtension(ext) {
writeError(w, http.StatusBadRequest, fmt.Sprintf("unsupported file extension: %s", ext)) writeError(w, http.StatusBadRequest, fmt.Sprintf("unsupported file extension: %s", ext))
return return
} }
@@ -483,6 +578,40 @@ func (h *Handler) BackfillRefs(w http.ResponseWriter, r *http.Request) {
writeJSON(w, map[string]int{"updated": n}) writeJSON(w, map[string]int{"updated": n})
} }
// Pending handles GET /pending — list raw/ notes awaiting promotion.
func (h *Handler) Pending(w http.ResponseWriter, _ *http.Request) {
pending, err := ListPending(h.brainDir)
if err != nil {
h.logger.Error("pending failed", "err", err)
writeError(w, http.StatusInternalServerError, "pending error")
return
}
writeJSON(w, map[string]any{"pending": pending})
}
type promoteRequest struct {
Filename string `json:"filename"`
Wing string `json:"wing"`
Hall string `json:"hall"`
Slug string `json:"slug,omitempty"`
}
// Promote handles POST /promote — move a raw/ note into the wiki. A bad
// hall / collision / missing source is a 400 (caller error), not a 500.
func (h *Handler) Promote(w http.ResponseWriter, r *http.Request) {
var req promoteRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid JSON")
return
}
rel, err := PromoteNote(h.brainDir, PromoteOptions(req))
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
writeJSON(w, map[string]string{"path": rel})
}
func writeJSON(w http.ResponseWriter, v any) { func writeJSON(w http.ResponseWriter, v any) {
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(v) //nolint:errcheck json.NewEncoder(w).Encode(v) //nolint:errcheck
+110
View File
@@ -118,6 +118,116 @@ func TestWrite_IncludesFrontmatterWhenTypeProvided(t *testing.T) {
assert.Contains(t, string(content), "Some learning.") assert.Contains(t, string(content), "Some learning.")
} }
func TestWriteNote_HallRouteIncludesSourceTypeWhenSet(t *testing.T) {
dir := t.TempDir()
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "# Claude session abc (koala)\n\nBody.\n",
Filename: "session-koala-abc",
Wing: "claude-sessions",
Hall: "facts",
Type: "source",
SourceType: "internal",
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.Contains(t, string(got), "source_type: internal")
assert.Contains(t, string(got), "wing: claude-sessions")
}
func TestWriteNote_HallFactsDefaultsSourceTypeInternalWhenUnset(t *testing.T) {
dir := t.TempDir()
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "manually captured fact.\n",
Wing: "agentsquad",
Hall: "facts",
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
// Most hall=facts entries are first-party (an eval/benchmark the agent ran
// itself), not external claims — default to internal, require explicit
// opt-out for the rare case that does need a citation (brain-gardener#7).
assert.Contains(t, string(got), "source_type: internal")
}
func TestWriteNote_HallFactsPreservesExplicitExternalSourceType(t *testing.T) {
dir := t.TempDir()
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "vendor pricing claim, needs a citation.\n",
Wing: "agentsquad",
Hall: "facts",
SourceType: "external",
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.Contains(t, string(got), "source_type: external")
}
func TestWriteNote_HallRouteOmitsSourceTypeForNonFactsHalls(t *testing.T) {
dir := t.TempDir()
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "a decision record.\n",
Wing: "agentsquad",
Hall: "decisions",
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.NotContains(t, string(got), "source_type")
}
func TestWriteNote_HallRouteMergesExistingFrontmatterInsteadOfStacking(t *testing.T) {
dir := t.TempDir()
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "---\ntitle: act_runner host-executor\ntags: [gitea-actions, act_runner]\n---\n\n# Body\n\nSome content.\n",
Filename: "act-runner-host-executor",
Wing: "homelab",
Hall: "failures",
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
body := string(got)
// exactly one frontmatter block: only two "---" delimiter lines total
assert.Equal(t, 2, strings.Count(body, "---\n"), "expected a single merged frontmatter block, not stacked blocks")
assert.Contains(t, body, "wing: homelab")
assert.Contains(t, body, "hall: failures")
assert.Contains(t, body, "title: act_runner host-executor")
assert.Contains(t, body, "tags: [gitea-actions, act_runner]")
assert.Contains(t, body, "# Body")
}
func TestWriteNote_HallRouteExistingTypeWinsOverOptsType(t *testing.T) {
dir := t.TempDir()
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "---\ntype: hypothesis\n---\n\nBody.\n",
Filename: "note",
Wing: "agentsquad",
Hall: "decisions",
Type: "decision", // should lose to content's own "type: hypothesis"
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.Contains(t, string(got), "type: hypothesis")
assert.NotContains(t, string(got), "type: decision")
}
func TestWrite_GeneratesFilenameIfAbsent(t *testing.T) { func TestWrite_GeneratesFilenameIfAbsent(t *testing.T) {
dir, h := setup(t) dir, h := setup(t)
body, _ := json.Marshal(map[string]any{"content": "auto name"}) body, _ := json.Marshal(map[string]any{"content": "auto name"})
@@ -0,0 +1,61 @@
// ingestion/internal/api/ingestpath_docmark_test.go
package api_test
import (
"bytes"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestIngestPath_DocxUnsupportedWhenDocmarkNotConfigured(t *testing.T) {
t.Setenv("DOCMARK_URL", "")
_, h := setup(t)
dir := t.TempDir()
f := filepath.Join(dir, "doc.docx")
require.NoError(t, os.WriteFile(f, []byte("fake docx"), 0o644))
body, _ := json.Marshal(map[string]any{"path": f, "source": "test-doc", "dry_run": true})
req := httptest.NewRequest(http.MethodPost, "/ingest-path", bytes.NewReader(body))
rec := httptest.NewRecorder()
h.IngestPath(rec, req)
assert.Equal(t, http.StatusBadRequest, rec.Code, rec.Body.String())
}
func TestIngestPath_DocxSupportedWhenDocmarkConfigured(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"jsonrpc":"2.0","id":1,"result":{"content":[{"type":"text","text":"# Converted Doc"}]}}`))
}))
defer srv.Close()
t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
t.Setenv("DOCMARK_BEARER_TOKEN", "tok")
_, h := setup(t)
dir := t.TempDir()
f := filepath.Join(dir, "doc.docx")
require.NoError(t, os.WriteFile(f, []byte("fake docx"), 0o644))
body, _ := json.Marshal(map[string]any{"path": f, "source": "test-doc", "dry_run": true})
req := httptest.NewRequest(http.MethodPost, "/ingest-path", bytes.NewReader(body))
rec := httptest.NewRecorder()
h.IngestPath(rec, req)
require.Equal(t, http.StatusOK, rec.Code, rec.Body.String())
var resp map[string]any
require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &resp))
pages, ok := resp["pages"].([]any)
require.True(t, ok)
assert.NotEmpty(t, pages)
}
+156
View File
@@ -0,0 +1,156 @@
package api
import (
"fmt"
"os"
"path/filepath"
"regexp"
"sort"
"strings"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
)
// PendingNote describes a raw/ note awaiting human promotion to the wiki.
type PendingNote struct {
Filename string `json:"filename"`
CreatedAt string `json:"created_at"`
SizeBytes int64 `json:"size_bytes"`
Excerpt string `json:"excerpt"`
}
// datePrefix matches a leading YYYY-MM-DD- on a raw filename, stripped when
// deriving the default promoted slug.
var datePrefix = regexp.MustCompile(`^\d{4}-\d{2}-\d{2}-`)
// ListPending returns the notes in brain/raw/ awaiting review, oldest-first
// (natural review order). An absent raw/ dir yields an empty slice, not an
// error. Only .md files are listed; tunnel-candidate files are skipped.
func ListPending(brainDir string) ([]PendingNote, error) {
dir := filepath.Join(brainDir, "raw")
entries, err := os.ReadDir(dir)
if err != nil {
if os.IsNotExist(err) {
return []PendingNote{}, nil
}
return nil, fmt.Errorf("read raw dir: %w", err)
}
out := make([]PendingNote, 0, len(entries))
for _, e := range entries {
if e.IsDir() || !strings.HasSuffix(e.Name(), ".md") || strings.HasPrefix(e.Name(), "tunnel-candidates-") {
continue
}
info, statErr := e.Info()
if statErr != nil {
continue
}
raw, readErr := os.ReadFile(filepath.Join(dir, e.Name()))
if readErr != nil {
continue
}
fm, body := parseFrontmatter(string(raw))
created := fm.get("created_at")
if created == "" {
created = info.ModTime().UTC().Format(time.RFC3339)
}
out = append(out, PendingNote{
Filename: e.Name(),
CreatedAt: created,
SizeBytes: info.Size(),
Excerpt: excerpt(body, 200),
})
}
sort.SliceStable(out, func(i, j int) bool { return out[i].CreatedAt < out[j].CreatedAt })
return out, nil
}
// PromoteOptions identifies a raw note to promote and its wiki destination.
type PromoteOptions struct {
Filename string // basename in brain/raw/
Wing string
Hall string
Slug string // optional; defaults to Filename minus date prefix + .md
}
// PromoteNote moves a note from brain/raw/ into the structured wiki: it
// rewrites frontmatter (sets wing/hall/promoted_at, preserves created_at and
// any custom fields), writes to brain/wiki/<wing>/<hall>/<slug>.md, deletes
// the source, then rebuilds the wing index and runs auto-tunnel detection.
//
// It is atomic from the caller's view: validation (hall, wing, slug,
// collision) happens before any filesystem change, and the source is deleted
// only after the destination write succeeds (write-then-delete, never move).
// Returns the promoted note's path relative to brainDir.
func PromoteNote(brainDir string, opts PromoteOptions) (string, error) {
// Validate filename (basename only — no traversal) before touching fs.
base := filepath.Base(opts.Filename)
if base != opts.Filename || base == "." || base == ".." || strings.ContainsAny(opts.Filename, `/\`) {
return "", fmt.Errorf("invalid filename %q", opts.Filename)
}
slug := opts.Slug
if slug == "" {
slug = datePrefix.ReplaceAllString(strings.TrimSuffix(base, ".md"), "")
}
// NotePath validates hall + wing + slug; do this before reading anything.
dest, err := brain.NotePath(brainDir, opts.Wing, opts.Hall, slug)
if err != nil {
return "", err
}
src := filepath.Join(brainDir, "raw", base)
raw, err := os.ReadFile(src)
if err != nil {
if os.IsNotExist(err) {
return "", fmt.Errorf("pending note %q does not exist in raw/", base)
}
return "", fmt.Errorf("read source: %w", err)
}
// Collision: never silently overwrite an existing promoted note.
if _, statErr := os.Stat(dest); statErr == nil {
rel, _ := filepath.Rel(brainDir, dest)
return "", fmt.Errorf("target %s already exists; choose a different slug", filepath.ToSlash(rel))
}
fm, body := parseFrontmatter(string(raw))
now := time.Now().UTC().Format(time.RFC3339)
fm.set("wing", brain.Sanitise(opts.Wing))
fm.set("hall", opts.Hall)
if fm.get("created_at") == "" {
fm.set("created_at", now)
}
fm.set("promoted_at", now)
if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
return "", fmt.Errorf("create wing dir: %w", err)
}
// Write-then-delete: the source survives any write failure.
if err := os.WriteFile(dest, []byte(fm.render()+body), 0o644); err != nil {
return "", fmt.Errorf("write promoted note: %w", err)
}
if err := os.Remove(src); err != nil {
return "", fmt.Errorf("promoted note written but source removal failed: %w", err)
}
rel, _ := filepath.Rel(brainDir, dest)
relSlash := filepath.ToSlash(rel)
// Best-effort wiki upkeep — the note is already promoted.
_ = brain.BuildWingIndex(brainDir, opts.Wing)
_ = brain.AutoTunnel(brainDir, relSlash, body)
return relSlash, nil
}
// excerpt returns the first n runes of s, trimmed, single-spaced.
func excerpt(s string, n int) string {
s = strings.TrimSpace(s)
r := []rune(s)
if len(r) > n {
r = r[:n]
}
return strings.TrimSpace(string(r))
}
+121
View File
@@ -0,0 +1,121 @@
package api
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func writeRaw(t *testing.T, brainDir, name, content string) {
t.Helper()
dir := filepath.Join(brainDir, "raw")
require.NoError(t, os.MkdirAll(dir, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, name), []byte(content), 0o644))
}
func TestListPendingEmptyWhenAbsent(t *testing.T) {
got, err := ListPending(t.TempDir())
require.NoError(t, err, "absent raw/ is not an error")
assert.Empty(t, got)
}
func TestListPendingReturnsOldestFirstWithExcerpt(t *testing.T) {
dir := t.TempDir()
writeRaw(t, dir, "2026-06-02-newer.md", "---\ncreated_at: 2026-06-02T00:00:00Z\n---\nNewer body here.\n")
writeRaw(t, dir, "2026-06-01-older.md", "---\ncreated_at: 2026-06-01T00:00:00Z\n---\nOlder body content.\n")
// non-md ignored
writeRaw(t, dir, "notes.txt", "ignore me")
got, err := ListPending(dir)
require.NoError(t, err)
require.Len(t, got, 2)
assert.Equal(t, "2026-06-01-older.md", got[0].Filename, "oldest first")
assert.Equal(t, "2026-06-02-newer.md", got[1].Filename)
assert.Contains(t, got[0].Excerpt, "Older body content")
assert.NotContains(t, got[0].Excerpt, "---", "excerpt is body, not frontmatter")
assert.Positive(t, got[0].SizeBytes)
}
func TestPromoteHappyPath(t *testing.T) {
dir := t.TempDir()
writeRaw(t, dir, "2026-06-01-lejpa-decision.md",
"---\ncreated_at: 2026-06-01T09:00:00Z\ncustom_field: keep-me\n---\n# LeJEPA\n\nbody.\n")
rel, err := PromoteNote(dir, PromoteOptions{
Filename: "2026-06-01-lejpa-decision.md", Wing: "jepa-fx", Hall: "decisions",
})
require.NoError(t, err)
assert.Equal(t, "wiki/jepa-fx/decisions/lejpa-decision.md", rel, "slug defaults to filename minus date prefix")
// Source deleted.
_, statErr := os.Stat(filepath.Join(dir, "raw", "2026-06-01-lejpa-decision.md"))
assert.True(t, os.IsNotExist(statErr), "source removed after promote")
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
s := string(got)
assert.Contains(t, s, "wing: jepa-fx")
assert.Contains(t, s, "hall: decisions")
assert.Contains(t, s, "created_at: 2026-06-01T09:00:00Z", "original created_at preserved")
assert.Contains(t, s, "promoted_at:")
assert.Contains(t, s, "custom_field: keep-me", "custom frontmatter preserved")
assert.Contains(t, s, "# LeJEPA")
}
func TestPromoteExplicitSlug(t *testing.T) {
dir := t.TempDir()
writeRaw(t, dir, "2026-06-01-x.md", "body\n")
rel, err := PromoteNote(dir, PromoteOptions{Filename: "2026-06-01-x.md", Wing: "a", Hall: "facts", Slug: "custom-slug"})
require.NoError(t, err)
assert.Equal(t, "wiki/a/facts/custom-slug.md", rel)
}
func TestPromoteInvalidHallErrorsBeforeTouchingFS(t *testing.T) {
dir := t.TempDir()
writeRaw(t, dir, "2026-06-01-x.md", "body\n")
_, err := PromoteNote(dir, PromoteOptions{Filename: "2026-06-01-x.md", Wing: "a", Hall: "garbage"})
require.Error(t, err)
// Source untouched.
_, statErr := os.Stat(filepath.Join(dir, "raw", "2026-06-01-x.md"))
assert.NoError(t, statErr, "invalid hall must not delete or move the source")
}
func TestPromoteMissingSourceErrors(t *testing.T) {
_, err := PromoteNote(t.TempDir(), PromoteOptions{Filename: "ghost.md", Wing: "a", Hall: "facts"})
require.Error(t, err)
}
func TestPromoteSlugCollisionNoOverwrite(t *testing.T) {
dir := t.TempDir()
// Pre-existing target.
dest := filepath.Join(dir, "wiki", "a", "facts", "x.md")
require.NoError(t, os.MkdirAll(filepath.Dir(dest), 0o755))
require.NoError(t, os.WriteFile(dest, []byte("EXISTING\n"), 0o644))
writeRaw(t, dir, "2026-06-01-x.md", "NEW\n")
_, err := PromoteNote(dir, PromoteOptions{Filename: "2026-06-01-x.md", Wing: "a", Hall: "facts"})
require.Error(t, err, "collision must error, not overwrite")
got, _ := os.ReadFile(dest)
assert.Equal(t, "EXISTING\n", string(got), "target not overwritten")
_, statErr := os.Stat(filepath.Join(dir, "raw", "2026-06-01-x.md"))
assert.NoError(t, statErr, "source preserved on collision (atomic: no delete without write)")
}
func TestPromoteRejectsTraversalFilename(t *testing.T) {
_, err := PromoteNote(t.TempDir(), PromoteOptions{Filename: "../escape.md", Wing: "a", Hall: "facts"})
require.Error(t, err)
}
func TestPromoteRebuildsWingIndex(t *testing.T) {
dir := t.TempDir()
writeRaw(t, dir, "2026-06-01-x.md", "---\ntitle: X Note\n---\nbody\n")
_, err := PromoteNote(dir, PromoteOptions{Filename: "2026-06-01-x.md", Wing: "a", Hall: "facts"})
require.NoError(t, err)
idx, err := os.ReadFile(filepath.Join(dir, "wiki", "a", "_index.md"))
require.NoError(t, err, "wing _index regenerated")
assert.Contains(t, string(idx), "x", "promoted note appears in the index")
}
+131
View File
@@ -0,0 +1,131 @@
package api
import (
"crypto/sha256"
"encoding/hex"
"fmt"
"os"
"path/filepath"
"strings"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
)
// ContentHash returns the lowercase hex sha256 of b. It is the note's
// content_hash handle: brain_write / brain_update return it, brain_get
// recomputes it from the file on disk, and brain_update stamps the prior
// note's hash into the new note's `supersedes` frontmatter.
func ContentHash(b []byte) string {
sum := sha256.Sum256(b)
return hex.EncodeToString(sum[:])
}
// resolveWithin maps a brainDir-relative path to an absolute path and
// guarantees it does not escape brainDir. Returns the cleaned relPath
// (forward-slashed) and the absolute path.
func resolveWithin(brainDir, relPath string) (rel, abs string, err error) {
clean := filepath.Clean("/" + filepath.ToSlash(relPath))
rel = strings.TrimPrefix(clean, "/")
abs = filepath.Join(brainDir, filepath.FromSlash(rel))
check, err := filepath.Rel(brainDir, abs)
if err != nil || check == ".." || strings.HasPrefix(check, ".."+string(filepath.Separator)) {
return "", "", fmt.Errorf("path %q escapes brain dir", relPath)
}
return rel, abs, nil
}
// UpdateNoteOptions identifies the note to supersede and supplies its new
// body. Path takes precedence; otherwise the target is resolved from
// Wing/Hall/Slug via brain.NotePath.
type UpdateNoteOptions struct {
Path string // brainDir-relative path; takes precedence over wing/hall/slug
Wing string
Hall string
Slug string
Content string // new full body (whole-note replace)
Reason string // optional; stamped as supersede_reason
}
// UpdateNote supersedes an existing note in place. It replaces the body
// with opts.Content, preserves the existing frontmatter (created_at,
// wing, hall, and any custom fields), and stamps updated_at, supersedes
// (the prior content hash), and supersede_reason (when given).
//
// It never creates: if the target does not exist, it returns an error so
// the caller can fall back to brain_write. Returns the note's relPath,
// the new content hash, and the prior content hash.
//
// Embeddings are NOT refreshed here. The rewritten file's mtime advances,
// which the mtime-driven vectorstore.Sync ticker uses to re-embed it on
// its next pass — the same out-of-band mechanism brain_write relies on.
func UpdateNote(brainDir string, opts UpdateNoteOptions) (relPath, contentHash, priorHash string, err error) {
if opts.Content == "" {
return "", "", "", fmt.Errorf("content is required")
}
var rel string
if opts.Path != "" {
rel = opts.Path
} else {
full, perr := brain.NotePath(brainDir, opts.Wing, opts.Hall, opts.Slug)
if perr != nil {
return "", "", "", perr
}
rel, _ = filepath.Rel(brainDir, full)
rel = filepath.ToSlash(rel)
}
rel, abs, err := resolveWithin(brainDir, rel)
if err != nil {
return "", "", "", err
}
prior, err := os.ReadFile(abs)
if err != nil {
if os.IsNotExist(err) {
return "", "", "", fmt.Errorf("note %q does not exist: use brain_write to create", rel)
}
return "", "", "", fmt.Errorf("read target: %w", err)
}
priorHash = ContentHash(prior)
fm, _ := parseFrontmatter(string(prior))
fm.set("updated_at", time.Now().UTC().Format(time.RFC3339))
fm.set("supersedes", priorHash)
if opts.Reason != "" {
fm.set("supersede_reason", opts.Reason)
}
out := []byte(fm.render() + opts.Content)
if err := os.WriteFile(abs, out, 0o644); err != nil {
return "", "", "", fmt.Errorf("write: %w", err)
}
return rel, ContentHash(out), priorHash, nil
}
// ReadNote reads the note at the brainDir-relative relPath and returns
// its parsed frontmatter, body, and content hash. It is the read-after-
// write primitive behind brain_get: the hash it returns equals the hash
// brain_write / brain_update returned for the same bytes.
func ReadNote(brainDir, relPath string) (fm map[string]string, body, contentHash string, err error) {
_, abs, err := resolveWithin(brainDir, relPath)
if err != nil {
return nil, "", "", err
}
raw, err := os.ReadFile(abs)
if err != nil {
if os.IsNotExist(err) {
return nil, "", "", fmt.Errorf("note %q does not exist", relPath)
}
return nil, "", "", fmt.Errorf("read note: %w", err)
}
parsed, body := parseFrontmatter(string(raw))
fm = make(map[string]string, len(parsed.lines))
for _, l := range parsed.lines {
if l.key != "" {
fm[l.key] = l.value
}
}
return fm, body, ContentHash(raw), nil
}
+130
View File
@@ -0,0 +1,130 @@
package api
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// seedNote writes a note directly to disk and returns its relPath.
func seedNote(t *testing.T, brainDir, rel, content string) string {
t.Helper()
full := filepath.Join(brainDir, filepath.FromSlash(rel))
require.NoError(t, os.MkdirAll(filepath.Dir(full), 0o755))
require.NoError(t, os.WriteFile(full, []byte(content), 0o644))
return rel
}
func TestUpdateNoteSupersedesAndStamps(t *testing.T) {
brainDir := t.TempDir()
rel := seedNote(t, brainDir, "wiki/jepa-fx/facts/val-vol.md",
"---\nwing: jepa-fx\nhall: facts\ncreated_at: 2026-01-01T00:00:00Z\ncustom: keep-me\n---\n# Old\n\nold body\n")
relPath, hash, priorHash, err := UpdateNote(brainDir, UpdateNoteOptions{
Path: rel,
Content: "# New\n\nnew body\n",
Reason: "facts changed",
})
require.NoError(t, err)
assert.Equal(t, rel, relPath)
assert.NotEmpty(t, hash)
assert.NotEmpty(t, priorHash)
assert.NotEqual(t, hash, priorHash)
got, err := os.ReadFile(filepath.Join(brainDir, filepath.FromSlash(rel)))
require.NoError(t, err)
s := string(got)
// Body replaced.
assert.Contains(t, s, "# New")
assert.NotContains(t, s, "old body")
// Prior fields preserved.
assert.Contains(t, s, "wing: jepa-fx")
assert.Contains(t, s, "hall: facts")
assert.Contains(t, s, "created_at: 2026-01-01T00:00:00Z")
assert.Contains(t, s, "custom: keep-me")
// Supersession stamped.
assert.Contains(t, s, "updated_at:")
assert.Contains(t, s, "supersedes: "+priorHash)
assert.Contains(t, s, "supersede_reason: facts changed")
}
func TestUpdateNoteResolvesByWingHallSlug(t *testing.T) {
brainDir := t.TempDir()
seedNote(t, brainDir, "wiki/jepa-fx/facts/val-vol.md",
"---\nwing: jepa-fx\nhall: facts\n---\nold\n")
relPath, _, _, err := UpdateNote(brainDir, UpdateNoteOptions{
Wing: "jepa-fx", Hall: "facts", Slug: "val-vol",
Content: "new\n",
})
require.NoError(t, err)
assert.Equal(t, "wiki/jepa-fx/facts/val-vol.md", relPath)
}
func TestUpdateNoteErrorsOnMissingAndDoesNotCreate(t *testing.T) {
brainDir := t.TempDir()
_, _, _, err := UpdateNote(brainDir, UpdateNoteOptions{
Wing: "jepa-fx", Hall: "facts", Slug: "ghost",
Content: "x\n",
})
require.Error(t, err)
assert.Contains(t, err.Error(), "does not exist")
// No file created.
_, statErr := os.Stat(filepath.Join(brainDir, "wiki/jepa-fx/facts/ghost.md"))
assert.True(t, os.IsNotExist(statErr), "missing-target update must not create a note")
}
func TestUpdateNoteRejectsTraversal(t *testing.T) {
brainDir := t.TempDir()
_, _, _, err := UpdateNote(brainDir, UpdateNoteOptions{
Path: "../escape.md",
Content: "x\n",
})
require.Error(t, err)
}
func TestReadNoteReturnsFrontmatterBodyHash(t *testing.T) {
brainDir := t.TempDir()
rel := seedNote(t, brainDir, "wiki/jepa-fx/facts/n.md",
"---\nwing: jepa-fx\nhall: facts\n---\n# Body\n\ntext\n")
fm, body, hash, err := ReadNote(brainDir, rel)
require.NoError(t, err)
assert.Equal(t, "jepa-fx", fm["wing"])
assert.Equal(t, "facts", fm["hall"])
assert.Equal(t, "# Body\n\ntext\n", body)
// Hash matches ContentHash of the raw bytes on disk (round-trip).
raw, _ := os.ReadFile(filepath.Join(brainDir, filepath.FromSlash(rel)))
assert.Equal(t, ContentHash(raw), hash)
}
func TestReadNoteRejectsTraversal(t *testing.T) {
brainDir := t.TempDir()
_, _, _, err := ReadNote(brainDir, "../../etc/passwd")
require.Error(t, err)
}
func TestUpdateThenReadRoundTripsHash(t *testing.T) {
brainDir := t.TempDir()
rel := seedNote(t, brainDir, "wiki/a/facts/n.md", "---\nwing: a\nhall: facts\n---\nold\n")
_, hash, _, err := UpdateNote(brainDir, UpdateNoteOptions{Path: rel, Content: "new\n"})
require.NoError(t, err)
_, _, readHash, err := ReadNote(brainDir, rel)
require.NoError(t, err)
assert.Equal(t, hash, readHash, "update content_hash must round-trip through ReadNote")
}
func TestContentHashStable(t *testing.T) {
assert.Equal(t, ContentHash([]byte("abc")), ContentHash([]byte("abc")))
assert.NotEqual(t, ContentHash([]byte("abc")), ContentHash([]byte("abd")))
assert.True(t, strings.HasPrefix(ContentHash([]byte("")), "")) // hex, non-panicking
}
+167
View File
@@ -0,0 +1,167 @@
package audit
import (
"bufio"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"os"
"path/filepath"
"sync"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
)
// FileBuffer is a durable, restart-surviving audit buffer backed by a
// JSONL file: one {id, entry} record per line. It is the internal/public
// tier fallback when loki is unreachable. Confirm rewrites the file
// without the confirmed record, so a record is cleared only after its
// central write is confirmed.
//
// Access is serialised by a mutex; the buffer is low-throughput (only
// written during a loki outage), so a whole-file rewrite on Confirm is
// acceptable and keeps the on-disk format trivially correct.
type FileBuffer struct {
path string
mu sync.Mutex
}
type bufferLine struct {
ID string `json:"id"`
Entry capture.AuditEntry `json:"entry"`
}
// NewFileBuffer returns a buffer backed by path. The parent directory is
// created if needed. The file itself is created lazily on first Append.
func NewFileBuffer(path string) (*FileBuffer, error) {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return nil, fmt.Errorf("create buffer dir: %w", err)
}
return &FileBuffer{path: path}, nil
}
// Writable reports whether the buffer file can be appended to. It probes
// by opening the file for append (creating it if absent) — the same
// operation Append performs — so Reserve's check matches Append's reality.
func (b *FileBuffer) Writable() error {
b.mu.Lock()
defer b.mu.Unlock()
f, err := os.OpenFile(b.path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o644)
if err != nil {
return err
}
return f.Close()
}
// Append durably writes one audit record. The ID is derived from the
// content + timestamp so it is stable and unique per record.
func (b *FileBuffer) Append(e capture.AuditEntry) error {
b.mu.Lock()
defer b.mu.Unlock()
line := bufferLine{ID: recordID(e), Entry: e}
data, err := json.Marshal(line)
if err != nil {
return fmt.Errorf("marshal buffer line: %w", err)
}
f, err := os.OpenFile(b.path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o644)
if err != nil {
return err
}
defer func() { _ = f.Close() }()
if _, err := f.Write(append(data, '\n')); err != nil {
return err
}
return f.Sync()
}
// Pending reads all buffered records. A missing file means none.
func (b *FileBuffer) Pending() ([]Buffered, error) {
b.mu.Lock()
defer b.mu.Unlock()
return b.readAllLocked()
}
func (b *FileBuffer) readAllLocked() ([]Buffered, error) {
f, err := os.Open(b.path)
if err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, err
}
defer func() { _ = f.Close() }()
var out []Buffered
sc := bufio.NewScanner(f)
sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
for sc.Scan() {
raw := sc.Bytes()
if len(raw) == 0 {
continue
}
var l bufferLine
if err := json.Unmarshal(raw, &l); err != nil {
return nil, fmt.Errorf("parse buffer line: %w", err)
}
out = append(out, Buffered(l))
}
return out, sc.Err()
}
// Confirm removes a single record after its central write is confirmed, by
// rewriting the file without it. Unknown IDs are a no-op.
func (b *FileBuffer) Confirm(id string) error {
b.mu.Lock()
defer b.mu.Unlock()
all, err := b.readAllLocked()
if err != nil {
return err
}
tmp := b.path + ".tmp"
f, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o644)
if err != nil {
return err
}
w := bufio.NewWriter(f)
kept := 0
for _, rec := range all {
if rec.ID == id {
continue
}
data, _ := json.Marshal(bufferLine(rec))
if _, err := w.Write(append(data, '\n')); err != nil {
_ = f.Close()
return err
}
kept++
}
if err := w.Flush(); err != nil {
_ = f.Close()
return err
}
if err := f.Sync(); err != nil {
_ = f.Close()
return err
}
if err := f.Close(); err != nil {
return err
}
// Empty buffer → remove the file entirely so Pending sees nothing.
if kept == 0 {
_ = os.Remove(tmp)
return os.Remove(b.path)
}
return os.Rename(tmp, b.path)
}
// recordID is a stable per-record identifier: sha256 of the principal,
// timestamp, and item list. Distinct captures never collide; the same
// buffered record always hashes the same.
func recordID(e capture.AuditEntry) string {
h := sha256.New()
_, _ = fmt.Fprintf(h, "%s|%s|%v|%s", e.Principal, e.Timestamp.UTC().Format("2006-01-02T15:04:05.000000000Z07:00"), e.Items, e.SessionRef)
return hex.EncodeToString(h.Sum(nil))[:16]
}
+96
View File
@@ -0,0 +1,96 @@
package audit
import (
"context"
"fmt"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
"github.com/mathiasbq/hyperguild/ingestion/internal/classification"
)
// Central is the central audit substrate (loki). Ready is a cheap
// reachability probe used by the pre-write reserve; Push writes a record.
type Central interface {
Ready(ctx context.Context) error
Push(ctx context.Context, e capture.AuditEntry) error
}
// Buffer is the durable local fallback for internal/public-tier records
// when the central sink is unreachable. It must survive process restart.
type Buffer interface {
// Writable reports whether the buffer can currently be appended to.
Writable() error
Append(e capture.AuditEntry) error
// Pending returns buffered records awaiting reconciliation, each with a
// stable ID used to Confirm (delete) it after a confirmed central write.
Pending() ([]Buffered, error)
Confirm(id string) error
}
// Buffered is a buffered audit record plus its stable buffer ID.
type Buffered struct {
ID string
Entry capture.AuditEntry
}
// Notifier raises an out-of-band alert (ntfy) about a degraded state.
type Notifier interface {
Notify(ctx context.Context, msg string) error
}
// DegradingSink is the classification-aware AuditSink (§4.4):
//
// - central reachable → AuditCentral (all tiers).
// - central down + confidential → refuse (no buffer): confidential must
// be centrally auditable at write time.
// - central down + internal/public + buffer writable → AuditBuffered.
// - central down + (confidential, or buffer not writable) → refuse (floor).
//
// The decision is made in Reserve, before any write; Record then executes it.
type DegradingSink struct {
central Central
buffer Buffer
notifier Notifier
}
// NewDegradingSink wires the central sink, durable buffer, and notifier.
func NewDegradingSink(central Central, buffer Buffer, notifier Notifier) *DegradingSink {
return &DegradingSink{central: central, buffer: buffer, notifier: notifier}
}
// Reserve decides, before any write, how the capture will be audited — or
// returns an error to refuse it.
func (d *DegradingSink) Reserve(ctx context.Context, level classification.Level) (capture.AuditOutcome, error) {
if err := d.central.Ready(ctx); err == nil {
return capture.AuditCentral, nil
}
// Central sink is down.
if level == classification.Confidential {
return 0, fmt.Errorf("confidential capture requires the central audit sink, which is unreachable")
}
if err := d.buffer.Writable(); err != nil {
// Floor: neither central nor local buffer can record the audit.
return 0, fmt.Errorf("audit floor: central sink down and local buffer unwritable: %w", err)
}
return capture.AuditBuffered, nil
}
// Record persists the entry per the reserved outcome. For AuditBuffered it
// also fires the degraded-state alert.
func (d *DegradingSink) Record(ctx context.Context, e capture.AuditEntry, outcome capture.AuditOutcome) error {
switch outcome {
case capture.AuditBuffered:
if err := d.buffer.Append(e); err != nil {
return fmt.Errorf("buffer audit record: %w", err)
}
// Best-effort alert; the record is already durably buffered.
if d.notifier != nil {
_ = d.notifier.Notify(ctx, fmt.Sprintf(
"capture audit BUFFERED LOCALLY (loki unreachable) — principal=%s class=%s items=%d",
e.Principal, e.EffectiveClassification, len(e.Items)))
}
return nil
default:
return d.central.Push(ctx, e)
}
}
+191
View File
@@ -0,0 +1,191 @@
package audit_test
import (
"context"
"errors"
"path/filepath"
"testing"
"github.com/mathiasbq/hyperguild/ingestion/internal/audit"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
"github.com/mathiasbq/hyperguild/ingestion/internal/classification"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// --- fakes ---
type fakeCentral struct {
down bool
pushed []capture.AuditEntry
pushErr error
}
func (f *fakeCentral) Ready(context.Context) error {
if f.down {
return errors.New("loki down")
}
return nil
}
func (f *fakeCentral) Push(_ context.Context, e capture.AuditEntry) error {
if f.pushErr != nil {
return f.pushErr
}
f.pushed = append(f.pushed, e)
return nil
}
type fakeNotifier struct{ msgs []string }
func (f *fakeNotifier) Notify(_ context.Context, msg string) error {
f.msgs = append(f.msgs, msg)
return nil
}
// unwritableBuffer always reports it cannot be written (floor condition).
type unwritableBuffer struct{}
func (unwritableBuffer) Writable() error { return errors.New("disk full") }
func (unwritableBuffer) Append(capture.AuditEntry) error { return errors.New("disk full") }
func (unwritableBuffer) Pending() ([]audit.Buffered, error) { return nil, nil }
func (unwritableBuffer) Confirm(string) error { return nil }
func newFileBuffer(t *testing.T) *audit.FileBuffer {
t.Helper()
b, err := audit.NewFileBuffer(filepath.Join(t.TempDir(), "audit-buffer.jsonl"))
require.NoError(t, err)
return b
}
func entry(principal string) capture.AuditEntry {
return capture.AuditEntry{Principal: principal, EffectiveClassification: "internal", Items: []string{"insight:x"}}
}
// --- Reserve: classification-aware decision ---
func TestReserveCentralUpGrantsCentral(t *testing.T) {
d := audit.NewDegradingSink(&fakeCentral{}, newFileBuffer(t), &fakeNotifier{})
for _, lvl := range []classification.Level{classification.Public, classification.Internal, classification.Confidential} {
out, err := d.Reserve(context.Background(), lvl)
require.NoError(t, err)
assert.Equal(t, capture.AuditCentral, out)
}
}
func TestReserveConfidentialSinkDownRefuses(t *testing.T) {
d := audit.NewDegradingSink(&fakeCentral{down: true}, newFileBuffer(t), &fakeNotifier{})
_, err := d.Reserve(context.Background(), classification.Confidential)
require.Error(t, err, "confidential + sink down → refuse, no buffer")
}
func TestReserveInternalSinkDownBuffers(t *testing.T) {
d := audit.NewDegradingSink(&fakeCentral{down: true}, newFileBuffer(t), &fakeNotifier{})
out, err := d.Reserve(context.Background(), classification.Internal)
require.NoError(t, err)
assert.Equal(t, capture.AuditBuffered, out)
}
func TestReserveFloorRefusesWhenNothingCanRecord(t *testing.T) {
d := audit.NewDegradingSink(&fakeCentral{down: true}, unwritableBuffer{}, &fakeNotifier{})
_, err := d.Reserve(context.Background(), classification.Internal)
require.Error(t, err, "central down AND buffer unwritable → floor refuse")
}
// --- Record: executes the reserved outcome ---
func TestRecordCentralPushes(t *testing.T) {
c := &fakeCentral{}
d := audit.NewDegradingSink(c, newFileBuffer(t), &fakeNotifier{})
require.NoError(t, d.Record(context.Background(), entry("p"), capture.AuditCentral))
assert.Len(t, c.pushed, 1)
}
func TestRecordBufferedAppendsAndNotifies(t *testing.T) {
buf := newFileBuffer(t)
nt := &fakeNotifier{}
d := audit.NewDegradingSink(&fakeCentral{down: true}, buf, nt)
require.NoError(t, d.Record(context.Background(), entry("p"), capture.AuditBuffered))
pending, err := buf.Pending()
require.NoError(t, err)
assert.Len(t, pending, 1)
assert.NotEmpty(t, nt.msgs, "degraded state alerts via ntfy")
}
// --- FileBuffer durability + Confirm ---
func TestFileBufferSurvivesRestart(t *testing.T) {
path := filepath.Join(t.TempDir(), "buf.jsonl")
b1, err := audit.NewFileBuffer(path)
require.NoError(t, err)
require.NoError(t, b1.Append(entry("p1")))
require.NoError(t, b1.Append(entry("p2")))
// "restart": a fresh FileBuffer over the same file sees the records.
b2, err := audit.NewFileBuffer(path)
require.NoError(t, err)
pending, err := b2.Pending()
require.NoError(t, err)
assert.Len(t, pending, 2)
}
func TestFileBufferConfirmRemovesOnlyThatRecord(t *testing.T) {
buf := newFileBuffer(t)
require.NoError(t, buf.Append(entry("keep")))
require.NoError(t, buf.Append(entry("drop")))
pending, _ := buf.Pending()
require.Len(t, pending, 2)
var dropID string
for _, p := range pending {
if p.Entry.Principal == "drop" {
dropID = p.ID
}
}
require.NoError(t, buf.Confirm(dropID))
after, _ := buf.Pending()
require.Len(t, after, 1)
assert.Equal(t, "keep", after[0].Entry.Principal)
}
// --- Reconcile ---
func TestReconcileReplaysAndClearsOnlyAfterConfirmedWrite(t *testing.T) {
buf := newFileBuffer(t)
require.NoError(t, buf.Append(entry("a")))
require.NoError(t, buf.Append(entry("b")))
c := &fakeCentral{} // up
nt := &fakeNotifier{}
n, err := audit.Reconcile(context.Background(), c, buf, nt)
require.NoError(t, err)
assert.Equal(t, 2, n)
assert.Len(t, c.pushed, 2, "buffered records replayed to central")
pending, _ := buf.Pending()
assert.Empty(t, pending, "buffer cleared after confirmed central writes")
}
func TestReconcileNoopWhenCentralDown(t *testing.T) {
buf := newFileBuffer(t)
require.NoError(t, buf.Append(entry("a")))
n, err := audit.Reconcile(context.Background(), &fakeCentral{down: true}, buf, &fakeNotifier{})
require.NoError(t, err)
assert.Equal(t, 0, n)
pending, _ := buf.Pending()
assert.Len(t, pending, 1, "records stay buffered while central is down")
}
func TestReconcileKeepsRecordWhenPushFails(t *testing.T) {
buf := newFileBuffer(t)
require.NoError(t, buf.Append(entry("a")))
// Ready ok but Push fails → record must remain buffered (not lost).
c := &fakeCentral{pushErr: errors.New("push rejected")}
n, err := audit.Reconcile(context.Background(), c, buf, &fakeNotifier{})
require.NoError(t, err)
assert.Equal(t, 0, n)
pending, _ := buf.Pending()
assert.Len(t, pending, 1)
}
+99
View File
@@ -0,0 +1,99 @@
package audit
import (
"bytes"
"context"
"encoding/json"
"fmt"
"net/http"
"strconv"
"strings"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
)
// LokiCentral pushes capture audit records to a Grafana Loki instance via
// its push API, and probes readiness via /ready. It is the central audit
// substrate behind DegradingSink.
type LokiCentral struct {
baseURL string
labels map[string]string
http *http.Client
}
// NewLokiCentral constructs a LokiCentral for the given base URL (e.g.
// http://loki:3100). Returns nil when baseURL is empty so callers can
// treat missing config as "no central sink" with a single nil check.
func NewLokiCentral(baseURL string) *LokiCentral {
if baseURL == "" {
return nil
}
return &LokiCentral{
baseURL: strings.TrimRight(baseURL, "/"),
labels: map[string]string{"service": "brain-capture", "kind": "audit"},
http: &http.Client{Timeout: 10 * time.Second},
}
}
// Ready probes Loki's readiness endpoint.
func (l *LokiCentral) Ready(ctx context.Context) error {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, l.baseURL+"/ready", nil)
if err != nil {
return err
}
resp, err := l.http.Do(req)
if err != nil {
return fmt.Errorf("loki not ready: %w", err)
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("loki not ready: status %d", resp.StatusCode)
}
return nil
}
// pushPayload is the Loki push API body: one stream, one entry whose line
// is the JSON-encoded audit record.
type pushPayload struct {
Streams []lokiStream `json:"streams"`
}
type lokiStream struct {
Stream map[string]string `json:"stream"`
Values [][2]string `json:"values"`
}
// Push writes one audit record to Loki as a structured log line.
func (l *LokiCentral) Push(ctx context.Context, e capture.AuditEntry) error {
line, err := json.Marshal(e)
if err != nil {
return fmt.Errorf("marshal audit entry: %w", err)
}
ts := e.Timestamp
if ts.IsZero() {
ts = time.Now()
}
body, err := json.Marshal(pushPayload{Streams: []lokiStream{{
Stream: l.labels,
Values: [][2]string{{strconv.FormatInt(ts.UTC().UnixNano(), 10), string(line)}},
}}})
if err != nil {
return err
}
req, err := http.NewRequestWithContext(ctx, http.MethodPost,
l.baseURL+"/loki/api/v1/push", bytes.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
resp, err := l.http.Do(req)
if err != nil {
return fmt.Errorf("loki push: %w", err)
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("loki push: status %d", resp.StatusCode)
}
return nil
}
@@ -0,0 +1,86 @@
package audit_test
import (
"context"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/mathiasbq/hyperguild/ingestion/internal/audit"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestLokiReadyAndPush(t *testing.T) {
var pushBody string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/ready":
w.WriteHeader(http.StatusOK)
case "/loki/api/v1/push":
b, _ := io.ReadAll(r.Body)
pushBody = string(b)
w.WriteHeader(http.StatusNoContent)
default:
w.WriteHeader(http.StatusNotFound)
}
}))
defer srv.Close()
c := audit.NewLokiCentral(srv.URL)
require.NotNil(t, c)
require.NoError(t, c.Ready(context.Background()))
err := c.Push(context.Background(), capture.AuditEntry{
Principal: "koala-cli", EffectiveClassification: "internal", Items: []string{"insight:x"},
})
require.NoError(t, err)
assert.Contains(t, pushBody, "streams")
assert.Contains(t, pushBody, "koala-cli", "audit entry serialised into the loki line")
}
func TestLokiReadyFailsWhenDown(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusServiceUnavailable)
}))
defer srv.Close()
require.Error(t, audit.NewLokiCentral(srv.URL).Ready(context.Background()))
}
func TestLokiNilWhenUnconfigured(t *testing.T) {
assert.Nil(t, audit.NewLokiCentral(""))
}
func TestNtfyNotify(t *testing.T) {
var gotBody, gotAuth string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
b, _ := io.ReadAll(r.Body)
gotBody = string(b)
gotAuth = r.Header.Get("Authorization")
w.WriteHeader(http.StatusOK)
}))
defer srv.Close()
n := audit.NewNtfyNotifier(srv.URL, "ntfy-token")
require.NotNil(t, n)
require.NoError(t, n.Notify(context.Background(), "audit buffered locally"))
assert.Contains(t, gotBody, "audit buffered locally")
assert.Equal(t, "Bearer ntfy-token", gotAuth)
}
func TestNtfyDoesNotLeakTokenOnError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
}))
defer srv.Close()
err := audit.NewNtfyNotifier(srv.URL, "secret-token").Notify(context.Background(), "x")
require.Error(t, err)
assert.False(t, strings.Contains(err.Error(), "secret-token"), "token must not leak into errors")
}
func TestNtfyNilWhenUnconfigured(t *testing.T) {
assert.Nil(t, audit.NewNtfyNotifier("", "tok"))
}
+55
View File
@@ -0,0 +1,55 @@
package audit
import (
"context"
"fmt"
"net/http"
"strings"
"time"
)
// NtfyNotifier posts alerts to an ntfy topic URL. Used to surface a
// degraded audit state (records buffered locally during a loki outage).
type NtfyNotifier struct {
topicURL string
token string
http *http.Client
}
// NewNtfyNotifier constructs a notifier for the given ntfy topic URL
// (e.g. https://ntfy.sh/my-topic). token is an optional bearer for
// protected ntfy instances; it is held here and only sent in the
// Authorization header, never logged. Returns nil when topicURL is empty.
func NewNtfyNotifier(topicURL, token string) *NtfyNotifier {
if topicURL == "" {
return nil
}
return &NtfyNotifier{
topicURL: strings.TrimRight(topicURL, "/"),
token: token,
http: &http.Client{Timeout: 10 * time.Second},
}
}
// Notify posts a message to the ntfy topic.
func (n *NtfyNotifier) Notify(ctx context.Context, msg string) error {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, n.topicURL, strings.NewReader(msg))
if err != nil {
return err
}
req.Header.Set("Title", "brain-capture audit degraded")
req.Header.Set("Priority", "high")
req.Header.Set("Tags", "warning,brain")
if n.token != "" {
req.Header.Set("Authorization", "Bearer "+n.token)
}
resp, err := n.http.Do(req)
if err != nil {
return fmt.Errorf("ntfy notify: %w", err)
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("ntfy notify: status %d", resp.StatusCode)
}
return nil
}
+65
View File
@@ -0,0 +1,65 @@
package audit
import (
"context"
"fmt"
"log/slog"
"time"
)
// Reconcile replays locally-buffered audit records to the central sink
// when it is reachable again. A record is removed from the buffer ONLY
// after its central write is confirmed, so a crash mid-reconcile re-plays
// rather than loses. Returns the number of records reconciled.
//
// A no-op (0, nil) when the central sink is still unreachable or the
// buffer is empty.
func Reconcile(ctx context.Context, central Central, buffer Buffer, notifier Notifier) (int, error) {
if err := central.Ready(ctx); err != nil {
return 0, nil // still down; try again next tick
}
pending, err := buffer.Pending()
if err != nil {
return 0, fmt.Errorf("read buffer: %w", err)
}
reconciled := 0
for _, rec := range pending {
if err := central.Push(ctx, rec.Entry); err != nil {
// Central went away mid-drain; stop and keep the rest buffered.
break
}
if err := buffer.Confirm(rec.ID); err != nil {
return reconciled, fmt.Errorf("confirm buffered record %s: %w", rec.ID, err)
}
reconciled++
}
if reconciled > 0 && notifier != nil {
_ = notifier.Notify(ctx, fmt.Sprintf("reconciled %d buffered capture audit record(s) to loki", reconciled))
}
return reconciled, nil
}
// StartReconcile runs Reconcile on a ticker until ctx is cancelled. It is
// the recovery half of the degrade-and-buffer path; pair it with a
// DegradingSink sharing the same buffer + central.
func StartReconcile(ctx context.Context, central Central, buffer Buffer, notifier Notifier, interval time.Duration) {
if interval <= 0 {
interval = time.Minute
}
go func() {
t := time.NewTicker(interval)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
if n, err := Reconcile(ctx, central, buffer, notifier); err != nil {
slog.Warn("audit reconcile failed", "err", err)
} else if n > 0 {
slog.Info("audit reconcile", "reconciled", n)
}
}
}
}()
}
+56
View File
@@ -0,0 +1,56 @@
// Package audit provides AuditSink implementations for the capture
// capability (I5). This file ships the minimal slog-backed sink used in
// #53: it emits the request-level audit record to structured logs, which
// the alloy/loki substrate already scrapes. The classification-aware
// degradation/refusal sink (confidential fails closed, internal buffers +
// reconciles) lands in #54 and replaces this behind the same interface.
package audit
import (
"context"
"log/slog"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
"github.com/mathiasbq/hyperguild/ingestion/internal/classification"
)
// SlogSink records audit entries to an slog.Logger. It never fails and is
// always centrally available, so its Reserve always grants AuditCentral —
// it does not exercise the I5 degradation/floor. That is DegradingSink's
// job (loki + durable buffer). SlogSink is the default for deployments
// without a loki endpoint configured. A nil logger ⇒ slog.Default().
type SlogSink struct {
logger *slog.Logger
}
// NewSlogSink constructs a SlogSink. nil logger ⇒ slog.Default().
func NewSlogSink(logger *slog.Logger) *SlogSink {
if logger == nil {
logger = slog.Default()
}
return &SlogSink{logger: logger}
}
// Reserve always grants central recording — slog is always available.
func (s *SlogSink) Reserve(_ context.Context, _ classification.Level) (capture.AuditOutcome, error) {
return capture.AuditCentral, nil
}
// Record emits the audit entry at info level. Security events, when
// present, are logged at warn level so they surface independently of the
// routine audit stream.
func (s *SlogSink) Record(_ context.Context, e capture.AuditEntry, _ capture.AuditOutcome) error {
s.logger.Info("capture audit",
"principal", e.Principal,
"actor", e.Actor,
"harness", e.Harness,
"session_ref", e.SessionRef,
"classification", e.EffectiveClassification,
"items", e.Items,
"ts", e.Timestamp,
)
for _, ev := range e.SecurityEvents {
s.logger.Warn("capture security event", "principal", e.Principal, "event", ev)
}
return nil
}
+41
View File
@@ -0,0 +1,41 @@
package audit_test
import (
"bytes"
"context"
"log/slog"
"testing"
"github.com/mathiasbq/hyperguild/ingestion/internal/audit"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestSlogSinkRecordsEntryAndSecurityEvents(t *testing.T) {
var buf bytes.Buffer
sink := audit.NewSlogSink(slog.New(slog.NewTextHandler(&buf, nil)))
err := sink.Record(context.Background(), capture.AuditEntry{
Principal: "koala-cli",
Harness: "claude-code",
EffectiveClassification: "confidential",
Items: []string{"insight:wiki/a/facts/x.md"},
SecurityEvents: []string{"asserted-vs-derived origin mismatch"},
}, capture.AuditCentral)
require.NoError(t, err)
out := buf.String()
assert.Contains(t, out, "capture audit")
assert.Contains(t, out, "koala-cli")
assert.Contains(t, out, "confidential")
assert.Contains(t, out, "capture security event")
assert.Contains(t, out, "asserted-vs-derived origin mismatch")
}
func TestSlogSinkNilLoggerDefaults(t *testing.T) {
// nil logger must not panic.
require.NotPanics(t, func() {
_ = audit.NewSlogSink(nil).Record(context.Background(), capture.AuditEntry{}, capture.AuditCentral)
})
}
+129
View File
@@ -0,0 +1,129 @@
// Package brainstore is the concrete BrainStore: the single shared
// implementation of the #45 write/update/get verbs, used by BOTH the MCP
// handlers and the capture use-case so there is one implementation, not
// two (the Clean-Architecture / DRY payoff of #51).
//
// It composes the file-level primitives in package api (WriteNote,
// UpdateNote, ReadNote — the read-after-write contract) with the wiki
// upkeep that must accompany a write: wing _index rebuild, cross-wing
// auto-tunnel, and graph re-index. Embedding refresh is intentionally
// out-of-band (mtime-driven vectorstore.Sync) and not triggered here —
// see the brain note on out-of-band sync.
package brainstore
import (
"context"
"log/slog"
"strings"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
"github.com/mathiasbq/hyperguild/ingestion/internal/graphsync"
)
// Store implements capture.BrainStore against a brain directory on disk,
// optionally re-indexing each write into the knowledge graph.
type Store struct {
brainDir string
graph graphsync.Store // nil = graph re-index disabled
}
// New constructs a Store bound to brainDir with graph indexing disabled.
func New(brainDir string) *Store {
return &Store{brainDir: brainDir}
}
// WithGraph enables graph re-index on every write/update. nil disables it.
func (s *Store) WithGraph(g graphsync.Store) *Store {
s.graph = g
return s
}
// Write creates a brain note and returns its read-after-write handle.
func (s *Store) Write(ctx context.Context, n capture.Note) (capture.Ref, error) {
relPath, err := api.WriteNote(s.brainDir, api.WriteNoteOptions{
Content: n.Content,
Filename: n.Filename,
Type: n.Type,
Domain: n.Domain,
Wing: n.Wing,
Hall: n.Hall,
})
if err != nil {
return capture.Ref{}, err
}
s.wikiUpkeep(relPath, n.Wing, n.Content)
s.indexInGraph(ctx, "brain_write", relPath)
_, _, hash, _ := api.ReadNote(s.brainDir, relPath)
return capture.Ref{ID: relPath, Path: relPath, ContentHash: hash}, nil
}
// Update supersedes an existing note in place. slug may be a bare slug
// (resolved against n.Wing/n.Hall) or a full brain-relative path (when it
// contains a slash). It never creates — a missing target is an error.
func (s *Store) Update(ctx context.Context, slug string, n capture.Note) (capture.Ref, error) {
opts := api.UpdateNoteOptions{Content: n.Content, Reason: n.Reason}
if strings.Contains(slug, "/") {
opts.Path = slug
} else {
opts.Wing, opts.Hall, opts.Slug = n.Wing, n.Hall, slug
}
relPath, hash, _, err := api.UpdateNote(s.brainDir, opts)
if err != nil {
return capture.Ref{}, err
}
if wing := wingFromRelPath(relPath); wing != "" {
s.wikiUpkeep(relPath, wing, n.Content)
}
s.indexInGraph(ctx, "brain_update", relPath)
return capture.Ref{ID: relPath, Path: relPath, ContentHash: hash, Superseded: true}, nil
}
// Get fetches a note by id/path — the read-after-write confirmation
// primitive (a direct fetch, never a semantic query).
func (s *Store) Get(_ context.Context, id string) (capture.StoredNote, error) {
fm, body, hash, err := api.ReadNote(s.brainDir, id)
if err != nil {
return capture.StoredNote{}, err
}
return capture.StoredNote{ID: id, Path: id, ContentHash: hash, Frontmatter: fm, Body: body}, nil
}
// wikiUpkeep rebuilds the wing _index and re-tunnels cross-wing matches
// when a note lands in the structured wiki. Both are best-effort: the
// note is already written, so a failure here is logged, not propagated.
func (s *Store) wikiUpkeep(relPath, wing, content string) {
if wing == "" {
return
}
if err := brain.BuildWingIndex(s.brainDir, wing); err != nil {
slog.Warn("brainstore: auto-index failed", "wing", wing, "err", err)
}
if err := brain.AutoTunnel(s.brainDir, relPath, content); err != nil {
slog.Warn("brainstore: auto-tunnel failed", "src", relPath, "err", err)
}
}
// indexInGraph re-indexes a written doc into the graph, best-effort.
func (s *Store) indexInGraph(ctx context.Context, op, relPath string) {
if s.graph == nil || relPath == "" {
return
}
if err := graphsync.IndexDoc(ctx, s.graph, s.brainDir, relPath); err != nil {
slog.Warn(op+": graph index failed", "path", relPath, "err", err)
}
}
// wingFromRelPath extracts the wing from a structured wiki path
// (wiki/<wing>/<hall>/<slug>.md). Returns "" for legacy/non-wiki paths.
func wingFromRelPath(relPath string) string {
parts := strings.Split(relPath, "/")
if len(parts) >= 4 && parts[0] == "wiki" {
return parts[1]
}
return ""
}
@@ -0,0 +1,85 @@
package brainstore_test
import (
"context"
"os"
"path/filepath"
"testing"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainstore"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestStoreWriteReturnsHandle(t *testing.T) {
dir := t.TempDir()
s := brainstore.New(dir)
ref, err := s.Write(context.Background(), capture.Note{
Content: "# X\n\nbody\n", Filename: "x", Wing: "a", Hall: "facts",
})
require.NoError(t, err)
assert.Equal(t, "wiki/a/facts/x.md", ref.Path)
assert.Equal(t, ref.Path, ref.ID)
assert.NotEmpty(t, ref.ContentHash)
assert.False(t, ref.Superseded)
_, err = os.Stat(filepath.Join(dir, "wiki/a/facts/x.md"))
require.NoError(t, err)
}
func TestStoreUpdateSupersedes(t *testing.T) {
dir := t.TempDir()
s := brainstore.New(dir)
_, err := s.Write(context.Background(), capture.Note{
Content: "old\n", Filename: "n", Wing: "a", Hall: "facts",
})
require.NoError(t, err)
ref, err := s.Update(context.Background(), "n", capture.Note{
Content: "new\n", Wing: "a", Hall: "facts", Reason: "changed",
})
require.NoError(t, err)
assert.True(t, ref.Superseded)
assert.Equal(t, "wiki/a/facts/n.md", ref.Path)
got, _ := os.ReadFile(filepath.Join(dir, "wiki/a/facts/n.md"))
assert.Contains(t, string(got), "new")
assert.Contains(t, string(got), "supersede_reason: changed")
}
func TestStoreUpdateByFullPath(t *testing.T) {
dir := t.TempDir()
s := brainstore.New(dir)
_, err := s.Write(context.Background(), capture.Note{Content: "old\n", Filename: "n", Wing: "a", Hall: "facts"})
require.NoError(t, err)
ref, err := s.Update(context.Background(), "wiki/a/facts/n.md", capture.Note{Content: "fresh\n"})
require.NoError(t, err)
assert.Equal(t, "wiki/a/facts/n.md", ref.Path)
}
func TestStoreUpdateMissingErrors(t *testing.T) {
dir := t.TempDir()
s := brainstore.New(dir)
_, err := s.Update(context.Background(), "ghost", capture.Note{Content: "x\n", Wing: "a", Hall: "facts"})
require.Error(t, err)
_, statErr := os.Stat(filepath.Join(dir, "wiki/a/facts/ghost.md"))
assert.True(t, os.IsNotExist(statErr), "update must not create")
}
func TestStoreGetRoundTripsHash(t *testing.T) {
dir := t.TempDir()
s := brainstore.New(dir)
ref, err := s.Write(context.Background(), capture.Note{
Content: "# Body\n\ntext\n", Filename: "n", Wing: "a", Hall: "facts",
})
require.NoError(t, err)
note, err := s.Get(context.Background(), ref.ID)
require.NoError(t, err)
assert.Equal(t, ref.ContentHash, note.ContentHash, "write→get hash round-trips")
assert.Equal(t, "a", note.Frontmatter["wing"])
assert.Contains(t, note.Body, "# Body")
}
+148
View File
@@ -0,0 +1,148 @@
// Package capture is the Clean-Architecture use-case for the uniform
// capture capability (issue #49/#51): persist a finished session's
// valuable output — insights → brain, action items → Gitea tickets,
// optional summary → ai-sessions — with one invocation, identical core
// behaviour across every harness.
//
// This package is pure orchestration. It depends only on ports
// (interfaces) and plain entities — no HTTP, no live Gitea, no embedding
// or audit I/O. The real adapters are wired in #52 (Gitea tracker), #53
// (REST + I1 origin gate), and #54/#55 (audit path + relay). The I1
// sovereignty refusal and the classification-aware audit degradation are
// deliberately NOT here — those need the server-derived principal origin
// (#53) and the loki/buffer machinery (#54). What lives here is everything
// testable against fakes: validation, effective-classification resolution
// (stricter wins), best-effort orchestration, and the partial receipt.
package capture
// Zone is the trust zone a capture originates from, server-derived from
// the authenticated principal (spec §4.2 / I1). It is NEVER taken from
// caller input — context.Harness is descriptive telemetry only.
type Zone int
const (
// ZoneUnknown means the origin was not set. The REST adapter always
// sets a concrete zone; the service treats Unknown as "not gated" (only
// an explicit ZoneUSNexus triggers the I1 refusal) so the gate can
// never fire on a caller-controllable default.
ZoneUnknown Zone = iota
// ZoneSovereign is sovereign soil (homelab / Tailscale CLI callers).
ZoneSovereign
// ZoneUSNexus is a non-sovereign US-jurisdiction surface (e.g.
// claude.ai). Confidential captures through it are refused (I1).
ZoneUSNexus
)
// String renders the zone for audit/refusal messages.
func (z Zone) String() string {
switch z {
case ZoneSovereign:
return "sovereign-soil"
case ZoneUSNexus:
return "us-nexus"
default:
return "unknown"
}
}
// CaptureContext is the per-session metadata accompanying a capture.
//
// Classification is the caller-declared sensitivity (model C, spec §4.1):
// the server independently derives the target's classification and gates
// on the stricter of the two. Principal and Origin are server-derived from
// the authenticated identity (the REST adapter populates them); they are
// never caller-asserted. Harness is descriptive telemetry only — never a
// gate input.
type CaptureContext struct {
Harness string
SessionRef string
Fidelity string
Actor string
Classification string // caller-declared level token ("" = unspecified)
Principal string // server-derived (auth); audit identity
Origin Zone // server-derived trust zone; the I1 gate input
}
// Insight is one piece of session knowledge bound for the brain. A
// non-empty SupersedeSlug routes to Update (revise in place); otherwise
// Write (create).
type Insight struct {
Text string
Wing string
Hall string
SupersedeSlug string
}
// Ticket is one action item bound for a Gitea repo. Owner is always the
// operator (set by the tracker adapter), never carried here.
type Ticket struct {
Repo string
Action string // create | close | comment
Number int // required for close/comment
Title string // required for create
Body string
}
// Summary is an optional session summary bound for ai-sessions.
type Summary struct {
Title string
Body string
ReposTouched []string
}
// CaptureInput is the whole capture request.
type CaptureInput struct {
Context CaptureContext
Insights []Insight
Tickets []Ticket
Summary *Summary
DryRun bool
}
// InsightResult is the per-insight outcome in the receipt.
type InsightResult struct {
ID string `json:"id,omitempty"`
Path string `json:"path,omitempty"`
ContentHash string `json:"content_hash,omitempty"`
Superseded bool `json:"superseded"`
OK bool `json:"ok"`
}
// TicketResult is the per-ticket outcome in the receipt.
type TicketResult struct {
Repo string `json:"repo"`
Number int `json:"number,omitempty"`
Action string `json:"action"`
URL string `json:"url,omitempty"`
OK bool `json:"ok"`
}
// SummaryResult is the summary outcome in the receipt.
type SummaryResult struct {
Path string `json:"path,omitempty"`
OK bool `json:"ok"`
}
// ItemError pins a failure to a specific request item for the partial
// receipt. Item is a stable locator like "insight[1]" or "ticket[0]".
type ItemError struct {
Item string `json:"item"`
Error string `json:"error"`
}
// CaptureReceipt is the structured, partial-aware result. Per-item ok
// flags plus a flat Errors list make partial success explicit; the
// caller never has to infer what landed.
type CaptureReceipt struct {
Insights []InsightResult `json:"insights"`
Tickets []TicketResult `json:"tickets"`
Summary *SummaryResult `json:"summary,omitempty"`
Errors []ItemError `json:"errors"`
EffectiveClassification string `json:"effective_classification,omitempty"`
DryRun bool `json:"dry_run"`
// AuditBuffered is true when the central audit sink was unreachable and
// this capture's audit record was written to the durable local buffer
// instead (internal/public tier). Surfaces the degraded state to the
// caller per §4.4.
AuditBuffered bool `json:"audit_buffered,omitempty"`
}
+126
View File
@@ -0,0 +1,126 @@
package capture
import (
"context"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/classification"
)
// Ref is the read-after-write handle returned by a brain write/update —
// the #45 contract. ContentHash lets the caller confirm what landed
// without a re-query; for an Update, Superseded is true.
type Ref struct {
ID string
Path string
ContentHash string
Superseded bool
}
// StoredNote is a brain note fetched by Get: the read-after-write
// confirmation primitive (a direct fetch, never a semantic query).
type StoredNote struct {
ID string
Path string
ContentHash string
Frontmatter map[string]string
Body string
}
// Note is the brain-write payload. It carries both the wing/hall taxonomy
// and the legacy type/domain fields so a single BrainStore serves both
// capture insights and the existing MCP brain_write surface. Reason is
// the supersede rationale, used only by Update.
type Note struct {
Content string
Filename string
Wing string
Hall string
Type string
Domain string
Reason string
}
// BrainStore is the brain persistence port — the shared implementation of
// the #45 write/update/get verbs that both the MCP handlers and capture
// call, so there is one implementation, not two. The read-after-write +
// staleness discipline lives behind this interface so no caller carries
// the rule.
type BrainStore interface {
Write(ctx context.Context, n Note) (Ref, error)
Update(ctx context.Context, slug string, n Note) (Ref, error)
Get(ctx context.Context, id string) (StoredNote, error)
}
// IssueRef identifies a ticket touched by the tracker.
type IssueRef struct {
Repo string
Number int
URL string
}
// IssueTracker is the Gitea ticket port. The implementation (#52) always
// scopes to owner "mathias"; the port deliberately omits owner.
type IssueTracker interface {
CreateIssue(ctx context.Context, repo, title, body string) (IssueRef, error)
// CloseIssue closes an issue, optionally posting a closing comment
// first (empty comment ⇒ close only).
CloseIssue(ctx context.Context, repo string, number int, comment string) (IssueRef, error)
CommentIssue(ctx context.Context, repo string, number int, body string) (IssueRef, error)
}
// SummaryWriter is the ai-sessions summary port.
type SummaryWriter interface {
WriteFile(ctx context.Context, repo, path, content string) error
}
// ClassificationPolicy derives a target's sensitivity (model C). The
// "stricter wins" combination of declared vs derived is use-case policy
// and lives in the service, so the port stays minimal. Satisfied by
// classification.Config (#50).
type ClassificationPolicy interface {
Derive(target classification.Target) classification.Level
}
// AuditEntry is the request-level audit record (I5): who/what captured
// what, when, via which principal. SecurityEvents carries anomalies such
// as a caller under-declaring sensitivity relative to the target floor.
type AuditEntry struct {
Timestamp time.Time
Principal string
Actor string
Harness string
SessionRef string
EffectiveClassification string
Items []string
SecurityEvents []string
}
// AuditOutcome is how a capture's audit record was (or will be) persisted.
type AuditOutcome int
const (
// AuditCentral means the record goes to the central sink (loki).
AuditCentral AuditOutcome = iota
// AuditBuffered means the central sink was unreachable and the record
// is written to a durable local buffer for later reconciliation
// (internal/public tier only).
AuditBuffered
)
// AuditSink is the two-phase, classification-aware audit port (I5, §4.4).
//
// Reserve runs BEFORE any write and decides whether the capture can be
// audited at its effective classification: it returns the outcome to use,
// or an error to refuse the capture before anything is written
// (confidential + central sink down → refuse; the all-tiers floor when
// nothing can record → refuse). Record runs AFTER the writes and persists
// the final entry per the reserved outcome.
//
// Splitting reserve from record is what lets "confidential + sink-down →
// refuse before any write" be literally true while the record itself
// (which lists what landed) is necessarily written afterwards.
type AuditSink interface {
Reserve(ctx context.Context, level classification.Level) (AuditOutcome, error)
Record(ctx context.Context, e AuditEntry, outcome AuditOutcome) error
}
+388
View File
@@ -0,0 +1,388 @@
package capture
import (
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"strings"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
"github.com/mathiasbq/hyperguild/ingestion/internal/classification"
)
// Service is the CaptureSession use-case. It depends only on ports.
type Service struct {
brain BrainStore
issues IssueTracker
summaries SummaryWriter
policy ClassificationPolicy
audit AuditSink
// now is the clock, injectable for deterministic summary paths and
// audit timestamps in tests.
now func() time.Time
}
// NewService constructs a Service from its ports. summaries may be nil
// when no summary persistence is wired; a CaptureInput with a Summary
// then fails that item rather than panicking.
func NewService(b BrainStore, tr IssueTracker, sw SummaryWriter, p ClassificationPolicy, a AuditSink) *Service {
return &Service{brain: b, issues: tr, summaries: sw, policy: p, audit: a, now: time.Now}
}
var validActions = map[string]bool{"create": true, "close": true, "comment": true}
// ErrSovereigntyRefused is returned when the I1 gate refuses a capture
// (confidential effective classification through a us-nexus origin). The
// REST adapter maps it to HTTP 403. Callers test with errors.Is.
var ErrSovereigntyRefused = fmt.Errorf("capture refused by I1 sovereignty gate")
// ErrAuditUnavailable is returned when the I5 audit gate refuses a capture
// before any write: a confidential capture whose central audit sink is
// unreachable, or the all-tiers floor where nothing can record the audit.
// The REST adapter maps it to HTTP 503. Callers test with errors.Is.
var ErrAuditUnavailable = fmt.Errorf("capture refused: audit substrate unavailable")
// assertedZoneMismatch returns a security-event string when the caller's
// harness label asserts a trust zone that contradicts the server-derived
// origin. A harness label that names no zone (the normal case, e.g.
// "claude-code") returns "". The label is never used as a gate input —
// this only flags the discrepancy for the audit trail.
func assertedZoneMismatch(harness string, derived Zone) string {
var asserted Zone
switch strings.ToLower(strings.TrimSpace(harness)) {
case "sovereign-soil", "sovereign":
asserted = ZoneSovereign
case "us-nexus", "usnexus":
asserted = ZoneUSNexus
default:
return "" // no zone claim
}
if asserted != derived {
return fmt.Sprintf("asserted-vs-derived origin mismatch: harness asserted %s, principal resolves to %s",
asserted, derived)
}
return ""
}
// Capture runs the use-case: validate (fail-closed), resolve effective
// classification (stricter of declared vs target-derived), then persist
// insights → tickets → summary best-effort, emit an audit record, and
// return a partial-aware receipt.
//
// A validation failure returns a non-nil error with nothing written. A
// per-item execution failure is recorded in the receipt (no rollback);
// the call still returns a nil error so the caller gets the partial
// receipt. The I1 origin gate and audit-down degradation are layered on
// by #53/#54 around this core.
func (s *Service) Capture(ctx context.Context, in CaptureInput) (CaptureReceipt, error) {
if err := s.validate(in); err != nil {
return CaptureReceipt{}, err
}
declared := classification.Public // unspecified ⇒ lowest ⇒ target floor governs
if in.Context.Classification != "" {
// Already validated parseable.
declared, _ = classification.ParseLevel(in.Context.Classification)
}
effective, securityEvents := s.resolveClassification(declared, in)
// Server-derived origin governs the I1 gate; a caller-asserted harness
// label that names a different zone is descriptive-only and logged as a
// security event (spec §4.2: a control keyed on attacker-suppliable
// input is not a control).
if ev := assertedZoneMismatch(in.Context.Harness, in.Context.Origin); ev != "" {
securityEvents = append(securityEvents, ev)
}
// I1 sovereignty gate: a confidential capture through a us-nexus origin
// is refused before ANY write. The refusal itself is audited (best
// effort) — refusals must be reconstructable too.
if effective == classification.Confidential && in.Context.Origin == ZoneUSNexus {
_ = s.audit.Record(ctx, AuditEntry{
Timestamp: s.now().UTC(),
Principal: in.Context.Principal,
Actor: in.Context.Actor,
Harness: in.Context.Harness,
SessionRef: in.Context.SessionRef,
EffectiveClassification: effective.String(),
Items: nil, // refused before any write
SecurityEvents: append(securityEvents, "I1 refusal: confidential capture via us-nexus origin"),
}, AuditCentral)
return CaptureReceipt{}, fmt.Errorf("%w: effective classification confidential through %s origin",
ErrSovereigntyRefused, in.Context.Origin)
}
receipt := CaptureReceipt{
Errors: []ItemError{},
EffectiveClassification: effective.String(),
DryRun: in.DryRun,
}
if in.DryRun {
// Would-be receipt: mark planned items ok, write nothing (not even
// audit — dry_run touches nothing).
for range in.Insights {
receipt.Insights = append(receipt.Insights, InsightResult{OK: true})
}
for _, tk := range in.Tickets {
receipt.Tickets = append(receipt.Tickets, TicketResult{Repo: tk.Repo, Action: tk.Action, Number: tk.Number, OK: true})
}
if in.Summary != nil {
receipt.Summary = &SummaryResult{Path: s.summaryPath(in.Context, in.Summary), OK: true}
}
return receipt, nil
}
// I5 audit gate: decide BEFORE any write whether this capture can be
// audited at its effective classification. Confidential + central sink
// down → refuse here, before writing anything; the all-tiers floor
// (nothing can record) likewise refuses. Internal/public degrade to the
// durable local buffer (signalled by AuditBuffered).
outcome, err := s.audit.Reserve(ctx, effective)
if err != nil {
return CaptureReceipt{}, fmt.Errorf("%w: %v", ErrAuditUnavailable, err)
}
var landed []string
for i, ins := range in.Insights {
res, item, err := s.persistInsight(ctx, ins)
receipt.Insights = append(receipt.Insights, res)
if err != nil {
receipt.Errors = append(receipt.Errors, ItemError{Item: fmt.Sprintf("insight[%d]", i), Error: err.Error()})
continue
}
landed = append(landed, item)
}
for i, tk := range in.Tickets {
res, err := s.persistTicket(ctx, tk)
receipt.Tickets = append(receipt.Tickets, res)
if err != nil {
receipt.Errors = append(receipt.Errors, ItemError{Item: fmt.Sprintf("ticket[%d]", i), Error: err.Error()})
continue
}
landed = append(landed, fmt.Sprintf("ticket:%s#%d", tk.Repo, res.Number))
}
if in.Summary != nil {
res, err := s.persistSummary(ctx, in.Context, in.Summary)
receipt.Summary = &res
if err != nil {
receipt.Errors = append(receipt.Errors, ItemError{Item: "summary", Error: err.Error()})
} else {
landed = append(landed, "summary:"+res.Path)
}
}
// I5: persist the request-level audit record of exactly what landed,
// using the outcome reserved before the writes. AuditBuffered surfaces
// the degraded (locally-buffered) state on the receipt.
if err := s.audit.Record(ctx, AuditEntry{
Timestamp: s.now().UTC(),
Principal: in.Context.Principal,
Actor: in.Context.Actor,
Harness: in.Context.Harness,
SessionRef: in.Context.SessionRef,
EffectiveClassification: effective.String(),
Items: landed,
SecurityEvents: securityEvents,
}, outcome); err != nil {
receipt.Errors = append(receipt.Errors, ItemError{Item: "audit", Error: err.Error()})
}
if outcome == AuditBuffered {
receipt.AuditBuffered = true
}
return receipt, nil
}
// validate enforces fail-closed structural validity over the whole
// request before any write. A bad declared classification, an invalid
// wing/hall, an empty insight, or a malformed ticket aborts the capture
// with nothing written.
func (s *Service) validate(in CaptureInput) error {
if in.Context.Classification != "" {
if _, err := classification.ParseLevel(in.Context.Classification); err != nil {
return fmt.Errorf("context.classification: %w", err)
}
}
for i, ins := range in.Insights {
if strings.TrimSpace(ins.Text) == "" {
return fmt.Errorf("insight[%d]: text is required", i)
}
if strings.TrimSpace(ins.Wing) == "" {
return fmt.Errorf("insight[%d]: wing is required", i)
}
if !brain.IsValidHall(ins.Hall) {
return fmt.Errorf("insight[%d]: invalid hall %q", i, ins.Hall)
}
}
for i, tk := range in.Tickets {
if strings.TrimSpace(tk.Repo) == "" {
return fmt.Errorf("ticket[%d]: repo is required", i)
}
if !validActions[tk.Action] {
return fmt.Errorf("ticket[%d]: invalid action %q (want create/close/comment)", i, tk.Action)
}
if tk.Action == "create" && strings.TrimSpace(tk.Title) == "" {
return fmt.Errorf("ticket[%d]: create requires a title", i)
}
if (tk.Action == "close" || tk.Action == "comment") && tk.Number <= 0 {
return fmt.Errorf("ticket[%d]: %s requires an issue number", i, tk.Action)
}
}
return nil
}
// resolveClassification computes the effective level (stricter of
// declared and every target's derived level) and collects a security
// event whenever the caller under-declared relative to a target floor.
func (s *Service) resolveClassification(declared classification.Level, in CaptureInput) (classification.Level, []string) {
effective := declared
var events []string
consider := func(kind classification.TargetKind, name string) {
derived := s.policy.Derive(classification.Target{Kind: kind, Name: name})
effective = classification.Stricter(effective, derived)
if declared < derived {
events = append(events, fmt.Sprintf("classification under-declared: declared=%s target=%s(%s) derived=%s",
declared, name, kindString(kind), derived))
}
}
for _, ins := range in.Insights {
consider(classification.WingTarget, ins.Wing)
}
for _, tk := range in.Tickets {
consider(classification.RepoTarget, tk.Repo)
}
if in.Summary != nil {
for _, repo := range in.Summary.ReposTouched {
consider(classification.RepoTarget, repo)
}
}
return effective, events
}
func (s *Service) persistInsight(ctx context.Context, ins Insight) (InsightResult, string, error) {
note := Note{Content: ins.Text, Wing: ins.Wing, Hall: ins.Hall, Filename: brain.Sanitise(firstLine(ins.Text))}
var ref Ref
var err error
if ins.SupersedeSlug != "" {
note.Reason = "superseded via capture"
ref, err = s.brain.Update(ctx, ins.SupersedeSlug, note)
} else {
ref, err = s.brain.Write(ctx, note)
}
if err != nil {
return InsightResult{OK: false, Superseded: ins.SupersedeSlug != ""}, "", err
}
return InsightResult{
ID: ref.ID, Path: ref.Path, ContentHash: ref.ContentHash,
Superseded: ref.Superseded, OK: true,
}, "insight:" + ref.ID, nil
}
func (s *Service) persistTicket(ctx context.Context, tk Ticket) (TicketResult, error) {
res := TicketResult{Repo: tk.Repo, Action: tk.Action, Number: tk.Number}
var ref IssueRef
var err error
switch tk.Action {
case "create":
ref, err = s.issues.CreateIssue(ctx, tk.Repo, tk.Title, tk.Body)
case "close":
ref, err = s.issues.CloseIssue(ctx, tk.Repo, tk.Number, tk.Body)
case "comment":
ref, err = s.issues.CommentIssue(ctx, tk.Repo, tk.Number, tk.Body)
}
if err != nil {
return res, err
}
if ref.Number != 0 {
res.Number = ref.Number
}
res.URL = ref.URL
res.OK = true
return res, nil
}
func (s *Service) persistSummary(ctx context.Context, c CaptureContext, sum *Summary) (SummaryResult, error) {
if s.summaries == nil {
return SummaryResult{OK: false}, fmt.Errorf("no summary writer configured")
}
path := s.summaryPath(c, sum)
content := s.renderSummary(c, sum)
repo := "ai-sessions"
if err := s.summaries.WriteFile(ctx, repo, path, content); err != nil {
return SummaryResult{Path: path, OK: false}, err
}
return SummaryResult{Path: path, OK: true}, nil
}
// summaryPath builds summaries/<harness>/<YYYY-MM>/<date>-<slug>-<ref8>.md.
// The ref8 disambiguator is derived from the session_ref (or the title
// when no ref is present) so distinct sessions never collide.
func (s *Service) summaryPath(c CaptureContext, sum *Summary) string {
t := s.now().UTC()
slug := brain.Sanitise(sum.Title)
if slug == "" {
slug = "summary"
}
seed := c.SessionRef
if seed == "" {
seed = sum.Title + sum.Body
}
sum8 := shortHash(seed)
return fmt.Sprintf("summaries/%s/%s/%s-%s-%s.md",
brain.Sanitise(c.Harness), t.Format("2006-01"), t.Format("2006-01-02"), slug, sum8)
}
// renderSummary stamps fidelity + session metadata into frontmatter so the
// richer-fidelity-supersedes-thinner collision rule has the data it needs.
func (s *Service) renderSummary(c CaptureContext, sum *Summary) string {
var b strings.Builder
b.WriteString("---\n")
fmt.Fprintf(&b, "title: %s\n", sum.Title)
fmt.Fprintf(&b, "harness: %s\n", c.Harness)
if c.SessionRef != "" {
fmt.Fprintf(&b, "session_ref: %s\n", c.SessionRef)
}
fmt.Fprintf(&b, "fidelity: %s\n", c.Fidelity)
fmt.Fprintf(&b, "captured_at: %s\n", s.now().UTC().Format(time.RFC3339))
if len(sum.ReposTouched) > 0 {
fmt.Fprintf(&b, "repos_touched: [%s]\n", strings.Join(sum.ReposTouched, ", "))
}
b.WriteString("---\n\n")
b.WriteString(sum.Body)
if !strings.HasSuffix(sum.Body, "\n") {
b.WriteByte('\n')
}
return b.String()
}
func kindString(k classification.TargetKind) string {
if k == classification.RepoTarget {
return "repo"
}
return "wing"
}
func firstLine(s string) string {
s = strings.TrimSpace(s)
if i := strings.IndexByte(s, '\n'); i >= 0 {
s = s[:i]
}
s = strings.TrimLeft(s, "# ")
if len(s) > 60 {
s = s[:60]
}
return s
}
func shortHash(s string) string {
sum := sha256.Sum256([]byte(s))
return hex.EncodeToString(sum[:])[:8]
}
+471
View File
@@ -0,0 +1,471 @@
package capture
import (
"context"
"errors"
"strings"
"testing"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/classification"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// --- fakes ---
type fakeBrain struct {
writes []Note
updates []Note
gets []string
failOn func(Note) error // nil = always succeed
hashSeq int
}
func (f *fakeBrain) ref(prefix string, n Note, superseded bool) Ref {
f.hashSeq++
path := "wiki/" + n.Wing + "/" + n.Hall + "/" + n.Filename + ".md"
return Ref{ID: path, Path: path, ContentHash: prefix + string(rune('0'+f.hashSeq)), Superseded: superseded}
}
func (f *fakeBrain) Write(_ context.Context, n Note) (Ref, error) {
if f.failOn != nil {
if err := f.failOn(n); err != nil {
return Ref{}, err
}
}
f.writes = append(f.writes, n)
return f.ref("w", n, false), nil
}
func (f *fakeBrain) Update(_ context.Context, slug string, n Note) (Ref, error) {
if f.failOn != nil {
if err := f.failOn(n); err != nil {
return Ref{}, err
}
}
n.Filename = slug
f.updates = append(f.updates, n)
return f.ref("u", n, true), nil
}
func (f *fakeBrain) Get(_ context.Context, id string) (StoredNote, error) {
f.gets = append(f.gets, id)
return StoredNote{ID: id, Path: id}, nil
}
type fakeTracker struct {
created []string
closed []int
comments []int
err error
}
func (f *fakeTracker) CreateIssue(_ context.Context, repo, title, _ string) (IssueRef, error) {
if f.err != nil {
return IssueRef{}, f.err
}
f.created = append(f.created, repo+":"+title)
return IssueRef{Repo: repo, Number: 100 + len(f.created), URL: "https://git/" + repo + "/issues/x"}, nil
}
func (f *fakeTracker) CloseIssue(_ context.Context, repo string, number int, _ string) (IssueRef, error) {
if f.err != nil {
return IssueRef{}, f.err
}
f.closed = append(f.closed, number)
return IssueRef{Repo: repo, Number: number}, nil
}
func (f *fakeTracker) CommentIssue(_ context.Context, repo string, number int, _ string) (IssueRef, error) {
if f.err != nil {
return IssueRef{}, f.err
}
f.comments = append(f.comments, number)
return IssueRef{Repo: repo, Number: number}, nil
}
type fakeSummary struct {
paths []string
content []string
err error
}
func (f *fakeSummary) WriteFile(_ context.Context, _, path, content string) error {
if f.err != nil {
return f.err
}
f.paths = append(f.paths, path)
f.content = append(f.content, content)
return nil
}
// fakePolicy derives from an explicit map; default Internal so tests pin
// behaviour without depending on the real defaulting.
type fakePolicy struct{ tags map[string]classification.Level }
func (p fakePolicy) Derive(t classification.Target) classification.Level {
if lvl, ok := p.tags[t.Name]; ok {
return lvl
}
return classification.Internal
}
type fakeAudit struct {
entries []AuditEntry
err error // Record error
reserveErr error // Reserve error (refuse before write)
reserveMode AuditOutcome
}
func (f *fakeAudit) Reserve(_ context.Context, _ classification.Level) (AuditOutcome, error) {
if f.reserveErr != nil {
return 0, f.reserveErr
}
return f.reserveMode, nil
}
func (f *fakeAudit) Record(_ context.Context, e AuditEntry, _ AuditOutcome) error {
if f.err != nil {
return f.err
}
f.entries = append(f.entries, e)
return nil
}
// --- helpers ---
func newSvc(b BrainStore, tr IssueTracker, sw SummaryWriter, p ClassificationPolicy, a AuditSink) *Service {
s := NewService(b, tr, sw, p, a)
s.now = func() time.Time { return time.Date(2026, 6, 22, 12, 0, 0, 0, time.UTC) }
return s
}
func baseCtx() CaptureContext {
return CaptureContext{Harness: "claude-code", Actor: "mathias", Principal: "mathias", Classification: "internal"}
}
// --- scenarios ---
func TestCaptureHappyPath(t *testing.T) {
b := &fakeBrain{}
tr := &fakeTracker{}
au := &fakeAudit{}
svc := newSvc(b, tr, nil, fakePolicy{}, au)
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
Insights: []Insight{
{Text: "a", Wing: "hyperguild", Hall: "decisions", SupersedeSlug: ""},
{Text: "b", Wing: "hyperguild", Hall: "facts"},
},
Tickets: []Ticket{{Repo: "hyperguild", Action: "create", Title: "do x", Body: "y"}},
})
require.NoError(t, err)
require.Len(t, rec.Insights, 2)
for _, r := range rec.Insights {
assert.True(t, r.OK)
assert.NotEmpty(t, r.ContentHash, "read-after-write hash returned")
}
require.Len(t, rec.Tickets, 1)
assert.True(t, rec.Tickets[0].OK)
assert.Equal(t, 2, len(b.writes))
assert.Empty(t, rec.Errors)
// Audit emitted naming principal/harness + items that landed.
require.Len(t, au.entries, 1)
assert.Equal(t, "mathias", au.entries[0].Principal)
assert.Equal(t, "claude-code", au.entries[0].Harness)
assert.Len(t, au.entries[0].Items, 3)
}
func TestCaptureSupersedeNotDuplicate(t *testing.T) {
b := &fakeBrain{}
svc := newSvc(b, &fakeTracker{}, nil, fakePolicy{}, &fakeAudit{})
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
Insights: []Insight{{Text: "revised", Wing: "hyperguild", Hall: "facts", SupersedeSlug: "prior-note"}},
})
require.NoError(t, err)
assert.Empty(t, b.writes, "supersede must not create")
require.Len(t, b.updates, 1)
assert.Equal(t, "prior-note", b.updates[0].Filename)
assert.True(t, rec.Insights[0].Superseded)
}
func TestCaptureValidationFailClosed(t *testing.T) {
b := &fakeBrain{}
tr := &fakeTracker{}
au := &fakeAudit{}
svc := newSvc(b, tr, nil, fakePolicy{}, au)
_, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
Insights: []Insight{
{Text: "ok", Wing: "hyperguild", Hall: "facts"},
{Text: "bad", Wing: "hyperguild", Hall: "garbage-hall"}, // invalid hall
},
Tickets: []Ticket{{Repo: "hyperguild", Action: "create", Title: "t"}},
})
require.Error(t, err)
// Nothing written anywhere.
assert.Empty(t, b.writes)
assert.Empty(t, b.updates)
assert.Empty(t, tr.created)
assert.Empty(t, au.entries)
}
func TestCaptureValidationRejectsBadTicket(t *testing.T) {
svc := newSvc(&fakeBrain{}, &fakeTracker{}, nil, fakePolicy{}, &fakeAudit{})
_, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
Tickets: []Ticket{{Repo: "hyperguild", Action: "frobnicate"}}, // bad action
})
require.Error(t, err)
_, err = svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
Tickets: []Ticket{{Repo: "hyperguild", Action: "close"}}, // close needs number
})
require.Error(t, err)
}
func TestCapturePartialFailureBestEffort(t *testing.T) {
b := &fakeBrain{failOn: func(n Note) error {
if strings.Contains(n.Content, "FAIL") {
return errors.New("disk full")
}
return nil
}}
tr := &fakeTracker{}
au := &fakeAudit{}
svc := newSvc(b, tr, nil, fakePolicy{}, au)
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
Insights: []Insight{
{Text: "good one", Wing: "hyperguild", Hall: "facts"},
{Text: "FAIL here", Wing: "hyperguild", Hall: "facts"},
},
Tickets: []Ticket{{Repo: "hyperguild", Action: "create", Title: "t"}},
})
require.NoError(t, err, "partial failure is not a request-level error")
assert.True(t, rec.Insights[0].OK)
assert.False(t, rec.Insights[1].OK)
assert.True(t, rec.Tickets[0].OK, "ticket still persisted; no rollback")
require.Len(t, rec.Errors, 1)
assert.Equal(t, "insight[1]", rec.Errors[0].Item)
// Audit reflects exactly what landed: 1 insight + 1 ticket.
require.Len(t, au.entries, 1)
assert.Len(t, au.entries[0].Items, 2)
}
func TestCaptureDryRunWritesNothing(t *testing.T) {
b := &fakeBrain{}
tr := &fakeTracker{}
au := &fakeAudit{}
svc := newSvc(b, tr, nil, fakePolicy{}, au)
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
DryRun: true,
Insights: []Insight{{Text: "a", Wing: "hyperguild", Hall: "facts"}},
Tickets: []Ticket{{Repo: "hyperguild", Action: "create", Title: "t"}},
})
require.NoError(t, err)
assert.True(t, rec.DryRun)
assert.Len(t, rec.Insights, 1)
assert.True(t, rec.Insights[0].OK, "would-be receipt marks planned items ok")
// Nothing written anywhere, including audit.
assert.Empty(t, b.writes)
assert.Empty(t, tr.created)
assert.Empty(t, au.entries)
}
func TestCaptureStricterClassificationWins(t *testing.T) {
// Caller declares internal; target wing tagged confidential → effective confidential + security event.
b := &fakeBrain{}
au := &fakeAudit{}
pol := fakePolicy{tags: map[string]classification.Level{"client-seb": classification.Confidential}}
svc := newSvc(b, &fakeTracker{}, nil, pol, au)
ctx := baseCtx()
ctx.Classification = "internal"
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: ctx,
Insights: []Insight{{Text: "x", Wing: "client-seb", Hall: "facts"}},
})
require.NoError(t, err)
assert.Equal(t, "confidential", rec.EffectiveClassification)
require.Len(t, au.entries, 1)
assert.NotEmpty(t, au.entries[0].SecurityEvents, "under-declaration logged as security event")
assert.Equal(t, "confidential", au.entries[0].EffectiveClassification)
}
func TestCaptureCallerRaisingSensitivityHonoured(t *testing.T) {
// Caller declares confidential; target internal → effective confidential, NOT a security event.
au := &fakeAudit{}
pol := fakePolicy{tags: map[string]classification.Level{"hyperguild": classification.Internal}}
svc := newSvc(&fakeBrain{}, &fakeTracker{}, nil, pol, au)
ctx := baseCtx()
ctx.Classification = "confidential"
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: ctx,
Insights: []Insight{{Text: "x", Wing: "hyperguild", Hall: "facts"}},
})
require.NoError(t, err)
assert.Equal(t, "confidential", rec.EffectiveClassification)
assert.Empty(t, au.entries[0].SecurityEvents, "raising sensitivity is honoured, not flagged")
}
func TestCaptureSummaryPathAndFidelity(t *testing.T) {
sw := &fakeSummary{}
svc := newSvc(&fakeBrain{}, &fakeTracker{}, sw, fakePolicy{}, &fakeAudit{})
ctx := baseCtx()
ctx.Fidelity = "transcript-parse"
ctx.SessionRef = "abc123def456"
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: ctx,
Summary: &Summary{Title: "Session Wrap", Body: "did stuff", ReposTouched: []string{"hyperguild"}},
})
require.NoError(t, err)
require.NotNil(t, rec.Summary)
assert.True(t, rec.Summary.OK)
require.Len(t, sw.paths, 1)
assert.True(t, strings.HasPrefix(sw.paths[0], "summaries/claude-code/2026-06/"), "path: %s", sw.paths[0])
assert.Contains(t, sw.paths[0], "session-wrap")
assert.Contains(t, sw.content[0], "fidelity: transcript-parse", "fidelity stamped in frontmatter")
}
// --- I1 sovereignty gate (#53) ---
func TestCaptureRefusesConfidentialViaUSNexus(t *testing.T) {
b := &fakeBrain{}
tr := &fakeTracker{}
au := &fakeAudit{}
pol := fakePolicy{tags: map[string]classification.Level{"client-seb": classification.Confidential}}
svc := newSvc(b, tr, nil, pol, au)
ctx := baseCtx()
ctx.Classification = "confidential"
ctx.Origin = ZoneUSNexus
_, err := svc.Capture(context.Background(), CaptureInput{
Context: ctx,
Insights: []Insight{{Text: "x", Wing: "client-seb", Hall: "facts"}},
})
require.Error(t, err)
assert.ErrorIs(t, err, ErrSovereigntyRefused)
// Refused before any write.
assert.Empty(t, b.writes)
assert.Empty(t, tr.created)
// Refusal is audited.
require.Len(t, au.entries, 1)
assert.Empty(t, au.entries[0].Items, "no items landed on refusal")
}
func TestCaptureAllowsConfidentialViaSovereign(t *testing.T) {
b := &fakeBrain{}
pol := fakePolicy{tags: map[string]classification.Level{"client-seb": classification.Confidential}}
svc := newSvc(b, &fakeTracker{}, nil, pol, &fakeAudit{})
ctx := baseCtx()
ctx.Classification = "confidential"
ctx.Origin = ZoneSovereign
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: ctx,
Insights: []Insight{{Text: "x", Wing: "client-seb", Hall: "facts"}},
})
require.NoError(t, err)
assert.True(t, rec.Insights[0].OK)
assert.Len(t, b.writes, 1)
}
func TestCaptureAssertedLabelIgnoredAndLogged(t *testing.T) {
// Caller asserts harness "sovereign-soil" but principal resolves to
// us-nexus; confidential ⇒ refused, and the discrepancy is a security event.
au := &fakeAudit{}
pol := fakePolicy{tags: map[string]classification.Level{"client-seb": classification.Confidential}}
svc := newSvc(&fakeBrain{}, &fakeTracker{}, nil, pol, au)
ctx := baseCtx()
ctx.Harness = "sovereign-soil" // asserted
ctx.Origin = ZoneUSNexus // server-derived
ctx.Classification = "confidential"
_, err := svc.Capture(context.Background(), CaptureInput{
Context: ctx,
Insights: []Insight{{Text: "x", Wing: "client-seb", Hall: "facts"}},
})
require.ErrorIs(t, err, ErrSovereigntyRefused)
require.Len(t, au.entries, 1)
joined := strings.Join(au.entries[0].SecurityEvents, " | ")
assert.Contains(t, joined, "asserted-vs-derived origin mismatch")
assert.Contains(t, joined, "I1 refusal")
}
func TestCaptureInternalViaUSNexusAllowed(t *testing.T) {
// us-nexus origin is fine for non-confidential data.
b := &fakeBrain{}
svc := newSvc(b, &fakeTracker{}, nil, fakePolicy{}, &fakeAudit{})
ctx := baseCtx()
ctx.Origin = ZoneUSNexus // internal classification, so gate doesn't fire
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: ctx,
Insights: []Insight{{Text: "x", Wing: "hyperguild", Hall: "facts"}},
})
require.NoError(t, err)
assert.True(t, rec.Insights[0].OK)
}
// --- I5 audit gate (#54) ---
func TestCaptureRefusesWhenAuditReserveFails(t *testing.T) {
// Reserve refusing (e.g. confidential + central sink down, or the floor)
// aborts the capture before any write.
b := &fakeBrain{}
tr := &fakeTracker{}
au := &fakeAudit{reserveErr: errors.New("central sink unreachable")}
svc := newSvc(b, tr, nil, fakePolicy{}, au)
_, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
Insights: []Insight{{Text: "x", Wing: "hyperguild", Hall: "facts"}},
})
require.Error(t, err)
assert.ErrorIs(t, err, ErrAuditUnavailable)
assert.Empty(t, b.writes, "nothing written when audit unavailable")
assert.Empty(t, tr.created)
}
func TestCaptureFlagsLocallyBufferedAudit(t *testing.T) {
// Reserve returns AuditBuffered (internal/public, central down) → capture
// proceeds and the receipt flags the degraded audit state.
b := &fakeBrain{}
au := &fakeAudit{reserveMode: AuditBuffered}
svc := newSvc(b, &fakeTracker{}, nil, fakePolicy{}, au)
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
Insights: []Insight{{Text: "x", Wing: "hyperguild", Hall: "facts"}},
})
require.NoError(t, err)
assert.True(t, rec.Insights[0].OK, "capture proceeds on degraded audit")
assert.True(t, rec.AuditBuffered, "receipt flags locally-buffered audit")
require.Len(t, au.entries, 1)
}
func TestCaptureDryRunSkipsAuditGate(t *testing.T) {
// dry_run must not even probe the audit sink (writes nothing anywhere).
au := &fakeAudit{reserveErr: errors.New("would refuse")}
svc := newSvc(&fakeBrain{}, &fakeTracker{}, nil, fakePolicy{}, au)
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
DryRun: true,
Insights: []Insight{{Text: "x", Wing: "hyperguild", Hall: "facts"}},
})
require.NoError(t, err, "dry-run does not hit the audit gate")
assert.True(t, rec.DryRun)
assert.Empty(t, au.entries)
}
+232
View File
@@ -0,0 +1,232 @@
// Package capturehttp is the REST adapter for the capture use-case: the
// POST /capture door (#53). It is deliberately thin — authenticate, derive
// the trust-zone origin from the authenticated principal, decode the
// request, call capture.Service, map the receipt to an HTTP status. No
// business logic lives here; the I1 gate, validation, and orchestration
// are all in the use-case.
package capturehttp
import (
"context"
"crypto/subtle"
"encoding/json"
"errors"
"io"
"net/http"
"strings"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
)
// Validator validates a Bearer JWT and returns its subject. The chassis
// *auth.JWTValidator satisfies it (including its nil-receiver "disabled"
// behaviour), and tests can substitute a fake without a live JWKS.
type Validator interface {
Validate(ctx context.Context, rawToken string) (string, error)
}
// Handler serves POST /capture.
type Handler struct {
svc *capture.Service
validator Validator // nil ⇒ JWT auth disabled
staticToken string // "" ⇒ static auth disabled
staticPrincipal string // principal name attributed to static-token callers
resolver OriginResolver
}
// New constructs a capture HTTP handler. staticToken callers are
// attributed to staticPrincipal (a sovereign homelab identity); JWT
// callers are attributed to their token subject.
func New(svc *capture.Service, validator Validator, staticToken, staticPrincipal string, resolver OriginResolver) *Handler {
if staticPrincipal == "" {
staticPrincipal = "local-cli"
}
return &Handler{
svc: svc,
validator: validator,
staticToken: staticToken,
staticPrincipal: staticPrincipal,
resolver: resolver,
}
}
// wire types — the POST /capture request body.
type request struct {
Context contextBody `json:"context"`
Insights []insightBody `json:"insights"`
Tickets []ticketBody `json:"tickets"`
Summary *summaryBody `json:"summary,omitempty"`
DryRun bool `json:"dry_run"`
}
type contextBody struct {
Harness string `json:"harness"`
SessionRef string `json:"session_ref"`
Fidelity string `json:"fidelity"`
Actor string `json:"actor"`
Classification string `json:"classification"`
}
type insightBody struct {
Text string `json:"text"`
Wing string `json:"wing"`
Hall string `json:"hall"`
SupersedeSlug string `json:"supersede_slug,omitempty"`
}
type ticketBody struct {
Repo string `json:"repo"`
Action string `json:"action"`
Number int `json:"number,omitempty"`
Title string `json:"title,omitempty"`
Body string `json:"body,omitempty"`
}
type summaryBody struct {
Title string `json:"title"`
Body string `json:"body"`
ReposTouched []string `json:"repos_touched,omitempty"`
}
// ServeHTTP authenticates, derives origin, runs the use-case, and maps the
// result to an HTTP status.
func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
principal, viaStatic, ok := Authenticate(r, h.staticToken, h.staticPrincipal, h.validator)
if !ok {
http.Error(w, "unauthorized", http.StatusUnauthorized)
return
}
body, err := io.ReadAll(r.Body)
if err != nil {
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "read body"})
return
}
in, err := DecodeRequest(body)
if err != nil {
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid JSON"})
return
}
// Principal and origin are server-derived — overwrite anything the
// caller may have tried to put in the body.
in.Context.Principal = principal
in.Context.Origin = h.resolver.Resolve(principal, viaStatic)
rec, err := h.svc.Capture(r.Context(), in)
switch {
case errors.Is(err, capture.ErrSovereigntyRefused):
writeJSON(w, http.StatusForbidden, map[string]string{"error": err.Error()})
return
case errors.Is(err, capture.ErrAuditUnavailable):
// I5 refusal: confidential + audit sink down, or the all-tiers floor.
writeJSON(w, http.StatusServiceUnavailable, map[string]string{"error": err.Error()})
return
case err != nil:
// Pre-write validation failure (fail-closed).
writeJSON(w, http.StatusBadRequest, map[string]string{"error": err.Error()})
return
}
writeJSON(w, statusFor(rec), rec)
}
// Authenticate mirrors the chassis Bearer precedence (static token wins,
// then Dex JWT) and returns the resolved principal plus whether the static
// path was taken — the chassis middleware hides both, and capture (REST or
// MCP) needs them to derive the trust-zone origin. ok is false when no
// credential matched.
func Authenticate(r *http.Request, staticToken, staticPrincipal string, validator Validator) (principal string, viaStatic, ok bool) {
raw, found := strings.CutPrefix(r.Header.Get("Authorization"), "Bearer ")
if !found || raw == "" {
return "", false, false
}
if staticToken != "" && subtle.ConstantTimeCompare([]byte(raw), []byte(staticToken)) == 1 {
return staticPrincipal, true, true
}
if validator != nil {
if sub, err := validator.Validate(r.Context(), raw); err == nil && sub != "" {
return sub, false, true
}
}
return "", false, false
}
// DecodeRequest parses a capture request body into a CaptureInput. Shared
// by the REST adapter and the MCP capture tool so the wire shape has one
// definition. Principal and Origin are NOT set here — the caller sets them
// from the authenticated identity.
func DecodeRequest(data []byte) (capture.CaptureInput, error) {
var b request
if err := json.Unmarshal(data, &b); err != nil {
return capture.CaptureInput{}, err
}
return b.toInput(), nil
}
func (b request) toInput() capture.CaptureInput {
in := capture.CaptureInput{
Context: capture.CaptureContext{
Harness: b.Context.Harness,
SessionRef: b.Context.SessionRef,
Fidelity: b.Context.Fidelity,
Actor: b.Context.Actor,
Classification: b.Context.Classification,
},
DryRun: b.DryRun,
}
for _, i := range b.Insights {
in.Insights = append(in.Insights, capture.Insight{
Text: i.Text, Wing: i.Wing, Hall: i.Hall, SupersedeSlug: i.SupersedeSlug,
})
}
for _, t := range b.Tickets {
in.Tickets = append(in.Tickets, capture.Ticket{
Repo: t.Repo, Action: t.Action, Number: t.Number, Title: t.Title, Body: t.Body,
})
}
if b.Summary != nil {
in.Summary = &capture.Summary{
Title: b.Summary.Title, Body: b.Summary.Body, ReposTouched: b.Summary.ReposTouched,
}
}
return in
}
// statusFor maps a receipt to an HTTP status: 200 all-ok (or dry-run),
// 207 partial, 502 everything-failed.
func statusFor(rec capture.CaptureReceipt) int {
if rec.DryRun {
return http.StatusOK
}
var ok, fail int
for _, i := range rec.Insights {
count(&ok, &fail, i.OK)
}
for _, t := range rec.Tickets {
count(&ok, &fail, t.OK)
}
if rec.Summary != nil {
count(&ok, &fail, rec.Summary.OK)
}
switch {
case fail == 0:
return http.StatusOK
case ok == 0:
return http.StatusBadGateway // every persistence attempt failed
default:
return http.StatusMultiStatus // 207: partial success
}
}
func count(ok, fail *int, isOK bool) {
if isOK {
*ok++
} else {
*fail++
}
}
func writeJSON(w http.ResponseWriter, status int, v any) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(v)
}
@@ -0,0 +1,198 @@
package capturehttp_test
import (
"bytes"
"context"
"encoding/json"
"errors"
"net/http"
"net/http/httptest"
"testing"
"github.com/mathiasbq/hyperguild/ingestion/internal/audit"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainstore"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
"github.com/mathiasbq/hyperguild/ingestion/internal/capturehttp"
"github.com/mathiasbq/hyperguild/ingestion/internal/classification"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
const staticTok = "static-secret"
// fakeValidator stands in for the chassis JWT validator.
type fakeValidator struct {
subject string
err error
}
func (f fakeValidator) Validate(context.Context, string) (string, error) {
return f.subject, f.err
}
type fakeTracker struct{ failCreate bool }
func (f fakeTracker) CreateIssue(context.Context, string, string, string) (capture.IssueRef, error) {
if f.failCreate {
return capture.IssueRef{}, errors.New("gitea down")
}
return capture.IssueRef{Repo: "hyperguild", Number: 1, URL: "https://git/1"}, nil
}
func (fakeTracker) CloseIssue(context.Context, string, int, string) (capture.IssueRef, error) {
return capture.IssueRef{}, nil
}
func (fakeTracker) CommentIssue(context.Context, string, int, string) (capture.IssueRef, error) {
return capture.IssueRef{}, nil
}
func newHandler(t *testing.T, v capturehttp.Validator, tr capture.IssueTracker, sovereign []string) *capturehttp.Handler {
t.Helper()
cfg, err := classification.Load(t.TempDir())
require.NoError(t, err)
svc := capture.NewService(brainstore.New(t.TempDir()), tr, nil, cfg, audit.NewSlogSink(nil))
return capturehttp.New(svc, v, staticTok, "local-cli", capturehttp.NewOriginResolver(sovereign))
}
func do(t *testing.T, h *capturehttp.Handler, authz string, body any) *httptest.ResponseRecorder {
t.Helper()
b, _ := json.Marshal(body)
req := httptest.NewRequest(http.MethodPost, "/capture", bytes.NewReader(b))
if authz != "" {
req.Header.Set("Authorization", authz)
}
rr := httptest.NewRecorder()
h.ServeHTTP(rr, req)
return rr
}
func internalReq() map[string]any {
return map[string]any{
"context": map[string]any{"harness": "claude-code", "actor": "mathias", "classification": "internal"},
"insights": []map[string]any{{"text": "a fact", "wing": "hyperguild", "hall": "facts"}},
}
}
func TestUnauthorizedWithoutToken(t *testing.T) {
h := newHandler(t, fakeValidator{err: errors.New("no")}, fakeTracker{}, nil)
rr := do(t, h, "", internalReq())
assert.Equal(t, http.StatusUnauthorized, rr.Code)
}
func TestUnauthorizedBadToken(t *testing.T) {
h := newHandler(t, fakeValidator{err: errors.New("bad jwt")}, fakeTracker{}, nil)
rr := do(t, h, "Bearer wrong", internalReq())
assert.Equal(t, http.StatusUnauthorized, rr.Code)
}
func TestHappyPathStaticToken(t *testing.T) {
h := newHandler(t, nil, fakeTracker{}, nil)
rr := do(t, h, "Bearer "+staticTok, map[string]any{
"context": map[string]any{"harness": "claude-code", "actor": "mathias", "classification": "internal"},
"insights": []map[string]any{{"text": "a fact", "wing": "hyperguild", "hall": "facts"}},
"tickets": []map[string]any{{"repo": "hyperguild", "action": "create", "title": "t"}},
})
require.Equal(t, http.StatusOK, rr.Code)
var rec capture.CaptureReceipt
require.NoError(t, json.Unmarshal(rr.Body.Bytes(), &rec))
assert.True(t, rec.Insights[0].OK)
assert.True(t, rec.Tickets[0].OK)
assert.Empty(t, rec.Errors)
}
func TestConfidentialViaUSNexusRefused(t *testing.T) {
// JWT principal not in the sovereign allowlist ⇒ us-nexus; confidential ⇒ 403.
h := newHandler(t, fakeValidator{subject: "claudeai-oauth-client"}, fakeTracker{}, nil)
rr := do(t, h, "Bearer jwt-token", map[string]any{
"context": map[string]any{"harness": "claudeai-chat", "actor": "mathias", "classification": "confidential"},
"insights": []map[string]any{{"text": "secret", "wing": "client-seb", "hall": "facts"}},
})
assert.Equal(t, http.StatusForbidden, rr.Code)
assert.Contains(t, rr.Body.String(), "sovereignty")
}
func TestConfidentialViaSovereignJWTAllowed(t *testing.T) {
// Same confidential payload, but the principal is allowlisted sovereign ⇒ allowed.
h := newHandler(t, fakeValidator{subject: "koala-cli"}, fakeTracker{}, []string{"koala-cli"})
rr := do(t, h, "Bearer jwt-token", map[string]any{
"context": map[string]any{"harness": "claude-code", "actor": "mathias", "classification": "confidential"},
"insights": []map[string]any{{"text": "secret", "wing": "client-seb", "hall": "facts"}},
})
require.Equal(t, http.StatusOK, rr.Code)
}
func TestStaticTokenIsSovereignSoConfidentialAllowed(t *testing.T) {
h := newHandler(t, nil, fakeTracker{}, nil)
rr := do(t, h, "Bearer "+staticTok, map[string]any{
"context": map[string]any{"harness": "claude-code", "actor": "mathias", "classification": "confidential"},
"insights": []map[string]any{{"text": "secret", "wing": "client-seb", "hall": "facts"}},
})
assert.Equal(t, http.StatusOK, rr.Code)
}
func TestValidationRejectedBeforeWrite(t *testing.T) {
h := newHandler(t, nil, fakeTracker{}, nil)
rr := do(t, h, "Bearer "+staticTok, map[string]any{
"context": map[string]any{"actor": "mathias", "classification": "internal"},
"insights": []map[string]any{{"text": "x", "wing": "hyperguild", "hall": "not-a-hall"}},
})
assert.Equal(t, http.StatusBadRequest, rr.Code)
}
func TestPartialFailureIs207(t *testing.T) {
h := newHandler(t, nil, fakeTracker{failCreate: true}, nil)
rr := do(t, h, "Bearer "+staticTok, map[string]any{
"context": map[string]any{"harness": "claude-code", "actor": "mathias", "classification": "internal"},
"insights": []map[string]any{{"text": "ok insight", "wing": "hyperguild", "hall": "facts"}},
"tickets": []map[string]any{{"repo": "hyperguild", "action": "create", "title": "fails"}},
})
assert.Equal(t, http.StatusMultiStatus, rr.Code)
var rec capture.CaptureReceipt
require.NoError(t, json.Unmarshal(rr.Body.Bytes(), &rec))
assert.True(t, rec.Insights[0].OK)
assert.False(t, rec.Tickets[0].OK)
assert.Len(t, rec.Errors, 1)
}
func TestDryRunWritesNothing(t *testing.T) {
h := newHandler(t, nil, fakeTracker{}, nil)
rr := do(t, h, "Bearer "+staticTok, map[string]any{
"context": map[string]any{"harness": "claude-code", "actor": "mathias", "classification": "internal"},
"insights": []map[string]any{{"text": "a", "wing": "hyperguild", "hall": "facts"}},
"dry_run": true,
})
require.Equal(t, http.StatusOK, rr.Code)
var rec capture.CaptureReceipt
require.NoError(t, json.Unmarshal(rr.Body.Bytes(), &rec))
assert.True(t, rec.DryRun)
}
func TestCallerCannotForgeOrigin(t *testing.T) {
// Even if the body tried to assert a sovereign harness, a us-nexus JWT
// principal + confidential ⇒ refused. (Origin is server-derived.)
h := newHandler(t, fakeValidator{subject: "claudeai-oauth-client"}, fakeTracker{}, nil)
rr := do(t, h, "Bearer jwt", map[string]any{
"context": map[string]any{"harness": "sovereign-soil", "actor": "mathias", "classification": "confidential"},
"insights": []map[string]any{{"text": "secret", "wing": "client-seb", "hall": "facts"}},
})
assert.Equal(t, http.StatusForbidden, rr.Code)
}
// refusingAudit refuses at Reserve (e.g. confidential + loki down, or floor).
type refusingAudit struct{}
func (refusingAudit) Reserve(context.Context, classification.Level) (capture.AuditOutcome, error) {
return 0, errors.New("central audit sink unreachable")
}
func (refusingAudit) Record(context.Context, capture.AuditEntry, capture.AuditOutcome) error {
return nil
}
func TestAuditUnavailableIs503(t *testing.T) {
cfg, err := classification.Load(t.TempDir())
require.NoError(t, err)
svc := capture.NewService(brainstore.New(t.TempDir()), fakeTracker{}, nil, cfg, refusingAudit{})
h := capturehttp.New(svc, nil, staticTok, "local-cli", capturehttp.NewOriginResolver(nil))
rr := do(t, h, "Bearer "+staticTok, internalReq())
assert.Equal(t, http.StatusServiceUnavailable, rr.Code)
}
+42
View File
@@ -0,0 +1,42 @@
package capturehttp
import "github.com/mathiasbq/hyperguild/ingestion/internal/capture"
// OriginResolver maps an authenticated principal to its trust zone
// (spec §4.2). The mapping is server-side and never reads caller input.
//
// Rules:
// - The static-token path is a homelab CLI caller on sovereign soil →
// ZoneSovereign.
// - A JWT principal in the sovereign allowlist → ZoneSovereign.
// - Any other JWT principal (e.g. claude.ai's OAuth identity, or any
// unrecognised subject) → ZoneUSNexus.
//
// The default is the strict one: an unknown principal is treated as
// us-nexus so the I1 gate fails safe (refuses confidential), exactly as
// an untagged classification target fails safe to confidential (#50).
type OriginResolver struct {
sovereign map[string]bool
}
// NewOriginResolver builds a resolver whose JWT sovereign principals are
// the given subjects. The static-token caller is always sovereign and
// need not be listed.
func NewOriginResolver(sovereignPrincipals []string) OriginResolver {
m := make(map[string]bool, len(sovereignPrincipals))
for _, p := range sovereignPrincipals {
if p != "" {
m[p] = true
}
}
return OriginResolver{sovereign: m}
}
// Resolve returns the trust zone for a principal. viaStatic is true when
// the static-token auth path was taken.
func (r OriginResolver) Resolve(principal string, viaStatic bool) capture.Zone {
if viaStatic || r.sovereign[principal] {
return capture.ZoneSovereign
}
return capture.ZoneUSNexus
}
@@ -0,0 +1,189 @@
// Package classification defines the data-sensitivity taxonomy and the
// per-wing / per-repo tagging the capture server reads to enforce the I1
// sovereignty gate (issue #50, capture spec §4.1).
//
// The single load-bearing property is fail-safe-to-strictest: a target
// with no explicit tag and no known default classifies as Confidential,
// never as something more permissive. A missing tag must never silently
// downgrade — that would turn the I1 gate into theatre.
//
// Classification is read from an optional classification.yaml at the
// brain root. A central, Flux-reconcilable file is deliberate: it is
// auditable in one place (I2/I5), it does not require a live Gitea client
// to classify a repo (so this package has no dependency on the gitea
// tracker work), and it avoids tagging a wing's _index.md frontmatter —
// which BuildWingIndex regenerates and would clobber.
package classification
import (
"fmt"
"os"
"path/filepath"
"strings"
"gopkg.in/yaml.v3"
)
// Level is a data-sensitivity tier. Higher is stricter, so the "stricter
// wins" rule (spec §4.1 model C) is a plain max.
type Level int
const (
Public Level = iota
Internal
Confidential
)
// String returns the canonical lowercase token for a level.
func (l Level) String() string {
switch l {
case Public:
return "public"
case Internal:
return "internal"
case Confidential:
return "confidential"
default:
return fmt.Sprintf("level(%d)", int(l))
}
}
// ParseLevel parses a level token (case-insensitive, surrounding space
// tolerated). An unknown token is an error — callers must decide what to
// do with bad input rather than have it silently coerced.
func ParseLevel(s string) (Level, error) {
switch strings.ToLower(strings.TrimSpace(s)) {
case "public":
return Public, nil
case "internal":
return Internal, nil
case "confidential":
return Confidential, nil
default:
return Confidential, fmt.Errorf("unknown classification level %q (want public/internal/confidential)", s)
}
}
// Stricter returns the more restrictive of two levels.
func Stricter(a, b Level) Level {
if a > b {
return a
}
return b
}
// TargetKind distinguishes the two kinds of capture destination.
type TargetKind int
const (
WingTarget TargetKind = iota // a brain wing (insights land here)
RepoTarget // a Gitea repo (tickets / summaries land here)
)
// Target names a capture destination to classify.
type Target struct {
Kind TargetKind
Name string
}
// Config holds the explicit per-wing / per-repo classification tags read
// from classification.yaml. Absent entries fall through to the built-in
// defaults in defaultFor. The zero value (no file) is valid and applies
// defaults to everything.
type Config struct {
wings map[string]Level
repos map[string]Level
}
// rawConfig is the on-disk YAML shape: string→string maps, parsed into
// validated levels by Load.
type rawConfig struct {
Wings map[string]string `yaml:"wings"`
Repos map[string]string `yaml:"repos"`
}
// Load reads classification.yaml from brainDir. An absent file is not an
// error — it yields an empty config where every target classifies by the
// built-in defaults. A malformed file, or any unparseable level token in
// it, is a hard error: a classification source the server cannot trust
// must fail loud, not degrade silently.
func Load(brainDir string) (*Config, error) {
cfg := &Config{wings: map[string]Level{}, repos: map[string]Level{}}
data, err := os.ReadFile(filepath.Join(brainDir, "classification.yaml"))
if err != nil {
if os.IsNotExist(err) {
return cfg, nil
}
return nil, fmt.Errorf("read classification.yaml: %w", err)
}
var raw rawConfig
if err := yaml.Unmarshal(data, &raw); err != nil {
return nil, fmt.Errorf("parse classification.yaml: %w", err)
}
for name, lvl := range raw.Wings {
parsed, perr := ParseLevel(lvl)
if perr != nil {
return nil, fmt.Errorf("wing %q: %w", name, perr)
}
cfg.wings[normalise(name)] = parsed
}
for name, lvl := range raw.Repos {
parsed, perr := ParseLevel(lvl)
if perr != nil {
return nil, fmt.Errorf("repo %q: %w", name, perr)
}
cfg.repos[normalise(name)] = parsed
}
return cfg, nil
}
// Derive returns the classification for any target — the function the
// capture use-case calls per item.
func (c *Config) Derive(t Target) Level {
if t.Kind == RepoTarget {
return c.Repo(t.Name)
}
return c.Wing(t.Name)
}
// Wing classifies a brain wing: an explicit tag wins, else defaults.
func (c *Config) Wing(name string) Level {
if lvl, ok := c.wings[normalise(name)]; ok {
return lvl
}
return defaultFor(name)
}
// Repo classifies a Gitea repo: an explicit tag wins, else defaults.
func (c *Config) Repo(name string) Level {
if lvl, ok := c.repos[normalise(name)]; ok {
return lvl
}
return defaultFor(name)
}
// defaultFor applies the built-in defaulting rules when a target has no
// explicit tag:
// - client-* → Confidential (client work is confidential by default)
// - hyperguild / homelab → Internal (the operator's own infra)
// - everything else → Confidential (fail safe to strictest)
func defaultFor(name string) Level {
n := normalise(name)
if strings.HasPrefix(n, "client-") {
return Confidential
}
switch n {
case "hyperguild", "homelab":
return Internal
default:
return Confidential
}
}
// normalise lowercases and trims a wing/repo name so matching and the
// client-* prefix check are case-insensitive.
func normalise(name string) string {
return strings.ToLower(strings.TrimSpace(name))
}
@@ -0,0 +1,112 @@
package classification
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestLevelOrderingAndString(t *testing.T) {
assert.True(t, Public < Internal)
assert.True(t, Internal < Confidential)
assert.Equal(t, "public", Public.String())
assert.Equal(t, "internal", Internal.String())
assert.Equal(t, "confidential", Confidential.String())
}
func TestParseLevel(t *testing.T) {
for s, want := range map[string]Level{
"public": Public, "internal": Internal, "confidential": Confidential,
"PUBLIC": Public, " Confidential ": Confidential,
} {
got, err := ParseLevel(s)
require.NoError(t, err, s)
assert.Equal(t, want, got, s)
}
_, err := ParseLevel("secret")
require.Error(t, err, "unknown level must error, not silently default")
_, err = ParseLevel("")
require.Error(t, err)
}
func TestStricterReturnsMax(t *testing.T) {
assert.Equal(t, Confidential, Stricter(Internal, Confidential))
assert.Equal(t, Confidential, Stricter(Confidential, Public))
assert.Equal(t, Internal, Stricter(Public, Internal))
assert.Equal(t, Public, Stricter(Public, Public))
}
func TestLoadAbsentFileIsDefaultsOnly(t *testing.T) {
cfg, err := Load(t.TempDir())
require.NoError(t, err, "absent classification.yaml must not be an error — defaults apply")
require.NotNil(t, cfg)
// Pure defaulting still works.
assert.Equal(t, Internal, cfg.Wing("hyperguild"))
assert.Equal(t, Confidential, cfg.Wing("anything-unknown"))
}
func TestLoadParsesExplicitTags(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "classification.yaml"), []byte(
"wings:\n research-public: public\n hyperguild: confidential\nrepos:\n infra: internal\n research-public: public\n",
), 0o644))
cfg, err := Load(dir)
require.NoError(t, err)
// Explicit tag wins over the built-in default (hyperguild default is internal).
assert.Equal(t, Confidential, cfg.Wing("hyperguild"))
// Explicit public is honoured.
assert.Equal(t, Public, cfg.Wing("research-public"))
assert.Equal(t, Internal, cfg.Repo("infra"))
assert.Equal(t, Public, cfg.Repo("research-public"))
}
func TestLoadRejectsUnknownLevelInFile(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "classification.yaml"),
[]byte("wings:\n x: top-secret\n"), 0o644))
_, err := Load(dir)
require.Error(t, err, "an unparseable level in the config must fail loud, not be ignored")
}
func TestWingDefaulting(t *testing.T) {
cfg, err := Load(t.TempDir())
require.NoError(t, err)
cases := map[string]Level{
"client-seb": Confidential, // client-* → confidential
"client-mastercard": Confidential,
"hyperguild": Internal,
"homelab": Internal,
"jepa-fx": Confidential, // unknown → fail safe to strictest
"": Confidential, // empty → fail safe
}
for wing, want := range cases {
assert.Equal(t, want, cfg.Wing(wing), "wing %q", wing)
}
}
func TestRepoDefaulting(t *testing.T) {
cfg, err := Load(t.TempDir())
require.NoError(t, err)
assert.Equal(t, Confidential, cfg.Repo("client-seb-pipeline"))
assert.Equal(t, Internal, cfg.Repo("hyperguild"))
assert.Equal(t, Confidential, cfg.Repo("some-unknown-repo"), "untagged repo → confidential (fail safe)")
}
func TestDeriveUnifiedTarget(t *testing.T) {
cfg, err := Load(t.TempDir())
require.NoError(t, err)
assert.Equal(t, Internal, cfg.Derive(Target{Kind: WingTarget, Name: "homelab"}))
assert.Equal(t, Confidential, cfg.Derive(Target{Kind: RepoTarget, Name: "client-x"}))
assert.Equal(t, Confidential, cfg.Derive(Target{Kind: WingTarget, Name: "untagged"}))
}
func TestCaseInsensitiveMatching(t *testing.T) {
cfg, err := Load(t.TempDir())
require.NoError(t, err)
assert.Equal(t, Confidential, cfg.Wing("Client-SEB"), "client- prefix match is case-insensitive")
assert.Equal(t, Internal, cfg.Wing("HyperGuild"))
}
+13 -1
View File
@@ -38,11 +38,23 @@ var DefaultRules = []Rule{
// specific match name in logs. // specific match name in logs.
{Name: "authorization-header", RE: regexp.MustCompile(`(?i)Authorization\s*:\s*[A-Za-z]+\s+\S{8,}`)}, {Name: "authorization-header", RE: regexp.MustCompile(`(?i)Authorization\s*:\s*[A-Za-z]+\s+\S{8,}`)},
{Name: "bearer-token", RE: regexp.MustCompile(`(?i)Bearer\s+[A-Za-z0-9._\-]{16,}`)}, {Name: "bearer-token", RE: regexp.MustCompile(`(?i)Bearer\s+[A-Za-z0-9._\-]{16,}`)},
// JWT (header.payload.sig), e.g. a Dex/OAuth token dumped to stdout
// without a "Bearer " prefix. Both header and payload base64url-encode
// JSON, so both segments begin with "eyJ".
{Name: "jwt", RE: regexp.MustCompile(`eyJ[A-Za-z0-9_\-]{8,}\.eyJ[A-Za-z0-9_\-]{8,}\.[A-Za-z0-9_\-]{8,}`)},
{Name: "postgres-uri-with-password", RE: regexp.MustCompile(`postgres(?:ql)?://[^:\s/]+:[^@\s/]+@`)}, {Name: "postgres-uri-with-password", RE: regexp.MustCompile(`postgres(?:ql)?://[^:\s/]+:[^@\s/]+@`)},
{Name: "private-key", RE: regexp.MustCompile(`-----BEGIN[^-]*PRIVATE KEY-----`)}, {Name: "private-key", RE: regexp.MustCompile(`-----BEGIN[^-]*PRIVATE KEY-----`)},
{Name: "ssh-key", RE: regexp.MustCompile(`ssh-(?:rsa|ed25519|ecdsa)\s+[A-Za-z0-9+/=]{40,}`)}, {Name: "ssh-key", RE: regexp.MustCompile(`ssh-(?:rsa|ed25519|ecdsa)\s+[A-Za-z0-9+/=]{40,}`)},
{Name: "github-pat", RE: regexp.MustCompile(`\b(?:ghp|gho|ghu|ghr|gha)_[A-Za-z0-9]{30,}\b`)}, {Name: "github-pat", RE: regexp.MustCompile(`\b(?:ghp|gho|ghu|ghr|gha)_[A-Za-z0-9]{30,}\b`)},
{Name: "openai-sk", RE: regexp.MustCompile(`\bsk-(?:proj-)?[A-Za-z0-9]{32,}\b`)}, // 1Password service-account token (ops_<base64url>). Long, high-value root
// credential; guard the bare value (the _TOKEN= form also hits homelab-env-token).
{Name: "op-service-account", RE: regexp.MustCompile(`\bops_[A-Za-z0-9_\-]{40,}`)},
// No leading \b: a shell mangle can glue the key to a preceding word
// ("yes"+"sk-...") which has no word boundary, and that exact case
// leaked a LiteLLM master key past this rule (2026-06-11). Match the
// sk- shape wherever it appears; the {32,} length floor keeps short
// "task-"/"disk-" words from tripping it.
{Name: "openai-sk", RE: regexp.MustCompile(`sk-(?:proj-)?[A-Za-z0-9]{32,}`)},
{Name: "anthropic-sk", RE: regexp.MustCompile(`\bsk-ant-[A-Za-z0-9_\-]{32,}\b`)}, {Name: "anthropic-sk", RE: regexp.MustCompile(`\bsk-ant-[A-Za-z0-9_\-]{32,}\b`)},
{Name: "aws-access-key", RE: regexp.MustCompile(`\bAKIA[0-9A-Z]{16}\b`)}, {Name: "aws-access-key", RE: regexp.MustCompile(`\bAKIA[0-9A-Z]{16}\b`)},
{Name: "homelab-env-token", RE: regexp.MustCompile(`(?i)(?:_TOKEN|_PASSWORD|_API_KEY|_SECRET)\s*[:=]\s*['"]?[A-Za-z0-9._/+\-]{12,}`)}, {Name: "homelab-env-token", RE: regexp.MustCompile(`(?i)(?:_TOKEN|_PASSWORD|_API_KEY|_SECRET)\s*[:=]\s*['"]?[A-Za-z0-9._/+\-]{12,}`)},
@@ -25,6 +25,18 @@ func TestScrub_PoisonedFixtures(t *testing.T) {
{"aws-access-key", "AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE", "aws-access-key"}, {"aws-access-key", "AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE", "aws-access-key"},
{"homelab-env", "POSTGRES_PASSWORD=hunter2supersecretvalue", "homelab-env-token"}, {"homelab-env", "POSTGRES_PASSWORD=hunter2supersecretvalue", "homelab-env-token"},
{"sops-marker", "value: ENC[AES256_GCM,data:abc123def456,iv:zzz]", "sops-encrypted-marker"}, {"sops-marker", "value: ENC[AES256_GCM,data:abc123def456,iv:zzz]", "sops-encrypted-marker"},
// Regression: a shell mangle glued the key to a preceding word
// ("yes"+"sk-..."), defeating the leading \b in the sk- rule and
// leaking a LiteLLM master key past the scrubber (2026-06-11).
{"sk-glued-to-word", "master key resolved: yessk-7181ca984603239d8c4819361bf33b94b9c3c07018791868", "openai-sk"},
{"sk-standalone-hex", "sk-7181ca984603239d8c4819361bf33b94b9c3c07018791868", "openai-sk"},
// Bare JWT not preceded by "Bearer" (e.g. a Dex token dumped to stdout).
{"jwt-bare", "token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0.dQw4w9WgXcQabcdef", "jwt"},
// 1Password service-account token (ops_<base64url>), env-assigned and bare.
// Both hit the dedicated op-service-account rule (ordered before the
// generic homelab-env-token). Guards ~/.zshrc reads etc. (2026-06-14).
{"op-sa-env", "export OP_SERVICE_ACCOUNT_TOKEN=ops_eyJzaWduSW5BZGRyZXNzIjoibXkuMXBhc3N3b3JkLmNvbSJ9", "op-service-account"},
{"op-sa-bare", "ops_eyJzaWduSW5BZGRyZXNzIjoibXkuMXBhc3N3b3JkLmNvbSIsInVzZXJBdXRoIjp7fX0aGVsbG8", "op-service-account"},
} }
for _, tc := range cases { for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
@@ -43,6 +55,9 @@ func TestScrub_CleanContentPassesThrough(t *testing.T) {
"file at ~/.ssh/id_ed25519", "file at ~/.ssh/id_ed25519",
"the function Authorization() takes no args", "the function Authorization() takes no args",
"comment: see API key in 1Password", "comment: see API key in 1Password",
// loosened sk- rule must not trip on short "task-"/"disk-" words
"run task-build then task-test in the pipeline",
"mounted /dev/disk-by-id/wwn-0x5000",
} }
for _, c := range cases { for _, c := range cases {
assert.Empty(t, Scrub(c), "expected clean for %q", c) assert.Empty(t, Scrub(c), "expected clean for %q", c)
+148
View File
@@ -0,0 +1,148 @@
// ingestion/internal/extract/docmark.go
package extract
import (
"bytes"
"encoding/base64"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"time"
)
// docmarkRequest is a JSON-RPC 2.0 tools/call request for docmark's
// convert_to_markdown tool.
type docmarkRequest struct {
JSONRPC string `json:"jsonrpc"`
ID int `json:"id"`
Method string `json:"method"`
Params struct {
Name string `json:"name"`
Arguments struct {
ContentBase64 string `json:"content_base64"`
Filename string `json:"filename"`
} `json:"arguments"`
} `json:"params"`
}
type docmarkResponse struct {
Result *struct {
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
} `json:"content"`
IsError bool `json:"isError"`
} `json:"result"`
Error *struct {
Message string `json:"message"`
} `json:"error"`
}
// extractViaDocmark converts path (PDF/DOCX/XLSX/PPTX/image) to Markdown by
// calling the docmark MCP server with a single self-contained tools/call
// request (docmark runs stateless_http -- no initialize handshake or session
// ID needed). DOCMARK_URL must be set (e.g.
// http://docmark.docmark.svc.cluster.local:3001/mcp); DOCMARK_BEARER_TOKEN
// is docmark's static bearer (network is docmark's primary auth boundary,
// this is defense-in-depth — ADR-0013).
func extractViaDocmark(path string) (string, error) {
url := os.Getenv("DOCMARK_URL")
if url == "" {
return "", fmt.Errorf("extractViaDocmark: DOCMARK_URL is not set")
}
raw, err := os.ReadFile(path)
if err != nil {
return "", fmt.Errorf("read %s: %w", path, err)
}
var reqBody docmarkRequest
reqBody.JSONRPC = "2.0"
reqBody.ID = 1
reqBody.Method = "tools/call"
reqBody.Params.Name = "convert_to_markdown"
reqBody.Params.Arguments.ContentBase64 = base64.StdEncoding.EncodeToString(raw)
reqBody.Params.Arguments.Filename = fileBase(path)
payload, err := json.Marshal(reqBody)
if err != nil {
return "", fmt.Errorf("marshal docmark request: %w", err)
}
httpReq, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(payload))
if err != nil {
return "", fmt.Errorf("build docmark request: %w", err)
}
httpReq.Header.Set("Content-Type", "application/json")
httpReq.Header.Set("Accept", "application/json, text/event-stream")
if tok := os.Getenv("DOCMARK_BEARER_TOKEN"); tok != "" {
httpReq.Header.Set("Authorization", "Bearer "+tok)
}
client := &http.Client{Timeout: 60 * time.Second}
resp, err := client.Do(httpReq)
if err != nil {
return "", fmt.Errorf("call docmark: %w", err)
}
defer func() { _ = resp.Body.Close() }()
body, err := io.ReadAll(resp.Body)
if err != nil {
return "", fmt.Errorf("read docmark response: %w", err)
}
if resp.StatusCode != http.StatusOK {
return "", fmt.Errorf("docmark: HTTP %d: %s", resp.StatusCode, string(body))
}
jsonBody := body
if data := sseDataPayload(body); data != nil {
jsonBody = data
}
var out docmarkResponse
if err := json.Unmarshal(jsonBody, &out); err != nil {
return "", fmt.Errorf("decode docmark response: %w", err)
}
if out.Error != nil {
return "", fmt.Errorf("docmark: %s", out.Error.Message)
}
if out.Result == nil || len(out.Result.Content) == 0 {
return "", fmt.Errorf("docmark: empty response")
}
text := out.Result.Content[0].Text
if out.Result.IsError {
return "", fmt.Errorf("docmark: %s", text)
}
return text, nil
}
// sseDataPayload extracts the JSON payload from an SSE-framed response body
// ("event: message\r\ndata: {...}\r\n\r\n"). docmark's Streamable-HTTP
// transport frames every response this way (Content-Type: text/event-stream)
// regardless of stateless_http — that flag removes the session/initialize
// requirement, not the SSE wire framing. Returns nil if body isn't SSE-framed
// (e.g. a plain-JSON response, kept as a fallback for forward-compatibility).
func sseDataPayload(body []byte) []byte {
const prefix = "data: "
for _, line := range bytes.Split(body, []byte("\n")) {
line = bytes.TrimRight(line, "\r")
if bytes.HasPrefix(line, []byte(prefix)) {
return bytes.TrimPrefix(line, []byte(prefix))
}
}
return nil
}
// fileBase returns the final path segment (like filepath.Base, kept local to
// avoid importing path/filepath just for this one call).
func fileBase(path string) string {
for i := len(path) - 1; i >= 0; i-- {
if path[i] == '/' || path[i] == '\\' {
return path[i+1:]
}
}
return path
}
+189
View File
@@ -0,0 +1,189 @@
// ingestion/internal/extract/docmark_test.go
package extract
import (
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// mcpToolResult mirrors the shape of docmark's JSON-RPC tools/call response.
type mcpToolResult struct {
JSONRPC string `json:"jsonrpc"`
ID int `json:"id"`
Result *struct {
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
} `json:"content"`
IsError bool `json:"isError"`
} `json:"result,omitempty"`
Error *struct {
Message string `json:"message"`
} `json:"error,omitempty"`
}
// writeMCPResponse mirrors docmark's REAL response framing (empirically
// confirmed against the live server): Content-Type: text/event-stream,
// body is SSE-framed ("event: message\r\ndata: {...}\r\n\r\n"), not bare
// JSON -- inherent to MCP Streamable-HTTP, independent of stateless_http.
func writeMCPResponse(w http.ResponseWriter, body mcpToolResult) {
payload, _ := json.Marshal(body)
w.Header().Set("Content-Type", "text/event-stream")
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte("event: message\r\ndata: "))
_, _ = w.Write(payload)
_, _ = w.Write([]byte("\r\n\r\n"))
}
func TestExtractViaDocmark_Success(t *testing.T) {
var gotAuth, gotAccept string
var gotBody map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotAuth = r.Header.Get("Authorization")
gotAccept = r.Header.Get("Accept")
b, _ := io.ReadAll(r.Body)
_ = json.Unmarshal(b, &gotBody)
writeMCPResponse(w, mcpToolResult{
JSONRPC: "2.0", ID: 1,
Result: &struct {
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
} `json:"content"`
IsError bool `json:"isError"`
}{
Content: []struct {
Type string `json:"type"`
Text string `json:"text"`
}{{Type: "text", Text: "# Converted\n\nhello"}},
},
})
}))
defer srv.Close()
t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
t.Setenv("DOCMARK_BEARER_TOKEN", "test-token-123")
dir := t.TempDir()
path := filepath.Join(dir, "doc.docx")
require.NoError(t, os.WriteFile(path, []byte("fake docx bytes"), 0o644))
got, err := extractViaDocmark(path)
require.NoError(t, err)
assert.Equal(t, "# Converted\n\nhello", got)
assert.Equal(t, "Bearer test-token-123", gotAuth)
assert.Contains(t, gotAccept, "application/json")
params, _ := gotBody["params"].(map[string]any)
require.NotNil(t, params)
assert.Equal(t, "convert_to_markdown", params["name"])
args, _ := params["arguments"].(map[string]any)
require.NotNil(t, args)
assert.Equal(t, "doc.docx", args["filename"])
assert.NotEmpty(t, args["content_base64"])
}
func TestExtractViaDocmark_NotConfigured(t *testing.T) {
t.Setenv("DOCMARK_URL", "")
dir := t.TempDir()
path := filepath.Join(dir, "doc.docx")
require.NoError(t, os.WriteFile(path, []byte("x"), 0o644))
_, err := extractViaDocmark(path)
require.Error(t, err)
assert.Contains(t, err.Error(), "DOCMARK_URL")
}
func TestExtractViaDocmark_ToolErrorSurfacesMessage(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
writeMCPResponse(w, mcpToolResult{
JSONRPC: "2.0", ID: 1,
Result: &struct {
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
} `json:"content"`
IsError bool `json:"isError"`
}{
Content: []struct {
Type string `json:"type"`
Text string `json:"text"`
}{{Type: "text", Text: "unsupported format for 'doc.docx'"}},
IsError: true,
},
})
}))
defer srv.Close()
t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
t.Setenv("DOCMARK_BEARER_TOKEN", "tok")
dir := t.TempDir()
path := filepath.Join(dir, "doc.docx")
require.NoError(t, os.WriteFile(path, []byte("x"), 0o644))
_, err := extractViaDocmark(path)
require.Error(t, err)
assert.Contains(t, err.Error(), "unsupported format")
}
func TestExtractViaDocmark_HTTPErrorSurfaces(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusUnauthorized)
_, _ = w.Write([]byte("unauthorized"))
}))
defer srv.Close()
t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
t.Setenv("DOCMARK_BEARER_TOKEN", "wrong")
dir := t.TempDir()
path := filepath.Join(dir, "doc.docx")
require.NoError(t, os.WriteFile(path, []byte("x"), 0o644))
_, err := extractViaDocmark(path)
require.Error(t, err)
assert.Contains(t, err.Error(), "401")
}
func TestText_RoutesDocxXlsxPptxImagesToDocmark(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
writeMCPResponse(w, mcpToolResult{
JSONRPC: "2.0", ID: 1,
Result: &struct {
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
} `json:"content"`
IsError bool `json:"isError"`
}{
Content: []struct {
Type string `json:"type"`
Text string `json:"text"`
}{{Type: "text", Text: "converted"}},
},
})
}))
defer srv.Close()
t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
t.Setenv("DOCMARK_BEARER_TOKEN", "tok")
for _, ext := range []string{".docx", ".xlsx", ".pptx", ".png", ".jpg", ".jpeg"} {
t.Run(ext, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "f"+ext)
require.NoError(t, os.WriteFile(path, []byte("x"), 0o644))
got, err := Text(path)
require.NoError(t, err)
assert.Equal(t, "converted", got)
})
}
}
+5 -1
View File
@@ -8,7 +8,9 @@ import (
) )
// Text reads the file at path and returns its plain-text content. // Text reads the file at path and returns its plain-text content.
// Supported extensions: .md, .txt (passthrough), .pdf (via pdftotext). // Supported extensions: .md, .txt (passthrough), .pdf (via pdftotext),
// .docx/.xlsx/.pptx/.png/.jpg/.jpeg (via docmark, ADR-0013 -- requires
// DOCMARK_URL to be set; see docmark.go).
func Text(path string) (string, error) { func Text(path string) (string, error) {
ext := strings.ToLower(fileExt(path)) ext := strings.ToLower(fileExt(path))
switch ext { switch ext {
@@ -20,6 +22,8 @@ func Text(path string) (string, error) {
return string(b), nil return string(b), nil
case ".pdf": case ".pdf":
return extractPDF(path) return extractPDF(path)
case ".docx", ".xlsx", ".pptx", ".png", ".jpg", ".jpeg":
return extractViaDocmark(path)
default: default:
return "", fmt.Errorf("unsupported file extension: %s", ext) return "", fmt.Errorf("unsupported file extension: %s", ext)
} }
+198
View File
@@ -0,0 +1,198 @@
// Package gitea implements capture.IssueTracker against a Gitea instance
// over its REST API. It is the new outbound dependency the brain server
// gains for the capture capability (#49c/#52): the server otherwise does
// brain-local file ops only.
//
// Owner is hard-coded to the operator and never taken from caller input.
// The API token is read once at construction, held in the struct, and
// never logged or placed in argv — it travels only in the Authorization
// header of outbound requests (AGENTS.md secret-handling).
package gitea
import (
"bytes"
"context"
"encoding/base64"
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
)
// owner is the fixed repository owner for every ticket operation. It is a
// constant, not a parameter, so a caller can never redirect a write to
// another owner's repo.
const owner = "mathias"
// Client is a Gitea REST API IssueTracker.
type Client struct {
baseURL string
token string
http *http.Client
}
// New constructs a Client. It returns nil when either baseURL or token is
// empty, so callers can treat missing config as "tracker disabled" with a
// single nil check (mirrors embed.New).
func New(baseURL, token string) *Client {
if baseURL == "" || token == "" {
return nil
}
return &Client{
baseURL: strings.TrimRight(baseURL, "/"),
token: token,
http: &http.Client{Timeout: 15 * time.Second},
}
}
// issueResponse is the subset of a Gitea issue/comment payload we read.
type issueResponse struct {
Number int `json:"number"`
HTMLURL string `json:"html_url"`
}
// CreateIssue opens a new issue under the fixed owner.
func (c *Client) CreateIssue(ctx context.Context, repo, title, body string) (capture.IssueRef, error) {
var out issueResponse
if err := c.do(ctx, http.MethodPost,
fmt.Sprintf("/api/v1/repos/%s/%s/issues", owner, repo),
map[string]any{"title": title, "body": body}, &out); err != nil {
return capture.IssueRef{}, err
}
return capture.IssueRef{Repo: repo, Number: out.Number, URL: out.HTMLURL}, nil
}
// CommentIssue posts a comment on an existing issue.
func (c *Client) CommentIssue(ctx context.Context, repo string, number int, body string) (capture.IssueRef, error) {
var out issueResponse
if err := c.do(ctx, http.MethodPost,
fmt.Sprintf("/api/v1/repos/%s/%s/issues/%d/comments", owner, repo, number),
map[string]any{"body": body}, &out); err != nil {
return capture.IssueRef{}, err
}
return capture.IssueRef{Repo: repo, Number: number, URL: out.HTMLURL}, nil
}
// CloseIssue closes an issue, first posting a closing comment when one is
// given (empty comment ⇒ close only).
func (c *Client) CloseIssue(ctx context.Context, repo string, number int, comment string) (capture.IssueRef, error) {
if strings.TrimSpace(comment) != "" {
if _, err := c.CommentIssue(ctx, repo, number, comment); err != nil {
return capture.IssueRef{}, err
}
}
var out issueResponse
if err := c.do(ctx, http.MethodPatch,
fmt.Sprintf("/api/v1/repos/%s/%s/issues/%d", owner, repo, number),
map[string]any{"state": "closed"}, &out); err != nil {
return capture.IssueRef{}, err
}
return capture.IssueRef{Repo: repo, Number: number, URL: out.HTMLURL}, nil
}
// WriteFile creates or updates a file in repo at path via the Gitea
// contents API — the SummaryWriter port (#66). It upserts: a GET resolves
// the current blob sha (if any) so an existing file is updated rather than
// rejected (the richer-fidelity-supersedes rule for re-captured sessions).
// Owner is the fixed const, like every other call.
func (c *Client) WriteFile(ctx context.Context, repo, path, content string) error {
cpath := fmt.Sprintf("/api/v1/repos/%s/%s/contents/%s", owner, repo, path)
sha, err := c.fileSHA(ctx, cpath)
if err != nil {
return err
}
payload := map[string]any{
"message": "capture: " + path,
"content": base64.StdEncoding.EncodeToString([]byte(content)),
}
// Gitea contents API: POST creates a new file, PUT updates an existing
// one (PUT requires the current sha). Pick by whether the file exists.
method := http.MethodPost
if sha != "" {
method = http.MethodPut
payload["sha"] = sha
}
status, body, err := c.request(ctx, method, cpath, payload)
if err != nil {
return err
}
if status < 200 || status >= 300 {
return fmt.Errorf("gitea %s %s: status %d: %s", method, cpath, status, strings.TrimSpace(string(body)))
}
return nil
}
// fileSHA returns the current blob sha for a contents path, or "" when the
// file does not exist (404). Any other non-2xx is an error.
func (c *Client) fileSHA(ctx context.Context, cpath string) (string, error) {
status, body, err := c.request(ctx, http.MethodGet, cpath, nil)
if err != nil {
return "", err
}
if status == http.StatusNotFound {
return "", nil
}
if status < 200 || status >= 300 {
return "", fmt.Errorf("gitea GET %s: status %d: %s", cpath, status, strings.TrimSpace(string(body)))
}
var meta struct {
SHA string `json:"sha"`
}
if err := json.Unmarshal(body, &meta); err != nil {
return "", fmt.Errorf("gitea GET %s: decode: %w", cpath, err)
}
return meta.SHA, nil
}
// do performs a JSON request against the Gitea API and decodes a 2xx
// response into out. Errors carry the status and a truncated body for
// diagnosis but never the token.
func (c *Client) do(ctx context.Context, method, path string, payload any, out *issueResponse) error {
status, body, err := c.request(ctx, method, path, payload)
if err != nil {
return err
}
if status < 200 || status >= 300 {
return fmt.Errorf("gitea %s %s: status %d: %s", method, path, status, strings.TrimSpace(string(body)))
}
if out != nil && len(body) > 0 {
if err := json.Unmarshal(body, out); err != nil {
return fmt.Errorf("gitea %s %s: decode response: %w", method, path, err)
}
}
return nil
}
// request is the shared HTTP path: marshals an optional JSON payload,
// attaches auth (token only ever in the header), and returns the status +
// body so callers can branch on status (e.g. 404) without it being an
// error. Never logs the token.
func (c *Client) request(ctx context.Context, method, path string, payload any) (int, []byte, error) {
var reader io.Reader
if payload != nil {
reqBody, err := json.Marshal(payload)
if err != nil {
return 0, nil, fmt.Errorf("marshal request: %w", err)
}
reader = bytes.NewReader(reqBody)
}
req, err := http.NewRequestWithContext(ctx, method, c.baseURL+path, reader)
if err != nil {
return 0, nil, err
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Accept", "application/json")
req.Header.Set("Authorization", "token "+c.token)
resp, err := c.http.Do(req)
if err != nil {
return 0, nil, fmt.Errorf("gitea %s %s: %w", method, path, err)
}
defer func() { _ = resp.Body.Close() }()
body, _ := io.ReadAll(io.LimitReader(resp.Body, 8192))
return resp.StatusCode, body, nil
}
+183
View File
@@ -0,0 +1,183 @@
package gitea_test
import (
"context"
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/mathiasbq/hyperguild/ingestion/internal/gitea"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
const testToken = "super-secret-token-value"
func TestNewNilWhenUnconfigured(t *testing.T) {
assert.Nil(t, gitea.New("", testToken))
assert.Nil(t, gitea.New("https://git.example", ""))
}
func TestCreateIssueForcesOwnerAndAuth(t *testing.T) {
var gotPath, gotAuth, gotBody string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
gotAuth = r.Header.Get("Authorization")
b, _ := io.ReadAll(r.Body)
gotBody = string(b)
assert.Equal(t, http.MethodPost, r.Method)
w.WriteHeader(http.StatusCreated)
_ = json.NewEncoder(w).Encode(map[string]any{"number": 42, "html_url": "https://git.d-ma.be/mathias/hyperguild/issues/42"})
}))
defer srv.Close()
c := gitea.New(srv.URL, testToken)
require.NotNil(t, c)
ref, err := c.CreateIssue(context.Background(), "hyperguild", "Do the thing", "details")
require.NoError(t, err)
assert.Equal(t, "/api/v1/repos/mathias/hyperguild/issues", gotPath, "owner forced to mathias")
assert.Equal(t, "token "+testToken, gotAuth)
assert.Contains(t, gotBody, "Do the thing")
assert.Equal(t, "hyperguild", ref.Repo)
assert.Equal(t, 42, ref.Number)
assert.Contains(t, ref.URL, "/issues/42")
}
func TestCommentIssue(t *testing.T) {
var gotPath string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
w.WriteHeader(http.StatusCreated)
_ = json.NewEncoder(w).Encode(map[string]any{"html_url": "https://git/c/1"})
}))
defer srv.Close()
ref, err := gitea.New(srv.URL, testToken).CommentIssue(context.Background(), "hyperguild", 7, "a comment")
require.NoError(t, err)
assert.Equal(t, "/api/v1/repos/mathias/hyperguild/issues/7/comments", gotPath)
assert.Equal(t, 7, ref.Number)
}
func TestCloseIssueWithComment(t *testing.T) {
var paths []string
var states []string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
paths = append(paths, r.Method+" "+r.URL.Path)
if r.Method == http.MethodPatch {
var body map[string]any
b, _ := io.ReadAll(r.Body)
_ = json.Unmarshal(b, &body)
states = append(states, body["state"].(string))
}
w.WriteHeader(http.StatusOK)
_ = json.NewEncoder(w).Encode(map[string]any{"number": 9, "html_url": "https://git/i/9"})
}))
defer srv.Close()
ref, err := gitea.New(srv.URL, testToken).CloseIssue(context.Background(), "hyperguild", 9, "closing because done")
require.NoError(t, err)
assert.Equal(t, 9, ref.Number)
// Comment posted first, then state PATCHed to closed.
assert.Contains(t, paths, "POST /api/v1/repos/mathias/hyperguild/issues/9/comments")
assert.Contains(t, paths, "PATCH /api/v1/repos/mathias/hyperguild/issues/9")
assert.Equal(t, []string{"closed"}, states)
}
func TestCloseIssueNoComment(t *testing.T) {
var commented bool
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if strings.HasSuffix(r.URL.Path, "/comments") {
commented = true
}
w.WriteHeader(http.StatusOK)
_ = json.NewEncoder(w).Encode(map[string]any{"number": 3, "html_url": "https://git/i/3"})
}))
defer srv.Close()
_, err := gitea.New(srv.URL, testToken).CloseIssue(context.Background(), "hyperguild", 3, "")
require.NoError(t, err)
assert.False(t, commented, "empty comment ⇒ no comment POST")
}
func TestErrorPathDoesNotLeakToken(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
_, _ = w.Write([]byte("boom"))
}))
defer srv.Close()
_, err := gitea.New(srv.URL, testToken).CreateIssue(context.Background(), "hyperguild", "t", "b")
require.Error(t, err)
assert.NotContains(t, err.Error(), testToken, "token must never appear in an error message")
assert.Contains(t, err.Error(), "500")
}
func TestWriteFileCreatesNewFile(t *testing.T) {
var getPath, postPath, postBody string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.Method {
case http.MethodGet:
getPath = r.URL.Path
w.WriteHeader(http.StatusNotFound) // file does not exist yet
case http.MethodPost: // gitea contents API: POST = create
postPath = r.URL.Path
b, _ := io.ReadAll(r.Body)
postBody = string(b)
w.WriteHeader(http.StatusCreated)
_ = json.NewEncoder(w).Encode(map[string]any{"content": map[string]any{"html_url": "https://git/x"}})
default:
t.Errorf("create must POST, got %s", r.Method)
}
}))
defer srv.Close()
err := gitea.New(srv.URL, testToken).WriteFile(context.Background(),
"ai-sessions", "summaries/claude-code/2026-06/2026-06-23-x-abcd1234.md", "# Summary\n\nbody\n")
require.NoError(t, err)
assert.Equal(t, "/api/v1/repos/mathias/ai-sessions/contents/summaries/claude-code/2026-06/2026-06-23-x-abcd1234.md", getPath)
assert.Equal(t, getPath, postPath)
// base64 of the content, no sha on create.
assert.Contains(t, postBody, "IyBTdW1tYXJ5") // base64("# Summary")
assert.NotContains(t, postBody, `"sha"`)
}
func TestWriteFileUpdatesExisting(t *testing.T) {
var putBody string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.Method {
case http.MethodGet:
w.WriteHeader(http.StatusOK)
_ = json.NewEncoder(w).Encode(map[string]any{"sha": "deadbeef"})
case http.MethodPut:
b, _ := io.ReadAll(r.Body)
putBody = string(b)
w.WriteHeader(http.StatusOK)
_ = json.NewEncoder(w).Encode(map[string]any{"content": map[string]any{"html_url": "https://git/x"}})
}
}))
defer srv.Close()
err := gitea.New(srv.URL, testToken).WriteFile(context.Background(), "ai-sessions", "p/x.md", "new")
require.NoError(t, err)
assert.Contains(t, putBody, `"sha":"deadbeef"`, "existing file → update with sha")
}
func TestWriteFileErrorNoTokenLeak(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodGet {
w.WriteHeader(http.StatusNotFound)
return
}
w.WriteHeader(http.StatusUnprocessableEntity)
_, _ = w.Write([]byte("bad"))
}))
defer srv.Close()
err := gitea.New(srv.URL, testToken).WriteFile(context.Background(), "ai-sessions", "p/x.md", "x")
require.Error(t, err)
assert.NotContains(t, err.Error(), testToken)
assert.Contains(t, err.Error(), "422")
}
+204
View File
@@ -0,0 +1,204 @@
package mcp_test
import (
"context"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/mcp"
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// callResult parses the JSON text payload of a successful tool call.
func callResult(t *testing.T, resp map[string]any) map[string]any {
t.Helper()
require.Nil(t, resp["error"], "tool returned error: %v", resp["error"])
text := resp["result"].(map[string]any)["content"].([]any)[0].(map[string]any)["text"].(string)
var out map[string]any
require.NoError(t, json.Unmarshal([]byte(text), &out))
return out
}
func TestBrainUpdateSupersedesExisting(t *testing.T) {
brainDir := t.TempDir()
srv := mcp.NewServer(brainDir, nil, nil, nil)
// Seed via brain_write so the note carries real frontmatter.
callResult(t, toolCall(t, srv, "brain_write", map[string]any{
"content": "# Old\n\nold body\n", "filename": "val-vol",
"wing": "jepa-fx", "hall": "facts",
}))
out := callResult(t, toolCall(t, srv, "brain_update", map[string]any{
"wing": "jepa-fx", "hall": "facts", "slug": "val-vol",
"content": "# New\n\nnew body\n", "reason": "facts changed",
}))
assert.Equal(t, "wiki/jepa-fx/facts/val-vol.md", out["path"])
assert.Equal(t, out["path"], out["id"])
assert.NotEmpty(t, out["content_hash"])
assert.Equal(t, true, out["superseded"])
got, err := os.ReadFile(filepath.Join(brainDir, "wiki/jepa-fx/facts/val-vol.md"))
require.NoError(t, err)
s := string(got)
assert.Contains(t, s, "# New")
assert.NotContains(t, s, "old body")
assert.Contains(t, s, "wing: jepa-fx")
assert.Contains(t, s, "supersede_reason: facts changed")
assert.Contains(t, s, "supersedes:")
}
func TestBrainUpdateMissingTargetErrorsNoCreate(t *testing.T) {
brainDir := t.TempDir()
srv := mcp.NewServer(brainDir, nil, nil, nil)
resp := toolCall(t, srv, "brain_update", map[string]any{
"wing": "jepa-fx", "hall": "facts", "slug": "ghost",
"content": "x\n",
})
require.NotNil(t, resp["error"])
assert.Contains(t, resp["error"].(map[string]any)["message"].(string), "does not exist")
_, statErr := os.Stat(filepath.Join(brainDir, "wiki/jepa-fx/facts/ghost.md"))
assert.True(t, os.IsNotExist(statErr))
}
func TestBrainUpdateByFullPath(t *testing.T) {
brainDir := t.TempDir()
srv := mcp.NewServer(brainDir, nil, nil, nil)
callResult(t, toolCall(t, srv, "brain_write", map[string]any{
"content": "old\n", "filename": "n", "wing": "a", "hall": "facts",
}))
out := callResult(t, toolCall(t, srv, "brain_update", map[string]any{
"slug": "wiki/a/facts/n.md", "content": "fresh\n",
}))
assert.Equal(t, "wiki/a/facts/n.md", out["path"])
}
func TestBrainGetByIDAndPath(t *testing.T) {
brainDir := t.TempDir()
srv := mcp.NewServer(brainDir, nil, nil, nil)
w := callResult(t, toolCall(t, srv, "brain_write", map[string]any{
"content": "# Body\n\ntext\n", "filename": "n", "wing": "a", "hall": "facts",
}))
id := w["id"].(string)
hash := w["content_hash"].(string)
require.NotEmpty(t, id)
require.NotEmpty(t, hash)
// by id
g1 := callResult(t, toolCall(t, srv, "brain_get", map[string]any{"id": id}))
assert.Equal(t, id, g1["path"])
assert.Equal(t, hash, g1["content_hash"], "content_hash must round-trip write→get")
assert.Contains(t, g1["body"].(string), "# Body")
fm := g1["frontmatter"].(map[string]any)
assert.Equal(t, "a", fm["wing"])
// by path
g2 := callResult(t, toolCall(t, srv, "brain_get", map[string]any{"path": id}))
assert.Equal(t, hash, g2["content_hash"])
}
func TestBrainGetMissingArgsErrors(t *testing.T) {
srv := mcp.NewServer(t.TempDir(), nil, nil, nil)
resp := toolCall(t, srv, "brain_get", map[string]any{})
require.NotNil(t, resp["error"])
}
func TestBrainWriteReturnsHandle(t *testing.T) {
brainDir := t.TempDir()
srv := mcp.NewServer(brainDir, nil, nil, nil)
out := callResult(t, toolCall(t, srv, "brain_write", map[string]any{
"content": "# X\n\nbody\n", "filename": "x", "wing": "a", "hall": "facts",
}))
assert.Equal(t, "wiki/a/facts/x.md", out["path"])
assert.Equal(t, out["path"], out["id"])
assert.NotEmpty(t, out["content_hash"])
}
// --- retrieval-reflects-new-content: exercises the real mtime-driven Sync ---
type fakeVecStore struct {
chunks map[string][]float32
deleted []string
}
func (f *fakeVecStore) KnownPathsWithTime(_ context.Context) (map[string]time.Time, error) {
m := make(map[string]time.Time, len(f.chunks))
for p := range f.chunks {
m[p] = time.Unix(0, 0) // always stale → mtime(now) is always newer
}
return m, nil
}
func (f *fakeVecStore) Upsert(_ context.Context, path string, vec []float32) error {
f.chunks[path] = vec
return nil
}
func (f *fakeVecStore) Delete(_ context.Context, path string) error {
delete(f.chunks, path)
f.deleted = append(f.deleted, path)
return nil
}
type fakeEmbedder struct{ seen []string }
func (e *fakeEmbedder) Embed(_ context.Context, text string) ([]float32, error) {
e.seen = append(e.seen, text)
return []float32{1, 0, 0}, nil
}
// TestBrainUpdateReembedsNewContent proves the supersede contract end to
// end against the actual embedding mechanism: brain_update rewrites the
// file, advancing its mtime, and the next vectorstore.Sync pass re-embeds
// the NEW body and drops the stale chunk. No stub of the re-index path.
func TestBrainUpdateReembedsNewContent(t *testing.T) {
brainDir := t.TempDir()
srv := mcp.NewServer(brainDir, nil, nil, nil)
ctx := context.Background()
callResult(t, toolCall(t, srv, "brain_write", map[string]any{
"content": "# Note\n\nthe OLD distinctive payload\n",
"filename": "n", "wing": "a", "hall": "facts",
}))
store := &fakeVecStore{chunks: map[string][]float32{}}
emb := &fakeEmbedder{}
// First sync embeds the original content.
_, err := vectorstore.Sync(ctx, brainDir, store, emb)
require.NoError(t, err)
require.NotEmpty(t, store.chunks)
require.True(t, anyContains(emb.seen, "OLD distinctive payload"))
callResult(t, toolCall(t, srv, "brain_update", map[string]any{
"wing": "a", "hall": "facts", "slug": "n",
"content": "# Note\n\nthe NEW distinctive payload\n",
}))
emb.seen = nil // only watch what the second pass embeds
_, err = vectorstore.Sync(ctx, brainDir, store, emb)
require.NoError(t, err)
assert.True(t, anyContains(emb.seen, "NEW distinctive payload"),
"Sync must re-embed the superseded body; saw %v", emb.seen)
assert.False(t, anyContains(emb.seen, "OLD distinctive payload"),
"the old body must not be re-embedded")
assert.NotEmpty(t, store.deleted, "stale chunks must be deleted before re-embed")
}
func anyContains(ss []string, sub string) bool {
for _, s := range ss {
if strings.Contains(s, sub) {
return true
}
}
return false
}
+159 -13
View File
@@ -11,6 +11,7 @@ import (
"github.com/mathiasbq/hyperguild/ingestion/internal/api" "github.com/mathiasbq/hyperguild/ingestion/internal/api"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain" "github.com/mathiasbq/hyperguild/ingestion/internal/brain"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
"github.com/mathiasbq/hyperguild/ingestion/internal/extract" "github.com/mathiasbq/hyperguild/ingestion/internal/extract"
"github.com/mathiasbq/hyperguild/ingestion/internal/graphsync" "github.com/mathiasbq/hyperguild/ingestion/internal/graphsync"
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline" "github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
@@ -38,7 +39,7 @@ func (s *Server) tools() []map[string]any {
return b return b
} }
return []map[string]any{ tools := []map[string]any{
{ {
"name": "brain_query", "name": "brain_query",
"description": "BM25 full-text search across brain/knowledge/ and brain/wiki/ markdown files. Optionally scope by wing (topic domain) and hall (memory type).", "description": "BM25 full-text search across brain/knowledge/ and brain/wiki/ markdown files. Optionally scope by wing (topic domain) and hall (memory type).",
@@ -61,6 +62,41 @@ func (s *Server) tools() []map[string]any {
"hall": enum("optional memory type (requires wing)", halls...), "hall": enum("optional memory type (requires wing)", halls...),
}), }),
}, },
{
"name": "brain_update",
"description": "Supersede an existing brain note in place: whole-note body replace + frontmatter re-stamp (updated_at, supersedes=prior content hash, supersede_reason). Errors if the target does not exist — use brain_write to create. Returns {id, path, content_hash, superseded}. Prior version recoverable from git.",
"inputSchema": schema([]string{"content"}, map[string]any{
"content": str("new full body (whole-note replace)"),
"slug": str("target note slug within wing/hall, OR a full brain-relative path (e.g. wiki/jepa-fx/facts/x.md)"),
"wing": str("wing of the target (required unless slug/path is a full path)"),
"hall": enum("hall of the target (required unless slug/path is a full path)", halls...),
"path": str("full brain-relative path to the target; takes precedence over slug/wing/hall"),
"reason": str("optional short note on why superseded — stamped into frontmatter"),
}),
},
{
"name": "brain_get",
"description": "Fetch a single brain note by id or path (both are the brain-relative path — the note handle). Returns {id, path, content_hash, frontmatter, body}. Read-after-write confirmation without a lexical re-query.",
"inputSchema": schema([]string{}, map[string]any{
"id": str("note id (brain-relative path) as returned by brain_write/brain_update"),
"path": str("brain-relative path to the note; equivalent to id"),
}),
},
{
"name": "brain_pending",
"description": "List notes in brain/raw/ awaiting human promotion to the wiki, oldest-first. Returns filename, created_at, size_bytes, excerpt. The human-review queue complement to brain_promote.",
"inputSchema": schema([]string{}, map[string]any{}),
},
{
"name": "brain_promote",
"description": "Promote a brain/raw/ note into brain/wiki/<wing>/<hall>/: rewrites frontmatter (sets wing/hall/promoted_at, preserves created_at + custom fields), deletes the source, rebuilds the wing index, runs auto-tunnel. Errors (without touching the fs) on invalid hall or a slug collision. Returns {path}.",
"inputSchema": schema([]string{"filename", "wing", "hall"}, map[string]any{
"filename": str("basename in brain/raw/, e.g. 2026-06-01-lejpa-decision.md"),
"wing": str("target wing, e.g. jepa-fx"),
"hall": enum("target hall", halls...),
"slug": str("optional target slug; defaults to filename minus date prefix"),
}),
},
{ {
"name": "brain_tunnel", "name": "brain_tunnel",
"description": "Create an explicit bidirectional [[wikilink]] between two notes in different wings. Idempotent.", "description": "Create an explicit bidirectional [[wikilink]] between two notes in different wings. Idempotent.",
@@ -151,6 +187,13 @@ func (s *Server) tools() []map[string]any {
}), }),
}, },
} }
// The capture relay tool (#55) is advertised only when wired via
// WithCapture — MCP-native harnesses (claude.ai, Crush, Pi, LLM Council)
// reach capture through it.
if s.capture != nil {
tools = append(tools, captureToolDescriptor())
}
return tools
} }
type brainQueryArgs struct { type brainQueryArgs struct {
@@ -199,7 +242,11 @@ func (s *Server) brainWrite(ctx context.Context, args json.RawMessage) (json.Raw
if err := json.Unmarshal(args, &a); err != nil { if err := json.Unmarshal(args, &a); err != nil {
return nil, fmt.Errorf("parse args: %w", err) return nil, fmt.Errorf("parse args: %w", err)
} }
relPath, err := api.WriteNote(s.brainDir, api.WriteNoteOptions{ // Delegate to the shared BrainStore so write+index+tunnel+graph live in
// one implementation (capture uses the same store). The read-after-write
// handle {id, path, content_hash} comes back from the store; path is kept
// for backward compatibility.
ref, err := s.store.Write(ctx, capture.Note{
Content: a.Content, Content: a.Content,
Filename: a.Filename, Filename: a.Filename,
Type: a.Type, Type: a.Type,
@@ -210,18 +257,117 @@ func (s *Server) brainWrite(ctx context.Context, args json.RawMessage) (json.Raw
if err != nil { if err != nil {
return nil, err return nil, err
} }
// Auto-regenerate the wing _index.md when the write landed in the return json.Marshal(map[string]string{"id": ref.ID, "path": ref.Path, "content_hash": ref.ContentHash})
// structured wiki, and auto-tunnel cross-wing matches. Both are }
// best-effort: the note is already written.
if a.Wing != "" && a.Hall != "" { type brainUpdateArgs struct {
if err := brain.BuildWingIndex(s.brainDir, a.Wing); err != nil { Slug string `json:"slug,omitempty"`
slog.Warn("brain_write: auto-index failed", "wing", a.Wing, "err", err) Wing string `json:"wing,omitempty"`
} Hall string `json:"hall,omitempty"`
if err := brain.AutoTunnel(s.brainDir, relPath, a.Content); err != nil { Path string `json:"path,omitempty"`
slog.Warn("brain_write: auto-tunnel failed", "src", relPath, "err", err) Content string `json:"content"`
} Reason string `json:"reason,omitempty"`
}
// brainUpdate supersedes an existing note in place: whole-note body
// replace, frontmatter re-stamp (updated_at/supersedes/supersede_reason),
// graph re-index, and wing _index rebuild. It never creates — a missing
// target is an error so the caller can fall back to brain_write.
//
// Embedding re-sync is delegated to the out-of-band vectorstore.Sync
// ticker: the rewritten file's mtime advances, so the next pass re-embeds
// it. This mirrors brain_write, which likewise does not embed in-handler.
func (s *Server) brainUpdate(ctx context.Context, args json.RawMessage) (json.RawMessage, error) {
var a brainUpdateArgs
if err := json.Unmarshal(args, &a); err != nil {
return nil, fmt.Errorf("parse args: %w", err)
} }
s.indexInGraph(ctx, "brain_write", relPath) if a.Content == "" {
return nil, fmt.Errorf("content is required")
}
// path takes precedence over slug; the store treats any slug containing
// a slash as a full brain-relative path (issue #45: "slug ... OR path").
slug := a.Slug
if a.Path != "" {
slug = a.Path
}
ref, err := s.store.Update(ctx, slug, capture.Note{
Content: a.Content,
Wing: a.Wing,
Hall: a.Hall,
Reason: a.Reason,
})
if err != nil {
return nil, err
}
return json.Marshal(map[string]any{
"id": ref.ID, "path": ref.Path, "content_hash": ref.ContentHash, "superseded": ref.Superseded,
})
}
type brainGetArgs struct {
ID string `json:"id,omitempty"`
Path string `json:"path,omitempty"`
}
// brainGet fetches a note by id or path (both are the brainDir-relative
// path — the de-facto handle). Read-only; the create-path read-after-
// write primitive that lets callers confirm a write landed without a
// lexical re-query.
func (s *Server) brainGet(ctx context.Context, args json.RawMessage) (json.RawMessage, error) {
var a brainGetArgs
if err := json.Unmarshal(args, &a); err != nil {
return nil, fmt.Errorf("parse args: %w", err)
}
target := a.Path
if target == "" {
target = a.ID
}
if target == "" {
return nil, fmt.Errorf("id or path is required")
}
note, err := s.store.Get(ctx, target)
if err != nil {
return nil, err
}
return json.Marshal(map[string]any{
"id": note.ID, "path": note.Path, "content_hash": note.ContentHash,
"frontmatter": note.Frontmatter, "body": note.Body,
})
}
// brainPending lists the raw/ review queue (oldest-first).
func (s *Server) brainPending(_ context.Context, _ json.RawMessage) (json.RawMessage, error) {
pending, err := api.ListPending(s.brainDir)
if err != nil {
return nil, err
}
return json.Marshal(map[string]any{"pending": pending})
}
type brainPromoteArgs struct {
Filename string `json:"filename"`
Wing string `json:"wing"`
Hall string `json:"hall"`
Slug string `json:"slug,omitempty"`
}
// brainPromote moves a raw/ note into the structured wiki (frontmatter
// rewrite + index + auto-tunnel, all owned by api.PromoteNote) and then
// re-indexes it into the graph. The human-facing complement to brain_write.
func (s *Server) brainPromote(ctx context.Context, args json.RawMessage) (json.RawMessage, error) {
var a brainPromoteArgs
if err := json.Unmarshal(args, &a); err != nil {
return nil, fmt.Errorf("parse args: %w", err)
}
relPath, err := api.PromoteNote(s.brainDir, api.PromoteOptions{
Filename: a.Filename, Wing: a.Wing, Hall: a.Hall, Slug: a.Slug,
})
if err != nil {
return nil, err
}
s.indexInGraph(ctx, "brain_promote", relPath)
return json.Marshal(map[string]string{"path": relPath}) return json.Marshal(map[string]string{"path": relPath})
} }
+58
View File
@@ -332,3 +332,61 @@ func TestSessionLogRequiresSessionID(t *testing.T) {
resp := toolCall(t, srv, "session_log", map[string]any{"skill": "tdd"}) resp := toolCall(t, srv, "session_log", map[string]any{"skill": "tdd"})
require.NotNil(t, resp["error"]) require.NotNil(t, resp["error"])
} }
func TestBrainPendingListsRaw(t *testing.T) {
brainDir := t.TempDir()
raw := filepath.Join(brainDir, "raw")
require.NoError(t, os.MkdirAll(raw, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(raw, "2026-06-01-x.md"),
[]byte("---\ncreated_at: 2026-06-01T00:00:00Z\n---\npending body\n"), 0o644))
srv := mcp.NewServer(brainDir, nil, nil, nil)
resp := toolCall(t, srv, "brain_pending", map[string]any{})
require.Nil(t, resp["error"])
text := resp["result"].(map[string]any)["content"].([]any)[0].(map[string]any)["text"].(string)
assert.Contains(t, text, "2026-06-01-x.md")
assert.Contains(t, text, "pending body")
}
func TestBrainPendingEmpty(t *testing.T) {
srv := mcp.NewServer(t.TempDir(), nil, nil, nil)
resp := toolCall(t, srv, "brain_pending", map[string]any{})
require.Nil(t, resp["error"])
text := resp["result"].(map[string]any)["content"].([]any)[0].(map[string]any)["text"].(string)
assert.Contains(t, text, `"pending":[]`)
}
func TestBrainPromoteMovesToWiki(t *testing.T) {
brainDir := t.TempDir()
raw := filepath.Join(brainDir, "raw")
require.NoError(t, os.MkdirAll(raw, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(raw, "2026-06-01-decision.md"),
[]byte("---\ncreated_at: 2026-06-01T00:00:00Z\n---\n# D\n\nbody\n"), 0o644))
srv := mcp.NewServer(brainDir, nil, nil, nil)
resp := toolCall(t, srv, "brain_promote", map[string]any{
"filename": "2026-06-01-decision.md", "wing": "jepa-fx", "hall": "decisions",
})
require.Nil(t, resp["error"], "got: %v", resp["error"])
text := resp["result"].(map[string]any)["content"].([]any)[0].(map[string]any)["text"].(string)
assert.Contains(t, text, "wiki/jepa-fx/decisions/decision.md")
_, err := os.Stat(filepath.Join(brainDir, "wiki/jepa-fx/decisions/decision.md"))
require.NoError(t, err)
_, srcErr := os.Stat(filepath.Join(raw, "2026-06-01-decision.md"))
assert.True(t, os.IsNotExist(srcErr), "source removed")
}
func TestBrainPromoteInvalidHallErrors(t *testing.T) {
brainDir := t.TempDir()
raw := filepath.Join(brainDir, "raw")
require.NoError(t, os.MkdirAll(raw, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(raw, "x.md"), []byte("body\n"), 0o644))
srv := mcp.NewServer(brainDir, nil, nil, nil)
resp := toolCall(t, srv, "brain_promote", map[string]any{
"filename": "x.md", "wing": "a", "hall": "garbage",
})
require.NotNil(t, resp["error"])
_, srcErr := os.Stat(filepath.Join(raw, "x.md"))
assert.NoError(t, srcErr, "source untouched on validation error")
}
+98 -5
View File
@@ -1,7 +1,9 @@
// Package mcp implements an MCP HTTP handler for the ingestion service. // Package mcp implements an MCP HTTP handler for the ingestion service.
// Exposed tools: brain_query, brain_write, brain_index, brain_tunnel, // Exposed tools: brain_query, brain_write, brain_update, brain_get,
// brain_ingest, brain_ingest_raw, brain_answer, brain_classify, // brain_pending, brain_promote, brain_index, brain_tunnel, brain_ingest,
// brain_graph, brain_context, session_log. // brain_ingest_raw, brain_answer, brain_classify, brain_graph,
// brain_context, session_log, and capture (the #55 relay tool, registered
// only when WithCapture is set).
package mcp package mcp
import ( import (
@@ -10,6 +12,9 @@ import (
"fmt" "fmt"
"net/http" "net/http"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainstore"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
"github.com/mathiasbq/hyperguild/ingestion/internal/capturehttp"
"github.com/mathiasbq/hyperguild/ingestion/internal/graphstore" "github.com/mathiasbq/hyperguild/ingestion/internal/graphstore"
"github.com/mathiasbq/hyperguild/ingestion/internal/graphsync" "github.com/mathiasbq/hyperguild/ingestion/internal/graphsync"
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline" "github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
@@ -46,6 +51,21 @@ type Server struct {
vector search.VectorSearcher // nil = BM25-only retrieval vector search.VectorSearcher // nil = BM25-only retrieval
embedder search.Embedder // nil = BM25-only retrieval embedder search.Embedder // nil = BM25-only retrieval
graph graphsync.Store // nil = brain_graph and GraphRAG augmentation disabled graph graphsync.Store // nil = brain_graph and GraphRAG augmentation disabled
store *brainstore.Store // shared brain write/update/get impl (also used by capture)
tracker capture.IssueTracker // nil = no Gitea ticket integration; wired for capture (#53)
capture *captureDeps // nil = capture MCP tool disabled (#55 relay)
}
// captureDeps holds what the MCP `capture` tool (the #55 relay door for
// MCP-native harnesses like claude.ai) needs: the use-case, the auth bits
// to re-derive the caller's principal from the Bearer header (the chassis
// middleware gates but discards the principal), and the origin resolver.
type captureDeps struct {
svc *capture.Service
validator capturehttp.Validator
staticToken string
staticPrincipal string
resolver capturehttp.OriginResolver
} }
// NewServer constructs a Server bound to brainDir. pipelineCfg supplies the // NewServer constructs a Server bound to brainDir. pipelineCfg supplies the
@@ -56,7 +76,13 @@ func NewServer(brainDir string, pipelineCfg *pipeline.Config, llm pipeline.Compl
if pipelineCfg != nil { if pipelineCfg != nil {
cfg = *pipelineCfg cfg = *pipelineCfg
} }
return &Server{brainDir: brainDir, pipeline: cfg, llm: llm, answerLLM: answerLLM} return &Server{
brainDir: brainDir,
pipeline: cfg,
llm: llm,
answerLLM: answerLLM,
store: brainstore.New(brainDir),
}
} }
// WithReranker installs an opt-in cross-encoder reranker. When set, // WithReranker installs an opt-in cross-encoder reranker. When set,
@@ -84,9 +110,55 @@ func (s *Server) WithHybridRetrieval(v search.VectorSearcher, e search.Embedder)
func (s *Server) WithGraph(g *graphstore.PGStore) *Server { func (s *Server) WithGraph(g *graphstore.PGStore) *Server {
if g == nil { if g == nil {
s.graph = nil s.graph = nil
s.store.WithGraph(nil)
return s return s
} }
s.graph = g s.graph = g
s.store.WithGraph(g)
return s
}
// WithIssueTracker injects the Gitea ticket tracker behind the
// capture.IssueTracker interface. nil leaves ticket integration off. The
// use-case (capture) consumes this in #53; it is wired here so the
// dependency is constructed once and stays swappable/testable.
func (s *Server) WithIssueTracker(t capture.IssueTracker) *Server {
s.tracker = t
return s
}
// IssueTracker returns the injected ticket tracker (nil when unconfigured).
func (s *Server) IssueTracker() capture.IssueTracker {
return s.tracker
}
// BrainStore returns the shared brain store (graph-wired once WithGraph
// has run), so the capture use-case writes through the exact same
// implementation as the MCP handlers.
func (s *Server) BrainStore() *brainstore.Store {
return s.store
}
// WithCapture enables the MCP `capture` tool (#55) — the relay door for
// MCP-native harnesses (claude.ai, Crush, Pi, LLM Council) that cannot run
// the use-case in-process. It forwards to the same CaptureService as
// POST /capture, deriving the caller's principal + origin from the same
// auth credentials that gate /mcp. nil svc leaves the tool unregistered.
func (s *Server) WithCapture(svc *capture.Service, validator capturehttp.Validator, staticToken, staticPrincipal string, resolver capturehttp.OriginResolver) *Server {
if svc == nil {
s.capture = nil
return s
}
if staticPrincipal == "" {
staticPrincipal = "local-cli"
}
s.capture = &captureDeps{
svc: svc,
validator: validator,
staticToken: staticToken,
staticPrincipal: staticPrincipal,
resolver: resolver,
}
return s return s
} }
@@ -139,7 +211,18 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
rpcErr = &rpcError{Code: -32602, Message: "invalid params"} rpcErr = &rpcError{Code: -32602, Message: "invalid params"}
break break
} }
out, err := s.handleCall(r.Context(), p.Name, p.Arguments) // Re-derive the authenticated principal from the Bearer header so
// the capture tool can compute the trust-zone origin. The request
// is already gated by BearerMiddleware; this only recovers the
// identity that middleware discards.
ctx := r.Context()
if s.capture != nil {
if principal, viaStatic, ok := capturehttp.Authenticate(
r, s.capture.staticToken, s.capture.staticPrincipal, s.capture.validator); ok {
ctx = withPrincipal(ctx, principal, viaStatic)
}
}
out, err := s.handleCall(ctx, p.Name, p.Arguments)
if err != nil { if err != nil {
rpcErr = &rpcError{Code: -32000, Message: err.Error()} rpcErr = &rpcError{Code: -32000, Message: err.Error()}
break break
@@ -177,6 +260,16 @@ func (s *Server) handleCall(ctx context.Context, name string, args json.RawMessa
return s.brainQuery(ctx, args) return s.brainQuery(ctx, args)
case "brain_write": case "brain_write":
return s.brainWrite(ctx, args) return s.brainWrite(ctx, args)
case "brain_update":
return s.brainUpdate(ctx, args)
case "brain_get":
return s.brainGet(ctx, args)
case "brain_pending":
return s.brainPending(ctx, args)
case "brain_promote":
return s.brainPromote(ctx, args)
case "capture":
return s.brainCapture(ctx, args)
case "brain_index": case "brain_index":
return s.brainIndex(ctx, args) return s.brainIndex(ctx, args)
case "brain_tunnel": case "brain_tunnel":
+24 -1
View File
@@ -2,12 +2,14 @@ package mcp_test
import ( import (
"bytes" "bytes"
"context"
"encoding/json" "encoding/json"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"strings" "strings"
"testing" "testing"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
"github.com/mathiasbq/hyperguild/ingestion/internal/mcp" "github.com/mathiasbq/hyperguild/ingestion/internal/mcp"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
@@ -55,7 +57,9 @@ func TestServerToolsList(t *testing.T) {
names = append(names, t.(map[string]any)["name"].(string)) names = append(names, t.(map[string]any)["name"].(string))
} }
assert.ElementsMatch(t, []string{ assert.ElementsMatch(t, []string{
"brain_query", "brain_write", "brain_index", "brain_tunnel", "brain_query", "brain_write", "brain_update", "brain_get",
"brain_pending", "brain_promote",
"brain_index", "brain_tunnel",
"brain_ingest_raw", "brain_ingest", "brain_ingest_raw", "brain_ingest",
"brain_answer", "brain_classify", "brain_graph", "brain_context", "brain_answer", "brain_classify", "brain_graph", "brain_context",
"session_log", "session_log",
@@ -92,3 +96,22 @@ func TestServerUnknownMethodReturnsError(t *testing.T) {
assert.Equal(t, float64(-32601), errObj["code"]) assert.Equal(t, float64(-32601), errObj["code"])
assert.Contains(t, errObj["message"].(string), "unknown/method") assert.Contains(t, errObj["message"].(string), "unknown/method")
} }
type stubTracker struct{}
func (stubTracker) CreateIssue(context.Context, string, string, string) (capture.IssueRef, error) {
return capture.IssueRef{}, nil
}
func (stubTracker) CloseIssue(context.Context, string, int, string) (capture.IssueRef, error) {
return capture.IssueRef{}, nil
}
func (stubTracker) CommentIssue(context.Context, string, int, string) (capture.IssueRef, error) {
return capture.IssueRef{}, nil
}
func TestWithIssueTrackerInjects(t *testing.T) {
srv := mcp.NewServer(t.TempDir(), nil, nil, nil)
assert.Nil(t, srv.IssueTracker(), "tracker is off by default")
srv = srv.WithIssueTracker(stubTracker{})
assert.NotNil(t, srv.IssueTracker(), "tracker injected behind the interface")
}
+16 -3
View File
@@ -77,9 +77,22 @@ func (s *Server) brainAnswer(ctx context.Context, args json.RawMessage) (json.Ra
return nil, fmt.Errorf("search: %w", err) return nil, fmt.Errorf("search: %w", err)
} }
if s.reranker != nil && len(results) > 0 { if s.reranker != nil && len(results) > 0 {
results, err = rerankResults(ctx, s.reranker, a.Query, results, 5) reranked, rerr := rerankResults(ctx, s.reranker, a.Query, results, 5)
if err != nil { if rerr != nil {
return nil, fmt.Errorf("rerank: %w", err) return nil, fmt.Errorf("rerank: %w", rerr)
}
// The reranker is a filter, not a gate. The Qwen3-Reranker is a
// web-search cross-encoder: against a conversational / personal-
// intent query ("what am I optimizing toward?") it scores even
// on-topic notes as "no", which would collapse the whole answer to
// "no relevant content" despite BM25 having retrieved relevant
// content. When the reranker keeps nothing, fall back to the
// BM25/vector ordering (capped to the no-reranker depth) rather
// than returning an empty answer.
if len(reranked) > 0 {
results = reranked
} else if len(results) > 10 {
results = results[:10]
} }
} }
if len(results) == 0 { if len(results) == 0 {
@@ -98,6 +98,42 @@ func TestBrainAnswer_RerankerFiltersBeforeLLM(t *testing.T) {
assert.NotContains(t, sawSources, "noise.md") assert.NotContains(t, sawSources, "noise.md")
} }
func TestBrainAnswer_RerankerKeepsNone_FallsBackToBM25(t *testing.T) {
brainDir := brainDirWithContent(t) // test.md BM25-matches "pass-rate logging"
// Reranker rejects every candidate ("no" to all) — models a
// web-search cross-encoder facing a conversational / personal-intent
// query, which is exactly when it wrongly scores on-topic notes as
// irrelevant. The answer must still synthesize from the BM25 hits, not
// collapse to "no relevant content".
rrSrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_ = json.NewEncoder(w).Encode(map[string]any{"response": "no", "done": true})
}))
defer rrSrv.Close()
var sawSources string
llm := func(_ context.Context, _, user string) (string, error) {
sawSources = user
return "fallback answer", nil
}
srv := mcp.NewServer(brainDir, nil, nil, llm).
WithReranker(reranker.New(rrSrv.URL, "qwen3"))
ts := httptest.NewServer(srv)
defer ts.Close()
rpc := callTool(t, ts, "brain_answer", map[string]any{"query": "pass-rate logging"})
require.Nil(t, rpc["error"])
content := rpc["result"].(map[string]any)["content"].([]any)[0].(map[string]any)["text"].(string)
var result map[string]any
require.NoError(t, json.Unmarshal([]byte(content), &result))
assert.Equal(t, "fallback answer", result["answer"])
assert.NotEmpty(t, result["sources"], "reranker keeping nothing must fall back to BM25, not empty")
assert.Contains(t, sawSources, "test.md")
}
func TestBrainAnswer_NoLLM(t *testing.T) { func TestBrainAnswer_NoLLM(t *testing.T) {
srv := mcp.NewServer(t.TempDir(), nil, nil, nil) srv := mcp.NewServer(t.TempDir(), nil, nil, nil)
ts := httptest.NewServer(srv) ts := httptest.NewServer(srv)
+105
View File
@@ -0,0 +1,105 @@
package mcp
import (
"context"
"encoding/json"
"fmt"
"github.com/mathiasbq/hyperguild/ingestion/internal/capturehttp"
)
// principalKey is the context key under which the authenticated principal
// (re-derived in ServeHTTP) is stashed for the capture tool.
type principalKeyT struct{}
var principalKey principalKeyT
type principalInfo struct {
principal string
viaStatic bool
}
func withPrincipal(ctx context.Context, principal string, viaStatic bool) context.Context {
return context.WithValue(ctx, principalKey, principalInfo{principal: principal, viaStatic: viaStatic})
}
// captureToolDescriptor is the tools/list entry for the capture relay.
// Appended only when WithCapture has wired the tool.
func captureToolDescriptor() map[string]any {
str := func(d string) map[string]any { return map[string]any{"type": "string", "description": d} }
insightItem := map[string]any{
"type": "object",
"properties": map[string]any{
"text": str("the insight body"), "wing": str("brain wing"),
"hall": str("brain hall (facts/decisions/failures/hypotheses/sources)"),
"supersede_slug": str("optional: slug of a prior note to revise in place instead of creating"),
},
"required": []string{"text", "wing", "hall"},
}
ticketItem := map[string]any{
"type": "object",
"properties": map[string]any{
"repo": str("gitea repo (owner is always mathias)"), "action": str("create|close|comment"),
"number": map[string]any{"type": "integer", "description": "issue number (close/comment)"},
"title": str("issue title (create)"), "body": str("issue/comment body"),
},
"required": []string{"repo", "action"},
}
schema := map[string]any{
"type": "object",
"properties": map[string]any{
"context": map[string]any{
"type": "object",
"properties": map[string]any{
"harness": str("descriptive harness label (telemetry only, never a gate input)"),
"session_ref": str("optional session reference"), "fidelity": str("live-capture|transcript-parse|agent-runlog"),
"actor": str("acting user/agent"), "classification": str("caller-declared sensitivity: public|internal|confidential"),
},
},
"insights": map[string]any{"type": "array", "items": insightItem},
"tickets": map[string]any{"type": "array", "items": ticketItem},
"summary": map[string]any{"type": "object", "properties": map[string]any{
"title": str("summary title"), "body": str("summary body"),
"repos_touched": map[string]any{"type": "array", "items": map[string]any{"type": "string"}},
}},
"dry_run": map[string]any{"type": "boolean", "description": "validate + return the would-be receipt, write nothing"},
},
}
b, _ := json.Marshal(schema)
return map[string]any{
"name": "capture",
"description": "Persist a session's value uniformly: insights → brain (write or supersede), action items → Gitea tickets, optional summary → ai-sessions. The relay door for MCP-native harnesses. Origin is server-derived from your authenticated identity; confidential captures through a us-nexus surface are refused (I1). Returns a partial-aware receipt.",
"inputSchema": json.RawMessage(b),
}
}
// brainCapture is the MCP capture tool: the #55 relay for MCP-native
// harnesses. It re-uses the same CaptureService, principal-derivation, and
// origin resolver as POST /capture — only the transport differs. It holds
// no state and retains nothing beyond the I5 audit record.
func (s *Server) brainCapture(ctx context.Context, args json.RawMessage) (json.RawMessage, error) {
if s.capture == nil {
return nil, fmt.Errorf("capture tool not configured")
}
info, ok := ctx.Value(principalKey).(principalInfo)
if !ok || info.principal == "" {
// No authenticated principal ⇒ cannot derive origin ⇒ cannot gate.
return nil, fmt.Errorf("capture requires an authenticated principal")
}
in, err := capturehttp.DecodeRequest(args)
if err != nil {
return nil, fmt.Errorf("invalid capture request: %w", err)
}
// Principal and origin are server-derived — never taken from the body.
in.Context.Principal = info.principal
in.Context.Origin = s.capture.resolver.Resolve(info.principal, info.viaStatic)
rec, err := s.capture.svc.Capture(ctx, in)
if err != nil {
// Surface I1/I5 refusals and validation failures verbatim; errors.Is
// markers (ErrSovereigntyRefused / ErrAuditUnavailable) ride in the message.
return nil, err
}
return json.Marshal(rec)
}
@@ -0,0 +1,150 @@
package mcp_test
import (
"bytes"
"context"
"encoding/json"
"errors"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"github.com/mathiasbq/hyperguild/ingestion/internal/audit"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainstore"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
"github.com/mathiasbq/hyperguild/ingestion/internal/capturehttp"
"github.com/mathiasbq/hyperguild/ingestion/internal/classification"
"github.com/mathiasbq/hyperguild/ingestion/internal/mcp"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
const capStaticTok = "cap-static-tok"
type capFakeTracker struct{}
func (capFakeTracker) CreateIssue(context.Context, string, string, string) (capture.IssueRef, error) {
return capture.IssueRef{Repo: "hyperguild", Number: 1, URL: "https://git/1"}, nil
}
func (capFakeTracker) CloseIssue(context.Context, string, int, string) (capture.IssueRef, error) {
return capture.IssueRef{}, nil
}
func (capFakeTracker) CommentIssue(context.Context, string, int, string) (capture.IssueRef, error) {
return capture.IssueRef{}, nil
}
type capFakeValidator struct {
subject string
err error
}
func (v capFakeValidator) Validate(context.Context, string) (string, error) {
return v.subject, v.err
}
func captureServer(t *testing.T, validator capturehttp.Validator, sovereign []string) (*mcp.Server, string) {
t.Helper()
brainDir := t.TempDir()
cfg, err := classification.Load(brainDir)
require.NoError(t, err)
svc := capture.NewService(brainstore.New(brainDir), capFakeTracker{}, nil, cfg, audit.NewSlogSink(nil))
srv := mcp.NewServer(brainDir, nil, nil, nil)
srv.WithCapture(svc, validator, capStaticTok, "local-cli", capturehttp.NewOriginResolver(sovereign))
return srv, brainDir
}
func captureCall(t *testing.T, srv http.Handler, authz string, args map[string]any) map[string]any {
t.Helper()
body, _ := json.Marshal(map[string]any{
"jsonrpc": "2.0", "id": 1, "method": "tools/call",
"params": map[string]any{"name": "capture", "arguments": args},
})
req := httptest.NewRequest(http.MethodPost, "/mcp", bytes.NewReader(body))
if authz != "" {
req.Header.Set("Authorization", authz)
}
rr := httptest.NewRecorder()
srv.ServeHTTP(rr, req)
var resp map[string]any
require.NoError(t, json.Unmarshal(rr.Body.Bytes(), &resp))
return resp
}
func TestCaptureToolListedWhenWired(t *testing.T) {
srv, _ := captureServer(t, nil, nil)
body, _ := json.Marshal(map[string]any{"jsonrpc": "2.0", "id": 1, "method": "tools/list"})
req := httptest.NewRequest(http.MethodPost, "/mcp", bytes.NewReader(body))
rr := httptest.NewRecorder()
srv.ServeHTTP(rr, req)
assert.Contains(t, rr.Body.String(), `"capture"`)
}
func TestCaptureToolNotListedByDefault(t *testing.T) {
srv := mcp.NewServer(t.TempDir(), nil, nil, nil) // no WithCapture
body, _ := json.Marshal(map[string]any{"jsonrpc": "2.0", "id": 1, "method": "tools/list"})
req := httptest.NewRequest(http.MethodPost, "/mcp", bytes.NewReader(body))
rr := httptest.NewRecorder()
srv.ServeHTTP(rr, req)
assert.NotContains(t, rr.Body.String(), `"capture"`)
}
func TestCaptureToolForwardsViaStaticPrincipal(t *testing.T) {
srv, brainDir := captureServer(t, nil, nil)
resp := captureCall(t, srv, "Bearer "+capStaticTok, map[string]any{
"context": map[string]any{"harness": "claude-code", "actor": "mathias", "classification": "internal"},
"insights": []map[string]any{{"text": "a fact", "wing": "hyperguild", "hall": "facts"}},
"tickets": []map[string]any{{"repo": "hyperguild", "action": "create", "title": "t"}},
})
require.Nil(t, resp["error"], "got error: %v", resp["error"])
text := resp["result"].(map[string]any)["content"].([]any)[0].(map[string]any)["text"].(string)
var rec capture.CaptureReceipt
require.NoError(t, json.Unmarshal([]byte(text), &rec))
assert.True(t, rec.Insights[0].OK)
assert.True(t, rec.Tickets[0].OK)
// Forwarded to the real brain store.
_, statErr := os.Stat(filepath.Join(brainDir, "wiki/hyperguild/facts"))
require.NoError(t, statErr)
}
func TestCaptureToolRefusesConfidentialViaUSNexus(t *testing.T) {
// JWT principal not in the sovereign allowlist ⇒ us-nexus origin.
srv, _ := captureServer(t, capFakeValidator{subject: "claudeai-oauth"}, nil)
resp := captureCall(t, srv, "Bearer jwt-token", map[string]any{
"context": map[string]any{"harness": "claudeai-chat", "actor": "mathias", "classification": "confidential"},
"insights": []map[string]any{{"text": "secret", "wing": "client-seb", "hall": "facts"}},
})
require.NotNil(t, resp["error"])
assert.Contains(t, resp["error"].(map[string]any)["message"].(string), "sovereignty")
}
func TestCaptureToolAllowsConfidentialViaSovereignJWT(t *testing.T) {
srv, _ := captureServer(t, capFakeValidator{subject: "koala-cli"}, []string{"koala-cli"})
resp := captureCall(t, srv, "Bearer jwt-token", map[string]any{
"context": map[string]any{"harness": "claude-code", "actor": "mathias", "classification": "confidential"},
"insights": []map[string]any{{"text": "secret", "wing": "client-seb", "hall": "facts"}},
})
assert.Nil(t, resp["error"], "sovereign JWT principal should be allowed: %v", resp["error"])
}
func TestCaptureToolRejectsUnauthenticated(t *testing.T) {
srv, _ := captureServer(t, capFakeValidator{err: errors.New("no jwt")}, nil)
resp := captureCall(t, srv, "", map[string]any{ // no Authorization
"context": map[string]any{"harness": "x", "classification": "internal"},
"insights": []map[string]any{{"text": "a", "wing": "hyperguild", "hall": "facts"}},
})
require.NotNil(t, resp["error"])
assert.Contains(t, resp["error"].(map[string]any)["message"].(string), "authenticated principal")
}
func TestCaptureToolCallerCannotForgeOrigin(t *testing.T) {
// Body asserts sovereign harness, but the us-nexus JWT principal governs.
srv, _ := captureServer(t, capFakeValidator{subject: "claudeai-oauth"}, nil)
resp := captureCall(t, srv, "Bearer jwt", map[string]any{
"context": map[string]any{"harness": "sovereign-soil", "classification": "confidential"},
"insights": []map[string]any{{"text": "secret", "wing": "client-seb", "hall": "facts"}},
})
require.NotNil(t, resp["error"])
assert.Contains(t, resp["error"].(map[string]any)["message"].(string), "sovereignty")
}
+9
View File
@@ -76,6 +76,15 @@ func buildFrontmatter(rp RawPage, date string) string {
} }
fmt.Fprintf(&sb, "date_ingested: %s\n", date) fmt.Fprintf(&sb, "date_ingested: %s\n", date)
fmt.Fprintf(&sb, "last_updated: %s\n", date) fmt.Fprintf(&sb, "last_updated: %s\n", date)
if rp.Source != "" {
fmt.Fprintf(&sb, "source: %s\n", yamlScalar(rp.Source))
}
if rp.Author != "" {
fmt.Fprintf(&sb, "author: %s\n", yamlScalar(rp.Author))
}
if rp.Published != "" {
fmt.Fprintf(&sb, "published: %s\n", yamlScalar(rp.Published))
}
case "concept": case "concept":
if rp.Domain != "" { if rp.Domain != "" {
fmt.Fprintf(&sb, "domain: %s\n", yamlScalar(rp.Domain)) fmt.Fprintf(&sb, "domain: %s\n", yamlScalar(rp.Domain))
+46
View File
@@ -154,6 +154,52 @@ func TestBuildPages_EntityNoSubtype(t *testing.T) {
assert.Contains(t, pages[0].Content, "title: 'Basecamp'") assert.Contains(t, pages[0].Content, "title: 'Basecamp'")
} }
func TestBuildPages_SourcePageCarriesSourceAuthorPublished(t *testing.T) {
raw := []RawPage{
{
Title: "Ornith",
Type: "source",
Subtype: "article",
Content: "## Summary\n\nAn agentic coding model.\n",
Source: "https://example.com/ornith",
Author: "Jane Doe",
Published: "2026-07-20",
},
}
pages, warnings := BuildPages(raw, "ornith", "2026-07-26")
require.Len(t, pages, 1)
assert.Empty(t, warnings)
p := pages[0]
assert.Contains(t, p.Content, "source: 'https://example.com/ornith'")
assert.Contains(t, p.Content, "author: 'Jane Doe'")
assert.Contains(t, p.Content, "published: '2026-07-20'")
}
func TestBuildPages_SourcePageOmitsSourceAuthorPublishedWhenEmpty(t *testing.T) {
raw := []RawPage{
{Title: "Shape Up", Type: "source", Subtype: "book", Content: "## Summary\n\nA book.\n"},
}
pages, _ := BuildPages(raw, "shape-up", "2026-04-23")
require.Len(t, pages, 1)
assert.NotContains(t, pages[0].Content, "source:")
assert.NotContains(t, pages[0].Content, "author:")
assert.NotContains(t, pages[0].Content, "published:")
}
func TestBuildPages_ConceptPageIgnoresSourceAuthorPublished(t *testing.T) {
// source/author/published are source-note-only metadata; a concept page
// shouldn't carry them even if somehow set on the RawPage.
raw := []RawPage{
{Title: "Betting", Type: "concept", Content: "## Definition\n\nFoo.\n", Source: "x", Author: "y", Published: "z"},
}
pages, _ := BuildPages(raw, "src", "2026-04-23")
require.Len(t, pages, 1)
assert.NotContains(t, pages[0].Content, "source:")
assert.NotContains(t, pages[0].Content, "author:")
assert.NotContains(t, pages[0].Content, "published:")
}
func TestBuildPages_EmptyTitleSkippedWithWarning(t *testing.T) { func TestBuildPages_EmptyTitleSkippedWithWarning(t *testing.T) {
raw := []RawPage{ raw := []RawPage{
{Title: "", Type: "concept", Content: "## Definition\n\nFoo.\n"}, {Title: "", Type: "concept", Content: "## Definition\n\nFoo.\n"},
+25 -5
View File
@@ -51,6 +51,21 @@ func buildTitleMap(pages []wiki.Page, inventory map[wiki.PageType][]wiki.Entry)
return m return m
} }
// pathStylePrefixes are known root prefixes the LLM extraction step
// sometimes bakes into a wikilink target instead of emitting a clean
// wing/hall/slug path (or bare title). Stripping them repairs the link
// in place — see hyperguild#87.
var pathStylePrefixes = []string{"wing:", "wiki/"}
func stripPathStylePrefix(displayName string) (string, bool) {
for _, prefix := range pathStylePrefixes {
if stripped, ok := strings.CutPrefix(displayName, prefix); ok {
return stripped, true
}
}
return displayName, false
}
func canonicalizeContent(content string, titleToSlug map[string]string) (string, []string) { func canonicalizeContent(content string, titleToSlug map[string]string) (string, []string) {
var warnings []string var warnings []string
result := plainLinkRE.ReplaceAllStringFunc(content, func(match string) string { result := plainLinkRE.ReplaceAllStringFunc(content, func(match string) string {
@@ -59,12 +74,17 @@ func canonicalizeContent(content string, titleToSlug map[string]string) (string,
return match return match
} }
displayName := sub[1] displayName := sub[1]
slug, ok := titleToSlug[strings.ToLower(displayName)]
if !ok { if slug, ok := titleToSlug[strings.ToLower(displayName)]; ok {
warnings = append(warnings, fmt.Sprintf("unknown wikilink: [[%s]]", displayName)) return "[[" + slug + "|" + displayName + "]]"
return match
} }
return "[[" + slug + "|" + displayName + "]]"
if stripped, hadPrefix := stripPathStylePrefix(displayName); hadPrefix {
return "[[" + stripped + "]]"
}
warnings = append(warnings, fmt.Sprintf("unknown wikilink: [[%s]]", displayName))
return match
}) })
return result, warnings return result, warnings
} }
+47
View File
@@ -102,6 +102,53 @@ func TestCanonicalizeLinks_CurrentBatchPagesResolved(t *testing.T) {
assert.Contains(t, got[0].Content, "[[betting|Betting]]") assert.Contains(t, got[0].Content, "[[betting|Betting]]")
} }
func TestCanonicalizeLinks_StripsWingColonPrefix(t *testing.T) {
pages := []wiki.Page{
{
Path: "wiki/homelab/failures/act-runner-host-mode-container-needs-node-and-libatomic.md",
Content: "---\ntitle: 'act_runner host-mode'\n---\n\nSee [[wing:homelab/failures/rootless-buildah-act-runner-run-containers-denied]].\n",
},
}
got, warnings := CanonicalizeLinks(pages, map[wiki.PageType][]wiki.Entry{})
require.Len(t, got, 1)
assert.Empty(t, warnings)
assert.Contains(t, got[0].Content, "[[homelab/failures/rootless-buildah-act-runner-run-containers-denied]]")
assert.NotContains(t, got[0].Content, "wing:")
}
func TestCanonicalizeLinks_StripsWikiSlashPrefix(t *testing.T) {
pages := []wiki.Page{
{
Path: "wiki/agentsquad/hypotheses/council-consolidation-standalone-deliberation-service.md",
Content: "---\ntitle: 'council consolidation'\n---\n\nSee [[wiki/agentsquad/decisions/autoresearch-council-sibling-pipe]].\n",
},
}
got, warnings := CanonicalizeLinks(pages, map[wiki.PageType][]wiki.Entry{})
require.Len(t, got, 1)
assert.Empty(t, warnings)
assert.Contains(t, got[0].Content, "[[agentsquad/decisions/autoresearch-council-sibling-pipe]]")
assert.NotContains(t, got[0].Content, "wiki/agentsquad/decisions/autoresearch-council-sibling-pipe]]\n\n") // no leftover wiki/ prefix
assert.NotContains(t, got[0].Content, "[[wiki/")
}
func TestCanonicalizeLinks_TitleLookupStillTakesPriorityOverPrefixStrip(t *testing.T) {
// A plain link that resolves via the title map must still use the
// normal slug|Display form, not fall through to prefix-strip repair.
pages := []wiki.Page{
{
Path: "wiki/sources/shape-up.md",
Content: "---\ntitle: 'Shape Up'\n---\n\nSee [[Betting]].\n",
},
}
inventory := map[wiki.PageType][]wiki.Entry{
wiki.PageTypeConcept: {{Slug: "betting", Title: "Betting"}},
}
got, warnings := CanonicalizeLinks(pages, inventory)
require.Len(t, got, 1)
assert.Empty(t, warnings)
assert.Contains(t, got[0].Content, "[[betting|Betting]]")
}
func TestCanonicalizeLinks_MultipleLinksInOnePage(t *testing.T) { func TestCanonicalizeLinks_MultipleLinksInOnePage(t *testing.T) {
pages := []wiki.Page{ pages := []wiki.Page{
{ {
+62
View File
@@ -15,6 +15,14 @@ type RawPage struct {
Subtype string `json:"subtype"` // entity: person|company|tool|model|framework|technology; source: article|pdf|book|video|note|project Subtype string `json:"subtype"` // entity: person|company|tool|model|framework|technology; source: article|pdf|book|video|note|project
Domain string `json:"domain"` Domain string `json:"domain"`
Content string `json:"content"` // Markdown body only — no frontmatter Content string `json:"content"` // Markdown body only — no frontmatter
// Source, Author, Published are deterministic passthrough from the raw
// ingested content's own frontmatter (see parseContentFrontmatter) — never
// set by the LLM. json:"-" keeps them immune to same-named keys the LLM
// might emit. Only meaningful for Type == "source".
Source string `json:"-"`
Author string `json:"-"`
Published string `json:"-"`
} }
// ParseRawPages parses LLM output as a JSON array of RawPage objects. // ParseRawPages parses LLM output as a JSON array of RawPage objects.
@@ -98,6 +106,60 @@ func repairJSON(s string) string {
return b.String() return b.String()
} }
// sourceMeta is source/author/published pulled from the raw ingested
// content's own frontmatter — deterministic passthrough, never LLM output.
type sourceMeta struct {
Source string
Author string
Published string
}
// parseContentFrontmatter extracts source/author/published from a leading
// "---\n...\n---" YAML block in raw ingested content. Only these three flat
// scalar keys are recognised; anything else in the block is ignored. Returns
// a zero-value sourceMeta if content has no frontmatter block.
func parseContentFrontmatter(content string) sourceMeta {
var meta sourceMeta
if !strings.HasPrefix(content, "---\n") && !strings.HasPrefix(content, "---\r\n") {
return meta
}
lines := strings.Split(content, "\n")
for _, line := range lines[1:] {
if strings.TrimSpace(line) == "---" {
break
}
key, val, ok := strings.Cut(line, ":")
if !ok {
continue
}
key = strings.TrimSpace(key)
val = strings.Trim(strings.TrimSpace(val), `"'`)
switch key {
case "source":
meta.Source = val
case "author":
meta.Author = val
case "published":
meta.Published = val
}
}
return meta
}
// applySourceMeta deterministically overwrites Source/Author/Published on
// every "source"-type page with meta — the LLM never controls these fields.
func applySourceMeta(pages []RawPage, meta sourceMeta) {
for i := range pages {
if pages[i].Type != "source" {
continue
}
pages[i].Source = meta.Source
pages[i].Author = meta.Author
pages[i].Published = meta.Published
}
}
func stripFences(s string) string { func stripFences(s string) string {
for _, prefix := range []string{"```json\n", "```json\r\n", "```\n", "```\r\n"} { for _, prefix := range []string{"```json\n", "```json\r\n", "```\n", "```\r\n"} {
if strings.HasPrefix(s, prefix) { if strings.HasPrefix(s, prefix) {
+2
View File
@@ -59,6 +59,8 @@ func Run(ctx context.Context, cfg Config, brainDir, content, source string, dryR
allWarnings = append(allWarnings, warnings...) allWarnings = append(allWarnings, warnings...)
} }
applySourceMeta(allRaw, parseContentFrontmatter(content))
return buildAndWrite(allRaw, sourceSlug, date, brainDir, source, inventory, allWarnings, dryRun) return buildAndWrite(allRaw, sourceSlug, date, brainDir, source, inventory, allWarnings, dryRun)
} }
@@ -130,6 +130,40 @@ func TestRun_MergesDuplicatePaths(t *testing.T) {
assert.Contains(t, string(content), "[[Baz]]") assert.Contains(t, string(content), "[[Baz]]")
} }
func TestRun_ThreadsSourceAuthorPublishedFromContentFrontmatter(t *testing.T) {
brainDir := t.TempDir()
for _, sub := range []string{"wiki/concepts", "wiki/entities", "wiki/sources"} {
require.NoError(t, os.MkdirAll(filepath.Join(brainDir, sub), 0o755))
}
llmResponse := mustJSON([]RawPage{{
Title: "Ornith",
Type: "source",
Subtype: "article",
Content: "## Summary\n\nAn agentic coding model.\n",
}})
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_ = json.NewEncoder(w).Encode(map[string]any{
"choices": []map[string]any{{"message": map[string]any{"content": llmResponse}}},
})
}))
defer srv.Close()
cfg := Config{Complete: llm.New(srv.URL, "", "m", 30*time.Second).Complete}
rawContent := "---\nsource: https://example.com/ornith\nauthor: Jane Doe\npublished: 2026-07-20\n---\n\nAn agentic coding model that runs on your laptop.\n"
result, err := Run(context.Background(), cfg, brainDir, rawContent, "ornith", false)
require.NoError(t, err)
require.Len(t, result.Pages, 1)
content, err := os.ReadFile(filepath.Join(brainDir, "wiki", "sources", "ornith.md"))
require.NoError(t, err)
assert.Contains(t, string(content), "source: 'https://example.com/ornith'")
assert.Contains(t, string(content), "author: 'Jane Doe'")
assert.Contains(t, string(content), "published: '2026-07-20'")
}
func mustJSON(v any) string { func mustJSON(v any) string {
b, err := json.Marshal(v) b, err := json.Marshal(v)
if err != nil { if err != nil {
+65
View File
@@ -3,6 +3,7 @@ package vectorstore
import ( import (
"fmt" "fmt"
"strings" "strings"
"unicode/utf8"
) )
// NumberedChunk pairs a chunk's body with the storage path it will use // NumberedChunk pairs a chunk's body with the storage path it will use
@@ -66,6 +67,70 @@ func ChunkMarkdown(content string, maxBytes int) []string {
} }
out = append(out, splitAtParagraphs(s, maxBytes)...) out = append(out, splitAtParagraphs(s, maxBytes)...)
} }
// Final guarantee: no chunk exceeds maxBytes. A single heading-less,
// paragraph-less block (JSON-lines, minified content) survives the two
// passes above whole — splitAtParagraphs emits an over-budget paragraph
// rather than truncating prose. Hard-split any such chunk at line/rune
// boundaries so the embedder never rejects an over-context chunk.
final := make([]string, 0, len(out))
for _, c := range out {
if len(c) <= maxBytes {
final = append(final, c)
continue
}
final = append(final, hardSplit(c, maxBytes)...)
}
return final
}
// hardSplit slices s into pieces no larger than maxBytes, breaking at line
// boundaries where possible and otherwise mid-line at a UTF-8 rune boundary.
// Last resort for content that has neither headings nor blank-line paragraphs.
func hardSplit(s string, maxBytes int) []string {
var out []string
var cur strings.Builder
flush := func() {
if cur.Len() > 0 {
out = append(out, cur.String())
cur.Reset()
}
}
for _, line := range strings.SplitAfter(s, "\n") {
if line == "" {
continue
}
if len(line) > maxBytes {
flush()
out = append(out, runeSplit(line, maxBytes)...)
continue
}
if cur.Len() > 0 && cur.Len()+len(line) > maxBytes {
flush()
}
cur.WriteString(line)
}
flush()
return out
}
// runeSplit slices s into <=maxBytes pieces without splitting a UTF-8 rune.
func runeSplit(s string, maxBytes int) []string {
var out []string
for len(s) > maxBytes {
cut := maxBytes
for cut > 0 && !utf8.RuneStart(s[cut]) {
cut--
}
if cut == 0 { // single rune wider than the budget; emit it whole
cut = maxBytes
}
out = append(out, s[:cut])
s = s[cut:]
}
if len(s) > 0 {
out = append(out, s)
}
return out return out
} }
@@ -30,6 +30,23 @@ func TestChunkMarkdown_SplitsAtHeadings(t *testing.T) {
} }
} }
func TestChunkMarkdown_HardSplitsHeadinglessOversizedBlock(t *testing.T) {
// A document with no headings and no blank-line paragraph breaks (e.g.
// JSON-lines like wiki/telos/decisions/human-intent-column.md). The old
// chunker emitted it as one over-budget chunk → nomic-embed returned
// "input length exceeds the context length" (400). Every chunk must now
// fit the budget, with no content lost.
maxBytes := 200
src := strings.Repeat("x", 1000) // one 1000-byte blob, no headings, no \n\n
out := vectorstore.ChunkMarkdown(src, maxBytes)
require.Greater(t, len(out), 1, "oversized blob must be split")
for i, c := range out {
assert.LessOrEqual(t, len(c), maxBytes, "chunk %d over budget: %d bytes", i, len(c))
}
assert.Equal(t, 1000, strings.Count(strings.Join(out, ""), "x"), "no content lost")
}
func TestChunkMarkdown_FurtherSplitsOversizedSection(t *testing.T) { func TestChunkMarkdown_FurtherSplitsOversizedSection(t *testing.T) {
// One H2 section with 4 paragraphs of ~80 chars each, limit 100. // One H2 section with 4 paragraphs of ~80 chars each, limit 100.
src := "## big\n\n" + src := "## big\n\n" +
+7
View File
@@ -74,6 +74,13 @@ func processDir(ctx context.Context, cfg Config, date string) []error {
return nil return nil
} }
// AutoTunnel's own fuzzy-match human-review queue, not source
// material to extract (hyperguild#88) — same exclusion as
// api.ListPending already applies when listing raw/ for promotion.
if strings.HasPrefix(d.Name(), "tunnel-candidates-") {
return nil
}
// Skip files that have already been processed or permanently failed. // Skip files that have already been processed or permanently failed.
if _, err := os.Stat(path + ".processed"); err == nil { if _, err := os.Stat(path + ".processed"); err == nil {
return nil return nil
@@ -229,3 +229,49 @@ func TestProcessDir_SkipsSubdirs(t *testing.T) {
_, err = os.Stat(failedFile) _, err = os.Stat(failedFile)
assert.NoError(t, err, "failed subdir file should be untouched") assert.NoError(t, err, "failed subdir file should be untouched")
} }
// TestProcessDir_SkipsTunnelCandidateFiles guards against hyperguild#88:
// AutoTunnel's own human-review queue (brain/raw/tunnel-candidates-*.md)
// must never be re-ingested as if it were external source material — doing
// so fed the raw "(term: X)" log entries back through the LLM extraction
// pipeline, which then legitimately (from its own perspective) surfaced
// [[X]] as a wikilink for any term that happened to match a real page
// title, however generic (e.g. "mission").
func TestProcessDir_SkipsTunnelCandidateFiles(t *testing.T) {
brainDir := setupBrainDir(t)
tunnelFile := filepath.Join(brainDir, "raw", "tunnel-candidates-2026-07-19.md")
require.NoError(t, os.WriteFile(tunnelFile, []byte(
"# Tunnel candidates 2026-07-19\n\n- `wiki/homelab/decisions/foo.md` ↔ `wiki/telos/decisions/mission.md` (term: \"mission\")\n",
), 0o644))
var completeCalls int
completeFn := func(ctx context.Context, system, user string) (string, error) {
completeCalls++
raw := pipeline.RawPage{Title: "Should not be written", Type: "source", Subtype: "article", Content: "## Summary\n\nx.\n"}
b, _ := json.Marshal([]pipeline.RawPage{raw})
return string(b), nil
}
cfg := Config{
BrainDir: brainDir,
Interval: time.Hour, // not used; we call processDir directly
Pipeline: pipeline.Config{
Complete: completeFn,
ChunkSize: 0,
Schema: "# Schema\nThree page types.",
},
}
date := time.Now().UTC().Format("2006-01-02")
errs := processDir(context.Background(), cfg, date)
assert.Empty(t, errs)
assert.Zero(t, completeCalls, "tunnel-candidates file must never reach the LLM extraction pipeline")
// File must be left alone in raw/ — not moved to processed/, no marker written.
_, err := os.Stat(tunnelFile + ".processed")
assert.True(t, os.IsNotExist(err), "tunnel-candidates file should not get a .processed marker")
_, err = os.Stat(filepath.Join(brainDir, "raw", "processed", date, "tunnel-candidates-2026-07-19.md"))
assert.True(t, os.IsNotExist(err), "tunnel-candidates file should not be copied to processed/")
}
+109
View File
@@ -0,0 +1,109 @@
// Package webhook triggers an on-demand brain-sync Job when Gitea pushes to
// mathias/brain, instead of waiting for the next 15-minute CronJob poll.
package webhook
import (
"context"
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"log/slog"
"net/http"
batchv1 "k8s.io/api/batch/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
)
// VerifySignature checks a Gitea webhook's X-Gitea-Signature header: a
// hex-encoded HMAC-SHA256 of the raw request body, keyed by the shared
// webhook secret. Constant-time compare — timing must not leak how much of
// the signature matched.
func VerifySignature(payload []byte, signatureHeader, secret string) bool {
if signatureHeader == "" {
return false
}
mac := hmac.New(sha256.New, []byte(secret))
mac.Write(payload)
expected := hex.EncodeToString(mac.Sum(nil))
return hmac.Equal([]byte(expected), []byte(signatureHeader))
}
// pushEvent is the subset of Gitea's push webhook payload this handler needs.
type pushEvent struct {
Ref string `json:"ref"`
Repo struct {
FullName string `json:"full_name"`
} `json:"repository"`
}
// TriggerJobFromCronJob reads the named CronJob's job template and creates a
// new, uniquely-named Job from it — the same thing `kubectl create job
// --from=cronjob/<name>` does. Reuses the CronJob's already-tested script
// rather than re-implementing sync logic here.
func TriggerJobFromCronJob(ctx context.Context, cs kubernetes.Interface, namespace, cronJobName string) (string, error) {
cj, err := cs.BatchV1().CronJobs(namespace).Get(ctx, cronJobName, metav1.GetOptions{})
if err != nil {
return "", fmt.Errorf("get cronjob %s/%s: %w", namespace, cronJobName, err)
}
job := &batchv1.Job{
ObjectMeta: metav1.ObjectMeta{
GenerateName: cronJobName + "-webhook-",
Namespace: namespace,
Annotations: map[string]string{
"triggered-by": "brain-webhook",
},
},
Spec: cj.Spec.JobTemplate.Spec,
}
created, err := cs.BatchV1().Jobs(namespace).Create(ctx, job, metav1.CreateOptions{})
if err != nil {
return "", fmt.Errorf("create job from cronjob %s/%s: %w", namespace, cronJobName, err)
}
return created.Name, nil
}
// Handler is the HTTP handler for Gitea's push webhook on mathias/brain.
type Handler struct {
Secret string
Clientset kubernetes.Interface
Namespace string // e.g. "brain"
CronJobName string // e.g. "brain-sync"
WatchRepo string // e.g. "mathias/brain"
Logger *slog.Logger
}
func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
body, err := io.ReadAll(r.Body)
if err != nil {
http.Error(w, "bad body", http.StatusBadRequest)
return
}
sig := r.Header.Get("X-Gitea-Signature")
if !VerifySignature(body, sig, h.Secret) {
http.Error(w, "bad signature", http.StatusUnauthorized)
return
}
var ev pushEvent
if err := json.Unmarshal(body, &ev); err != nil {
http.Error(w, "bad payload", http.StatusBadRequest)
return
}
if ev.Repo.FullName != h.WatchRepo || ev.Ref != "refs/heads/main" {
w.WriteHeader(http.StatusOK)
_, _ = fmt.Fprintf(w, "ignored: repo=%s ref=%s", ev.Repo.FullName, ev.Ref)
return
}
jobName, err := TriggerJobFromCronJob(r.Context(), h.Clientset, h.Namespace, h.CronJobName)
if err != nil {
h.Logger.Error("webhook: trigger job failed", "err", err)
http.Error(w, "trigger failed", http.StatusInternalServerError)
return
}
h.Logger.Info("webhook: triggered brain-sync job", "job", jobName)
w.WriteHeader(http.StatusOK)
_, _ = fmt.Fprintf(w, "triggered %s", jobName)
}
+222
View File
@@ -0,0 +1,222 @@
package webhook
import (
"bytes"
"context"
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"log/slog"
"net/http"
"net/http/httptest"
"os"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
batchv1 "k8s.io/api/batch/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/client-go/kubernetes/fake"
k8stesting "k8s.io/client-go/testing"
)
// newFakeClientset simulates the real API server's GenerateName expansion,
// which the plain fake clientset tracker does not do on its own -- without
// this, every created Job keeps an empty Name (a fake-clientset limitation,
// not real API server behavior).
func newFakeClientset(objects ...runtime.Object) *fake.Clientset {
cs := fake.NewSimpleClientset(objects...)
cs.PrependReactor("create", "jobs", func(action k8stesting.Action) (bool, runtime.Object, error) {
createAction := action.(k8stesting.CreateAction)
job, ok := createAction.GetObject().(*batchv1.Job)
if ok && job.Name == "" && job.GenerateName != "" {
job.Name = job.GenerateName + "test0001"
}
return false, nil, nil // not "handled" -- let the default reactor store it
})
return cs
}
func sign(t *testing.T, payload []byte, secret string) string {
t.Helper()
mac := hmac.New(sha256.New, []byte(secret))
mac.Write(payload)
return hex.EncodeToString(mac.Sum(nil))
}
func testLogger() *slog.Logger {
return slog.New(slog.NewTextHandler(os.Stderr, nil))
}
func fakeCronJob(namespace, name string) *batchv1.CronJob {
return &batchv1.CronJob{
ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: namespace},
Spec: batchv1.CronJobSpec{
JobTemplate: batchv1.JobTemplateSpec{
Spec: batchv1.JobSpec{
Template: corev1.PodTemplateSpec{
Spec: corev1.PodSpec{
Containers: []corev1.Container{
{Name: "sync", Image: "alpine/git:v2.47.2"},
},
RestartPolicy: corev1.RestartPolicyNever,
},
},
},
},
},
}
}
// --------------------------------------------------------------------------- #
// VerifySignature
// --------------------------------------------------------------------------- #
func TestVerifySignature_AcceptsCorrectHMAC(t *testing.T) {
payload := []byte(`{"ref":"refs/heads/main"}`)
secret := "s3cret"
sig := sign(t, payload, secret)
assert.True(t, VerifySignature(payload, sig, secret))
}
func TestVerifySignature_RejectsWrongSecret(t *testing.T) {
payload := []byte(`{"ref":"refs/heads/main"}`)
sig := sign(t, payload, "right-secret")
assert.False(t, VerifySignature(payload, sig, "wrong-secret"))
}
func TestVerifySignature_RejectsTamperedPayload(t *testing.T) {
secret := "s3cret"
sig := sign(t, []byte(`{"ref":"refs/heads/main"}`), secret)
assert.False(t, VerifySignature([]byte(`{"ref":"refs/heads/evil"}`), sig, secret))
}
func TestVerifySignature_RejectsEmptySignature(t *testing.T) {
assert.False(t, VerifySignature([]byte("payload"), "", "secret"))
}
// --------------------------------------------------------------------------- #
// TriggerJobFromCronJob
// --------------------------------------------------------------------------- #
func TestTriggerJobFromCronJob_CreatesJobMatchingTemplate(t *testing.T) {
cs := newFakeClientset(fakeCronJob("brain", "brain-sync"))
jobName, err := TriggerJobFromCronJob(context.Background(), cs, "brain", "brain-sync")
require.NoError(t, err)
assert.NotEmpty(t, jobName)
jobs, err := cs.BatchV1().Jobs("brain").List(context.Background(), metav1.ListOptions{})
require.NoError(t, err)
require.Len(t, jobs.Items, 1)
assert.Equal(t, "alpine/git:v2.47.2", jobs.Items[0].Spec.Template.Spec.Containers[0].Image)
}
func TestTriggerJobFromCronJob_ErrorsWhenCronJobMissing(t *testing.T) {
cs := fake.NewSimpleClientset()
_, err := TriggerJobFromCronJob(context.Background(), cs, "brain", "brain-sync")
assert.Error(t, err)
}
// --------------------------------------------------------------------------- #
// Handler.ServeHTTP
// --------------------------------------------------------------------------- #
func newTestHandler(cs *fake.Clientset) *Handler {
return &Handler{
Secret: "s3cret",
Clientset: cs,
Namespace: "brain",
CronJobName: "brain-sync",
WatchRepo: "mathias/brain",
Logger: testLogger(),
}
}
func pushPayload(t *testing.T, repo, ref string) []byte {
t.Helper()
body := map[string]any{
"ref": ref,
"repository": map[string]any{
"full_name": repo,
},
}
b, err := json.Marshal(body)
require.NoError(t, err)
return b
}
func TestHandler_RejectsMissingSignature(t *testing.T) {
cs := fake.NewSimpleClientset(fakeCronJob("brain", "brain-sync"))
h := newTestHandler(cs)
payload := pushPayload(t, "mathias/brain", "refs/heads/main")
req := httptest.NewRequest(http.MethodPost, "/webhooks/brain-sync", bytes.NewReader(payload))
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
assert.Equal(t, http.StatusUnauthorized, rec.Code)
jobs, _ := cs.BatchV1().Jobs("brain").List(context.Background(), metav1.ListOptions{})
assert.Empty(t, jobs.Items, "must not trigger a job on an unsigned request")
}
func TestHandler_RejectsWrongSignature(t *testing.T) {
cs := fake.NewSimpleClientset(fakeCronJob("brain", "brain-sync"))
h := newTestHandler(cs)
payload := pushPayload(t, "mathias/brain", "refs/heads/main")
req := httptest.NewRequest(http.MethodPost, "/webhooks/brain-sync", bytes.NewReader(payload))
req.Header.Set("X-Gitea-Signature", sign(t, payload, "not-the-real-secret"))
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
assert.Equal(t, http.StatusUnauthorized, rec.Code)
jobs, _ := cs.BatchV1().Jobs("brain").List(context.Background(), metav1.ListOptions{})
assert.Empty(t, jobs.Items)
}
func TestHandler_IgnoresOtherRepos(t *testing.T) {
cs := fake.NewSimpleClientset(fakeCronJob("brain", "brain-sync"))
h := newTestHandler(cs)
payload := pushPayload(t, "mathias/some-other-repo", "refs/heads/main")
req := httptest.NewRequest(http.MethodPost, "/webhooks/brain-sync", bytes.NewReader(payload))
req.Header.Set("X-Gitea-Signature", sign(t, payload, "s3cret"))
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
assert.Equal(t, http.StatusOK, rec.Code)
jobs, _ := cs.BatchV1().Jobs("brain").List(context.Background(), metav1.ListOptions{})
assert.Empty(t, jobs.Items, "must not trigger for a push to an unrelated repo")
}
func TestHandler_IgnoresNonMainBranch(t *testing.T) {
cs := fake.NewSimpleClientset(fakeCronJob("brain", "brain-sync"))
h := newTestHandler(cs)
payload := pushPayload(t, "mathias/brain", "refs/heads/some-feature-branch")
req := httptest.NewRequest(http.MethodPost, "/webhooks/brain-sync", bytes.NewReader(payload))
req.Header.Set("X-Gitea-Signature", sign(t, payload, "s3cret"))
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
assert.Equal(t, http.StatusOK, rec.Code)
jobs, _ := cs.BatchV1().Jobs("brain").List(context.Background(), metav1.ListOptions{})
assert.Empty(t, jobs.Items, "must not trigger for a push to a non-main branch")
}
func TestHandler_TriggersJobOnValidMainPush(t *testing.T) {
cs := fake.NewSimpleClientset(fakeCronJob("brain", "brain-sync"))
h := newTestHandler(cs)
payload := pushPayload(t, "mathias/brain", "refs/heads/main")
req := httptest.NewRequest(http.MethodPost, "/webhooks/brain-sync", bytes.NewReader(payload))
req.Header.Set("X-Gitea-Signature", sign(t, payload, "s3cret"))
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
assert.Equal(t, http.StatusOK, rec.Code)
jobs, err := cs.BatchV1().Jobs("brain").List(context.Background(), metav1.ListOptions{})
require.NoError(t, err)
assert.Len(t, jobs.Items, 1, "a valid push to main on the watched repo must trigger exactly one job")
}
+1 -1
View File
@@ -13,7 +13,7 @@ import (
"github.com/lestrrat-go/jwx/v2/jwa" "github.com/lestrrat-go/jwx/v2/jwa"
"github.com/lestrrat-go/jwx/v2/jwk" "github.com/lestrrat-go/jwx/v2/jwk"
"github.com/lestrrat-go/jwx/v2/jwt" "github.com/lestrrat-go/jwx/v2/jwt"
"github.com/mathiasbq/supervisor/internal/auth" "git.d-ma.be/mathias/hyperguild/internal/auth"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
+1 -1
View File
@@ -6,7 +6,7 @@ import (
"net/http/httptest" "net/http/httptest"
"testing" "testing"
"github.com/mathiasbq/supervisor/internal/auth" "git.d-ma.be/mathias/hyperguild/internal/auth"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
+1 -1
View File
@@ -7,7 +7,7 @@ import (
"net/http/httptest" "net/http/httptest"
"testing" "testing"
"github.com/mathiasbq/supervisor/internal/brain" "git.d-ma.be/mathias/hyperguild/internal/brain"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
+1 -1
View File
@@ -3,7 +3,7 @@ package config_test
import ( import (
"testing" "testing"
"github.com/mathiasbq/supervisor/internal/config" "git.d-ma.be/mathias/hyperguild/internal/config"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
+1 -1
View File
@@ -5,7 +5,7 @@ import (
"path/filepath" "path/filepath"
"testing" "testing"
"github.com/mathiasbq/supervisor/internal/config" "git.d-ma.be/mathias/hyperguild/internal/config"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
+2
View File
@@ -14,6 +14,7 @@ type RoutingConfig struct {
LiteLLMBaseURL string // LITELLM_BASE_URL, default https://llm-api.d-ma.be LiteLLMBaseURL string // LITELLM_BASE_URL, default https://llm-api.d-ma.be
LiteLLMAPIKey string // LITELLM_API_KEY LiteLLMAPIKey string // LITELLM_API_KEY
BrainURL string // BRAIN_URL, default http://ingestion.supervisor:3300 BrainURL string // BRAIN_URL, default http://ingestion.supervisor:3300
BrainMCPToken string // BRAIN_MCP_TOKEN, bearer for the auth-gated ingestion /mcp (session_log)
FastModel string // HYPERGUILD_FAST_MODEL, default koala/qwen35-9b-fast FastModel string // HYPERGUILD_FAST_MODEL, default koala/qwen35-9b-fast
ThinkingModel string // HYPERGUILD_THINKING_MODEL, default iguana/gemma4-26b ThinkingModel string // HYPERGUILD_THINKING_MODEL, default iguana/gemma4-26b
// RouteLocalFloor and RouteLocalCeil intentionally invert the usual // RouteLocalFloor and RouteLocalCeil intentionally invert the usual
@@ -44,6 +45,7 @@ func LoadRouting() (RoutingConfig, error) {
LiteLLMBaseURL: envOr("LITELLM_BASE_URL", "https://llm-api.d-ma.be"), LiteLLMBaseURL: envOr("LITELLM_BASE_URL", "https://llm-api.d-ma.be"),
LiteLLMAPIKey: os.Getenv("LITELLM_API_KEY"), LiteLLMAPIKey: os.Getenv("LITELLM_API_KEY"),
BrainURL: envOr("BRAIN_URL", "http://ingestion.supervisor:3300"), BrainURL: envOr("BRAIN_URL", "http://ingestion.supervisor:3300"),
BrainMCPToken: os.Getenv("BRAIN_MCP_TOKEN"),
FastModel: envOr("HYPERGUILD_FAST_MODEL", "koala/qwen35-9b-fast"), FastModel: envOr("HYPERGUILD_FAST_MODEL", "koala/qwen35-9b-fast"),
ThinkingModel: envOr("HYPERGUILD_THINKING_MODEL", "iguana/gemma4-26b"), ThinkingModel: envOr("HYPERGUILD_THINKING_MODEL", "iguana/gemma4-26b"),
} }
+1 -1
View File
@@ -3,7 +3,7 @@ package config_test
import ( import (
"testing" "testing"
"github.com/mathiasbq/supervisor/internal/config" "git.d-ma.be/mathias/hyperguild/internal/config"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"testing" "testing"
"time" "time"
iexec "github.com/mathiasbq/supervisor/internal/exec" iexec "git.d-ma.be/mathias/hyperguild/internal/exec"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
+1 -1
View File
@@ -9,7 +9,7 @@ import (
"net/http/httptest" "net/http/httptest"
"testing" "testing"
"github.com/mathiasbq/supervisor/internal/githubclient" "git.d-ma.be/mathias/hyperguild/internal/githubclient"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
+2 -2
View File
@@ -8,8 +8,8 @@ import (
"net/http" "net/http"
"strings" "strings"
"github.com/mathiasbq/supervisor/internal/auth" "git.d-ma.be/mathias/hyperguild/internal/auth"
"github.com/mathiasbq/supervisor/internal/registry" "git.d-ma.be/mathias/hyperguild/internal/registry"
) )
type request struct { type request struct {
+2 -2
View File
@@ -8,8 +8,8 @@ import (
"strings" "strings"
"testing" "testing"
"github.com/mathiasbq/supervisor/internal/mcp" "git.d-ma.be/mathias/hyperguild/internal/mcp"
"github.com/mathiasbq/supervisor/internal/registry" "git.d-ma.be/mathias/hyperguild/internal/registry"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
+1 -1
View File
@@ -9,7 +9,7 @@ import (
"net/http/httptest" "net/http/httptest"
"testing" "testing"
"github.com/mathiasbq/supervisor/internal/mcpclient" "git.d-ma.be/mathias/hyperguild/internal/mcpclient"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
+1 -1
View File
@@ -5,7 +5,7 @@ import (
"encoding/json" "encoding/json"
"testing" "testing"
"github.com/mathiasbq/supervisor/internal/registry" "git.d-ma.be/mathias/hyperguild/internal/registry"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
-21
View File
@@ -1,21 +0,0 @@
package routing
import (
"crypto/sha256"
"encoding/binary"
)
// CanonicalHash returns a deterministic 64-bit hash of (system, user).
// Used to make sample-band routing decisions reproducible: identical input
// strings produce the same hash on every call, independent of process state.
//
// Inputs are joined with a 0x00 byte separator before hashing — distinguishes
// (system="ab", user="cd") from (system="abcd", user="").
func CanonicalHash(system, user string) uint64 {
h := sha256.New()
h.Write([]byte(system))
h.Write([]byte{0})
h.Write([]byte(user))
sum := h.Sum(nil)
return binary.BigEndian.Uint64(sum[:8])
}
-46
View File
@@ -1,46 +0,0 @@
package routing_test
import (
"testing"
"github.com/mathiasbq/supervisor/internal/routing"
"github.com/stretchr/testify/assert"
)
func TestCanonicalHashDeterministic(t *testing.T) {
a := routing.CanonicalHash("system one", "user one")
b := routing.CanonicalHash("system one", "user one")
assert.Equal(t, a, b, "same inputs must produce same hash")
}
func TestCanonicalHashDistinguishesInputs(t *testing.T) {
cases := [][2]string{
{"sys", "user"},
{"sys", "user2"},
{"sys2", "user"},
{"", "system\x00user"}, // separator collision attempt
{"system\x00user", ""},
}
seen := make(map[uint64]bool)
for _, c := range cases {
h := routing.CanonicalHash(c[0], c[1])
assert.False(t, seen[h], "collision on %v", c)
seen[h] = true
}
}
func TestCanonicalHashLowBitDistribution(t *testing.T) {
// Sanity check: across 1000 distinct inputs, low-bit split is roughly even.
zeros, ones := 0, 0
for i := 0; i < 1000; i++ {
h := routing.CanonicalHash("sys", string(rune('a'+(i%26)))+string(rune(i)))
if h&1 == 0 {
zeros++
} else {
ones++
}
}
// Allow ±15% deviation from 500/500. Tighter would be flaky on real data.
assert.InDelta(t, 500, zeros, 150)
assert.InDelta(t, 500, ones, 150)
}

Some files were not shown because too many files have changed in this diff Show More