Compare commits

..
Author SHA1 Message Date
mathiasandClaude Opus 5 db00644cab fix(capture): name what escalated the classification in an I1 refusal (#91)
CI / Lint / Test / Vet (pull_request) Successful in 30s
CI / Mirror to GitHub (pull_request) Skipped
infra#265 reported "capture with a tickets array fails": 20 tickets
errored, 5 errored, the caller fell back to 20 individual issue_create
calls and filed a bug against the array.

The array was never the failing dimension. Every ticket in that batch
targeted a client repo, and the I1 sovereignty gate refuses a
confidential capture through a us-nexus origin at ANY length — reproduced
live today against brain-mcp: length 1 succeeds for an internal repo,
length 20 validates clean, and a single ticket targeting
trucker-ecommerce is refused.

What made it look like an array bug is that the refusal named neither the
offending target nor the fact that a policy said no:

    capture refused by I1 sovereignty gate: effective classification
    confidential through us-nexus origin

With twenty tickets a caller cannot tell which repo tripped it, so
"the array is broken" is a reasonable conclusion from the evidence
available. resolveClassification now also returns the targets carrying
the effective level, and the refusal names them:

    ... through us-nexus origin (required by: trucker-ecommerce)

Only targets that actually reach the effective level are named — at
twenty tickets, listing the innocent ones hides the one that matters —
and a caller who declared confidential themselves is told exactly that,
because "you asked for this" and "a repo you touched required it" call
for different fixes.

Regression tests at length 1 and 20 pin the array path that was never
broken, and a third pins per-item partial success: infra#265 asked "if
ticket 3 of 5 fails, what happened to 1 and 2?" — they land, and the
receipt says so per item.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeAp5LTscnz2W7Ubt4eJ6m
2026-08-18 15:38:12 +02:00
mathiasandClaude Opus 5 125c99bb46 fix(text): truncate on rune boundaries, not byte offsets (#93)
CI / Mirror to GitHub (push) Successful in 4s
CI / Lint / Test / Vet (push) Successful in 31s
Four call sites did `s[:limit] + "…"`. Both halves are byte operations:
len() counts bytes and a slice cuts bytes. On Swedish text — å ä ö are
two bytes, an em dash three — the cut lands mid-character and leaves an
orphaned lead byte, producing invalid UTF-8.

It failed loudly in exactly one place: wiki/index.md is written raw to
disk, where every strict decoder then rejects the whole file. The other
three (brain_context excerpts x2, brain_query excerpts) feed MCP
responses, where encoding/json silently substitutes U+FFFD — so those had
been quietly eating the last character of truncated excerpts, with no
error anywhere and nothing to notice.

textutil.Truncate counts runes and appends the marker only when content
was actually dropped; an ellipsis on complete text is a lie about the
content. The test is a property, not an example: it walks the limit
across every offset of a multi-byte string and asserts utf8.Valid at each
one, so a future off-by-one cannot pass by picking a lucky boundary.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeAp5LTscnz2W7Ubt4eJ6m
2026-08-18 07:13:15 +02:00
mathiasandClaude Opus 5 3e537bd582 feat(brain): surface an aging review queue in brain_context (#82)
CI / Lint / Test / Vet (push) Successful in 30s
CI / Mirror to GitHub (push) Successful in 4s
brain_pending existed the whole time four real items sat unreviewed from
2026-04-23 to 2026-07-15. The tool was never the gap — nothing ever
prompted anyone to call it. The system should be what remembers the queue
exists.

Option 2 from the issue: ride along with brain_context, which sessions
already call, so noticing an aging note costs nobody a decision to go
looking. No scheduled job, no ntfy wiring, nothing new to keep running.

Threshold-gated at 14 days, and silent below it — a notice on every
context call is one people learn to skip, which reproduces the original
failure with extra steps. It reports the whole queue once it speaks: the
threshold decides whether to talk, not what to count.

It never promotes, rejects or reorders. Age is a reason to ask a human,
not a verdict; an old note may be exactly right to promote once someone
looks. A test asserts the queue is untouched and that the payload carries
a hint rather than an action.

An unreadable raw/ dir degrades to silence — brain_context is a hot path
and must not fail because the review queue is broken.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeAp5LTscnz2W7Ubt4eJ6m
2026-08-18 00:37:48 +02:00
mathiasandClaude Opus 5 4e6fc7aeee chore(brain): de-surface brain_index from the model-facing toolset (#78)
CI / Lint / Test / Vet (push) Successful in 29s
CI / Mirror to GitHub (push) Successful in 4s
brain_index is auto-called after every write, and its own description
said so. A verb surfaced to callers that callers should never call is
cognitive load with no decision behind it — ADR-0012 rule 4.

Removed from tools/list only. The dispatch case stays, so maintenance
can still rebuild a Map of Content by hand: de-surfacing hides the verb,
not the operation. A test pins both halves, since removing the descriptor
and removing the capability look identical from the tool list.

Nothing called it directly — grepped the skills, specs and docs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeAp5LTscnz2W7Ubt4eJ6m
2026-08-18 00:31:06 +02:00
mathiasandClaude Opus 5 9a232ad444 fix(ingestion): retry OIDC discovery instead of exiting on cold-boot 502 (#83)
CI / Lint / Test / Vet (push) Successful in 36s
CI / Mirror to GitHub (push) Successful in 4s
On a cluster restart this pod can beat dex to serving, gets a 502 from
ingress, and exits — after connecting postgres, backfilling 794 graph
nodes and ingesting three session batches, all of which the restart
redoes. Nothing was misconfigured; it started first.

The validator now retries with bounded exponential backoff (8 attempts,
1s doubling to 30s) and is built immediately after ctx, before the
watcher and the backfill. That second part is the other half of the
issue: a genuine config error now fails in seconds rather than after all
that work.

Bounded on purpose. Retrying forever would convert a wrong issuer URL
into a process that looks alive and never serves, which is harder to
diagnose than a crash loop.

startupretry.Do is generic and tested directly: transient-then-success,
budget exhaustion surfacing the last error via errors.Is, prompt return
on context cancellation, growing-and-capped delays, and a zero-value
policy still making one attempt rather than silently succeeding.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeAp5LTscnz2W7Ubt4eJ6m
2026-08-18 00:21:56 +02:00
mathiasandClaude Opus 5 3da0eca9b4 fix(ingestion): decide quoting with YAML itself, not a colon heuristic (#90)
CI / Lint / Test / Vet (push) Successful in 29s
CI / Mirror to GitHub (push) Successful in 4s
`title: "Foo" — bar` opens with a quote and is not quoted; the previous
prefix check read it as already-quoted and left two notes broken. Probing
the value as the right-hand side of a mapping answers the real question —
does this load as written — and leaves anything that already parses alone,
whatever it parses into.

Repairs the last two quotable files in mathias/brain. Only wiki/index.md
remains unparseable, for an unrelated reason: a truncated UTF-8 sequence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeAp5LTscnz2W7Ubt4eJ6m
2026-08-18 00:10:11 +02:00
mathiasandClaude Opus 5 7d1213451c chore: re-sync context adapters
CI / Lint / Test / Vet (push) Successful in 29s
CI / Mirror to GitHub (push) Successful in 3s
Root .context/AGENT.md gained the CAD sprint patterns section; the derived
adapters had not been regenerated since. Pure `task context:sync` output.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeAp5LTscnz2W7Ubt4eJ6m
2026-08-18 00:08:07 +02:00
mathiasandClaude Opus 5 7c9293f4ed feat(ingestion): lint and repair unparseable note frontmatter (#90)
Hand-authored and promoted notes skip pipeline.BuildPages, so nothing
quotes their scalars. A title carrying an unescaped ": " yields a block
yaml.Unmarshal rejects, and every consumer that reads frontmatter drops
the note silently — 18 files repo-wide today, up from the 8 in #90.

frontmatter.Lint reports it; frontmatter.Fix quotes only what quoting can
rescue and refuses to return a document that still fails to parse, so an
unfixable file comes back byte-identical rather than half-repaired. Scan
walks a tree and reports an unreadable file as a finding instead of
aborting, so one poisoned note cannot hide the rest.

Sequence items are deliberately left alone: "- foo: bar" is valid YAML
that silently becomes a map, which is a type corruption rather than a
parse failure, and repairing it would rewrite notes that currently load.

    task lint:frontmatter          # report
    task lint:frontmatter FIX=1    # repair in place

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeAp5LTscnz2W7Ubt4eJ6m
2026-08-18 00:07:27 +02:00
mathiasandClaude Sonnet 5 6e0155a2ab feat(security): add gitleaks pre-commit scan, wire into task check (infra#39)
CI / Lint / Test / Vet (push) Successful in 1m41s
CI / Mirror to GitHub (push) Successful in 5s
Ported from mathias/infra's .gitleaks.toml (2026-08-04) -- same
homelab bearer/MCP-token rules, plus a BRAIN_PG_DSN pattern for the
hyperguild#20 leak class. Triaged 10 initial findings: 9 were
claudewatcher's own scrubber test fixtures (deliberately fake), 1 was
gitleaks matching the literal placeholder word "REDACTED" in a plan
doc. Both allowlisted with rationale, clean scan confirmed before
wiring into `task check`.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016FQX4RwxyxuQY4yf4XaACL
2026-08-04 08:32:42 +02:00
mathiasandClaude Sonnet 5 1cea2c9f78 feat(capture): remove summary->ai-sessions write path
CI / Lint / Test / Vet (push) Successful in 29s
CI / Mirror to GitHub (push) Successful in 4s
ai-sessions#13: capture's `summary` field wrote a frontmatter shape
(title/harness/fidelity/captured_at/repos_touched) that ai-sessions'
own extract/audit pipeline can't parse -- silently invisible to that
repo's own audit tooling, and bypassing its redaction + Stage2
completeness gates by construction. Only 5 files ever landed this way
over 3 weeks; the summary capability's whole value (speed) fights
ai-sessions' whole value (redacted, audited, complete), so kill it
rather than build a second parse branch.

capture now persists insights -> brain and action items -> Gitea
tickets only. Removes Summary/SummaryResult/SummaryWriter and all
wiring (service, REST body, MCP tool schema); close-session updated to
stop assembling a summary payload. specs/capture-*.md kept as historical
record with a superseded note -- the feature shipped and is documented,
just no longer current behaviour.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WPdSHbp9Utb2hPFm9wDG59
2026-07-27 23:17:56 +02:00
mathiasandClaude Sonnet 5 9dcd60931a chore: retrigger CI after gitea-runner RBAC fix (round 6 of the CI/CD outage chain)
CI / Lint / Test / Vet (push) Successful in 30s
CI / Mirror to GitHub (push) Has been skipped
cd.yml's build+push+ssh chain fully worked last round (infra 707a3c0
landed via CD Bot). Final gap was RBAC: kubectl (new in baked-v6) had
no ServiceAccount permissions. Fixed in infra@949b333 (dedicated SA +
scoped Role/RoleBinding for flux-system + supervisor, verified live
via kubectl auth can-i before this retrigger). No code change here.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 22:29:57 +02:00
mathiasandClaude Sonnet 5 1fac90ed2a fix(cd): pass git.d-ma.be SSH override as -o flags, not an appended ~/.ssh/config line
CI / Lint / Test / Vet (push) Successful in 29s
CI / Mirror to GitHub (push) Successful in 4s
cd.yml's own $HOME (/data) is a PVC that persists across job runs on
this runner. The previous fix (cb9c251) appended the corrected
HostName to ~/.ssh/config, but a stale 127.0.0.1:30022 entry from an
earlier failed run was already there — ssh_config is first-match-wins,
so the stale entry silently shadowed the fix and the very next run
failed identically. Cleaned the persisted stale entry on the runner
directly; this commit removes the append entirely so the step no
longer depends on file state surviving (or not surviving) between
runs — CLI -o overrides always win. Verified end-to-end against the
real infra repo from inside the runner pod (git ls-remote, correctly
reached gitea and got a clean auth rejection, not a connection error).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 22:16:00 +02:00
mathiasandClaude Sonnet 5 cb9c2513a4 fix(cd): use in-cluster DNS for git.d-ma.be SSH, not 127.0.0.1:30022
CI / Lint / Test / Vet (push) Successful in 28s
CI / Mirror to GitHub (push) Successful in 4s
act_runner moved from koala's bare host network into a k3s pod
(2026-07-26); loopback inside that pod's own network namespace never
reaches the host, so the git.d-ma.be SSH override (used to push the
infra repo update) failed with "Connection refused". Verified
TCP+SSH-handshake reachability to gitea-ssh-nodeport.gitea.svc.cluster.local:22
from inside the actual runner pod before committing this.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 22:09:52 +02:00
mathiasandClaude Sonnet 5 3617a6c386 chore: retrigger CI after gitea-runner baked-v6 (ssh/skopeo/buildctl/kubectl)
CI / Lint / Test / Vet (push) Successful in 28s
CI / Mirror to GitHub (push) Successful in 4s
check job passed clean on baked-v5; mirror job then exposed missing
ssh/ssh-keyscan. Fixed in infra@707a3c0, which also preemptively adds
skopeo/buildctl/kubectl for cd.yml's deploy job. No code change here —
retriggering for the full CI->CD end-to-end proof.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 22:03:07 +02:00
mathiasandClaude Sonnet 5 1938170131 chore: retrigger CI after gitea-runner gcc/libc6-dev fix (baked-v5)
CI / Lint / Test / Vet (push) Successful in 1m12s
CI / Mirror to GitHub (push) Failing after 1s
Runner was missing gcc/libc6-dev too (-race requires cgo). Added in
infra@0f88aea. No code change here — retriggering CI.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 21:55:20 +02:00
mathiasandClaude Sonnet 5 b34717e6b8 chore: retrigger CI after gitea-runner image fix (infra#task-baked-v4)
CI / Lint / Test / Vet (push) Failing after 34s
CI / Mirror to GitHub (push) Has been skipped
Runner was missing task (taskfile.dev), added in infra@686c90f. No
code change here — retriggering CI to confirm the fix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 21:49:33 +02:00
mathiasandClaude Sonnet 5 d8d7e9a307 chore(context): re-sync derived adapters (Python fallback stack, error-handling note)
CI / Lint / Test / Vet (push) Failing after 1s
CI / Mirror to GitHub (push) Has been skipped
Canonical ~/dev/.context/AGENT.md drifted (Python fallback stack table
row, Python error-handling convention) without a hyperguild commit
re-running task context:sync since — task check's drift gate has been
failing CI on every push since the sprint-1 commits landed, blocking
CD entirely (unrelated to those commits' own content: lint/test/vet
all pass clean, including -race).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 15:18:38 +02:00
mathiasandClaude Sonnet 5 f0055483a3 fix(watcher): never re-ingest AutoTunnel's own tunnel-candidates log (#88)
CI / Lint / Test / Vet (push) Failing after 1s
CI / Mirror to GitHub (push) Has been skipped
Root cause was neither of the two mechanisms the issue named
(CanonicalizeLinks over-canonicalizing, or a schema/prompt bias) —
CanonicalizeLinks correctly resolved [[Mission]] because
wiki/concepts/mission.md genuinely exists; the extraction prompt has
no "mission" bias either. The real bug: watcher.processDir walks
brain/raw/ and feeds every .md/.txt/.pdf file to pipeline.Run as
source material to extract, with no exclusion for
tunnel-candidates-*.md — AutoTunnel's own fuzzy-match human-review
queue (logFuzzyCandidates writes it into that same raw/ directory).
The watcher's next poll re-ingested the raw candidate log itself, and
the LLM naturally surfaced "mission" as a wikilink because the log
literally lists `(term: "mission")` entries in its content — a
generic-word title match DetectTunnels correctly flagged as fuzzy
(not auto-tunneled) but which still leaked downstream once treated as
real source material.

api.ListPending already excludes tunnel-candidates-* when listing
raw/ for promotion; processDir gets the same exclusion.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 14:37:50 +02:00
mathiasandClaude Sonnet 5 6d014c1d0f fix(ingest): CanonicalizeLinks strips wing:/wiki/ prefix and repairs path-style links (#87)
CI / Mirror to GitHub (push) Has been skipped
CI / Lint / Test / Vet (push) Failing after 2s
plainLinkRE matches any [[...]] without a pipe, including path-prefixed
forms like [[wing:homelab/failures/foo]] or [[wiki/agentsquad/decisions/bar]]
that the LLM extraction step occasionally emits. Title-lookup against
titleToSlug always fails for these (they're paths, not titles), so they
were silently left broken.

Before falling to the unknown-wikilink warning, strip a known wing:/wiki/
root prefix and emit the clean wing/hall/slug path directly — matching
the brain-graph path-style link convention.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 14:32:31 +02:00
mathiasandClaude Sonnet 5 1001acfb44 fix(ingest): merge existing frontmatter in writeHallNote instead of stacking (#86)
CI / Lint / Test / Vet (push) Failing after 1s
CI / Mirror to GitHub (push) Has been skipped
opts.Content can already carry its own "---\n...\n---" block (e.g. from
an upstream extraction/promotion step). writeHallNote used to prepend a
second block unconditionally, making a standard frontmatter parser blind
to everything the first block declared (title, tags, ...).

splitFrontmatter now pulls the existing block's fields out first;
wing/hall/created_at are always fresh-injected, and type/domain/
source_type prefer the note's own existing value over the opts
fallback. Remaining existing fields are carried through verbatim into
the single merged block.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-26 23:45:36 +02:00
mathiasandClaude Sonnet 5 6ad275b505 fix(ingest): thread source/author/published frontmatter through extraction (#85)
CI / Lint / Test / Vet (push) Failing after 1s
CI / Mirror to GitHub (push) Has been skipped
pipeline.Run parses source/author/published from the raw content's own
frontmatter and applies them deterministically to source-type RawPages
after LLM extraction — never relies on the LLM to copy them through.
buildFrontmatter now emits all three (source pages only) when present.

Backfill of the 46 already-affected wiki/sources/*.md notes is a
separate concern per the issue, not done here.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-26 23:42:33 +02:00
mathiasandClaude Sonnet 5 b600cc986c chore(module): rename module github.com/mathiasbq/supervisor -> git.d-ma.be/mathias/hyperguild (#42)
CI / Lint / Test / Vet (push) Failing after 5s
CI / Mirror to GitHub (push) Has been skipped
Aligns module path with canonical Gitea source (infra ADR-0004) and
repo name. Binary only, no downstream consumers.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-26 23:37:11 +02:00
mathiasandClaude Opus 4.8 9bdab1c48c fix(ingest): parse docmark's SSE-framed response, not bare JSON
CI / Lint / Test / Vet (push) Successful in 16s
CI / Mirror to GitHub (push) Has been skipped
Found via a real live-cluster end-to-end test (POST /ingest-path with a real
.docx against the deployed ingestion + docmark): 'decode docmark response:
invalid character e looking for beginning of value'. docmark's Streamable-HTTP
transport frames every response as SSE (event: message\r\ndata: {...}\r\n\r\n,
Content-Type: text/event-stream) -- stateless_http=True removes the need for
an initialize handshake/session ID, it does NOT change the wire framing to
bare JSON. My original client + its own test mock both assumed bare JSON,
so the unit tests passed while the real call failed.

sseDataPayload() extracts the data: line before JSON-unmarshaling; falls back
to the raw body if no SSE framing is present (forward-compatible). Test mock
(docmark_test.go) now emits the SAME framing the live server actually sends,
confirmed by directly probing docmark's live response before writing the fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 08:53:24 +02:00
mathiasandClaude Opus 4.8 6520c2fc4b feat(ingest): route DOCX/XLSX/PPTX/images through docmark (ADR-0013)
CI / Lint / Test / Vet (push) Successful in 16s
CI / Mirror to GitHub (push) Successful in 4s
Wires the sovereign docmark MCP server (git.d-ma.be/mathias/docmark) as the
converter for formats ingestion previously couldn't handle at all -- DOCX,
XLSX, PPTX, PNG/JPG. A single self-contained tools/call POST (docmark runs
stateless_http, no session handshake needed) does the conversion.

- internal/extract/docmark.go: extractViaDocmark(path) -- reads DOCMARK_URL +
  DOCMARK_BEARER_TOKEN from env, base64-encodes the file, calls docmark's
  convert_to_markdown tool, surfaces tool/HTTP errors with context.
- internal/extract/extract.go: routes .docx/.xlsx/.pptx/.png/.jpg/.jpeg to it.
- internal/api/handler.go: isSupportedExtension() replaces the static
  supportedExtensions map -- the new extensions are gated on DOCMARK_URL being
  set, checked per-request. When DOCMARK_URL is unset (e.g. this repo's
  current deployed state), behavior is byte-for-byte unchanged from today:
  same 400/silent-skip as before. Zero risk until the env var is actually
  wired in infra.

TDD throughout: docmark.go tested via httptest mock (success, not-configured,
tool-error, HTTP-error, all 6 new extensions route correctly); handler-level
test proves the DOCMARK_URL gate (docx 400 when unset, 200 when a working
mock is configured). Full task check green (fmt/vet/govulncheck/all tests).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 08:46:03 +02:00
mathiasandClaude Opus 4.8 fcbd1072b6 fix(lint): check fmt.Fprintf errors in webhook handler (infra#190)
CI / Lint / Test / Vet (push) Successful in 16s
CI / Mirror to GitHub (push) Has been skipped
errcheck flagged two unchecked fmt.Fprintf return values in
internal/webhook/webhook.go, failing CI (run 310/311) for the whole
'trigger brain-sync on Gitea push' feature -- so no new ingestion image was
ever built, and the deployed image predates this feature entirely even
though infra's manifest (secret, RBAC, env wiring) was already live.

Both writes are best-effort informational text after WriteHeader has already
committed the status code -- a failed write here only happens on client
disconnect and nothing depends on it succeeding, so explicitly discard
(_, _ =) rather than log-and-continue noise.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 08:29:39 +02:00
mathias 3b7706b358 chore(context): re-sync derived adapters (skills-source paragraph) 2026-07-09 08:28:54 +02:00
mathias 394a227877 feat(webhook): trigger brain-sync on Gitea push instead of 15-min poll
CI / Mirror to GitHub (push) Has been skipped
CI / Lint / Test / Vet (push) Failing after 8s
Adds POST /webhooks/brain-sync: verifies Gitea's HMAC-SHA256 signature,
checks the push is to mathias/brain main, then creates a one-off Job from
the existing brain-sync CronJob's template (same script the 15-min poll
already runs, just triggered on-demand). Off by default -- opt in via
GITEA_WEBHOOK_SECRET, since it needs Job-create RBAC in the "brain"
namespace a fresh deploy won't have.

10 new tests (internal/webhook), including a fake-clientset reactor to
simulate server-side GenerateName expansion, which the plain fake tracker
doesn't do on its own.

Needs (follow-up, infra repo): RBAC granting ingestion's ServiceAccount
get on cronjobs/brain-sync + create on jobs in the brain namespace, the
GITEA_WEBHOOK_SECRET env, and the actual Gitea webhook registration.
2026-07-07 22:00:56 +02:00
mathias 0f84ab5eda fix(ingest): route raw claude-session dumps to non-indexed archive, not wiki
CI / Lint / Test / Vet (push) Successful in 11s
CI / Mirror to GitHub (push) Has been skipped
claudeSink wrote every raw transcript into brain/wiki/claude-sessions/facts/
(BM25-indexed), duplicating what ai-sessions' curated summaries already
cover and out-ranking them in search (177,818 vs 45,335 BM25 weight on the
same query, per ai-sessions#10 investigation). Raw dumps now land in
brain/archive/claude-sessions/<host>/ — kept for deep lookups, never
indexed, never deleted.

Refs: ai-sessions#10
2026-07-07 11:36:01 +02:00
mathias 5b57843346 fix(lint): revert redundant Write conversion (staticcheck S1016)
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 3s
writeRequest and WriteNoteOptions have matching fields again now that
both carry SourceType, so the explicit field-by-field literal added
in ee1204d is flagged as a redundant conversion. Back to the plain
type conversion. Caught by task check (golangci-lint), not run
locally before the previous push -- plain go test passed, task check
(which adds -race and golangci-lint) didn't.
2026-07-04 14:13:16 +02:00
mathias ee1204d76b fix(api): default hall=facts source_type to internal (brain-gardener#7)
CI / Lint / Test / Vet (push) Failing after 5s
CI / Mirror to GitHub (push) Has been skipped
The unsourced-check fix (source_type: internal) only covered
claudewatcher's writes. 41 residual findings on the audit's next run
were manually-authored hall=facts entries written via brain_write/
capture -- also first-party observations, just a different write path,
with no way to mark them internal short of hand-editing every entry.

writeHallNote now defaults source_type to internal whenever hall ==
"facts" and the caller didn't set it explicitly. An explicit
source_type: external (now threaded through the /write HTTP endpoint)
opts a genuinely citation-needing entry back out.
2026-07-04 14:09:20 +02:00
mathias 6d58336ce2 fix(claudewatcher): tag session-dump notes source_type: internal
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 3s
brain-gardener#5: the audit's unsourced check was flagging 59 raw
claudewatcher session dumps as high-severity hallucination risk,
because it can't distinguish a first-party observation (a session
transcript) from an external claim needing citation.

Adds WriteNoteOptions.SourceType, emitted as source_type: <value> in
the wing/hall frontmatter route. claudeSink (the claudewatcher sink)
now always sets source_type: internal on the notes it writes.
2026-07-04 13:53:08 +02:00
mathias 0785f14220 Merge pull request 'Consolidate to single harness: icebox cmd/routing, keep cmd/hyperguild + ingestion (#75)' (#76) from feat/consolidate-single-harness into main
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 4s
2026-07-01 16:27:06 +00:00
78 changed files with 3057 additions and 330 deletions
+16 -3
View File
@@ -78,6 +78,14 @@ These rules apply to every task across every project, regardless of harness.
- Surface the brain entries and issue numbers in the closing summary so the
trail is auditable.
## CAD sprint patterns
Two CAD sprint archetypes (defs: brain `wiki/homelab/decisions/continuous-agentic-development-cad-concept-2026-06-16`):
- **Inception Sprint** — birth a new repo. Oath-gated close; tag only once every clause holds. Brain: `.../inception-sprint-and-oath`.
- **Sweep Sprint** — clear an existing backlog. Parallel read-only research per issue → sequential one-issue-per-commit execution. Brain: `.../sweep-sprint-and-oath`.
Shared: decisions that are yours, not the agent's, surface as `DECISION NEEDED` (question/options/recommendation/blocks) — ask live if you're present, else comment and wait, never act past it. Live-breaking regressions get `git revert`, never forward-fixed; host-network work skips even read-only research (solo, synchronous, you present). Rule 6's closing ritual only fires once every Oath clause genuinely holds — blocked = descoped-and-tracked, never marked done.
## Default stack
| Layer | Default | Fallback | Last resort |
@@ -88,7 +96,7 @@ These rules apply to every task across every project, regardless of harness.
| Containers | Docker Compose (dev), k3s (prod) | — | — |
| DB | PostgreSQL + sqlc | SQLite | — |
| Search | pgvector (vector), BM25 | Qdrant (when >1M vectors or hybrid retrieval) | — |
| Logging | slog (structured) | — | — |
| Logging | slog (structured) | stdlib `logging` w/ structured `extra=` (or structlog) | — |
| Testing | Table-driven, testify | — | — |
| Agents (Go) | google.golang.org/adk + pkg/litellm adapter | — | — |
@@ -97,7 +105,12 @@ Exploratory: Rust, Zig — I'll tell you when I want these.
## Code conventions
- **Go style**: golines, gofumpt, golangci-lint
- **Errors**: `fmt.Errorf("operation: %w", err)` — never naked, never log-and-return
- **Python style** (fallback language): ruff (format+lint, one tool), mypy --strict (non-negotiable,
matches Go's static typing discipline), pytest + pytest-cov (table-driven via
`@pytest.mark.parametrize`), uv (venv+deps+lock, one tool), pydantic-settings (typed env-var config
— same principle as Go's typed structs), src-layout + `pyproject.toml` only (no `setup.py`)
- **Errors**: `fmt.Errorf("operation: %w", err)` — never naked, never log-and-return.
Python: `raise X from e` (exception chaining, same principle) — never bare `except`, never silent `pass`
- **Naming**: stdlib conventions, no stuttering
- **Architecture**: prefer stdlib over frameworks, constructor injection, env-var config parsed into typed structs
- **Git**: conventional commits (`feat:`, `fix:`, `chore:`), commit directly to main,
@@ -268,7 +281,7 @@ unconditionally on every host, every harness.
## Engineering Skills
Shared engineering skills are available in `~/dev/.skills/`. Load at task start — not "on demand" but on schedule, before writing code. See `~/dev/.skills/SKILLS_INDEX.md` for the full list.
Shared engineering skills live in the **`mathias/skills`** repo (`git.d-ma.be/mathias/skills`). Clone it to `~/dev/skills/` and run `SKILLS_CHECKOUT_DIR="$PWD" bash install.sh` there to wire every skill into your harnesses (Claude Code, Crush, Antigravity, Mistral Vibe) as native, on-demand skills. (Use `install.sh`, not `task install` — the latter is currently broken, skills#7.) Load at task start — not "on demand" but on schedule, before writing code. Browse `~/dev/skills/SKILLS_INDEX.md` for the full list.
**Skill trigger table — load before starting, not after getting stuck:**
+17 -3
View File
@@ -48,9 +48,23 @@ jobs:
mkdir -p ~/.ssh
echo "${{ secrets.INFRA_DEPLOY_KEY }}" > ~/.ssh/infra_deploy_key
chmod 600 ~/.ssh/infra_deploy_key
printf 'Host git.d-ma.be\n HostName 127.0.0.1\n Port 30022\n StrictHostKeyChecking no\n' >> ~/.ssh/config
GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes" \
# In-cluster DNS to gitea's SSH NodePort service, not 127.0.0.1:30022
# (that only worked when act_runner ran on koala's bare host network;
# from inside the containerized runner's own pod netns, loopback
# never reaches the host — "Connection refused", found 2026-07-27).
#
# Pass as -o overrides on the ssh invocation itself, NOT appended to
# ~/.ssh/config: $HOME (/data) is a PVC that persists across job
# runs on this runner (same "workspace not ephemeral" class as
# brain: act-runner-host-executor-tmp-persists), so an appended
# line here would pile up duplicate `Host git.d-ma.be` blocks
# across every run — ssh_config is first-match-wins, so a stale
# entry from an earlier failed run would silently shadow this
# fix forever (exactly what happened once already: this fix's
# own first attempt got appended AFTER an already-stale entry
# and lost). CLI -o options always win regardless of file state,
# so this step is safe to re-run any number of times.
GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes -o HostName=gitea-ssh-nodeport.gitea.svc.cluster.local -o Port=22 -o StrictHostKeyChecking=no" \
git clone "${INFRA_REPO}" /tmp/infra-update
cd /tmp/infra-update
+63
View File
@@ -0,0 +1,63 @@
# gitleaks config for the hyperguild repo (infra#39 — leak prevention pass,
# Phase 3 checklist item: "gitleaks pre-commit hook in infra AND hyperguild").
#
# Ported from mathias/infra's .gitleaks.toml (2026-08-04) — same homelab
# token-shape rules, minus the SOPS/searxng allowlists infra needed (this
# repo doesn't use SOPS).
title = "hyperguild gitleaks config"
[extend]
useDefault = true
# --- Homelab-specific rules -------------------------------------------------
[[rules]]
id = "homelab-static-bearer"
description = "Homelab MCP/LLM static bearer or API key assigned a long literal value"
regex = '''(?i)\b(DMABE_[A-Z0-9_]+|[A-Z0-9_]*MCP_TOKEN|ROUTING_MCP_TOKEN|INFRA_MCP_TOKEN|BRAIN_MCP_TOKEN|GITEA_MCP_TOKEN|LITELLM_MASTER_KEY|LITELLM_SALT_KEY|DMABE_LLMAPI_KEY|BRAIN_PG_DSN)\s*[:=]\s*['"]?([A-Za-z0-9/_+.\-]{16,})['"]?'''
keywords = ["dmabe_", "mcp_token", "litellm_master_key", "litellm_salt_key", "llmapi_key", "brain_pg_dsn"]
[[rules.allowlists]]
description = "Env indirection is not a literal secret"
regexes = [
'''os\.environ''',
'''valueFrom''',
'''secretKeyRef''',
'''\$\{?[A-Za-z_][A-Za-z0-9_]*\}?''',
'''REDACTED''',
'''<[A-Z_]+>''',
]
[[rules]]
id = "homelab-authorization-bearer"
description = "Hardcoded Authorization: Bearer header"
regex = '''(?i)authorization['"]?\s*[:=]\s*['"]?bearer\s+([A-Za-z0-9/_+.\-=]{16,})'''
keywords = ["authorization", "bearer"]
[[rules.allowlists]]
description = "Env indirection is not a literal secret"
regexes = [
'''\$\{?[A-Za-z_][A-Za-z0-9_]*\}?''',
'''os\.environ''',
'''REDACTED''',
'''<[A-Z_]+>''',
]
# --- Global allowlist: claudewatcher's own scrubber test fixtures ------------
# ingestion/internal/claudewatcher/{scrubber,watcher}_test.go deliberately
# contain fake secret-shaped literals to test that the scrubber detects and
# redacts them. Verified 2026-08-04: all 9 findings here are test fixtures
# (github-pat, jwt, generic-api-key, homelab-authorization-bearer rules) plus
# 1 doc finding that was gitleaks matching the literal placeholder word
# "REDACTED" in a plan doc — not a real secret in either case.
[[allowlists]]
description = "claudewatcher scrubber test fixtures — deliberately fake secrets"
paths = [
'''ingestion/internal/claudewatcher/scrubber_test\.go$''',
'''ingestion/internal/claudewatcher/watcher_test\.go$''',
]
[[allowlists]]
description = "Literal placeholder word REDACTED matched as if it were a token (verified 2026-08-04: extracted Secret == 'REDACTED' exactly, gitleaks' curl-auth-header rule matched the placeholder text itself, not a real credential)"
condition = "AND"
paths = ['''docs/superpowers/plans/2026-04-22-phase4-attempt-wiring\.md$''']
regexes = ['''REDACTED''']
+16 -3
View File
@@ -73,6 +73,14 @@ These rules apply to every task across every project, regardless of harness.
- Surface the brain entries and issue numbers in the closing summary so the
trail is auditable.
## CAD sprint patterns
Two CAD sprint archetypes (defs: brain `wiki/homelab/decisions/continuous-agentic-development-cad-concept-2026-06-16`):
- **Inception Sprint** — birth a new repo. Oath-gated close; tag only once every clause holds. Brain: `.../inception-sprint-and-oath`.
- **Sweep Sprint** — clear an existing backlog. Parallel read-only research per issue → sequential one-issue-per-commit execution. Brain: `.../sweep-sprint-and-oath`.
Shared: decisions that are yours, not the agent's, surface as `DECISION NEEDED` (question/options/recommendation/blocks) — ask live if you're present, else comment and wait, never act past it. Live-breaking regressions get `git revert`, never forward-fixed; host-network work skips even read-only research (solo, synchronous, you present). Rule 6's closing ritual only fires once every Oath clause genuinely holds — blocked = descoped-and-tracked, never marked done.
## Default stack
| Layer | Default | Fallback | Last resort |
@@ -83,7 +91,7 @@ These rules apply to every task across every project, regardless of harness.
| Containers | Docker Compose (dev), k3s (prod) | — | — |
| DB | PostgreSQL + sqlc | SQLite | — |
| Search | pgvector (vector), BM25 | Qdrant (when >1M vectors or hybrid retrieval) | — |
| Logging | slog (structured) | — | — |
| Logging | slog (structured) | stdlib `logging` w/ structured `extra=` (or structlog) | — |
| Testing | Table-driven, testify | — | — |
| Agents (Go) | google.golang.org/adk + pkg/litellm adapter | — | — |
@@ -92,7 +100,12 @@ Exploratory: Rust, Zig — I'll tell you when I want these.
## Code conventions
- **Go style**: golines, gofumpt, golangci-lint
- **Errors**: `fmt.Errorf("operation: %w", err)` — never naked, never log-and-return
- **Python style** (fallback language): ruff (format+lint, one tool), mypy --strict (non-negotiable,
matches Go's static typing discipline), pytest + pytest-cov (table-driven via
`@pytest.mark.parametrize`), uv (venv+deps+lock, one tool), pydantic-settings (typed env-var config
— same principle as Go's typed structs), src-layout + `pyproject.toml` only (no `setup.py`)
- **Errors**: `fmt.Errorf("operation: %w", err)` — never naked, never log-and-return.
Python: `raise X from e` (exception chaining, same principle) — never bare `except`, never silent `pass`
- **Naming**: stdlib conventions, no stuttering
- **Architecture**: prefer stdlib over frameworks, constructor injection, env-var config parsed into typed structs
- **Git**: conventional commits (`feat:`, `fix:`, `chore:`), commit directly to main,
@@ -263,7 +276,7 @@ unconditionally on every host, every harness.
## Engineering Skills
Shared engineering skills are available in `~/dev/.skills/`. Load at task start — not "on demand" but on schedule, before writing code. See `~/dev/.skills/SKILLS_INDEX.md` for the full list.
Shared engineering skills live in the **`mathias/skills`** repo (`git.d-ma.be/mathias/skills`). Clone it to `~/dev/skills/` and run `SKILLS_CHECKOUT_DIR="$PWD" bash install.sh` there to wire every skill into your harnesses (Claude Code, Crush, Antigravity, Mistral Vibe) as native, on-demand skills. (Use `install.sh`, not `task install` — the latter is currently broken, skills#7.) Load at task start — not "on demand" but on schedule, before writing code. Browse `~/dev/skills/SKILLS_INDEX.md` for the full list.
**Skill trigger table — load before starting, not after getting stuck:**
+36
View File
@@ -101,12 +101,48 @@ tasks:
- task: lint
- task: test
- task: vet
- task: security:gitleaks
# ── Security ─────────────────────────────────────────────
security:gitleaks:
desc: Scan the working tree for secrets (gitleaks, fail-closed; skipped if gitleaks absent)
dir: '{{.ROOT_DIR}}'
cmds:
- |
GL="$(command -v gitleaks || true)"
[ -z "$GL" ] && [ -x "$(go env GOPATH 2>/dev/null)/bin/gitleaks" ] && GL="$(go env GOPATH)/bin/gitleaks"
if [ -z "$GL" ]; then
echo "⚠ gitleaks not installed — skipping secret scan (CI enforces it)."
echo " Install: go install github.com/zricethezav/gitleaks/v8@latest"
exit 0
fi
"$GL" detect --no-git --redact --config .gitleaks.toml --source .
security:gitleaks:history:
desc: "One-time FULL-HISTORY secret audit (infra#39 rotation pass; not a per-push gate)"
dir: '{{.ROOT_DIR}}'
cmds:
- |
GL="$(command -v gitleaks || true)"
[ -z "$GL" ] && [ -x "$(go env GOPATH 2>/dev/null)/bin/gitleaks" ] && GL="$(go env GOPATH)/bin/gitleaks"
if [ -z "$GL" ]; then
echo "gitleaks not installed: go install github.com/zricethezav/gitleaks/v8@latest" >&2
exit 2
fi
echo "Scanning FULL git history (redacted). Known historical leaks are expected"
echo "until the infra#39 rotation pass completes — triage against the rotation list."
"$GL" detect --redact --config .gitleaks.toml
lint:
cmds:
- golangci-lint run ./...
- cd ingestion && golangci-lint run ./...
lint:frontmatter:
desc: Report brain notes whose YAML frontmatter does not parse (BRAIN=path, FIX=1 repairs in place)
cmds:
- cd ingestion && go run ./cmd/frontmatterlint {{if .FIX}}-fix{{end}} {{.BRAIN | default "../../brain"}}
test:
cmds:
- go test -race -count=1 ./...
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"io"
"os"
"github.com/mathiasbq/supervisor/internal/tier"
"git.d-ma.be/mathias/hyperguild/internal/tier"
)
const defaultAnthropicProbe = "https://api.anthropic.com"
+1 -1
View File
@@ -1,4 +1,4 @@
module github.com/mathiasbq/supervisor
module git.d-ma.be/mathias/hyperguild
go 1.26.1
+53
View File
@@ -0,0 +1,53 @@
// Command frontmatterlint reports markdown notes whose YAML frontmatter does
// not parse, and optionally repairs the ones that only need quoting.
//
// go run ./cmd/frontmatterlint ../../brain # report
// go run ./cmd/frontmatterlint -fix ../../brain # repair in place
//
// Exit codes: 0 clean (or everything repaired), 1 findings remain, 2 the scan
// itself failed.
package main
import (
"flag"
"fmt"
"os"
"github.com/mathiasbq/hyperguild/ingestion/internal/frontmatter"
)
func main() {
fix := flag.Bool("fix", false, "repair files that quoting can rescue, in place")
flag.Parse()
root := "."
if flag.NArg() > 0 {
root = flag.Arg(0)
}
findings, err := frontmatter.Scan(root, *fix)
if err != nil {
fmt.Fprintln(os.Stderr, "frontmatterlint:", err)
os.Exit(2)
}
var unresolved int
for _, f := range findings {
if f.Repaired {
fmt.Printf("repaired %s\n", f.Path)
continue
}
unresolved++
fmt.Printf("BROKEN %s: %v\n", f.Path, f.Err)
}
if unresolved > 0 {
fmt.Fprintf(os.Stderr, "\n%d file(s) with unparseable frontmatter", unresolved)
if !*fix {
fmt.Fprint(os.Stderr, " — re-run with -fix")
}
fmt.Fprintln(os.Stderr)
os.Exit(1)
}
fmt.Printf("✓ frontmatter: %d file(s) repaired, none broken\n", len(findings))
}
+89 -26
View File
@@ -32,14 +32,36 @@ import (
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
"github.com/mathiasbq/hyperguild/ingestion/internal/reranker"
"github.com/mathiasbq/hyperguild/ingestion/internal/search"
"github.com/mathiasbq/hyperguild/ingestion/internal/startupretry"
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
"github.com/mathiasbq/hyperguild/ingestion/internal/watcher"
"github.com/mathiasbq/hyperguild/ingestion/internal/webhook"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/rest"
"k8s.io/client-go/tools/clientcmd"
)
// claudeSink converts each claudewatcher.Batch into one wiki note under
// brain/wiki/claude-sessions/facts/. v1 emits one note per session
// keyed by host + session id; classifier-driven hall routing is a
// follow-up (hyperguild#27 v2).
// kubeClient builds an in-cluster Kubernetes client (falls back to
// $KUBECONFIG for local dev/testing against a real cluster).
func kubeClient() (kubernetes.Interface, error) {
if cfg, err := rest.InClusterConfig(); err == nil {
return kubernetes.NewForConfig(cfg)
}
rules := clientcmd.NewDefaultClientConfigLoadingRules()
cc := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(rules, &clientcmd.ConfigOverrides{})
cfg, err := cc.ClientConfig()
if err != nil {
return nil, fmt.Errorf("kube config (no in-cluster, no kubeconfig): %w", err)
}
return kubernetes.NewForConfig(cfg)
}
// claudeSink converts each claudewatcher.Batch into a raw session dump
// under brain/archive/claude-sessions/<host>/. Deliberately NOT a wiki
// note (api.WriteNote / brain/wiki/) — raw full transcripts out-ranked
// curated ai-sessions summaries in BM25 (177,818 vs 45,335 on the same
// query) and duplicated content already summarized elsewhere. Kept for
// deep lookups, never indexed. See ai-sessions#10.
type claudeSink struct {
brainDir string
logger *slog.Logger
@@ -67,16 +89,14 @@ func (s *claudeSink) Ingest(ctx context.Context, b claudewatcher.Batch) error {
sb.WriteString("\n\n")
}
slug := "session-" + b.Host + "-" + b.SessionID
if _, err := api.WriteNote(s.brainDir, api.WriteNoteOptions{
Filename: slug,
Wing: "claude-sessions",
Hall: "facts",
Type: "source",
Domain: b.ProjectID,
Content: sb.String(),
}); err != nil {
return fmt.Errorf("write claude session note: %w", err)
dest := filepath.Join(s.brainDir, "archive", "claude-sessions", b.Host, slug+".md")
if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
return fmt.Errorf("create claude-sessions archive dir: %w", err)
}
if err := os.WriteFile(dest, []byte(sb.String()), 0o644); err != nil {
return fmt.Errorf("write claude session archive: %w", err)
}
s.logger.Debug("claude session archived (non-indexed)", "path", dest)
return nil
}
@@ -295,6 +315,36 @@ func main() {
}
ctx := context.Background()
// Built before the watcher, the graph backfill and session ingestion, and
// retried rather than fatal (#83). Two separate reasons:
//
// - On a cold cluster boot this pod can beat dex to serving and gets a
// 502 from ingress. Nothing is misconfigured; it simply started first.
// Exiting there meant CrashLoopBackOff redoing the backfill on every
// attempt, with exponential backoff stretching the brain outage.
// - Constructing it first means a genuine config error (wrong issuer,
// bad audience) fails in seconds instead of after all that work.
//
// The budget stays bounded: retrying forever would turn a misconfiguration
// into a process that looks alive and never serves.
jwtValidator, err := startupretry.Do(ctx,
startupretry.Policy{Attempts: 8, Base: time.Second, Max: 30 * time.Second},
func(ctx context.Context) (*chassisauth.JWTValidator, error) {
v, err := chassisauth.NewJWTValidator(ctx, os.Getenv("DEX_ISSUER_URL"), os.Getenv("MCP_AUDIENCE"))
if err != nil {
logger.Warn("oidc discovery not ready, retrying", "err", err)
}
return v, err
})
if err != nil {
logger.Error("build jwt validator", "err", err)
os.Exit(1)
}
if jwtValidator != nil {
logger.Info("jwt auth enabled", "issuer", os.Getenv("DEX_ISSUER_URL"))
}
if watchInterval > 0 {
watcher.Start(ctx, watcher.Config{
BrainDir: brainDir,
@@ -352,6 +402,29 @@ func main() {
logger.Info("claudewatcher started",
"sessions_dir", claudeDir, "host", host, "interval", interval)
}
// Gitea push webhook -> on-demand brain-sync Job, instead of waiting up
// to 15 minutes for the next CronJob poll. Off by default (opt in via
// GITEA_WEBHOOK_SECRET) since it needs Job-create RBAC in the "brain"
// namespace that a fresh deploy won't have granted yet.
var webhookHandler *webhook.Handler
if webhookSecret := os.Getenv("GITEA_WEBHOOK_SECRET"); webhookSecret != "" {
kc, err := kubeClient()
if err != nil {
logger.Error("brain-sync webhook: kube client", "err", err)
os.Exit(1)
}
webhookHandler = &webhook.Handler{
Secret: webhookSecret,
Clientset: kc,
Namespace: envOr("BRAIN_SYNC_NAMESPACE", "brain"),
CronJobName: envOr("BRAIN_SYNC_CRONJOB", "brain-sync"),
WatchRepo: envOr("BRAIN_SYNC_WATCH_REPO", "mathias/brain"),
Logger: logger,
}
logger.Info("brain-sync webhook enabled", "namespace", webhookHandler.Namespace, "cronjob", webhookHandler.CronJobName)
}
if vectorStore != nil {
embedSyncInterval := envInt("BRAIN_EMBED_SYNC_INTERVAL", 300)
vectorstore.StartSync(ctx, brainDir, vectorStore,
@@ -373,15 +446,9 @@ func main() {
mux.HandleFunc("POST /promote", h.Promote)
mux.HandleFunc("POST /backfill-embeddings", h.BackfillEmbeddings)
mux.HandleFunc("GET /pass-rate", h.PassRate)
jwtValidator, err := chassisauth.NewJWTValidator(ctx, os.Getenv("DEX_ISSUER_URL"), os.Getenv("MCP_AUDIENCE"))
if err != nil {
logger.Error("build jwt validator", "err", err)
os.Exit(1)
if webhookHandler != nil {
mux.Handle("POST /webhooks/brain-sync", webhookHandler)
}
if jwtValidator != nil {
logger.Info("jwt auth enabled", "issuer", os.Getenv("DEX_ISSUER_URL"))
}
// Resource-metadata URL is only emitted on 401 when Dex OAuth is
// configured. Static-Bearer-only deployments leave this empty so
// clients never see an OAuth challenge.
@@ -412,12 +479,8 @@ func main() {
os.Exit(1)
}
auditSink := buildAuditSink(ctx, brainDir, logger)
// The Gitea client also satisfies SummaryWriter (#66): session
// summaries are written to mathias/ai-sessions over the same API
// token. nil only if a future tracker impl lacks file writes.
summaryWriter, _ := tracker.(capture.SummaryWriter)
captureSvc := capture.NewService(
mcpSrv.BrainStore(), tracker, summaryWriter, classCfg, auditSink)
mcpSrv.BrainStore(), tracker, classCfg, auditSink)
sovereign := splitList(os.Getenv("BRAIN_CAPTURE_SOVEREIGN_PRINCIPALS"))
resolver := capturehttp.NewOriginResolver(sovereign)
captureH := capturehttp.New(captureSvc, jwtValidator, mcpToken, "local-cli", resolver)
+38
View File
@@ -0,0 +1,38 @@
// ingestion/cmd/server/main_test.go
package main
import (
"context"
"log/slog"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/claudewatcher"
)
func TestClaudeSink_IngestWritesToNonIndexedArchiveNotWiki(t *testing.T) {
dir := t.TempDir()
sink := &claudeSink{brainDir: dir, logger: slog.New(slog.NewTextHandler(os.Stderr, nil))}
err := sink.Ingest(context.Background(), claudewatcher.Batch{
Host: "koala",
FilePath: "/host-home-claude/projects/-home-mathias-dev/abc.jsonl",
SessionID: "abc",
ProjectID: "-home-mathias-dev",
Turns: []claudewatcher.Turn{
{Type: "assistant", Content: "did a thing"},
},
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(dir, "archive", "claude-sessions", "koala", "session-koala-abc.md"))
require.NoError(t, err, "raw session dump must land in the non-indexed archive")
assert.Contains(t, string(got), "did a thing")
_, err = os.Stat(filepath.Join(dir, "wiki", "claude-sessions"))
assert.True(t, os.IsNotExist(err), "raw transcripts must never land under wiki/ (ai-sessions#10 — BM25 pollution)")
}
+44 -4
View File
@@ -2,17 +2,57 @@ module github.com/mathiasbq/hyperguild/ingestion
go 1.26.1
require github.com/stretchr/testify v1.11.1
require (
github.com/stretchr/testify v1.11.1
k8s.io/api v0.31.3
k8s.io/apimachinery v0.31.3
k8s.io/client-go v0.31.3
)
require (
github.com/kr/text v0.2.0 // indirect
github.com/emicklei/go-restful/v3 v3.11.0 // indirect
github.com/fxamacker/cbor/v2 v2.7.0 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-openapi/jsonpointer v0.19.6 // indirect
github.com/go-openapi/jsonreference v0.20.2 // indirect
github.com/go-openapi/swag v0.22.4 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/google/gnostic-models v0.6.8 // indirect
github.com/google/go-cmp v0.6.0 // indirect
github.com/google/gofuzz v1.2.0 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/imdario/mergo v0.3.6 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/lestrrat-go/jwx/v2 v2.1.6 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/rogpeppe/go-internal v1.15.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/x448/float16 v0.8.4 // indirect
golang.org/x/net v0.26.0 // indirect
golang.org/x/oauth2 v0.21.0 // indirect
golang.org/x/term v0.28.0 // indirect
golang.org/x/time v0.3.0 // indirect
google.golang.org/protobuf v1.34.2 // indirect
gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
k8s.io/klog/v2 v2.130.1 // indirect
k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 // indirect
k8s.io/utils v0.0.0-20240711033017-18e509b52bc8 // indirect
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect
sigs.k8s.io/structured-merge-diff/v4 v4.4.1 // indirect
sigs.k8s.io/yaml v1.4.0 // indirect
)
require (
git.d-ma.be/mathias/mcp-chassis v0.2.0
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0 // indirect
github.com/goccy/go-json v0.10.3 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
@@ -24,7 +64,7 @@ require (
github.com/lestrrat-go/httprc v1.0.6 // indirect
github.com/lestrrat-go/iter v1.0.2 // indirect
github.com/lestrrat-go/option v1.0.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/segmentio/asm v1.2.0 // indirect
golang.org/x/crypto v0.32.0 // indirect
golang.org/x/sync v0.17.0 // indirect
+135 -4
View File
@@ -2,12 +2,46 @@ git.d-ma.be/mathias/mcp-chassis v0.2.0 h1:6fLmb7xqRa2nNVWsHaUbbfbArgDXJw/gDhb09c
git.d-ma.be/mathias/mcp-chassis v0.2.0/go.mod h1:Ks7EK2UnGAN0H3rJjKUxUagX8/ZBdtLrOlcUbv0RwH8=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0 h1:NMZiJj8QnKe1LgsbDayM4UoHwbvwDRwnI3hwNaAHRnc=
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0/go.mod h1:ZXNYxsqcloTdSy/rNShjYzMhyjf0LaoftYK0p+A3h40=
github.com/emicklei/go-restful/v3 v3.11.0 h1:rAQeMHw1c7zTmncogyy8VvRZwtkmkZ4FxERmMY4rD+g=
github.com/emicklei/go-restful/v3 v3.11.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc=
github.com/fxamacker/cbor/v2 v2.7.0 h1:iM5WgngdRBanHcxugY4JySA0nk1wZorNOpTgCMedv5E=
github.com/fxamacker/cbor/v2 v2.7.0/go.mod h1:pxXPTn3joSm21Gbwsv0w9OSA2y1HFR9qXEeXQVeNoDQ=
github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-openapi/jsonpointer v0.19.6 h1:eCs3fxoIi3Wh6vtgmLTOjdhSpiqphQ+DaPn38N2ZdrE=
github.com/go-openapi/jsonpointer v0.19.6/go.mod h1:osyAmYz/mB/C3I+WsTTSgw1ONzaLJoLCyoi6/zppojs=
github.com/go-openapi/jsonreference v0.20.2 h1:3sVjiK66+uXK/6oQ8xgcRKcFgQ5KXa2KvnJRumpMGbE=
github.com/go-openapi/jsonreference v0.20.2/go.mod h1:Bl1zwGIM8/wsvqjsOQLJ/SH+En5Ap4rVB5KVcIDZG2k=
github.com/go-openapi/swag v0.22.3/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+VcZ0yl14=
github.com/go-openapi/swag v0.22.4 h1:QLMzNJnMGPRNDCbySlcj1x01tzU8/9LTTL9hZZZogBU=
github.com/go-openapi/swag v0.22.4/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+VcZ0yl14=
github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI=
github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8=
github.com/goccy/go-json v0.10.3 h1:KZ5WoDbxAIgm2HNbYckL0se1fHD6rz5j4ywS6ebzDqA=
github.com/goccy/go-json v0.10.3/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
github.com/google/gnostic-models v0.6.8 h1:yo/ABAfM5IMRsS1VnXjTBvUb61tFIHozhlYvRgGre9I=
github.com/google/gnostic-models v0.6.8/go.mod h1:5n7qKqH0f5wFt+aWF8CW6pZLLNOfYuF5OpfBSENuI8U=
github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0=
github.com/google/gofuzz v1.2.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/pprof v0.0.0-20240525223248-4bfdf5a9a2af h1:kmjWCqn2qkEml422C2Rrd27c3VGxi6a/6HNq8QmHRKM=
github.com/google/pprof v0.0.0-20240525223248-4bfdf5a9a2af/go.mod h1:K1liHPHnj73Fdn/EKuT8nrFqBihUSKXoLYU0BuatOYo=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/imdario/mergo v0.3.6 h1:xTNEAn+kxVO7dTZGu0CegyqKZmoWFI0rF8UxjlB2d28=
github.com/imdario/mergo v0.3.6/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
@@ -16,8 +50,17 @@ github.com/jackc/pgx/v5 v5.9.2 h1:3ZhOzMWnR4yJ+RW1XImIPsD1aNSz4T4fyP7zlQb56hw=
github.com/jackc/pgx/v5 v5.9.2/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0=
github.com/kr/pretty v0.3.0/go.mod h1:640gp4NfQd8pI5XOwp5fnNeVWj67G7CFk/SaSQn7NBk=
github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/lestrrat-go/blackmagic v1.0.3 h1:94HXkVLxkZO9vJI/w2u1T0DAoprShFd13xtnSINtDWs=
@@ -32,30 +75,118 @@ github.com/lestrrat-go/jwx/v2 v2.1.6 h1:hxM1gfDILk/l5ylers6BX/Eq1m/pnxe9NBwW6lVf
github.com/lestrrat-go/jwx/v2 v2.1.6/go.mod h1:Y722kU5r/8mV7fYDifjug0r8FK8mZdw0K0GpJw/l8pU=
github.com/lestrrat-go/option v1.0.1 h1:oAzP2fvZGQKWkvHa1/SAcFolBEca1oN+mQ7eooNBEYU=
github.com/lestrrat-go/option v1.0.1/go.mod h1:5ZHFbivi4xwXxhxY9XHDe2FHo6/Z7WWmtT7T5nBBp3I=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0=
github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/onsi/ginkgo/v2 v2.19.0 h1:9Cnnf7UHo57Hy3k6/m5k3dRfGTMXGvxhHFvkDTCTpvA=
github.com/onsi/ginkgo/v2 v2.19.0/go.mod h1:rlwLi9PilAFJ8jCg9UE1QP6VBpd6/xj3SRC0d6TU0To=
github.com/onsi/gomega v1.19.0 h1:4ieX6qQjPP/BfC3mpsAtIGGlxTWPeA3Inl/7DtXw1tw=
github.com/onsi/gomega v1.19.0/go.mod h1:LY+I3pBVzYsTBU1AnDwOSxaYi9WoWiqgwooUqq9yPro=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/rogpeppe/go-internal v1.15.0 h1:D0RCU5rMAp+SpgkiNdrjfJ+LX4J1M32V2NeCY7EJ6hc=
github.com/rogpeppe/go-internal v1.15.0/go.mod h1:DrUVZyrJU+txYW5/1kwtXQSMFio52ZOxX7yM1VHvnxs=
github.com/segmentio/asm v1.2.0 h1:9BQrFxC+YOHJlTlHGkTrFWf59nbL3XnCoFLTwDCI7ys=
github.com/segmentio/asm v1.2.0/go.mod h1:BqMnlJP91P8d+4ibuonYZw9mfnzI9HfxselHZr5aAcs=
github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM=
github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.32.0 h1:euUpcYgM8WcP71gNpTqQCn6rC2t6ULUPiOzfWaXVVfc=
golang.org/x/crypto v0.32.0/go.mod h1:ZnnJkOaASj8g0AjIduWNlq2NRxL0PlBrbKVyZ6V/Ugc=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.26.0 h1:soB7SVo0PWrY4vPW/+ay0jKDNScG2X9wFeYlXIvJsOQ=
golang.org/x/net v0.26.0/go.mod h1:5YKkiSynbBIh3p6iOc/vibscux0x38BZDkn8sCUPxHE=
golang.org/x/oauth2 v0.21.0 h1:tsimM75w1tF/uws5rbeHzIWxEqElMehnc+iW793zsZs=
golang.org/x/oauth2 v0.21.0/go.mod h1:XYTD2NtWslqkgxebSiOHnXEap4TF09sJSc7H1sXbhtI=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.17.0 h1:l60nONMj9l5drqw6jlhIELNv9I0A4OFgRsG9k2oT9Ug=
golang.org/x/sync v0.17.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik=
golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/term v0.28.0 h1:/Ts8HFuMR2E6IP/jlo7QVLZHggjKQbhu/7H0LJFr3Gg=
golang.org/x/term v0.28.0/go.mod h1:Sw/lC2IAUZ92udQNf3WodGtn4k/XoLyZoh8v/8uiwek=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.29.0 h1:1neNs90w9YzJ9BocxfsQNHKuAT4pkghyXc4nhZ6sJvk=
golang.org/x/text v0.29.0/go.mod h1:7MhJOA9CD2qZyOKYazxdYMF85OwPdEr9jTtBpO7ydH4=
golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4=
golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.36.0 h1:kWS0uv/zsvHEle1LbV5LE8QujrxB3wfQyxHfhOk0Qkg=
golang.org/x/tools v0.36.0/go.mod h1:WBDiHKJK8YgLHlcQPYQzNCkUxUypCaa5ZegCVutKm+s=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/protobuf v1.34.2 h1:6xV6lTsCfpGD21XK49h7MhtcApnLqkfYgPcdHftf6hg=
google.golang.org/protobuf v1.34.2/go.mod h1:qYOHts0dSfpeUzUFpOMr/WGzszTmLH+DiWniOlNbLDw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/evanphx/json-patch.v4 v4.12.0 h1:n6jtcsulIzXPJaxegRbvFNNrZDjbij7ny3gmSPG+6V4=
gopkg.in/evanphx/json-patch.v4 v4.12.0/go.mod h1:p8EYWUEYMpynmqDbY58zCKCFZw8pRWMG4EsWvDvM72M=
gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc=
gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
k8s.io/api v0.31.3 h1:umzm5o8lFbdN/hIXbrK9oRpOproJO62CV1zqxXrLgk8=
k8s.io/api v0.31.3/go.mod h1:UJrkIp9pnMOI9K2nlL6vwpxRzzEX5sWgn8kGQe92kCE=
k8s.io/apimachinery v0.31.3 h1:6l0WhcYgasZ/wk9ktLq5vLaoXJJr5ts6lkaQzgeYPq4=
k8s.io/apimachinery v0.31.3/go.mod h1:rsPdaZJfTfLsNJSQzNHQvYoTmxhoOEofxtOsF3rtsMo=
k8s.io/client-go v0.31.3 h1:CAlZuM+PH2cm+86LOBemaJI/lQ5linJ6UFxKX/SoG+4=
k8s.io/client-go v0.31.3/go.mod h1:2CgjPUTpv3fE5dNygAr2NcM8nhHzXvxB8KL5gYc3kJs=
k8s.io/klog/v2 v2.130.1 h1:n9Xl7H1Xvksem4KFG4PYbdQCQxqc/tTUyrgXaOhHSzk=
k8s.io/klog/v2 v2.130.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE=
k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 h1:BZqlfIlq5YbRMFko6/PM7FjZpUb45WallggurYhKGag=
k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340/go.mod h1:yD4MZYeKMBwQKVht279WycxKyM84kkAx2DPrTXaeb98=
k8s.io/utils v0.0.0-20240711033017-18e509b52bc8 h1:pUdcCO1Lk/tbT5ztQWOBi5HBgbBP1J8+AsQnQCKsi8A=
k8s.io/utils v0.0.0-20240711033017-18e509b52bc8/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0=
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd h1:EDPBXCAspyGV4jQlpZSudPeMmr1bNJefnuqLsRAsHZo=
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd/go.mod h1:B8JuhiUyNFVKdsE8h686QcCxMaH6HrOAZj4vswFpcB0=
sigs.k8s.io/structured-merge-diff/v4 v4.4.1 h1:150L+0vs/8DA78h1u02ooW1/fFq/Lwr+sGiqlzvrtq4=
sigs.k8s.io/structured-merge-diff/v4 v4.4.1/go.mod h1:N8hJocpFajUSSeSJ9bOZ77VzejKZaXsTtZo4/u7Io08=
sigs.k8s.io/yaml v1.4.0 h1:Mk1wCc2gy/F0THH0TAp1QYyJNzRm2KCLy3o5ASXVI5E=
sigs.k8s.io/yaml v1.4.0/go.mod h1:Ejl7/uTz7PSA4eKMyQCUTnhZYNmLIl+5c2lQPGR2BPY=
+119 -24
View File
@@ -51,12 +51,13 @@ type queryRequest struct {
}
type writeRequest struct {
Content string `json:"content"`
Filename string `json:"filename,omitempty"`
Type string `json:"type,omitempty"`
Domain string `json:"domain,omitempty"`
Wing string `json:"wing,omitempty"`
Hall string `json:"hall,omitempty"`
Content string `json:"content"`
Filename string `json:"filename,omitempty"`
Type string `json:"type,omitempty"`
Domain string `json:"domain,omitempty"`
Wing string `json:"wing,omitempty"`
Hall string `json:"hall,omitempty"`
SourceType string `json:"source_type,omitempty"` // "external" opts a hall=facts entry out of the internal default
}
type ingestRequest struct {
@@ -115,12 +116,13 @@ func (h *Handler) Query(w http.ResponseWriter, r *http.Request) {
// When either is empty, the note falls back to brain/knowledge/<filename>
// with optional type/domain frontmatter (legacy behaviour).
type WriteNoteOptions struct {
Content string
Filename string
Type string
Domain string
Wing string
Hall string
Content string
Filename string
Type string
Domain string
Wing string
Hall string
SourceType string // "internal" marks a first-party observation (e.g. claudewatcher) that needs no external citation
}
// WriteNote writes a markdown note into the brain. Returns the path
@@ -154,26 +156,111 @@ func writeHallNote(brainDir string, opts WriteNoteOptions) (string, error) {
return "", fmt.Errorf("create hall dir: %w", err)
}
existingFields, body := splitFrontmatter(opts.Content)
existingByKey := make(map[string]frontmatterField, len(existingFields))
for _, f := range existingFields {
existingByKey[f.key] = f
}
emitted := make(map[string]bool, 6)
var fm strings.Builder
fm.WriteString("---\n")
fmt.Fprintf(&fm, "wing: %s\n", brain.Sanitise(opts.Wing))
fmt.Fprintf(&fm, "hall: %s\n", opts.Hall)
fmt.Fprintf(&fm, "created_at: %s\n", time.Now().UTC().Format(time.RFC3339))
if opts.Type != "" {
fmt.Fprintf(&fm, "type: %s\n", opts.Type)
emitted["wing"], emitted["hall"], emitted["created_at"] = true, true, true
// writeField merges one key: opts.Content's own value (if the note already
// carries this field in its own frontmatter) always wins over the fallback,
// so promotion/extraction-step metadata survives verbatim instead of being
// shadowed by a second, stacked frontmatter block (#86).
writeField := func(key, fallback string) {
emitted[key] = true
if f, ok := existingByKey[key]; ok {
for _, line := range f.lines {
fm.WriteString(line)
fm.WriteString("\n")
}
return
}
if fallback != "" {
fmt.Fprintf(&fm, "%s: %s\n", key, fallback)
}
}
if opts.Domain != "" {
fmt.Fprintf(&fm, "domain: %s\n", opts.Domain)
writeField("type", opts.Type)
writeField("domain", opts.Domain)
sourceType := opts.SourceType
if sourceType == "" && opts.Hall == "facts" {
// Most hall=facts entries are first-party (an eval/benchmark the
// writer ran itself), not external claims — default to internal and
// require an explicit source_type: external opt-out for the rare
// citation-needing entry (brain-gardener#7).
sourceType = "internal"
}
writeField("source_type", sourceType)
for _, f := range existingFields {
if emitted[f.key] {
continue
}
for _, line := range f.lines {
fm.WriteString(line)
fm.WriteString("\n")
}
}
fm.WriteString("---\n")
if err := os.WriteFile(dest, []byte(fm.String()+opts.Content), 0o644); err != nil {
if err := os.WriteFile(dest, []byte(fm.String()+body), 0o644); err != nil {
return "", fmt.Errorf("write: %w", err)
}
rel, _ := filepath.Rel(brainDir, dest)
return filepath.ToSlash(rel), nil
}
// frontmatterField is one top-level YAML key from a frontmatter block,
// along with its raw line and any indented continuation lines (e.g. a
// bulleted list value spanning multiple lines).
type frontmatterField struct {
key string
lines []string
}
// splitFrontmatter splits a leading "---\n...\n---\n" YAML block out of
// content, returning its top-level fields in original order and the
// remaining body. If content has no leading frontmatter block, fields is
// nil and body is content unchanged.
func splitFrontmatter(content string) (fields []frontmatterField, body string) {
if !strings.HasPrefix(content, "---\n") {
return nil, content
}
lines := strings.Split(content, "\n")
i := 1
var cur *frontmatterField
for ; i < len(lines); i++ {
line := lines[i]
if strings.TrimSpace(line) == "---" {
i++
break
}
if line != "" && !strings.HasPrefix(line, " ") && !strings.HasPrefix(line, "\t") {
if cur != nil {
fields = append(fields, *cur)
}
key, _, _ := strings.Cut(line, ":")
cur = &frontmatterField{key: strings.TrimSpace(key), lines: []string{line}}
} else if cur != nil {
cur.lines = append(cur.lines, line)
}
}
if cur != nil {
fields = append(fields, *cur)
}
body = strings.Join(lines[i:], "\n")
return fields, body
}
// writeLegacyNote preserves the original brain/knowledge/ behaviour for
// callers that have not adopted the wing/hall taxonomy.
func writeLegacyNote(brainDir string, opts WriteNoteOptions) (string, error) {
@@ -332,11 +419,19 @@ func (h *Handler) Ingest(w http.ResponseWriter, r *http.Request) {
writeJSON(w, ingestResponse{Pages: pages, Warnings: warnings})
}
// supportedExtensions lists file extensions that IngestPath will process.
var supportedExtensions = map[string]bool{
".md": true,
".txt": true,
".pdf": true,
// isSupportedExtension reports whether IngestPath will process ext.
// .docx/.xlsx/.pptx/.png/.jpg/.jpeg require docmark (ADR-0013) and are only
// supported when DOCMARK_URL is configured — checked per-call (not cached at
// package init) so it reflects the environment at request time.
func isSupportedExtension(ext string) bool {
switch ext {
case ".md", ".txt", ".pdf":
return true
case ".docx", ".xlsx", ".pptx", ".png", ".jpg", ".jpeg":
return os.Getenv("DOCMARK_URL") != ""
default:
return false
}
}
// IngestPath handles POST /ingest-path — ingest a file or directory.
@@ -369,7 +464,7 @@ func (h *Handler) IngestPath(w http.ResponseWriter, r *http.Request) {
return nil
}
ext := strings.ToLower(filepath.Ext(path))
if !supportedExtensions[ext] {
if !isSupportedExtension(ext) {
return nil
}
content, readErr := extract.Text(path)
@@ -397,7 +492,7 @@ func (h *Handler) IngestPath(w http.ResponseWriter, r *http.Request) {
}
} else {
ext := strings.ToLower(filepath.Ext(req.Path))
if !supportedExtensions[ext] {
if !isSupportedExtension(ext) {
writeError(w, http.StatusBadRequest, fmt.Sprintf("unsupported file extension: %s", ext))
return
}
+110
View File
@@ -118,6 +118,116 @@ func TestWrite_IncludesFrontmatterWhenTypeProvided(t *testing.T) {
assert.Contains(t, string(content), "Some learning.")
}
func TestWriteNote_HallRouteIncludesSourceTypeWhenSet(t *testing.T) {
dir := t.TempDir()
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "# Claude session abc (koala)\n\nBody.\n",
Filename: "session-koala-abc",
Wing: "claude-sessions",
Hall: "facts",
Type: "source",
SourceType: "internal",
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.Contains(t, string(got), "source_type: internal")
assert.Contains(t, string(got), "wing: claude-sessions")
}
func TestWriteNote_HallFactsDefaultsSourceTypeInternalWhenUnset(t *testing.T) {
dir := t.TempDir()
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "manually captured fact.\n",
Wing: "agentsquad",
Hall: "facts",
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
// Most hall=facts entries are first-party (an eval/benchmark the agent ran
// itself), not external claims — default to internal, require explicit
// opt-out for the rare case that does need a citation (brain-gardener#7).
assert.Contains(t, string(got), "source_type: internal")
}
func TestWriteNote_HallFactsPreservesExplicitExternalSourceType(t *testing.T) {
dir := t.TempDir()
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "vendor pricing claim, needs a citation.\n",
Wing: "agentsquad",
Hall: "facts",
SourceType: "external",
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.Contains(t, string(got), "source_type: external")
}
func TestWriteNote_HallRouteOmitsSourceTypeForNonFactsHalls(t *testing.T) {
dir := t.TempDir()
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "a decision record.\n",
Wing: "agentsquad",
Hall: "decisions",
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.NotContains(t, string(got), "source_type")
}
func TestWriteNote_HallRouteMergesExistingFrontmatterInsteadOfStacking(t *testing.T) {
dir := t.TempDir()
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "---\ntitle: act_runner host-executor\ntags: [gitea-actions, act_runner]\n---\n\n# Body\n\nSome content.\n",
Filename: "act-runner-host-executor",
Wing: "homelab",
Hall: "failures",
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
body := string(got)
// exactly one frontmatter block: only two "---" delimiter lines total
assert.Equal(t, 2, strings.Count(body, "---\n"), "expected a single merged frontmatter block, not stacked blocks")
assert.Contains(t, body, "wing: homelab")
assert.Contains(t, body, "hall: failures")
assert.Contains(t, body, "title: act_runner host-executor")
assert.Contains(t, body, "tags: [gitea-actions, act_runner]")
assert.Contains(t, body, "# Body")
}
func TestWriteNote_HallRouteExistingTypeWinsOverOptsType(t *testing.T) {
dir := t.TempDir()
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "---\ntype: hypothesis\n---\n\nBody.\n",
Filename: "note",
Wing: "agentsquad",
Hall: "decisions",
Type: "decision", // should lose to content's own "type: hypothesis"
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.Contains(t, string(got), "type: hypothesis")
assert.NotContains(t, string(got), "type: decision")
}
func TestWrite_GeneratesFilenameIfAbsent(t *testing.T) {
dir, h := setup(t)
body, _ := json.Marshal(map[string]any{"content": "auto name"})
@@ -0,0 +1,61 @@
// ingestion/internal/api/ingestpath_docmark_test.go
package api_test
import (
"bytes"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestIngestPath_DocxUnsupportedWhenDocmarkNotConfigured(t *testing.T) {
t.Setenv("DOCMARK_URL", "")
_, h := setup(t)
dir := t.TempDir()
f := filepath.Join(dir, "doc.docx")
require.NoError(t, os.WriteFile(f, []byte("fake docx"), 0o644))
body, _ := json.Marshal(map[string]any{"path": f, "source": "test-doc", "dry_run": true})
req := httptest.NewRequest(http.MethodPost, "/ingest-path", bytes.NewReader(body))
rec := httptest.NewRecorder()
h.IngestPath(rec, req)
assert.Equal(t, http.StatusBadRequest, rec.Code, rec.Body.String())
}
func TestIngestPath_DocxSupportedWhenDocmarkConfigured(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"jsonrpc":"2.0","id":1,"result":{"content":[{"type":"text","text":"# Converted Doc"}]}}`))
}))
defer srv.Close()
t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
t.Setenv("DOCMARK_BEARER_TOKEN", "tok")
_, h := setup(t)
dir := t.TempDir()
f := filepath.Join(dir, "doc.docx")
require.NoError(t, os.WriteFile(f, []byte("fake docx"), 0o644))
body, _ := json.Marshal(map[string]any{"path": f, "source": "test-doc", "dry_run": true})
req := httptest.NewRequest(http.MethodPost, "/ingest-path", bytes.NewReader(body))
rec := httptest.NewRecorder()
h.IngestPath(rec, req)
require.Equal(t, http.StatusOK, rec.Code, rec.Body.String())
var resp map[string]any
require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &resp))
pages, ok := resp["pages"].([]any)
require.True(t, ok)
assert.NotEmpty(t, pages)
}
+2 -18
View File
@@ -1,8 +1,7 @@
// Package capture is the Clean-Architecture use-case for the uniform
// capture capability (issue #49/#51): persist a finished session's
// valuable output — insights → brain, action items → Gitea tickets,
// optional summary → ai-sessions — with one invocation, identical core
// behaviour across every harness.
// valuable output — insights → brain, action items → Gitea tickets —
// with one invocation, identical core behaviour across every harness.
//
// This package is pure orchestration. It depends only on ports
// (interfaces) and plain entities — no HTTP, no live Gitea, no embedding
@@ -83,19 +82,11 @@ type Ticket struct {
Body string
}
// Summary is an optional session summary bound for ai-sessions.
type Summary struct {
Title string
Body string
ReposTouched []string
}
// CaptureInput is the whole capture request.
type CaptureInput struct {
Context CaptureContext
Insights []Insight
Tickets []Ticket
Summary *Summary
DryRun bool
}
@@ -117,12 +108,6 @@ type TicketResult struct {
OK bool `json:"ok"`
}
// SummaryResult is the summary outcome in the receipt.
type SummaryResult struct {
Path string `json:"path,omitempty"`
OK bool `json:"ok"`
}
// ItemError pins a failure to a specific request item for the partial
// receipt. Item is a stable locator like "insight[1]" or "ticket[0]".
type ItemError struct {
@@ -136,7 +121,6 @@ type ItemError struct {
type CaptureReceipt struct {
Insights []InsightResult `json:"insights"`
Tickets []TicketResult `json:"tickets"`
Summary *SummaryResult `json:"summary,omitempty"`
Errors []ItemError `json:"errors"`
EffectiveClassification string `json:"effective_classification,omitempty"`
DryRun bool `json:"dry_run"`
-5
View File
@@ -69,11 +69,6 @@ type IssueTracker interface {
CommentIssue(ctx context.Context, repo string, number int, body string) (IssueRef, error)
}
// SummaryWriter is the ai-sessions summary port.
type SummaryWriter interface {
WriteFile(ctx context.Context, repo, path, content string) error
}
// ClassificationPolicy derives a target's sensitivity (model C). The
// "stricter wins" combination of declared vs derived is use-case policy
// and lives in the service, so the port stays minimal. Satisfied by
+49 -97
View File
@@ -2,8 +2,6 @@ package capture
import (
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"strings"
"time"
@@ -14,22 +12,19 @@ import (
// Service is the CaptureSession use-case. It depends only on ports.
type Service struct {
brain BrainStore
issues IssueTracker
summaries SummaryWriter
policy ClassificationPolicy
audit AuditSink
brain BrainStore
issues IssueTracker
policy ClassificationPolicy
audit AuditSink
// now is the clock, injectable for deterministic summary paths and
// audit timestamps in tests.
// now is the clock, injectable for deterministic audit timestamps in
// tests.
now func() time.Time
}
// NewService constructs a Service from its ports. summaries may be nil
// when no summary persistence is wired; a CaptureInput with a Summary
// then fails that item rather than panicking.
func NewService(b BrainStore, tr IssueTracker, sw SummaryWriter, p ClassificationPolicy, a AuditSink) *Service {
return &Service{brain: b, issues: tr, summaries: sw, policy: p, audit: a, now: time.Now}
// NewService constructs a Service from its ports.
func NewService(b BrainStore, tr IssueTracker, p ClassificationPolicy, a AuditSink) *Service {
return &Service{brain: b, issues: tr, policy: p, audit: a, now: time.Now}
}
var validActions = map[string]bool{"create": true, "close": true, "comment": true}
@@ -88,7 +83,7 @@ func (s *Service) Capture(ctx context.Context, in CaptureInput) (CaptureReceipt,
declared, _ = classification.ParseLevel(in.Context.Classification)
}
effective, securityEvents := s.resolveClassification(declared, in)
effective, securityEvents, escalators := s.resolveClassification(declared, in)
// Server-derived origin governs the I1 gate; a caller-asserted harness
// label that names a different zone is descriptive-only and logged as a
@@ -112,8 +107,8 @@ func (s *Service) Capture(ctx context.Context, in CaptureInput) (CaptureReceipt,
Items: nil, // refused before any write
SecurityEvents: append(securityEvents, "I1 refusal: confidential capture via us-nexus origin"),
}, AuditCentral)
return CaptureReceipt{}, fmt.Errorf("%w: effective classification confidential through %s origin",
ErrSovereigntyRefused, in.Context.Origin)
return CaptureReceipt{}, fmt.Errorf("%w: effective classification confidential through %s origin (%s)",
ErrSovereigntyRefused, in.Context.Origin, escalationReason(escalators))
}
receipt := CaptureReceipt{
@@ -131,9 +126,6 @@ func (s *Service) Capture(ctx context.Context, in CaptureInput) (CaptureReceipt,
for _, tk := range in.Tickets {
receipt.Tickets = append(receipt.Tickets, TicketResult{Repo: tk.Repo, Action: tk.Action, Number: tk.Number, OK: true})
}
if in.Summary != nil {
receipt.Summary = &SummaryResult{Path: s.summaryPath(in.Context, in.Summary), OK: true}
}
return receipt, nil
}
@@ -169,16 +161,6 @@ func (s *Service) Capture(ctx context.Context, in CaptureInput) (CaptureReceipt,
landed = append(landed, fmt.Sprintf("ticket:%s#%d", tk.Repo, res.Number))
}
if in.Summary != nil {
res, err := s.persistSummary(ctx, in.Context, in.Summary)
receipt.Summary = &res
if err != nil {
receipt.Errors = append(receipt.Errors, ItemError{Item: "summary", Error: err.Error()})
} else {
landed = append(landed, "summary:"+res.Path)
}
}
// I5: persist the request-level audit record of exactly what landed,
// using the outcome reserved before the writes. AuditBuffered surfaces
// the degraded (locally-buffered) state on the receipt.
@@ -239,15 +221,28 @@ func (s *Service) validate(in CaptureInput) error {
return nil
}
// resolveClassification computes the effective level (stricter of
// declared and every target's derived level) and collects a security
// event whenever the caller under-declared relative to a target floor.
func (s *Service) resolveClassification(declared classification.Level, in CaptureInput) (classification.Level, []string) {
// resolveClassification computes the effective level (stricter of declared
// and every target's derived level), collects a security event whenever the
// caller under-declared relative to a target floor, and names the targets
// that carry the strictest level.
//
// That third return is what makes an I1 refusal actionable. A caller sending
// twenty tickets gets one refusal; without the offending target it cannot
// tell which repo tripped the gate, and infra#265 shows what happens next —
// the caller concluded the tickets ARRAY was broken, built a fallback of
// twenty individual calls, and filed a bug against the wrong thing.
func (s *Service) resolveClassification(declared classification.Level, in CaptureInput) (classification.Level, []string, []string) {
effective := declared
var events []string
derivedFor := map[string]classification.Level{}
var order []string
consider := func(kind classification.TargetKind, name string) {
derived := s.policy.Derive(classification.Target{Kind: kind, Name: name})
effective = classification.Stricter(effective, derived)
if _, seen := derivedFor[name]; !seen {
derivedFor[name] = derived
order = append(order, name)
}
if declared < derived {
events = append(events, fmt.Sprintf("classification under-declared: declared=%s target=%s(%s) derived=%s",
declared, name, kindString(kind), derived))
@@ -259,12 +254,28 @@ func (s *Service) resolveClassification(declared classification.Level, in Captur
for _, tk := range in.Tickets {
consider(classification.RepoTarget, tk.Repo)
}
if in.Summary != nil {
for _, repo := range in.Summary.ReposTouched {
consider(classification.RepoTarget, repo)
// Only targets that actually reach the effective level are named: at
// twenty tickets, listing the innocent ones is noise that hides the one
// that matters.
var escalators []string
for _, name := range order {
if derivedFor[name] == effective && effective > declared {
escalators = append(escalators, name)
}
}
return effective, events
return effective, events, escalators
}
// escalationReason renders why the effective classification landed where it
// did, for an operator reading a refusal. Naming the caller's own declaration
// matters as much as naming a target: "you asked for this" and "a repo you
// touched required it" call for different fixes.
func escalationReason(escalators []string) string {
if len(escalators) == 0 {
return "declared by the caller; no target required it"
}
return "required by: " + strings.Join(escalators, ", ")
}
func (s *Service) persistInsight(ctx context.Context, ins Insight) (InsightResult, string, error) {
@@ -309,60 +320,6 @@ func (s *Service) persistTicket(ctx context.Context, tk Ticket) (TicketResult, e
return res, nil
}
func (s *Service) persistSummary(ctx context.Context, c CaptureContext, sum *Summary) (SummaryResult, error) {
if s.summaries == nil {
return SummaryResult{OK: false}, fmt.Errorf("no summary writer configured")
}
path := s.summaryPath(c, sum)
content := s.renderSummary(c, sum)
repo := "ai-sessions"
if err := s.summaries.WriteFile(ctx, repo, path, content); err != nil {
return SummaryResult{Path: path, OK: false}, err
}
return SummaryResult{Path: path, OK: true}, nil
}
// summaryPath builds summaries/<harness>/<YYYY-MM>/<date>-<slug>-<ref8>.md.
// The ref8 disambiguator is derived from the session_ref (or the title
// when no ref is present) so distinct sessions never collide.
func (s *Service) summaryPath(c CaptureContext, sum *Summary) string {
t := s.now().UTC()
slug := brain.Sanitise(sum.Title)
if slug == "" {
slug = "summary"
}
seed := c.SessionRef
if seed == "" {
seed = sum.Title + sum.Body
}
sum8 := shortHash(seed)
return fmt.Sprintf("summaries/%s/%s/%s-%s-%s.md",
brain.Sanitise(c.Harness), t.Format("2006-01"), t.Format("2006-01-02"), slug, sum8)
}
// renderSummary stamps fidelity + session metadata into frontmatter so the
// richer-fidelity-supersedes-thinner collision rule has the data it needs.
func (s *Service) renderSummary(c CaptureContext, sum *Summary) string {
var b strings.Builder
b.WriteString("---\n")
fmt.Fprintf(&b, "title: %s\n", sum.Title)
fmt.Fprintf(&b, "harness: %s\n", c.Harness)
if c.SessionRef != "" {
fmt.Fprintf(&b, "session_ref: %s\n", c.SessionRef)
}
fmt.Fprintf(&b, "fidelity: %s\n", c.Fidelity)
fmt.Fprintf(&b, "captured_at: %s\n", s.now().UTC().Format(time.RFC3339))
if len(sum.ReposTouched) > 0 {
fmt.Fprintf(&b, "repos_touched: [%s]\n", strings.Join(sum.ReposTouched, ", "))
}
b.WriteString("---\n\n")
b.WriteString(sum.Body)
if !strings.HasSuffix(sum.Body, "\n") {
b.WriteByte('\n')
}
return b.String()
}
func kindString(k classification.TargetKind) string {
if k == classification.RepoTarget {
return "repo"
@@ -381,8 +338,3 @@ func firstLine(s string) string {
}
return s
}
func shortHash(s string) string {
sum := sha256.Sum256([]byte(s))
return hex.EncodeToString(sum[:])[:8]
}
+122 -53
View File
@@ -3,6 +3,7 @@ package capture
import (
"context"
"errors"
"fmt"
"strings"
"testing"
"time"
@@ -59,12 +60,18 @@ type fakeTracker struct {
closed []int
comments []int
err error
// failOnTitle makes CreateIssue fail for one specific title, so a
// partial-failure receipt can be exercised without failing every item.
failOnTitle string
}
func (f *fakeTracker) CreateIssue(_ context.Context, repo, title, _ string) (IssueRef, error) {
if f.err != nil {
return IssueRef{}, f.err
}
if f.failOnTitle != "" && title == f.failOnTitle {
return IssueRef{}, errors.New("tracker refused this one")
}
f.created = append(f.created, repo+":"+title)
return IssueRef{Repo: repo, Number: 100 + len(f.created), URL: "https://git/" + repo + "/issues/x"}, nil
}
@@ -85,24 +92,11 @@ func (f *fakeTracker) CommentIssue(_ context.Context, repo string, number int, _
return IssueRef{Repo: repo, Number: number}, nil
}
type fakeSummary struct {
paths []string
content []string
err error
}
func (f *fakeSummary) WriteFile(_ context.Context, _, path, content string) error {
if f.err != nil {
return f.err
}
f.paths = append(f.paths, path)
f.content = append(f.content, content)
return nil
}
// fakePolicy derives from an explicit map; default Internal so tests pin
// behaviour without depending on the real defaulting.
type fakePolicy struct{ tags map[string]classification.Level }
type fakePolicy struct {
tags map[string]classification.Level
}
func (p fakePolicy) Derive(t classification.Target) classification.Level {
if lvl, ok := p.tags[t.Name]; ok {
@@ -135,8 +129,8 @@ func (f *fakeAudit) Record(_ context.Context, e AuditEntry, _ AuditOutcome) erro
// --- helpers ---
func newSvc(b BrainStore, tr IssueTracker, sw SummaryWriter, p ClassificationPolicy, a AuditSink) *Service {
s := NewService(b, tr, sw, p, a)
func newSvc(b BrainStore, tr IssueTracker, p ClassificationPolicy, a AuditSink) *Service {
s := NewService(b, tr, p, a)
s.now = func() time.Time { return time.Date(2026, 6, 22, 12, 0, 0, 0, time.UTC) }
return s
}
@@ -151,7 +145,7 @@ func TestCaptureHappyPath(t *testing.T) {
b := &fakeBrain{}
tr := &fakeTracker{}
au := &fakeAudit{}
svc := newSvc(b, tr, nil, fakePolicy{}, au)
svc := newSvc(b, tr, fakePolicy{}, au)
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
@@ -180,7 +174,7 @@ func TestCaptureHappyPath(t *testing.T) {
func TestCaptureSupersedeNotDuplicate(t *testing.T) {
b := &fakeBrain{}
svc := newSvc(b, &fakeTracker{}, nil, fakePolicy{}, &fakeAudit{})
svc := newSvc(b, &fakeTracker{}, fakePolicy{}, &fakeAudit{})
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
@@ -197,7 +191,7 @@ func TestCaptureValidationFailClosed(t *testing.T) {
b := &fakeBrain{}
tr := &fakeTracker{}
au := &fakeAudit{}
svc := newSvc(b, tr, nil, fakePolicy{}, au)
svc := newSvc(b, tr, fakePolicy{}, au)
_, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
@@ -216,7 +210,7 @@ func TestCaptureValidationFailClosed(t *testing.T) {
}
func TestCaptureValidationRejectsBadTicket(t *testing.T) {
svc := newSvc(&fakeBrain{}, &fakeTracker{}, nil, fakePolicy{}, &fakeAudit{})
svc := newSvc(&fakeBrain{}, &fakeTracker{}, fakePolicy{}, &fakeAudit{})
_, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
Tickets: []Ticket{{Repo: "hyperguild", Action: "frobnicate"}}, // bad action
@@ -239,7 +233,7 @@ func TestCapturePartialFailureBestEffort(t *testing.T) {
}}
tr := &fakeTracker{}
au := &fakeAudit{}
svc := newSvc(b, tr, nil, fakePolicy{}, au)
svc := newSvc(b, tr, fakePolicy{}, au)
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
@@ -264,7 +258,7 @@ func TestCaptureDryRunWritesNothing(t *testing.T) {
b := &fakeBrain{}
tr := &fakeTracker{}
au := &fakeAudit{}
svc := newSvc(b, tr, nil, fakePolicy{}, au)
svc := newSvc(b, tr, fakePolicy{}, au)
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
@@ -287,7 +281,7 @@ func TestCaptureStricterClassificationWins(t *testing.T) {
b := &fakeBrain{}
au := &fakeAudit{}
pol := fakePolicy{tags: map[string]classification.Level{"client-seb": classification.Confidential}}
svc := newSvc(b, &fakeTracker{}, nil, pol, au)
svc := newSvc(b, &fakeTracker{}, pol, au)
ctx := baseCtx()
ctx.Classification = "internal"
@@ -306,7 +300,7 @@ func TestCaptureCallerRaisingSensitivityHonoured(t *testing.T) {
// Caller declares confidential; target internal → effective confidential, NOT a security event.
au := &fakeAudit{}
pol := fakePolicy{tags: map[string]classification.Level{"hyperguild": classification.Internal}}
svc := newSvc(&fakeBrain{}, &fakeTracker{}, nil, pol, au)
svc := newSvc(&fakeBrain{}, &fakeTracker{}, pol, au)
ctx := baseCtx()
ctx.Classification = "confidential"
@@ -319,26 +313,6 @@ func TestCaptureCallerRaisingSensitivityHonoured(t *testing.T) {
assert.Empty(t, au.entries[0].SecurityEvents, "raising sensitivity is honoured, not flagged")
}
func TestCaptureSummaryPathAndFidelity(t *testing.T) {
sw := &fakeSummary{}
svc := newSvc(&fakeBrain{}, &fakeTracker{}, sw, fakePolicy{}, &fakeAudit{})
ctx := baseCtx()
ctx.Fidelity = "transcript-parse"
ctx.SessionRef = "abc123def456"
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: ctx,
Summary: &Summary{Title: "Session Wrap", Body: "did stuff", ReposTouched: []string{"hyperguild"}},
})
require.NoError(t, err)
require.NotNil(t, rec.Summary)
assert.True(t, rec.Summary.OK)
require.Len(t, sw.paths, 1)
assert.True(t, strings.HasPrefix(sw.paths[0], "summaries/claude-code/2026-06/"), "path: %s", sw.paths[0])
assert.Contains(t, sw.paths[0], "session-wrap")
assert.Contains(t, sw.content[0], "fidelity: transcript-parse", "fidelity stamped in frontmatter")
}
// --- I1 sovereignty gate (#53) ---
func TestCaptureRefusesConfidentialViaUSNexus(t *testing.T) {
@@ -346,7 +320,7 @@ func TestCaptureRefusesConfidentialViaUSNexus(t *testing.T) {
tr := &fakeTracker{}
au := &fakeAudit{}
pol := fakePolicy{tags: map[string]classification.Level{"client-seb": classification.Confidential}}
svc := newSvc(b, tr, nil, pol, au)
svc := newSvc(b, tr, pol, au)
ctx := baseCtx()
ctx.Classification = "confidential"
@@ -368,7 +342,7 @@ func TestCaptureRefusesConfidentialViaUSNexus(t *testing.T) {
func TestCaptureAllowsConfidentialViaSovereign(t *testing.T) {
b := &fakeBrain{}
pol := fakePolicy{tags: map[string]classification.Level{"client-seb": classification.Confidential}}
svc := newSvc(b, &fakeTracker{}, nil, pol, &fakeAudit{})
svc := newSvc(b, &fakeTracker{}, pol, &fakeAudit{})
ctx := baseCtx()
ctx.Classification = "confidential"
@@ -387,7 +361,7 @@ func TestCaptureAssertedLabelIgnoredAndLogged(t *testing.T) {
// us-nexus; confidential ⇒ refused, and the discrepancy is a security event.
au := &fakeAudit{}
pol := fakePolicy{tags: map[string]classification.Level{"client-seb": classification.Confidential}}
svc := newSvc(&fakeBrain{}, &fakeTracker{}, nil, pol, au)
svc := newSvc(&fakeBrain{}, &fakeTracker{}, pol, au)
ctx := baseCtx()
ctx.Harness = "sovereign-soil" // asserted
@@ -407,7 +381,7 @@ func TestCaptureAssertedLabelIgnoredAndLogged(t *testing.T) {
func TestCaptureInternalViaUSNexusAllowed(t *testing.T) {
// us-nexus origin is fine for non-confidential data.
b := &fakeBrain{}
svc := newSvc(b, &fakeTracker{}, nil, fakePolicy{}, &fakeAudit{})
svc := newSvc(b, &fakeTracker{}, fakePolicy{}, &fakeAudit{})
ctx := baseCtx()
ctx.Origin = ZoneUSNexus // internal classification, so gate doesn't fire
rec, err := svc.Capture(context.Background(), CaptureInput{
@@ -426,7 +400,7 @@ func TestCaptureRefusesWhenAuditReserveFails(t *testing.T) {
b := &fakeBrain{}
tr := &fakeTracker{}
au := &fakeAudit{reserveErr: errors.New("central sink unreachable")}
svc := newSvc(b, tr, nil, fakePolicy{}, au)
svc := newSvc(b, tr, fakePolicy{}, au)
_, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
@@ -443,7 +417,7 @@ func TestCaptureFlagsLocallyBufferedAudit(t *testing.T) {
// proceeds and the receipt flags the degraded audit state.
b := &fakeBrain{}
au := &fakeAudit{reserveMode: AuditBuffered}
svc := newSvc(b, &fakeTracker{}, nil, fakePolicy{}, au)
svc := newSvc(b, &fakeTracker{}, fakePolicy{}, au)
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
@@ -458,7 +432,7 @@ func TestCaptureFlagsLocallyBufferedAudit(t *testing.T) {
func TestCaptureDryRunSkipsAuditGate(t *testing.T) {
// dry_run must not even probe the audit sink (writes nothing anywhere).
au := &fakeAudit{reserveErr: errors.New("would refuse")}
svc := newSvc(&fakeBrain{}, &fakeTracker{}, nil, fakePolicy{}, au)
svc := newSvc(&fakeBrain{}, &fakeTracker{}, fakePolicy{}, au)
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
@@ -469,3 +443,98 @@ func TestCaptureDryRunSkipsAuditGate(t *testing.T) {
assert.True(t, rec.DryRun)
assert.Empty(t, au.entries)
}
// ---- hyperguild#91 part 1 -------------------------------------------------
//
// The reported symptom was "capture with a tickets array fails" (infra#265:
// 20 tickets errored, 5 errored, caller fell back to 20 individual
// issue_create calls). The array was never the failing dimension — every
// ticket targeted a client repo, and the I1 gate refuses that through a
// us-nexus origin at ANY length. The refusal did not say which target
// escalated the classification, so it read as a malfunction.
func TestCapture_TicketsArrayWorksAtLengthOne(t *testing.T) {
b, tr, au := &fakeBrain{}, &fakeTracker{}, &fakeAudit{}
svc := newSvc(b, tr, fakePolicy{}, au)
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
Tickets: []Ticket{{Repo: "hyperguild", Action: "create", Title: "one", Body: "b"}},
})
require.NoError(t, err)
require.Len(t, rec.Tickets, 1)
assert.True(t, rec.Tickets[0].OK)
assert.Len(t, tr.created, 1)
assert.Empty(t, rec.Errors)
}
func TestCapture_TicketsArrayWorksAtLengthMany(t *testing.T) {
b, tr, au := &fakeBrain{}, &fakeTracker{}, &fakeAudit{}
svc := newSvc(b, tr, fakePolicy{}, au)
in := CaptureInput{Context: baseCtx()}
for i := range 20 {
in.Tickets = append(in.Tickets, Ticket{
Repo: "hyperguild", Action: "create", Title: fmt.Sprintf("t%02d", i), Body: "b",
})
}
rec, err := svc.Capture(context.Background(), in)
require.NoError(t, err)
require.Len(t, rec.Tickets, 20, "one receipt entry per ticket")
for i, r := range rec.Tickets {
assert.True(t, r.OK, "ticket %d failed: %+v", i, rec.Errors)
}
assert.Len(t, tr.created, 20)
assert.Empty(t, rec.Errors)
}
func TestCapture_OneFailingTicketDoesNotSinkTheRest(t *testing.T) {
// infra#265 asked explicitly: "if ticket 3 of 5 fails, what happened to
// 1 and 2?" They land, and the receipt says so per item.
b, au := &fakeBrain{}, &fakeAudit{}
tr := &fakeTracker{failOnTitle: "boom"}
svc := newSvc(b, tr, fakePolicy{}, au)
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
Tickets: []Ticket{
{Repo: "hyperguild", Action: "create", Title: "ok-1", Body: "b"},
{Repo: "hyperguild", Action: "create", Title: "boom", Body: "b"},
{Repo: "hyperguild", Action: "create", Title: "ok-3", Body: "b"},
},
})
require.NoError(t, err, "a per-item failure is a partial receipt, not a call error")
require.Len(t, rec.Tickets, 3)
assert.True(t, rec.Tickets[0].OK)
assert.False(t, rec.Tickets[1].OK)
assert.True(t, rec.Tickets[2].OK)
require.Len(t, rec.Errors, 1)
assert.Contains(t, rec.Errors[0].Item, "ticket[1]")
}
func TestCapture_I1RefusalNamesWhatEscalatedTheClassification(t *testing.T) {
// The actual defect behind infra#265. Without the offending target in
// the message, a caller with 20 tickets cannot tell which repo tripped
// the gate — and reasonably concludes the array itself is broken.
b, tr, au := &fakeBrain{}, &fakeTracker{}, &fakeAudit{}
pol := fakePolicy{tags: map[string]classification.Level{"trucker-ecommerce": classification.Confidential}}
svc := newSvc(b, tr, pol, au)
ctx := baseCtx()
ctx.Origin = ZoneUSNexus
_, err := svc.Capture(context.Background(), CaptureInput{
Context: ctx,
Tickets: []Ticket{
{Repo: "hyperguild", Action: "create", Title: "fine", Body: "b"},
{Repo: "trucker-ecommerce", Action: "create", Title: "client work", Body: "b"},
},
})
require.ErrorIs(t, err, ErrSovereigntyRefused)
assert.Contains(t, err.Error(), "trucker-ecommerce",
"the refusal must name the target that escalated the classification")
assert.NotContains(t, err.Error(), "hyperguild",
"only the offending target, or the message is noise at 20 tickets")
assert.Empty(t, tr.created, "refused before any write")
}
-15
View File
@@ -55,7 +55,6 @@ type request struct {
Context contextBody `json:"context"`
Insights []insightBody `json:"insights"`
Tickets []ticketBody `json:"tickets"`
Summary *summaryBody `json:"summary,omitempty"`
DryRun bool `json:"dry_run"`
}
@@ -82,12 +81,6 @@ type ticketBody struct {
Body string `json:"body,omitempty"`
}
type summaryBody struct {
Title string `json:"title"`
Body string `json:"body"`
ReposTouched []string `json:"repos_touched,omitempty"`
}
// ServeHTTP authenticates, derives origin, runs the use-case, and maps the
// result to an HTTP status.
func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
@@ -183,11 +176,6 @@ func (b request) toInput() capture.CaptureInput {
Repo: t.Repo, Action: t.Action, Number: t.Number, Title: t.Title, Body: t.Body,
})
}
if b.Summary != nil {
in.Summary = &capture.Summary{
Title: b.Summary.Title, Body: b.Summary.Body, ReposTouched: b.Summary.ReposTouched,
}
}
return in
}
@@ -204,9 +192,6 @@ func statusFor(rec capture.CaptureReceipt) int {
for _, t := range rec.Tickets {
count(&ok, &fail, t.OK)
}
if rec.Summary != nil {
count(&ok, &fail, rec.Summary.OK)
}
switch {
case fail == 0:
return http.StatusOK
@@ -49,7 +49,7 @@ func newHandler(t *testing.T, v capturehttp.Validator, tr capture.IssueTracker,
t.Helper()
cfg, err := classification.Load(t.TempDir())
require.NoError(t, err)
svc := capture.NewService(brainstore.New(t.TempDir()), tr, nil, cfg, audit.NewSlogSink(nil))
svc := capture.NewService(brainstore.New(t.TempDir()), tr, cfg, audit.NewSlogSink(nil))
return capturehttp.New(svc, v, staticTok, "local-cli", capturehttp.NewOriginResolver(sovereign))
}
@@ -190,7 +190,7 @@ func (refusingAudit) Record(context.Context, capture.AuditEntry, capture.AuditOu
func TestAuditUnavailableIs503(t *testing.T) {
cfg, err := classification.Load(t.TempDir())
require.NoError(t, err)
svc := capture.NewService(brainstore.New(t.TempDir()), fakeTracker{}, nil, cfg, refusingAudit{})
svc := capture.NewService(brainstore.New(t.TempDir()), fakeTracker{}, cfg, refusingAudit{})
h := capturehttp.New(svc, nil, staticTok, "local-cli", capturehttp.NewOriginResolver(nil))
rr := do(t, h, "Bearer "+staticTok, internalReq())
+148
View File
@@ -0,0 +1,148 @@
// ingestion/internal/extract/docmark.go
package extract
import (
"bytes"
"encoding/base64"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"time"
)
// docmarkRequest is a JSON-RPC 2.0 tools/call request for docmark's
// convert_to_markdown tool.
type docmarkRequest struct {
JSONRPC string `json:"jsonrpc"`
ID int `json:"id"`
Method string `json:"method"`
Params struct {
Name string `json:"name"`
Arguments struct {
ContentBase64 string `json:"content_base64"`
Filename string `json:"filename"`
} `json:"arguments"`
} `json:"params"`
}
type docmarkResponse struct {
Result *struct {
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
} `json:"content"`
IsError bool `json:"isError"`
} `json:"result"`
Error *struct {
Message string `json:"message"`
} `json:"error"`
}
// extractViaDocmark converts path (PDF/DOCX/XLSX/PPTX/image) to Markdown by
// calling the docmark MCP server with a single self-contained tools/call
// request (docmark runs stateless_http -- no initialize handshake or session
// ID needed). DOCMARK_URL must be set (e.g.
// http://docmark.docmark.svc.cluster.local:3001/mcp); DOCMARK_BEARER_TOKEN
// is docmark's static bearer (network is docmark's primary auth boundary,
// this is defense-in-depth — ADR-0013).
func extractViaDocmark(path string) (string, error) {
url := os.Getenv("DOCMARK_URL")
if url == "" {
return "", fmt.Errorf("extractViaDocmark: DOCMARK_URL is not set")
}
raw, err := os.ReadFile(path)
if err != nil {
return "", fmt.Errorf("read %s: %w", path, err)
}
var reqBody docmarkRequest
reqBody.JSONRPC = "2.0"
reqBody.ID = 1
reqBody.Method = "tools/call"
reqBody.Params.Name = "convert_to_markdown"
reqBody.Params.Arguments.ContentBase64 = base64.StdEncoding.EncodeToString(raw)
reqBody.Params.Arguments.Filename = fileBase(path)
payload, err := json.Marshal(reqBody)
if err != nil {
return "", fmt.Errorf("marshal docmark request: %w", err)
}
httpReq, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(payload))
if err != nil {
return "", fmt.Errorf("build docmark request: %w", err)
}
httpReq.Header.Set("Content-Type", "application/json")
httpReq.Header.Set("Accept", "application/json, text/event-stream")
if tok := os.Getenv("DOCMARK_BEARER_TOKEN"); tok != "" {
httpReq.Header.Set("Authorization", "Bearer "+tok)
}
client := &http.Client{Timeout: 60 * time.Second}
resp, err := client.Do(httpReq)
if err != nil {
return "", fmt.Errorf("call docmark: %w", err)
}
defer func() { _ = resp.Body.Close() }()
body, err := io.ReadAll(resp.Body)
if err != nil {
return "", fmt.Errorf("read docmark response: %w", err)
}
if resp.StatusCode != http.StatusOK {
return "", fmt.Errorf("docmark: HTTP %d: %s", resp.StatusCode, string(body))
}
jsonBody := body
if data := sseDataPayload(body); data != nil {
jsonBody = data
}
var out docmarkResponse
if err := json.Unmarshal(jsonBody, &out); err != nil {
return "", fmt.Errorf("decode docmark response: %w", err)
}
if out.Error != nil {
return "", fmt.Errorf("docmark: %s", out.Error.Message)
}
if out.Result == nil || len(out.Result.Content) == 0 {
return "", fmt.Errorf("docmark: empty response")
}
text := out.Result.Content[0].Text
if out.Result.IsError {
return "", fmt.Errorf("docmark: %s", text)
}
return text, nil
}
// sseDataPayload extracts the JSON payload from an SSE-framed response body
// ("event: message\r\ndata: {...}\r\n\r\n"). docmark's Streamable-HTTP
// transport frames every response this way (Content-Type: text/event-stream)
// regardless of stateless_http — that flag removes the session/initialize
// requirement, not the SSE wire framing. Returns nil if body isn't SSE-framed
// (e.g. a plain-JSON response, kept as a fallback for forward-compatibility).
func sseDataPayload(body []byte) []byte {
const prefix = "data: "
for _, line := range bytes.Split(body, []byte("\n")) {
line = bytes.TrimRight(line, "\r")
if bytes.HasPrefix(line, []byte(prefix)) {
return bytes.TrimPrefix(line, []byte(prefix))
}
}
return nil
}
// fileBase returns the final path segment (like filepath.Base, kept local to
// avoid importing path/filepath just for this one call).
func fileBase(path string) string {
for i := len(path) - 1; i >= 0; i-- {
if path[i] == '/' || path[i] == '\\' {
return path[i+1:]
}
}
return path
}
+189
View File
@@ -0,0 +1,189 @@
// ingestion/internal/extract/docmark_test.go
package extract
import (
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// mcpToolResult mirrors the shape of docmark's JSON-RPC tools/call response.
type mcpToolResult struct {
JSONRPC string `json:"jsonrpc"`
ID int `json:"id"`
Result *struct {
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
} `json:"content"`
IsError bool `json:"isError"`
} `json:"result,omitempty"`
Error *struct {
Message string `json:"message"`
} `json:"error,omitempty"`
}
// writeMCPResponse mirrors docmark's REAL response framing (empirically
// confirmed against the live server): Content-Type: text/event-stream,
// body is SSE-framed ("event: message\r\ndata: {...}\r\n\r\n"), not bare
// JSON -- inherent to MCP Streamable-HTTP, independent of stateless_http.
func writeMCPResponse(w http.ResponseWriter, body mcpToolResult) {
payload, _ := json.Marshal(body)
w.Header().Set("Content-Type", "text/event-stream")
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte("event: message\r\ndata: "))
_, _ = w.Write(payload)
_, _ = w.Write([]byte("\r\n\r\n"))
}
func TestExtractViaDocmark_Success(t *testing.T) {
var gotAuth, gotAccept string
var gotBody map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotAuth = r.Header.Get("Authorization")
gotAccept = r.Header.Get("Accept")
b, _ := io.ReadAll(r.Body)
_ = json.Unmarshal(b, &gotBody)
writeMCPResponse(w, mcpToolResult{
JSONRPC: "2.0", ID: 1,
Result: &struct {
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
} `json:"content"`
IsError bool `json:"isError"`
}{
Content: []struct {
Type string `json:"type"`
Text string `json:"text"`
}{{Type: "text", Text: "# Converted\n\nhello"}},
},
})
}))
defer srv.Close()
t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
t.Setenv("DOCMARK_BEARER_TOKEN", "test-token-123")
dir := t.TempDir()
path := filepath.Join(dir, "doc.docx")
require.NoError(t, os.WriteFile(path, []byte("fake docx bytes"), 0o644))
got, err := extractViaDocmark(path)
require.NoError(t, err)
assert.Equal(t, "# Converted\n\nhello", got)
assert.Equal(t, "Bearer test-token-123", gotAuth)
assert.Contains(t, gotAccept, "application/json")
params, _ := gotBody["params"].(map[string]any)
require.NotNil(t, params)
assert.Equal(t, "convert_to_markdown", params["name"])
args, _ := params["arguments"].(map[string]any)
require.NotNil(t, args)
assert.Equal(t, "doc.docx", args["filename"])
assert.NotEmpty(t, args["content_base64"])
}
func TestExtractViaDocmark_NotConfigured(t *testing.T) {
t.Setenv("DOCMARK_URL", "")
dir := t.TempDir()
path := filepath.Join(dir, "doc.docx")
require.NoError(t, os.WriteFile(path, []byte("x"), 0o644))
_, err := extractViaDocmark(path)
require.Error(t, err)
assert.Contains(t, err.Error(), "DOCMARK_URL")
}
func TestExtractViaDocmark_ToolErrorSurfacesMessage(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
writeMCPResponse(w, mcpToolResult{
JSONRPC: "2.0", ID: 1,
Result: &struct {
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
} `json:"content"`
IsError bool `json:"isError"`
}{
Content: []struct {
Type string `json:"type"`
Text string `json:"text"`
}{{Type: "text", Text: "unsupported format for 'doc.docx'"}},
IsError: true,
},
})
}))
defer srv.Close()
t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
t.Setenv("DOCMARK_BEARER_TOKEN", "tok")
dir := t.TempDir()
path := filepath.Join(dir, "doc.docx")
require.NoError(t, os.WriteFile(path, []byte("x"), 0o644))
_, err := extractViaDocmark(path)
require.Error(t, err)
assert.Contains(t, err.Error(), "unsupported format")
}
func TestExtractViaDocmark_HTTPErrorSurfaces(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusUnauthorized)
_, _ = w.Write([]byte("unauthorized"))
}))
defer srv.Close()
t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
t.Setenv("DOCMARK_BEARER_TOKEN", "wrong")
dir := t.TempDir()
path := filepath.Join(dir, "doc.docx")
require.NoError(t, os.WriteFile(path, []byte("x"), 0o644))
_, err := extractViaDocmark(path)
require.Error(t, err)
assert.Contains(t, err.Error(), "401")
}
func TestText_RoutesDocxXlsxPptxImagesToDocmark(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
writeMCPResponse(w, mcpToolResult{
JSONRPC: "2.0", ID: 1,
Result: &struct {
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
} `json:"content"`
IsError bool `json:"isError"`
}{
Content: []struct {
Type string `json:"type"`
Text string `json:"text"`
}{{Type: "text", Text: "converted"}},
},
})
}))
defer srv.Close()
t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
t.Setenv("DOCMARK_BEARER_TOKEN", "tok")
for _, ext := range []string{".docx", ".xlsx", ".pptx", ".png", ".jpg", ".jpeg"} {
t.Run(ext, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "f"+ext)
require.NoError(t, os.WriteFile(path, []byte("x"), 0o644))
got, err := Text(path)
require.NoError(t, err)
assert.Equal(t, "converted", got)
})
}
}
+5 -1
View File
@@ -8,7 +8,9 @@ import (
)
// Text reads the file at path and returns its plain-text content.
// Supported extensions: .md, .txt (passthrough), .pdf (via pdftotext).
// Supported extensions: .md, .txt (passthrough), .pdf (via pdftotext),
// .docx/.xlsx/.pptx/.png/.jpg/.jpeg (via docmark, ADR-0013 -- requires
// DOCMARK_URL to be set; see docmark.go).
func Text(path string) (string, error) {
ext := strings.ToLower(fileExt(path))
switch ext {
@@ -20,6 +22,8 @@ func Text(path string) (string, error) {
return string(b), nil
case ".pdf":
return extractPDF(path)
case ".docx", ".xlsx", ".pptx", ".png", ".jpg", ".jpeg":
return extractViaDocmark(path)
default:
return "", fmt.Errorf("unsupported file extension: %s", ext)
}
@@ -0,0 +1,143 @@
// Package frontmatter lints and repairs the YAML frontmatter block of a
// markdown note.
//
// Notes written directly — hand-authored, or promoted from brain/raw/ — do not
// pass through pipeline.BuildPages, so nothing quotes their scalars. A title
// containing an unescaped ": " then produces a block that yaml.Unmarshal
// rejects with "mapping values are not allowed here", and every consumer that
// reads frontmatter silently skips the note.
//
// Fix is deliberately conservative: it only adds quoting, never reorders or
// rewrites, and it refuses to return a document that still does not parse.
package frontmatter
import (
"bytes"
"fmt"
"regexp"
"strings"
"gopkg.in/yaml.v3"
)
const fence = "---"
// Lint reports whether doc's frontmatter block parses as YAML.
//
// A document with no frontmatter block, or with an unterminated one, is not an
// error: there is nothing claiming to be YAML.
func Lint(doc []byte) error {
b, ok := block(doc)
if !ok {
return nil
}
var v any
if err := yaml.Unmarshal(b, &v); err != nil {
return fmt.Errorf("frontmatter: %w", err)
}
return nil
}
// Fix quotes unquoted scalars whose value breaks YAML parsing and returns the
// repaired document.
//
// It returns (doc, false, nil) when the document already parses, and
// (doc, false, err) when quoting cannot rescue it — an unfixable document is
// always returned byte-identical, never half-repaired.
func Fix(doc []byte) ([]byte, bool, error) {
if err := Lint(doc); err == nil {
return doc, false, nil
}
b, ok := block(doc)
if !ok {
// Lint only fails when there is a block, so this is unreachable in
// practice; return the original rather than guess.
return doc, false, nil
}
fixed := quoteScalars(b)
if bytes.Equal(fixed, b) {
return doc, false, fmt.Errorf("frontmatter: not repairable by quoting: %w", Lint(doc))
}
var v any
if err := yaml.Unmarshal(fixed, &v); err != nil {
return doc, false, fmt.Errorf("frontmatter: quoting did not repair the block: %w", err)
}
_, rest, _ := split(doc)
out := []byte(fence + "\n")
out = append(out, fixed...)
out = append(out, []byte(fence)...)
out = append(out, rest...)
return out, true, nil
}
// mappingLine matches "key: value" at any indentation, capturing the prefix so
// it can be re-emitted untouched.
//
// Sequence items ("- value") are deliberately not handled: "- foo: bar" is
// valid YAML, so a document containing one does not reach Fix in the first
// place. See the Lint table for why that gap is left open.
var mappingLine = regexp.MustCompile(`^(\s*[A-Za-z_][\w.-]*:[ \t]+)(\S.*)$`)
// quoteScalars single-quotes every unquoted scalar value in the block that
// contains a colon. Values that are already quoted, or that carry YAML syntax
// this package has no business rewriting, are left alone.
func quoteScalars(block []byte) []byte {
lines := strings.Split(string(block), "\n")
for i, line := range lines {
m := mappingLine.FindStringSubmatch(line)
if m == nil {
continue
}
prefix, value := m[1], m[2]
if !needsQuoting(value) {
continue
}
lines[i] = prefix + "'" + strings.ReplaceAll(value, "'", "''") + "'"
}
return []byte(strings.Join(lines, "\n"))
}
// needsQuoting reports whether a scalar value has to be quoted to load as the
// string it plainly is.
//
// The test is YAML's own parser rather than a pattern: ask whether the value
// stands alone as the right-hand side of a mapping. Heuristics get this wrong
// in both directions — `title: "Foo" — bar` opens with a quote but is not
// quoted, and a value may contain a colon inside quotes and be perfectly fine.
//
// Anything that already parses is left alone, whatever it parses into — a flow
// collection, a number, a bool. It is not broken, and quoting would change its
// type.
func needsQuoting(value string) bool {
if strings.HasPrefix(value, "|") || strings.HasPrefix(value, ">") {
return false // block scalar; its content is not parsed as YAML
}
var probe map[string]any
return yaml.Unmarshal([]byte("v: "+value+"\n"), &probe) != nil
}
// block returns the raw YAML between the opening and closing fences.
func block(doc []byte) ([]byte, bool) {
b, _, ok := split(doc)
return b, ok
}
// split returns the frontmatter block and everything from the closing fence
// onward (the closing fence itself excluded, its trailing newline included).
func split(doc []byte) (block, rest []byte, ok bool) {
s := string(doc)
if !strings.HasPrefix(s, fence+"\n") {
return nil, nil, false
}
body := s[len(fence)+1:]
idx := strings.Index(body, "\n"+fence)
if idx < 0 {
return nil, nil, false
}
return []byte(body[:idx+1]), []byte(body[idx+1+len(fence):]), true
}
@@ -0,0 +1,188 @@
// ingestion/internal/frontmatter/frontmatter_test.go
package frontmatter
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gopkg.in/yaml.v3"
)
func TestLint(t *testing.T) {
tests := []struct {
name string
doc string
wantErr bool
}{
{
name: "no frontmatter block is not an error",
doc: "# Just a heading\n\nSome body text.\n",
},
{
name: "quoted title parses",
doc: "---\ntitle: 'Headless dispatch runs: Bash is denied'\n---\n\nbody\n",
},
{
name: "plain title parses",
doc: "---\ntitle: A simple title\nwing: homelab\n---\n\nbody\n",
},
{
name: "unquoted colon in title does not parse",
doc: "---\ntitle: Headless dispatch runs: Bash is denied — use gitea MCP\nwing: homelab\n---\n\nbody\n",
wantErr: true,
},
{
// Known gap, deliberately not fixed here: this parses, but the
// item becomes map[string]any{"Headless dispatch": "denied"}
// rather than a string. It is a silent type corruption, not a
// parse failure, so repairing it would rewrite documents that
// currently load — out of scope for a lint about broken YAML.
name: "unquoted colon in a list item parses as a mapping, not an error",
doc: "---\ntitle: Fine\naliases:\n - Headless dispatch: denied\n---\n\nbody\n",
},
{
name: "unterminated frontmatter is not treated as frontmatter",
doc: "---\ntitle: no closing fence\n\nbody\n",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := Lint([]byte(tt.doc))
if tt.wantErr {
require.Error(t, err)
return
}
require.NoError(t, err)
})
}
}
func TestFix(t *testing.T) {
tests := []struct {
name string
doc string
wantChanged bool
wantKey string
wantValue string
}{
{
name: "quotes an unquoted title containing a colon",
doc: "---\ntitle: Headless dispatch runs: Bash is denied — use gitea MCP\nwing: homelab\n---\n\nbody\n",
wantChanged: true,
wantKey: "title",
wantValue: "Headless dispatch runs: Bash is denied — use gitea MCP",
},
{
// Parses already (see the Lint table), so Fix must leave it alone
// rather than rewrite a document that loads.
name: "leaves a colon-bearing list item alone because it already parses",
doc: "---\ntitle: Fine\naliases:\n - Headless dispatch: denied\n---\n\nbody\n",
wantChanged: false,
wantKey: "title",
wantValue: "Fine",
},
{
name: "escapes an embedded single quote when quoting",
doc: "---\ntitle: Don't do this: it breaks\n---\n\nbody\n",
wantChanged: true,
wantKey: "title",
wantValue: "Don't do this: it breaks",
},
{
// The value opens with a double quote and then keeps going after
// the closing one, so YAML reads it as a quoted scalar followed by
// junk. Treating "starts with a quote" as "already quoted" leaves
// this broken forever.
name: "quotes a value that only looks quoted",
doc: "---\ntitle: \"Evaluation Driven Generation\" — a TDD-style scaffold\ntags: [a, b]\n---\n\nbody\n",
wantChanged: true,
wantKey: "title",
wantValue: "\"Evaluation Driven Generation\" — a TDD-style scaffold",
},
{
name: "leaves an already-valid document untouched",
doc: "---\ntitle: 'Already quoted: fine'\nwing: homelab\n---\n\nbody\n",
wantChanged: false,
wantKey: "title",
wantValue: "Already quoted: fine",
},
{
name: "leaves a document with no frontmatter untouched",
doc: "# Heading\n\nbody: with a colon\n",
wantChanged: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, changed, err := Fix([]byte(tt.doc))
require.NoError(t, err)
assert.Equal(t, tt.wantChanged, changed)
if !tt.wantChanged {
assert.Equal(t, tt.doc, string(got), "unchanged docs must be returned byte-identical")
}
// Whatever came back must parse — that is the whole point.
require.NoError(t, Lint(got), "Fix must not return a document that still fails to parse")
if tt.wantKey == "" {
return
}
block, ok := block(got)
require.True(t, ok)
var m map[string]any
require.NoError(t, yaml.Unmarshal(block, &m))
assert.Equal(t, tt.wantValue, m[tt.wantKey], "the value must survive quoting unchanged")
})
}
}
func TestFix_LeavesStructuredValuesStructured(t *testing.T) {
// Only the broken scalar may be touched. A flow sequence on a neighbouring
// line still has to load as a sequence afterwards, not as a string.
doc := "---\ntitle: Broken: here\ntags: [prompting, llm]\ncount: 3\n---\n\nbody\n"
got, changed, err := Fix([]byte(doc))
require.NoError(t, err)
require.True(t, changed)
b, ok := block(got)
require.True(t, ok)
var m map[string]any
require.NoError(t, yaml.Unmarshal(b, &m))
assert.Equal(t, "Broken: here", m["title"])
assert.Equal(t, []any{"prompting", "llm"}, m["tags"])
assert.Equal(t, 3, m["count"])
}
func TestFix_PreservesBodyExactly(t *testing.T) {
body := "\nThe body has: colons, 'quotes', and\n---\na horizontal rule.\n"
doc := "---\ntitle: Broken: title\n---" + body
got, changed, err := Fix([]byte(doc))
require.NoError(t, err)
require.True(t, changed)
assert.True(t, len(got) > 0)
_, rest, ok := split([]byte(doc))
require.True(t, ok)
_, gotRest, ok := split(got)
require.True(t, ok)
assert.Equal(t, string(rest), string(gotRest), "the body must not be touched")
}
func TestFix_RefusesToReturnAnUnparseableResult(t *testing.T) {
// A frontmatter block that quoting cannot rescue: the structure itself is
// broken, not a scalar. Fix must return the original and an error rather
// than a half-repaired document.
doc := "---\ntitle: fine\n\tbad: tab-indented mapping\n - stray list item\n---\n\nbody\n"
got, changed, err := Fix([]byte(doc))
require.Error(t, err)
assert.False(t, changed)
assert.Equal(t, doc, string(got), "an unfixable document must come back byte-identical")
}
+84
View File
@@ -0,0 +1,84 @@
// ingestion/internal/frontmatter/scan.go
package frontmatter
import (
"fmt"
"io/fs"
"os"
"path/filepath"
"strings"
"unicode/utf8"
)
// Finding is one markdown file whose frontmatter does not parse.
type Finding struct {
Path string // relative to the scanned root
Err error // why it does not parse, or why the repair was refused
Repaired bool // true only when fix was requested and the file was rewritten
}
// Scan walks root for *.md files whose frontmatter fails to parse.
//
// With fix set, each finding that quoting can rescue is rewritten in place;
// anything quoting cannot rescue is reported and left byte-identical. A file
// that cannot be read or decoded is reported as a finding rather than aborting
// the walk — one poisoned note must not hide the other findings.
func Scan(root string, fix bool) ([]Finding, error) {
var found []Finding
err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() || !strings.EqualFold(filepath.Ext(path), ".md") {
return nil
}
rel, relErr := filepath.Rel(root, path)
if relErr != nil {
rel = path
}
doc, readErr := os.ReadFile(path)
if readErr != nil {
found = append(found, Finding{Path: rel, Err: readErr})
return nil
}
if !utf8.Valid(doc) {
found = append(found, Finding{Path: rel, Err: fmt.Errorf("not valid UTF-8")})
return nil
}
if lintErr := Lint(doc); lintErr == nil {
return nil
}
if !fix {
found = append(found, Finding{Path: rel, Err: Lint(doc)})
return nil
}
repaired, changed, fixErr := Fix(doc)
if fixErr != nil || !changed {
found = append(found, Finding{Path: rel, Err: fixErr})
return nil
}
if writeErr := writeFilePreservingMode(path, repaired); writeErr != nil {
found = append(found, Finding{Path: rel, Err: writeErr})
return nil
}
found = append(found, Finding{Path: rel, Repaired: true})
return nil
})
if err != nil {
return nil, fmt.Errorf("scan %s: %w", root, err)
}
return found, nil
}
func writeFilePreservingMode(path string, content []byte) error {
mode := fs.FileMode(0o644)
if info, err := os.Stat(path); err == nil {
mode = info.Mode().Perm()
}
return os.WriteFile(path, content, mode)
}
+103
View File
@@ -0,0 +1,103 @@
// ingestion/internal/frontmatter/scan_test.go
package frontmatter
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func writeTree(t *testing.T, files map[string]string) string {
t.Helper()
root := t.TempDir()
for name, content := range files {
path := filepath.Join(root, name)
require.NoError(t, os.MkdirAll(filepath.Dir(path), 0o755))
require.NoError(t, os.WriteFile(path, []byte(content), 0o644))
}
return root
}
const (
brokenDoc = "---\ntitle: Broken: title here\nwing: homelab\n---\n\nbody\n"
validDoc = "---\ntitle: 'Fine: quoted'\n---\n\nbody\n"
)
func TestScan_ReportsOnlyBrokenFiles(t *testing.T) {
root := writeTree(t, map[string]string{
"wiki/a/broken.md": brokenDoc,
"wiki/a/valid.md": validDoc,
"wiki/b/plain.md": "# no frontmatter\n\nbody: with colon\n",
"wiki/a/notes.txt": brokenDoc, // not markdown — must be ignored
})
found, err := Scan(root, false)
require.NoError(t, err)
require.Len(t, found, 1)
assert.Equal(t, "wiki/a/broken.md", found[0].Path)
assert.False(t, found[0].Repaired)
assert.Error(t, found[0].Err)
// Dry run must not touch the file.
got, err := os.ReadFile(filepath.Join(root, "wiki/a/broken.md"))
require.NoError(t, err)
assert.Equal(t, brokenDoc, string(got))
}
func TestScan_FixWritesRepairedFilesAndLeavesTheRestAlone(t *testing.T) {
root := writeTree(t, map[string]string{
"wiki/a/broken.md": brokenDoc,
"wiki/a/valid.md": validDoc,
})
found, err := Scan(root, true)
require.NoError(t, err)
require.Len(t, found, 1)
assert.True(t, found[0].Repaired)
assert.NoError(t, found[0].Err)
repaired, err := os.ReadFile(filepath.Join(root, "wiki/a/broken.md"))
require.NoError(t, err)
require.NoError(t, Lint(repaired), "the repaired file must parse")
assert.Contains(t, string(repaired), "title: 'Broken: title here'")
assert.Contains(t, string(repaired), "wing: homelab", "untouched keys must survive")
untouched, err := os.ReadFile(filepath.Join(root, "wiki/a/valid.md"))
require.NoError(t, err)
assert.Equal(t, validDoc, string(untouched))
// Second pass finds nothing — the repair is idempotent and complete.
again, err := Scan(root, true)
require.NoError(t, err)
assert.Empty(t, again)
}
func TestScan_ReportsUnfixableWithoutWriting(t *testing.T) {
unfixable := "---\ntitle: fine\n\tbad: tab-indented\n - stray\n---\n\nbody\n"
root := writeTree(t, map[string]string{"wiki/x.md": unfixable})
found, err := Scan(root, true)
require.NoError(t, err)
require.Len(t, found, 1)
assert.False(t, found[0].Repaired)
assert.Error(t, found[0].Err)
got, err := os.ReadFile(filepath.Join(root, "wiki/x.md"))
require.NoError(t, err)
assert.Equal(t, unfixable, string(got), "an unfixable file must be left exactly as it was")
}
func TestScan_ReportsUndecodableFilesInsteadOfFailingTheRun(t *testing.T) {
root := writeTree(t, map[string]string{"wiki/ok.md": validDoc})
require.NoError(t, os.WriteFile(filepath.Join(root, "wiki/bad-bytes.md"),
[]byte{'-', '-', '-', '\n', 0xff, 0xfe, '\n', '-', '-', '-', '\n'}, 0o644))
found, err := Scan(root, false)
require.NoError(t, err, "one unreadable file must not abort the whole scan")
require.Len(t, found, 1)
assert.Equal(t, "wiki/bad-bytes.md", found[0].Path)
}
-7
View File
@@ -105,13 +105,6 @@ func (s *Server) tools() []map[string]any {
"target": str("path of target note (must be in a different wing)"),
}),
},
{
"name": "brain_index",
"description": "Regenerate _index.md (Map of Content) for one or all wings under brain/wiki/. Auto-called after brain_write with wing+hall.",
"inputSchema": schema([]string{}, map[string]any{
"wing": str("optional wing to index; if absent, rebuilds every wing"),
}),
},
{
"name": "brain_ingest_raw",
"description": "Ingest pre-structured pages into the brain wiki, bypassing the LLM extraction step.",
+76
View File
@@ -0,0 +1,76 @@
package mcp
import (
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
)
// pendingNudgeThreshold is how old a raw/ note has to be before brain_context
// mentions the queue. Two weeks: long enough that a note written this sprint
// does not nag, short enough that four items cannot sit unnoticed for three
// months again (#82).
const pendingNudgeThreshold = 14 * 24 * time.Hour
// PendingNudge is the in-session reminder that the promote/reject queue has
// items waiting. It is advisory: a count, an age, and how to look.
type PendingNudge struct {
Pending int `json:"pending"` // everything in the queue
Aging int `json:"aging"` // those past the threshold
Oldest string `json:"oldest"` // filename of the oldest pending note
OldestDays int `json:"oldest_days"` // its age, in whole days
Hint string `json:"hint"`
}
// pendingNudge summarises the raw/ review queue, or returns nil when there is
// nothing worth saying.
//
// The trigger is the missing half of the queue (#82): brain_pending has always
// existed, and four real items still sat unreviewed from 2026-04-23 to
// 2026-07-15 because nothing ever prompted anyone to call it. The system
// should be what remembers the queue exists.
//
// Threshold-gated on purpose. A notice on every context call is one people
// learn to skip, which reproduces the original failure with extra steps.
//
// It never promotes, rejects, or reorders anything. Age is a reason to ask a
// human, not a verdict — an old note may be exactly right to promote once
// someone finally looks at it.
func pendingNudge(brainDir string, now time.Time, threshold time.Duration) *PendingNudge {
pending, err := api.ListPending(brainDir)
if err != nil || len(pending) == 0 {
// brain_context is a hot path; a broken queue degrades to silence
// rather than failing the call that asked for project context.
return nil
}
var aging int
var oldestName string
var oldestAge time.Duration
for _, p := range pending {
created, perr := time.Parse(time.RFC3339, p.CreatedAt)
if perr != nil {
continue // an unparseable timestamp is not evidence of age
}
age := now.Sub(created)
if age >= threshold {
aging++
}
if age > oldestAge {
oldestAge, oldestName = age, p.Filename
}
}
if aging == 0 {
return nil
}
return &PendingNudge{
Pending: len(pending),
Aging: aging,
Oldest: oldestName,
OldestDays: int(oldestAge.Hours() / 24),
Hint: "call brain_pending to review; nothing is promoted or rejected automatically",
}
}
@@ -0,0 +1,115 @@
package mcp
import (
"context"
"encoding/json"
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// writePending drops a raw/ note whose created_at is age old.
func writePending(t *testing.T, brainDir, name string, age time.Duration, now time.Time) {
t.Helper()
raw := filepath.Join(brainDir, "raw")
require.NoError(t, os.MkdirAll(raw, 0o755))
body := "---\ncreated_at: " + now.Add(-age).UTC().Format(time.RFC3339) + "\n---\n\nnote body\n"
require.NoError(t, os.WriteFile(filepath.Join(raw, name), []byte(body), 0o644))
}
func TestPendingNudge_SilentWhenNothingIsPending(t *testing.T) {
// No queue, no nudge. A notice that fires every session is one people
// learn to skip, which is how the queue got ignored for three months in
// the first place (#82).
got := pendingNudge(t.TempDir(), time.Now(), 14*24*time.Hour)
assert.Nil(t, got)
}
func TestPendingNudge_SilentWhileEverythingIsBelowTheThreshold(t *testing.T) {
now := time.Now()
dir := t.TempDir()
writePending(t, dir, "fresh.md", 2*24*time.Hour, now)
got := pendingNudge(dir, now, 14*24*time.Hour)
assert.Nil(t, got, "a two-day-old note is not aging")
}
func TestPendingNudge_ReportsCountAndOldestAgeOnceSomethingAges(t *testing.T) {
now := time.Now()
dir := t.TempDir()
writePending(t, dir, "old.md", 100*24*time.Hour, now)
writePending(t, dir, "older.md", 120*24*time.Hour, now)
writePending(t, dir, "fresh.md", 1*24*time.Hour, now)
got := pendingNudge(dir, now, 14*24*time.Hour)
require.NotNil(t, got)
// All three are pending; the threshold decides whether to speak, not
// what to count. Someone opening the queue sees the whole queue.
assert.Equal(t, 3, got.Pending)
assert.Equal(t, 2, got.Aging)
assert.Equal(t, 120, got.OldestDays)
assert.Equal(t, "older.md", got.Oldest)
}
func TestPendingNudge_NeverDecidesAnything(t *testing.T) {
// Aging is a trigger to ask a human, never a verdict. The nudge must
// carry no action and must leave the queue exactly as it found it.
now := time.Now()
dir := t.TempDir()
writePending(t, dir, "ancient.md", 365*24*time.Hour, now)
before, err := os.ReadDir(filepath.Join(dir, "raw"))
require.NoError(t, err)
got := pendingNudge(dir, now, 14*24*time.Hour)
require.NotNil(t, got)
after, err := os.ReadDir(filepath.Join(dir, "raw"))
require.NoError(t, err)
assert.Len(t, after, len(before), "the nudge must not promote or reject anything")
assert.Contains(t, got.Hint, "brain_pending", "the nudge should say how to look, not act")
}
func TestPendingNudge_SurvivesAnUnreadableQueue(t *testing.T) {
// brain_context is a hot path; a broken raw/ dir must degrade to silence
// rather than failing the whole context call.
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "raw"), []byte("not a directory"), 0o644))
assert.NotPanics(t, func() {
assert.Nil(t, pendingNudge(dir, time.Now(), 14*24*time.Hour))
})
}
func TestBrainContext_CarriesTheNudgeWithoutFailingWhenTheQueueIsEmpty(t *testing.T) {
// The wiring, not the summary: a nudge that never reaches a caller is the
// same as no nudge, and brain_context must still work with no queue.
dir := t.TempDir()
srv := NewServer(dir, nil, nil, nil)
raw, err := srv.brainContext(context.Background(),
json.RawMessage(`{"project_root":"/home/mathias/dev/AGENTS/hyperguild"}`))
require.NoError(t, err)
var empty map[string]any
require.NoError(t, json.Unmarshal(raw, &empty))
assert.NotContains(t, empty, "pending_review", "silence when nothing is aging")
writePending(t, dir, "ancient.md", 200*24*time.Hour, time.Now())
raw, err = srv.brainContext(context.Background(),
json.RawMessage(`{"project_root":"/home/mathias/dev/AGENTS/hyperguild"}`))
require.NoError(t, err)
var withQueue map[string]any
require.NoError(t, json.Unmarshal(raw, &withQueue))
nudge, ok := withQueue["pending_review"].(map[string]any)
require.True(t, ok, "the nudge must reach the caller: %v", withQueue)
assert.Equal(t, float64(1), nudge["aging"])
assert.Equal(t, "ancient.md", nudge["oldest"])
}
+6 -1
View File
@@ -1,9 +1,14 @@
// Package mcp implements an MCP HTTP handler for the ingestion service.
// Exposed tools: brain_query, brain_write, brain_update, brain_get,
// brain_pending, brain_promote, brain_index, brain_tunnel, brain_ingest,
// brain_pending, brain_promote, brain_tunnel, brain_ingest,
// brain_ingest_raw, brain_answer, brain_classify, brain_graph,
// brain_context, session_log, and capture (the #55 relay tool, registered
// only when WithCapture is set).
//
// brain_index dispatches but is deliberately absent from tools/list: it is
// auto-called after every write, so surfacing it as a caller verb is
// cognitive load with no decision behind it (ADR-0012 rule 4, #78). It stays
// callable for maintenance — de-surfacing hides the verb, not the operation.
package mcp
import (
+25 -1
View File
@@ -59,11 +59,35 @@ func TestServerToolsList(t *testing.T) {
assert.ElementsMatch(t, []string{
"brain_query", "brain_write", "brain_update", "brain_get",
"brain_pending", "brain_promote",
"brain_index", "brain_tunnel",
"brain_tunnel",
"brain_ingest_raw", "brain_ingest",
"brain_answer", "brain_classify", "brain_graph", "brain_context",
"session_log",
}, names)
// brain_index is auto-called after every brain_write, so surfacing it as
// a caller verb is cognitive load with no decision behind it (ADR-0012
// rule 4, #78). The capability stays reachable — see the admin test below.
assert.NotContains(t, names, "brain_index")
}
func TestBrainIndexStaysCallableForAdminUseAfterDeSurfacing(t *testing.T) {
// De-surfacing hides the verb from the model-facing list; it must not
// remove the operation. A maintenance caller still needs to rebuild a
// Map of Content by hand.
srv := mcp.NewServer(t.TempDir(), nil, nil, nil)
req := httptest.NewRequest(http.MethodPost, "/mcp", body(t, map[string]any{
"jsonrpc": "2.0", "id": 3, "method": "tools/call",
"params": map[string]any{"name": "brain_index", "arguments": map[string]any{}},
}))
rr := httptest.NewRecorder()
srv.ServeHTTP(rr, req)
require.Equal(t, http.StatusOK, rr.Code)
var resp map[string]any
require.NoError(t, json.Unmarshal(rr.Body.Bytes(), &resp))
assert.NotContains(t, resp, "error", "brain_index must still dispatch: %v", resp["error"])
}
func TestServerNotificationGetsNoBody(t *testing.T) {
+1 -5
View File
@@ -58,17 +58,13 @@ func captureToolDescriptor() map[string]any {
},
"insights": map[string]any{"type": "array", "items": insightItem},
"tickets": map[string]any{"type": "array", "items": ticketItem},
"summary": map[string]any{"type": "object", "properties": map[string]any{
"title": str("summary title"), "body": str("summary body"),
"repos_touched": map[string]any{"type": "array", "items": map[string]any{"type": "string"}},
}},
"dry_run": map[string]any{"type": "boolean", "description": "validate + return the would-be receipt, write nothing"},
},
}
b, _ := json.Marshal(schema)
return map[string]any{
"name": "capture",
"description": "Persist a session's value uniformly: insights → brain (write or supersede), action items → Gitea tickets, optional summary → ai-sessions. The relay door for MCP-native harnesses. Origin is server-derived from your authenticated identity; confidential captures through a us-nexus surface are refused (I1). Returns a partial-aware receipt.",
"description": "Persist a session's value uniformly: insights → brain (write or supersede), action items → Gitea tickets. The relay door for MCP-native harnesses. Origin is server-derived from your authenticated identity; confidential captures through a us-nexus surface are refused (I1). Returns a partial-aware receipt.",
"inputSchema": json.RawMessage(b),
}
}
+1 -1
View File
@@ -49,7 +49,7 @@ func captureServer(t *testing.T, validator capturehttp.Validator, sovereign []st
brainDir := t.TempDir()
cfg, err := classification.Load(brainDir)
require.NoError(t, err)
svc := capture.NewService(brainstore.New(brainDir), capFakeTracker{}, nil, cfg, audit.NewSlogSink(nil))
svc := capture.NewService(brainstore.New(brainDir), capFakeTracker{}, cfg, audit.NewSlogSink(nil))
srv := mcp.NewServer(brainDir, nil, nil, nil)
srv.WithCapture(svc, validator, capStaticTok, "local-cli", capturehttp.NewOriginResolver(sovereign))
return srv, brainDir
+12 -9
View File
@@ -8,8 +8,10 @@ import (
"path/filepath"
"sort"
"strings"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/search"
"github.com/mathiasbq/hyperguild/ingestion/internal/textutil"
)
// brainContextArgs is the input shape of brain_context. project_root is
@@ -155,17 +157,21 @@ func (s *Server) brainContext(ctx context.Context, args json.RawMessage) (json.R
entries = entries[:limit]
}
return json.Marshal(map[string]any{"entries": entries})
out := map[string]any{"entries": entries}
// The queue nudge rides along with a call the session already makes, so
// noticing an aging note costs nobody a decision to go looking (#82).
// Advisory only, and absent entirely when there is nothing to say.
if n := pendingNudge(s.brainDir, time.Now(), pendingNudgeThreshold); n != nil {
out["pending_review"] = n
}
return json.Marshal(out)
}
// truncateExcerpt clamps an already-stripped excerpt to maxLen characters
// without re-running the frontmatter parser. The ellipsis suffix matches
// the convention used in search.excerpt.
func truncateExcerpt(s string, maxLen int) string {
if len(s) <= maxLen {
return s
}
return s[:maxLen] + "…"
return textutil.Truncate(s, maxLen)
}
// readExcerpt loads a doc relative to brainDir, strips its frontmatter,
@@ -185,10 +191,7 @@ func readExcerpt(brainDir, relPath string, maxLen int) string {
if len(parts) == 3 {
body = strings.TrimSpace(parts[2])
}
if len(body) > maxLen {
return body[:maxLen] + "…"
}
return body
return textutil.Truncate(body, maxLen)
}
// max1 returns the maximum of n and 1, used to guard against divide-by-zero
+9
View File
@@ -76,6 +76,15 @@ func buildFrontmatter(rp RawPage, date string) string {
}
fmt.Fprintf(&sb, "date_ingested: %s\n", date)
fmt.Fprintf(&sb, "last_updated: %s\n", date)
if rp.Source != "" {
fmt.Fprintf(&sb, "source: %s\n", yamlScalar(rp.Source))
}
if rp.Author != "" {
fmt.Fprintf(&sb, "author: %s\n", yamlScalar(rp.Author))
}
if rp.Published != "" {
fmt.Fprintf(&sb, "published: %s\n", yamlScalar(rp.Published))
}
case "concept":
if rp.Domain != "" {
fmt.Fprintf(&sb, "domain: %s\n", yamlScalar(rp.Domain))
+46
View File
@@ -154,6 +154,52 @@ func TestBuildPages_EntityNoSubtype(t *testing.T) {
assert.Contains(t, pages[0].Content, "title: 'Basecamp'")
}
func TestBuildPages_SourcePageCarriesSourceAuthorPublished(t *testing.T) {
raw := []RawPage{
{
Title: "Ornith",
Type: "source",
Subtype: "article",
Content: "## Summary\n\nAn agentic coding model.\n",
Source: "https://example.com/ornith",
Author: "Jane Doe",
Published: "2026-07-20",
},
}
pages, warnings := BuildPages(raw, "ornith", "2026-07-26")
require.Len(t, pages, 1)
assert.Empty(t, warnings)
p := pages[0]
assert.Contains(t, p.Content, "source: 'https://example.com/ornith'")
assert.Contains(t, p.Content, "author: 'Jane Doe'")
assert.Contains(t, p.Content, "published: '2026-07-20'")
}
func TestBuildPages_SourcePageOmitsSourceAuthorPublishedWhenEmpty(t *testing.T) {
raw := []RawPage{
{Title: "Shape Up", Type: "source", Subtype: "book", Content: "## Summary\n\nA book.\n"},
}
pages, _ := BuildPages(raw, "shape-up", "2026-04-23")
require.Len(t, pages, 1)
assert.NotContains(t, pages[0].Content, "source:")
assert.NotContains(t, pages[0].Content, "author:")
assert.NotContains(t, pages[0].Content, "published:")
}
func TestBuildPages_ConceptPageIgnoresSourceAuthorPublished(t *testing.T) {
// source/author/published are source-note-only metadata; a concept page
// shouldn't carry them even if somehow set on the RawPage.
raw := []RawPage{
{Title: "Betting", Type: "concept", Content: "## Definition\n\nFoo.\n", Source: "x", Author: "y", Published: "z"},
}
pages, _ := BuildPages(raw, "src", "2026-04-23")
require.Len(t, pages, 1)
assert.NotContains(t, pages[0].Content, "source:")
assert.NotContains(t, pages[0].Content, "author:")
assert.NotContains(t, pages[0].Content, "published:")
}
func TestBuildPages_EmptyTitleSkippedWithWarning(t *testing.T) {
raw := []RawPage{
{Title: "", Type: "concept", Content: "## Definition\n\nFoo.\n"},
+25 -5
View File
@@ -51,6 +51,21 @@ func buildTitleMap(pages []wiki.Page, inventory map[wiki.PageType][]wiki.Entry)
return m
}
// pathStylePrefixes are known root prefixes the LLM extraction step
// sometimes bakes into a wikilink target instead of emitting a clean
// wing/hall/slug path (or bare title). Stripping them repairs the link
// in place — see hyperguild#87.
var pathStylePrefixes = []string{"wing:", "wiki/"}
func stripPathStylePrefix(displayName string) (string, bool) {
for _, prefix := range pathStylePrefixes {
if stripped, ok := strings.CutPrefix(displayName, prefix); ok {
return stripped, true
}
}
return displayName, false
}
func canonicalizeContent(content string, titleToSlug map[string]string) (string, []string) {
var warnings []string
result := plainLinkRE.ReplaceAllStringFunc(content, func(match string) string {
@@ -59,12 +74,17 @@ func canonicalizeContent(content string, titleToSlug map[string]string) (string,
return match
}
displayName := sub[1]
slug, ok := titleToSlug[strings.ToLower(displayName)]
if !ok {
warnings = append(warnings, fmt.Sprintf("unknown wikilink: [[%s]]", displayName))
return match
if slug, ok := titleToSlug[strings.ToLower(displayName)]; ok {
return "[[" + slug + "|" + displayName + "]]"
}
return "[[" + slug + "|" + displayName + "]]"
if stripped, hadPrefix := stripPathStylePrefix(displayName); hadPrefix {
return "[[" + stripped + "]]"
}
warnings = append(warnings, fmt.Sprintf("unknown wikilink: [[%s]]", displayName))
return match
})
return result, warnings
}
+47
View File
@@ -102,6 +102,53 @@ func TestCanonicalizeLinks_CurrentBatchPagesResolved(t *testing.T) {
assert.Contains(t, got[0].Content, "[[betting|Betting]]")
}
func TestCanonicalizeLinks_StripsWingColonPrefix(t *testing.T) {
pages := []wiki.Page{
{
Path: "wiki/homelab/failures/act-runner-host-mode-container-needs-node-and-libatomic.md",
Content: "---\ntitle: 'act_runner host-mode'\n---\n\nSee [[wing:homelab/failures/rootless-buildah-act-runner-run-containers-denied]].\n",
},
}
got, warnings := CanonicalizeLinks(pages, map[wiki.PageType][]wiki.Entry{})
require.Len(t, got, 1)
assert.Empty(t, warnings)
assert.Contains(t, got[0].Content, "[[homelab/failures/rootless-buildah-act-runner-run-containers-denied]]")
assert.NotContains(t, got[0].Content, "wing:")
}
func TestCanonicalizeLinks_StripsWikiSlashPrefix(t *testing.T) {
pages := []wiki.Page{
{
Path: "wiki/agentsquad/hypotheses/council-consolidation-standalone-deliberation-service.md",
Content: "---\ntitle: 'council consolidation'\n---\n\nSee [[wiki/agentsquad/decisions/autoresearch-council-sibling-pipe]].\n",
},
}
got, warnings := CanonicalizeLinks(pages, map[wiki.PageType][]wiki.Entry{})
require.Len(t, got, 1)
assert.Empty(t, warnings)
assert.Contains(t, got[0].Content, "[[agentsquad/decisions/autoresearch-council-sibling-pipe]]")
assert.NotContains(t, got[0].Content, "wiki/agentsquad/decisions/autoresearch-council-sibling-pipe]]\n\n") // no leftover wiki/ prefix
assert.NotContains(t, got[0].Content, "[[wiki/")
}
func TestCanonicalizeLinks_TitleLookupStillTakesPriorityOverPrefixStrip(t *testing.T) {
// A plain link that resolves via the title map must still use the
// normal slug|Display form, not fall through to prefix-strip repair.
pages := []wiki.Page{
{
Path: "wiki/sources/shape-up.md",
Content: "---\ntitle: 'Shape Up'\n---\n\nSee [[Betting]].\n",
},
}
inventory := map[wiki.PageType][]wiki.Entry{
wiki.PageTypeConcept: {{Slug: "betting", Title: "Betting"}},
}
got, warnings := CanonicalizeLinks(pages, inventory)
require.Len(t, got, 1)
assert.Empty(t, warnings)
assert.Contains(t, got[0].Content, "[[betting|Betting]]")
}
func TestCanonicalizeLinks_MultipleLinksInOnePage(t *testing.T) {
pages := []wiki.Page{
{
+62
View File
@@ -15,6 +15,14 @@ type RawPage struct {
Subtype string `json:"subtype"` // entity: person|company|tool|model|framework|technology; source: article|pdf|book|video|note|project
Domain string `json:"domain"`
Content string `json:"content"` // Markdown body only — no frontmatter
// Source, Author, Published are deterministic passthrough from the raw
// ingested content's own frontmatter (see parseContentFrontmatter) — never
// set by the LLM. json:"-" keeps them immune to same-named keys the LLM
// might emit. Only meaningful for Type == "source".
Source string `json:"-"`
Author string `json:"-"`
Published string `json:"-"`
}
// ParseRawPages parses LLM output as a JSON array of RawPage objects.
@@ -98,6 +106,60 @@ func repairJSON(s string) string {
return b.String()
}
// sourceMeta is source/author/published pulled from the raw ingested
// content's own frontmatter — deterministic passthrough, never LLM output.
type sourceMeta struct {
Source string
Author string
Published string
}
// parseContentFrontmatter extracts source/author/published from a leading
// "---\n...\n---" YAML block in raw ingested content. Only these three flat
// scalar keys are recognised; anything else in the block is ignored. Returns
// a zero-value sourceMeta if content has no frontmatter block.
func parseContentFrontmatter(content string) sourceMeta {
var meta sourceMeta
if !strings.HasPrefix(content, "---\n") && !strings.HasPrefix(content, "---\r\n") {
return meta
}
lines := strings.Split(content, "\n")
for _, line := range lines[1:] {
if strings.TrimSpace(line) == "---" {
break
}
key, val, ok := strings.Cut(line, ":")
if !ok {
continue
}
key = strings.TrimSpace(key)
val = strings.Trim(strings.TrimSpace(val), `"'`)
switch key {
case "source":
meta.Source = val
case "author":
meta.Author = val
case "published":
meta.Published = val
}
}
return meta
}
// applySourceMeta deterministically overwrites Source/Author/Published on
// every "source"-type page with meta — the LLM never controls these fields.
func applySourceMeta(pages []RawPage, meta sourceMeta) {
for i := range pages {
if pages[i].Type != "source" {
continue
}
pages[i].Source = meta.Source
pages[i].Author = meta.Author
pages[i].Published = meta.Published
}
}
func stripFences(s string) string {
for _, prefix := range []string{"```json\n", "```json\r\n", "```\n", "```\r\n"} {
if strings.HasPrefix(s, prefix) {
+2
View File
@@ -59,6 +59,8 @@ func Run(ctx context.Context, cfg Config, brainDir, content, source string, dryR
allWarnings = append(allWarnings, warnings...)
}
applySourceMeta(allRaw, parseContentFrontmatter(content))
return buildAndWrite(allRaw, sourceSlug, date, brainDir, source, inventory, allWarnings, dryRun)
}
@@ -130,6 +130,40 @@ func TestRun_MergesDuplicatePaths(t *testing.T) {
assert.Contains(t, string(content), "[[Baz]]")
}
func TestRun_ThreadsSourceAuthorPublishedFromContentFrontmatter(t *testing.T) {
brainDir := t.TempDir()
for _, sub := range []string{"wiki/concepts", "wiki/entities", "wiki/sources"} {
require.NoError(t, os.MkdirAll(filepath.Join(brainDir, sub), 0o755))
}
llmResponse := mustJSON([]RawPage{{
Title: "Ornith",
Type: "source",
Subtype: "article",
Content: "## Summary\n\nAn agentic coding model.\n",
}})
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_ = json.NewEncoder(w).Encode(map[string]any{
"choices": []map[string]any{{"message": map[string]any{"content": llmResponse}}},
})
}))
defer srv.Close()
cfg := Config{Complete: llm.New(srv.URL, "", "m", 30*time.Second).Complete}
rawContent := "---\nsource: https://example.com/ornith\nauthor: Jane Doe\npublished: 2026-07-20\n---\n\nAn agentic coding model that runs on your laptop.\n"
result, err := Run(context.Background(), cfg, brainDir, rawContent, "ornith", false)
require.NoError(t, err)
require.Len(t, result.Pages, 1)
content, err := os.ReadFile(filepath.Join(brainDir, "wiki", "sources", "ornith.md"))
require.NoError(t, err)
assert.Contains(t, string(content), "source: 'https://example.com/ornith'")
assert.Contains(t, string(content), "author: 'Jane Doe'")
assert.Contains(t, string(content), "published: '2026-07-20'")
}
func mustJSON(v any) string {
b, err := json.Marshal(v)
if err != nil {
+2 -4
View File
@@ -12,6 +12,7 @@ import (
"strings"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
"github.com/mathiasbq/hyperguild/ingestion/internal/textutil"
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
)
@@ -427,8 +428,5 @@ func excerpt(content string, maxLen int) string {
if len(parts) == 3 {
body = strings.TrimSpace(parts[2])
}
if len(body) > maxLen {
return body[:maxLen] + "…"
}
return body
return textutil.Truncate(body, maxLen)
}
@@ -0,0 +1,75 @@
// Package startupretry retries a startup dependency that may not be serving
// yet, instead of exiting on its first failure.
//
// It exists for the cold-boot ordering race (#83): on a cluster restart the
// ingestion pod can reach OIDC discovery before dex is answering, gets a 502
// from ingress, and dies — after having already connected postgres, backfilled
// the graph and ingested session batches, all of which the restart redoes.
// Nothing is misconfigured in that scenario; the pod simply started first.
//
// The budget is bounded on purpose. Retrying forever would turn a genuine
// misconfiguration (wrong issuer URL, bad audience) into a process that looks
// healthy and never serves, which is harder to diagnose than a crash loop.
package startupretry
import (
"context"
"fmt"
"time"
)
// Policy bounds the retry: at most Attempts calls, with the delay doubling
// from Base and capped at Max.
type Policy struct {
Attempts int // total calls, including the first; < 1 means 1
Base time.Duration // delay before the second attempt
Max time.Duration // ceiling for the doubling delay
}
// Do calls fn until it succeeds, the attempt budget is spent, or ctx ends.
//
// The returned error wraps the last failure, so callers keep the diagnosis
// they would have had without the retry.
func Do[T any](ctx context.Context, p Policy, fn func(context.Context) (T, error)) (T, error) {
var zero T
attempts := p.Attempts
if attempts < 1 {
attempts = 1
}
delay := p.Base
var lastErr error
for i := range attempts {
if err := ctx.Err(); err != nil {
return zero, fmt.Errorf("startupretry: %w", err)
}
v, err := fn(ctx)
if err == nil {
return v, nil
}
lastErr = err
if i == attempts-1 {
break
}
if delay <= 0 {
delay = time.Second
}
if p.Max > 0 && delay > p.Max {
delay = p.Max
}
select {
case <-ctx.Done():
return zero, fmt.Errorf("startupretry: %w", ctx.Err())
case <-time.After(delay):
}
delay *= 2
}
return zero, fmt.Errorf("startupretry: gave up after %d attempts: %w", attempts, lastErr)
}
@@ -0,0 +1,144 @@
// ingestion/internal/startupretry/startupretry_test.go
package startupretry
import (
"context"
"errors"
"testing"
"time"
)
func fastPolicy(attempts int) Policy {
return Policy{Attempts: attempts, Base: time.Millisecond, Max: 2 * time.Millisecond}
}
func TestDo_ReturnsImmediatelyOnFirstSuccess(t *testing.T) {
calls := 0
got, err := Do(context.Background(), fastPolicy(5), func(context.Context) (string, error) {
calls++
return "ready", nil
})
if err != nil {
t.Fatalf("Do: %v", err)
}
if got != "ready" {
t.Errorf("got %q, want %q", got, "ready")
}
if calls != 1 {
t.Errorf("calls = %d, want 1", calls)
}
}
func TestDo_SurvivesTransientFailuresThenSucceeds(t *testing.T) {
// The observed cold-boot race: dex is not serving yet, ingress answers
// 502 for the first few attempts, then discovery works.
calls := 0
got, err := Do(context.Background(), fastPolicy(5), func(context.Context) (string, error) {
calls++
if calls < 4 {
return "", errors.New("oidc discovery: status 502")
}
return "ready", nil
})
if err != nil {
t.Fatalf("Do returned %v — a transient 502 must not be fatal", err)
}
if got != "ready" {
t.Errorf("got %q, want %q", got, "ready")
}
if calls != 4 {
t.Errorf("calls = %d, want 4", calls)
}
}
func TestDo_GivesUpAfterTheAttemptBudgetAndReportsTheLastError(t *testing.T) {
// A genuinely broken issuer must still fail — retrying forever would turn
// a config error into a silent hang, which is worse than crash-looping.
last := errors.New("oidc discovery: status 404")
calls := 0
_, err := Do(context.Background(), fastPolicy(3), func(context.Context) (string, error) {
calls++
return "", last
})
if !errors.Is(err, last) {
t.Fatalf("err = %v, want it to wrap the last failure", err)
}
if calls != 3 {
t.Errorf("calls = %d, want 3 (the attempt budget)", calls)
}
}
func TestDo_StopsPromptlyWhenTheContextIsCancelled(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
calls := 0
done := make(chan struct{})
var err error
go func() {
defer close(done)
_, err = Do(ctx, Policy{Attempts: 100, Base: 50 * time.Millisecond, Max: time.Second},
func(context.Context) (string, error) {
calls++
if calls == 1 {
cancel()
}
return "", errors.New("still down")
})
}()
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("Do did not return after context cancellation")
}
if !errors.Is(err, context.Canceled) {
t.Errorf("err = %v, want context.Canceled", err)
}
}
func TestDo_BacksOffAndIsCappedByMax(t *testing.T) {
var delays []time.Duration
p := Policy{Attempts: 5, Base: 10 * time.Millisecond, Max: 25 * time.Millisecond}
last := time.Now()
_, _ = Do(context.Background(), p, func(context.Context) (string, error) {
now := time.Now()
delays = append(delays, now.Sub(last))
last = now
return "", errors.New("down")
})
// delays[0] is the first call (no wait); the rest are the sleeps.
if len(delays) != 5 {
t.Fatalf("attempts = %d, want 5", len(delays))
}
if delays[1] >= delays[2] {
t.Errorf("backoff is not growing: %v then %v", delays[1], delays[2])
}
for i, d := range delays[1:] {
if d > 2*p.Max {
t.Errorf("delay %d = %v, exceeds the Max cap %v with slack", i+1, d, p.Max)
}
}
}
func TestPolicy_ZeroValueStillMakesOneAttempt(t *testing.T) {
// A caller that forgets to configure the policy must not silently get a
// no-op that returns a nil validator and a nil error.
calls := 0
_, err := Do(context.Background(), Policy{}, func(context.Context) (string, error) {
calls++
return "", errors.New("down")
})
if calls != 1 {
t.Errorf("calls = %d, want 1", calls)
}
if err == nil {
t.Error("err = nil, want the failure surfaced")
}
}
+45
View File
@@ -0,0 +1,45 @@
// Package textutil holds small text helpers shared across the ingestion
// service.
package textutil
import "unicode/utf8"
// Ellipsis marks a value that was cut short.
const Ellipsis = "…"
// Truncate shortens s to at most limit CHARACTERS, appending an ellipsis when
// anything was dropped.
//
// It exists because four call sites did `s[:limit] + "…"` (#93). Both halves
// of that are byte operations: len() counts bytes and a slice cuts bytes, so
// on Swedish text — where å ä ö are two bytes and an em dash is three — the
// cut lands mid-character and leaves an orphaned lead byte. The result is
// invalid UTF-8.
//
// That failed loudly in exactly one place, wiki/index.md, which is written
// raw to disk and then rejected by every strict decoder. The other three fed
// MCP responses, where encoding/json silently substitutes U+FFFD — so those
// had been quietly eating the last character of truncated excerpts with no
// error anywhere.
//
// The marker is appended only when content was actually dropped: an ellipsis
// on complete text is a lie about the content.
func Truncate(s string, limit int) string {
if limit <= 0 {
return ""
}
if utf8.RuneCountInString(s) <= limit {
return s
}
// Walk runes rather than converting to []rune: the input can be a whole
// note body, and this runs on the read path.
count := 0
for i := range s {
if count == limit {
return s[:i] + Ellipsis
}
count++
}
return s + Ellipsis
}
@@ -0,0 +1,66 @@
// ingestion/internal/textutil/truncate_test.go
package textutil_test
import (
"strings"
"testing"
"unicode/utf8"
"github.com/mathiasbq/hyperguild/ingestion/internal/textutil"
)
func TestTruncate_ShortStringsAreUntouched(t *testing.T) {
for _, s := range []string{"", "kort", "räv", "a — b"} {
if got := textutil.Truncate(s, 100); got != s {
t.Errorf("Truncate(%q, 100) = %q, want it unchanged", s, got)
}
}
}
func TestTruncate_NeverCutsInsideACharacter(t *testing.T) {
// The property, not an example: whatever the limit, the result must be
// valid UTF-8. Byte-slicing Swedish text (å ä ö) or an em dash produces
// an orphaned lead byte, which is what corrupted wiki/index.md (#93).
s := strings.Repeat("räv — ål — ökäd ", 20)
for limit := 0; limit <= len(s)+5; limit++ {
got := textutil.Truncate(s, limit)
if !utf8.ValidString(got) {
t.Fatalf("Truncate(s, %d) produced invalid UTF-8: %q", limit, got)
}
}
}
func TestTruncate_CountsCharactersNotBytes(t *testing.T) {
// "räv" is 4 bytes, 3 runes. A byte-based limit of 3 would cut inside the
// "ä"; a rune-based one returns it whole.
if got := textutil.Truncate("räv", 3); got != "räv" {
t.Errorf("Truncate(\"räv\", 3) = %q, want the whole word", got)
}
got := textutil.Truncate("räven springer", 5)
if utf8.RuneCountInString(got) != 6 { // 5 runes + the ellipsis
t.Errorf("Truncate(...) = %q, want 5 runes plus the marker", got)
}
if !strings.HasSuffix(got, "…") {
t.Errorf("Truncate(...) = %q, want it marked as truncated", got)
}
}
func TestTruncate_MarksOnlyWhenSomethingWasDropped(t *testing.T) {
// An ellipsis on text that was not cut is a lie about the content.
if got := textutil.Truncate("räv", 10); strings.HasSuffix(got, "…") {
t.Errorf("Truncate(%q, 10) = %q, want no truncation marker", "räv", got)
}
if got := textutil.Truncate("räv", 3); strings.HasSuffix(got, "…") {
t.Errorf("exact-length input must not be marked: %q", got)
}
}
func TestTruncate_NonPositiveLimitYieldsNothingRatherThanPanicking(t *testing.T) {
for _, limit := range []int{0, -1, -100} {
if got := textutil.Truncate("räven", limit); got != "" {
t.Errorf("Truncate(_, %d) = %q, want empty", limit, got)
}
}
}
+7
View File
@@ -74,6 +74,13 @@ func processDir(ctx context.Context, cfg Config, date string) []error {
return nil
}
// AutoTunnel's own fuzzy-match human-review queue, not source
// material to extract (hyperguild#88) — same exclusion as
// api.ListPending already applies when listing raw/ for promotion.
if strings.HasPrefix(d.Name(), "tunnel-candidates-") {
return nil
}
// Skip files that have already been processed or permanently failed.
if _, err := os.Stat(path + ".processed"); err == nil {
return nil
@@ -229,3 +229,49 @@ func TestProcessDir_SkipsSubdirs(t *testing.T) {
_, err = os.Stat(failedFile)
assert.NoError(t, err, "failed subdir file should be untouched")
}
// TestProcessDir_SkipsTunnelCandidateFiles guards against hyperguild#88:
// AutoTunnel's own human-review queue (brain/raw/tunnel-candidates-*.md)
// must never be re-ingested as if it were external source material — doing
// so fed the raw "(term: X)" log entries back through the LLM extraction
// pipeline, which then legitimately (from its own perspective) surfaced
// [[X]] as a wikilink for any term that happened to match a real page
// title, however generic (e.g. "mission").
func TestProcessDir_SkipsTunnelCandidateFiles(t *testing.T) {
brainDir := setupBrainDir(t)
tunnelFile := filepath.Join(brainDir, "raw", "tunnel-candidates-2026-07-19.md")
require.NoError(t, os.WriteFile(tunnelFile, []byte(
"# Tunnel candidates 2026-07-19\n\n- `wiki/homelab/decisions/foo.md` ↔ `wiki/telos/decisions/mission.md` (term: \"mission\")\n",
), 0o644))
var completeCalls int
completeFn := func(ctx context.Context, system, user string) (string, error) {
completeCalls++
raw := pipeline.RawPage{Title: "Should not be written", Type: "source", Subtype: "article", Content: "## Summary\n\nx.\n"}
b, _ := json.Marshal([]pipeline.RawPage{raw})
return string(b), nil
}
cfg := Config{
BrainDir: brainDir,
Interval: time.Hour, // not used; we call processDir directly
Pipeline: pipeline.Config{
Complete: completeFn,
ChunkSize: 0,
Schema: "# Schema\nThree page types.",
},
}
date := time.Now().UTC().Format("2006-01-02")
errs := processDir(context.Background(), cfg, date)
assert.Empty(t, errs)
assert.Zero(t, completeCalls, "tunnel-candidates file must never reach the LLM extraction pipeline")
// File must be left alone in raw/ — not moved to processed/, no marker written.
_, err := os.Stat(tunnelFile + ".processed")
assert.True(t, os.IsNotExist(err), "tunnel-candidates file should not get a .processed marker")
_, err = os.Stat(filepath.Join(brainDir, "raw", "processed", date, "tunnel-candidates-2026-07-19.md"))
assert.True(t, os.IsNotExist(err), "tunnel-candidates file should not be copied to processed/")
}
+109
View File
@@ -0,0 +1,109 @@
// Package webhook triggers an on-demand brain-sync Job when Gitea pushes to
// mathias/brain, instead of waiting for the next 15-minute CronJob poll.
package webhook
import (
"context"
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"log/slog"
"net/http"
batchv1 "k8s.io/api/batch/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
)
// VerifySignature checks a Gitea webhook's X-Gitea-Signature header: a
// hex-encoded HMAC-SHA256 of the raw request body, keyed by the shared
// webhook secret. Constant-time compare — timing must not leak how much of
// the signature matched.
func VerifySignature(payload []byte, signatureHeader, secret string) bool {
if signatureHeader == "" {
return false
}
mac := hmac.New(sha256.New, []byte(secret))
mac.Write(payload)
expected := hex.EncodeToString(mac.Sum(nil))
return hmac.Equal([]byte(expected), []byte(signatureHeader))
}
// pushEvent is the subset of Gitea's push webhook payload this handler needs.
type pushEvent struct {
Ref string `json:"ref"`
Repo struct {
FullName string `json:"full_name"`
} `json:"repository"`
}
// TriggerJobFromCronJob reads the named CronJob's job template and creates a
// new, uniquely-named Job from it — the same thing `kubectl create job
// --from=cronjob/<name>` does. Reuses the CronJob's already-tested script
// rather than re-implementing sync logic here.
func TriggerJobFromCronJob(ctx context.Context, cs kubernetes.Interface, namespace, cronJobName string) (string, error) {
cj, err := cs.BatchV1().CronJobs(namespace).Get(ctx, cronJobName, metav1.GetOptions{})
if err != nil {
return "", fmt.Errorf("get cronjob %s/%s: %w", namespace, cronJobName, err)
}
job := &batchv1.Job{
ObjectMeta: metav1.ObjectMeta{
GenerateName: cronJobName + "-webhook-",
Namespace: namespace,
Annotations: map[string]string{
"triggered-by": "brain-webhook",
},
},
Spec: cj.Spec.JobTemplate.Spec,
}
created, err := cs.BatchV1().Jobs(namespace).Create(ctx, job, metav1.CreateOptions{})
if err != nil {
return "", fmt.Errorf("create job from cronjob %s/%s: %w", namespace, cronJobName, err)
}
return created.Name, nil
}
// Handler is the HTTP handler for Gitea's push webhook on mathias/brain.
type Handler struct {
Secret string
Clientset kubernetes.Interface
Namespace string // e.g. "brain"
CronJobName string // e.g. "brain-sync"
WatchRepo string // e.g. "mathias/brain"
Logger *slog.Logger
}
func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
body, err := io.ReadAll(r.Body)
if err != nil {
http.Error(w, "bad body", http.StatusBadRequest)
return
}
sig := r.Header.Get("X-Gitea-Signature")
if !VerifySignature(body, sig, h.Secret) {
http.Error(w, "bad signature", http.StatusUnauthorized)
return
}
var ev pushEvent
if err := json.Unmarshal(body, &ev); err != nil {
http.Error(w, "bad payload", http.StatusBadRequest)
return
}
if ev.Repo.FullName != h.WatchRepo || ev.Ref != "refs/heads/main" {
w.WriteHeader(http.StatusOK)
_, _ = fmt.Fprintf(w, "ignored: repo=%s ref=%s", ev.Repo.FullName, ev.Ref)
return
}
jobName, err := TriggerJobFromCronJob(r.Context(), h.Clientset, h.Namespace, h.CronJobName)
if err != nil {
h.Logger.Error("webhook: trigger job failed", "err", err)
http.Error(w, "trigger failed", http.StatusInternalServerError)
return
}
h.Logger.Info("webhook: triggered brain-sync job", "job", jobName)
w.WriteHeader(http.StatusOK)
_, _ = fmt.Fprintf(w, "triggered %s", jobName)
}
+222
View File
@@ -0,0 +1,222 @@
package webhook
import (
"bytes"
"context"
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"log/slog"
"net/http"
"net/http/httptest"
"os"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
batchv1 "k8s.io/api/batch/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/client-go/kubernetes/fake"
k8stesting "k8s.io/client-go/testing"
)
// newFakeClientset simulates the real API server's GenerateName expansion,
// which the plain fake clientset tracker does not do on its own -- without
// this, every created Job keeps an empty Name (a fake-clientset limitation,
// not real API server behavior).
func newFakeClientset(objects ...runtime.Object) *fake.Clientset {
cs := fake.NewSimpleClientset(objects...)
cs.PrependReactor("create", "jobs", func(action k8stesting.Action) (bool, runtime.Object, error) {
createAction := action.(k8stesting.CreateAction)
job, ok := createAction.GetObject().(*batchv1.Job)
if ok && job.Name == "" && job.GenerateName != "" {
job.Name = job.GenerateName + "test0001"
}
return false, nil, nil // not "handled" -- let the default reactor store it
})
return cs
}
func sign(t *testing.T, payload []byte, secret string) string {
t.Helper()
mac := hmac.New(sha256.New, []byte(secret))
mac.Write(payload)
return hex.EncodeToString(mac.Sum(nil))
}
func testLogger() *slog.Logger {
return slog.New(slog.NewTextHandler(os.Stderr, nil))
}
func fakeCronJob(namespace, name string) *batchv1.CronJob {
return &batchv1.CronJob{
ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: namespace},
Spec: batchv1.CronJobSpec{
JobTemplate: batchv1.JobTemplateSpec{
Spec: batchv1.JobSpec{
Template: corev1.PodTemplateSpec{
Spec: corev1.PodSpec{
Containers: []corev1.Container{
{Name: "sync", Image: "alpine/git:v2.47.2"},
},
RestartPolicy: corev1.RestartPolicyNever,
},
},
},
},
},
}
}
// --------------------------------------------------------------------------- #
// VerifySignature
// --------------------------------------------------------------------------- #
func TestVerifySignature_AcceptsCorrectHMAC(t *testing.T) {
payload := []byte(`{"ref":"refs/heads/main"}`)
secret := "s3cret"
sig := sign(t, payload, secret)
assert.True(t, VerifySignature(payload, sig, secret))
}
func TestVerifySignature_RejectsWrongSecret(t *testing.T) {
payload := []byte(`{"ref":"refs/heads/main"}`)
sig := sign(t, payload, "right-secret")
assert.False(t, VerifySignature(payload, sig, "wrong-secret"))
}
func TestVerifySignature_RejectsTamperedPayload(t *testing.T) {
secret := "s3cret"
sig := sign(t, []byte(`{"ref":"refs/heads/main"}`), secret)
assert.False(t, VerifySignature([]byte(`{"ref":"refs/heads/evil"}`), sig, secret))
}
func TestVerifySignature_RejectsEmptySignature(t *testing.T) {
assert.False(t, VerifySignature([]byte("payload"), "", "secret"))
}
// --------------------------------------------------------------------------- #
// TriggerJobFromCronJob
// --------------------------------------------------------------------------- #
func TestTriggerJobFromCronJob_CreatesJobMatchingTemplate(t *testing.T) {
cs := newFakeClientset(fakeCronJob("brain", "brain-sync"))
jobName, err := TriggerJobFromCronJob(context.Background(), cs, "brain", "brain-sync")
require.NoError(t, err)
assert.NotEmpty(t, jobName)
jobs, err := cs.BatchV1().Jobs("brain").List(context.Background(), metav1.ListOptions{})
require.NoError(t, err)
require.Len(t, jobs.Items, 1)
assert.Equal(t, "alpine/git:v2.47.2", jobs.Items[0].Spec.Template.Spec.Containers[0].Image)
}
func TestTriggerJobFromCronJob_ErrorsWhenCronJobMissing(t *testing.T) {
cs := fake.NewSimpleClientset()
_, err := TriggerJobFromCronJob(context.Background(), cs, "brain", "brain-sync")
assert.Error(t, err)
}
// --------------------------------------------------------------------------- #
// Handler.ServeHTTP
// --------------------------------------------------------------------------- #
func newTestHandler(cs *fake.Clientset) *Handler {
return &Handler{
Secret: "s3cret",
Clientset: cs,
Namespace: "brain",
CronJobName: "brain-sync",
WatchRepo: "mathias/brain",
Logger: testLogger(),
}
}
func pushPayload(t *testing.T, repo, ref string) []byte {
t.Helper()
body := map[string]any{
"ref": ref,
"repository": map[string]any{
"full_name": repo,
},
}
b, err := json.Marshal(body)
require.NoError(t, err)
return b
}
func TestHandler_RejectsMissingSignature(t *testing.T) {
cs := fake.NewSimpleClientset(fakeCronJob("brain", "brain-sync"))
h := newTestHandler(cs)
payload := pushPayload(t, "mathias/brain", "refs/heads/main")
req := httptest.NewRequest(http.MethodPost, "/webhooks/brain-sync", bytes.NewReader(payload))
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
assert.Equal(t, http.StatusUnauthorized, rec.Code)
jobs, _ := cs.BatchV1().Jobs("brain").List(context.Background(), metav1.ListOptions{})
assert.Empty(t, jobs.Items, "must not trigger a job on an unsigned request")
}
func TestHandler_RejectsWrongSignature(t *testing.T) {
cs := fake.NewSimpleClientset(fakeCronJob("brain", "brain-sync"))
h := newTestHandler(cs)
payload := pushPayload(t, "mathias/brain", "refs/heads/main")
req := httptest.NewRequest(http.MethodPost, "/webhooks/brain-sync", bytes.NewReader(payload))
req.Header.Set("X-Gitea-Signature", sign(t, payload, "not-the-real-secret"))
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
assert.Equal(t, http.StatusUnauthorized, rec.Code)
jobs, _ := cs.BatchV1().Jobs("brain").List(context.Background(), metav1.ListOptions{})
assert.Empty(t, jobs.Items)
}
func TestHandler_IgnoresOtherRepos(t *testing.T) {
cs := fake.NewSimpleClientset(fakeCronJob("brain", "brain-sync"))
h := newTestHandler(cs)
payload := pushPayload(t, "mathias/some-other-repo", "refs/heads/main")
req := httptest.NewRequest(http.MethodPost, "/webhooks/brain-sync", bytes.NewReader(payload))
req.Header.Set("X-Gitea-Signature", sign(t, payload, "s3cret"))
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
assert.Equal(t, http.StatusOK, rec.Code)
jobs, _ := cs.BatchV1().Jobs("brain").List(context.Background(), metav1.ListOptions{})
assert.Empty(t, jobs.Items, "must not trigger for a push to an unrelated repo")
}
func TestHandler_IgnoresNonMainBranch(t *testing.T) {
cs := fake.NewSimpleClientset(fakeCronJob("brain", "brain-sync"))
h := newTestHandler(cs)
payload := pushPayload(t, "mathias/brain", "refs/heads/some-feature-branch")
req := httptest.NewRequest(http.MethodPost, "/webhooks/brain-sync", bytes.NewReader(payload))
req.Header.Set("X-Gitea-Signature", sign(t, payload, "s3cret"))
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
assert.Equal(t, http.StatusOK, rec.Code)
jobs, _ := cs.BatchV1().Jobs("brain").List(context.Background(), metav1.ListOptions{})
assert.Empty(t, jobs.Items, "must not trigger for a push to a non-main branch")
}
func TestHandler_TriggersJobOnValidMainPush(t *testing.T) {
cs := fake.NewSimpleClientset(fakeCronJob("brain", "brain-sync"))
h := newTestHandler(cs)
payload := pushPayload(t, "mathias/brain", "refs/heads/main")
req := httptest.NewRequest(http.MethodPost, "/webhooks/brain-sync", bytes.NewReader(payload))
req.Header.Set("X-Gitea-Signature", sign(t, payload, "s3cret"))
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
assert.Equal(t, http.StatusOK, rec.Code)
jobs, err := cs.BatchV1().Jobs("brain").List(context.Background(), metav1.ListOptions{})
require.NoError(t, err)
assert.Len(t, jobs.Items, 1, "a valid push to main on the watched repo must trigger exactly one job")
}
+3 -4
View File
@@ -6,6 +6,8 @@ import (
"os"
"path/filepath"
"strings"
"github.com/mathiasbq/hyperguild/ingestion/internal/textutil"
)
// RebuildIndex writes brain/wiki/index.md from the current wiki contents.
@@ -62,10 +64,7 @@ func pageFirstSentence(brainDir string, e Entry) string {
if line == "" || strings.HasPrefix(line, "#") {
continue
}
if len(line) > 100 {
return line[:100] + "…"
}
return line
return textutil.Truncate(line, 100)
}
return ""
}
+1 -1
View File
@@ -13,7 +13,7 @@ import (
"github.com/lestrrat-go/jwx/v2/jwa"
"github.com/lestrrat-go/jwx/v2/jwk"
"github.com/lestrrat-go/jwx/v2/jwt"
"github.com/mathiasbq/supervisor/internal/auth"
"git.d-ma.be/mathias/hyperguild/internal/auth"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -6,7 +6,7 @@ import (
"net/http/httptest"
"testing"
"github.com/mathiasbq/supervisor/internal/auth"
"git.d-ma.be/mathias/hyperguild/internal/auth"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -7,7 +7,7 @@ import (
"net/http/httptest"
"testing"
"github.com/mathiasbq/supervisor/internal/brain"
"git.d-ma.be/mathias/hyperguild/internal/brain"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -3,7 +3,7 @@ package config_test
import (
"testing"
"github.com/mathiasbq/supervisor/internal/config"
"git.d-ma.be/mathias/hyperguild/internal/config"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -5,7 +5,7 @@ import (
"path/filepath"
"testing"
"github.com/mathiasbq/supervisor/internal/config"
"git.d-ma.be/mathias/hyperguild/internal/config"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -3,7 +3,7 @@ package config_test
import (
"testing"
"github.com/mathiasbq/supervisor/internal/config"
"git.d-ma.be/mathias/hyperguild/internal/config"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"testing"
"time"
iexec "github.com/mathiasbq/supervisor/internal/exec"
iexec "git.d-ma.be/mathias/hyperguild/internal/exec"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -9,7 +9,7 @@ import (
"net/http/httptest"
"testing"
"github.com/mathiasbq/supervisor/internal/githubclient"
"git.d-ma.be/mathias/hyperguild/internal/githubclient"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+2 -2
View File
@@ -8,8 +8,8 @@ import (
"net/http"
"strings"
"github.com/mathiasbq/supervisor/internal/auth"
"github.com/mathiasbq/supervisor/internal/registry"
"git.d-ma.be/mathias/hyperguild/internal/auth"
"git.d-ma.be/mathias/hyperguild/internal/registry"
)
type request struct {
+2 -2
View File
@@ -8,8 +8,8 @@ import (
"strings"
"testing"
"github.com/mathiasbq/supervisor/internal/mcp"
"github.com/mathiasbq/supervisor/internal/registry"
"git.d-ma.be/mathias/hyperguild/internal/mcp"
"git.d-ma.be/mathias/hyperguild/internal/registry"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -9,7 +9,7 @@ import (
"net/http/httptest"
"testing"
"github.com/mathiasbq/supervisor/internal/mcpclient"
"git.d-ma.be/mathias/hyperguild/internal/mcpclient"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -5,7 +5,7 @@ import (
"encoding/json"
"testing"
"github.com/mathiasbq/supervisor/internal/registry"
"git.d-ma.be/mathias/hyperguild/internal/registry"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -6,7 +6,7 @@ import (
"testing"
"time"
"github.com/mathiasbq/supervisor/internal/session"
"git.d-ma.be/mathias/hyperguild/internal/session"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"testing"
"time"
"github.com/mathiasbq/supervisor/internal/session"
"git.d-ma.be/mathias/hyperguild/internal/session"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"net/http/httptest"
"testing"
"github.com/mathiasbq/supervisor/internal/skills/brain"
"git.d-ma.be/mathias/hyperguild/internal/skills/brain"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -4,7 +4,7 @@ package brain
import (
"encoding/json"
"github.com/mathiasbq/supervisor/internal/registry"
"git.d-ma.be/mathias/hyperguild/internal/registry"
)
// Config holds brain skill configuration.
+2 -2
View File
@@ -6,8 +6,8 @@ import (
"encoding/json"
"testing"
"github.com/mathiasbq/supervisor/internal/skills/org"
"github.com/mathiasbq/supervisor/internal/tier"
"git.d-ma.be/mathias/hyperguild/internal/skills/org"
"git.d-ma.be/mathias/hyperguild/internal/tier"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+2 -2
View File
@@ -5,8 +5,8 @@ import (
"context"
"encoding/json"
"github.com/mathiasbq/supervisor/internal/registry"
"github.com/mathiasbq/supervisor/internal/tier"
"git.d-ma.be/mathias/hyperguild/internal/registry"
"git.d-ma.be/mathias/hyperguild/internal/tier"
)
// TierFn returns the current tier. Injected for testability.
+1 -1
View File
@@ -7,7 +7,7 @@ import (
"fmt"
"time"
"github.com/mathiasbq/supervisor/internal/session"
"git.d-ma.be/mathias/hyperguild/internal/session"
)
type logArgs struct {
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"path/filepath"
"testing"
"github.com/mathiasbq/supervisor/internal/skills/sessionlog"
"git.d-ma.be/mathias/hyperguild/internal/skills/sessionlog"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -4,7 +4,7 @@ package sessionlog
import (
"encoding/json"
"github.com/mathiasbq/supervisor/internal/registry"
"git.d-ma.be/mathias/hyperguild/internal/registry"
)
// Config holds sessionlog skill configuration.
+1 -1
View File
@@ -7,7 +7,7 @@ import (
"net/http/httptest"
"testing"
"github.com/mathiasbq/supervisor/internal/tier"
"git.d-ma.be/mathias/hyperguild/internal/tier"
"github.com/stretchr/testify/assert"
)
+8 -7
View File
@@ -42,34 +42,35 @@ These actions are **carried into the Phase 4 capture call** as `tickets[]` rathe
## Phase 4 — Capture (one uniform call)
Persist the session via a **single `capture` call** (the `brain:capture` MCP tool, live on the Claude.ai connector). Capture owns the writes server-side — insights → brain, action items → Gitea tickets, summary → ai-sessions — plus the I1 sovereignty gate, the I5 audit record, and the supersession/read-after-write discipline. The skill's job is to *assemble the payload*, not to write each store itself. Do NOT fall back to separate `gitea:file_write_branch` + `brain_write` steps unless `capture` is unreachable (see fallback below).
Persist the session via a **single `capture` call** (the `brain:capture` MCP tool, live on the Claude.ai connector). Capture owns the writes server-side — insights → brain, action items → Gitea tickets — plus the I1 sovereignty gate, the I5 audit record, and the supersession/read-after-write discipline. The skill's job is to *assemble the payload*, not to write each store itself. Do NOT fall back to separate `gitea:file_write_branch` + `brain_write` steps unless `capture` is unreachable (see fallback below).
**Note:** capture no longer writes a session summary anywhere (the `summary` → `ai-sessions` write path was removed — it produced a frontmatter shape `ai-sessions`' own pipeline couldn't parse, silently invisible to that repo's own audit tooling; see `ai-sessions#13`). If the session's decisions/artifacts are worth a durable narrative beyond the `insights[]` this skill writes to the brain, that's a separate, explicit call — not something this skill does implicitly.
**Assemble one payload:**
- **`insights[]`** — the generalizable learnings from Phase 1 (decisions/failures worth re-reading). Each: `{text, wing, hall}`; add `supersede_slug` to revise a prior note in place instead of creating a duplicate. `hall` ∈ facts/decisions/failures/hypotheses/sources.
- **`tickets[]`** — the issue actions from Phase 3: `{repo, action, ...}` where action ∈ create/close/comment. Owner is always `mathias` (server-forced).
- **`summary`** — `{title, body, repos_touched}`. Capture writes it to `ai-sessions` and stamps `fidelity` in frontmatter. Body stays reconstructable: one-paragraph summary, decisions, key artifacts, open threads.
- **`context`** — `{harness: "claudeai-chat", session_ref: <chatid8-or-slug>, fidelity: "live-capture", actor: "mathias", classification: <see gate below>}`.
**THE CLASSIFICATION GATE (read before calling — this is where capture refuses).**
Capture computes an **effective classification = the strictest across EVERY target it touches** (each insight's `wing`, each ticket's `repo`, and every entry in `summary.repos_touched`), then refuses if that effective level is `confidential` and the origin is us-nexus (claude.ai is us-nexus). Levels come from `classification.yaml` at the brain root (source of truth, #67), with the code defaults as the floor: `hyperguild`/`homelab` → internal; `client-*` → confidential; **anything untagged → confidential (fail-safe)**.
Capture computes an **effective classification = the strictest across EVERY target it touches** (each insight's `wing`, each ticket's `repo`), then refuses if that effective level is `confidential` and the origin is us-nexus (claude.ai is us-nexus). Levels come from `classification.yaml` at the brain root (source of truth, #67), with the code defaults as the floor: `hyperguild`/`homelab` → internal; `client-*` → confidential; **anything untagged → confidential (fail-safe)**.
- **Tagged `internal` today** (safe through claude.ai): wings `hyperguild`, `homelab`; repos `brain`, `ai-sessions`, `infra`, `hyperguild`, `homelab`, `tapir`, `agentsquad`, `jepa-fx-risk`, `swedsl`. Treat `classification.yaml` as authoritative — this list is a hint, not gospel.
- Declare `context.classification: "internal"` for normal homelab work.
- `summary.repos_touched`, insight `wing`s, and ticket `repo`s are classification INPUTS, not free-form metadata — every target must resolve `internal` or the whole capture escalates to `confidential` and the gate refuses via claude.ai. Listing the central homelab repos (incl. `brain`/`ai-sessions`) is now fine; they're tagged. The summary always lands in `ai-sessions` (internal), so the summary path itself never escalates.
- Insight `wing`s and ticket `repo`s are classification INPUTS, not free-form metadata — every target must resolve `internal` or the whole capture escalates to `confidential` and the gate refuses via claude.ai. Listing the central homelab repos (incl. `brain`/`ai-sessions`) is now fine; they're tagged.
- If a session genuinely touched **`client-*` or otherwise-untagged** material, it cannot be captured through claude.ai — note that in the verdict rather than trying to force it.
**GATE — dry-run first, then execute.**
1. Call `capture` with `dry_run: true`. It validates the whole payload and returns the would-be receipt + `effective_classification`, writing nothing.
2. **STOP. Show the dry-run receipt** (effective classification, the insights/tickets/summary that would land) and get explicit confirmation.
2. **STOP. Show the dry-run receipt** (effective classification, the insights/tickets that would land) and get explicit confirmation.
3. On confirmation, call `capture` again with `dry_run: false`. Read the returned receipt: it is partial-aware (`errors[]`, per-item `ok`). Report exactly what landed.
If `capture` is **unreachable** (tool not on the connector — e.g. a session that started before a deploy; a tool-list refresh usually fixes it): say so. Only then fall back to the legacy inline path (`gitea:file_write_branch` summary + `brain_write`/`brain_update` + `brain_get` confirm), and note in the verdict that the I5 audit record was NOT produced.
If `capture` is **unreachable** (tool not on the connector — e.g. a session that started before a deploy; a tool-list refresh usually fixes it): say so. Only then fall back to the legacy inline path (`brain_write`/`brain_update` + `brain_get` confirm), and note in the verdict that the I5 audit record was NOT produced.
## Phase 5 — Verdict
Deliver a final "safe to archive" verdict in the chat. Either:
- **SAFE TO ARCHIVE** — list what landed from the capture receipt (issues closed/filed with numbers, summary path, brain note ids/paths) so the trail is auditable. Then list anything still in the user's queue (e.g. a PR awaiting their merge, a decision owed next session).
- **SAFE TO ARCHIVE** — list what landed from the capture receipt (issues closed/filed with numbers, brain note ids/paths) so the trail is auditable. Then list anything still in the user's queue (e.g. a PR awaiting their merge, a decision owed next session).
- **NOT YET** — name the specific gate that wasn't passed, the capture refusal reason, or the per-item error from the receipt, and what to do about it.
Never claim safe-to-archive if the capture refused, any receipt item errored, or a gated confirmation was declined. The verdict is the skill's contract: if it says safe, the session can be lost without losing the work.
+2
View File
@@ -2,6 +2,8 @@
**Status:** Decisions resolved 2026-06-22 (§4). Ready for implementation scoping. `capture` is a
privileged cross-harness write path touching brain + Gitea + ai-sessions.
**Superseded 2026-07-27:** the ai-sessions summary write path specified below was removed post-ship
(see `specs/capture-implementation-report.md`). `capture` now touches brain + Gitea only.
**Tracks:** hyperguild #49.
**Governed by:** `infra/docs/architecture/01-invariants.md` (I1–I5), the admissibility test in
`00-synthesis-model.md`, and the distributed-consolidation shape mandated by
+7
View File
@@ -1,6 +1,13 @@
# Capture capability — implementation report (as-built)
**Status:** Shipped 2026-06-23, tagged `v0.11.0`. Epic hyperguild #49 (sub-issues #50–#55) closed.
**Superseded 2026-07-27:** the `summary` → `ai-sessions` write path documented below was removed.
It wrote a frontmatter shape (`title`/`harness`/`fidelity`/`captured_at`/`repos_touched`) that
`ai-sessions`' own extract/audit pipeline couldn't parse — invisible to that repo's own audit
tooling and bypassing its redaction/completeness gates (`ai-sessions#13`). `capture` now persists
insights → brain and action items → Gitea tickets only. This document is kept as the historical
as-built record of what shipped in #49; treat every `summary`/ai-sessions reference below as
retired, not current behaviour.
**Spec:** `specs/capture-bdd-spec.md` (the design contract this implements).
**Governed by:** `infra/docs/architecture/01-invariants.md` (I1–I5) + the I2 acceptance ledger entry in `infra/docs/security-baseline.md`.