Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
dc2265da0b | ||
|
|
c176b71b3d | ||
|
|
4e8bdc137e | ||
|
|
42235dcb6a | ||
|
|
05330ad54a | ||
|
|
692244d213 | ||
|
|
d98a305985 | ||
|
|
670325e519 | ||
|
|
54237af7ad | ||
|
|
b6ceccca26 | ||
|
|
36a45068ab | ||
|
|
055af368cd | ||
|
|
eb9e3e69a9 | ||
|
|
b7cf0efb9d | ||
|
|
125c99bb46 | ||
|
|
3e537bd582 | ||
|
|
4e6fc7aeee | ||
|
|
9a232ad444 | ||
|
|
3da0eca9b4 | ||
|
|
7d1213451c | ||
|
|
7c9293f4ed | ||
|
|
6e0155a2ab | ||
|
|
1cea2c9f78 | ||
|
|
9dcd60931a | ||
|
|
1fac90ed2a | ||
|
|
cb9c2513a4 | ||
|
|
3617a6c386 | ||
|
|
1938170131 | ||
|
|
b34717e6b8 | ||
|
|
d8d7e9a307 | ||
|
|
f0055483a3 | ||
|
|
6d014c1d0f | ||
|
|
1001acfb44 | ||
|
|
6ad275b505 | ||
|
|
b600cc986c | ||
|
|
9bdab1c48c | ||
|
|
6520c2fc4b | ||
|
|
fcbd1072b6 | ||
|
|
3b7706b358 | ||
|
|
394a227877 | ||
|
|
0f84ab5eda | ||
|
|
5b57843346 | ||
|
|
ee1204d76b | ||
|
|
6d58336ce2 | ||
|
|
0785f14220 | ||
|
|
a7db0dd00d | ||
|
|
fb59c390e2 | ||
|
|
00e5f62c8e |
+90
-21
@@ -53,14 +53,30 @@ These rules apply to every task across every project, regardless of harness.
|
|||||||
logical change (one tool, one fix, one test) with passing tests. Main is always
|
logical change (one tool, one fix, one test) with passing tests. Main is always
|
||||||
deployable. Never create long-lived feature branches.
|
deployable. Never create long-lived feature branches.
|
||||||
|
|
||||||
**Exception — parallel agents on same repo:** If another agent is known to be
|
**The PR trigger is RISK, not concurrency.** `risk/low` work commits straight to
|
||||||
actively working on the same repo simultaneously, create a short-lived branch
|
main with CI as the gate — that is most work and is unchanged. `risk/medium` and
|
||||||
(`agent/<description>`), finish the task, and merge to main within the same
|
above opens a PR, even solo, even with nobody waiting to review it. This mirrors
|
||||||
session. Do not leave agent branches open between sessions.
|
the tiered merge model in `assessor-loop`
|
||||||
|
`method/transfer-to-a-regulated-organisation.md` §4.1, so the estate practises
|
||||||
|
what it sells instead of merging everything on green and claiming otherwise.
|
||||||
|
|
||||||
**Exception — external contributor or client four-eyes requirement:** Use
|
**Concurrency is NOT a reason to branch.** Measured 2026-08-29 across three real
|
||||||
PR flow only when a human reviewer outside the project is required. Document
|
collisions between parallel sessions: two were not branching problems at all — a
|
||||||
the reason in PROJECT.md.
|
generated artifact written to the same path by both (branching turns that into a
|
||||||
|
merge conflict on a 33k-line JSON), and uncommitted work in a shared checkout
|
||||||
|
(a per-session worktree fixes it) — and the third was an ordinary non-fast-forward
|
||||||
|
push resolved by rebase in seconds. Use a worktree per session on shared repos and
|
||||||
|
timestamp generated artifacts. Do not reach for a branch.
|
||||||
|
|
||||||
|
Branch+PR flow carries documented costs on solo repos: merging auto-closes gated
|
||||||
|
issues, review comments live on a thread separate from the issue's, and a
|
||||||
|
merged-and-deleted branch cannot be amended (brain:
|
||||||
|
`gitea-pr-merge-autoclose-and-review-comment-footguns`). Those are the price of
|
||||||
|
the `risk/medium`+ lane — paid deliberately, not incurred by accident.
|
||||||
|
|
||||||
|
**Exception — external contributor or client four-eyes requirement:** Use PR flow
|
||||||
|
when a human reviewer outside the project is required. Document the reason in
|
||||||
|
PROJECT.md.
|
||||||
|
|
||||||
6. **Close the loop — every substantive task ends with the same ritual.** Shipping
|
6. **Close the loop — every substantive task ends with the same ritual.** Shipping
|
||||||
the code is not the end of the task; capturing it is. Run this unprompted:
|
the code is not the end of the task; capturing it is. Run this unprompted:
|
||||||
@@ -78,6 +94,26 @@ These rules apply to every task across every project, regardless of harness.
|
|||||||
- Surface the brain entries and issue numbers in the closing summary so the
|
- Surface the brain entries and issue numbers in the closing summary so the
|
||||||
trail is auditable.
|
trail is auditable.
|
||||||
|
|
||||||
|
**Chat is not a durable store. Never rely on Mathias remembering anything.**
|
||||||
|
Every gap, caveat, risk, open question and next step must land in one of four
|
||||||
|
places before the turn ends: **the brain**, **a repo** (code, docs, ADR),
|
||||||
|
**a tracker issue**, or **a plan file**. A finding stated only in conversation is
|
||||||
|
lost the moment the session closes — and this estate has repeatedly proved that a
|
||||||
|
remembered convention is an unreliable control.
|
||||||
|
|
||||||
|
This applies hardest to what is tempting to leave as a closing remark: "worth
|
||||||
|
knowing X isn't enforced yet", "this is a convention, not a gate", "we should
|
||||||
|
revisit Y". Those are exactly the items that need a locator. If it is worth
|
||||||
|
saying in a summary, it is worth writing where it can be found again without you.
|
||||||
|
|
||||||
|
## CAD sprint patterns
|
||||||
|
|
||||||
|
Two CAD sprint archetypes (defs: brain `wiki/homelab/decisions/continuous-agentic-development-cad-concept-2026-06-16`):
|
||||||
|
- **Inception Sprint** — birth a new repo. Oath-gated close; tag only once every clause holds. Brain: `.../inception-sprint-and-oath`.
|
||||||
|
- **Sweep Sprint** — clear an existing backlog. Parallel read-only research per issue → sequential one-issue-per-commit execution. Brain: `.../sweep-sprint-and-oath`.
|
||||||
|
|
||||||
|
Shared: decisions that are yours, not the agent's, surface as `DECISION NEEDED` (question/options/recommendation/blocks) — ask live if you're present, else comment and wait, never act past it. Live-breaking regressions get `git revert`, never forward-fixed; host-network work skips even read-only research (solo, synchronous, you present). Rule 6's closing ritual only fires once every Oath clause genuinely holds — blocked = descoped-and-tracked, never marked done.
|
||||||
|
|
||||||
## Default stack
|
## Default stack
|
||||||
|
|
||||||
| Layer | Default | Fallback | Last resort |
|
| Layer | Default | Fallback | Last resort |
|
||||||
@@ -88,7 +124,7 @@ These rules apply to every task across every project, regardless of harness.
|
|||||||
| Containers | Docker Compose (dev), k3s (prod) | — | — |
|
| Containers | Docker Compose (dev), k3s (prod) | — | — |
|
||||||
| DB | PostgreSQL + sqlc | SQLite | — |
|
| DB | PostgreSQL + sqlc | SQLite | — |
|
||||||
| Search | pgvector (vector), BM25 | Qdrant (when >1M vectors or hybrid retrieval) | — |
|
| Search | pgvector (vector), BM25 | Qdrant (when >1M vectors or hybrid retrieval) | — |
|
||||||
| Logging | slog (structured) | — | — |
|
| Logging | slog (structured) | stdlib `logging` w/ structured `extra=` (or structlog) | — |
|
||||||
| Testing | Table-driven, testify | — | — |
|
| Testing | Table-driven, testify | — | — |
|
||||||
| Agents (Go) | google.golang.org/adk + pkg/litellm adapter | — | — |
|
| Agents (Go) | google.golang.org/adk + pkg/litellm adapter | — | — |
|
||||||
|
|
||||||
@@ -97,7 +133,12 @@ Exploratory: Rust, Zig — I'll tell you when I want these.
|
|||||||
## Code conventions
|
## Code conventions
|
||||||
|
|
||||||
- **Go style**: golines, gofumpt, golangci-lint
|
- **Go style**: golines, gofumpt, golangci-lint
|
||||||
- **Errors**: `fmt.Errorf("operation: %w", err)` — never naked, never log-and-return
|
- **Python style** (fallback language): ruff (format+lint, one tool), mypy --strict (non-negotiable,
|
||||||
|
matches Go's static typing discipline), pytest + pytest-cov (table-driven via
|
||||||
|
`@pytest.mark.parametrize`), uv (venv+deps+lock, one tool), pydantic-settings (typed env-var config
|
||||||
|
— same principle as Go's typed structs), src-layout + `pyproject.toml` only (no `setup.py`)
|
||||||
|
- **Errors**: `fmt.Errorf("operation: %w", err)` — never naked, never log-and-return.
|
||||||
|
Python: `raise X from e` (exception chaining, same principle) — never bare `except`, never silent `pass`
|
||||||
- **Naming**: stdlib conventions, no stuttering
|
- **Naming**: stdlib conventions, no stuttering
|
||||||
- **Architecture**: prefer stdlib over frameworks, constructor injection, env-var config parsed into typed structs
|
- **Architecture**: prefer stdlib over frameworks, constructor injection, env-var config parsed into typed structs
|
||||||
- **Git**: conventional commits (`feat:`, `fix:`, `chore:`), commit directly to main,
|
- **Git**: conventional commits (`feat:`, `fix:`, `chore:`), commit directly to main,
|
||||||
@@ -135,10 +176,10 @@ Three machines on Tailscale:
|
|||||||
|---------|------|-----------|
|
|---------|------|-----------|
|
||||||
| koala | GPU inference, heavy compute | RTX 5070, runs k3s + llama-swap + shared postgres18/pgvector |
|
| koala | GPU inference, heavy compute | RTX 5070, runs k3s + llama-swap + shared postgres18/pgvector |
|
||||||
| iguana | Services, builds | M2 Ultra Mac |
|
| iguana | Services, builds | M2 Ultra Mac |
|
||||||
| flamingo | Daily driver, edge | Mac mini, ~/dev is here |
|
| flamingo | Daily driver, edge | Apple Silicon Mac, ~/dev is here |
|
||||||
|
|
||||||
- **Model routing**: LiteLLM in front of llama-swap (local) + cloud APIs (when permitted)
|
- **Model routing**: LiteLLM in front of llama-swap (local) + cloud APIs (when permitted)
|
||||||
- **Orchestration**: k3s cluster across all three machines
|
- **Orchestration**: k3s on koala only — single control-plane node, no workers (verified 2026-08-30). iguana and flamingo are on the tailnet but are not cluster members
|
||||||
- **Networking**: Tailscale mesh
|
- **Networking**: Tailscale mesh
|
||||||
|
|
||||||
## Project landscape
|
## Project landscape
|
||||||
@@ -234,10 +275,11 @@ When working on a project tagged with a client name:
|
|||||||
|
|
||||||
## Harness-agnostic principles
|
## Harness-agnostic principles
|
||||||
|
|
||||||
This context is designed to work with any AI coding tool:
|
Supported harnesses (2026-09-01) — Cursor and Aider were dropped; do not
|
||||||
- Claude Code, Cursor, Aider, Open WebUI, Charmbracelet Mods/Crush
|
re-add adapters for them:
|
||||||
- Pi Coding Agent, Mistral Vibe, Antigravity
|
- Claude Code, Antigravity, OpenCode, Mistral Vibe, Crush, Pi Coding Agent
|
||||||
- Any tool that accepts a system prompt or reads a markdown context file
|
- Open WebUI, Charmbracelet Mods, or any tool that takes a system prompt string
|
||||||
|
- Any other tool that reads `AGENTS.md`, which is the de facto convention
|
||||||
|
|
||||||
The canonical source is always `.context/AGENT.md` (root) and `.context/PROJECT.md` (per-project).
|
The canonical source is always `.context/AGENT.md` (root) and `.context/PROJECT.md` (per-project).
|
||||||
Derived files are committed (see *How context propagates* below) so a `git pull` on any host yields full agent context with no setup.
|
Derived files are committed (see *How context propagates* below) so a `git pull` on any host yields full agent context with no setup.
|
||||||
@@ -249,15 +291,29 @@ Canonical sources of truth:
|
|||||||
- Project: `<repo>/.context/PROJECT.md` (per-repo)
|
- Project: `<repo>/.context/PROJECT.md` (per-repo)
|
||||||
|
|
||||||
Derived files (committed, regenerated by `task context:sync`):
|
Derived files (committed, regenerated by `task context:sync`):
|
||||||
- `CLAUDE.md`, `AGENTS.md`, `.cursorrules`, `.aider.conventions.md`,
|
- `CLAUDE.md`, `AGENTS.md`, `.context/system-prompt.txt`
|
||||||
`.context/system-prompt.txt`
|
|
||||||
|
Repos still carrying `.cursorrules`, `.aider.conventions.md` or
|
||||||
|
`.aider.conf.yml` are stale: nothing regenerates them. Delete them the next time
|
||||||
|
you touch the repo, along with their entries in that repo's `task check`
|
||||||
|
assertion.
|
||||||
|
|
||||||
Workflow:
|
Workflow:
|
||||||
1. Edit a canonical file. Run `task context:sync`. Commit canonical and
|
1. Edit a canonical file. Run `task context:sync`. Commit canonical and
|
||||||
derived together. Push.
|
derived together. Push.
|
||||||
2. On any other host, `git pull` brings both. Claude Code (tree-walking)
|
2. On any other host, `git pull` brings both. Which file a harness reads does
|
||||||
uses `CLAUDE.md`; Crush / Pi / Antigravity (cwd-only) use `AGENTS.md`;
|
NOT split cleanly by lookup strategy, so do not reason from the strategy:
|
||||||
Cursor uses `.cursorrules`; Aider uses `.aider.conventions.md`.
|
- `CLAUDE.md` — Claude Code (walks up the tree)
|
||||||
|
- `AGENTS.md` — Crush, Pi, Antigravity (cwd-only), OpenCode (project root,
|
||||||
|
global at `~/.config/opencode/AGENTS.md`), and Mistral Vibe, which **walks
|
||||||
|
up** from cwd and takes the first `AGENTS.md` it finds
|
||||||
|
(user-level `~/.vibe/AGENTS.md`)
|
||||||
|
- `.context/system-prompt.txt` — Open WebUI, Mods, raw API calls; not read by
|
||||||
|
any harness, it is for pasting
|
||||||
|
|
||||||
|
Vibe tree-walks for `AGENTS.md`, so "tree-walkers use CLAUDE.md" is false.
|
||||||
|
Each repo's `AGENTS.md` embeds the canonical and is the first one found from
|
||||||
|
inside that repo, so this works — but by construction, not by luck.
|
||||||
3. `task check` runs `context:sync` then asserts `git status --porcelain`
|
3. `task check` runs `context:sync` then asserts `git status --porcelain`
|
||||||
is empty over the derived files (catches both modified-tracked drift
|
is empty over the derived files (catches both modified-tracked drift
|
||||||
and missing-untracked adapters). A drift fails the check with a
|
and missing-untracked adapters). A drift fails the check with a
|
||||||
@@ -268,7 +324,20 @@ unconditionally on every host, every harness.
|
|||||||
|
|
||||||
## Engineering Skills
|
## Engineering Skills
|
||||||
|
|
||||||
Shared engineering skills are available in `~/dev/.skills/`. Load at task start — not "on demand" but on schedule, before writing code. See `~/dev/.skills/SKILLS_INDEX.md` for the full list.
|
Shared engineering skills live in the **`mathias/skills`** repo (`git.d-ma.be/mathias/skills`). Load at task start — not "on demand" but on schedule, before writing code. Browse `SKILLS_INDEX.md` for the full list.
|
||||||
|
|
||||||
|
**Claude Code** installs the repo as a plugin from the marketplace the same repo hosts. That path also carries the `secret-guard` and `pre-task-ritual` hooks and the brain/gitea/infra MCP servers, which symlinks cannot:
|
||||||
|
|
||||||
|
```
|
||||||
|
/plugin marketplace add https://git.d-ma.be/mathias/skills.git
|
||||||
|
/plugin install skills@mathias
|
||||||
|
```
|
||||||
|
|
||||||
|
Skills are **namespaced** under Claude Code: load `tdd` as `skills:tdd`. The trigger table below uses bare names — prefix them.
|
||||||
|
|
||||||
|
**Crush, Pi and OpenCode** have no plugin system and are wired by `install.sh` as one directory symlink per harness via `~/.agents/skills`. Do not clone the checkout into `~/dev`: that folder is iCloud-shared between the Macs and spans two path roots (`/Users/mathias` vs `/home/mathias` on koala), so a checkout there is a git working tree with two concurrent writers, and any absolute symlink stored in it is broken on one side by construction (skills#23). Leave `SKILLS_CHECKOUT_DIR` at its host-local default.
|
||||||
|
|
||||||
|
Check a host with `task doctor` in the skills repo; it also runs automatically at session start via the plugin.
|
||||||
|
|
||||||
**Skill trigger table — load before starting, not after getting stuck:**
|
**Skill trigger table — load before starting, not after getting stuck:**
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,2 @@
|
|||||||
|
# Presence of this file marks this repo as opt-in for headless dispatch.
|
||||||
|
# See dispatch#3.
|
||||||
+26
-65
@@ -14,7 +14,6 @@ jobs:
|
|||||||
environment: staging
|
environment: staging
|
||||||
env:
|
env:
|
||||||
INGESTION_IMAGE: git.d-ma.be/mathias/ingestion
|
INGESTION_IMAGE: git.d-ma.be/mathias/ingestion
|
||||||
ROUTING_IMAGE: git.d-ma.be/mathias/routing
|
|
||||||
INFRA_REPO: git@git.d-ma.be:mathias/infra.git
|
INFRA_REPO: git@git.d-ma.be:mathias/infra.git
|
||||||
BUILDKIT_HOST: unix:///run/buildkit/buildkitd.sock
|
BUILDKIT_HOST: unix:///run/buildkit/buildkitd.sock
|
||||||
steps:
|
steps:
|
||||||
@@ -41,39 +40,37 @@ jobs:
|
|||||||
|
|
||||||
echo "Built and pushed ${INGESTION_IMAGE}:${IMAGE_TAG}"
|
echo "Built and pushed ${INGESTION_IMAGE}:${IMAGE_TAG}"
|
||||||
|
|
||||||
- name: Build and push routing image
|
|
||||||
run: |
|
|
||||||
set -e
|
|
||||||
trap 'rm -f /tmp/routing-image.tar' EXIT
|
|
||||||
IMAGE_TAG="${{ github.sha }}"
|
|
||||||
echo "Building ${ROUTING_IMAGE}:${IMAGE_TAG}"
|
|
||||||
|
|
||||||
buildctl --addr "${BUILDKIT_HOST}" build \
|
|
||||||
--frontend dockerfile.v0 \
|
|
||||||
--local context=. \
|
|
||||||
--local dockerfile=. \
|
|
||||||
--opt filename=Dockerfile.routing \
|
|
||||||
--opt build-arg:VERSION="${IMAGE_TAG}" \
|
|
||||||
--output type=oci,dest=/tmp/routing-image.tar
|
|
||||||
|
|
||||||
skopeo copy \
|
|
||||||
oci-archive:/tmp/routing-image.tar \
|
|
||||||
docker://${ROUTING_IMAGE}:${IMAGE_TAG} \
|
|
||||||
--dest-creds "${{ secrets.REGISTRY_CREDS }}"
|
|
||||||
|
|
||||||
echo "Built and pushed ${ROUTING_IMAGE}:${IMAGE_TAG}"
|
|
||||||
|
|
||||||
- name: Update infra repo
|
- name: Update infra repo
|
||||||
run: |
|
run: |
|
||||||
set -e
|
set -e
|
||||||
trap 'rm -rf /tmp/infra-update; rm -f ~/.ssh/infra_deploy_key' EXIT
|
trap 'rm -rf /tmp/infra-update; rm -f ~/.ssh/infra_deploy_key' EXIT
|
||||||
IMAGE_TAG="${{ github.sha }}"
|
IMAGE_TAG="${{ github.sha }}"
|
||||||
|
# infra#311: github.sha is never empty in practice, but the manifest
|
||||||
|
# patch below is the same shape that took Flux down estate-wide on
|
||||||
|
# 2026-08-25 -- an empty tag yields `image: <repo>:`, which is invalid
|
||||||
|
# YAML and fails the ENTIRE Kustomization, not just this app. The
|
||||||
|
# assert costs nothing and removes the class.
|
||||||
|
[ -n "${IMAGE_TAG:-}" ] || { echo "IMAGE_TAG is empty - refusing to patch"; exit 1; }
|
||||||
mkdir -p ~/.ssh
|
mkdir -p ~/.ssh
|
||||||
echo "${{ secrets.INFRA_DEPLOY_KEY }}" > ~/.ssh/infra_deploy_key
|
echo "${{ secrets.INFRA_DEPLOY_KEY }}" > ~/.ssh/infra_deploy_key
|
||||||
chmod 600 ~/.ssh/infra_deploy_key
|
chmod 600 ~/.ssh/infra_deploy_key
|
||||||
printf 'Host git.d-ma.be\n HostName 127.0.0.1\n Port 30022\n StrictHostKeyChecking no\n' >> ~/.ssh/config
|
# In-cluster DNS to gitea's SSH NodePort service, not 127.0.0.1:30022
|
||||||
|
# (that only worked when act_runner ran on koala's bare host network;
|
||||||
GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes" \
|
# from inside the containerized runner's own pod netns, loopback
|
||||||
|
# never reaches the host — "Connection refused", found 2026-07-27).
|
||||||
|
#
|
||||||
|
# Pass as -o overrides on the ssh invocation itself, NOT appended to
|
||||||
|
# ~/.ssh/config: $HOME (/data) is a PVC that persists across job
|
||||||
|
# runs on this runner (same "workspace not ephemeral" class as
|
||||||
|
# brain: act-runner-host-executor-tmp-persists), so an appended
|
||||||
|
# line here would pile up duplicate `Host git.d-ma.be` blocks
|
||||||
|
# across every run — ssh_config is first-match-wins, so a stale
|
||||||
|
# entry from an earlier failed run would silently shadow this
|
||||||
|
# fix forever (exactly what happened once already: this fix's
|
||||||
|
# own first attempt got appended AFTER an already-stale entry
|
||||||
|
# and lost). CLI -o options always win regardless of file state,
|
||||||
|
# so this step is safe to re-run any number of times.
|
||||||
|
GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes -o HostName=gitea-ssh-nodeport.gitea.svc.cluster.local -o Port=22 -o StrictHostKeyChecking=no" \
|
||||||
git clone "${INFRA_REPO}" /tmp/infra-update
|
git clone "${INFRA_REPO}" /tmp/infra-update
|
||||||
|
|
||||||
cd /tmp/infra-update
|
cd /tmp/infra-update
|
||||||
@@ -81,18 +78,14 @@ jobs:
|
|||||||
sed -i "s|git.d-ma.be/mathias/ingestion:.*|git.d-ma.be/mathias/ingestion:${IMAGE_TAG}|" \
|
sed -i "s|git.d-ma.be/mathias/ingestion:.*|git.d-ma.be/mathias/ingestion:${IMAGE_TAG}|" \
|
||||||
"k3s/apps/supervisor/ingestion-deployment.yaml"
|
"k3s/apps/supervisor/ingestion-deployment.yaml"
|
||||||
|
|
||||||
sed -i "s|git.d-ma.be/mathias/routing:.*|git.d-ma.be/mathias/routing:${IMAGE_TAG}|" \
|
|
||||||
"k3s/apps/routing/deployment.yaml"
|
|
||||||
|
|
||||||
git config user.email "cd-bot@d-ma.be"
|
git config user.email "cd-bot@d-ma.be"
|
||||||
git config user.name "CD Bot"
|
git config user.name "CD Bot"
|
||||||
git add "k3s/apps/supervisor/ingestion-deployment.yaml" \
|
git add "k3s/apps/supervisor/ingestion-deployment.yaml"
|
||||||
"k3s/apps/routing/deployment.yaml"
|
git commit -m "chore(deploy): ingestion → ${IMAGE_TAG}"
|
||||||
git commit -m "chore(deploy): ingestion+routing → ${IMAGE_TAG}"
|
|
||||||
GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes" \
|
GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes" \
|
||||||
git push
|
git push
|
||||||
|
|
||||||
echo "Infra repo updated: ingestion+routing → ${IMAGE_TAG}"
|
echo "Infra repo updated: ingestion → ${IMAGE_TAG}"
|
||||||
|
|
||||||
- name: Trigger Flux reconcile (immediate)
|
- name: Trigger Flux reconcile (immediate)
|
||||||
run: |
|
run: |
|
||||||
@@ -132,35 +125,3 @@ jobs:
|
|||||||
kubectl describe pods -n supervisor -l app=ingestion | tail -40
|
kubectl describe pods -n supervisor -l app=ingestion | tail -40
|
||||||
exit 1
|
exit 1
|
||||||
}
|
}
|
||||||
|
|
||||||
- name: Wait for Flux to apply new routing image
|
|
||||||
run: |
|
|
||||||
EXPECTED="git.d-ma.be/mathias/routing:${{ github.sha }}"
|
|
||||||
for i in $(seq 1 60); do
|
|
||||||
CURRENT=$(kubectl get deploy routing -n routing \
|
|
||||||
-o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null || echo "")
|
|
||||||
if [ "$CURRENT" = "$EXPECTED" ]; then
|
|
||||||
echo "✓ Flux applied routing image after ${i}s"
|
|
||||||
break
|
|
||||||
fi
|
|
||||||
sleep 1
|
|
||||||
done
|
|
||||||
kubectl get deploy routing -n routing \
|
|
||||||
-o jsonpath='{.spec.template.spec.containers[0].image}' \
|
|
||||||
| grep -qx "$EXPECTED" \
|
|
||||||
|| { echo "✗ Flux did not apply routing image within 60s"; exit 1; }
|
|
||||||
|
|
||||||
- name: Verify routing rollout
|
|
||||||
run: |
|
|
||||||
kubectl rollout status deployment/routing \
|
|
||||||
--namespace routing \
|
|
||||||
--timeout=120s \
|
|
||||||
|| {
|
|
||||||
echo "── pod status ──"
|
|
||||||
kubectl get pods -n routing -o wide
|
|
||||||
echo "── events ──"
|
|
||||||
kubectl get events -n routing --sort-by='.lastTimestamp' | tail -20
|
|
||||||
echo "── describe ──"
|
|
||||||
kubectl describe pods -n routing -l app=routing | tail -40
|
|
||||||
exit 1
|
|
||||||
}
|
|
||||||
|
|||||||
Executable
+34
@@ -0,0 +1,34 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Pre-commit secret scan for the hyperguild repo.
|
||||||
|
#
|
||||||
|
# task check -> security:gitleaks enforces this in CI, but CI is after the
|
||||||
|
# push. This catches a staged secret before it ever leaves the machine --
|
||||||
|
# the infra#39 rotation pass found exactly this class of leak (a printed
|
||||||
|
# credential, committed, archived into git) and there was no local gate to
|
||||||
|
# have caught it. Mirrors infra's own .githooks/pre-commit (infra#326).
|
||||||
|
#
|
||||||
|
# Enable once per clone: git config core.hooksPath .githooks
|
||||||
|
# Bypass if absolutely needed: git commit --no-verify
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
GITLEAKS="$(command -v gitleaks || true)"
|
||||||
|
if [[ -z "$GITLEAKS" && -x "$(go env GOPATH 2>/dev/null)/bin/gitleaks" ]]; then
|
||||||
|
GITLEAKS="$(go env GOPATH)/bin/gitleaks"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ -z "$GITLEAKS" ]]; then
|
||||||
|
echo "⚠ gitleaks not installed — skipping local secret scan (CI still enforces it)."
|
||||||
|
echo " Install: go install github.com/zricethezav/gitleaks/v8@latest"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "→ Scanning staged changes for secrets..."
|
||||||
|
if ! "$GITLEAKS" protect --staged --redact --config .gitleaks.toml >/dev/null 2>&1; then
|
||||||
|
echo " ✗ gitleaks found a secret in the staged changes."
|
||||||
|
echo " Review with: gitleaks protect --staged --redact --config .gitleaks.toml -v"
|
||||||
|
echo " If it is a false positive, allowlist it in .gitleaks.toml or add a"
|
||||||
|
echo " '# gitleaks:allow' comment. Do NOT --no-verify a real secret."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo " ✓ no staged secrets"
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
# gitleaks config for the hyperguild repo (infra#39 — leak prevention pass,
|
||||||
|
# Phase 3 checklist item: "gitleaks pre-commit hook in infra AND hyperguild").
|
||||||
|
#
|
||||||
|
# Ported from mathias/infra's .gitleaks.toml (2026-08-04) — same homelab
|
||||||
|
# token-shape rules, minus the SOPS/searxng allowlists infra needed (this
|
||||||
|
# repo doesn't use SOPS).
|
||||||
|
|
||||||
|
title = "hyperguild gitleaks config"
|
||||||
|
|
||||||
|
[extend]
|
||||||
|
useDefault = true
|
||||||
|
|
||||||
|
# --- Homelab-specific rules -------------------------------------------------
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "homelab-static-bearer"
|
||||||
|
description = "Homelab MCP/LLM static bearer or API key assigned a long literal value"
|
||||||
|
regex = '''(?i)\b(DMABE_[A-Z0-9_]+|[A-Z0-9_]*MCP_TOKEN|ROUTING_MCP_TOKEN|INFRA_MCP_TOKEN|BRAIN_MCP_TOKEN|GITEA_MCP_TOKEN|LITELLM_MASTER_KEY|LITELLM_SALT_KEY|DMABE_LLMAPI_KEY|BRAIN_PG_DSN)\s*[:=]\s*['"]?([A-Za-z0-9/_+.\-]{16,})['"]?'''
|
||||||
|
keywords = ["dmabe_", "mcp_token", "litellm_master_key", "litellm_salt_key", "llmapi_key", "brain_pg_dsn"]
|
||||||
|
[[rules.allowlists]]
|
||||||
|
description = "Env indirection is not a literal secret"
|
||||||
|
regexes = [
|
||||||
|
'''os\.environ''',
|
||||||
|
'''valueFrom''',
|
||||||
|
'''secretKeyRef''',
|
||||||
|
'''\$\{?[A-Za-z_][A-Za-z0-9_]*\}?''',
|
||||||
|
'''REDACTED''',
|
||||||
|
'''<[A-Z_]+>''',
|
||||||
|
]
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "homelab-authorization-bearer"
|
||||||
|
description = "Hardcoded Authorization: Bearer header"
|
||||||
|
regex = '''(?i)authorization['"]?\s*[:=]\s*['"]?bearer\s+([A-Za-z0-9/_+.\-=]{16,})'''
|
||||||
|
keywords = ["authorization", "bearer"]
|
||||||
|
[[rules.allowlists]]
|
||||||
|
description = "Env indirection is not a literal secret"
|
||||||
|
regexes = [
|
||||||
|
'''\$\{?[A-Za-z_][A-Za-z0-9_]*\}?''',
|
||||||
|
'''os\.environ''',
|
||||||
|
'''REDACTED''',
|
||||||
|
'''<[A-Z_]+>''',
|
||||||
|
]
|
||||||
|
|
||||||
|
# --- Global allowlist: claudewatcher's own scrubber test fixtures ------------
|
||||||
|
# ingestion/internal/claudewatcher/{scrubber,watcher}_test.go deliberately
|
||||||
|
# contain fake secret-shaped literals to test that the scrubber detects and
|
||||||
|
# redacts them. Verified 2026-08-04: all 9 findings here are test fixtures
|
||||||
|
# (github-pat, jwt, generic-api-key, homelab-authorization-bearer rules) plus
|
||||||
|
# 1 doc finding that was gitleaks matching the literal placeholder word
|
||||||
|
# "REDACTED" in a plan doc — not a real secret in either case.
|
||||||
|
[[allowlists]]
|
||||||
|
description = "claudewatcher scrubber test fixtures — deliberately fake secrets"
|
||||||
|
paths = [
|
||||||
|
'''ingestion/internal/claudewatcher/scrubber_test\.go$''',
|
||||||
|
'''ingestion/internal/claudewatcher/watcher_test\.go$''',
|
||||||
|
]
|
||||||
|
|
||||||
|
# infra#344: the allowlist that used to sit here is gone, and so is the reason
|
||||||
|
# for it. It suppressed docs/superpowers/plans/2026-04-22-phase4-attempt-wiring.md
|
||||||
|
# on the stated grounds that "extracted Secret == 'REDACTED' exactly ... the
|
||||||
|
# placeholder text itself, not a real credential" (verified 2026-08-04). That was
|
||||||
|
# not what the file contained: line 1054 carried a real 40-char hex Gitea PAT in
|
||||||
|
# an `Authorization: token ...` header. Confirmed dead by side-effect — GET
|
||||||
|
# /api/v1/user with it returns 401 — and redacted in that file, so there is now
|
||||||
|
# nothing to allowlist.
|
||||||
|
#
|
||||||
|
# The `paths` entry also made the block far broader than its description: gitleaks
|
||||||
|
# ORs global allowlist clauses and ignores `condition = "AND"`, so the whole file
|
||||||
|
# was immune, not just lines matching /REDACTED/. A live token added to that doc
|
||||||
|
# would have been suppressed on the strength of a note about a different value.
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# Historical gitleaks findings, dispositioned (infra#326 / infra#317).
|
||||||
|
#
|
||||||
|
# WHY THIS FILE AND NOT .gitleaks.toml:
|
||||||
|
# This is a finding in git HISTORY, surfaced by the full-history audit
|
||||||
|
# (`task security:gitleaks:history`). The working-tree scan that gates commits
|
||||||
|
# (`task security:gitleaks`, the pre-commit hook, CI) is clean. Suppressing by
|
||||||
|
# PATH in .gitleaks.toml would also blind the working-tree scan for that file
|
||||||
|
# forever; a fingerprint pins one finding in one commit, so a NEW secret in the
|
||||||
|
# same file still fires.
|
||||||
|
#
|
||||||
|
# History is not rewritten here, so the remediation for a real leak is
|
||||||
|
# ROTATION -- making the committed value worthless -- never scrubbing.
|
||||||
|
#
|
||||||
|
# Dispositioned 2026-08-29. Re-check with: task security:gitleaks:history
|
||||||
|
|
||||||
|
# --- Real credential, ALREADY DEAD -------------------------------------------
|
||||||
|
# A Gitea PAT pasted into a CD-watching command in a planning doc
|
||||||
|
# (`Authorization: token <40-hex>`, last_eight 3105a670).
|
||||||
|
#
|
||||||
|
# Verified dead 2026-08-29: authenticating against
|
||||||
|
# GET https://git.d-ma.be/api/v1/user with this exact value returns 401, so the
|
||||||
|
# committed token no longer grants anything. It was almost certainly invalidated
|
||||||
|
# by the early-August rotation of the `mathias` PATs (the same rotation whose
|
||||||
|
# fan-out caused infra#260, where CI copies of the old value kept failing).
|
||||||
|
#
|
||||||
|
# No rotation action outstanding. Kept as a fingerprint rather than removed from
|
||||||
|
# history because history cannot be rewritten here.
|
||||||
|
#
|
||||||
|
# The generalisable lesson is recorded separately: a credential printed into a
|
||||||
|
# doc/transcript is permanent, so the only remedy is rotating it -- which in
|
||||||
|
# this case had already happened for unrelated reasons, by luck rather than by
|
||||||
|
# the leak being noticed.
|
||||||
|
c9310b1079478bf4430b0a2dfb25445d7ccc9b59:docs/superpowers/plans/2026-04-22-phase4-attempt-wiring.md:curl-auth-header:1053
|
||||||
@@ -6,6 +6,13 @@
|
|||||||
"headers": {
|
"headers": {
|
||||||
"Authorization": "Bearer ${BRAIN_MCP_TOKEN}"
|
"Authorization": "Bearer ${BRAIN_MCP_TOKEN}"
|
||||||
}
|
}
|
||||||
|
},
|
||||||
|
"gitea": {
|
||||||
|
"type": "http",
|
||||||
|
"url": "https://git-mcp.d-ma.be/mcp",
|
||||||
|
"headers": {
|
||||||
|
"Authorization": "Bearer ${GITEA_MCP_TOKEN}"
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -48,14 +48,30 @@ These rules apply to every task across every project, regardless of harness.
|
|||||||
logical change (one tool, one fix, one test) with passing tests. Main is always
|
logical change (one tool, one fix, one test) with passing tests. Main is always
|
||||||
deployable. Never create long-lived feature branches.
|
deployable. Never create long-lived feature branches.
|
||||||
|
|
||||||
**Exception — parallel agents on same repo:** If another agent is known to be
|
**The PR trigger is RISK, not concurrency.** `risk/low` work commits straight to
|
||||||
actively working on the same repo simultaneously, create a short-lived branch
|
main with CI as the gate — that is most work and is unchanged. `risk/medium` and
|
||||||
(`agent/<description>`), finish the task, and merge to main within the same
|
above opens a PR, even solo, even with nobody waiting to review it. This mirrors
|
||||||
session. Do not leave agent branches open between sessions.
|
the tiered merge model in `assessor-loop`
|
||||||
|
`method/transfer-to-a-regulated-organisation.md` §4.1, so the estate practises
|
||||||
|
what it sells instead of merging everything on green and claiming otherwise.
|
||||||
|
|
||||||
**Exception — external contributor or client four-eyes requirement:** Use
|
**Concurrency is NOT a reason to branch.** Measured 2026-08-29 across three real
|
||||||
PR flow only when a human reviewer outside the project is required. Document
|
collisions between parallel sessions: two were not branching problems at all — a
|
||||||
the reason in PROJECT.md.
|
generated artifact written to the same path by both (branching turns that into a
|
||||||
|
merge conflict on a 33k-line JSON), and uncommitted work in a shared checkout
|
||||||
|
(a per-session worktree fixes it) — and the third was an ordinary non-fast-forward
|
||||||
|
push resolved by rebase in seconds. Use a worktree per session on shared repos and
|
||||||
|
timestamp generated artifacts. Do not reach for a branch.
|
||||||
|
|
||||||
|
Branch+PR flow carries documented costs on solo repos: merging auto-closes gated
|
||||||
|
issues, review comments live on a thread separate from the issue's, and a
|
||||||
|
merged-and-deleted branch cannot be amended (brain:
|
||||||
|
`gitea-pr-merge-autoclose-and-review-comment-footguns`). Those are the price of
|
||||||
|
the `risk/medium`+ lane — paid deliberately, not incurred by accident.
|
||||||
|
|
||||||
|
**Exception — external contributor or client four-eyes requirement:** Use PR flow
|
||||||
|
when a human reviewer outside the project is required. Document the reason in
|
||||||
|
PROJECT.md.
|
||||||
|
|
||||||
6. **Close the loop — every substantive task ends with the same ritual.** Shipping
|
6. **Close the loop — every substantive task ends with the same ritual.** Shipping
|
||||||
the code is not the end of the task; capturing it is. Run this unprompted:
|
the code is not the end of the task; capturing it is. Run this unprompted:
|
||||||
@@ -73,6 +89,26 @@ These rules apply to every task across every project, regardless of harness.
|
|||||||
- Surface the brain entries and issue numbers in the closing summary so the
|
- Surface the brain entries and issue numbers in the closing summary so the
|
||||||
trail is auditable.
|
trail is auditable.
|
||||||
|
|
||||||
|
**Chat is not a durable store. Never rely on Mathias remembering anything.**
|
||||||
|
Every gap, caveat, risk, open question and next step must land in one of four
|
||||||
|
places before the turn ends: **the brain**, **a repo** (code, docs, ADR),
|
||||||
|
**a tracker issue**, or **a plan file**. A finding stated only in conversation is
|
||||||
|
lost the moment the session closes — and this estate has repeatedly proved that a
|
||||||
|
remembered convention is an unreliable control.
|
||||||
|
|
||||||
|
This applies hardest to what is tempting to leave as a closing remark: "worth
|
||||||
|
knowing X isn't enforced yet", "this is a convention, not a gate", "we should
|
||||||
|
revisit Y". Those are exactly the items that need a locator. If it is worth
|
||||||
|
saying in a summary, it is worth writing where it can be found again without you.
|
||||||
|
|
||||||
|
## CAD sprint patterns
|
||||||
|
|
||||||
|
Two CAD sprint archetypes (defs: brain `wiki/homelab/decisions/continuous-agentic-development-cad-concept-2026-06-16`):
|
||||||
|
- **Inception Sprint** — birth a new repo. Oath-gated close; tag only once every clause holds. Brain: `.../inception-sprint-and-oath`.
|
||||||
|
- **Sweep Sprint** — clear an existing backlog. Parallel read-only research per issue → sequential one-issue-per-commit execution. Brain: `.../sweep-sprint-and-oath`.
|
||||||
|
|
||||||
|
Shared: decisions that are yours, not the agent's, surface as `DECISION NEEDED` (question/options/recommendation/blocks) — ask live if you're present, else comment and wait, never act past it. Live-breaking regressions get `git revert`, never forward-fixed; host-network work skips even read-only research (solo, synchronous, you present). Rule 6's closing ritual only fires once every Oath clause genuinely holds — blocked = descoped-and-tracked, never marked done.
|
||||||
|
|
||||||
## Default stack
|
## Default stack
|
||||||
|
|
||||||
| Layer | Default | Fallback | Last resort |
|
| Layer | Default | Fallback | Last resort |
|
||||||
@@ -83,7 +119,7 @@ These rules apply to every task across every project, regardless of harness.
|
|||||||
| Containers | Docker Compose (dev), k3s (prod) | — | — |
|
| Containers | Docker Compose (dev), k3s (prod) | — | — |
|
||||||
| DB | PostgreSQL + sqlc | SQLite | — |
|
| DB | PostgreSQL + sqlc | SQLite | — |
|
||||||
| Search | pgvector (vector), BM25 | Qdrant (when >1M vectors or hybrid retrieval) | — |
|
| Search | pgvector (vector), BM25 | Qdrant (when >1M vectors or hybrid retrieval) | — |
|
||||||
| Logging | slog (structured) | — | — |
|
| Logging | slog (structured) | stdlib `logging` w/ structured `extra=` (or structlog) | — |
|
||||||
| Testing | Table-driven, testify | — | — |
|
| Testing | Table-driven, testify | — | — |
|
||||||
| Agents (Go) | google.golang.org/adk + pkg/litellm adapter | — | — |
|
| Agents (Go) | google.golang.org/adk + pkg/litellm adapter | — | — |
|
||||||
|
|
||||||
@@ -92,7 +128,12 @@ Exploratory: Rust, Zig — I'll tell you when I want these.
|
|||||||
## Code conventions
|
## Code conventions
|
||||||
|
|
||||||
- **Go style**: golines, gofumpt, golangci-lint
|
- **Go style**: golines, gofumpt, golangci-lint
|
||||||
- **Errors**: `fmt.Errorf("operation: %w", err)` — never naked, never log-and-return
|
- **Python style** (fallback language): ruff (format+lint, one tool), mypy --strict (non-negotiable,
|
||||||
|
matches Go's static typing discipline), pytest + pytest-cov (table-driven via
|
||||||
|
`@pytest.mark.parametrize`), uv (venv+deps+lock, one tool), pydantic-settings (typed env-var config
|
||||||
|
— same principle as Go's typed structs), src-layout + `pyproject.toml` only (no `setup.py`)
|
||||||
|
- **Errors**: `fmt.Errorf("operation: %w", err)` — never naked, never log-and-return.
|
||||||
|
Python: `raise X from e` (exception chaining, same principle) — never bare `except`, never silent `pass`
|
||||||
- **Naming**: stdlib conventions, no stuttering
|
- **Naming**: stdlib conventions, no stuttering
|
||||||
- **Architecture**: prefer stdlib over frameworks, constructor injection, env-var config parsed into typed structs
|
- **Architecture**: prefer stdlib over frameworks, constructor injection, env-var config parsed into typed structs
|
||||||
- **Git**: conventional commits (`feat:`, `fix:`, `chore:`), commit directly to main,
|
- **Git**: conventional commits (`feat:`, `fix:`, `chore:`), commit directly to main,
|
||||||
@@ -130,10 +171,10 @@ Three machines on Tailscale:
|
|||||||
|---------|------|-----------|
|
|---------|------|-----------|
|
||||||
| koala | GPU inference, heavy compute | RTX 5070, runs k3s + llama-swap + shared postgres18/pgvector |
|
| koala | GPU inference, heavy compute | RTX 5070, runs k3s + llama-swap + shared postgres18/pgvector |
|
||||||
| iguana | Services, builds | M2 Ultra Mac |
|
| iguana | Services, builds | M2 Ultra Mac |
|
||||||
| flamingo | Daily driver, edge | Mac mini, ~/dev is here |
|
| flamingo | Daily driver, edge | Apple Silicon Mac, ~/dev is here |
|
||||||
|
|
||||||
- **Model routing**: LiteLLM in front of llama-swap (local) + cloud APIs (when permitted)
|
- **Model routing**: LiteLLM in front of llama-swap (local) + cloud APIs (when permitted)
|
||||||
- **Orchestration**: k3s cluster across all three machines
|
- **Orchestration**: k3s on koala only — single control-plane node, no workers (verified 2026-08-30). iguana and flamingo are on the tailnet but are not cluster members
|
||||||
- **Networking**: Tailscale mesh
|
- **Networking**: Tailscale mesh
|
||||||
|
|
||||||
## Project landscape
|
## Project landscape
|
||||||
@@ -229,10 +270,11 @@ When working on a project tagged with a client name:
|
|||||||
|
|
||||||
## Harness-agnostic principles
|
## Harness-agnostic principles
|
||||||
|
|
||||||
This context is designed to work with any AI coding tool:
|
Supported harnesses (2026-09-01) — Cursor and Aider were dropped; do not
|
||||||
- Claude Code, Cursor, Aider, Open WebUI, Charmbracelet Mods/Crush
|
re-add adapters for them:
|
||||||
- Pi Coding Agent, Mistral Vibe, Antigravity
|
- Claude Code, Antigravity, OpenCode, Mistral Vibe, Crush, Pi Coding Agent
|
||||||
- Any tool that accepts a system prompt or reads a markdown context file
|
- Open WebUI, Charmbracelet Mods, or any tool that takes a system prompt string
|
||||||
|
- Any other tool that reads `AGENTS.md`, which is the de facto convention
|
||||||
|
|
||||||
The canonical source is always `.context/AGENT.md` (root) and `.context/PROJECT.md` (per-project).
|
The canonical source is always `.context/AGENT.md` (root) and `.context/PROJECT.md` (per-project).
|
||||||
Derived files are committed (see *How context propagates* below) so a `git pull` on any host yields full agent context with no setup.
|
Derived files are committed (see *How context propagates* below) so a `git pull` on any host yields full agent context with no setup.
|
||||||
@@ -244,15 +286,29 @@ Canonical sources of truth:
|
|||||||
- Project: `<repo>/.context/PROJECT.md` (per-repo)
|
- Project: `<repo>/.context/PROJECT.md` (per-repo)
|
||||||
|
|
||||||
Derived files (committed, regenerated by `task context:sync`):
|
Derived files (committed, regenerated by `task context:sync`):
|
||||||
- `CLAUDE.md`, `AGENTS.md`, `.cursorrules`, `.aider.conventions.md`,
|
- `CLAUDE.md`, `AGENTS.md`, `.context/system-prompt.txt`
|
||||||
`.context/system-prompt.txt`
|
|
||||||
|
Repos still carrying `.cursorrules`, `.aider.conventions.md` or
|
||||||
|
`.aider.conf.yml` are stale: nothing regenerates them. Delete them the next time
|
||||||
|
you touch the repo, along with their entries in that repo's `task check`
|
||||||
|
assertion.
|
||||||
|
|
||||||
Workflow:
|
Workflow:
|
||||||
1. Edit a canonical file. Run `task context:sync`. Commit canonical and
|
1. Edit a canonical file. Run `task context:sync`. Commit canonical and
|
||||||
derived together. Push.
|
derived together. Push.
|
||||||
2. On any other host, `git pull` brings both. Claude Code (tree-walking)
|
2. On any other host, `git pull` brings both. Which file a harness reads does
|
||||||
uses `CLAUDE.md`; Crush / Pi / Antigravity (cwd-only) use `AGENTS.md`;
|
NOT split cleanly by lookup strategy, so do not reason from the strategy:
|
||||||
Cursor uses `.cursorrules`; Aider uses `.aider.conventions.md`.
|
- `CLAUDE.md` — Claude Code (walks up the tree)
|
||||||
|
- `AGENTS.md` — Crush, Pi, Antigravity (cwd-only), OpenCode (project root,
|
||||||
|
global at `~/.config/opencode/AGENTS.md`), and Mistral Vibe, which **walks
|
||||||
|
up** from cwd and takes the first `AGENTS.md` it finds
|
||||||
|
(user-level `~/.vibe/AGENTS.md`)
|
||||||
|
- `.context/system-prompt.txt` — Open WebUI, Mods, raw API calls; not read by
|
||||||
|
any harness, it is for pasting
|
||||||
|
|
||||||
|
Vibe tree-walks for `AGENTS.md`, so "tree-walkers use CLAUDE.md" is false.
|
||||||
|
Each repo's `AGENTS.md` embeds the canonical and is the first one found from
|
||||||
|
inside that repo, so this works — but by construction, not by luck.
|
||||||
3. `task check` runs `context:sync` then asserts `git status --porcelain`
|
3. `task check` runs `context:sync` then asserts `git status --porcelain`
|
||||||
is empty over the derived files (catches both modified-tracked drift
|
is empty over the derived files (catches both modified-tracked drift
|
||||||
and missing-untracked adapters). A drift fails the check with a
|
and missing-untracked adapters). A drift fails the check with a
|
||||||
@@ -263,7 +319,20 @@ unconditionally on every host, every harness.
|
|||||||
|
|
||||||
## Engineering Skills
|
## Engineering Skills
|
||||||
|
|
||||||
Shared engineering skills are available in `~/dev/.skills/`. Load at task start — not "on demand" but on schedule, before writing code. See `~/dev/.skills/SKILLS_INDEX.md` for the full list.
|
Shared engineering skills live in the **`mathias/skills`** repo (`git.d-ma.be/mathias/skills`). Load at task start — not "on demand" but on schedule, before writing code. Browse `SKILLS_INDEX.md` for the full list.
|
||||||
|
|
||||||
|
**Claude Code** installs the repo as a plugin from the marketplace the same repo hosts. That path also carries the `secret-guard` and `pre-task-ritual` hooks and the brain/gitea/infra MCP servers, which symlinks cannot:
|
||||||
|
|
||||||
|
```
|
||||||
|
/plugin marketplace add https://git.d-ma.be/mathias/skills.git
|
||||||
|
/plugin install skills@mathias
|
||||||
|
```
|
||||||
|
|
||||||
|
Skills are **namespaced** under Claude Code: load `tdd` as `skills:tdd`. The trigger table below uses bare names — prefix them.
|
||||||
|
|
||||||
|
**Crush, Pi and OpenCode** have no plugin system and are wired by `install.sh` as one directory symlink per harness via `~/.agents/skills`. Do not clone the checkout into `~/dev`: that folder is iCloud-shared between the Macs and spans two path roots (`/Users/mathias` vs `/home/mathias` on koala), so a checkout there is a git working tree with two concurrent writers, and any absolute symlink stored in it is broken on one side by construction (skills#23). Leave `SKILLS_CHECKOUT_DIR` at its host-local default.
|
||||||
|
|
||||||
|
Check a host with `task doctor` in the skills repo; it also runs automatically at session start via the plugin.
|
||||||
|
|
||||||
**Skill trigger table — load before starting, not after getting stuck:**
|
**Skill trigger table — load before starting, not after getting stuck:**
|
||||||
|
|
||||||
|
|||||||
@@ -1,30 +0,0 @@
|
|||||||
# syntax=docker/dockerfile:1
|
|
||||||
|
|
||||||
# ── Build stage ───────────────────────────────────────────────────────────────
|
|
||||||
FROM golang:1.26-bookworm AS builder
|
|
||||||
|
|
||||||
ARG VERSION=dev
|
|
||||||
WORKDIR /src
|
|
||||||
|
|
||||||
COPY go.mod go.sum ./
|
|
||||||
RUN go mod download
|
|
||||||
|
|
||||||
COPY . .
|
|
||||||
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
|
|
||||||
go build -trimpath -ldflags="-s -w -X main.version=${VERSION}" \
|
|
||||||
-o /out/routing ./cmd/routing
|
|
||||||
|
|
||||||
# ── Runtime stage ─────────────────────────────────────────────────────────────
|
|
||||||
FROM gcr.io/distroless/base-debian12
|
|
||||||
|
|
||||||
COPY --from=builder /out/routing /usr/local/bin/routing
|
|
||||||
COPY config/ /app/config/
|
|
||||||
|
|
||||||
ENV SUPERVISOR_CONFIG_DIR=/app/config/supervisor
|
|
||||||
ENV ROUTING_PORT=3210
|
|
||||||
|
|
||||||
EXPOSE 3210
|
|
||||||
|
|
||||||
USER 65532:65532
|
|
||||||
|
|
||||||
ENTRYPOINT ["/usr/local/bin/routing"]
|
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
# Icebox — retired code (recoverable, not destroyed)
|
||||||
|
|
||||||
|
Per issue #75 (consolidate to a single harness), the routing-pod path was removed
|
||||||
|
from the live tree. It is **preserved and recoverable**, not deleted without trace.
|
||||||
|
|
||||||
|
## What was iceboxed (2026-07-01, issue #75)
|
||||||
|
|
||||||
|
| Path | Why |
|
||||||
|
|------|-----|
|
||||||
|
| `cmd/routing/` | The routing MCP-server binary; every skill call was wrapped through the broken pass-rate router (`wrap(skillName)`). |
|
||||||
|
| `internal/routing/` | Router / Fetcher / Policy / pass-rate. Signal is survivorship-biased and unusable as-is (infra#174). |
|
||||||
|
| `internal/skills/{review,debug,retrospective,trainer,project}/` | Skill handlers usable **only** through `cmd/routing` (verified: each imported solely by `cmd/routing`). |
|
||||||
|
| `Dockerfile.routing` | Built `cmd/routing` exclusively. |
|
||||||
|
| `.gitea/workflows/cd.yml` (routing steps only) | Removed the routing image build + infra image-bump + Flux-wait/rollout-verify for routing; **ingestion build/deploy is unchanged**. |
|
||||||
|
|
||||||
|
## Why
|
||||||
|
|
||||||
|
The live minimal harness is `cmd/hyperguild` + `brain-mcp` (+ `gitea-mcp` available) —
|
||||||
|
that is what ran the infra#170 loop-1 experiment (routing/injection machinery off) and
|
||||||
|
closed it twice. `cmd/hyperguild` has **zero** transitive dependency on `internal/routing`
|
||||||
|
or `cmd/routing`'s packages (it imports only `internal/tier`). The routing pass-rate signal
|
||||||
|
is being retired, not resurrected (fresh start, per infra#174).
|
||||||
|
|
||||||
|
## How to recover
|
||||||
|
|
||||||
|
Everything above is preserved at commit `00e5f62` under:
|
||||||
|
|
||||||
|
- **tag** `icebox/cmd-routing-2026-07-01`
|
||||||
|
- **branch** `icebox/cmd-routing`
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# inspect
|
||||||
|
git checkout icebox/cmd-routing-2026-07-01
|
||||||
|
|
||||||
|
# restore specific packages onto a branch
|
||||||
|
git checkout icebox/cmd-routing-2026-07-01 -- cmd/routing internal/routing \
|
||||||
|
internal/skills/review internal/skills/debug internal/skills/retrospective \
|
||||||
|
internal/skills/trainer internal/skills/project Dockerfile.routing
|
||||||
|
```
|
||||||
|
|
||||||
|
## Deliberately NOT touched here (separate scope)
|
||||||
|
|
||||||
|
- **Live k8s routing deployment** (`infra` repo, `k3s/apps/routing/`) still runs its last
|
||||||
|
image; CD no longer rebuilds/redeploys it. Tearing down that deployment is a separate
|
||||||
|
infra-repo task.
|
||||||
|
- **`internal/skills/{brain,org,sessionlog}/`** — kept per #75; already had no importer
|
||||||
|
(orphaned before this cut), harmless, compile + test green.
|
||||||
|
- **`config/supervisor/{review,debug,retrospective,trainer-*}.md`** — routing skill prompts,
|
||||||
|
now orphaned data; left in place (not code, no build impact).
|
||||||
@@ -5,6 +5,11 @@ Instead of letting Claude Code do whatever it wants, hyperguild enforces structu
|
|||||||
workflows (TDD red/green/refactor), logs every session, and accumulates learnings
|
workflows (TDD red/green/refactor), logs every session, and accumulates learnings
|
||||||
into a searchable brain.
|
into a searchable brain.
|
||||||
|
|
||||||
|
**This repo is also where `brain-mcp` lives** — the MCP server at
|
||||||
|
`brain-mcp.d-ma.be` that `brain_query`, `brain_write`, `brain_read`, `brain_edit`
|
||||||
|
and `capture` are served from. It is not a separate deployable; see
|
||||||
|
[Where brain-mcp lives](#where-brain-mcp-lives).
|
||||||
|
|
||||||
## Hypothesis
|
## Hypothesis
|
||||||
|
|
||||||
> We believe routing skill tasks through local models, backed by brain context,
|
> We believe routing skill tasks through local models, backed by brain context,
|
||||||
@@ -20,7 +25,7 @@ usage, the fast-model routing path should be reconsidered.
|
|||||||
|
|
||||||
**hyperguild = Claude Code + MCP.** This is a supervisor for Claude Code sessions specifically.
|
**hyperguild = Claude Code + MCP.** This is a supervisor for Claude Code sessions specifically.
|
||||||
For multi-agent orchestration (OpenCode + LiteLLM, executor/reviewer pipelines), see
|
For multi-agent orchestration (OpenCode + LiteLLM, executor/reviewer pipelines), see
|
||||||
[agentsquad](http://gitea.d-ma.be/mathias/agentsquad) — a separate harness for a different
|
[agentsquad](https://git.d-ma.be/mathias/agentsquad) — a separate harness for a different
|
||||||
orchestration model. Skills (mathias/skills) are shared between both.
|
orchestration model. Skills (mathias/skills) are shared between both.
|
||||||
|
|
||||||
## How it works
|
## How it works
|
||||||
@@ -44,6 +49,33 @@ brain/
|
|||||||
└── knowledge/ — legacy flat notes (migration pending: hyperguild#22)
|
└── knowledge/ — legacy flat notes (migration pending: hyperguild#22)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Where brain-mcp lives
|
||||||
|
|
||||||
|
`brain-mcp` is served from **`ingestion/internal/mcp`** in this repo, at
|
||||||
|
**`brain-mcp.d-ma.be`**. It is a component *inside* the `ingestion` service, not
|
||||||
|
a standalone deployable — one pod in the `supervisor` namespace serves the MCP
|
||||||
|
endpoint, the REST API, and the claudewatcher together.
|
||||||
|
|
||||||
|
| You are looking for | It is here |
|
||||||
|
|---|---|
|
||||||
|
| The MCP tool surface (`brain_query`, `brain_write`, `brain_read`, `brain_edit`, `capture`, …) | `ingestion/internal/mcp` |
|
||||||
|
| The HTTP/REST API (`/query`, `/write`, `/ingest`) | `ingestion/internal/api` |
|
||||||
|
| Session ingestion from Claude Code transcripts | `ingestion/internal/claudewatcher` |
|
||||||
|
| The brain *content* (notes, wings, halls) | [`mathias/brain`](https://git.d-ma.be/mathias/brain) — a separate repo |
|
||||||
|
| Deployment manifests | `config/supervisor` |
|
||||||
|
|
||||||
|
### Why it is not its own repo
|
||||||
|
|
||||||
|
Asked and decided in [#91](https://git.d-ma.be/mathias/hyperguild/issues/91)
|
||||||
|
(part 2). `internal/mcp` imports 17 of ingestion's other internal packages and
|
||||||
|
has exactly one importer — it is the MCP facade over the service's own
|
||||||
|
internals, so "move it out" resolves to renaming `ingestion`, publishing 17
|
||||||
|
packages as a shared library, or maintaining a shim. The discoverability problem
|
||||||
|
that prompted the question is answered by naming it here instead.
|
||||||
|
|
||||||
|
Full reasoning, including the reversal trigger: brain
|
||||||
|
`wiki/hyperguild/decisions/brain-mcp-stays-in-ingestion-fix-discoverability-by-name`.
|
||||||
|
|
||||||
## Phase 1 tools (available now)
|
## Phase 1 tools (available now)
|
||||||
|
|
||||||
| Tool | What it does |
|
| Tool | What it does |
|
||||||
@@ -58,7 +90,7 @@ brain/
|
|||||||
| `tier` | Returns the current connectivity tier (1=cloud, 2=LAN, 3=offline) |
|
| `tier` | Returns the current connectivity tier (1=cloud, 2=LAN, 3=offline) |
|
||||||
|
|
||||||
> **Note:** `tdd_red/green/refactor` and `spec` were retired in Plan 7 (2026-05-12).
|
> **Note:** `tdd_red/green/refactor` and `spec` were retired in Plan 7 (2026-05-12).
|
||||||
> They are now SKILL.md files in [mathias/skills](http://gitea.d-ma.be/mathias/skills).
|
> They are now SKILL.md files in [mathias/skills](https://git.d-ma.be/mathias/skills).
|
||||||
|
|
||||||
## Connect a project
|
## Connect a project
|
||||||
|
|
||||||
@@ -147,7 +179,7 @@ The routing pod selects models per skill call based on historical pass rate:
|
|||||||
|
|
||||||
## Open issues
|
## Open issues
|
||||||
|
|
||||||
See [issues](http://gitea.d-ma.be/mathias/hyperguild/issues) — key open items:
|
See [issues](https://git.d-ma.be/mathias/hyperguild/issues) — key open items:
|
||||||
|
|
||||||
- **#25** — skills platform overhaul (audit first, then lazy loading + brain feedback loop)
|
- **#25** — skills platform overhaul (audit first, then lazy loading + brain feedback loop)
|
||||||
- **#24** — reduce context burn from skill listing
|
- **#24** — reduce context burn from skill listing
|
||||||
|
|||||||
+65
-10
@@ -86,27 +86,82 @@ tasks:
|
|||||||
check:
|
check:
|
||||||
desc: Run all checks (context freshness + lint + test + vet) across all modules
|
desc: Run all checks (context freshness + lint + test + vet) across all modules
|
||||||
cmds:
|
cmds:
|
||||||
- task: context:sync
|
# Context sync is skipped under CI, matching dma-cli, gitea-mcp, and the
|
||||||
|
# rest of the estate. It is not a preference: context-sync.sh walks UP
|
||||||
|
# the tree for the canonical .context/AGENT.md, which lives in local-dev
|
||||||
|
# and does not exist on a CI runner. Without this guard, the script
|
||||||
|
# silently falls back to "project context only" and regenerates AGENTS.md
|
||||||
|
# and .context/system-prompt.txt shorter than their committed, correct
|
||||||
|
# versions -- so the drift assertion fails every CI run for a purely
|
||||||
|
# environmental reason, not real drift. Verified, not assumed: this is
|
||||||
|
# exactly what was happening here (AGENTS.md 430 lines committed vs. an
|
||||||
|
# 82-line regeneration on the runner).
|
||||||
|
#
|
||||||
|
# The cost is real and worth stating: drift is therefore only ever caught
|
||||||
|
# on a developer machine, or by infra's estate-wide context-sync-drift
|
||||||
|
# CronJob (ADR-0019), which does have canonical access and is the actual
|
||||||
|
# enforcement mechanism now.
|
||||||
- cmd: |
|
- cmd: |
|
||||||
drift=$(git status --porcelain -- AGENTS.md CLAUDE.md .context/system-prompt.txt 2>/dev/null)
|
if [ -n "${CI:-}" ]; then
|
||||||
if [ -n "$drift" ]; then
|
echo "✓ context sync: skipped in CI (no canonical AGENT.md on the runner)"
|
||||||
echo "ERROR: derived adapters drifted from canonical context." >&2
|
else
|
||||||
echo "$drift" >&2
|
bash scripts/context-sync.sh
|
||||||
echo "" >&2
|
drift=$(git status --porcelain -- AGENTS.md CLAUDE.md .context/system-prompt.txt 2>/dev/null)
|
||||||
echo "Run: git add AGENTS.md CLAUDE.md .context/system-prompt.txt" >&2
|
if [ -n "$drift" ]; then
|
||||||
echo " git commit -m 'chore: re-sync context adapters'" >&2
|
echo "ERROR: derived adapters drifted from canonical context." >&2
|
||||||
exit 1
|
echo "$drift" >&2
|
||||||
|
echo "" >&2
|
||||||
|
echo "Run: git add AGENTS.md CLAUDE.md .context/system-prompt.txt" >&2
|
||||||
|
echo " git commit -m 'chore: re-sync context adapters'" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "✓ context: canonical and adapters are in sync"
|
||||||
fi
|
fi
|
||||||
echo "✓ context: canonical and adapters are in sync"
|
|
||||||
- task: lint
|
- task: lint
|
||||||
- task: test
|
- task: test
|
||||||
- task: vet
|
- task: vet
|
||||||
|
- task: security:gitleaks
|
||||||
|
|
||||||
|
# ── Security ─────────────────────────────────────────────
|
||||||
|
security:gitleaks:
|
||||||
|
desc: Scan the working tree for secrets (gitleaks, fail-closed; skipped if gitleaks absent)
|
||||||
|
dir: '{{.ROOT_DIR}}'
|
||||||
|
cmds:
|
||||||
|
- |
|
||||||
|
GL="$(command -v gitleaks || true)"
|
||||||
|
[ -z "$GL" ] && [ -x "$(go env GOPATH 2>/dev/null)/bin/gitleaks" ] && GL="$(go env GOPATH)/bin/gitleaks"
|
||||||
|
if [ -z "$GL" ]; then
|
||||||
|
echo "⚠ gitleaks not installed — skipping secret scan (CI enforces it)."
|
||||||
|
echo " Install: go install github.com/zricethezav/gitleaks/v8@latest"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
"$GL" detect --no-git --redact --config .gitleaks.toml --source .
|
||||||
|
|
||||||
|
security:gitleaks:history:
|
||||||
|
desc: "One-time FULL-HISTORY secret audit (infra#39 rotation pass; not a per-push gate)"
|
||||||
|
dir: '{{.ROOT_DIR}}'
|
||||||
|
cmds:
|
||||||
|
- |
|
||||||
|
GL="$(command -v gitleaks || true)"
|
||||||
|
[ -z "$GL" ] && [ -x "$(go env GOPATH 2>/dev/null)/bin/gitleaks" ] && GL="$(go env GOPATH)/bin/gitleaks"
|
||||||
|
if [ -z "$GL" ]; then
|
||||||
|
echo "gitleaks not installed: go install github.com/zricethezav/gitleaks/v8@latest" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
echo "Scanning FULL git history (redacted). Known historical leaks are expected"
|
||||||
|
echo "until the infra#39 rotation pass completes — triage against the rotation list."
|
||||||
|
"$GL" detect --redact --config .gitleaks.toml
|
||||||
|
|
||||||
lint:
|
lint:
|
||||||
cmds:
|
cmds:
|
||||||
- golangci-lint run ./...
|
- golangci-lint run ./...
|
||||||
- cd ingestion && golangci-lint run ./...
|
- cd ingestion && golangci-lint run ./...
|
||||||
|
|
||||||
|
lint:frontmatter:
|
||||||
|
desc: Report brain notes whose YAML frontmatter does not parse (BRAIN=path, FIX=1 repairs in place)
|
||||||
|
cmds:
|
||||||
|
- cd ingestion && go run ./cmd/frontmatterlint {{if .FIX}}-fix{{end}} {{.BRAIN | default "../../brain"}}
|
||||||
|
|
||||||
test:
|
test:
|
||||||
cmds:
|
cmds:
|
||||||
- go test -race -count=1 ./...
|
- go test -race -count=1 ./...
|
||||||
|
|||||||
@@ -112,16 +112,15 @@ Flags:
|
|||||||
- `--out PATH` — output file (default `./.mcp.json`)
|
- `--out PATH` — output file (default `./.mcp.json`)
|
||||||
- `--force` — overwrite an existing file
|
- `--force` — overwrite an existing file
|
||||||
|
|
||||||
Modes:
|
Modes (all list **brain + gitea** — Gitea is the audit-trail invariant of the
|
||||||
|
consolidated single harness, #75):
|
||||||
|
|
||||||
- **cloud** — brain MCP only. Claude Code with no routing.
|
- **cloud** — brain + gitea MCP.
|
||||||
- **client-local** — brain + routing pod. The `routing` entry points at
|
- **client-local** — brain + gitea MCP. (The former `routing` entry pointing at
|
||||||
`koala:30310/mcp` (the routing pod, deployed in Plan 6). The
|
`koala:30310/mcp` was removed — the routing pod is iceboxed, #75.)
|
||||||
`X-Hyperguild-Mode: client-local` header is forward-compat for future
|
- **sovereign** — brain + gitea, with a `_mode_note` explaining that this mode
|
||||||
modes; the pod treats absent or unknown values as `client-local`.
|
primarily uses Crush + LiteLLM and the `.mcp.json` is a Claude Code fallback
|
||||||
- **sovereign** — brain only, with a `_mode_note` explaining that this
|
for emergency offline use.
|
||||||
mode primarily uses Crush + LiteLLM and the `.mcp.json` is a Claude
|
|
||||||
Code fallback for emergency offline use.
|
|
||||||
|
|
||||||
## Environment
|
## Environment
|
||||||
|
|
||||||
|
|||||||
+26
-19
@@ -59,31 +59,40 @@ func runMode(ctx context.Context, args []string, _ io.Reader, stdout, stderr io.
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// giteaMCPURL is the public Gitea MCP endpoint (OAuth via Dex/Authentik). Gitea
|
||||||
|
// is the audit-trail invariant of the consolidated single harness (#75), so every
|
||||||
|
// mode lists it as an available connection.
|
||||||
|
const giteaMCPURL = "https://git-mcp.d-ma.be/mcp"
|
||||||
|
|
||||||
|
func brainEntry(brainURL string) map[string]any {
|
||||||
|
return map[string]any{
|
||||||
|
"url": brainURL + "/mcp",
|
||||||
|
"description": "Brain MCP — knowledge query, write, ingestion, session log",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func giteaEntry() map[string]any {
|
||||||
|
return map[string]any{
|
||||||
|
"url": giteaMCPURL,
|
||||||
|
"description": "Gitea MCP — issues/PRs/repo ops (audit-trail invariant)",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func modeCloud(brainURL string) map[string]any {
|
func modeCloud(brainURL string) map[string]any {
|
||||||
return map[string]any{
|
return map[string]any{
|
||||||
"mcpServers": map[string]any{
|
"mcpServers": map[string]any{
|
||||||
"brain": map[string]any{
|
"brain": brainEntry(brainURL),
|
||||||
"url": brainURL + "/mcp",
|
"gitea": giteaEntry(),
|
||||||
"description": "Brain MCP — knowledge query, write, ingestion, session log",
|
|
||||||
},
|
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func modeClientLocal(brainURL string) map[string]any {
|
func modeClientLocal(brainURL string) map[string]any {
|
||||||
|
// The routing pod is iceboxed (#75); the consolidated harness is brain + gitea.
|
||||||
return map[string]any{
|
return map[string]any{
|
||||||
"mcpServers": map[string]any{
|
"mcpServers": map[string]any{
|
||||||
"brain": map[string]any{
|
"brain": brainEntry(brainURL),
|
||||||
"url": brainURL + "/mcp",
|
"gitea": giteaEntry(),
|
||||||
"description": "Brain MCP — knowledge query, write, ingestion, session log",
|
|
||||||
},
|
|
||||||
"routing": map[string]any{
|
|
||||||
"url": "http://koala:30310/mcp",
|
|
||||||
"description": "Mode 2 routing pod — routes skill calls to LiteLLM/local",
|
|
||||||
"headers": map[string]any{
|
|
||||||
"X-Hyperguild-Mode": "client-local",
|
|
||||||
},
|
|
||||||
},
|
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -92,10 +101,8 @@ func modeSovereign(brainURL string) map[string]any {
|
|||||||
return map[string]any{
|
return map[string]any{
|
||||||
"_mode_note": "Sovereign mode primarily uses Crush + LiteLLM. This .mcp.json is provided as Claude Code fallback (e.g. emergency offline editing).",
|
"_mode_note": "Sovereign mode primarily uses Crush + LiteLLM. This .mcp.json is provided as Claude Code fallback (e.g. emergency offline editing).",
|
||||||
"mcpServers": map[string]any{
|
"mcpServers": map[string]any{
|
||||||
"brain": map[string]any{
|
"brain": brainEntry(brainURL),
|
||||||
"url": brainURL + "/mcp",
|
"gitea": giteaEntry(),
|
||||||
"description": "Brain MCP — knowledge query, write, ingestion, session log",
|
|
||||||
},
|
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+11
-17
@@ -35,11 +35,13 @@ func TestRunMode_Cloud_Default(t *testing.T) {
|
|||||||
servers, ok := got["mcpServers"].(map[string]any)
|
servers, ok := got["mcpServers"].(map[string]any)
|
||||||
require.True(t, ok, "mcpServers must be a JSON object")
|
require.True(t, ok, "mcpServers must be a JSON object")
|
||||||
assert.Contains(t, servers, "brain")
|
assert.Contains(t, servers, "brain")
|
||||||
|
assert.Contains(t, servers, "gitea")
|
||||||
assert.NotContains(t, servers, "routing")
|
assert.NotContains(t, servers, "routing")
|
||||||
assert.NotContains(t, got, "_mode_note")
|
assert.NotContains(t, got, "_mode_note")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunMode_ClientLocal_HasRoutingEntry(t *testing.T) {
|
func TestRunMode_ClientLocal_NoRouting_HasGitea(t *testing.T) {
|
||||||
|
// #75: the routing pod is iceboxed; the consolidated harness is brain + gitea.
|
||||||
dir := t.TempDir()
|
dir := t.TempDir()
|
||||||
outPath := filepath.Join(dir, ".mcp.json")
|
outPath := filepath.Join(dir, ".mcp.json")
|
||||||
t.Setenv("BRAIN_URL", "http://koala:30330")
|
t.Setenv("BRAIN_URL", "http://koala:30330")
|
||||||
@@ -51,17 +53,11 @@ func TestRunMode_ClientLocal_HasRoutingEntry(t *testing.T) {
|
|||||||
got := readJSON(t, outPath)
|
got := readJSON(t, outPath)
|
||||||
servers := got["mcpServers"].(map[string]any)
|
servers := got["mcpServers"].(map[string]any)
|
||||||
require.Contains(t, servers, "brain")
|
require.Contains(t, servers, "brain")
|
||||||
require.Contains(t, servers, "routing")
|
require.Contains(t, servers, "gitea")
|
||||||
|
assert.NotContains(t, servers, "routing", "routing pod iceboxed (#75)")
|
||||||
routing := servers["routing"].(map[string]any)
|
|
||||||
assert.NotContains(t, routing, "_routing_pending", "placeholder should be removed once Plan 6 ships")
|
|
||||||
|
|
||||||
headers, ok := routing["headers"].(map[string]any)
|
|
||||||
require.True(t, ok, "routing entry should have headers block")
|
|
||||||
assert.Equal(t, "client-local", headers["X-Hyperguild-Mode"])
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestModeClientLocalHasRoutingHeader(t *testing.T) {
|
func TestModeGiteaEntryPresentAndWellFormed(t *testing.T) {
|
||||||
tmp := t.TempDir() + "/mcp.json"
|
tmp := t.TempDir() + "/mcp.json"
|
||||||
out := &bytes.Buffer{}
|
out := &bytes.Buffer{}
|
||||||
stderr := &bytes.Buffer{}
|
stderr := &bytes.Buffer{}
|
||||||
@@ -73,13 +69,10 @@ func TestModeClientLocalHasRoutingHeader(t *testing.T) {
|
|||||||
require.NoError(t, json.Unmarshal(body, &doc))
|
require.NoError(t, json.Unmarshal(body, &doc))
|
||||||
|
|
||||||
servers := doc["mcpServers"].(map[string]any)
|
servers := doc["mcpServers"].(map[string]any)
|
||||||
routing := servers["routing"].(map[string]any)
|
require.NotContains(t, servers, "routing")
|
||||||
assert.Equal(t, "http://koala:30310/mcp", routing["url"])
|
gitea, ok := servers["gitea"].(map[string]any)
|
||||||
assert.NotContains(t, routing, "_routing_pending", "placeholder should be removed once Plan 6 ships")
|
require.True(t, ok, "gitea entry must be present (audit-trail invariant)")
|
||||||
|
assert.Equal(t, "https://git-mcp.d-ma.be/mcp", gitea["url"])
|
||||||
headers, ok := routing["headers"].(map[string]any)
|
|
||||||
require.True(t, ok, "routing entry should have headers block")
|
|
||||||
assert.Equal(t, "client-local", headers["X-Hyperguild-Mode"])
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunMode_Sovereign_HasModeNote(t *testing.T) {
|
func TestRunMode_Sovereign_HasModeNote(t *testing.T) {
|
||||||
@@ -94,6 +87,7 @@ func TestRunMode_Sovereign_HasModeNote(t *testing.T) {
|
|||||||
assert.Contains(t, got, "_mode_note")
|
assert.Contains(t, got, "_mode_note")
|
||||||
servers := got["mcpServers"].(map[string]any)
|
servers := got["mcpServers"].(map[string]any)
|
||||||
assert.Contains(t, servers, "brain")
|
assert.Contains(t, servers, "brain")
|
||||||
|
assert.Contains(t, servers, "gitea")
|
||||||
assert.NotContains(t, servers, "routing")
|
assert.NotContains(t, servers, "routing")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ import (
|
|||||||
"io"
|
"io"
|
||||||
"os"
|
"os"
|
||||||
|
|
||||||
"github.com/mathiasbq/supervisor/internal/tier"
|
"git.d-ma.be/mathias/hyperguild/internal/tier"
|
||||||
)
|
)
|
||||||
|
|
||||||
const defaultAnthropicProbe = "https://api.anthropic.com"
|
const defaultAnthropicProbe = "https://api.anthropic.com"
|
||||||
|
|||||||
@@ -1,170 +0,0 @@
|
|||||||
package main
|
|
||||||
|
|
||||||
// The internal/skills/{debug,retrospective,review,trainer} packages imported
|
|
||||||
// below are also imported by cmd/supervisor. Plan 7 (supervisor retirement)
|
|
||||||
// MUST NOT delete these four packages — the routing pod is their second
|
|
||||||
// consumer. Plan 7 deletes only internal/skills/{tdd,spec,tier} (the skills
|
|
||||||
// that don't route to local), the supervisor binary, and supervisor manifests.
|
|
||||||
// See docs/superpowers/specs/2026-05-04-mode-2-routing-pod-design.md (Constraints).
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"log/slog"
|
|
||||||
"net/http"
|
|
||||||
"os"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/mathiasbq/supervisor/internal/auth"
|
|
||||||
"github.com/mathiasbq/supervisor/internal/config"
|
|
||||||
iexec "github.com/mathiasbq/supervisor/internal/exec"
|
|
||||||
"github.com/mathiasbq/supervisor/internal/githubclient"
|
|
||||||
"github.com/mathiasbq/supervisor/internal/mcp"
|
|
||||||
"github.com/mathiasbq/supervisor/internal/mcpclient"
|
|
||||||
"github.com/mathiasbq/supervisor/internal/registry"
|
|
||||||
"github.com/mathiasbq/supervisor/internal/routing"
|
|
||||||
"github.com/mathiasbq/supervisor/internal/skills/debug"
|
|
||||||
"github.com/mathiasbq/supervisor/internal/skills/project"
|
|
||||||
"github.com/mathiasbq/supervisor/internal/skills/retrospective"
|
|
||||||
"github.com/mathiasbq/supervisor/internal/skills/review"
|
|
||||||
"github.com/mathiasbq/supervisor/internal/skills/trainer"
|
|
||||||
)
|
|
||||||
|
|
||||||
func main() {
|
|
||||||
logger := slog.New(slog.NewTextHandler(os.Stderr, nil))
|
|
||||||
slog.SetDefault(logger)
|
|
||||||
|
|
||||||
cfg, err := config.LoadRouting()
|
|
||||||
if err != nil {
|
|
||||||
logger.Error("config load failed", "err", err)
|
|
||||||
os.Exit(1)
|
|
||||||
}
|
|
||||||
|
|
||||||
configDir := envOr("SUPERVISOR_CONFIG_DIR", "/app/config/supervisor")
|
|
||||||
mustRead := func(path string) string {
|
|
||||||
b, err := os.ReadFile(configDir + "/" + path)
|
|
||||||
if err != nil {
|
|
||||||
logger.Error("read prompt failed", "path", path, "err", err)
|
|
||||||
os.Exit(1)
|
|
||||||
}
|
|
||||||
return string(b)
|
|
||||||
}
|
|
||||||
|
|
||||||
llm := iexec.NewLiteLLM(cfg.LiteLLMBaseURL, cfg.LiteLLMAPIKey, 0)
|
|
||||||
|
|
||||||
router := &routing.Router{
|
|
||||||
Fetcher: routing.NewFetcher(cfg.BrainURL, "7d", time.Duration(cfg.PassRateTTLSeconds)*time.Second),
|
|
||||||
Logger: routing.NewLogger(cfg.BrainURL, cfg.BrainMCPToken),
|
|
||||||
Policy: routing.Policy{Floor: cfg.RouteLocalFloor, Ceil: cfg.RouteLocalCeil},
|
|
||||||
FastModel: cfg.FastModel,
|
|
||||||
ThinkingModel: cfg.ThinkingModel,
|
|
||||||
Complete: llm.Complete,
|
|
||||||
}
|
|
||||||
|
|
||||||
// Skill packages call CompleteFunc(ctx, model, system, user) — no session_id
|
|
||||||
// or project_root in the signature. Rather than modifying every skill's API
|
|
||||||
// (and inflating Plan 6's blast radius), the routing pod logs every decision
|
|
||||||
// under a fixed session_id "_routing". Operators query
|
|
||||||
// `GET /pass-rate?skill=_routing&window=...` to inspect routing health.
|
|
||||||
const routingSessionID = "_routing"
|
|
||||||
wrap := func(skillName string) routing.CompleteFunc {
|
|
||||||
return func(ctx context.Context, _, system, user string) (string, int64, error) {
|
|
||||||
// The model param is ignored: the router picks the model based on policy.
|
|
||||||
return router.Run(ctx, routing.RunInput{
|
|
||||||
Skill: skillName,
|
|
||||||
System: system,
|
|
||||||
User: user,
|
|
||||||
SessionID: routingSessionID,
|
|
||||||
ProjectRoot: "",
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
reg := registry.New()
|
|
||||||
reg.Register(review.New(review.Config{
|
|
||||||
SkillPrompt: mustRead("review.md"),
|
|
||||||
DefaultModel: cfg.FastModel,
|
|
||||||
CompleteFunc: review.CompleteFunc(wrap("review")),
|
|
||||||
}))
|
|
||||||
reg.Register(debug.New(debug.Config{
|
|
||||||
SkillPrompt: mustRead("debug.md"),
|
|
||||||
DefaultModel: cfg.FastModel,
|
|
||||||
CompleteFunc: debug.CompleteFunc(wrap("debug")),
|
|
||||||
}))
|
|
||||||
reg.Register(retrospective.New(retrospective.Config{
|
|
||||||
SkillPrompt: mustRead("retrospective.md"),
|
|
||||||
DefaultModel: cfg.FastModel,
|
|
||||||
CompleteFunc: retrospective.CompleteFunc(wrap("retrospective")),
|
|
||||||
}))
|
|
||||||
reg.Register(trainer.New(trainer.Config{
|
|
||||||
ReaderPrompt: mustRead("trainer-reader.md"),
|
|
||||||
WriterPrompt: mustRead("trainer-writer.md"),
|
|
||||||
DefaultModel: cfg.FastModel,
|
|
||||||
CompleteFunc: trainer.CompleteFunc(wrap("trainer")),
|
|
||||||
}))
|
|
||||||
|
|
||||||
if cfg.GiteaMCPURL != "" {
|
|
||||||
mcpC, err := mcpclient.New(cfg.GiteaMCPURL, cfg.GiteaMCPToken)
|
|
||||||
if err != nil {
|
|
||||||
logger.Error("mcpclient init for project_create — GITEA_MCP_URL is set but GITEA_MCP_TOKEN is empty (check routing-secrets)", "err", err)
|
|
||||||
os.Exit(1)
|
|
||||||
}
|
|
||||||
var ghClient *githubclient.Client
|
|
||||||
if cfg.GitHubPAT != "" {
|
|
||||||
ghClient = githubclient.New(cfg.GitHubPAT)
|
|
||||||
}
|
|
||||||
reg.Register(project.New(project.Config{
|
|
||||||
Client: mcpC,
|
|
||||||
GitHub: ghClient,
|
|
||||||
GiteaOwner: cfg.GiteaOwner,
|
|
||||||
GitHubOwner: cfg.GitHubOwner,
|
|
||||||
GitHubPAT: cfg.GitHubPAT,
|
|
||||||
InfraRepo: cfg.InfraRepo,
|
|
||||||
}))
|
|
||||||
logger.Info("project_create registered", "gitea_mcp_url", cfg.GiteaMCPURL,
|
|
||||||
"gitea_owner", cfg.GiteaOwner, "github_owner", cfg.GitHubOwner,
|
|
||||||
"infra_repo", cfg.InfraRepo, "github_pat_set", cfg.GitHubPAT != "")
|
|
||||||
} else {
|
|
||||||
logger.Info("project_create skipped — GITEA_MCP_URL not set")
|
|
||||||
}
|
|
||||||
|
|
||||||
var validator *auth.Validator
|
|
||||||
if dexURL := os.Getenv("DEX_ISSUER_URL"); dexURL != "" {
|
|
||||||
audience := os.Getenv("MCP_AUDIENCE")
|
|
||||||
v, err := auth.NewValidator(dexURL, audience)
|
|
||||||
if err != nil {
|
|
||||||
logger.Error("build jwt validator", "err", err)
|
|
||||||
os.Exit(1)
|
|
||||||
}
|
|
||||||
validator = v
|
|
||||||
logger.Info("jwt auth enabled", "issuer", dexURL)
|
|
||||||
}
|
|
||||||
|
|
||||||
srv := mcp.NewServer(reg, cfg.MCPAuthToken, validator)
|
|
||||||
mux := http.NewServeMux()
|
|
||||||
mux.Handle("/mcp", srv)
|
|
||||||
mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) {
|
|
||||||
w.WriteHeader(http.StatusOK)
|
|
||||||
})
|
|
||||||
|
|
||||||
if dexURL := os.Getenv("DEX_ISSUER_URL"); dexURL != "" {
|
|
||||||
resourceURL := os.Getenv("MCP_RESOURCE_URL")
|
|
||||||
mux.HandleFunc("GET /.well-known/oauth-protected-resource",
|
|
||||||
auth.ProtectedResourceHandler(resourceURL, dexURL))
|
|
||||||
}
|
|
||||||
|
|
||||||
addr := ":" + cfg.Port
|
|
||||||
logger.Info("routing pod starting", "addr", addr,
|
|
||||||
"fast", cfg.FastModel, "thinking", cfg.ThinkingModel,
|
|
||||||
"floor", cfg.RouteLocalFloor, "ceil", cfg.RouteLocalCeil)
|
|
||||||
if err := http.ListenAndServe(addr, mux); err != nil { //nolint:gosec
|
|
||||||
logger.Error("server stopped", "err", err)
|
|
||||||
os.Exit(1)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func envOr(key, def string) string {
|
|
||||||
if v := os.Getenv(key); v != "" {
|
|
||||||
return v
|
|
||||||
}
|
|
||||||
return def
|
|
||||||
}
|
|
||||||
@@ -1,135 +0,0 @@
|
|||||||
package main_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"encoding/json"
|
|
||||||
"io"
|
|
||||||
"net"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"os"
|
|
||||||
"os/exec"
|
|
||||||
"strconv"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
)
|
|
||||||
|
|
||||||
// TestRoutingPodEndToEnd boots the binary against fake LiteLLM + brain servers,
|
|
||||||
// calls tools/list and one tools/call, and verifies the brain saw a session_log POST.
|
|
||||||
func TestRoutingPodEndToEnd(t *testing.T) {
|
|
||||||
if testing.Short() {
|
|
||||||
t.Skip("end-to-end binary boot")
|
|
||||||
}
|
|
||||||
|
|
||||||
var brainHits int
|
|
||||||
llm := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
|
||||||
_ = json.NewEncoder(w).Encode(map[string]any{
|
|
||||||
"choices": []map[string]any{{"message": map[string]any{"role": "assistant", "content": "stub"}}},
|
|
||||||
})
|
|
||||||
}))
|
|
||||||
defer llm.Close()
|
|
||||||
|
|
||||||
brain := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
||||||
switch r.URL.Path {
|
|
||||||
case "/pass-rate":
|
|
||||||
brainHits++
|
|
||||||
_ = json.NewEncoder(w).Encode(map[string]any{"pass_rate": 0.95})
|
|
||||||
case "/mcp":
|
|
||||||
brainHits++
|
|
||||||
_ = json.NewEncoder(w).Encode(map[string]any{"jsonrpc": "2.0", "id": 1, "result": map[string]any{}})
|
|
||||||
}
|
|
||||||
}))
|
|
||||||
defer brain.Close()
|
|
||||||
|
|
||||||
port := freePort(t)
|
|
||||||
addr := "127.0.0.1:" + port
|
|
||||||
baseURL := "http://" + addr
|
|
||||||
|
|
||||||
bin := buildRouting(t)
|
|
||||||
cmd := exec.Command(bin)
|
|
||||||
cmd.Env = []string{
|
|
||||||
"ROUTING_PORT=" + port,
|
|
||||||
"LITELLM_BASE_URL=" + llm.URL,
|
|
||||||
"LITELLM_API_KEY=stub",
|
|
||||||
"BRAIN_URL=" + brain.URL,
|
|
||||||
"SUPERVISOR_CONFIG_DIR=../../config/supervisor",
|
|
||||||
"PATH=" + os.Getenv("PATH"),
|
|
||||||
"HOME=" + os.Getenv("HOME"),
|
|
||||||
}
|
|
||||||
require.NoError(t, cmd.Start())
|
|
||||||
t.Cleanup(func() { _ = cmd.Process.Kill() })
|
|
||||||
|
|
||||||
require.NoError(t, waitForPort(t, addr, 30*time.Second))
|
|
||||||
|
|
||||||
resp := mcpCall(t, baseURL+"/mcp", `{"jsonrpc":"2.0","id":1,"method":"tools/list"}`)
|
|
||||||
assert.Contains(t, resp, `"review"`)
|
|
||||||
assert.Contains(t, resp, `"debug"`)
|
|
||||||
assert.Contains(t, resp, `"retrospective"`)
|
|
||||||
assert.Contains(t, resp, `"trainer"`)
|
|
||||||
|
|
||||||
resp = mcpCall(t, baseURL+"/mcp", `{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"review","arguments":{"project_root":"/tmp","files":["README.md"]}}}`)
|
|
||||||
_ = resp // shape varies by skill; we only need a 200
|
|
||||||
|
|
||||||
// Wait briefly for the async session_log to land.
|
|
||||||
deadline := time.Now().Add(2 * time.Second)
|
|
||||||
for time.Now().Before(deadline) && brainHits < 2 {
|
|
||||||
time.Sleep(50 * time.Millisecond)
|
|
||||||
}
|
|
||||||
assert.GreaterOrEqual(t, brainHits, 2, "expected at least one /pass-rate hit and one /mcp session_log hit")
|
|
||||||
}
|
|
||||||
|
|
||||||
func buildRouting(t *testing.T) string {
|
|
||||||
t.Helper()
|
|
||||||
bin := t.TempDir() + "/routing"
|
|
||||||
out, err := exec.Command("go", "build", "-o", bin, "github.com/mathiasbq/supervisor/cmd/routing").CombinedOutput()
|
|
||||||
require.NoError(t, err, "build failed: %s", out)
|
|
||||||
return bin
|
|
||||||
}
|
|
||||||
|
|
||||||
func waitForPort(_ *testing.T, addr string, dur time.Duration) error {
|
|
||||||
deadline := time.Now().Add(dur)
|
|
||||||
for time.Now().Before(deadline) {
|
|
||||||
c, err := http.Get("http://" + addr + "/healthz") //nolint:noctx
|
|
||||||
if err == nil {
|
|
||||||
_ = c.Body.Close()
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
conn, err := http.NewRequest(http.MethodPost, "http://"+addr+"/mcp", strings.NewReader(`{}`))
|
|
||||||
if err == nil {
|
|
||||||
r, err := http.DefaultClient.Do(conn)
|
|
||||||
if err == nil {
|
|
||||||
_ = r.Body.Close()
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
}
|
|
||||||
time.Sleep(50 * time.Millisecond)
|
|
||||||
}
|
|
||||||
return context.DeadlineExceeded
|
|
||||||
}
|
|
||||||
|
|
||||||
func mcpCall(t *testing.T, url, body string) string {
|
|
||||||
t.Helper()
|
|
||||||
r, err := http.Post(url, "application/json", strings.NewReader(body)) //nolint:noctx
|
|
||||||
require.NoError(t, err)
|
|
||||||
defer func() { _ = r.Body.Close() }()
|
|
||||||
raw, err := io.ReadAll(r.Body)
|
|
||||||
require.NoError(t, err)
|
|
||||||
return string(raw)
|
|
||||||
}
|
|
||||||
|
|
||||||
// freePort grabs an OS-assigned TCP port and releases it. There is a small
|
|
||||||
// race window before the subprocess re-binds it, but it is acceptable for
|
|
||||||
// test isolation against a hardcoded port colliding with another test or
|
|
||||||
// stray process.
|
|
||||||
func freePort(t *testing.T) string {
|
|
||||||
t.Helper()
|
|
||||||
l, err := net.Listen("tcp", "127.0.0.1:0")
|
|
||||||
require.NoError(t, err)
|
|
||||||
port := l.Addr().(*net.TCPAddr).Port
|
|
||||||
require.NoError(t, l.Close())
|
|
||||||
return strconv.Itoa(port)
|
|
||||||
}
|
|
||||||
@@ -55,26 +55,35 @@ In normal sessions, invoke the pod's tools instead of reviewing/debugging inline
|
|||||||
|
|
||||||
Each call logs an outcome to ingestion → `/pass-rate` ticks up.
|
Each call logs an outcome to ingestion → `/pass-rate` ticks up.
|
||||||
|
|
||||||
## Step 3 — the flywheel (cold-start behavior is the gate)
|
## Step 3 — how routing actually picks the model
|
||||||
|
|
||||||
The router's cold start **is** the data gate — same loop:
|
Per `internal/routing/policy.go`:
|
||||||
|
|
||||||
1. With pass-rate `nil` (cold), the policy routes to the **thinking / cloud**
|
1. pass-rate `nil` (cold) → **local** fast tier. The router defaults to local
|
||||||
model — it won't trust the local fast model without history.
|
from invocation #1, not to cloud — so the fast tier is exercised immediately.
|
||||||
2. As `review` / `debug` accumulate **passes**, pass-rate climbs past the floor
|
2. pass-rate `>= 0.90` (floor) → **local**; `< 0.70` (ceil) → **cloud/thinking**;
|
||||||
(`HYPERGUILD_ROUTE_LOCAL_FLOOR=0.90`) and the router starts sending those
|
in the `[0.70, 0.90)` band a request-hash bit samples 50/50.
|
||||||
skills to the **local fast tier**.
|
3. On a local execution error the router falls open to the thinking model for
|
||||||
3. So early calls "pay in" on cloud to build the record; then the local tier
|
that one call (logged `thinking_fallback`).
|
||||||
activates automatically. Exercising the traffic is what spins it up.
|
|
||||||
|
|
||||||
The fast tier is now **`koala/qwen36-35b-a3b`** (Qwen3.6-35B-A3B MTP), promoted
|
So you are not "paying in on cloud" — cold calls already run on the (validated)
|
||||||
2026-06-29 after clearing the tool-call gate (infra `c66a195`,
|
local fast tier **`koala/qwen36-35b-a3b`** (Qwen3.6-35B-A3B MTP, promoted
|
||||||
`HYPERGUILD_FAST_MODEL`).
|
2026-06-29, infra `c66a195`, `HYPERGUILD_FAST_MODEL`). Accumulating passes just
|
||||||
|
keeps it there once real pass-rate is computed.
|
||||||
|
|
||||||
|
> **Instrumentation note (#73, fixed 2026-06-30):** until v0.11.1 the pod logged
|
||||||
|
> successes as `"skip"` (not `"pass"`), under `skill:"_routing"`, via an
|
||||||
|
> unauthenticated POST that silently 401'd — so `/pass-rate` stayed at zero no
|
||||||
|
> matter how much you used it. That's fixed and verified (a real review call now
|
||||||
|
> moves `/pass-rate?skill=review` 0→1). If you see traffic not registering,
|
||||||
|
> re-check #73's three failure modes first.
|
||||||
|
|
||||||
## Target & verification
|
## Target & verification
|
||||||
|
|
||||||
- **50 logged invocations** across `review` + `debug` within the 14-day window
|
- **50 logged invocations** across `review` + `debug` within the 14-day window.
|
||||||
(opened 2026-06-26, **kill-date 2026-07-10**) → ~4–5 calls/day.
|
The clock restarts **2026-06-30** (the day instrumentation was verified working;
|
||||||
|
the original 2026-06-26→07-10 window measured broken plumbing) → **kill-date
|
||||||
|
2026-07-14**, ~4 calls/day (1 already logged from the #73 smoke test).
|
||||||
- Check progress anytime:
|
- Check progress anytime:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
@@ -1051,7 +1051,7 @@ git push
|
|||||||
```bash
|
```bash
|
||||||
# Poll until complete (replace RUN_ID with the new run):
|
# Poll until complete (replace RUN_ID with the new run):
|
||||||
curl -s "https://gitea.d-ma.be/api/v1/repos/mathias/hyperguild/actions/runs?limit=2" \
|
curl -s "https://gitea.d-ma.be/api/v1/repos/mathias/hyperguild/actions/runs?limit=2" \
|
||||||
-H "Authorization: token 736a8c36adcc6ecb41fff56e5ae4d0eb3105a670" \
|
-H "Authorization: token <REDACTED-dead-gitea-pat-infra#344>" \
|
||||||
| python3 -c "import sys,json; [print(r['id'],r['status'],r.get('conclusion','—')) for r in json.load(sys.stdin)['workflow_runs']]"
|
| python3 -c "import sys,json; [print(r['id'],r['status'],r.get('conclusion','—')) for r in json.load(sys.stdin)['workflow_runs']]"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
module github.com/mathiasbq/supervisor
|
module git.d-ma.be/mathias/hyperguild
|
||||||
|
|
||||||
go 1.26.1
|
go 1.26.1
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,53 @@
|
|||||||
|
// Command frontmatterlint reports markdown notes whose YAML frontmatter does
|
||||||
|
// not parse, and optionally repairs the ones that only need quoting.
|
||||||
|
//
|
||||||
|
// go run ./cmd/frontmatterlint ../../brain # report
|
||||||
|
// go run ./cmd/frontmatterlint -fix ../../brain # repair in place
|
||||||
|
//
|
||||||
|
// Exit codes: 0 clean (or everything repaired), 1 findings remain, 2 the scan
|
||||||
|
// itself failed.
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
|
||||||
|
"github.com/mathiasbq/hyperguild/ingestion/internal/frontmatter"
|
||||||
|
)
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
fix := flag.Bool("fix", false, "repair files that quoting can rescue, in place")
|
||||||
|
flag.Parse()
|
||||||
|
|
||||||
|
root := "."
|
||||||
|
if flag.NArg() > 0 {
|
||||||
|
root = flag.Arg(0)
|
||||||
|
}
|
||||||
|
|
||||||
|
findings, err := frontmatter.Scan(root, *fix)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(os.Stderr, "frontmatterlint:", err)
|
||||||
|
os.Exit(2)
|
||||||
|
}
|
||||||
|
|
||||||
|
var unresolved int
|
||||||
|
for _, f := range findings {
|
||||||
|
if f.Repaired {
|
||||||
|
fmt.Printf("repaired %s\n", f.Path)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
unresolved++
|
||||||
|
fmt.Printf("BROKEN %s: %v\n", f.Path, f.Err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if unresolved > 0 {
|
||||||
|
fmt.Fprintf(os.Stderr, "\n%d file(s) with unparseable frontmatter", unresolved)
|
||||||
|
if !*fix {
|
||||||
|
fmt.Fprint(os.Stderr, " — re-run with -fix")
|
||||||
|
}
|
||||||
|
fmt.Fprintln(os.Stderr)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
fmt.Printf("✓ frontmatter: %d file(s) repaired, none broken\n", len(findings))
|
||||||
|
}
|
||||||
+112
-33
@@ -32,14 +32,36 @@ import (
|
|||||||
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
|
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
|
||||||
"github.com/mathiasbq/hyperguild/ingestion/internal/reranker"
|
"github.com/mathiasbq/hyperguild/ingestion/internal/reranker"
|
||||||
"github.com/mathiasbq/hyperguild/ingestion/internal/search"
|
"github.com/mathiasbq/hyperguild/ingestion/internal/search"
|
||||||
|
"github.com/mathiasbq/hyperguild/ingestion/internal/startupretry"
|
||||||
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
|
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
|
||||||
"github.com/mathiasbq/hyperguild/ingestion/internal/watcher"
|
"github.com/mathiasbq/hyperguild/ingestion/internal/watcher"
|
||||||
|
"github.com/mathiasbq/hyperguild/ingestion/internal/webhook"
|
||||||
|
"k8s.io/client-go/kubernetes"
|
||||||
|
"k8s.io/client-go/rest"
|
||||||
|
"k8s.io/client-go/tools/clientcmd"
|
||||||
)
|
)
|
||||||
|
|
||||||
// claudeSink converts each claudewatcher.Batch into one wiki note under
|
// kubeClient builds an in-cluster Kubernetes client (falls back to
|
||||||
// brain/wiki/claude-sessions/facts/. v1 emits one note per session
|
// $KUBECONFIG for local dev/testing against a real cluster).
|
||||||
// keyed by host + session id; classifier-driven hall routing is a
|
func kubeClient() (kubernetes.Interface, error) {
|
||||||
// follow-up (hyperguild#27 v2).
|
if cfg, err := rest.InClusterConfig(); err == nil {
|
||||||
|
return kubernetes.NewForConfig(cfg)
|
||||||
|
}
|
||||||
|
rules := clientcmd.NewDefaultClientConfigLoadingRules()
|
||||||
|
cc := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(rules, &clientcmd.ConfigOverrides{})
|
||||||
|
cfg, err := cc.ClientConfig()
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("kube config (no in-cluster, no kubeconfig): %w", err)
|
||||||
|
}
|
||||||
|
return kubernetes.NewForConfig(cfg)
|
||||||
|
}
|
||||||
|
|
||||||
|
// claudeSink converts each claudewatcher.Batch into a raw session dump
|
||||||
|
// under brain/archive/claude-sessions/<host>/. Deliberately NOT a wiki
|
||||||
|
// note (api.WriteNote / brain/wiki/) — raw full transcripts out-ranked
|
||||||
|
// curated ai-sessions summaries in BM25 (177,818 vs 45,335 on the same
|
||||||
|
// query) and duplicated content already summarized elsewhere. Kept for
|
||||||
|
// deep lookups, never indexed. See ai-sessions#10.
|
||||||
type claudeSink struct {
|
type claudeSink struct {
|
||||||
brainDir string
|
brainDir string
|
||||||
logger *slog.Logger
|
logger *slog.Logger
|
||||||
@@ -67,16 +89,14 @@ func (s *claudeSink) Ingest(ctx context.Context, b claudewatcher.Batch) error {
|
|||||||
sb.WriteString("\n\n")
|
sb.WriteString("\n\n")
|
||||||
}
|
}
|
||||||
slug := "session-" + b.Host + "-" + b.SessionID
|
slug := "session-" + b.Host + "-" + b.SessionID
|
||||||
if _, err := api.WriteNote(s.brainDir, api.WriteNoteOptions{
|
dest := filepath.Join(s.brainDir, "archive", "claude-sessions", b.Host, slug+".md")
|
||||||
Filename: slug,
|
if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
|
||||||
Wing: "claude-sessions",
|
return fmt.Errorf("create claude-sessions archive dir: %w", err)
|
||||||
Hall: "facts",
|
|
||||||
Type: "source",
|
|
||||||
Domain: b.ProjectID,
|
|
||||||
Content: sb.String(),
|
|
||||||
}); err != nil {
|
|
||||||
return fmt.Errorf("write claude session note: %w", err)
|
|
||||||
}
|
}
|
||||||
|
if err := os.WriteFile(dest, []byte(sb.String()), 0o644); err != nil {
|
||||||
|
return fmt.Errorf("write claude session archive: %w", err)
|
||||||
|
}
|
||||||
|
s.logger.Debug("claude session archived (non-indexed)", "path", dest)
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -124,6 +144,16 @@ func envInt(key string, fallback int) int {
|
|||||||
return fallback
|
return fallback
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// buildNotifier returns the ntfy notifier, or a nil interface when
|
||||||
|
// BRAIN_NTFY_URL is unset. A nil *interface* (not a typed-nil) is required so
|
||||||
|
// DegradingSink/Reconcile and the governance observer skip it cleanly.
|
||||||
|
func buildNotifier() audit.Notifier {
|
||||||
|
if n := audit.NewNtfyNotifier(os.Getenv("BRAIN_NTFY_URL"), os.Getenv("BRAIN_NTFY_TOKEN")); n != nil {
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
// buildAuditSink selects the capture audit sink. When BRAIN_LOKI_URL is
|
// buildAuditSink selects the capture audit sink. When BRAIN_LOKI_URL is
|
||||||
// set it builds the classification-aware DegradingSink (loki central +
|
// set it builds the classification-aware DegradingSink (loki central +
|
||||||
// durable file buffer + optional ntfy) and starts the reconcile loop;
|
// durable file buffer + optional ntfy) and starts the reconcile loop;
|
||||||
@@ -141,12 +171,7 @@ func buildAuditSink(ctx context.Context, brainDir string, logger *slog.Logger) c
|
|||||||
logger.Error("capture audit buffer init", "err", err)
|
logger.Error("capture audit buffer init", "err", err)
|
||||||
os.Exit(1)
|
os.Exit(1)
|
||||||
}
|
}
|
||||||
// Keep notifier as a nil interface (not a typed-nil) when unconfigured
|
notifier := buildNotifier()
|
||||||
// so DegradingSink/Reconcile skip it cleanly.
|
|
||||||
var notifier audit.Notifier
|
|
||||||
if n := audit.NewNtfyNotifier(os.Getenv("BRAIN_NTFY_URL"), os.Getenv("BRAIN_NTFY_TOKEN")); n != nil {
|
|
||||||
notifier = n
|
|
||||||
}
|
|
||||||
reconcileInterval := time.Duration(envInt("BRAIN_AUDIT_RECONCILE_INTERVAL", 60)) * time.Second
|
reconcileInterval := time.Duration(envInt("BRAIN_AUDIT_RECONCILE_INTERVAL", 60)) * time.Second
|
||||||
audit.StartReconcile(ctx, central, buffer, notifier, reconcileInterval)
|
audit.StartReconcile(ctx, central, buffer, notifier, reconcileInterval)
|
||||||
logger.Info("capture audit: loki+buffer sink", "loki", lokiURL, "reconcile_s", int(reconcileInterval.Seconds()))
|
logger.Info("capture audit: loki+buffer sink", "loki", lokiURL, "reconcile_s", int(reconcileInterval.Seconds()))
|
||||||
@@ -211,6 +236,11 @@ func main() {
|
|||||||
fallbackModel := envOr("BRAIN_LLM_FALLBACK_MODEL", "gemma4:31b")
|
fallbackModel := envOr("BRAIN_LLM_FALLBACK_MODEL", "gemma4:31b")
|
||||||
router.Fallback = llm.New(fallbackURL, "", fallbackModel, timeout)
|
router.Fallback = llm.New(fallbackURL, "", fallbackModel, timeout)
|
||||||
}
|
}
|
||||||
|
if tertiaryURL := os.Getenv("BRAIN_LLM_TERTIARY_URL"); tertiaryURL != "" {
|
||||||
|
tertiaryModel := envOr("BRAIN_LLM_TERTIARY_MODEL", "gemini-2.5-flash")
|
||||||
|
tertiaryKey := os.Getenv("GEMINI_API_KEY")
|
||||||
|
router.Tertiary = llm.New(tertiaryURL, tertiaryKey, tertiaryModel, timeout)
|
||||||
|
}
|
||||||
answerComplete = router.Complete
|
answerComplete = router.Complete
|
||||||
logger.Info("brain answer LLM configured", "primary", primaryURL, "model", primaryModel)
|
logger.Info("brain answer LLM configured", "primary", primaryURL, "model", primaryModel)
|
||||||
}
|
}
|
||||||
@@ -295,6 +325,36 @@ func main() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
|
|
||||||
|
// Built before the watcher, the graph backfill and session ingestion, and
|
||||||
|
// retried rather than fatal (#83). Two separate reasons:
|
||||||
|
//
|
||||||
|
// - On a cold cluster boot this pod can beat dex to serving and gets a
|
||||||
|
// 502 from ingress. Nothing is misconfigured; it simply started first.
|
||||||
|
// Exiting there meant CrashLoopBackOff redoing the backfill on every
|
||||||
|
// attempt, with exponential backoff stretching the brain outage.
|
||||||
|
// - Constructing it first means a genuine config error (wrong issuer,
|
||||||
|
// bad audience) fails in seconds instead of after all that work.
|
||||||
|
//
|
||||||
|
// The budget stays bounded: retrying forever would turn a misconfiguration
|
||||||
|
// into a process that looks alive and never serves.
|
||||||
|
jwtValidator, err := startupretry.Do(ctx,
|
||||||
|
startupretry.Policy{Attempts: 8, Base: time.Second, Max: 30 * time.Second},
|
||||||
|
func(ctx context.Context) (*chassisauth.JWTValidator, error) {
|
||||||
|
v, err := chassisauth.NewJWTValidator(ctx, os.Getenv("DEX_ISSUER_URL"), os.Getenv("MCP_AUDIENCE"))
|
||||||
|
if err != nil {
|
||||||
|
logger.Warn("oidc discovery not ready, retrying", "err", err)
|
||||||
|
}
|
||||||
|
return v, err
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
logger.Error("build jwt validator", "err", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
if jwtValidator != nil {
|
||||||
|
logger.Info("jwt auth enabled", "issuer", os.Getenv("DEX_ISSUER_URL"))
|
||||||
|
}
|
||||||
|
|
||||||
if watchInterval > 0 {
|
if watchInterval > 0 {
|
||||||
watcher.Start(ctx, watcher.Config{
|
watcher.Start(ctx, watcher.Config{
|
||||||
BrainDir: brainDir,
|
BrainDir: brainDir,
|
||||||
@@ -352,6 +412,29 @@ func main() {
|
|||||||
logger.Info("claudewatcher started",
|
logger.Info("claudewatcher started",
|
||||||
"sessions_dir", claudeDir, "host", host, "interval", interval)
|
"sessions_dir", claudeDir, "host", host, "interval", interval)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Gitea push webhook -> on-demand brain-sync Job, instead of waiting up
|
||||||
|
// to 15 minutes for the next CronJob poll. Off by default (opt in via
|
||||||
|
// GITEA_WEBHOOK_SECRET) since it needs Job-create RBAC in the "brain"
|
||||||
|
// namespace that a fresh deploy won't have granted yet.
|
||||||
|
var webhookHandler *webhook.Handler
|
||||||
|
if webhookSecret := os.Getenv("GITEA_WEBHOOK_SECRET"); webhookSecret != "" {
|
||||||
|
kc, err := kubeClient()
|
||||||
|
if err != nil {
|
||||||
|
logger.Error("brain-sync webhook: kube client", "err", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
webhookHandler = &webhook.Handler{
|
||||||
|
Secret: webhookSecret,
|
||||||
|
Clientset: kc,
|
||||||
|
Namespace: envOr("BRAIN_SYNC_NAMESPACE", "brain"),
|
||||||
|
CronJobName: envOr("BRAIN_SYNC_CRONJOB", "brain-sync"),
|
||||||
|
WatchRepo: envOr("BRAIN_SYNC_WATCH_REPO", "mathias/brain"),
|
||||||
|
Logger: logger,
|
||||||
|
}
|
||||||
|
logger.Info("brain-sync webhook enabled", "namespace", webhookHandler.Namespace, "cronjob", webhookHandler.CronJobName)
|
||||||
|
}
|
||||||
|
|
||||||
if vectorStore != nil {
|
if vectorStore != nil {
|
||||||
embedSyncInterval := envInt("BRAIN_EMBED_SYNC_INTERVAL", 300)
|
embedSyncInterval := envInt("BRAIN_EMBED_SYNC_INTERVAL", 300)
|
||||||
vectorstore.StartSync(ctx, brainDir, vectorStore,
|
vectorstore.StartSync(ctx, brainDir, vectorStore,
|
||||||
@@ -373,15 +456,9 @@ func main() {
|
|||||||
mux.HandleFunc("POST /promote", h.Promote)
|
mux.HandleFunc("POST /promote", h.Promote)
|
||||||
mux.HandleFunc("POST /backfill-embeddings", h.BackfillEmbeddings)
|
mux.HandleFunc("POST /backfill-embeddings", h.BackfillEmbeddings)
|
||||||
mux.HandleFunc("GET /pass-rate", h.PassRate)
|
mux.HandleFunc("GET /pass-rate", h.PassRate)
|
||||||
jwtValidator, err := chassisauth.NewJWTValidator(ctx, os.Getenv("DEX_ISSUER_URL"), os.Getenv("MCP_AUDIENCE"))
|
if webhookHandler != nil {
|
||||||
if err != nil {
|
mux.Handle("POST /webhooks/brain-sync", webhookHandler)
|
||||||
logger.Error("build jwt validator", "err", err)
|
|
||||||
os.Exit(1)
|
|
||||||
}
|
}
|
||||||
if jwtValidator != nil {
|
|
||||||
logger.Info("jwt auth enabled", "issuer", os.Getenv("DEX_ISSUER_URL"))
|
|
||||||
}
|
|
||||||
|
|
||||||
// Resource-metadata URL is only emitted on 401 when Dex OAuth is
|
// Resource-metadata URL is only emitted on 401 when Dex OAuth is
|
||||||
// configured. Static-Bearer-only deployments leave this empty so
|
// configured. Static-Bearer-only deployments leave this empty so
|
||||||
// clients never see an OAuth challenge.
|
// clients never see an OAuth challenge.
|
||||||
@@ -412,12 +489,8 @@ func main() {
|
|||||||
os.Exit(1)
|
os.Exit(1)
|
||||||
}
|
}
|
||||||
auditSink := buildAuditSink(ctx, brainDir, logger)
|
auditSink := buildAuditSink(ctx, brainDir, logger)
|
||||||
// The Gitea client also satisfies SummaryWriter (#66): session
|
|
||||||
// summaries are written to mathias/ai-sessions over the same API
|
|
||||||
// token. nil only if a future tracker impl lacks file writes.
|
|
||||||
summaryWriter, _ := tracker.(capture.SummaryWriter)
|
|
||||||
captureSvc := capture.NewService(
|
captureSvc := capture.NewService(
|
||||||
mcpSrv.BrainStore(), tracker, summaryWriter, classCfg, auditSink)
|
mcpSrv.BrainStore(), tracker, classCfg, auditSink)
|
||||||
sovereign := splitList(os.Getenv("BRAIN_CAPTURE_SOVEREIGN_PRINCIPALS"))
|
sovereign := splitList(os.Getenv("BRAIN_CAPTURE_SOVEREIGN_PRINCIPALS"))
|
||||||
resolver := capturehttp.NewOriginResolver(sovereign)
|
resolver := capturehttp.NewOriginResolver(sovereign)
|
||||||
captureH := capturehttp.New(captureSvc, jwtValidator, mcpToken, "local-cli", resolver)
|
captureH := capturehttp.New(captureSvc, jwtValidator, mcpToken, "local-cli", resolver)
|
||||||
@@ -427,7 +500,13 @@ func main() {
|
|||||||
// the existing /mcp OAuth connector. mcpSrv is already wrapped above;
|
// the existing /mcp OAuth connector. mcpSrv is already wrapped above;
|
||||||
// WithCapture mutates the same instance, so the tool appears live.
|
// WithCapture mutates the same instance, so the tool appears live.
|
||||||
mcpSrv.WithCapture(captureSvc, jwtValidator, mcpToken, "local-cli", resolver)
|
mcpSrv.WithCapture(captureSvc, jwtValidator, mcpToken, "local-cli", resolver)
|
||||||
logger.Info("capture enabled (REST + MCP tool)", "sovereign_principals", len(sovereign))
|
// hyperguild#97: a governance-file rewrite gets the same trail a
|
||||||
|
// capture already gets, plus an out-of-band alert. Wired here because
|
||||||
|
// brain_edit_root needs the capture plumbing anyway — that is what
|
||||||
|
// re-derives the static principal its gate checks.
|
||||||
|
mcpSrv.WithGovernanceObserver(auditSink, buildNotifier())
|
||||||
|
logger.Info("capture enabled (REST + MCP tool)", "sovereign_principals", len(sovereign),
|
||||||
|
"governance_observer", true)
|
||||||
} else {
|
} else {
|
||||||
logger.Info("capture endpoint disabled (BRAIN_GITEA_TOKEN unset)")
|
logger.Info("capture endpoint disabled (BRAIN_GITEA_TOKEN unset)")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,38 @@
|
|||||||
|
// ingestion/cmd/server/main_test.go
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"log/slog"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"github.com/mathiasbq/hyperguild/ingestion/internal/claudewatcher"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestClaudeSink_IngestWritesToNonIndexedArchiveNotWiki(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
sink := &claudeSink{brainDir: dir, logger: slog.New(slog.NewTextHandler(os.Stderr, nil))}
|
||||||
|
|
||||||
|
err := sink.Ingest(context.Background(), claudewatcher.Batch{
|
||||||
|
Host: "koala",
|
||||||
|
FilePath: "/host-home-claude/projects/-home-mathias-dev/abc.jsonl",
|
||||||
|
SessionID: "abc",
|
||||||
|
ProjectID: "-home-mathias-dev",
|
||||||
|
Turns: []claudewatcher.Turn{
|
||||||
|
{Type: "assistant", Content: "did a thing"},
|
||||||
|
},
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
got, err := os.ReadFile(filepath.Join(dir, "archive", "claude-sessions", "koala", "session-koala-abc.md"))
|
||||||
|
require.NoError(t, err, "raw session dump must land in the non-indexed archive")
|
||||||
|
assert.Contains(t, string(got), "did a thing")
|
||||||
|
|
||||||
|
_, err = os.Stat(filepath.Join(dir, "wiki", "claude-sessions"))
|
||||||
|
assert.True(t, os.IsNotExist(err), "raw transcripts must never land under wiki/ (ai-sessions#10 — BM25 pollution)")
|
||||||
|
}
|
||||||
+44
-4
@@ -2,17 +2,57 @@ module github.com/mathiasbq/hyperguild/ingestion
|
|||||||
|
|
||||||
go 1.26.1
|
go 1.26.1
|
||||||
|
|
||||||
require github.com/stretchr/testify v1.11.1
|
require (
|
||||||
|
github.com/stretchr/testify v1.11.1
|
||||||
|
k8s.io/api v0.31.3
|
||||||
|
k8s.io/apimachinery v0.31.3
|
||||||
|
k8s.io/client-go v0.31.3
|
||||||
|
)
|
||||||
|
|
||||||
require (
|
require (
|
||||||
github.com/kr/text v0.2.0 // indirect
|
github.com/emicklei/go-restful/v3 v3.11.0 // indirect
|
||||||
|
github.com/fxamacker/cbor/v2 v2.7.0 // indirect
|
||||||
|
github.com/go-logr/logr v1.4.2 // indirect
|
||||||
|
github.com/go-openapi/jsonpointer v0.19.6 // indirect
|
||||||
|
github.com/go-openapi/jsonreference v0.20.2 // indirect
|
||||||
|
github.com/go-openapi/swag v0.22.4 // indirect
|
||||||
|
github.com/gogo/protobuf v1.3.2 // indirect
|
||||||
|
github.com/golang/protobuf v1.5.4 // indirect
|
||||||
|
github.com/google/gnostic-models v0.6.8 // indirect
|
||||||
|
github.com/google/go-cmp v0.6.0 // indirect
|
||||||
|
github.com/google/gofuzz v1.2.0 // indirect
|
||||||
|
github.com/google/uuid v1.6.0 // indirect
|
||||||
|
github.com/imdario/mergo v0.3.6 // indirect
|
||||||
|
github.com/josharian/intern v1.0.0 // indirect
|
||||||
|
github.com/json-iterator/go v1.1.12 // indirect
|
||||||
github.com/lestrrat-go/jwx/v2 v2.1.6 // indirect
|
github.com/lestrrat-go/jwx/v2 v2.1.6 // indirect
|
||||||
|
github.com/mailru/easyjson v0.7.7 // indirect
|
||||||
|
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
|
||||||
|
github.com/modern-go/reflect2 v1.0.2 // indirect
|
||||||
|
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
|
||||||
|
github.com/pkg/errors v0.9.1 // indirect
|
||||||
github.com/rogpeppe/go-internal v1.15.0 // indirect
|
github.com/rogpeppe/go-internal v1.15.0 // indirect
|
||||||
|
github.com/spf13/pflag v1.0.5 // indirect
|
||||||
|
github.com/x448/float16 v0.8.4 // indirect
|
||||||
|
golang.org/x/net v0.26.0 // indirect
|
||||||
|
golang.org/x/oauth2 v0.21.0 // indirect
|
||||||
|
golang.org/x/term v0.28.0 // indirect
|
||||||
|
golang.org/x/time v0.3.0 // indirect
|
||||||
|
google.golang.org/protobuf v1.34.2 // indirect
|
||||||
|
gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
|
||||||
|
gopkg.in/inf.v0 v0.9.1 // indirect
|
||||||
|
gopkg.in/yaml.v2 v2.4.0 // indirect
|
||||||
|
k8s.io/klog/v2 v2.130.1 // indirect
|
||||||
|
k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 // indirect
|
||||||
|
k8s.io/utils v0.0.0-20240711033017-18e509b52bc8 // indirect
|
||||||
|
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect
|
||||||
|
sigs.k8s.io/structured-merge-diff/v4 v4.4.1 // indirect
|
||||||
|
sigs.k8s.io/yaml v1.4.0 // indirect
|
||||||
)
|
)
|
||||||
|
|
||||||
require (
|
require (
|
||||||
git.d-ma.be/mathias/mcp-chassis v0.2.0
|
git.d-ma.be/mathias/mcp-chassis v0.2.0
|
||||||
github.com/davecgh/go-spew v1.1.1 // indirect
|
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
|
||||||
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0 // indirect
|
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0 // indirect
|
||||||
github.com/goccy/go-json v0.10.3 // indirect
|
github.com/goccy/go-json v0.10.3 // indirect
|
||||||
github.com/jackc/pgpassfile v1.0.0 // indirect
|
github.com/jackc/pgpassfile v1.0.0 // indirect
|
||||||
@@ -24,7 +64,7 @@ require (
|
|||||||
github.com/lestrrat-go/httprc v1.0.6 // indirect
|
github.com/lestrrat-go/httprc v1.0.6 // indirect
|
||||||
github.com/lestrrat-go/iter v1.0.2 // indirect
|
github.com/lestrrat-go/iter v1.0.2 // indirect
|
||||||
github.com/lestrrat-go/option v1.0.1 // indirect
|
github.com/lestrrat-go/option v1.0.1 // indirect
|
||||||
github.com/pmezard/go-difflib v1.0.0 // indirect
|
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
|
||||||
github.com/segmentio/asm v1.2.0 // indirect
|
github.com/segmentio/asm v1.2.0 // indirect
|
||||||
golang.org/x/crypto v0.32.0 // indirect
|
golang.org/x/crypto v0.32.0 // indirect
|
||||||
golang.org/x/sync v0.17.0 // indirect
|
golang.org/x/sync v0.17.0 // indirect
|
||||||
|
|||||||
+135
-4
@@ -2,12 +2,46 @@ git.d-ma.be/mathias/mcp-chassis v0.2.0 h1:6fLmb7xqRa2nNVWsHaUbbfbArgDXJw/gDhb09c
|
|||||||
git.d-ma.be/mathias/mcp-chassis v0.2.0/go.mod h1:Ks7EK2UnGAN0H3rJjKUxUagX8/ZBdtLrOlcUbv0RwH8=
|
git.d-ma.be/mathias/mcp-chassis v0.2.0/go.mod h1:Ks7EK2UnGAN0H3rJjKUxUagX8/ZBdtLrOlcUbv0RwH8=
|
||||||
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
|
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
|
||||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
|
||||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
|
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
|
||||||
|
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0 h1:NMZiJj8QnKe1LgsbDayM4UoHwbvwDRwnI3hwNaAHRnc=
|
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0 h1:NMZiJj8QnKe1LgsbDayM4UoHwbvwDRwnI3hwNaAHRnc=
|
||||||
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0/go.mod h1:ZXNYxsqcloTdSy/rNShjYzMhyjf0LaoftYK0p+A3h40=
|
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0/go.mod h1:ZXNYxsqcloTdSy/rNShjYzMhyjf0LaoftYK0p+A3h40=
|
||||||
|
github.com/emicklei/go-restful/v3 v3.11.0 h1:rAQeMHw1c7zTmncogyy8VvRZwtkmkZ4FxERmMY4rD+g=
|
||||||
|
github.com/emicklei/go-restful/v3 v3.11.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc=
|
||||||
|
github.com/fxamacker/cbor/v2 v2.7.0 h1:iM5WgngdRBanHcxugY4JySA0nk1wZorNOpTgCMedv5E=
|
||||||
|
github.com/fxamacker/cbor/v2 v2.7.0/go.mod h1:pxXPTn3joSm21Gbwsv0w9OSA2y1HFR9qXEeXQVeNoDQ=
|
||||||
|
github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
|
||||||
|
github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
|
||||||
|
github.com/go-openapi/jsonpointer v0.19.6 h1:eCs3fxoIi3Wh6vtgmLTOjdhSpiqphQ+DaPn38N2ZdrE=
|
||||||
|
github.com/go-openapi/jsonpointer v0.19.6/go.mod h1:osyAmYz/mB/C3I+WsTTSgw1ONzaLJoLCyoi6/zppojs=
|
||||||
|
github.com/go-openapi/jsonreference v0.20.2 h1:3sVjiK66+uXK/6oQ8xgcRKcFgQ5KXa2KvnJRumpMGbE=
|
||||||
|
github.com/go-openapi/jsonreference v0.20.2/go.mod h1:Bl1zwGIM8/wsvqjsOQLJ/SH+En5Ap4rVB5KVcIDZG2k=
|
||||||
|
github.com/go-openapi/swag v0.22.3/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+VcZ0yl14=
|
||||||
|
github.com/go-openapi/swag v0.22.4 h1:QLMzNJnMGPRNDCbySlcj1x01tzU8/9LTTL9hZZZogBU=
|
||||||
|
github.com/go-openapi/swag v0.22.4/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+VcZ0yl14=
|
||||||
|
github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI=
|
||||||
|
github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8=
|
||||||
github.com/goccy/go-json v0.10.3 h1:KZ5WoDbxAIgm2HNbYckL0se1fHD6rz5j4ywS6ebzDqA=
|
github.com/goccy/go-json v0.10.3 h1:KZ5WoDbxAIgm2HNbYckL0se1fHD6rz5j4ywS6ebzDqA=
|
||||||
github.com/goccy/go-json v0.10.3/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
|
github.com/goccy/go-json v0.10.3/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
|
||||||
|
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
|
||||||
|
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
|
||||||
|
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
|
||||||
|
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
|
||||||
|
github.com/google/gnostic-models v0.6.8 h1:yo/ABAfM5IMRsS1VnXjTBvUb61tFIHozhlYvRgGre9I=
|
||||||
|
github.com/google/gnostic-models v0.6.8/go.mod h1:5n7qKqH0f5wFt+aWF8CW6pZLLNOfYuF5OpfBSENuI8U=
|
||||||
|
github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
|
||||||
|
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
|
||||||
|
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
|
||||||
|
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
|
||||||
|
github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0=
|
||||||
|
github.com/google/gofuzz v1.2.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
|
||||||
|
github.com/google/pprof v0.0.0-20240525223248-4bfdf5a9a2af h1:kmjWCqn2qkEml422C2Rrd27c3VGxi6a/6HNq8QmHRKM=
|
||||||
|
github.com/google/pprof v0.0.0-20240525223248-4bfdf5a9a2af/go.mod h1:K1liHPHnj73Fdn/EKuT8nrFqBihUSKXoLYU0BuatOYo=
|
||||||
|
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||||
|
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||||
|
github.com/imdario/mergo v0.3.6 h1:xTNEAn+kxVO7dTZGu0CegyqKZmoWFI0rF8UxjlB2d28=
|
||||||
|
github.com/imdario/mergo v0.3.6/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
|
||||||
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
|
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
|
||||||
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
|
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
|
||||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
|
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
|
||||||
@@ -16,8 +50,17 @@ github.com/jackc/pgx/v5 v5.9.2 h1:3ZhOzMWnR4yJ+RW1XImIPsD1aNSz4T4fyP7zlQb56hw=
|
|||||||
github.com/jackc/pgx/v5 v5.9.2/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
|
github.com/jackc/pgx/v5 v5.9.2/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
|
||||||
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
||||||
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||||
github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0=
|
github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
|
||||||
github.com/kr/pretty v0.3.0/go.mod h1:640gp4NfQd8pI5XOwp5fnNeVWj67G7CFk/SaSQn7NBk=
|
github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
|
||||||
|
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
|
||||||
|
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
|
||||||
|
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
|
||||||
|
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
|
||||||
|
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
|
||||||
|
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
|
||||||
|
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
|
||||||
|
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
|
||||||
|
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
|
||||||
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
|
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
|
||||||
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
|
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
|
||||||
github.com/lestrrat-go/blackmagic v1.0.3 h1:94HXkVLxkZO9vJI/w2u1T0DAoprShFd13xtnSINtDWs=
|
github.com/lestrrat-go/blackmagic v1.0.3 h1:94HXkVLxkZO9vJI/w2u1T0DAoprShFd13xtnSINtDWs=
|
||||||
@@ -32,30 +75,118 @@ github.com/lestrrat-go/jwx/v2 v2.1.6 h1:hxM1gfDILk/l5ylers6BX/Eq1m/pnxe9NBwW6lVf
|
|||||||
github.com/lestrrat-go/jwx/v2 v2.1.6/go.mod h1:Y722kU5r/8mV7fYDifjug0r8FK8mZdw0K0GpJw/l8pU=
|
github.com/lestrrat-go/jwx/v2 v2.1.6/go.mod h1:Y722kU5r/8mV7fYDifjug0r8FK8mZdw0K0GpJw/l8pU=
|
||||||
github.com/lestrrat-go/option v1.0.1 h1:oAzP2fvZGQKWkvHa1/SAcFolBEca1oN+mQ7eooNBEYU=
|
github.com/lestrrat-go/option v1.0.1 h1:oAzP2fvZGQKWkvHa1/SAcFolBEca1oN+mQ7eooNBEYU=
|
||||||
github.com/lestrrat-go/option v1.0.1/go.mod h1:5ZHFbivi4xwXxhxY9XHDe2FHo6/Z7WWmtT7T5nBBp3I=
|
github.com/lestrrat-go/option v1.0.1/go.mod h1:5ZHFbivi4xwXxhxY9XHDe2FHo6/Z7WWmtT7T5nBBp3I=
|
||||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0=
|
||||||
|
github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
|
||||||
|
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
|
||||||
|
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
|
||||||
|
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
|
||||||
|
github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
|
||||||
|
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
|
||||||
|
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
|
||||||
|
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
|
||||||
|
github.com/onsi/ginkgo/v2 v2.19.0 h1:9Cnnf7UHo57Hy3k6/m5k3dRfGTMXGvxhHFvkDTCTpvA=
|
||||||
|
github.com/onsi/ginkgo/v2 v2.19.0/go.mod h1:rlwLi9PilAFJ8jCg9UE1QP6VBpd6/xj3SRC0d6TU0To=
|
||||||
|
github.com/onsi/gomega v1.19.0 h1:4ieX6qQjPP/BfC3mpsAtIGGlxTWPeA3Inl/7DtXw1tw=
|
||||||
|
github.com/onsi/gomega v1.19.0/go.mod h1:LY+I3pBVzYsTBU1AnDwOSxaYi9WoWiqgwooUqq9yPro=
|
||||||
|
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
|
||||||
|
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
|
||||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||||
|
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
|
||||||
|
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||||
github.com/rogpeppe/go-internal v1.15.0 h1:D0RCU5rMAp+SpgkiNdrjfJ+LX4J1M32V2NeCY7EJ6hc=
|
github.com/rogpeppe/go-internal v1.15.0 h1:D0RCU5rMAp+SpgkiNdrjfJ+LX4J1M32V2NeCY7EJ6hc=
|
||||||
github.com/rogpeppe/go-internal v1.15.0/go.mod h1:DrUVZyrJU+txYW5/1kwtXQSMFio52ZOxX7yM1VHvnxs=
|
github.com/rogpeppe/go-internal v1.15.0/go.mod h1:DrUVZyrJU+txYW5/1kwtXQSMFio52ZOxX7yM1VHvnxs=
|
||||||
github.com/segmentio/asm v1.2.0 h1:9BQrFxC+YOHJlTlHGkTrFWf59nbL3XnCoFLTwDCI7ys=
|
github.com/segmentio/asm v1.2.0 h1:9BQrFxC+YOHJlTlHGkTrFWf59nbL3XnCoFLTwDCI7ys=
|
||||||
github.com/segmentio/asm v1.2.0/go.mod h1:BqMnlJP91P8d+4ibuonYZw9mfnzI9HfxselHZr5aAcs=
|
github.com/segmentio/asm v1.2.0/go.mod h1:BqMnlJP91P8d+4ibuonYZw9mfnzI9HfxselHZr5aAcs=
|
||||||
|
github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
|
||||||
|
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
|
||||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||||
|
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
|
||||||
|
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
|
||||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||||
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||||
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||||
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||||
|
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
|
||||||
|
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
|
||||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||||
|
github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM=
|
||||||
|
github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg=
|
||||||
|
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
|
||||||
|
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
|
||||||
|
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
|
||||||
|
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
|
||||||
|
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
|
||||||
golang.org/x/crypto v0.32.0 h1:euUpcYgM8WcP71gNpTqQCn6rC2t6ULUPiOzfWaXVVfc=
|
golang.org/x/crypto v0.32.0 h1:euUpcYgM8WcP71gNpTqQCn6rC2t6ULUPiOzfWaXVVfc=
|
||||||
golang.org/x/crypto v0.32.0/go.mod h1:ZnnJkOaASj8g0AjIduWNlq2NRxL0PlBrbKVyZ6V/Ugc=
|
golang.org/x/crypto v0.32.0/go.mod h1:ZnnJkOaASj8g0AjIduWNlq2NRxL0PlBrbKVyZ6V/Ugc=
|
||||||
|
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
|
||||||
|
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
|
||||||
|
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
|
||||||
|
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
|
||||||
|
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
|
||||||
|
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
|
||||||
|
golang.org/x/net v0.26.0 h1:soB7SVo0PWrY4vPW/+ay0jKDNScG2X9wFeYlXIvJsOQ=
|
||||||
|
golang.org/x/net v0.26.0/go.mod h1:5YKkiSynbBIh3p6iOc/vibscux0x38BZDkn8sCUPxHE=
|
||||||
|
golang.org/x/oauth2 v0.21.0 h1:tsimM75w1tF/uws5rbeHzIWxEqElMehnc+iW793zsZs=
|
||||||
|
golang.org/x/oauth2 v0.21.0/go.mod h1:XYTD2NtWslqkgxebSiOHnXEap4TF09sJSc7H1sXbhtI=
|
||||||
|
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||||
|
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||||
|
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||||
golang.org/x/sync v0.17.0 h1:l60nONMj9l5drqw6jlhIELNv9I0A4OFgRsG9k2oT9Ug=
|
golang.org/x/sync v0.17.0 h1:l60nONMj9l5drqw6jlhIELNv9I0A4OFgRsG9k2oT9Ug=
|
||||||
golang.org/x/sync v0.17.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
|
golang.org/x/sync v0.17.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
|
||||||
|
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||||
|
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||||
|
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||||
golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik=
|
golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik=
|
||||||
golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
|
golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
|
||||||
|
golang.org/x/term v0.28.0 h1:/Ts8HFuMR2E6IP/jlo7QVLZHggjKQbhu/7H0LJFr3Gg=
|
||||||
|
golang.org/x/term v0.28.0/go.mod h1:Sw/lC2IAUZ92udQNf3WodGtn4k/XoLyZoh8v/8uiwek=
|
||||||
|
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||||
|
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
|
||||||
golang.org/x/text v0.29.0 h1:1neNs90w9YzJ9BocxfsQNHKuAT4pkghyXc4nhZ6sJvk=
|
golang.org/x/text v0.29.0 h1:1neNs90w9YzJ9BocxfsQNHKuAT4pkghyXc4nhZ6sJvk=
|
||||||
golang.org/x/text v0.29.0/go.mod h1:7MhJOA9CD2qZyOKYazxdYMF85OwPdEr9jTtBpO7ydH4=
|
golang.org/x/text v0.29.0/go.mod h1:7MhJOA9CD2qZyOKYazxdYMF85OwPdEr9jTtBpO7ydH4=
|
||||||
|
golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4=
|
||||||
|
golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
|
||||||
|
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||||
|
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
|
||||||
|
golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
|
||||||
|
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
|
||||||
|
golang.org/x/tools v0.36.0 h1:kWS0uv/zsvHEle1LbV5LE8QujrxB3wfQyxHfhOk0Qkg=
|
||||||
|
golang.org/x/tools v0.36.0/go.mod h1:WBDiHKJK8YgLHlcQPYQzNCkUxUypCaa5ZegCVutKm+s=
|
||||||
|
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||||
|
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||||
|
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||||
|
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||||
|
google.golang.org/protobuf v1.34.2 h1:6xV6lTsCfpGD21XK49h7MhtcApnLqkfYgPcdHftf6hg=
|
||||||
|
google.golang.org/protobuf v1.34.2/go.mod h1:qYOHts0dSfpeUzUFpOMr/WGzszTmLH+DiWniOlNbLDw=
|
||||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||||
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
|
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
|
||||||
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
|
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
|
||||||
|
gopkg.in/evanphx/json-patch.v4 v4.12.0 h1:n6jtcsulIzXPJaxegRbvFNNrZDjbij7ny3gmSPG+6V4=
|
||||||
|
gopkg.in/evanphx/json-patch.v4 v4.12.0/go.mod h1:p8EYWUEYMpynmqDbY58zCKCFZw8pRWMG4EsWvDvM72M=
|
||||||
|
gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc=
|
||||||
|
gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
|
||||||
|
gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||||
|
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
|
||||||
|
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
|
||||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||||
|
k8s.io/api v0.31.3 h1:umzm5o8lFbdN/hIXbrK9oRpOproJO62CV1zqxXrLgk8=
|
||||||
|
k8s.io/api v0.31.3/go.mod h1:UJrkIp9pnMOI9K2nlL6vwpxRzzEX5sWgn8kGQe92kCE=
|
||||||
|
k8s.io/apimachinery v0.31.3 h1:6l0WhcYgasZ/wk9ktLq5vLaoXJJr5ts6lkaQzgeYPq4=
|
||||||
|
k8s.io/apimachinery v0.31.3/go.mod h1:rsPdaZJfTfLsNJSQzNHQvYoTmxhoOEofxtOsF3rtsMo=
|
||||||
|
k8s.io/client-go v0.31.3 h1:CAlZuM+PH2cm+86LOBemaJI/lQ5linJ6UFxKX/SoG+4=
|
||||||
|
k8s.io/client-go v0.31.3/go.mod h1:2CgjPUTpv3fE5dNygAr2NcM8nhHzXvxB8KL5gYc3kJs=
|
||||||
|
k8s.io/klog/v2 v2.130.1 h1:n9Xl7H1Xvksem4KFG4PYbdQCQxqc/tTUyrgXaOhHSzk=
|
||||||
|
k8s.io/klog/v2 v2.130.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE=
|
||||||
|
k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 h1:BZqlfIlq5YbRMFko6/PM7FjZpUb45WallggurYhKGag=
|
||||||
|
k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340/go.mod h1:yD4MZYeKMBwQKVht279WycxKyM84kkAx2DPrTXaeb98=
|
||||||
|
k8s.io/utils v0.0.0-20240711033017-18e509b52bc8 h1:pUdcCO1Lk/tbT5ztQWOBi5HBgbBP1J8+AsQnQCKsi8A=
|
||||||
|
k8s.io/utils v0.0.0-20240711033017-18e509b52bc8/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0=
|
||||||
|
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd h1:EDPBXCAspyGV4jQlpZSudPeMmr1bNJefnuqLsRAsHZo=
|
||||||
|
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd/go.mod h1:B8JuhiUyNFVKdsE8h686QcCxMaH6HrOAZj4vswFpcB0=
|
||||||
|
sigs.k8s.io/structured-merge-diff/v4 v4.4.1 h1:150L+0vs/8DA78h1u02ooW1/fFq/Lwr+sGiqlzvrtq4=
|
||||||
|
sigs.k8s.io/structured-merge-diff/v4 v4.4.1/go.mod h1:N8hJocpFajUSSeSJ9bOZ77VzejKZaXsTtZo4/u7Io08=
|
||||||
|
sigs.k8s.io/yaml v1.4.0 h1:Mk1wCc2gy/F0THH0TAp1QYyJNzRm2KCLy3o5ASXVI5E=
|
||||||
|
sigs.k8s.io/yaml v1.4.0/go.mod h1:Ejl7/uTz7PSA4eKMyQCUTnhZYNmLIl+5c2lQPGR2BPY=
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
|
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
|
||||||
|
"github.com/mathiasbq/hyperguild/ingestion/internal/claim"
|
||||||
"github.com/mathiasbq/hyperguild/ingestion/internal/extract"
|
"github.com/mathiasbq/hyperguild/ingestion/internal/extract"
|
||||||
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
|
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
|
||||||
"github.com/mathiasbq/hyperguild/ingestion/internal/search"
|
"github.com/mathiasbq/hyperguild/ingestion/internal/search"
|
||||||
@@ -51,12 +52,13 @@ type queryRequest struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type writeRequest struct {
|
type writeRequest struct {
|
||||||
Content string `json:"content"`
|
Content string `json:"content"`
|
||||||
Filename string `json:"filename,omitempty"`
|
Filename string `json:"filename,omitempty"`
|
||||||
Type string `json:"type,omitempty"`
|
Type string `json:"type,omitempty"`
|
||||||
Domain string `json:"domain,omitempty"`
|
Domain string `json:"domain,omitempty"`
|
||||||
Wing string `json:"wing,omitempty"`
|
Wing string `json:"wing,omitempty"`
|
||||||
Hall string `json:"hall,omitempty"`
|
Hall string `json:"hall,omitempty"`
|
||||||
|
SourceType string `json:"source_type,omitempty"` // "external" opts a hall=facts entry out of the internal default
|
||||||
}
|
}
|
||||||
|
|
||||||
type ingestRequest struct {
|
type ingestRequest struct {
|
||||||
@@ -115,12 +117,13 @@ func (h *Handler) Query(w http.ResponseWriter, r *http.Request) {
|
|||||||
// When either is empty, the note falls back to brain/knowledge/<filename>
|
// When either is empty, the note falls back to brain/knowledge/<filename>
|
||||||
// with optional type/domain frontmatter (legacy behaviour).
|
// with optional type/domain frontmatter (legacy behaviour).
|
||||||
type WriteNoteOptions struct {
|
type WriteNoteOptions struct {
|
||||||
Content string
|
Content string
|
||||||
Filename string
|
Filename string
|
||||||
Type string
|
Type string
|
||||||
Domain string
|
Domain string
|
||||||
Wing string
|
Wing string
|
||||||
Hall string
|
Hall string
|
||||||
|
SourceType string // "internal" marks a first-party observation (e.g. claudewatcher) that needs no external citation
|
||||||
}
|
}
|
||||||
|
|
||||||
// WriteNote writes a markdown note into the brain. Returns the path
|
// WriteNote writes a markdown note into the brain. Returns the path
|
||||||
@@ -150,30 +153,143 @@ func writeHallNote(brainDir string, opts WriteNoteOptions) (string, error) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return "", err
|
return "", err
|
||||||
}
|
}
|
||||||
|
existingFields, body := splitFrontmatter(opts.Content)
|
||||||
|
existingByKey := make(map[string]frontmatterField, len(existingFields))
|
||||||
|
for _, f := range existingFields {
|
||||||
|
existingByKey[f.key] = f
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate the claim-property envelope BEFORE creating the directory, so a
|
||||||
|
// rejected write leaves nothing behind — not even an empty hall dir
|
||||||
|
// (brain#21). Fail closed: a value outside the enum is worse than an absent
|
||||||
|
// one, because a lint skips what it cannot parse and the note then reads as
|
||||||
|
// classified while nothing acts on it.
|
||||||
|
if err := claim.ValidateEnvelope(scalarValues(existingFields)); err != nil {
|
||||||
|
return "", fmt.Errorf("claim envelope: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
|
if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
|
||||||
return "", fmt.Errorf("create hall dir: %w", err)
|
return "", fmt.Errorf("create hall dir: %w", err)
|
||||||
}
|
}
|
||||||
|
emitted := make(map[string]bool, 6)
|
||||||
|
|
||||||
var fm strings.Builder
|
var fm strings.Builder
|
||||||
fm.WriteString("---\n")
|
fm.WriteString("---\n")
|
||||||
fmt.Fprintf(&fm, "wing: %s\n", brain.Sanitise(opts.Wing))
|
fmt.Fprintf(&fm, "wing: %s\n", brain.Sanitise(opts.Wing))
|
||||||
fmt.Fprintf(&fm, "hall: %s\n", opts.Hall)
|
fmt.Fprintf(&fm, "hall: %s\n", opts.Hall)
|
||||||
fmt.Fprintf(&fm, "created_at: %s\n", time.Now().UTC().Format(time.RFC3339))
|
fmt.Fprintf(&fm, "created_at: %s\n", time.Now().UTC().Format(time.RFC3339))
|
||||||
if opts.Type != "" {
|
emitted["wing"], emitted["hall"], emitted["created_at"] = true, true, true
|
||||||
fmt.Fprintf(&fm, "type: %s\n", opts.Type)
|
|
||||||
|
// writeField merges one key: opts.Content's own value (if the note already
|
||||||
|
// carries this field in its own frontmatter) always wins over the fallback,
|
||||||
|
// so promotion/extraction-step metadata survives verbatim instead of being
|
||||||
|
// shadowed by a second, stacked frontmatter block (#86).
|
||||||
|
writeField := func(key, fallback string) {
|
||||||
|
emitted[key] = true
|
||||||
|
if f, ok := existingByKey[key]; ok {
|
||||||
|
for _, line := range f.lines {
|
||||||
|
fm.WriteString(line)
|
||||||
|
fm.WriteString("\n")
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if fallback != "" {
|
||||||
|
fmt.Fprintf(&fm, "%s: %s\n", key, fallback)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
if opts.Domain != "" {
|
writeField("type", opts.Type)
|
||||||
fmt.Fprintf(&fm, "domain: %s\n", opts.Domain)
|
writeField("domain", opts.Domain)
|
||||||
|
|
||||||
|
sourceType := opts.SourceType
|
||||||
|
if sourceType == "" && opts.Hall == "facts" {
|
||||||
|
// Most hall=facts entries are first-party (an eval/benchmark the
|
||||||
|
// writer ran itself), not external claims — default to internal and
|
||||||
|
// require an explicit source_type: external opt-out for the rare
|
||||||
|
// citation-needing entry (brain-gardener#7).
|
||||||
|
sourceType = "internal"
|
||||||
|
}
|
||||||
|
writeField("source_type", sourceType)
|
||||||
|
|
||||||
|
for _, f := range existingFields {
|
||||||
|
if emitted[f.key] {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
for _, line := range f.lines {
|
||||||
|
fm.WriteString(line)
|
||||||
|
fm.WriteString("\n")
|
||||||
|
}
|
||||||
}
|
}
|
||||||
fm.WriteString("---\n")
|
fm.WriteString("---\n")
|
||||||
|
|
||||||
if err := os.WriteFile(dest, []byte(fm.String()+opts.Content), 0o644); err != nil {
|
if err := os.WriteFile(dest, []byte(fm.String()+body), 0o644); err != nil {
|
||||||
return "", fmt.Errorf("write: %w", err)
|
return "", fmt.Errorf("write: %w", err)
|
||||||
}
|
}
|
||||||
rel, _ := filepath.Rel(brainDir, dest)
|
rel, _ := filepath.Rel(brainDir, dest)
|
||||||
return filepath.ToSlash(rel), nil
|
return filepath.ToSlash(rel), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// scalarValues flattens frontmatter fields to key -> scalar value for
|
||||||
|
// validation. Multi-line values (lists, nested blocks) yield the text after
|
||||||
|
// the colon on the first line, which is empty for a list — correct here,
|
||||||
|
// since every claim-property field is a scalar enum.
|
||||||
|
func scalarValues(fields []frontmatterField) map[string]string {
|
||||||
|
out := make(map[string]string, len(fields))
|
||||||
|
for _, f := range fields {
|
||||||
|
if len(f.lines) == 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
_, value, found := strings.Cut(f.lines[0], ":")
|
||||||
|
if !found {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out[f.key] = strings.TrimSpace(value)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// frontmatterField is one top-level YAML key from a frontmatter block,
|
||||||
|
// along with its raw line and any indented continuation lines (e.g. a
|
||||||
|
// bulleted list value spanning multiple lines).
|
||||||
|
type frontmatterField struct {
|
||||||
|
key string
|
||||||
|
lines []string
|
||||||
|
}
|
||||||
|
|
||||||
|
// splitFrontmatter splits a leading "---\n...\n---\n" YAML block out of
|
||||||
|
// content, returning its top-level fields in original order and the
|
||||||
|
// remaining body. If content has no leading frontmatter block, fields is
|
||||||
|
// nil and body is content unchanged.
|
||||||
|
func splitFrontmatter(content string) (fields []frontmatterField, body string) {
|
||||||
|
if !strings.HasPrefix(content, "---\n") {
|
||||||
|
return nil, content
|
||||||
|
}
|
||||||
|
|
||||||
|
lines := strings.Split(content, "\n")
|
||||||
|
i := 1
|
||||||
|
var cur *frontmatterField
|
||||||
|
for ; i < len(lines); i++ {
|
||||||
|
line := lines[i]
|
||||||
|
if strings.TrimSpace(line) == "---" {
|
||||||
|
i++
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if line != "" && !strings.HasPrefix(line, " ") && !strings.HasPrefix(line, "\t") {
|
||||||
|
if cur != nil {
|
||||||
|
fields = append(fields, *cur)
|
||||||
|
}
|
||||||
|
key, _, _ := strings.Cut(line, ":")
|
||||||
|
cur = &frontmatterField{key: strings.TrimSpace(key), lines: []string{line}}
|
||||||
|
} else if cur != nil {
|
||||||
|
cur.lines = append(cur.lines, line)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if cur != nil {
|
||||||
|
fields = append(fields, *cur)
|
||||||
|
}
|
||||||
|
body = strings.Join(lines[i:], "\n")
|
||||||
|
return fields, body
|
||||||
|
}
|
||||||
|
|
||||||
// writeLegacyNote preserves the original brain/knowledge/ behaviour for
|
// writeLegacyNote preserves the original brain/knowledge/ behaviour for
|
||||||
// callers that have not adopted the wing/hall taxonomy.
|
// callers that have not adopted the wing/hall taxonomy.
|
||||||
func writeLegacyNote(brainDir string, opts WriteNoteOptions) (string, error) {
|
func writeLegacyNote(brainDir string, opts WriteNoteOptions) (string, error) {
|
||||||
@@ -332,11 +448,19 @@ func (h *Handler) Ingest(w http.ResponseWriter, r *http.Request) {
|
|||||||
writeJSON(w, ingestResponse{Pages: pages, Warnings: warnings})
|
writeJSON(w, ingestResponse{Pages: pages, Warnings: warnings})
|
||||||
}
|
}
|
||||||
|
|
||||||
// supportedExtensions lists file extensions that IngestPath will process.
|
// isSupportedExtension reports whether IngestPath will process ext.
|
||||||
var supportedExtensions = map[string]bool{
|
// .docx/.xlsx/.pptx/.png/.jpg/.jpeg require docmark (ADR-0013) and are only
|
||||||
".md": true,
|
// supported when DOCMARK_URL is configured — checked per-call (not cached at
|
||||||
".txt": true,
|
// package init) so it reflects the environment at request time.
|
||||||
".pdf": true,
|
func isSupportedExtension(ext string) bool {
|
||||||
|
switch ext {
|
||||||
|
case ".md", ".txt", ".pdf":
|
||||||
|
return true
|
||||||
|
case ".docx", ".xlsx", ".pptx", ".png", ".jpg", ".jpeg":
|
||||||
|
return os.Getenv("DOCMARK_URL") != ""
|
||||||
|
default:
|
||||||
|
return false
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// IngestPath handles POST /ingest-path — ingest a file or directory.
|
// IngestPath handles POST /ingest-path — ingest a file or directory.
|
||||||
@@ -369,7 +493,7 @@ func (h *Handler) IngestPath(w http.ResponseWriter, r *http.Request) {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
ext := strings.ToLower(filepath.Ext(path))
|
ext := strings.ToLower(filepath.Ext(path))
|
||||||
if !supportedExtensions[ext] {
|
if !isSupportedExtension(ext) {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
content, readErr := extract.Text(path)
|
content, readErr := extract.Text(path)
|
||||||
@@ -397,7 +521,7 @@ func (h *Handler) IngestPath(w http.ResponseWriter, r *http.Request) {
|
|||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
ext := strings.ToLower(filepath.Ext(req.Path))
|
ext := strings.ToLower(filepath.Ext(req.Path))
|
||||||
if !supportedExtensions[ext] {
|
if !isSupportedExtension(ext) {
|
||||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("unsupported file extension: %s", ext))
|
writeError(w, http.StatusBadRequest, fmt.Sprintf("unsupported file extension: %s", ext))
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -118,6 +118,116 @@ func TestWrite_IncludesFrontmatterWhenTypeProvided(t *testing.T) {
|
|||||||
assert.Contains(t, string(content), "Some learning.")
|
assert.Contains(t, string(content), "Some learning.")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestWriteNote_HallRouteIncludesSourceTypeWhenSet(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
|
||||||
|
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
|
||||||
|
Content: "# Claude session abc (koala)\n\nBody.\n",
|
||||||
|
Filename: "session-koala-abc",
|
||||||
|
Wing: "claude-sessions",
|
||||||
|
Hall: "facts",
|
||||||
|
Type: "source",
|
||||||
|
SourceType: "internal",
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Contains(t, string(got), "source_type: internal")
|
||||||
|
assert.Contains(t, string(got), "wing: claude-sessions")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWriteNote_HallFactsDefaultsSourceTypeInternalWhenUnset(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
|
||||||
|
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
|
||||||
|
Content: "manually captured fact.\n",
|
||||||
|
Wing: "agentsquad",
|
||||||
|
Hall: "facts",
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
|
||||||
|
require.NoError(t, err)
|
||||||
|
// Most hall=facts entries are first-party (an eval/benchmark the agent ran
|
||||||
|
// itself), not external claims — default to internal, require explicit
|
||||||
|
// opt-out for the rare case that does need a citation (brain-gardener#7).
|
||||||
|
assert.Contains(t, string(got), "source_type: internal")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWriteNote_HallFactsPreservesExplicitExternalSourceType(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
|
||||||
|
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
|
||||||
|
Content: "vendor pricing claim, needs a citation.\n",
|
||||||
|
Wing: "agentsquad",
|
||||||
|
Hall: "facts",
|
||||||
|
SourceType: "external",
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Contains(t, string(got), "source_type: external")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWriteNote_HallRouteOmitsSourceTypeForNonFactsHalls(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
|
||||||
|
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
|
||||||
|
Content: "a decision record.\n",
|
||||||
|
Wing: "agentsquad",
|
||||||
|
Hall: "decisions",
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.NotContains(t, string(got), "source_type")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWriteNote_HallRouteMergesExistingFrontmatterInsteadOfStacking(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
|
||||||
|
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
|
||||||
|
Content: "---\ntitle: act_runner host-executor\ntags: [gitea-actions, act_runner]\n---\n\n# Body\n\nSome content.\n",
|
||||||
|
Filename: "act-runner-host-executor",
|
||||||
|
Wing: "homelab",
|
||||||
|
Hall: "failures",
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
|
||||||
|
require.NoError(t, err)
|
||||||
|
body := string(got)
|
||||||
|
|
||||||
|
// exactly one frontmatter block: only two "---" delimiter lines total
|
||||||
|
assert.Equal(t, 2, strings.Count(body, "---\n"), "expected a single merged frontmatter block, not stacked blocks")
|
||||||
|
assert.Contains(t, body, "wing: homelab")
|
||||||
|
assert.Contains(t, body, "hall: failures")
|
||||||
|
assert.Contains(t, body, "title: act_runner host-executor")
|
||||||
|
assert.Contains(t, body, "tags: [gitea-actions, act_runner]")
|
||||||
|
assert.Contains(t, body, "# Body")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWriteNote_HallRouteExistingTypeWinsOverOptsType(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
|
||||||
|
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
|
||||||
|
Content: "---\ntype: hypothesis\n---\n\nBody.\n",
|
||||||
|
Filename: "note",
|
||||||
|
Wing: "agentsquad",
|
||||||
|
Hall: "decisions",
|
||||||
|
Type: "decision", // should lose to content's own "type: hypothesis"
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Contains(t, string(got), "type: hypothesis")
|
||||||
|
assert.NotContains(t, string(got), "type: decision")
|
||||||
|
}
|
||||||
|
|
||||||
func TestWrite_GeneratesFilenameIfAbsent(t *testing.T) {
|
func TestWrite_GeneratesFilenameIfAbsent(t *testing.T) {
|
||||||
dir, h := setup(t)
|
dir, h := setup(t)
|
||||||
body, _ := json.Marshal(map[string]any{"content": "auto name"})
|
body, _ := json.Marshal(map[string]any{"content": "auto name"})
|
||||||
@@ -330,3 +440,60 @@ func TestIngestPath_Directory(t *testing.T) {
|
|||||||
require.True(t, ok, "pages must be an array")
|
require.True(t, ok, "pages must be an array")
|
||||||
assert.NotEmpty(t, pagesSlice)
|
assert.NotEmpty(t, pagesSlice)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// --- brain#21: claim-property envelope on the write path -------------------
|
||||||
|
|
||||||
|
func TestWriteNote_PreservesClaimEnvelopeVerbatim(t *testing.T) {
|
||||||
|
// writeHallNote already passes unknown keys through; this pins that
|
||||||
|
// behaviour for the envelope specifically, so a future refactor of the
|
||||||
|
// merge loop cannot silently drop it.
|
||||||
|
dir := t.TempDir()
|
||||||
|
|
||||||
|
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
|
||||||
|
Content: "---\nhalf_life: durable\napplies: universal\nclaim_provenance: asserted\n---\n\nA durable claim.\n",
|
||||||
|
Wing: "homelab",
|
||||||
|
Hall: "decisions",
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Contains(t, string(got), "half_life: durable")
|
||||||
|
assert.Contains(t, string(got), "applies: universal")
|
||||||
|
assert.Contains(t, string(got), "claim_provenance: asserted")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWriteNote_RejectsInvalidClaimEnvelopeAndWritesNothing(t *testing.T) {
|
||||||
|
// Fail closed: a value outside the enum must not land. A lint skips what
|
||||||
|
// it cannot parse, so an unvalidated write produces a note that reads as
|
||||||
|
// classified while nothing acts on it.
|
||||||
|
dir := t.TempDir()
|
||||||
|
|
||||||
|
_, err := api.WriteNote(dir, api.WriteNoteOptions{
|
||||||
|
Content: "---\nhalf_life: forever\nclaim_provenance: asserted\n---\n\nBody.\n",
|
||||||
|
Filename: "bad-envelope",
|
||||||
|
Wing: "homelab",
|
||||||
|
Hall: "facts",
|
||||||
|
})
|
||||||
|
require.Error(t, err)
|
||||||
|
assert.Contains(t, err.Error(), "half_life")
|
||||||
|
|
||||||
|
_, statErr := os.Stat(filepath.Join(dir, "wiki/homelab/facts/bad-envelope.md"))
|
||||||
|
assert.True(t, os.IsNotExist(statErr), "a rejected write must leave no file behind")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWriteNote_RejectsClaimFieldWithoutProvenance(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
|
||||||
|
_, err := api.WriteNote(dir, api.WriteNoteOptions{
|
||||||
|
Content: "---\napplies: homelab\n---\n\nBody.\n",
|
||||||
|
Filename: "no-provenance",
|
||||||
|
Wing: "homelab",
|
||||||
|
Hall: "facts",
|
||||||
|
})
|
||||||
|
require.Error(t, err)
|
||||||
|
assert.Contains(t, err.Error(), "claim_provenance")
|
||||||
|
|
||||||
|
_, statErr := os.Stat(filepath.Join(dir, "wiki/homelab/facts/no-provenance.md"))
|
||||||
|
assert.True(t, os.IsNotExist(statErr))
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,61 @@
|
|||||||
|
// ingestion/internal/api/ingestpath_docmark_test.go
|
||||||
|
package api_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestIngestPath_DocxUnsupportedWhenDocmarkNotConfigured(t *testing.T) {
|
||||||
|
t.Setenv("DOCMARK_URL", "")
|
||||||
|
_, h := setup(t)
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
f := filepath.Join(dir, "doc.docx")
|
||||||
|
require.NoError(t, os.WriteFile(f, []byte("fake docx"), 0o644))
|
||||||
|
|
||||||
|
body, _ := json.Marshal(map[string]any{"path": f, "source": "test-doc", "dry_run": true})
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/ingest-path", bytes.NewReader(body))
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.IngestPath(rec, req)
|
||||||
|
|
||||||
|
assert.Equal(t, http.StatusBadRequest, rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestIngestPath_DocxSupportedWhenDocmarkConfigured(t *testing.T) {
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
_, _ = w.Write([]byte(`{"jsonrpc":"2.0","id":1,"result":{"content":[{"type":"text","text":"# Converted Doc"}]}}`))
|
||||||
|
}))
|
||||||
|
defer srv.Close()
|
||||||
|
t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
|
||||||
|
t.Setenv("DOCMARK_BEARER_TOKEN", "tok")
|
||||||
|
|
||||||
|
_, h := setup(t)
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
f := filepath.Join(dir, "doc.docx")
|
||||||
|
require.NoError(t, os.WriteFile(f, []byte("fake docx"), 0o644))
|
||||||
|
|
||||||
|
body, _ := json.Marshal(map[string]any{"path": f, "source": "test-doc", "dry_run": true})
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/ingest-path", bytes.NewReader(body))
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.IngestPath(rec, req)
|
||||||
|
|
||||||
|
require.Equal(t, http.StatusOK, rec.Code, rec.Body.String())
|
||||||
|
var resp map[string]any
|
||||||
|
require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &resp))
|
||||||
|
pages, ok := resp["pages"].([]any)
|
||||||
|
require.True(t, ok)
|
||||||
|
assert.NotEmpty(t, pages)
|
||||||
|
}
|
||||||
@@ -1,8 +1,7 @@
|
|||||||
// Package capture is the Clean-Architecture use-case for the uniform
|
// Package capture is the Clean-Architecture use-case for the uniform
|
||||||
// capture capability (issue #49/#51): persist a finished session's
|
// capture capability (issue #49/#51): persist a finished session's
|
||||||
// valuable output — insights → brain, action items → Gitea tickets,
|
// valuable output — insights → brain, action items → Gitea tickets —
|
||||||
// optional summary → ai-sessions — with one invocation, identical core
|
// with one invocation, identical core behaviour across every harness.
|
||||||
// behaviour across every harness.
|
|
||||||
//
|
//
|
||||||
// This package is pure orchestration. It depends only on ports
|
// This package is pure orchestration. It depends only on ports
|
||||||
// (interfaces) and plain entities — no HTTP, no live Gitea, no embedding
|
// (interfaces) and plain entities — no HTTP, no live Gitea, no embedding
|
||||||
@@ -83,19 +82,11 @@ type Ticket struct {
|
|||||||
Body string
|
Body string
|
||||||
}
|
}
|
||||||
|
|
||||||
// Summary is an optional session summary bound for ai-sessions.
|
|
||||||
type Summary struct {
|
|
||||||
Title string
|
|
||||||
Body string
|
|
||||||
ReposTouched []string
|
|
||||||
}
|
|
||||||
|
|
||||||
// CaptureInput is the whole capture request.
|
// CaptureInput is the whole capture request.
|
||||||
type CaptureInput struct {
|
type CaptureInput struct {
|
||||||
Context CaptureContext
|
Context CaptureContext
|
||||||
Insights []Insight
|
Insights []Insight
|
||||||
Tickets []Ticket
|
Tickets []Ticket
|
||||||
Summary *Summary
|
|
||||||
DryRun bool
|
DryRun bool
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -117,12 +108,6 @@ type TicketResult struct {
|
|||||||
OK bool `json:"ok"`
|
OK bool `json:"ok"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// SummaryResult is the summary outcome in the receipt.
|
|
||||||
type SummaryResult struct {
|
|
||||||
Path string `json:"path,omitempty"`
|
|
||||||
OK bool `json:"ok"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// ItemError pins a failure to a specific request item for the partial
|
// ItemError pins a failure to a specific request item for the partial
|
||||||
// receipt. Item is a stable locator like "insight[1]" or "ticket[0]".
|
// receipt. Item is a stable locator like "insight[1]" or "ticket[0]".
|
||||||
type ItemError struct {
|
type ItemError struct {
|
||||||
@@ -136,7 +121,6 @@ type ItemError struct {
|
|||||||
type CaptureReceipt struct {
|
type CaptureReceipt struct {
|
||||||
Insights []InsightResult `json:"insights"`
|
Insights []InsightResult `json:"insights"`
|
||||||
Tickets []TicketResult `json:"tickets"`
|
Tickets []TicketResult `json:"tickets"`
|
||||||
Summary *SummaryResult `json:"summary,omitempty"`
|
|
||||||
Errors []ItemError `json:"errors"`
|
Errors []ItemError `json:"errors"`
|
||||||
EffectiveClassification string `json:"effective_classification,omitempty"`
|
EffectiveClassification string `json:"effective_classification,omitempty"`
|
||||||
DryRun bool `json:"dry_run"`
|
DryRun bool `json:"dry_run"`
|
||||||
|
|||||||
@@ -69,11 +69,6 @@ type IssueTracker interface {
|
|||||||
CommentIssue(ctx context.Context, repo string, number int, body string) (IssueRef, error)
|
CommentIssue(ctx context.Context, repo string, number int, body string) (IssueRef, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
// SummaryWriter is the ai-sessions summary port.
|
|
||||||
type SummaryWriter interface {
|
|
||||||
WriteFile(ctx context.Context, repo, path, content string) error
|
|
||||||
}
|
|
||||||
|
|
||||||
// ClassificationPolicy derives a target's sensitivity (model C). The
|
// ClassificationPolicy derives a target's sensitivity (model C). The
|
||||||
// "stricter wins" combination of declared vs derived is use-case policy
|
// "stricter wins" combination of declared vs derived is use-case policy
|
||||||
// and lives in the service, so the port stays minimal. Satisfied by
|
// and lives in the service, so the port stays minimal. Satisfied by
|
||||||
|
|||||||
@@ -2,8 +2,6 @@ package capture
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"crypto/sha256"
|
|
||||||
"encoding/hex"
|
|
||||||
"fmt"
|
"fmt"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
@@ -14,22 +12,19 @@ import (
|
|||||||
|
|
||||||
// Service is the CaptureSession use-case. It depends only on ports.
|
// Service is the CaptureSession use-case. It depends only on ports.
|
||||||
type Service struct {
|
type Service struct {
|
||||||
brain BrainStore
|
brain BrainStore
|
||||||
issues IssueTracker
|
issues IssueTracker
|
||||||
summaries SummaryWriter
|
policy ClassificationPolicy
|
||||||
policy ClassificationPolicy
|
audit AuditSink
|
||||||
audit AuditSink
|
|
||||||
|
|
||||||
// now is the clock, injectable for deterministic summary paths and
|
// now is the clock, injectable for deterministic audit timestamps in
|
||||||
// audit timestamps in tests.
|
// tests.
|
||||||
now func() time.Time
|
now func() time.Time
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewService constructs a Service from its ports. summaries may be nil
|
// NewService constructs a Service from its ports.
|
||||||
// when no summary persistence is wired; a CaptureInput with a Summary
|
func NewService(b BrainStore, tr IssueTracker, p ClassificationPolicy, a AuditSink) *Service {
|
||||||
// then fails that item rather than panicking.
|
return &Service{brain: b, issues: tr, policy: p, audit: a, now: time.Now}
|
||||||
func NewService(b BrainStore, tr IssueTracker, sw SummaryWriter, p ClassificationPolicy, a AuditSink) *Service {
|
|
||||||
return &Service{brain: b, issues: tr, summaries: sw, policy: p, audit: a, now: time.Now}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
var validActions = map[string]bool{"create": true, "close": true, "comment": true}
|
var validActions = map[string]bool{"create": true, "close": true, "comment": true}
|
||||||
@@ -88,7 +83,7 @@ func (s *Service) Capture(ctx context.Context, in CaptureInput) (CaptureReceipt,
|
|||||||
declared, _ = classification.ParseLevel(in.Context.Classification)
|
declared, _ = classification.ParseLevel(in.Context.Classification)
|
||||||
}
|
}
|
||||||
|
|
||||||
effective, securityEvents := s.resolveClassification(declared, in)
|
effective, securityEvents, escalators := s.resolveClassification(declared, in)
|
||||||
|
|
||||||
// Server-derived origin governs the I1 gate; a caller-asserted harness
|
// Server-derived origin governs the I1 gate; a caller-asserted harness
|
||||||
// label that names a different zone is descriptive-only and logged as a
|
// label that names a different zone is descriptive-only and logged as a
|
||||||
@@ -112,8 +107,8 @@ func (s *Service) Capture(ctx context.Context, in CaptureInput) (CaptureReceipt,
|
|||||||
Items: nil, // refused before any write
|
Items: nil, // refused before any write
|
||||||
SecurityEvents: append(securityEvents, "I1 refusal: confidential capture via us-nexus origin"),
|
SecurityEvents: append(securityEvents, "I1 refusal: confidential capture via us-nexus origin"),
|
||||||
}, AuditCentral)
|
}, AuditCentral)
|
||||||
return CaptureReceipt{}, fmt.Errorf("%w: effective classification confidential through %s origin",
|
return CaptureReceipt{}, fmt.Errorf("%w: effective classification confidential through %s origin (%s)",
|
||||||
ErrSovereigntyRefused, in.Context.Origin)
|
ErrSovereigntyRefused, in.Context.Origin, escalationReason(escalators))
|
||||||
}
|
}
|
||||||
|
|
||||||
receipt := CaptureReceipt{
|
receipt := CaptureReceipt{
|
||||||
@@ -131,9 +126,6 @@ func (s *Service) Capture(ctx context.Context, in CaptureInput) (CaptureReceipt,
|
|||||||
for _, tk := range in.Tickets {
|
for _, tk := range in.Tickets {
|
||||||
receipt.Tickets = append(receipt.Tickets, TicketResult{Repo: tk.Repo, Action: tk.Action, Number: tk.Number, OK: true})
|
receipt.Tickets = append(receipt.Tickets, TicketResult{Repo: tk.Repo, Action: tk.Action, Number: tk.Number, OK: true})
|
||||||
}
|
}
|
||||||
if in.Summary != nil {
|
|
||||||
receipt.Summary = &SummaryResult{Path: s.summaryPath(in.Context, in.Summary), OK: true}
|
|
||||||
}
|
|
||||||
return receipt, nil
|
return receipt, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -169,16 +161,6 @@ func (s *Service) Capture(ctx context.Context, in CaptureInput) (CaptureReceipt,
|
|||||||
landed = append(landed, fmt.Sprintf("ticket:%s#%d", tk.Repo, res.Number))
|
landed = append(landed, fmt.Sprintf("ticket:%s#%d", tk.Repo, res.Number))
|
||||||
}
|
}
|
||||||
|
|
||||||
if in.Summary != nil {
|
|
||||||
res, err := s.persistSummary(ctx, in.Context, in.Summary)
|
|
||||||
receipt.Summary = &res
|
|
||||||
if err != nil {
|
|
||||||
receipt.Errors = append(receipt.Errors, ItemError{Item: "summary", Error: err.Error()})
|
|
||||||
} else {
|
|
||||||
landed = append(landed, "summary:"+res.Path)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// I5: persist the request-level audit record of exactly what landed,
|
// I5: persist the request-level audit record of exactly what landed,
|
||||||
// using the outcome reserved before the writes. AuditBuffered surfaces
|
// using the outcome reserved before the writes. AuditBuffered surfaces
|
||||||
// the degraded (locally-buffered) state on the receipt.
|
// the degraded (locally-buffered) state on the receipt.
|
||||||
@@ -239,15 +221,28 @@ func (s *Service) validate(in CaptureInput) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// resolveClassification computes the effective level (stricter of
|
// resolveClassification computes the effective level (stricter of declared
|
||||||
// declared and every target's derived level) and collects a security
|
// and every target's derived level), collects a security event whenever the
|
||||||
// event whenever the caller under-declared relative to a target floor.
|
// caller under-declared relative to a target floor, and names the targets
|
||||||
func (s *Service) resolveClassification(declared classification.Level, in CaptureInput) (classification.Level, []string) {
|
// that carry the strictest level.
|
||||||
|
//
|
||||||
|
// That third return is what makes an I1 refusal actionable. A caller sending
|
||||||
|
// twenty tickets gets one refusal; without the offending target it cannot
|
||||||
|
// tell which repo tripped the gate, and infra#265 shows what happens next —
|
||||||
|
// the caller concluded the tickets ARRAY was broken, built a fallback of
|
||||||
|
// twenty individual calls, and filed a bug against the wrong thing.
|
||||||
|
func (s *Service) resolveClassification(declared classification.Level, in CaptureInput) (classification.Level, []string, []string) {
|
||||||
effective := declared
|
effective := declared
|
||||||
var events []string
|
var events []string
|
||||||
|
derivedFor := map[string]classification.Level{}
|
||||||
|
var order []string
|
||||||
consider := func(kind classification.TargetKind, name string) {
|
consider := func(kind classification.TargetKind, name string) {
|
||||||
derived := s.policy.Derive(classification.Target{Kind: kind, Name: name})
|
derived := s.policy.Derive(classification.Target{Kind: kind, Name: name})
|
||||||
effective = classification.Stricter(effective, derived)
|
effective = classification.Stricter(effective, derived)
|
||||||
|
if _, seen := derivedFor[name]; !seen {
|
||||||
|
derivedFor[name] = derived
|
||||||
|
order = append(order, name)
|
||||||
|
}
|
||||||
if declared < derived {
|
if declared < derived {
|
||||||
events = append(events, fmt.Sprintf("classification under-declared: declared=%s target=%s(%s) derived=%s",
|
events = append(events, fmt.Sprintf("classification under-declared: declared=%s target=%s(%s) derived=%s",
|
||||||
declared, name, kindString(kind), derived))
|
declared, name, kindString(kind), derived))
|
||||||
@@ -259,12 +254,28 @@ func (s *Service) resolveClassification(declared classification.Level, in Captur
|
|||||||
for _, tk := range in.Tickets {
|
for _, tk := range in.Tickets {
|
||||||
consider(classification.RepoTarget, tk.Repo)
|
consider(classification.RepoTarget, tk.Repo)
|
||||||
}
|
}
|
||||||
if in.Summary != nil {
|
|
||||||
for _, repo := range in.Summary.ReposTouched {
|
// Only targets that actually reach the effective level are named: at
|
||||||
consider(classification.RepoTarget, repo)
|
// twenty tickets, listing the innocent ones is noise that hides the one
|
||||||
|
// that matters.
|
||||||
|
var escalators []string
|
||||||
|
for _, name := range order {
|
||||||
|
if derivedFor[name] == effective && effective > declared {
|
||||||
|
escalators = append(escalators, name)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return effective, events
|
return effective, events, escalators
|
||||||
|
}
|
||||||
|
|
||||||
|
// escalationReason renders why the effective classification landed where it
|
||||||
|
// did, for an operator reading a refusal. Naming the caller's own declaration
|
||||||
|
// matters as much as naming a target: "you asked for this" and "a repo you
|
||||||
|
// touched required it" call for different fixes.
|
||||||
|
func escalationReason(escalators []string) string {
|
||||||
|
if len(escalators) == 0 {
|
||||||
|
return "declared by the caller; no target required it"
|
||||||
|
}
|
||||||
|
return "required by: " + strings.Join(escalators, ", ")
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *Service) persistInsight(ctx context.Context, ins Insight) (InsightResult, string, error) {
|
func (s *Service) persistInsight(ctx context.Context, ins Insight) (InsightResult, string, error) {
|
||||||
@@ -309,60 +320,6 @@ func (s *Service) persistTicket(ctx context.Context, tk Ticket) (TicketResult, e
|
|||||||
return res, nil
|
return res, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *Service) persistSummary(ctx context.Context, c CaptureContext, sum *Summary) (SummaryResult, error) {
|
|
||||||
if s.summaries == nil {
|
|
||||||
return SummaryResult{OK: false}, fmt.Errorf("no summary writer configured")
|
|
||||||
}
|
|
||||||
path := s.summaryPath(c, sum)
|
|
||||||
content := s.renderSummary(c, sum)
|
|
||||||
repo := "ai-sessions"
|
|
||||||
if err := s.summaries.WriteFile(ctx, repo, path, content); err != nil {
|
|
||||||
return SummaryResult{Path: path, OK: false}, err
|
|
||||||
}
|
|
||||||
return SummaryResult{Path: path, OK: true}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// summaryPath builds summaries/<harness>/<YYYY-MM>/<date>-<slug>-<ref8>.md.
|
|
||||||
// The ref8 disambiguator is derived from the session_ref (or the title
|
|
||||||
// when no ref is present) so distinct sessions never collide.
|
|
||||||
func (s *Service) summaryPath(c CaptureContext, sum *Summary) string {
|
|
||||||
t := s.now().UTC()
|
|
||||||
slug := brain.Sanitise(sum.Title)
|
|
||||||
if slug == "" {
|
|
||||||
slug = "summary"
|
|
||||||
}
|
|
||||||
seed := c.SessionRef
|
|
||||||
if seed == "" {
|
|
||||||
seed = sum.Title + sum.Body
|
|
||||||
}
|
|
||||||
sum8 := shortHash(seed)
|
|
||||||
return fmt.Sprintf("summaries/%s/%s/%s-%s-%s.md",
|
|
||||||
brain.Sanitise(c.Harness), t.Format("2006-01"), t.Format("2006-01-02"), slug, sum8)
|
|
||||||
}
|
|
||||||
|
|
||||||
// renderSummary stamps fidelity + session metadata into frontmatter so the
|
|
||||||
// richer-fidelity-supersedes-thinner collision rule has the data it needs.
|
|
||||||
func (s *Service) renderSummary(c CaptureContext, sum *Summary) string {
|
|
||||||
var b strings.Builder
|
|
||||||
b.WriteString("---\n")
|
|
||||||
fmt.Fprintf(&b, "title: %s\n", sum.Title)
|
|
||||||
fmt.Fprintf(&b, "harness: %s\n", c.Harness)
|
|
||||||
if c.SessionRef != "" {
|
|
||||||
fmt.Fprintf(&b, "session_ref: %s\n", c.SessionRef)
|
|
||||||
}
|
|
||||||
fmt.Fprintf(&b, "fidelity: %s\n", c.Fidelity)
|
|
||||||
fmt.Fprintf(&b, "captured_at: %s\n", s.now().UTC().Format(time.RFC3339))
|
|
||||||
if len(sum.ReposTouched) > 0 {
|
|
||||||
fmt.Fprintf(&b, "repos_touched: [%s]\n", strings.Join(sum.ReposTouched, ", "))
|
|
||||||
}
|
|
||||||
b.WriteString("---\n\n")
|
|
||||||
b.WriteString(sum.Body)
|
|
||||||
if !strings.HasSuffix(sum.Body, "\n") {
|
|
||||||
b.WriteByte('\n')
|
|
||||||
}
|
|
||||||
return b.String()
|
|
||||||
}
|
|
||||||
|
|
||||||
func kindString(k classification.TargetKind) string {
|
func kindString(k classification.TargetKind) string {
|
||||||
if k == classification.RepoTarget {
|
if k == classification.RepoTarget {
|
||||||
return "repo"
|
return "repo"
|
||||||
@@ -381,8 +338,3 @@ func firstLine(s string) string {
|
|||||||
}
|
}
|
||||||
return s
|
return s
|
||||||
}
|
}
|
||||||
|
|
||||||
func shortHash(s string) string {
|
|
||||||
sum := sha256.Sum256([]byte(s))
|
|
||||||
return hex.EncodeToString(sum[:])[:8]
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ package capture
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
|
"fmt"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
@@ -59,12 +60,18 @@ type fakeTracker struct {
|
|||||||
closed []int
|
closed []int
|
||||||
comments []int
|
comments []int
|
||||||
err error
|
err error
|
||||||
|
// failOnTitle makes CreateIssue fail for one specific title, so a
|
||||||
|
// partial-failure receipt can be exercised without failing every item.
|
||||||
|
failOnTitle string
|
||||||
}
|
}
|
||||||
|
|
||||||
func (f *fakeTracker) CreateIssue(_ context.Context, repo, title, _ string) (IssueRef, error) {
|
func (f *fakeTracker) CreateIssue(_ context.Context, repo, title, _ string) (IssueRef, error) {
|
||||||
if f.err != nil {
|
if f.err != nil {
|
||||||
return IssueRef{}, f.err
|
return IssueRef{}, f.err
|
||||||
}
|
}
|
||||||
|
if f.failOnTitle != "" && title == f.failOnTitle {
|
||||||
|
return IssueRef{}, errors.New("tracker refused this one")
|
||||||
|
}
|
||||||
f.created = append(f.created, repo+":"+title)
|
f.created = append(f.created, repo+":"+title)
|
||||||
return IssueRef{Repo: repo, Number: 100 + len(f.created), URL: "https://git/" + repo + "/issues/x"}, nil
|
return IssueRef{Repo: repo, Number: 100 + len(f.created), URL: "https://git/" + repo + "/issues/x"}, nil
|
||||||
}
|
}
|
||||||
@@ -85,24 +92,11 @@ func (f *fakeTracker) CommentIssue(_ context.Context, repo string, number int, _
|
|||||||
return IssueRef{Repo: repo, Number: number}, nil
|
return IssueRef{Repo: repo, Number: number}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
type fakeSummary struct {
|
|
||||||
paths []string
|
|
||||||
content []string
|
|
||||||
err error
|
|
||||||
}
|
|
||||||
|
|
||||||
func (f *fakeSummary) WriteFile(_ context.Context, _, path, content string) error {
|
|
||||||
if f.err != nil {
|
|
||||||
return f.err
|
|
||||||
}
|
|
||||||
f.paths = append(f.paths, path)
|
|
||||||
f.content = append(f.content, content)
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// fakePolicy derives from an explicit map; default Internal so tests pin
|
// fakePolicy derives from an explicit map; default Internal so tests pin
|
||||||
// behaviour without depending on the real defaulting.
|
// behaviour without depending on the real defaulting.
|
||||||
type fakePolicy struct{ tags map[string]classification.Level }
|
type fakePolicy struct {
|
||||||
|
tags map[string]classification.Level
|
||||||
|
}
|
||||||
|
|
||||||
func (p fakePolicy) Derive(t classification.Target) classification.Level {
|
func (p fakePolicy) Derive(t classification.Target) classification.Level {
|
||||||
if lvl, ok := p.tags[t.Name]; ok {
|
if lvl, ok := p.tags[t.Name]; ok {
|
||||||
@@ -135,8 +129,8 @@ func (f *fakeAudit) Record(_ context.Context, e AuditEntry, _ AuditOutcome) erro
|
|||||||
|
|
||||||
// --- helpers ---
|
// --- helpers ---
|
||||||
|
|
||||||
func newSvc(b BrainStore, tr IssueTracker, sw SummaryWriter, p ClassificationPolicy, a AuditSink) *Service {
|
func newSvc(b BrainStore, tr IssueTracker, p ClassificationPolicy, a AuditSink) *Service {
|
||||||
s := NewService(b, tr, sw, p, a)
|
s := NewService(b, tr, p, a)
|
||||||
s.now = func() time.Time { return time.Date(2026, 6, 22, 12, 0, 0, 0, time.UTC) }
|
s.now = func() time.Time { return time.Date(2026, 6, 22, 12, 0, 0, 0, time.UTC) }
|
||||||
return s
|
return s
|
||||||
}
|
}
|
||||||
@@ -151,7 +145,7 @@ func TestCaptureHappyPath(t *testing.T) {
|
|||||||
b := &fakeBrain{}
|
b := &fakeBrain{}
|
||||||
tr := &fakeTracker{}
|
tr := &fakeTracker{}
|
||||||
au := &fakeAudit{}
|
au := &fakeAudit{}
|
||||||
svc := newSvc(b, tr, nil, fakePolicy{}, au)
|
svc := newSvc(b, tr, fakePolicy{}, au)
|
||||||
|
|
||||||
rec, err := svc.Capture(context.Background(), CaptureInput{
|
rec, err := svc.Capture(context.Background(), CaptureInput{
|
||||||
Context: baseCtx(),
|
Context: baseCtx(),
|
||||||
@@ -180,7 +174,7 @@ func TestCaptureHappyPath(t *testing.T) {
|
|||||||
|
|
||||||
func TestCaptureSupersedeNotDuplicate(t *testing.T) {
|
func TestCaptureSupersedeNotDuplicate(t *testing.T) {
|
||||||
b := &fakeBrain{}
|
b := &fakeBrain{}
|
||||||
svc := newSvc(b, &fakeTracker{}, nil, fakePolicy{}, &fakeAudit{})
|
svc := newSvc(b, &fakeTracker{}, fakePolicy{}, &fakeAudit{})
|
||||||
|
|
||||||
rec, err := svc.Capture(context.Background(), CaptureInput{
|
rec, err := svc.Capture(context.Background(), CaptureInput{
|
||||||
Context: baseCtx(),
|
Context: baseCtx(),
|
||||||
@@ -197,7 +191,7 @@ func TestCaptureValidationFailClosed(t *testing.T) {
|
|||||||
b := &fakeBrain{}
|
b := &fakeBrain{}
|
||||||
tr := &fakeTracker{}
|
tr := &fakeTracker{}
|
||||||
au := &fakeAudit{}
|
au := &fakeAudit{}
|
||||||
svc := newSvc(b, tr, nil, fakePolicy{}, au)
|
svc := newSvc(b, tr, fakePolicy{}, au)
|
||||||
|
|
||||||
_, err := svc.Capture(context.Background(), CaptureInput{
|
_, err := svc.Capture(context.Background(), CaptureInput{
|
||||||
Context: baseCtx(),
|
Context: baseCtx(),
|
||||||
@@ -216,7 +210,7 @@ func TestCaptureValidationFailClosed(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestCaptureValidationRejectsBadTicket(t *testing.T) {
|
func TestCaptureValidationRejectsBadTicket(t *testing.T) {
|
||||||
svc := newSvc(&fakeBrain{}, &fakeTracker{}, nil, fakePolicy{}, &fakeAudit{})
|
svc := newSvc(&fakeBrain{}, &fakeTracker{}, fakePolicy{}, &fakeAudit{})
|
||||||
_, err := svc.Capture(context.Background(), CaptureInput{
|
_, err := svc.Capture(context.Background(), CaptureInput{
|
||||||
Context: baseCtx(),
|
Context: baseCtx(),
|
||||||
Tickets: []Ticket{{Repo: "hyperguild", Action: "frobnicate"}}, // bad action
|
Tickets: []Ticket{{Repo: "hyperguild", Action: "frobnicate"}}, // bad action
|
||||||
@@ -239,7 +233,7 @@ func TestCapturePartialFailureBestEffort(t *testing.T) {
|
|||||||
}}
|
}}
|
||||||
tr := &fakeTracker{}
|
tr := &fakeTracker{}
|
||||||
au := &fakeAudit{}
|
au := &fakeAudit{}
|
||||||
svc := newSvc(b, tr, nil, fakePolicy{}, au)
|
svc := newSvc(b, tr, fakePolicy{}, au)
|
||||||
|
|
||||||
rec, err := svc.Capture(context.Background(), CaptureInput{
|
rec, err := svc.Capture(context.Background(), CaptureInput{
|
||||||
Context: baseCtx(),
|
Context: baseCtx(),
|
||||||
@@ -264,7 +258,7 @@ func TestCaptureDryRunWritesNothing(t *testing.T) {
|
|||||||
b := &fakeBrain{}
|
b := &fakeBrain{}
|
||||||
tr := &fakeTracker{}
|
tr := &fakeTracker{}
|
||||||
au := &fakeAudit{}
|
au := &fakeAudit{}
|
||||||
svc := newSvc(b, tr, nil, fakePolicy{}, au)
|
svc := newSvc(b, tr, fakePolicy{}, au)
|
||||||
|
|
||||||
rec, err := svc.Capture(context.Background(), CaptureInput{
|
rec, err := svc.Capture(context.Background(), CaptureInput{
|
||||||
Context: baseCtx(),
|
Context: baseCtx(),
|
||||||
@@ -287,7 +281,7 @@ func TestCaptureStricterClassificationWins(t *testing.T) {
|
|||||||
b := &fakeBrain{}
|
b := &fakeBrain{}
|
||||||
au := &fakeAudit{}
|
au := &fakeAudit{}
|
||||||
pol := fakePolicy{tags: map[string]classification.Level{"client-seb": classification.Confidential}}
|
pol := fakePolicy{tags: map[string]classification.Level{"client-seb": classification.Confidential}}
|
||||||
svc := newSvc(b, &fakeTracker{}, nil, pol, au)
|
svc := newSvc(b, &fakeTracker{}, pol, au)
|
||||||
|
|
||||||
ctx := baseCtx()
|
ctx := baseCtx()
|
||||||
ctx.Classification = "internal"
|
ctx.Classification = "internal"
|
||||||
@@ -306,7 +300,7 @@ func TestCaptureCallerRaisingSensitivityHonoured(t *testing.T) {
|
|||||||
// Caller declares confidential; target internal → effective confidential, NOT a security event.
|
// Caller declares confidential; target internal → effective confidential, NOT a security event.
|
||||||
au := &fakeAudit{}
|
au := &fakeAudit{}
|
||||||
pol := fakePolicy{tags: map[string]classification.Level{"hyperguild": classification.Internal}}
|
pol := fakePolicy{tags: map[string]classification.Level{"hyperguild": classification.Internal}}
|
||||||
svc := newSvc(&fakeBrain{}, &fakeTracker{}, nil, pol, au)
|
svc := newSvc(&fakeBrain{}, &fakeTracker{}, pol, au)
|
||||||
|
|
||||||
ctx := baseCtx()
|
ctx := baseCtx()
|
||||||
ctx.Classification = "confidential"
|
ctx.Classification = "confidential"
|
||||||
@@ -319,26 +313,6 @@ func TestCaptureCallerRaisingSensitivityHonoured(t *testing.T) {
|
|||||||
assert.Empty(t, au.entries[0].SecurityEvents, "raising sensitivity is honoured, not flagged")
|
assert.Empty(t, au.entries[0].SecurityEvents, "raising sensitivity is honoured, not flagged")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestCaptureSummaryPathAndFidelity(t *testing.T) {
|
|
||||||
sw := &fakeSummary{}
|
|
||||||
svc := newSvc(&fakeBrain{}, &fakeTracker{}, sw, fakePolicy{}, &fakeAudit{})
|
|
||||||
|
|
||||||
ctx := baseCtx()
|
|
||||||
ctx.Fidelity = "transcript-parse"
|
|
||||||
ctx.SessionRef = "abc123def456"
|
|
||||||
rec, err := svc.Capture(context.Background(), CaptureInput{
|
|
||||||
Context: ctx,
|
|
||||||
Summary: &Summary{Title: "Session Wrap", Body: "did stuff", ReposTouched: []string{"hyperguild"}},
|
|
||||||
})
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.NotNil(t, rec.Summary)
|
|
||||||
assert.True(t, rec.Summary.OK)
|
|
||||||
require.Len(t, sw.paths, 1)
|
|
||||||
assert.True(t, strings.HasPrefix(sw.paths[0], "summaries/claude-code/2026-06/"), "path: %s", sw.paths[0])
|
|
||||||
assert.Contains(t, sw.paths[0], "session-wrap")
|
|
||||||
assert.Contains(t, sw.content[0], "fidelity: transcript-parse", "fidelity stamped in frontmatter")
|
|
||||||
}
|
|
||||||
|
|
||||||
// --- I1 sovereignty gate (#53) ---
|
// --- I1 sovereignty gate (#53) ---
|
||||||
|
|
||||||
func TestCaptureRefusesConfidentialViaUSNexus(t *testing.T) {
|
func TestCaptureRefusesConfidentialViaUSNexus(t *testing.T) {
|
||||||
@@ -346,7 +320,7 @@ func TestCaptureRefusesConfidentialViaUSNexus(t *testing.T) {
|
|||||||
tr := &fakeTracker{}
|
tr := &fakeTracker{}
|
||||||
au := &fakeAudit{}
|
au := &fakeAudit{}
|
||||||
pol := fakePolicy{tags: map[string]classification.Level{"client-seb": classification.Confidential}}
|
pol := fakePolicy{tags: map[string]classification.Level{"client-seb": classification.Confidential}}
|
||||||
svc := newSvc(b, tr, nil, pol, au)
|
svc := newSvc(b, tr, pol, au)
|
||||||
|
|
||||||
ctx := baseCtx()
|
ctx := baseCtx()
|
||||||
ctx.Classification = "confidential"
|
ctx.Classification = "confidential"
|
||||||
@@ -368,7 +342,7 @@ func TestCaptureRefusesConfidentialViaUSNexus(t *testing.T) {
|
|||||||
func TestCaptureAllowsConfidentialViaSovereign(t *testing.T) {
|
func TestCaptureAllowsConfidentialViaSovereign(t *testing.T) {
|
||||||
b := &fakeBrain{}
|
b := &fakeBrain{}
|
||||||
pol := fakePolicy{tags: map[string]classification.Level{"client-seb": classification.Confidential}}
|
pol := fakePolicy{tags: map[string]classification.Level{"client-seb": classification.Confidential}}
|
||||||
svc := newSvc(b, &fakeTracker{}, nil, pol, &fakeAudit{})
|
svc := newSvc(b, &fakeTracker{}, pol, &fakeAudit{})
|
||||||
|
|
||||||
ctx := baseCtx()
|
ctx := baseCtx()
|
||||||
ctx.Classification = "confidential"
|
ctx.Classification = "confidential"
|
||||||
@@ -387,7 +361,7 @@ func TestCaptureAssertedLabelIgnoredAndLogged(t *testing.T) {
|
|||||||
// us-nexus; confidential ⇒ refused, and the discrepancy is a security event.
|
// us-nexus; confidential ⇒ refused, and the discrepancy is a security event.
|
||||||
au := &fakeAudit{}
|
au := &fakeAudit{}
|
||||||
pol := fakePolicy{tags: map[string]classification.Level{"client-seb": classification.Confidential}}
|
pol := fakePolicy{tags: map[string]classification.Level{"client-seb": classification.Confidential}}
|
||||||
svc := newSvc(&fakeBrain{}, &fakeTracker{}, nil, pol, au)
|
svc := newSvc(&fakeBrain{}, &fakeTracker{}, pol, au)
|
||||||
|
|
||||||
ctx := baseCtx()
|
ctx := baseCtx()
|
||||||
ctx.Harness = "sovereign-soil" // asserted
|
ctx.Harness = "sovereign-soil" // asserted
|
||||||
@@ -407,7 +381,7 @@ func TestCaptureAssertedLabelIgnoredAndLogged(t *testing.T) {
|
|||||||
func TestCaptureInternalViaUSNexusAllowed(t *testing.T) {
|
func TestCaptureInternalViaUSNexusAllowed(t *testing.T) {
|
||||||
// us-nexus origin is fine for non-confidential data.
|
// us-nexus origin is fine for non-confidential data.
|
||||||
b := &fakeBrain{}
|
b := &fakeBrain{}
|
||||||
svc := newSvc(b, &fakeTracker{}, nil, fakePolicy{}, &fakeAudit{})
|
svc := newSvc(b, &fakeTracker{}, fakePolicy{}, &fakeAudit{})
|
||||||
ctx := baseCtx()
|
ctx := baseCtx()
|
||||||
ctx.Origin = ZoneUSNexus // internal classification, so gate doesn't fire
|
ctx.Origin = ZoneUSNexus // internal classification, so gate doesn't fire
|
||||||
rec, err := svc.Capture(context.Background(), CaptureInput{
|
rec, err := svc.Capture(context.Background(), CaptureInput{
|
||||||
@@ -426,7 +400,7 @@ func TestCaptureRefusesWhenAuditReserveFails(t *testing.T) {
|
|||||||
b := &fakeBrain{}
|
b := &fakeBrain{}
|
||||||
tr := &fakeTracker{}
|
tr := &fakeTracker{}
|
||||||
au := &fakeAudit{reserveErr: errors.New("central sink unreachable")}
|
au := &fakeAudit{reserveErr: errors.New("central sink unreachable")}
|
||||||
svc := newSvc(b, tr, nil, fakePolicy{}, au)
|
svc := newSvc(b, tr, fakePolicy{}, au)
|
||||||
|
|
||||||
_, err := svc.Capture(context.Background(), CaptureInput{
|
_, err := svc.Capture(context.Background(), CaptureInput{
|
||||||
Context: baseCtx(),
|
Context: baseCtx(),
|
||||||
@@ -443,7 +417,7 @@ func TestCaptureFlagsLocallyBufferedAudit(t *testing.T) {
|
|||||||
// proceeds and the receipt flags the degraded audit state.
|
// proceeds and the receipt flags the degraded audit state.
|
||||||
b := &fakeBrain{}
|
b := &fakeBrain{}
|
||||||
au := &fakeAudit{reserveMode: AuditBuffered}
|
au := &fakeAudit{reserveMode: AuditBuffered}
|
||||||
svc := newSvc(b, &fakeTracker{}, nil, fakePolicy{}, au)
|
svc := newSvc(b, &fakeTracker{}, fakePolicy{}, au)
|
||||||
|
|
||||||
rec, err := svc.Capture(context.Background(), CaptureInput{
|
rec, err := svc.Capture(context.Background(), CaptureInput{
|
||||||
Context: baseCtx(),
|
Context: baseCtx(),
|
||||||
@@ -458,7 +432,7 @@ func TestCaptureFlagsLocallyBufferedAudit(t *testing.T) {
|
|||||||
func TestCaptureDryRunSkipsAuditGate(t *testing.T) {
|
func TestCaptureDryRunSkipsAuditGate(t *testing.T) {
|
||||||
// dry_run must not even probe the audit sink (writes nothing anywhere).
|
// dry_run must not even probe the audit sink (writes nothing anywhere).
|
||||||
au := &fakeAudit{reserveErr: errors.New("would refuse")}
|
au := &fakeAudit{reserveErr: errors.New("would refuse")}
|
||||||
svc := newSvc(&fakeBrain{}, &fakeTracker{}, nil, fakePolicy{}, au)
|
svc := newSvc(&fakeBrain{}, &fakeTracker{}, fakePolicy{}, au)
|
||||||
|
|
||||||
rec, err := svc.Capture(context.Background(), CaptureInput{
|
rec, err := svc.Capture(context.Background(), CaptureInput{
|
||||||
Context: baseCtx(),
|
Context: baseCtx(),
|
||||||
@@ -469,3 +443,98 @@ func TestCaptureDryRunSkipsAuditGate(t *testing.T) {
|
|||||||
assert.True(t, rec.DryRun)
|
assert.True(t, rec.DryRun)
|
||||||
assert.Empty(t, au.entries)
|
assert.Empty(t, au.entries)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ---- hyperguild#91 part 1 -------------------------------------------------
|
||||||
|
//
|
||||||
|
// The reported symptom was "capture with a tickets array fails" (infra#265:
|
||||||
|
// 20 tickets errored, 5 errored, caller fell back to 20 individual
|
||||||
|
// issue_create calls). The array was never the failing dimension — every
|
||||||
|
// ticket targeted a client repo, and the I1 gate refuses that through a
|
||||||
|
// us-nexus origin at ANY length. The refusal did not say which target
|
||||||
|
// escalated the classification, so it read as a malfunction.
|
||||||
|
|
||||||
|
func TestCapture_TicketsArrayWorksAtLengthOne(t *testing.T) {
|
||||||
|
b, tr, au := &fakeBrain{}, &fakeTracker{}, &fakeAudit{}
|
||||||
|
svc := newSvc(b, tr, fakePolicy{}, au)
|
||||||
|
|
||||||
|
rec, err := svc.Capture(context.Background(), CaptureInput{
|
||||||
|
Context: baseCtx(),
|
||||||
|
Tickets: []Ticket{{Repo: "hyperguild", Action: "create", Title: "one", Body: "b"}},
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, rec.Tickets, 1)
|
||||||
|
assert.True(t, rec.Tickets[0].OK)
|
||||||
|
assert.Len(t, tr.created, 1)
|
||||||
|
assert.Empty(t, rec.Errors)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCapture_TicketsArrayWorksAtLengthMany(t *testing.T) {
|
||||||
|
b, tr, au := &fakeBrain{}, &fakeTracker{}, &fakeAudit{}
|
||||||
|
svc := newSvc(b, tr, fakePolicy{}, au)
|
||||||
|
|
||||||
|
in := CaptureInput{Context: baseCtx()}
|
||||||
|
for i := range 20 {
|
||||||
|
in.Tickets = append(in.Tickets, Ticket{
|
||||||
|
Repo: "hyperguild", Action: "create", Title: fmt.Sprintf("t%02d", i), Body: "b",
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
rec, err := svc.Capture(context.Background(), in)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, rec.Tickets, 20, "one receipt entry per ticket")
|
||||||
|
for i, r := range rec.Tickets {
|
||||||
|
assert.True(t, r.OK, "ticket %d failed: %+v", i, rec.Errors)
|
||||||
|
}
|
||||||
|
assert.Len(t, tr.created, 20)
|
||||||
|
assert.Empty(t, rec.Errors)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCapture_OneFailingTicketDoesNotSinkTheRest(t *testing.T) {
|
||||||
|
// infra#265 asked explicitly: "if ticket 3 of 5 fails, what happened to
|
||||||
|
// 1 and 2?" They land, and the receipt says so per item.
|
||||||
|
b, au := &fakeBrain{}, &fakeAudit{}
|
||||||
|
tr := &fakeTracker{failOnTitle: "boom"}
|
||||||
|
svc := newSvc(b, tr, fakePolicy{}, au)
|
||||||
|
|
||||||
|
rec, err := svc.Capture(context.Background(), CaptureInput{
|
||||||
|
Context: baseCtx(),
|
||||||
|
Tickets: []Ticket{
|
||||||
|
{Repo: "hyperguild", Action: "create", Title: "ok-1", Body: "b"},
|
||||||
|
{Repo: "hyperguild", Action: "create", Title: "boom", Body: "b"},
|
||||||
|
{Repo: "hyperguild", Action: "create", Title: "ok-3", Body: "b"},
|
||||||
|
},
|
||||||
|
})
|
||||||
|
require.NoError(t, err, "a per-item failure is a partial receipt, not a call error")
|
||||||
|
require.Len(t, rec.Tickets, 3)
|
||||||
|
assert.True(t, rec.Tickets[0].OK)
|
||||||
|
assert.False(t, rec.Tickets[1].OK)
|
||||||
|
assert.True(t, rec.Tickets[2].OK)
|
||||||
|
require.Len(t, rec.Errors, 1)
|
||||||
|
assert.Contains(t, rec.Errors[0].Item, "ticket[1]")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCapture_I1RefusalNamesWhatEscalatedTheClassification(t *testing.T) {
|
||||||
|
// The actual defect behind infra#265. Without the offending target in
|
||||||
|
// the message, a caller with 20 tickets cannot tell which repo tripped
|
||||||
|
// the gate — and reasonably concludes the array itself is broken.
|
||||||
|
b, tr, au := &fakeBrain{}, &fakeTracker{}, &fakeAudit{}
|
||||||
|
pol := fakePolicy{tags: map[string]classification.Level{"trucker-ecommerce": classification.Confidential}}
|
||||||
|
svc := newSvc(b, tr, pol, au)
|
||||||
|
|
||||||
|
ctx := baseCtx()
|
||||||
|
ctx.Origin = ZoneUSNexus
|
||||||
|
_, err := svc.Capture(context.Background(), CaptureInput{
|
||||||
|
Context: ctx,
|
||||||
|
Tickets: []Ticket{
|
||||||
|
{Repo: "hyperguild", Action: "create", Title: "fine", Body: "b"},
|
||||||
|
{Repo: "trucker-ecommerce", Action: "create", Title: "client work", Body: "b"},
|
||||||
|
},
|
||||||
|
})
|
||||||
|
|
||||||
|
require.ErrorIs(t, err, ErrSovereigntyRefused)
|
||||||
|
assert.Contains(t, err.Error(), "trucker-ecommerce",
|
||||||
|
"the refusal must name the target that escalated the classification")
|
||||||
|
assert.NotContains(t, err.Error(), "hyperguild",
|
||||||
|
"only the offending target, or the message is noise at 20 tickets")
|
||||||
|
assert.Empty(t, tr.created, "refused before any write")
|
||||||
|
}
|
||||||
|
|||||||
@@ -55,7 +55,6 @@ type request struct {
|
|||||||
Context contextBody `json:"context"`
|
Context contextBody `json:"context"`
|
||||||
Insights []insightBody `json:"insights"`
|
Insights []insightBody `json:"insights"`
|
||||||
Tickets []ticketBody `json:"tickets"`
|
Tickets []ticketBody `json:"tickets"`
|
||||||
Summary *summaryBody `json:"summary,omitempty"`
|
|
||||||
DryRun bool `json:"dry_run"`
|
DryRun bool `json:"dry_run"`
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -82,12 +81,6 @@ type ticketBody struct {
|
|||||||
Body string `json:"body,omitempty"`
|
Body string `json:"body,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type summaryBody struct {
|
|
||||||
Title string `json:"title"`
|
|
||||||
Body string `json:"body"`
|
|
||||||
ReposTouched []string `json:"repos_touched,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// ServeHTTP authenticates, derives origin, runs the use-case, and maps the
|
// ServeHTTP authenticates, derives origin, runs the use-case, and maps the
|
||||||
// result to an HTTP status.
|
// result to an HTTP status.
|
||||||
func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||||
@@ -183,11 +176,6 @@ func (b request) toInput() capture.CaptureInput {
|
|||||||
Repo: t.Repo, Action: t.Action, Number: t.Number, Title: t.Title, Body: t.Body,
|
Repo: t.Repo, Action: t.Action, Number: t.Number, Title: t.Title, Body: t.Body,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
if b.Summary != nil {
|
|
||||||
in.Summary = &capture.Summary{
|
|
||||||
Title: b.Summary.Title, Body: b.Summary.Body, ReposTouched: b.Summary.ReposTouched,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return in
|
return in
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -204,9 +192,6 @@ func statusFor(rec capture.CaptureReceipt) int {
|
|||||||
for _, t := range rec.Tickets {
|
for _, t := range rec.Tickets {
|
||||||
count(&ok, &fail, t.OK)
|
count(&ok, &fail, t.OK)
|
||||||
}
|
}
|
||||||
if rec.Summary != nil {
|
|
||||||
count(&ok, &fail, rec.Summary.OK)
|
|
||||||
}
|
|
||||||
switch {
|
switch {
|
||||||
case fail == 0:
|
case fail == 0:
|
||||||
return http.StatusOK
|
return http.StatusOK
|
||||||
|
|||||||
@@ -49,7 +49,7 @@ func newHandler(t *testing.T, v capturehttp.Validator, tr capture.IssueTracker,
|
|||||||
t.Helper()
|
t.Helper()
|
||||||
cfg, err := classification.Load(t.TempDir())
|
cfg, err := classification.Load(t.TempDir())
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
svc := capture.NewService(brainstore.New(t.TempDir()), tr, nil, cfg, audit.NewSlogSink(nil))
|
svc := capture.NewService(brainstore.New(t.TempDir()), tr, cfg, audit.NewSlogSink(nil))
|
||||||
return capturehttp.New(svc, v, staticTok, "local-cli", capturehttp.NewOriginResolver(sovereign))
|
return capturehttp.New(svc, v, staticTok, "local-cli", capturehttp.NewOriginResolver(sovereign))
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -190,7 +190,7 @@ func (refusingAudit) Record(context.Context, capture.AuditEntry, capture.AuditOu
|
|||||||
func TestAuditUnavailableIs503(t *testing.T) {
|
func TestAuditUnavailableIs503(t *testing.T) {
|
||||||
cfg, err := classification.Load(t.TempDir())
|
cfg, err := classification.Load(t.TempDir())
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
svc := capture.NewService(brainstore.New(t.TempDir()), fakeTracker{}, nil, cfg, refusingAudit{})
|
svc := capture.NewService(brainstore.New(t.TempDir()), fakeTracker{}, cfg, refusingAudit{})
|
||||||
h := capturehttp.New(svc, nil, staticTok, "local-cli", capturehttp.NewOriginResolver(nil))
|
h := capturehttp.New(svc, nil, staticTok, "local-cli", capturehttp.NewOriginResolver(nil))
|
||||||
|
|
||||||
rr := do(t, h, "Bearer "+staticTok, internalReq())
|
rr := do(t, h, "Bearer "+staticTok, internalReq())
|
||||||
|
|||||||
@@ -0,0 +1,98 @@
|
|||||||
|
// Package claim validates the claim-property frontmatter that says when and
|
||||||
|
// where a note's claim holds, and who decided that (brain#21).
|
||||||
|
//
|
||||||
|
// Two families were designed together and land separately. This is the
|
||||||
|
// applicability envelope:
|
||||||
|
//
|
||||||
|
// half_life: volatile | seasonal | durable — how fast it decays
|
||||||
|
// applies: local | homelab | universal — how widely it holds
|
||||||
|
// claim_provenance: asserted | inferred — who set the above
|
||||||
|
//
|
||||||
|
// The strength family (confidence, contested) follows once the ingest-time
|
||||||
|
// contradiction detector exists to populate it.
|
||||||
|
//
|
||||||
|
// Why the envelope is validated rather than merely accepted: a value outside
|
||||||
|
// the enum is worse than an absent one. A lint skips what it cannot parse, so
|
||||||
|
// the note reads as classified while nothing acts on it — the field becomes
|
||||||
|
// decoration that suppresses the very review it was added to trigger.
|
||||||
|
//
|
||||||
|
// Why `applies` and not `scope`: `scope:` has been in use since brain#5 for
|
||||||
|
// glossary terms (`homelab-core`, `project:<repo>`) — a disjoint value space
|
||||||
|
// on the same key. The shipped schema keeps the name; this one takes another.
|
||||||
|
package claim
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Field names, exported so writers and lints agree on the spelling.
|
||||||
|
const (
|
||||||
|
FieldHalfLife = "half_life"
|
||||||
|
FieldApplies = "applies"
|
||||||
|
FieldProvenance = "claim_provenance"
|
||||||
|
)
|
||||||
|
|
||||||
|
var (
|
||||||
|
halfLives = []string{"volatile", "seasonal", "durable"}
|
||||||
|
appliesTo = []string{"local", "homelab", "universal"}
|
||||||
|
provenances = []string{"asserted", "inferred"}
|
||||||
|
)
|
||||||
|
|
||||||
|
// ValidateEnvelope checks the envelope fields of one note's frontmatter.
|
||||||
|
//
|
||||||
|
// Absent fields are valid: the envelope is opt-in for new writes, and a
|
||||||
|
// corpus-wide backfill is deliberately out of scope (brain#21) — inferring
|
||||||
|
// 1113 notes' properties in bulk has the same failure mode as the extraction
|
||||||
|
// pass that produced 28 wrong wikilinks out of 67.
|
||||||
|
func ValidateEnvelope(fm map[string]string) error {
|
||||||
|
halfLife, hasHalfLife := present(fm, FieldHalfLife)
|
||||||
|
applies, hasApplies := present(fm, FieldApplies)
|
||||||
|
prov, hasProv := present(fm, FieldProvenance)
|
||||||
|
|
||||||
|
if hasHalfLife {
|
||||||
|
if err := oneOf(FieldHalfLife, halfLife, halfLives); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if hasApplies {
|
||||||
|
if err := oneOf(FieldApplies, applies, appliesTo); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if hasProv {
|
||||||
|
if err := oneOf(FieldProvenance, prov, provenances); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The field and its provenance travel together or not at all. A value
|
||||||
|
// whose author is unknown cannot be trusted differently from one Mathias
|
||||||
|
// asserted, which is how an inferred guess hardens into accepted fact.
|
||||||
|
switch {
|
||||||
|
case (hasHalfLife || hasApplies) && !hasProv:
|
||||||
|
return fmt.Errorf("%s is required when %s or %s is set",
|
||||||
|
FieldProvenance, FieldHalfLife, FieldApplies)
|
||||||
|
case hasProv && !hasHalfLife && !hasApplies:
|
||||||
|
return fmt.Errorf("%s is set but neither %s nor %s is",
|
||||||
|
FieldProvenance, FieldHalfLife, FieldApplies)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// present reports a trimmed field value and whether it was set to anything
|
||||||
|
// non-empty. A key written with an empty value is treated as absent rather
|
||||||
|
// than as an error: that is a caller emitting a placeholder, not asserting.
|
||||||
|
func present(fm map[string]string, key string) (string, bool) {
|
||||||
|
v := strings.TrimSpace(fm[key])
|
||||||
|
return v, v != ""
|
||||||
|
}
|
||||||
|
|
||||||
|
func oneOf(field, value string, allowed []string) error {
|
||||||
|
for _, a := range allowed {
|
||||||
|
if a == value {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return fmt.Errorf("%s: %q is not one of %s", field, value, strings.Join(allowed, ", "))
|
||||||
|
}
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
// Package claim_test covers the claim-property envelope (brain#21).
|
||||||
|
package claim_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/mathiasbq/hyperguild/ingestion/internal/claim"
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestValidateEnvelope(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
fm map[string]string
|
||||||
|
wantErr string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "absent is fine — the fields are opt-in on new writes",
|
||||||
|
fm: map[string]string{"wing": "homelab", "hall": "facts"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "complete envelope with provenance",
|
||||||
|
fm: map[string]string{
|
||||||
|
"half_life": "durable", "applies": "universal", "claim_provenance": "asserted",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "inferred provenance is equally valid — it just must be stated",
|
||||||
|
fm: map[string]string{
|
||||||
|
"half_life": "volatile", "applies": "local", "claim_provenance": "inferred",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// A value nobody can act on is worse than no value: a lint
|
||||||
|
// silently skips it, so the note looks classified and isn't.
|
||||||
|
name: "unknown half_life value",
|
||||||
|
fm: map[string]string{"half_life": "forever", "claim_provenance": "asserted"},
|
||||||
|
wantErr: `half_life: "forever" is not one of volatile, seasonal, durable`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "unknown applies value",
|
||||||
|
fm: map[string]string{"applies": "galaxy", "claim_provenance": "asserted"},
|
||||||
|
wantErr: `applies: "galaxy" is not one of local, homelab, universal`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// brain#21: a value a model inferred must never be
|
||||||
|
// indistinguishable from one Mathias asserted. Writing the field
|
||||||
|
// without saying who set it is exactly that laundering.
|
||||||
|
name: "envelope field without provenance",
|
||||||
|
fm: map[string]string{"half_life": "durable"},
|
||||||
|
wantErr: "claim_provenance is required when half_life or applies is set",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "applies without provenance",
|
||||||
|
fm: map[string]string{"applies": "homelab"},
|
||||||
|
wantErr: "claim_provenance is required when half_life or applies is set",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "unknown provenance value",
|
||||||
|
fm: map[string]string{"half_life": "durable", "claim_provenance": "probably"},
|
||||||
|
wantErr: `claim_provenance: "probably" is not one of asserted, inferred`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// Provenance alone says who set fields that aren't there.
|
||||||
|
name: "provenance without any envelope field",
|
||||||
|
fm: map[string]string{"claim_provenance": "asserted"},
|
||||||
|
wantErr: "claim_provenance is set but neither half_life nor applies is",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// The glossary wing has used scope: homelab-core since #5. The
|
||||||
|
// envelope deliberately does NOT claim that key (brain#21).
|
||||||
|
name: "glossary scope is untouched",
|
||||||
|
fm: map[string]string{"scope": "homelab-core", "term": "koala"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
err := claim.ValidateEnvelope(tt.fm)
|
||||||
|
if tt.wantErr == "" {
|
||||||
|
assert.NoError(t, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
require.Error(t, err)
|
||||||
|
assert.Contains(t, err.Error(), tt.wantErr)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,148 @@
|
|||||||
|
// ingestion/internal/extract/docmark.go
|
||||||
|
package extract
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"encoding/base64"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// docmarkRequest is a JSON-RPC 2.0 tools/call request for docmark's
|
||||||
|
// convert_to_markdown tool.
|
||||||
|
type docmarkRequest struct {
|
||||||
|
JSONRPC string `json:"jsonrpc"`
|
||||||
|
ID int `json:"id"`
|
||||||
|
Method string `json:"method"`
|
||||||
|
Params struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
Arguments struct {
|
||||||
|
ContentBase64 string `json:"content_base64"`
|
||||||
|
Filename string `json:"filename"`
|
||||||
|
} `json:"arguments"`
|
||||||
|
} `json:"params"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type docmarkResponse struct {
|
||||||
|
Result *struct {
|
||||||
|
Content []struct {
|
||||||
|
Type string `json:"type"`
|
||||||
|
Text string `json:"text"`
|
||||||
|
} `json:"content"`
|
||||||
|
IsError bool `json:"isError"`
|
||||||
|
} `json:"result"`
|
||||||
|
Error *struct {
|
||||||
|
Message string `json:"message"`
|
||||||
|
} `json:"error"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// extractViaDocmark converts path (PDF/DOCX/XLSX/PPTX/image) to Markdown by
|
||||||
|
// calling the docmark MCP server with a single self-contained tools/call
|
||||||
|
// request (docmark runs stateless_http -- no initialize handshake or session
|
||||||
|
// ID needed). DOCMARK_URL must be set (e.g.
|
||||||
|
// http://docmark.docmark.svc.cluster.local:3001/mcp); DOCMARK_BEARER_TOKEN
|
||||||
|
// is docmark's static bearer (network is docmark's primary auth boundary,
|
||||||
|
// this is defense-in-depth — ADR-0013).
|
||||||
|
func extractViaDocmark(path string) (string, error) {
|
||||||
|
url := os.Getenv("DOCMARK_URL")
|
||||||
|
if url == "" {
|
||||||
|
return "", fmt.Errorf("extractViaDocmark: DOCMARK_URL is not set")
|
||||||
|
}
|
||||||
|
|
||||||
|
raw, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("read %s: %w", path, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var reqBody docmarkRequest
|
||||||
|
reqBody.JSONRPC = "2.0"
|
||||||
|
reqBody.ID = 1
|
||||||
|
reqBody.Method = "tools/call"
|
||||||
|
reqBody.Params.Name = "convert_to_markdown"
|
||||||
|
reqBody.Params.Arguments.ContentBase64 = base64.StdEncoding.EncodeToString(raw)
|
||||||
|
reqBody.Params.Arguments.Filename = fileBase(path)
|
||||||
|
|
||||||
|
payload, err := json.Marshal(reqBody)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("marshal docmark request: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
httpReq, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(payload))
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("build docmark request: %w", err)
|
||||||
|
}
|
||||||
|
httpReq.Header.Set("Content-Type", "application/json")
|
||||||
|
httpReq.Header.Set("Accept", "application/json, text/event-stream")
|
||||||
|
if tok := os.Getenv("DOCMARK_BEARER_TOKEN"); tok != "" {
|
||||||
|
httpReq.Header.Set("Authorization", "Bearer "+tok)
|
||||||
|
}
|
||||||
|
|
||||||
|
client := &http.Client{Timeout: 60 * time.Second}
|
||||||
|
resp, err := client.Do(httpReq)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("call docmark: %w", err)
|
||||||
|
}
|
||||||
|
defer func() { _ = resp.Body.Close() }()
|
||||||
|
|
||||||
|
body, err := io.ReadAll(resp.Body)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("read docmark response: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
return "", fmt.Errorf("docmark: HTTP %d: %s", resp.StatusCode, string(body))
|
||||||
|
}
|
||||||
|
|
||||||
|
jsonBody := body
|
||||||
|
if data := sseDataPayload(body); data != nil {
|
||||||
|
jsonBody = data
|
||||||
|
}
|
||||||
|
|
||||||
|
var out docmarkResponse
|
||||||
|
if err := json.Unmarshal(jsonBody, &out); err != nil {
|
||||||
|
return "", fmt.Errorf("decode docmark response: %w", err)
|
||||||
|
}
|
||||||
|
if out.Error != nil {
|
||||||
|
return "", fmt.Errorf("docmark: %s", out.Error.Message)
|
||||||
|
}
|
||||||
|
if out.Result == nil || len(out.Result.Content) == 0 {
|
||||||
|
return "", fmt.Errorf("docmark: empty response")
|
||||||
|
}
|
||||||
|
text := out.Result.Content[0].Text
|
||||||
|
if out.Result.IsError {
|
||||||
|
return "", fmt.Errorf("docmark: %s", text)
|
||||||
|
}
|
||||||
|
return text, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// sseDataPayload extracts the JSON payload from an SSE-framed response body
|
||||||
|
// ("event: message\r\ndata: {...}\r\n\r\n"). docmark's Streamable-HTTP
|
||||||
|
// transport frames every response this way (Content-Type: text/event-stream)
|
||||||
|
// regardless of stateless_http — that flag removes the session/initialize
|
||||||
|
// requirement, not the SSE wire framing. Returns nil if body isn't SSE-framed
|
||||||
|
// (e.g. a plain-JSON response, kept as a fallback for forward-compatibility).
|
||||||
|
func sseDataPayload(body []byte) []byte {
|
||||||
|
const prefix = "data: "
|
||||||
|
for _, line := range bytes.Split(body, []byte("\n")) {
|
||||||
|
line = bytes.TrimRight(line, "\r")
|
||||||
|
if bytes.HasPrefix(line, []byte(prefix)) {
|
||||||
|
return bytes.TrimPrefix(line, []byte(prefix))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// fileBase returns the final path segment (like filepath.Base, kept local to
|
||||||
|
// avoid importing path/filepath just for this one call).
|
||||||
|
func fileBase(path string) string {
|
||||||
|
for i := len(path) - 1; i >= 0; i-- {
|
||||||
|
if path[i] == '/' || path[i] == '\\' {
|
||||||
|
return path[i+1:]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return path
|
||||||
|
}
|
||||||
@@ -0,0 +1,189 @@
|
|||||||
|
// ingestion/internal/extract/docmark_test.go
|
||||||
|
package extract
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
// mcpToolResult mirrors the shape of docmark's JSON-RPC tools/call response.
|
||||||
|
type mcpToolResult struct {
|
||||||
|
JSONRPC string `json:"jsonrpc"`
|
||||||
|
ID int `json:"id"`
|
||||||
|
Result *struct {
|
||||||
|
Content []struct {
|
||||||
|
Type string `json:"type"`
|
||||||
|
Text string `json:"text"`
|
||||||
|
} `json:"content"`
|
||||||
|
IsError bool `json:"isError"`
|
||||||
|
} `json:"result,omitempty"`
|
||||||
|
Error *struct {
|
||||||
|
Message string `json:"message"`
|
||||||
|
} `json:"error,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// writeMCPResponse mirrors docmark's REAL response framing (empirically
|
||||||
|
// confirmed against the live server): Content-Type: text/event-stream,
|
||||||
|
// body is SSE-framed ("event: message\r\ndata: {...}\r\n\r\n"), not bare
|
||||||
|
// JSON -- inherent to MCP Streamable-HTTP, independent of stateless_http.
|
||||||
|
func writeMCPResponse(w http.ResponseWriter, body mcpToolResult) {
|
||||||
|
payload, _ := json.Marshal(body)
|
||||||
|
w.Header().Set("Content-Type", "text/event-stream")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
_, _ = w.Write([]byte("event: message\r\ndata: "))
|
||||||
|
_, _ = w.Write(payload)
|
||||||
|
_, _ = w.Write([]byte("\r\n\r\n"))
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractViaDocmark_Success(t *testing.T) {
|
||||||
|
var gotAuth, gotAccept string
|
||||||
|
var gotBody map[string]any
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
gotAuth = r.Header.Get("Authorization")
|
||||||
|
gotAccept = r.Header.Get("Accept")
|
||||||
|
b, _ := io.ReadAll(r.Body)
|
||||||
|
_ = json.Unmarshal(b, &gotBody)
|
||||||
|
writeMCPResponse(w, mcpToolResult{
|
||||||
|
JSONRPC: "2.0", ID: 1,
|
||||||
|
Result: &struct {
|
||||||
|
Content []struct {
|
||||||
|
Type string `json:"type"`
|
||||||
|
Text string `json:"text"`
|
||||||
|
} `json:"content"`
|
||||||
|
IsError bool `json:"isError"`
|
||||||
|
}{
|
||||||
|
Content: []struct {
|
||||||
|
Type string `json:"type"`
|
||||||
|
Text string `json:"text"`
|
||||||
|
}{{Type: "text", Text: "# Converted\n\nhello"}},
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}))
|
||||||
|
defer srv.Close()
|
||||||
|
|
||||||
|
t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
|
||||||
|
t.Setenv("DOCMARK_BEARER_TOKEN", "test-token-123")
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
path := filepath.Join(dir, "doc.docx")
|
||||||
|
require.NoError(t, os.WriteFile(path, []byte("fake docx bytes"), 0o644))
|
||||||
|
|
||||||
|
got, err := extractViaDocmark(path)
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, "# Converted\n\nhello", got)
|
||||||
|
assert.Equal(t, "Bearer test-token-123", gotAuth)
|
||||||
|
assert.Contains(t, gotAccept, "application/json")
|
||||||
|
params, _ := gotBody["params"].(map[string]any)
|
||||||
|
require.NotNil(t, params)
|
||||||
|
assert.Equal(t, "convert_to_markdown", params["name"])
|
||||||
|
args, _ := params["arguments"].(map[string]any)
|
||||||
|
require.NotNil(t, args)
|
||||||
|
assert.Equal(t, "doc.docx", args["filename"])
|
||||||
|
assert.NotEmpty(t, args["content_base64"])
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractViaDocmark_NotConfigured(t *testing.T) {
|
||||||
|
t.Setenv("DOCMARK_URL", "")
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
path := filepath.Join(dir, "doc.docx")
|
||||||
|
require.NoError(t, os.WriteFile(path, []byte("x"), 0o644))
|
||||||
|
|
||||||
|
_, err := extractViaDocmark(path)
|
||||||
|
require.Error(t, err)
|
||||||
|
assert.Contains(t, err.Error(), "DOCMARK_URL")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractViaDocmark_ToolErrorSurfacesMessage(t *testing.T) {
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
writeMCPResponse(w, mcpToolResult{
|
||||||
|
JSONRPC: "2.0", ID: 1,
|
||||||
|
Result: &struct {
|
||||||
|
Content []struct {
|
||||||
|
Type string `json:"type"`
|
||||||
|
Text string `json:"text"`
|
||||||
|
} `json:"content"`
|
||||||
|
IsError bool `json:"isError"`
|
||||||
|
}{
|
||||||
|
Content: []struct {
|
||||||
|
Type string `json:"type"`
|
||||||
|
Text string `json:"text"`
|
||||||
|
}{{Type: "text", Text: "unsupported format for 'doc.docx'"}},
|
||||||
|
IsError: true,
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}))
|
||||||
|
defer srv.Close()
|
||||||
|
|
||||||
|
t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
|
||||||
|
t.Setenv("DOCMARK_BEARER_TOKEN", "tok")
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
path := filepath.Join(dir, "doc.docx")
|
||||||
|
require.NoError(t, os.WriteFile(path, []byte("x"), 0o644))
|
||||||
|
|
||||||
|
_, err := extractViaDocmark(path)
|
||||||
|
require.Error(t, err)
|
||||||
|
assert.Contains(t, err.Error(), "unsupported format")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractViaDocmark_HTTPErrorSurfaces(t *testing.T) {
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.WriteHeader(http.StatusUnauthorized)
|
||||||
|
_, _ = w.Write([]byte("unauthorized"))
|
||||||
|
}))
|
||||||
|
defer srv.Close()
|
||||||
|
|
||||||
|
t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
|
||||||
|
t.Setenv("DOCMARK_BEARER_TOKEN", "wrong")
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
path := filepath.Join(dir, "doc.docx")
|
||||||
|
require.NoError(t, os.WriteFile(path, []byte("x"), 0o644))
|
||||||
|
|
||||||
|
_, err := extractViaDocmark(path)
|
||||||
|
require.Error(t, err)
|
||||||
|
assert.Contains(t, err.Error(), "401")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestText_RoutesDocxXlsxPptxImagesToDocmark(t *testing.T) {
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
writeMCPResponse(w, mcpToolResult{
|
||||||
|
JSONRPC: "2.0", ID: 1,
|
||||||
|
Result: &struct {
|
||||||
|
Content []struct {
|
||||||
|
Type string `json:"type"`
|
||||||
|
Text string `json:"text"`
|
||||||
|
} `json:"content"`
|
||||||
|
IsError bool `json:"isError"`
|
||||||
|
}{
|
||||||
|
Content: []struct {
|
||||||
|
Type string `json:"type"`
|
||||||
|
Text string `json:"text"`
|
||||||
|
}{{Type: "text", Text: "converted"}},
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}))
|
||||||
|
defer srv.Close()
|
||||||
|
t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
|
||||||
|
t.Setenv("DOCMARK_BEARER_TOKEN", "tok")
|
||||||
|
|
||||||
|
for _, ext := range []string{".docx", ".xlsx", ".pptx", ".png", ".jpg", ".jpeg"} {
|
||||||
|
t.Run(ext, func(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
path := filepath.Join(dir, "f"+ext)
|
||||||
|
require.NoError(t, os.WriteFile(path, []byte("x"), 0o644))
|
||||||
|
got, err := Text(path)
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, "converted", got)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -8,7 +8,9 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// Text reads the file at path and returns its plain-text content.
|
// Text reads the file at path and returns its plain-text content.
|
||||||
// Supported extensions: .md, .txt (passthrough), .pdf (via pdftotext).
|
// Supported extensions: .md, .txt (passthrough), .pdf (via pdftotext),
|
||||||
|
// .docx/.xlsx/.pptx/.png/.jpg/.jpeg (via docmark, ADR-0013 -- requires
|
||||||
|
// DOCMARK_URL to be set; see docmark.go).
|
||||||
func Text(path string) (string, error) {
|
func Text(path string) (string, error) {
|
||||||
ext := strings.ToLower(fileExt(path))
|
ext := strings.ToLower(fileExt(path))
|
||||||
switch ext {
|
switch ext {
|
||||||
@@ -20,6 +22,8 @@ func Text(path string) (string, error) {
|
|||||||
return string(b), nil
|
return string(b), nil
|
||||||
case ".pdf":
|
case ".pdf":
|
||||||
return extractPDF(path)
|
return extractPDF(path)
|
||||||
|
case ".docx", ".xlsx", ".pptx", ".png", ".jpg", ".jpeg":
|
||||||
|
return extractViaDocmark(path)
|
||||||
default:
|
default:
|
||||||
return "", fmt.Errorf("unsupported file extension: %s", ext)
|
return "", fmt.Errorf("unsupported file extension: %s", ext)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,143 @@
|
|||||||
|
// Package frontmatter lints and repairs the YAML frontmatter block of a
|
||||||
|
// markdown note.
|
||||||
|
//
|
||||||
|
// Notes written directly — hand-authored, or promoted from brain/raw/ — do not
|
||||||
|
// pass through pipeline.BuildPages, so nothing quotes their scalars. A title
|
||||||
|
// containing an unescaped ": " then produces a block that yaml.Unmarshal
|
||||||
|
// rejects with "mapping values are not allowed here", and every consumer that
|
||||||
|
// reads frontmatter silently skips the note.
|
||||||
|
//
|
||||||
|
// Fix is deliberately conservative: it only adds quoting, never reorders or
|
||||||
|
// rewrites, and it refuses to return a document that still does not parse.
|
||||||
|
package frontmatter
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"fmt"
|
||||||
|
"regexp"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gopkg.in/yaml.v3"
|
||||||
|
)
|
||||||
|
|
||||||
|
const fence = "---"
|
||||||
|
|
||||||
|
// Lint reports whether doc's frontmatter block parses as YAML.
|
||||||
|
//
|
||||||
|
// A document with no frontmatter block, or with an unterminated one, is not an
|
||||||
|
// error: there is nothing claiming to be YAML.
|
||||||
|
func Lint(doc []byte) error {
|
||||||
|
b, ok := block(doc)
|
||||||
|
if !ok {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
var v any
|
||||||
|
if err := yaml.Unmarshal(b, &v); err != nil {
|
||||||
|
return fmt.Errorf("frontmatter: %w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fix quotes unquoted scalars whose value breaks YAML parsing and returns the
|
||||||
|
// repaired document.
|
||||||
|
//
|
||||||
|
// It returns (doc, false, nil) when the document already parses, and
|
||||||
|
// (doc, false, err) when quoting cannot rescue it — an unfixable document is
|
||||||
|
// always returned byte-identical, never half-repaired.
|
||||||
|
func Fix(doc []byte) ([]byte, bool, error) {
|
||||||
|
if err := Lint(doc); err == nil {
|
||||||
|
return doc, false, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
b, ok := block(doc)
|
||||||
|
if !ok {
|
||||||
|
// Lint only fails when there is a block, so this is unreachable in
|
||||||
|
// practice; return the original rather than guess.
|
||||||
|
return doc, false, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
fixed := quoteScalars(b)
|
||||||
|
if bytes.Equal(fixed, b) {
|
||||||
|
return doc, false, fmt.Errorf("frontmatter: not repairable by quoting: %w", Lint(doc))
|
||||||
|
}
|
||||||
|
|
||||||
|
var v any
|
||||||
|
if err := yaml.Unmarshal(fixed, &v); err != nil {
|
||||||
|
return doc, false, fmt.Errorf("frontmatter: quoting did not repair the block: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, rest, _ := split(doc)
|
||||||
|
out := []byte(fence + "\n")
|
||||||
|
out = append(out, fixed...)
|
||||||
|
out = append(out, []byte(fence)...)
|
||||||
|
out = append(out, rest...)
|
||||||
|
return out, true, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// mappingLine matches "key: value" at any indentation, capturing the prefix so
|
||||||
|
// it can be re-emitted untouched.
|
||||||
|
//
|
||||||
|
// Sequence items ("- value") are deliberately not handled: "- foo: bar" is
|
||||||
|
// valid YAML, so a document containing one does not reach Fix in the first
|
||||||
|
// place. See the Lint table for why that gap is left open.
|
||||||
|
var mappingLine = regexp.MustCompile(`^(\s*[A-Za-z_][\w.-]*:[ \t]+)(\S.*)$`)
|
||||||
|
|
||||||
|
// quoteScalars single-quotes every unquoted scalar value in the block that
|
||||||
|
// contains a colon. Values that are already quoted, or that carry YAML syntax
|
||||||
|
// this package has no business rewriting, are left alone.
|
||||||
|
func quoteScalars(block []byte) []byte {
|
||||||
|
lines := strings.Split(string(block), "\n")
|
||||||
|
for i, line := range lines {
|
||||||
|
m := mappingLine.FindStringSubmatch(line)
|
||||||
|
if m == nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
prefix, value := m[1], m[2]
|
||||||
|
if !needsQuoting(value) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
lines[i] = prefix + "'" + strings.ReplaceAll(value, "'", "''") + "'"
|
||||||
|
}
|
||||||
|
return []byte(strings.Join(lines, "\n"))
|
||||||
|
}
|
||||||
|
|
||||||
|
// needsQuoting reports whether a scalar value has to be quoted to load as the
|
||||||
|
// string it plainly is.
|
||||||
|
//
|
||||||
|
// The test is YAML's own parser rather than a pattern: ask whether the value
|
||||||
|
// stands alone as the right-hand side of a mapping. Heuristics get this wrong
|
||||||
|
// in both directions — `title: "Foo" — bar` opens with a quote but is not
|
||||||
|
// quoted, and a value may contain a colon inside quotes and be perfectly fine.
|
||||||
|
//
|
||||||
|
// Anything that already parses is left alone, whatever it parses into — a flow
|
||||||
|
// collection, a number, a bool. It is not broken, and quoting would change its
|
||||||
|
// type.
|
||||||
|
func needsQuoting(value string) bool {
|
||||||
|
if strings.HasPrefix(value, "|") || strings.HasPrefix(value, ">") {
|
||||||
|
return false // block scalar; its content is not parsed as YAML
|
||||||
|
}
|
||||||
|
var probe map[string]any
|
||||||
|
return yaml.Unmarshal([]byte("v: "+value+"\n"), &probe) != nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// block returns the raw YAML between the opening and closing fences.
|
||||||
|
func block(doc []byte) ([]byte, bool) {
|
||||||
|
b, _, ok := split(doc)
|
||||||
|
return b, ok
|
||||||
|
}
|
||||||
|
|
||||||
|
// split returns the frontmatter block and everything from the closing fence
|
||||||
|
// onward (the closing fence itself excluded, its trailing newline included).
|
||||||
|
func split(doc []byte) (block, rest []byte, ok bool) {
|
||||||
|
s := string(doc)
|
||||||
|
if !strings.HasPrefix(s, fence+"\n") {
|
||||||
|
return nil, nil, false
|
||||||
|
}
|
||||||
|
body := s[len(fence)+1:]
|
||||||
|
|
||||||
|
idx := strings.Index(body, "\n"+fence)
|
||||||
|
if idx < 0 {
|
||||||
|
return nil, nil, false
|
||||||
|
}
|
||||||
|
return []byte(body[:idx+1]), []byte(body[idx+1+len(fence):]), true
|
||||||
|
}
|
||||||
@@ -0,0 +1,188 @@
|
|||||||
|
// ingestion/internal/frontmatter/frontmatter_test.go
|
||||||
|
package frontmatter
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"gopkg.in/yaml.v3"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestLint(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
doc string
|
||||||
|
wantErr bool
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "no frontmatter block is not an error",
|
||||||
|
doc: "# Just a heading\n\nSome body text.\n",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "quoted title parses",
|
||||||
|
doc: "---\ntitle: 'Headless dispatch runs: Bash is denied'\n---\n\nbody\n",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "plain title parses",
|
||||||
|
doc: "---\ntitle: A simple title\nwing: homelab\n---\n\nbody\n",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "unquoted colon in title does not parse",
|
||||||
|
doc: "---\ntitle: Headless dispatch runs: Bash is denied — use gitea MCP\nwing: homelab\n---\n\nbody\n",
|
||||||
|
wantErr: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// Known gap, deliberately not fixed here: this parses, but the
|
||||||
|
// item becomes map[string]any{"Headless dispatch": "denied"}
|
||||||
|
// rather than a string. It is a silent type corruption, not a
|
||||||
|
// parse failure, so repairing it would rewrite documents that
|
||||||
|
// currently load — out of scope for a lint about broken YAML.
|
||||||
|
name: "unquoted colon in a list item parses as a mapping, not an error",
|
||||||
|
doc: "---\ntitle: Fine\naliases:\n - Headless dispatch: denied\n---\n\nbody\n",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "unterminated frontmatter is not treated as frontmatter",
|
||||||
|
doc: "---\ntitle: no closing fence\n\nbody\n",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
err := Lint([]byte(tt.doc))
|
||||||
|
if tt.wantErr {
|
||||||
|
require.Error(t, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
require.NoError(t, err)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFix(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
doc string
|
||||||
|
wantChanged bool
|
||||||
|
wantKey string
|
||||||
|
wantValue string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "quotes an unquoted title containing a colon",
|
||||||
|
doc: "---\ntitle: Headless dispatch runs: Bash is denied — use gitea MCP\nwing: homelab\n---\n\nbody\n",
|
||||||
|
wantChanged: true,
|
||||||
|
wantKey: "title",
|
||||||
|
wantValue: "Headless dispatch runs: Bash is denied — use gitea MCP",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// Parses already (see the Lint table), so Fix must leave it alone
|
||||||
|
// rather than rewrite a document that loads.
|
||||||
|
name: "leaves a colon-bearing list item alone because it already parses",
|
||||||
|
doc: "---\ntitle: Fine\naliases:\n - Headless dispatch: denied\n---\n\nbody\n",
|
||||||
|
wantChanged: false,
|
||||||
|
wantKey: "title",
|
||||||
|
wantValue: "Fine",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "escapes an embedded single quote when quoting",
|
||||||
|
doc: "---\ntitle: Don't do this: it breaks\n---\n\nbody\n",
|
||||||
|
wantChanged: true,
|
||||||
|
wantKey: "title",
|
||||||
|
wantValue: "Don't do this: it breaks",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// The value opens with a double quote and then keeps going after
|
||||||
|
// the closing one, so YAML reads it as a quoted scalar followed by
|
||||||
|
// junk. Treating "starts with a quote" as "already quoted" leaves
|
||||||
|
// this broken forever.
|
||||||
|
name: "quotes a value that only looks quoted",
|
||||||
|
doc: "---\ntitle: \"Evaluation Driven Generation\" — a TDD-style scaffold\ntags: [a, b]\n---\n\nbody\n",
|
||||||
|
wantChanged: true,
|
||||||
|
wantKey: "title",
|
||||||
|
wantValue: "\"Evaluation Driven Generation\" — a TDD-style scaffold",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "leaves an already-valid document untouched",
|
||||||
|
doc: "---\ntitle: 'Already quoted: fine'\nwing: homelab\n---\n\nbody\n",
|
||||||
|
wantChanged: false,
|
||||||
|
wantKey: "title",
|
||||||
|
wantValue: "Already quoted: fine",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "leaves a document with no frontmatter untouched",
|
||||||
|
doc: "# Heading\n\nbody: with a colon\n",
|
||||||
|
wantChanged: false,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
got, changed, err := Fix([]byte(tt.doc))
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, tt.wantChanged, changed)
|
||||||
|
|
||||||
|
if !tt.wantChanged {
|
||||||
|
assert.Equal(t, tt.doc, string(got), "unchanged docs must be returned byte-identical")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Whatever came back must parse — that is the whole point.
|
||||||
|
require.NoError(t, Lint(got), "Fix must not return a document that still fails to parse")
|
||||||
|
|
||||||
|
if tt.wantKey == "" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
block, ok := block(got)
|
||||||
|
require.True(t, ok)
|
||||||
|
var m map[string]any
|
||||||
|
require.NoError(t, yaml.Unmarshal(block, &m))
|
||||||
|
assert.Equal(t, tt.wantValue, m[tt.wantKey], "the value must survive quoting unchanged")
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFix_LeavesStructuredValuesStructured(t *testing.T) {
|
||||||
|
// Only the broken scalar may be touched. A flow sequence on a neighbouring
|
||||||
|
// line still has to load as a sequence afterwards, not as a string.
|
||||||
|
doc := "---\ntitle: Broken: here\ntags: [prompting, llm]\ncount: 3\n---\n\nbody\n"
|
||||||
|
|
||||||
|
got, changed, err := Fix([]byte(doc))
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.True(t, changed)
|
||||||
|
|
||||||
|
b, ok := block(got)
|
||||||
|
require.True(t, ok)
|
||||||
|
var m map[string]any
|
||||||
|
require.NoError(t, yaml.Unmarshal(b, &m))
|
||||||
|
|
||||||
|
assert.Equal(t, "Broken: here", m["title"])
|
||||||
|
assert.Equal(t, []any{"prompting", "llm"}, m["tags"])
|
||||||
|
assert.Equal(t, 3, m["count"])
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFix_PreservesBodyExactly(t *testing.T) {
|
||||||
|
body := "\nThe body has: colons, 'quotes', and\n---\na horizontal rule.\n"
|
||||||
|
doc := "---\ntitle: Broken: title\n---" + body
|
||||||
|
|
||||||
|
got, changed, err := Fix([]byte(doc))
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.True(t, changed)
|
||||||
|
assert.True(t, len(got) > 0)
|
||||||
|
|
||||||
|
_, rest, ok := split([]byte(doc))
|
||||||
|
require.True(t, ok)
|
||||||
|
_, gotRest, ok := split(got)
|
||||||
|
require.True(t, ok)
|
||||||
|
assert.Equal(t, string(rest), string(gotRest), "the body must not be touched")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFix_RefusesToReturnAnUnparseableResult(t *testing.T) {
|
||||||
|
// A frontmatter block that quoting cannot rescue: the structure itself is
|
||||||
|
// broken, not a scalar. Fix must return the original and an error rather
|
||||||
|
// than a half-repaired document.
|
||||||
|
doc := "---\ntitle: fine\n\tbad: tab-indented mapping\n - stray list item\n---\n\nbody\n"
|
||||||
|
|
||||||
|
got, changed, err := Fix([]byte(doc))
|
||||||
|
require.Error(t, err)
|
||||||
|
assert.False(t, changed)
|
||||||
|
assert.Equal(t, doc, string(got), "an unfixable document must come back byte-identical")
|
||||||
|
}
|
||||||
@@ -0,0 +1,84 @@
|
|||||||
|
// ingestion/internal/frontmatter/scan.go
|
||||||
|
package frontmatter
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"io/fs"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"unicode/utf8"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Finding is one markdown file whose frontmatter does not parse.
|
||||||
|
type Finding struct {
|
||||||
|
Path string // relative to the scanned root
|
||||||
|
Err error // why it does not parse, or why the repair was refused
|
||||||
|
Repaired bool // true only when fix was requested and the file was rewritten
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scan walks root for *.md files whose frontmatter fails to parse.
|
||||||
|
//
|
||||||
|
// With fix set, each finding that quoting can rescue is rewritten in place;
|
||||||
|
// anything quoting cannot rescue is reported and left byte-identical. A file
|
||||||
|
// that cannot be read or decoded is reported as a finding rather than aborting
|
||||||
|
// the walk — one poisoned note must not hide the other findings.
|
||||||
|
func Scan(root string, fix bool) ([]Finding, error) {
|
||||||
|
var found []Finding
|
||||||
|
|
||||||
|
err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if d.IsDir() || !strings.EqualFold(filepath.Ext(path), ".md") {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
rel, relErr := filepath.Rel(root, path)
|
||||||
|
if relErr != nil {
|
||||||
|
rel = path
|
||||||
|
}
|
||||||
|
|
||||||
|
doc, readErr := os.ReadFile(path)
|
||||||
|
if readErr != nil {
|
||||||
|
found = append(found, Finding{Path: rel, Err: readErr})
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if !utf8.Valid(doc) {
|
||||||
|
found = append(found, Finding{Path: rel, Err: fmt.Errorf("not valid UTF-8")})
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if lintErr := Lint(doc); lintErr == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
if !fix {
|
||||||
|
found = append(found, Finding{Path: rel, Err: Lint(doc)})
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
repaired, changed, fixErr := Fix(doc)
|
||||||
|
if fixErr != nil || !changed {
|
||||||
|
found = append(found, Finding{Path: rel, Err: fixErr})
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if writeErr := writeFilePreservingMode(path, repaired); writeErr != nil {
|
||||||
|
found = append(found, Finding{Path: rel, Err: writeErr})
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
found = append(found, Finding{Path: rel, Repaired: true})
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("scan %s: %w", root, err)
|
||||||
|
}
|
||||||
|
return found, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeFilePreservingMode(path string, content []byte) error {
|
||||||
|
mode := fs.FileMode(0o644)
|
||||||
|
if info, err := os.Stat(path); err == nil {
|
||||||
|
mode = info.Mode().Perm()
|
||||||
|
}
|
||||||
|
return os.WriteFile(path, content, mode)
|
||||||
|
}
|
||||||
@@ -0,0 +1,103 @@
|
|||||||
|
// ingestion/internal/frontmatter/scan_test.go
|
||||||
|
package frontmatter
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
func writeTree(t *testing.T, files map[string]string) string {
|
||||||
|
t.Helper()
|
||||||
|
root := t.TempDir()
|
||||||
|
for name, content := range files {
|
||||||
|
path := filepath.Join(root, name)
|
||||||
|
require.NoError(t, os.MkdirAll(filepath.Dir(path), 0o755))
|
||||||
|
require.NoError(t, os.WriteFile(path, []byte(content), 0o644))
|
||||||
|
}
|
||||||
|
return root
|
||||||
|
}
|
||||||
|
|
||||||
|
const (
|
||||||
|
brokenDoc = "---\ntitle: Broken: title here\nwing: homelab\n---\n\nbody\n"
|
||||||
|
validDoc = "---\ntitle: 'Fine: quoted'\n---\n\nbody\n"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestScan_ReportsOnlyBrokenFiles(t *testing.T) {
|
||||||
|
root := writeTree(t, map[string]string{
|
||||||
|
"wiki/a/broken.md": brokenDoc,
|
||||||
|
"wiki/a/valid.md": validDoc,
|
||||||
|
"wiki/b/plain.md": "# no frontmatter\n\nbody: with colon\n",
|
||||||
|
"wiki/a/notes.txt": brokenDoc, // not markdown — must be ignored
|
||||||
|
})
|
||||||
|
|
||||||
|
found, err := Scan(root, false)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
require.Len(t, found, 1)
|
||||||
|
assert.Equal(t, "wiki/a/broken.md", found[0].Path)
|
||||||
|
assert.False(t, found[0].Repaired)
|
||||||
|
assert.Error(t, found[0].Err)
|
||||||
|
|
||||||
|
// Dry run must not touch the file.
|
||||||
|
got, err := os.ReadFile(filepath.Join(root, "wiki/a/broken.md"))
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, brokenDoc, string(got))
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestScan_FixWritesRepairedFilesAndLeavesTheRestAlone(t *testing.T) {
|
||||||
|
root := writeTree(t, map[string]string{
|
||||||
|
"wiki/a/broken.md": brokenDoc,
|
||||||
|
"wiki/a/valid.md": validDoc,
|
||||||
|
})
|
||||||
|
|
||||||
|
found, err := Scan(root, true)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, found, 1)
|
||||||
|
assert.True(t, found[0].Repaired)
|
||||||
|
assert.NoError(t, found[0].Err)
|
||||||
|
|
||||||
|
repaired, err := os.ReadFile(filepath.Join(root, "wiki/a/broken.md"))
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NoError(t, Lint(repaired), "the repaired file must parse")
|
||||||
|
assert.Contains(t, string(repaired), "title: 'Broken: title here'")
|
||||||
|
assert.Contains(t, string(repaired), "wing: homelab", "untouched keys must survive")
|
||||||
|
|
||||||
|
untouched, err := os.ReadFile(filepath.Join(root, "wiki/a/valid.md"))
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, validDoc, string(untouched))
|
||||||
|
|
||||||
|
// Second pass finds nothing — the repair is idempotent and complete.
|
||||||
|
again, err := Scan(root, true)
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Empty(t, again)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestScan_ReportsUnfixableWithoutWriting(t *testing.T) {
|
||||||
|
unfixable := "---\ntitle: fine\n\tbad: tab-indented\n - stray\n---\n\nbody\n"
|
||||||
|
root := writeTree(t, map[string]string{"wiki/x.md": unfixable})
|
||||||
|
|
||||||
|
found, err := Scan(root, true)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, found, 1)
|
||||||
|
assert.False(t, found[0].Repaired)
|
||||||
|
assert.Error(t, found[0].Err)
|
||||||
|
|
||||||
|
got, err := os.ReadFile(filepath.Join(root, "wiki/x.md"))
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, unfixable, string(got), "an unfixable file must be left exactly as it was")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestScan_ReportsUndecodableFilesInsteadOfFailingTheRun(t *testing.T) {
|
||||||
|
root := writeTree(t, map[string]string{"wiki/ok.md": validDoc})
|
||||||
|
require.NoError(t, os.WriteFile(filepath.Join(root, "wiki/bad-bytes.md"),
|
||||||
|
[]byte{'-', '-', '-', '\n', 0xff, 0xfe, '\n', '-', '-', '-', '\n'}, 0o644))
|
||||||
|
|
||||||
|
found, err := Scan(root, false)
|
||||||
|
require.NoError(t, err, "one unreadable file must not abort the whole scan")
|
||||||
|
require.Len(t, found, 1)
|
||||||
|
assert.Equal(t, "wiki/bad-bytes.md", found[0].Path)
|
||||||
|
}
|
||||||
@@ -5,14 +5,18 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Router calls Primary first; on any error falls back to Fallback.
|
// Router calls Primary first; on any error falls back to Fallback, then to
|
||||||
// Fallback may be nil, in which case primary errors are returned directly.
|
// Tertiary. Fallback and Tertiary may each be nil — a nil tier is skipped and
|
||||||
|
// the chain returns the errors accumulated so far.
|
||||||
type Router struct {
|
type Router struct {
|
||||||
Primary *Client
|
Primary *Client
|
||||||
Fallback *Client
|
Fallback *Client
|
||||||
|
Tertiary *Client
|
||||||
}
|
}
|
||||||
|
|
||||||
// Complete implements pipeline.CompleteFunc, routing through Primary then Fallback.
|
// Complete implements pipeline.CompleteFunc, routing through Primary, then
|
||||||
|
// Fallback, then Tertiary — trying each configured tier in order and
|
||||||
|
// returning the first success.
|
||||||
func (r *Router) Complete(ctx context.Context, system, user string) (string, error) {
|
func (r *Router) Complete(ctx context.Context, system, user string) (string, error) {
|
||||||
out, err := r.Primary.Complete(ctx, system, user)
|
out, err := r.Primary.Complete(ctx, system, user)
|
||||||
if err == nil {
|
if err == nil {
|
||||||
@@ -22,8 +26,15 @@ func (r *Router) Complete(ctx context.Context, system, user string) (string, err
|
|||||||
return "", fmt.Errorf("primary llm: %w", err)
|
return "", fmt.Errorf("primary llm: %w", err)
|
||||||
}
|
}
|
||||||
out, err2 := r.Fallback.Complete(ctx, system, user)
|
out, err2 := r.Fallback.Complete(ctx, system, user)
|
||||||
if err2 != nil {
|
if err2 == nil {
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
if r.Tertiary == nil {
|
||||||
return "", fmt.Errorf("primary llm: %w; fallback llm: %v", err, err2)
|
return "", fmt.Errorf("primary llm: %w; fallback llm: %v", err, err2)
|
||||||
}
|
}
|
||||||
|
out, err3 := r.Tertiary.Complete(ctx, system, user)
|
||||||
|
if err3 != nil {
|
||||||
|
return "", fmt.Errorf("primary llm: %w; fallback llm: %v; tertiary llm: %v", err, err2, err3)
|
||||||
|
}
|
||||||
return out, nil
|
return out, nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -69,3 +69,78 @@ func TestRouter_NilFallback(t *testing.T) {
|
|||||||
_, err := r.Complete(context.Background(), "sys", "user")
|
_, err := r.Complete(context.Background(), "sys", "user")
|
||||||
assert.Error(t, err)
|
assert.Error(t, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRouter_FallsBackToTertiaryWhenPrimaryAndFallbackFail(t *testing.T) {
|
||||||
|
fail := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
http.Error(w, "unavailable", http.StatusServiceUnavailable)
|
||||||
|
}))
|
||||||
|
defer fail.Close()
|
||||||
|
tertiary := mockServer(t, "from-tertiary")
|
||||||
|
defer tertiary.Close()
|
||||||
|
|
||||||
|
r := &Router{
|
||||||
|
Primary: New(fail.URL, "", "m", time.Second),
|
||||||
|
Fallback: New(fail.URL, "", "m", time.Second),
|
||||||
|
Tertiary: New(tertiary.URL, "", "m", time.Second),
|
||||||
|
}
|
||||||
|
out, err := r.Complete(context.Background(), "sys", "user")
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, "from-tertiary", out)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRouter_TertiaryNotCalledWhenFallbackSucceeds(t *testing.T) {
|
||||||
|
fail := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
http.Error(w, "unavailable", http.StatusServiceUnavailable)
|
||||||
|
}))
|
||||||
|
defer fail.Close()
|
||||||
|
fallback := mockServer(t, "from-fallback")
|
||||||
|
defer fallback.Close()
|
||||||
|
tertiary := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
t.Error("tertiary must not be called when fallback succeeds")
|
||||||
|
}))
|
||||||
|
defer tertiary.Close()
|
||||||
|
|
||||||
|
r := &Router{
|
||||||
|
Primary: New(fail.URL, "", "m", time.Second),
|
||||||
|
Fallback: New(fallback.URL, "", "m", time.Second),
|
||||||
|
Tertiary: New(tertiary.URL, "", "m", time.Second),
|
||||||
|
}
|
||||||
|
out, err := r.Complete(context.Background(), "sys", "user")
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, "from-fallback", out)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRouter_AllThreeFail(t *testing.T) {
|
||||||
|
fail := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
http.Error(w, "err", http.StatusBadGateway)
|
||||||
|
}))
|
||||||
|
defer fail.Close()
|
||||||
|
|
||||||
|
r := &Router{
|
||||||
|
Primary: New(fail.URL, "", "m", time.Second),
|
||||||
|
Fallback: New(fail.URL, "", "m", time.Second),
|
||||||
|
Tertiary: New(fail.URL, "", "m", time.Second),
|
||||||
|
}
|
||||||
|
_, err := r.Complete(context.Background(), "sys", "user")
|
||||||
|
assert.Error(t, err)
|
||||||
|
assert.Contains(t, err.Error(), "primary llm")
|
||||||
|
assert.Contains(t, err.Error(), "fallback llm")
|
||||||
|
assert.Contains(t, err.Error(), "tertiary llm")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRouter_NilTertiary(t *testing.T) {
|
||||||
|
// Existing 2-tier deployments (no BRAIN_LLM_TERTIARY_URL) must keep working
|
||||||
|
// exactly as before — Tertiary is opt-in, not required.
|
||||||
|
fail := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
http.Error(w, "err", http.StatusBadGateway)
|
||||||
|
}))
|
||||||
|
defer fail.Close()
|
||||||
|
|
||||||
|
r := &Router{
|
||||||
|
Primary: New(fail.URL, "", "m", time.Second),
|
||||||
|
Fallback: New(fail.URL, "", "m", time.Second),
|
||||||
|
}
|
||||||
|
_, err := r.Complete(context.Background(), "sys", "user")
|
||||||
|
assert.Error(t, err)
|
||||||
|
assert.NotContains(t, err.Error(), "tertiary")
|
||||||
|
}
|
||||||
|
|||||||
@@ -165,7 +165,7 @@ func TestBrainUpdateReembedsNewContent(t *testing.T) {
|
|||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
|
|
||||||
callResult(t, toolCall(t, srv, "brain_write", map[string]any{
|
callResult(t, toolCall(t, srv, "brain_write", map[string]any{
|
||||||
"content": "# Note\n\nthe OLD distinctive payload\n",
|
"content": "# Note\n\nthe OLD distinctive payload\n",
|
||||||
"filename": "n", "wing": "a", "hall": "facts",
|
"filename": "n", "wing": "a", "hall": "facts",
|
||||||
}))
|
}))
|
||||||
|
|
||||||
|
|||||||
@@ -14,6 +14,7 @@ import (
|
|||||||
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
|
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
|
||||||
"github.com/mathiasbq/hyperguild/ingestion/internal/extract"
|
"github.com/mathiasbq/hyperguild/ingestion/internal/extract"
|
||||||
"github.com/mathiasbq/hyperguild/ingestion/internal/graphsync"
|
"github.com/mathiasbq/hyperguild/ingestion/internal/graphsync"
|
||||||
|
"github.com/mathiasbq/hyperguild/ingestion/internal/notes"
|
||||||
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
|
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
|
||||||
"github.com/mathiasbq/hyperguild/ingestion/internal/search"
|
"github.com/mathiasbq/hyperguild/ingestion/internal/search"
|
||||||
"github.com/mathiasbq/hyperguild/ingestion/internal/session"
|
"github.com/mathiasbq/hyperguild/ingestion/internal/session"
|
||||||
@@ -97,6 +98,8 @@ func (s *Server) tools() []map[string]any {
|
|||||||
"slug": str("optional target slug; defaults to filename minus date prefix"),
|
"slug": str("optional target slug; defaults to filename minus date prefix"),
|
||||||
}),
|
}),
|
||||||
},
|
},
|
||||||
|
noteToolDescriptors()[0],
|
||||||
|
noteToolDescriptors()[1],
|
||||||
{
|
{
|
||||||
"name": "brain_tunnel",
|
"name": "brain_tunnel",
|
||||||
"description": "Create an explicit bidirectional [[wikilink]] between two notes in different wings. Idempotent.",
|
"description": "Create an explicit bidirectional [[wikilink]] between two notes in different wings. Idempotent.",
|
||||||
@@ -105,13 +108,6 @@ func (s *Server) tools() []map[string]any {
|
|||||||
"target": str("path of target note (must be in a different wing)"),
|
"target": str("path of target note (must be in a different wing)"),
|
||||||
}),
|
}),
|
||||||
},
|
},
|
||||||
{
|
|
||||||
"name": "brain_index",
|
|
||||||
"description": "Regenerate _index.md (Map of Content) for one or all wings under brain/wiki/. Auto-called after brain_write with wing+hall.",
|
|
||||||
"inputSchema": schema([]string{}, map[string]any{
|
|
||||||
"wing": str("optional wing to index; if absent, rebuilds every wing"),
|
|
||||||
}),
|
|
||||||
},
|
|
||||||
{
|
{
|
||||||
"name": "brain_ingest_raw",
|
"name": "brain_ingest_raw",
|
||||||
"description": "Ingest pre-structured pages into the brain wiki, bypassing the LLM extraction step.",
|
"description": "Ingest pre-structured pages into the brain wiki, bypassing the LLM extraction step.",
|
||||||
@@ -327,6 +323,15 @@ func (s *Server) brainGet(ctx context.Context, args json.RawMessage) (json.RawMe
|
|||||||
if target == "" {
|
if target == "" {
|
||||||
return nil, fmt.Errorf("id or path is required")
|
return nil, fmt.Errorf("id or path is required")
|
||||||
}
|
}
|
||||||
|
// Containment: the store resolves any brain-relative path, and
|
||||||
|
// parseFrontmatter returns the WHOLE file as body when there is no
|
||||||
|
// frontmatter fence — so without this, brain_get reads verbatim session
|
||||||
|
// transcripts under archive/ and sessions/, which BM25 never indexes and
|
||||||
|
// which nobody therefore thinks of as reachable. Same allowlist as
|
||||||
|
// brain_read (brain#4).
|
||||||
|
if _, err := notes.Resolve(s.brainDir, target, notes.ScopeNotes); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
note, err := s.store.Get(ctx, target)
|
note, err := s.store.Get(ctx, target)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
|
|||||||
@@ -0,0 +1,76 @@
|
|||||||
|
package mcp
|
||||||
|
|
||||||
|
import (
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
|
||||||
|
)
|
||||||
|
|
||||||
|
// pendingNudgeThreshold is how old a raw/ note has to be before brain_context
|
||||||
|
// mentions the queue. Two weeks: long enough that a note written this sprint
|
||||||
|
// does not nag, short enough that four items cannot sit unnoticed for three
|
||||||
|
// months again (#82).
|
||||||
|
const pendingNudgeThreshold = 14 * 24 * time.Hour
|
||||||
|
|
||||||
|
// PendingNudge is the in-session reminder that the promote/reject queue has
|
||||||
|
// items waiting. It is advisory: a count, an age, and how to look.
|
||||||
|
type PendingNudge struct {
|
||||||
|
Pending int `json:"pending"` // everything in the queue
|
||||||
|
Aging int `json:"aging"` // those past the threshold
|
||||||
|
Oldest string `json:"oldest"` // filename of the oldest pending note
|
||||||
|
OldestDays int `json:"oldest_days"` // its age, in whole days
|
||||||
|
Hint string `json:"hint"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// pendingNudge summarises the raw/ review queue, or returns nil when there is
|
||||||
|
// nothing worth saying.
|
||||||
|
//
|
||||||
|
// The trigger is the missing half of the queue (#82): brain_pending has always
|
||||||
|
// existed, and four real items still sat unreviewed from 2026-04-23 to
|
||||||
|
// 2026-07-15 because nothing ever prompted anyone to call it. The system
|
||||||
|
// should be what remembers the queue exists.
|
||||||
|
//
|
||||||
|
// Threshold-gated on purpose. A notice on every context call is one people
|
||||||
|
// learn to skip, which reproduces the original failure with extra steps.
|
||||||
|
//
|
||||||
|
// It never promotes, rejects, or reorders anything. Age is a reason to ask a
|
||||||
|
// human, not a verdict — an old note may be exactly right to promote once
|
||||||
|
// someone finally looks at it.
|
||||||
|
func pendingNudge(brainDir string, now time.Time, threshold time.Duration) *PendingNudge {
|
||||||
|
pending, err := api.ListPending(brainDir)
|
||||||
|
if err != nil || len(pending) == 0 {
|
||||||
|
// brain_context is a hot path; a broken queue degrades to silence
|
||||||
|
// rather than failing the call that asked for project context.
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
var aging int
|
||||||
|
var oldestName string
|
||||||
|
var oldestAge time.Duration
|
||||||
|
|
||||||
|
for _, p := range pending {
|
||||||
|
created, perr := time.Parse(time.RFC3339, p.CreatedAt)
|
||||||
|
if perr != nil {
|
||||||
|
continue // an unparseable timestamp is not evidence of age
|
||||||
|
}
|
||||||
|
age := now.Sub(created)
|
||||||
|
if age >= threshold {
|
||||||
|
aging++
|
||||||
|
}
|
||||||
|
if age > oldestAge {
|
||||||
|
oldestAge, oldestName = age, p.Filename
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if aging == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return &PendingNudge{
|
||||||
|
Pending: len(pending),
|
||||||
|
Aging: aging,
|
||||||
|
Oldest: oldestName,
|
||||||
|
OldestDays: int(oldestAge.Hours() / 24),
|
||||||
|
Hint: "call brain_pending to review; nothing is promoted or rejected automatically",
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,115 @@
|
|||||||
|
package mcp
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
// writePending drops a raw/ note whose created_at is age old.
|
||||||
|
func writePending(t *testing.T, brainDir, name string, age time.Duration, now time.Time) {
|
||||||
|
t.Helper()
|
||||||
|
raw := filepath.Join(brainDir, "raw")
|
||||||
|
require.NoError(t, os.MkdirAll(raw, 0o755))
|
||||||
|
body := "---\ncreated_at: " + now.Add(-age).UTC().Format(time.RFC3339) + "\n---\n\nnote body\n"
|
||||||
|
require.NoError(t, os.WriteFile(filepath.Join(raw, name), []byte(body), 0o644))
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPendingNudge_SilentWhenNothingIsPending(t *testing.T) {
|
||||||
|
// No queue, no nudge. A notice that fires every session is one people
|
||||||
|
// learn to skip, which is how the queue got ignored for three months in
|
||||||
|
// the first place (#82).
|
||||||
|
got := pendingNudge(t.TempDir(), time.Now(), 14*24*time.Hour)
|
||||||
|
assert.Nil(t, got)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPendingNudge_SilentWhileEverythingIsBelowTheThreshold(t *testing.T) {
|
||||||
|
now := time.Now()
|
||||||
|
dir := t.TempDir()
|
||||||
|
writePending(t, dir, "fresh.md", 2*24*time.Hour, now)
|
||||||
|
|
||||||
|
got := pendingNudge(dir, now, 14*24*time.Hour)
|
||||||
|
assert.Nil(t, got, "a two-day-old note is not aging")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPendingNudge_ReportsCountAndOldestAgeOnceSomethingAges(t *testing.T) {
|
||||||
|
now := time.Now()
|
||||||
|
dir := t.TempDir()
|
||||||
|
writePending(t, dir, "old.md", 100*24*time.Hour, now)
|
||||||
|
writePending(t, dir, "older.md", 120*24*time.Hour, now)
|
||||||
|
writePending(t, dir, "fresh.md", 1*24*time.Hour, now)
|
||||||
|
|
||||||
|
got := pendingNudge(dir, now, 14*24*time.Hour)
|
||||||
|
require.NotNil(t, got)
|
||||||
|
|
||||||
|
// All three are pending; the threshold decides whether to speak, not
|
||||||
|
// what to count. Someone opening the queue sees the whole queue.
|
||||||
|
assert.Equal(t, 3, got.Pending)
|
||||||
|
assert.Equal(t, 2, got.Aging)
|
||||||
|
assert.Equal(t, 120, got.OldestDays)
|
||||||
|
assert.Equal(t, "older.md", got.Oldest)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPendingNudge_NeverDecidesAnything(t *testing.T) {
|
||||||
|
// Aging is a trigger to ask a human, never a verdict. The nudge must
|
||||||
|
// carry no action and must leave the queue exactly as it found it.
|
||||||
|
now := time.Now()
|
||||||
|
dir := t.TempDir()
|
||||||
|
writePending(t, dir, "ancient.md", 365*24*time.Hour, now)
|
||||||
|
|
||||||
|
before, err := os.ReadDir(filepath.Join(dir, "raw"))
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
got := pendingNudge(dir, now, 14*24*time.Hour)
|
||||||
|
require.NotNil(t, got)
|
||||||
|
|
||||||
|
after, err := os.ReadDir(filepath.Join(dir, "raw"))
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Len(t, after, len(before), "the nudge must not promote or reject anything")
|
||||||
|
assert.Contains(t, got.Hint, "brain_pending", "the nudge should say how to look, not act")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPendingNudge_SurvivesAnUnreadableQueue(t *testing.T) {
|
||||||
|
// brain_context is a hot path; a broken raw/ dir must degrade to silence
|
||||||
|
// rather than failing the whole context call.
|
||||||
|
dir := t.TempDir()
|
||||||
|
require.NoError(t, os.WriteFile(filepath.Join(dir, "raw"), []byte("not a directory"), 0o644))
|
||||||
|
|
||||||
|
assert.NotPanics(t, func() {
|
||||||
|
assert.Nil(t, pendingNudge(dir, time.Now(), 14*24*time.Hour))
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBrainContext_CarriesTheNudgeWithoutFailingWhenTheQueueIsEmpty(t *testing.T) {
|
||||||
|
// The wiring, not the summary: a nudge that never reaches a caller is the
|
||||||
|
// same as no nudge, and brain_context must still work with no queue.
|
||||||
|
dir := t.TempDir()
|
||||||
|
srv := NewServer(dir, nil, nil, nil)
|
||||||
|
|
||||||
|
raw, err := srv.brainContext(context.Background(),
|
||||||
|
json.RawMessage(`{"project_root":"/home/mathias/dev/AGENTS/hyperguild"}`))
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
var empty map[string]any
|
||||||
|
require.NoError(t, json.Unmarshal(raw, &empty))
|
||||||
|
assert.NotContains(t, empty, "pending_review", "silence when nothing is aging")
|
||||||
|
|
||||||
|
writePending(t, dir, "ancient.md", 200*24*time.Hour, time.Now())
|
||||||
|
|
||||||
|
raw, err = srv.brainContext(context.Background(),
|
||||||
|
json.RawMessage(`{"project_root":"/home/mathias/dev/AGENTS/hyperguild"}`))
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
var withQueue map[string]any
|
||||||
|
require.NoError(t, json.Unmarshal(raw, &withQueue))
|
||||||
|
nudge, ok := withQueue["pending_review"].(map[string]any)
|
||||||
|
require.True(t, ok, "the nudge must reach the caller: %v", withQueue)
|
||||||
|
assert.Equal(t, float64(1), nudge["aging"])
|
||||||
|
assert.Equal(t, "ancient.md", nudge["oldest"])
|
||||||
|
}
|
||||||
@@ -1,9 +1,14 @@
|
|||||||
// Package mcp implements an MCP HTTP handler for the ingestion service.
|
// Package mcp implements an MCP HTTP handler for the ingestion service.
|
||||||
// Exposed tools: brain_query, brain_write, brain_update, brain_get,
|
// Exposed tools: brain_query, brain_write, brain_update, brain_get,
|
||||||
// brain_pending, brain_promote, brain_index, brain_tunnel, brain_ingest,
|
// brain_pending, brain_promote, brain_tunnel, brain_ingest,
|
||||||
// brain_ingest_raw, brain_answer, brain_classify, brain_graph,
|
// brain_ingest_raw, brain_answer, brain_classify, brain_graph,
|
||||||
// brain_context, session_log, and capture (the #55 relay tool, registered
|
// brain_context, session_log, and capture (the #55 relay tool, registered
|
||||||
// only when WithCapture is set).
|
// only when WithCapture is set).
|
||||||
|
//
|
||||||
|
// brain_index dispatches but is deliberately absent from tools/list: it is
|
||||||
|
// auto-called after every write, so surfacing it as a caller verb is
|
||||||
|
// cognitive load with no decision behind it (ADR-0012 rule 4, #78). It stays
|
||||||
|
// callable for maintenance — de-surfacing hides the verb, not the operation.
|
||||||
package mcp
|
package mcp
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -44,6 +49,8 @@ type rpcError struct {
|
|||||||
// Server handles MCP JSON-RPC over HTTP for the ingestion service.
|
// Server handles MCP JSON-RPC over HTTP for the ingestion service.
|
||||||
type Server struct {
|
type Server struct {
|
||||||
brainDir string
|
brainDir string
|
||||||
|
govAudit capture.AuditSink
|
||||||
|
govNotify govNotifier
|
||||||
pipeline pipeline.Config
|
pipeline pipeline.Config
|
||||||
llm pipeline.CompleteFunc
|
llm pipeline.CompleteFunc
|
||||||
answerLLM pipeline.CompleteFunc // nil = brain_answer and brain_classify unavailable
|
answerLLM pipeline.CompleteFunc // nil = brain_answer and brain_classify unavailable
|
||||||
@@ -270,6 +277,15 @@ func (s *Server) handleCall(ctx context.Context, name string, args json.RawMessa
|
|||||||
return s.brainPromote(ctx, args)
|
return s.brainPromote(ctx, args)
|
||||||
case "capture":
|
case "capture":
|
||||||
return s.brainCapture(ctx, args)
|
return s.brainCapture(ctx, args)
|
||||||
|
case "brain_read":
|
||||||
|
return s.brainRead(ctx, args)
|
||||||
|
case "brain_edit":
|
||||||
|
return s.brainEdit(ctx, args)
|
||||||
|
case "brain_edit_root":
|
||||||
|
// Dispatches but is intentionally absent from tools/list — see
|
||||||
|
// tools_notes.go for why a governance-file editor must not be
|
||||||
|
// discoverable by a model.
|
||||||
|
return s.brainEditRoot(ctx, args)
|
||||||
case "brain_index":
|
case "brain_index":
|
||||||
return s.brainIndex(ctx, args)
|
return s.brainIndex(ctx, args)
|
||||||
case "brain_tunnel":
|
case "brain_tunnel":
|
||||||
|
|||||||
@@ -59,11 +59,43 @@ func TestServerToolsList(t *testing.T) {
|
|||||||
assert.ElementsMatch(t, []string{
|
assert.ElementsMatch(t, []string{
|
||||||
"brain_query", "brain_write", "brain_update", "brain_get",
|
"brain_query", "brain_write", "brain_update", "brain_get",
|
||||||
"brain_pending", "brain_promote",
|
"brain_pending", "brain_promote",
|
||||||
"brain_index", "brain_tunnel",
|
"brain_read", "brain_edit",
|
||||||
|
"brain_tunnel",
|
||||||
"brain_ingest_raw", "brain_ingest",
|
"brain_ingest_raw", "brain_ingest",
|
||||||
"brain_answer", "brain_classify", "brain_graph", "brain_context",
|
"brain_answer", "brain_classify", "brain_graph", "brain_context",
|
||||||
"session_log",
|
"session_log",
|
||||||
}, names)
|
}, names)
|
||||||
|
|
||||||
|
// brain_index is auto-called after every brain_write, so surfacing it as
|
||||||
|
// a caller verb is cognitive load with no decision behind it (ADR-0012
|
||||||
|
// rule 4, #78). The capability stays reachable — see the admin test below.
|
||||||
|
assert.NotContains(t, names, "brain_index")
|
||||||
|
|
||||||
|
// brain_edit_root is de-surfaced for a sharper reason than brain_index:
|
||||||
|
// brain content is ingested from the outside world and reaches a model's
|
||||||
|
// context, so a listed governance-file editor would be a durable
|
||||||
|
// prompt-injection path — one poisoned page rewriting the CLAUDE.md a
|
||||||
|
// .dispatch-allow agent later loads as instructions.
|
||||||
|
assert.NotContains(t, names, "brain_edit_root")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBrainIndexStaysCallableForAdminUseAfterDeSurfacing(t *testing.T) {
|
||||||
|
// De-surfacing hides the verb from the model-facing list; it must not
|
||||||
|
// remove the operation. A maintenance caller still needs to rebuild a
|
||||||
|
// Map of Content by hand.
|
||||||
|
srv := mcp.NewServer(t.TempDir(), nil, nil, nil)
|
||||||
|
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/mcp", body(t, map[string]any{
|
||||||
|
"jsonrpc": "2.0", "id": 3, "method": "tools/call",
|
||||||
|
"params": map[string]any{"name": "brain_index", "arguments": map[string]any{}},
|
||||||
|
}))
|
||||||
|
rr := httptest.NewRecorder()
|
||||||
|
srv.ServeHTTP(rr, req)
|
||||||
|
|
||||||
|
require.Equal(t, http.StatusOK, rr.Code)
|
||||||
|
var resp map[string]any
|
||||||
|
require.NoError(t, json.Unmarshal(rr.Body.Bytes(), &resp))
|
||||||
|
assert.NotContains(t, resp, "error", "brain_index must still dispatch: %v", resp["error"])
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestServerNotificationGetsNoBody(t *testing.T) {
|
func TestServerNotificationGetsNoBody(t *testing.T) {
|
||||||
|
|||||||
@@ -31,7 +31,7 @@ func captureToolDescriptor() map[string]any {
|
|||||||
"type": "object",
|
"type": "object",
|
||||||
"properties": map[string]any{
|
"properties": map[string]any{
|
||||||
"text": str("the insight body"), "wing": str("brain wing"),
|
"text": str("the insight body"), "wing": str("brain wing"),
|
||||||
"hall": str("brain hall (facts/decisions/failures/hypotheses/sources)"),
|
"hall": str("brain hall (facts/decisions/failures/hypotheses/sources)"),
|
||||||
"supersede_slug": str("optional: slug of a prior note to revise in place instead of creating"),
|
"supersede_slug": str("optional: slug of a prior note to revise in place instead of creating"),
|
||||||
},
|
},
|
||||||
"required": []string{"text", "wing", "hall"},
|
"required": []string{"text", "wing", "hall"},
|
||||||
@@ -41,7 +41,7 @@ func captureToolDescriptor() map[string]any {
|
|||||||
"properties": map[string]any{
|
"properties": map[string]any{
|
||||||
"repo": str("gitea repo (owner is always mathias)"), "action": str("create|close|comment"),
|
"repo": str("gitea repo (owner is always mathias)"), "action": str("create|close|comment"),
|
||||||
"number": map[string]any{"type": "integer", "description": "issue number (close/comment)"},
|
"number": map[string]any{"type": "integer", "description": "issue number (close/comment)"},
|
||||||
"title": str("issue title (create)"), "body": str("issue/comment body"),
|
"title": str("issue title (create)"), "body": str("issue/comment body"),
|
||||||
},
|
},
|
||||||
"required": []string{"repo", "action"},
|
"required": []string{"repo", "action"},
|
||||||
}
|
}
|
||||||
@@ -51,24 +51,20 @@ func captureToolDescriptor() map[string]any {
|
|||||||
"context": map[string]any{
|
"context": map[string]any{
|
||||||
"type": "object",
|
"type": "object",
|
||||||
"properties": map[string]any{
|
"properties": map[string]any{
|
||||||
"harness": str("descriptive harness label (telemetry only, never a gate input)"),
|
"harness": str("descriptive harness label (telemetry only, never a gate input)"),
|
||||||
"session_ref": str("optional session reference"), "fidelity": str("live-capture|transcript-parse|agent-runlog"),
|
"session_ref": str("optional session reference"), "fidelity": str("live-capture|transcript-parse|agent-runlog"),
|
||||||
"actor": str("acting user/agent"), "classification": str("caller-declared sensitivity: public|internal|confidential"),
|
"actor": str("acting user/agent"), "classification": str("caller-declared sensitivity: public|internal|confidential"),
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
"insights": map[string]any{"type": "array", "items": insightItem},
|
"insights": map[string]any{"type": "array", "items": insightItem},
|
||||||
"tickets": map[string]any{"type": "array", "items": ticketItem},
|
"tickets": map[string]any{"type": "array", "items": ticketItem},
|
||||||
"summary": map[string]any{"type": "object", "properties": map[string]any{
|
"dry_run": map[string]any{"type": "boolean", "description": "validate + return the would-be receipt, write nothing"},
|
||||||
"title": str("summary title"), "body": str("summary body"),
|
|
||||||
"repos_touched": map[string]any{"type": "array", "items": map[string]any{"type": "string"}},
|
|
||||||
}},
|
|
||||||
"dry_run": map[string]any{"type": "boolean", "description": "validate + return the would-be receipt, write nothing"},
|
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
b, _ := json.Marshal(schema)
|
b, _ := json.Marshal(schema)
|
||||||
return map[string]any{
|
return map[string]any{
|
||||||
"name": "capture",
|
"name": "capture",
|
||||||
"description": "Persist a session's value uniformly: insights → brain (write or supersede), action items → Gitea tickets, optional summary → ai-sessions. The relay door for MCP-native harnesses. Origin is server-derived from your authenticated identity; confidential captures through a us-nexus surface are refused (I1). Returns a partial-aware receipt.",
|
"description": "Persist a session's value uniformly: insights → brain (write or supersede), action items → Gitea tickets. The relay door for MCP-native harnesses. Origin is server-derived from your authenticated identity; confidential captures through a us-nexus surface are refused (I1). Returns a partial-aware receipt.",
|
||||||
"inputSchema": json.RawMessage(b),
|
"inputSchema": json.RawMessage(b),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -49,7 +49,7 @@ func captureServer(t *testing.T, validator capturehttp.Validator, sovereign []st
|
|||||||
brainDir := t.TempDir()
|
brainDir := t.TempDir()
|
||||||
cfg, err := classification.Load(brainDir)
|
cfg, err := classification.Load(brainDir)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
svc := capture.NewService(brainstore.New(brainDir), capFakeTracker{}, nil, cfg, audit.NewSlogSink(nil))
|
svc := capture.NewService(brainstore.New(brainDir), capFakeTracker{}, cfg, audit.NewSlogSink(nil))
|
||||||
srv := mcp.NewServer(brainDir, nil, nil, nil)
|
srv := mcp.NewServer(brainDir, nil, nil, nil)
|
||||||
srv.WithCapture(svc, validator, capStaticTok, "local-cli", capturehttp.NewOriginResolver(sovereign))
|
srv.WithCapture(svc, validator, capStaticTok, "local-cli", capturehttp.NewOriginResolver(sovereign))
|
||||||
return srv, brainDir
|
return srv, brainDir
|
||||||
|
|||||||
@@ -8,8 +8,10 @@ import (
|
|||||||
"path/filepath"
|
"path/filepath"
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
"github.com/mathiasbq/hyperguild/ingestion/internal/search"
|
"github.com/mathiasbq/hyperguild/ingestion/internal/search"
|
||||||
|
"github.com/mathiasbq/hyperguild/ingestion/internal/textutil"
|
||||||
)
|
)
|
||||||
|
|
||||||
// brainContextArgs is the input shape of brain_context. project_root is
|
// brainContextArgs is the input shape of brain_context. project_root is
|
||||||
@@ -155,17 +157,21 @@ func (s *Server) brainContext(ctx context.Context, args json.RawMessage) (json.R
|
|||||||
entries = entries[:limit]
|
entries = entries[:limit]
|
||||||
}
|
}
|
||||||
|
|
||||||
return json.Marshal(map[string]any{"entries": entries})
|
out := map[string]any{"entries": entries}
|
||||||
|
// The queue nudge rides along with a call the session already makes, so
|
||||||
|
// noticing an aging note costs nobody a decision to go looking (#82).
|
||||||
|
// Advisory only, and absent entirely when there is nothing to say.
|
||||||
|
if n := pendingNudge(s.brainDir, time.Now(), pendingNudgeThreshold); n != nil {
|
||||||
|
out["pending_review"] = n
|
||||||
|
}
|
||||||
|
return json.Marshal(out)
|
||||||
}
|
}
|
||||||
|
|
||||||
// truncateExcerpt clamps an already-stripped excerpt to maxLen characters
|
// truncateExcerpt clamps an already-stripped excerpt to maxLen characters
|
||||||
// without re-running the frontmatter parser. The ellipsis suffix matches
|
// without re-running the frontmatter parser. The ellipsis suffix matches
|
||||||
// the convention used in search.excerpt.
|
// the convention used in search.excerpt.
|
||||||
func truncateExcerpt(s string, maxLen int) string {
|
func truncateExcerpt(s string, maxLen int) string {
|
||||||
if len(s) <= maxLen {
|
return textutil.Truncate(s, maxLen)
|
||||||
return s
|
|
||||||
}
|
|
||||||
return s[:maxLen] + "…"
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// readExcerpt loads a doc relative to brainDir, strips its frontmatter,
|
// readExcerpt loads a doc relative to brainDir, strips its frontmatter,
|
||||||
@@ -185,10 +191,7 @@ func readExcerpt(brainDir, relPath string, maxLen int) string {
|
|||||||
if len(parts) == 3 {
|
if len(parts) == 3 {
|
||||||
body = strings.TrimSpace(parts[2])
|
body = strings.TrimSpace(parts[2])
|
||||||
}
|
}
|
||||||
if len(body) > maxLen {
|
return textutil.Truncate(body, maxLen)
|
||||||
return body[:maxLen] + "…"
|
|
||||||
}
|
|
||||||
return body
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// max1 returns the maximum of n and 1, used to guard against divide-by-zero
|
// max1 returns the maximum of n and 1, used to guard against divide-by-zero
|
||||||
|
|||||||
@@ -0,0 +1,222 @@
|
|||||||
|
package mcp
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"log/slog"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
|
||||||
|
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
|
||||||
|
"github.com/mathiasbq/hyperguild/ingestion/internal/classification"
|
||||||
|
"github.com/mathiasbq/hyperguild/ingestion/internal/notes"
|
||||||
|
)
|
||||||
|
|
||||||
|
// noteToolDescriptors are the model-facing halves of brain#4.
|
||||||
|
//
|
||||||
|
// brain_edit_root is deliberately absent: it dispatches but is not listed, the
|
||||||
|
// same de-surfacing ADR-0012 rule 4 applies to brain_index (#78). Hiding the
|
||||||
|
// verb matters here for a specific reason — brain content is ingested from the
|
||||||
|
// outside world and reaches a model's context, so a listed governance-file
|
||||||
|
// editor is a durable prompt-injection path: one poisoned page could rewrite
|
||||||
|
// the CLAUDE.md that a .dispatch-allow agent later obeys. A human who knows
|
||||||
|
// the tool exists can still call it; a model reading tools/list cannot
|
||||||
|
// discover it.
|
||||||
|
func noteToolDescriptors() []map[string]any {
|
||||||
|
str := func(d string) map[string]any { return map[string]any{"type": "string", "description": d} }
|
||||||
|
schema := func(required []string, props map[string]any) json.RawMessage {
|
||||||
|
b, _ := json.Marshal(map[string]any{"type": "object", "required": required, "properties": props})
|
||||||
|
return b
|
||||||
|
}
|
||||||
|
return []map[string]any{
|
||||||
|
{
|
||||||
|
"name": "brain_read",
|
||||||
|
"description": "Return a note's raw markdown verbatim — no synthesis, no excerpting. " +
|
||||||
|
"Use this to obtain the exact text before brain_edit; never build old_str from a " +
|
||||||
|
"brain_query excerpt, which is truncated and normalised and will not match.",
|
||||||
|
"inputSchema": schema([]string{"path"}, map[string]any{
|
||||||
|
"path": str("brain-relative path under knowledge/ or wiki/, e.g. wiki/homelab/facts/x.md"),
|
||||||
|
}),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "brain_edit",
|
||||||
|
"description": "Replace exactly one occurrence of old_str with new_str in a note, leaving " +
|
||||||
|
"every other byte identical. Fails loudly if old_str is absent or matches more than " +
|
||||||
|
"once — extend old_str until it is unique. Does NOT stamp updated_at or supersedes " +
|
||||||
|
"(brain_update does); it is a byte-level patch, not a metadata operation.",
|
||||||
|
"inputSchema": schema([]string{"path", "old_str", "new_str"}, map[string]any{
|
||||||
|
"path": str("brain-relative path under knowledge/ or wiki/"),
|
||||||
|
"old_str": str("exact text to replace; must occur exactly once"),
|
||||||
|
"new_str": str("replacement text; empty string deletes old_str"),
|
||||||
|
}),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- hyperguild#97: governance-rewrite observability -----------------------
|
||||||
|
|
||||||
|
// govNotifier raises an out-of-band alert. Declared here rather than importing
|
||||||
|
// internal/audit so the MCP package keeps depending on capture alone.
|
||||||
|
type govNotifier interface {
|
||||||
|
Notify(ctx context.Context, msg string) error
|
||||||
|
}
|
||||||
|
|
||||||
|
// WithGovernanceObserver attaches the audit sink and notifier that observe a
|
||||||
|
// successful brain_edit_root. Both are optional; either may be nil.
|
||||||
|
func (s *Server) WithGovernanceObserver(sink capture.AuditSink, n govNotifier) {
|
||||||
|
s.govAudit, s.govNotify = sink, n
|
||||||
|
}
|
||||||
|
|
||||||
|
type brainReadArgs struct {
|
||||||
|
Path string `json:"path"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Server) brainRead(_ context.Context, args json.RawMessage) (json.RawMessage, error) {
|
||||||
|
var a brainReadArgs
|
||||||
|
if err := json.Unmarshal(args, &a); err != nil {
|
||||||
|
return nil, fmt.Errorf("parse args: %w", err)
|
||||||
|
}
|
||||||
|
content, err := notes.Read(s.brainDir, a.Path)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return json.Marshal(map[string]any{"path": a.Path, "content": string(content)})
|
||||||
|
}
|
||||||
|
|
||||||
|
type brainEditArgs struct {
|
||||||
|
Path string `json:"path"`
|
||||||
|
OldStr string `json:"old_str"`
|
||||||
|
NewStr string `json:"new_str"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// brainEdit is the model-facing editor: notes only.
|
||||||
|
//
|
||||||
|
// Deliberately NOT audited (#97). A note edit is already fully reconstructible
|
||||||
|
// from git history, which carries the same before/after bytes plus a diff.
|
||||||
|
// Governance files get the extra trail because they change what later agents
|
||||||
|
// obey — the asymmetry is about instruction-carrying content, not about how
|
||||||
|
// hard the change is to reconstruct. Auditing every note edit would also bury
|
||||||
|
// the rare governance record under routine traffic, which is how an alert
|
||||||
|
// stops being read.
|
||||||
|
func (s *Server) brainEdit(_ context.Context, args json.RawMessage) (json.RawMessage, error) {
|
||||||
|
_, out, err := s.editWithScope(args, notes.ScopeNotes)
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// brainEditRoot edits a governance file (CLAUDE.md). De-surfaced AND gated on
|
||||||
|
// the authenticated principal: only the static local-CLI token may call it, so
|
||||||
|
// a claude.ai-origin session — the one that may have just read an untrusted
|
||||||
|
// page — cannot rewrite the instructions later agents load.
|
||||||
|
func (s *Server) brainEditRoot(ctx context.Context, args json.RawMessage) (json.RawMessage, error) {
|
||||||
|
info, ok := ctx.Value(principalKey).(principalInfo)
|
||||||
|
if !ok || !info.viaStatic {
|
||||||
|
return nil, fmt.Errorf("brain_edit_root requires the local static principal; " +
|
||||||
|
"a governance file is not editable from a model-facing origin")
|
||||||
|
}
|
||||||
|
res, out, err := s.editWithScope(args, notes.ScopeRoot)
|
||||||
|
if err != nil {
|
||||||
|
// A refusal is not a rewrite. Recording or alerting on one would cry
|
||||||
|
// wolf, and the gate already logs its own denials.
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
// A durable instruction rewrite should be loud, not silent.
|
||||||
|
slog.Warn("brain_edit_root applied", "principal", info.principal, "result", string(out))
|
||||||
|
s.observeGovernanceEdit(ctx, info.principal, res)
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// observeGovernanceEdit records and alerts on a rewrite that already landed.
|
||||||
|
//
|
||||||
|
// Best-effort by construction: the bytes are written and brain-sync will commit
|
||||||
|
// them within the quarter hour, so failing the call here would report a
|
||||||
|
// non-rewrite to the caller while the rewrite sits on disk. A failure to
|
||||||
|
// observe is therefore logged at ERROR — the one case where the trail itself is
|
||||||
|
// what broke, and the only remaining signal.
|
||||||
|
func (s *Server) observeGovernanceEdit(ctx context.Context, principal string, res notes.Result) {
|
||||||
|
// Path and hashes only. The notification leaves the cluster for a phone
|
||||||
|
// via a third-party topic; a hash identifies the rewrite, the bytes would
|
||||||
|
// leak whatever was written into an instruction file.
|
||||||
|
msg := fmt.Sprintf("brain_edit_root applied\npath=%s\nprincipal=%s\nbefore=%s\nafter=%s",
|
||||||
|
res.Path, principal, res.PrevContentHash, res.ContentHash)
|
||||||
|
|
||||||
|
if s.govNotify != nil {
|
||||||
|
if err := s.govNotify.Notify(ctx, msg); err != nil {
|
||||||
|
slog.Error("brain_edit_root notify failed", "path", res.Path, "err", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if s.govAudit == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// Internal, not Confidential: a governance file is not client data, and
|
||||||
|
// Confidential would make DegradingSink REFUSE to record while loki is
|
||||||
|
// down rather than buffer — losing exactly the record that matters most.
|
||||||
|
outcome, err := s.govAudit.Reserve(ctx, classification.Internal)
|
||||||
|
if err != nil {
|
||||||
|
slog.Error("brain_edit_root audit reserve failed", "path", res.Path, "err", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
entry := capture.AuditEntry{
|
||||||
|
Timestamp: time.Now().UTC(),
|
||||||
|
Principal: principal,
|
||||||
|
Actor: "brain_edit_root",
|
||||||
|
Harness: "mcp",
|
||||||
|
EffectiveClassification: classification.Internal.String(),
|
||||||
|
Items: []string{res.Path},
|
||||||
|
SecurityEvents: []string{fmt.Sprintf(
|
||||||
|
"governance file rewritten: %s before=%s after=%s",
|
||||||
|
res.Path, res.PrevContentHash, res.ContentHash)},
|
||||||
|
}
|
||||||
|
if err := s.govAudit.Record(ctx, entry, outcome); err != nil {
|
||||||
|
slog.Error("brain_edit_root audit record failed", "path", res.Path, "err", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Server) editWithScope(args json.RawMessage, scope notes.Scope) (notes.Result, json.RawMessage, error) {
|
||||||
|
var a brainEditArgs
|
||||||
|
if err := json.Unmarshal(args, &a); err != nil {
|
||||||
|
return notes.Result{}, nil, fmt.Errorf("parse args: %w", err)
|
||||||
|
}
|
||||||
|
res, err := notes.Edit(s.brainDir, a.Path, a.OldStr, a.NewStr, scope)
|
||||||
|
if err != nil {
|
||||||
|
return notes.Result{}, nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Reindex only when the edit could have changed what the index reads.
|
||||||
|
// BuildWingIndex consumes title/created_at from the frontmatter block;
|
||||||
|
// BM25 walks the filesystem live and embeddings are mtime-driven, so an
|
||||||
|
// unconditional rebuild would buy nothing and dirty a second file —
|
||||||
|
// _index.md being precisely the class brain-sync already needs a bespoke
|
||||||
|
// conflict resolver for.
|
||||||
|
if wing := wingOf(res.Path); wing != "" && touchesFrontmatter(a.OldStr, a.NewStr) {
|
||||||
|
if ierr := brain.BuildWingIndex(s.brainDir, wing); ierr != nil {
|
||||||
|
// Best-effort, matching every other index hook here: the write
|
||||||
|
// already landed and will be committed, so failing the call would
|
||||||
|
// make the caller retry an edit whose old_str no longer exists.
|
||||||
|
res.ReindexError = ierr.Error()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out, merr := json.Marshal(res)
|
||||||
|
return res, out, merr
|
||||||
|
}
|
||||||
|
|
||||||
|
// wingOf returns the wing for a wiki path, or "" for knowledge/ and root.
|
||||||
|
func wingOf(rel string) string {
|
||||||
|
parts := strings.Split(rel, "/")
|
||||||
|
if len(parts) < 3 || parts[0] != "wiki" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return parts[1]
|
||||||
|
}
|
||||||
|
|
||||||
|
// touchesFrontmatter reports whether either side of the edit mentions a field
|
||||||
|
// the wing index reads.
|
||||||
|
func touchesFrontmatter(oldStr, newStr string) bool {
|
||||||
|
for _, s := range []string{oldStr, newStr} {
|
||||||
|
if strings.Contains(s, "title:") || strings.Contains(s, "created_at:") || strings.Contains(s, "---") {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
@@ -0,0 +1,297 @@
|
|||||||
|
// ingestion/internal/mcp/tools_notes_test.go
|
||||||
|
package mcp_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"encoding/json"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/mathiasbq/hyperguild/ingestion/internal/audit"
|
||||||
|
"github.com/mathiasbq/hyperguild/ingestion/internal/brainstore"
|
||||||
|
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
|
||||||
|
"github.com/mathiasbq/hyperguild/ingestion/internal/capturehttp"
|
||||||
|
"github.com/mathiasbq/hyperguild/ingestion/internal/classification"
|
||||||
|
"github.com/mathiasbq/hyperguild/ingestion/internal/mcp"
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ---- brain#4: brain_read / brain_edit through the JSON-RPC surface --------
|
||||||
|
|
||||||
|
func newBrainRepo(t *testing.T) string {
|
||||||
|
t.Helper()
|
||||||
|
dir := t.TempDir()
|
||||||
|
require.NoError(t, os.MkdirAll(filepath.Join(dir, ".git"), 0o755))
|
||||||
|
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki/homelab/facts"), 0o755))
|
||||||
|
require.NoError(t, os.WriteFile(filepath.Join(dir, "wiki/homelab/facts/n.md"),
|
||||||
|
[]byte("---\ntitle: 'n'\n---\n\nalpha\nbeta\n"), 0o644))
|
||||||
|
require.NoError(t, os.WriteFile(filepath.Join(dir, "CLAUDE.md"), []byte("# rules\n"), 0o644))
|
||||||
|
return dir
|
||||||
|
}
|
||||||
|
|
||||||
|
func call(t *testing.T, srv *mcp.Server, name string, args map[string]any) map[string]any {
|
||||||
|
t.Helper()
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/mcp", body(t, map[string]any{
|
||||||
|
"jsonrpc": "2.0", "id": 1, "method": "tools/call",
|
||||||
|
"params": map[string]any{"name": name, "arguments": args},
|
||||||
|
}))
|
||||||
|
rr := httptest.NewRecorder()
|
||||||
|
srv.ServeHTTP(rr, req)
|
||||||
|
var resp map[string]any
|
||||||
|
require.NoError(t, json.Unmarshal(rr.Body.Bytes(), &resp))
|
||||||
|
return resp
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBrainReadAndEdit_RoundTripThroughJSONRPC(t *testing.T) {
|
||||||
|
// The Go functions are tested in internal/notes; this pins the transport,
|
||||||
|
// because a tool that works in-process and is unreachable over JSON-RPC
|
||||||
|
// is indistinguishable from a working one in a unit test.
|
||||||
|
dir := newBrainRepo(t)
|
||||||
|
srv := mcp.NewServer(dir, nil, nil, nil)
|
||||||
|
|
||||||
|
resp := call(t, srv, "brain_read", map[string]any{"path": "wiki/homelab/facts/n.md"})
|
||||||
|
require.NotContains(t, resp, "error", "%v", resp["error"])
|
||||||
|
|
||||||
|
resp = call(t, srv, "brain_edit", map[string]any{
|
||||||
|
"path": "wiki/homelab/facts/n.md", "old_str": "beta", "new_str": "BETA",
|
||||||
|
})
|
||||||
|
require.NotContains(t, resp, "error", "%v", resp["error"])
|
||||||
|
|
||||||
|
got, err := os.ReadFile(filepath.Join(dir, "wiki/homelab/facts/n.md"))
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, "---\ntitle: 'n'\n---\n\nalpha\nBETA\n", string(got),
|
||||||
|
"only the matched bytes may change")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBrainEdit_RefusesGovernanceFilesFromTheModelFacingTool(t *testing.T) {
|
||||||
|
dir := newBrainRepo(t)
|
||||||
|
srv := mcp.NewServer(dir, nil, nil, nil)
|
||||||
|
|
||||||
|
resp := call(t, srv, "brain_edit", map[string]any{
|
||||||
|
"path": "CLAUDE.md", "old_str": "# rules", "new_str": "# rewritten",
|
||||||
|
})
|
||||||
|
require.Contains(t, resp, "error", "brain_edit must not reach CLAUDE.md")
|
||||||
|
|
||||||
|
got, err := os.ReadFile(filepath.Join(dir, "CLAUDE.md"))
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, "# rules\n", string(got))
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBrainEditRoot_RefusedWithoutTheLocalStaticPrincipal(t *testing.T) {
|
||||||
|
// No principal in context is the model-facing case (and the default in
|
||||||
|
// this test server): the privileged door must stay shut.
|
||||||
|
dir := newBrainRepo(t)
|
||||||
|
srv := mcp.NewServer(dir, nil, nil, nil)
|
||||||
|
|
||||||
|
resp := call(t, srv, "brain_edit_root", map[string]any{
|
||||||
|
"path": "CLAUDE.md", "old_str": "# rules", "new_str": "# rewritten",
|
||||||
|
})
|
||||||
|
require.Contains(t, resp, "error")
|
||||||
|
|
||||||
|
got, err := os.ReadFile(filepath.Join(dir, "CLAUDE.md"))
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, "# rules\n", string(got), "governance file must be untouched")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBrainGet_CannotReachSessionTranscripts(t *testing.T) {
|
||||||
|
// Pre-existing hole, not introduced here: api.ReadNote resolves any
|
||||||
|
// brain-relative path and parseFrontmatter returns the WHOLE file as body
|
||||||
|
// when there is no frontmatter fence. archive/ holds verbatim Claude
|
||||||
|
// session transcripts (up to 500 KB) that BM25 never indexes, so they are
|
||||||
|
// invisible to search and trivially readable by path — and CLAUDE.md
|
||||||
|
// documents the layout.
|
||||||
|
dir := newBrainRepo(t)
|
||||||
|
require.NoError(t, os.MkdirAll(filepath.Join(dir, "archive/claude-sessions/koala"), 0o755))
|
||||||
|
require.NoError(t, os.WriteFile(
|
||||||
|
filepath.Join(dir, "archive/claude-sessions/koala/session-1.md"),
|
||||||
|
[]byte("verbatim transcript with tool output\n"), 0o644))
|
||||||
|
|
||||||
|
srv := mcp.NewServer(dir, nil, nil, nil)
|
||||||
|
resp := call(t, srv, "brain_get", map[string]any{"path": "archive/claude-sessions/koala/session-1.md"})
|
||||||
|
|
||||||
|
require.Contains(t, resp, "error", "brain_get must not read outside knowledge/ and wiki/")
|
||||||
|
assert.NotContains(t, string(mustJSON(t, resp)), "verbatim transcript")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBrainGet_StillReadsNotes(t *testing.T) {
|
||||||
|
dir := newBrainRepo(t)
|
||||||
|
srv := mcp.NewServer(dir, nil, nil, nil)
|
||||||
|
|
||||||
|
resp := call(t, srv, "brain_get", map[string]any{"path": "wiki/homelab/facts/n.md"})
|
||||||
|
assert.NotContains(t, resp, "error", "%v", resp["error"])
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustJSON(t *testing.T, v any) []byte {
|
||||||
|
t.Helper()
|
||||||
|
b, err := json.Marshal(v)
|
||||||
|
require.NoError(t, err)
|
||||||
|
return b
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- hyperguild#97: a governance rewrite must be observed, not just gated ----
|
||||||
|
|
||||||
|
// govSink captures audit records. Governance edits are rare, so the whole
|
||||||
|
// history is kept and asserted on rather than sampled.
|
||||||
|
type govSink struct {
|
||||||
|
entries []capture.AuditEntry
|
||||||
|
outcomes []capture.AuditOutcome
|
||||||
|
recErr error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *govSink) Reserve(context.Context, classification.Level) (capture.AuditOutcome, error) {
|
||||||
|
return capture.AuditCentral, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *govSink) Record(_ context.Context, e capture.AuditEntry, o capture.AuditOutcome) error {
|
||||||
|
s.entries = append(s.entries, e)
|
||||||
|
s.outcomes = append(s.outcomes, o)
|
||||||
|
return s.recErr
|
||||||
|
}
|
||||||
|
|
||||||
|
type govNotifier struct {
|
||||||
|
msgs []string
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (n *govNotifier) Notify(_ context.Context, msg string) error {
|
||||||
|
n.msgs = append(n.msgs, msg)
|
||||||
|
return n.err
|
||||||
|
}
|
||||||
|
|
||||||
|
// govServer wires capture (which is what re-derives the static principal) plus
|
||||||
|
// the #97 observer, and returns the brain dir so the test can read the file.
|
||||||
|
func govServer(t *testing.T) (*mcp.Server, string, *govSink, *govNotifier) {
|
||||||
|
t.Helper()
|
||||||
|
dir := newBrainRepo(t)
|
||||||
|
cfg, err := classification.Load(dir)
|
||||||
|
require.NoError(t, err)
|
||||||
|
svc := capture.NewService(brainstore.New(dir), capFakeTracker{}, cfg, audit.NewSlogSink(nil))
|
||||||
|
|
||||||
|
srv := mcp.NewServer(dir, nil, nil, nil)
|
||||||
|
srv.WithCapture(svc, nil, capStaticTok, "local-cli", capturehttp.NewOriginResolver(nil))
|
||||||
|
sink, notifier := &govSink{}, &govNotifier{}
|
||||||
|
srv.WithGovernanceObserver(sink, notifier)
|
||||||
|
return srv, dir, sink, notifier
|
||||||
|
}
|
||||||
|
|
||||||
|
// authCall is call() with an Authorization header, so the static principal is
|
||||||
|
// present and brain_edit_root's gate opens.
|
||||||
|
func authCall(t *testing.T, srv *mcp.Server, authz, name string, args map[string]any) map[string]any {
|
||||||
|
t.Helper()
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/mcp", body(t, map[string]any{
|
||||||
|
"jsonrpc": "2.0", "id": 1, "method": "tools/call",
|
||||||
|
"params": map[string]any{"name": name, "arguments": args},
|
||||||
|
}))
|
||||||
|
if authz != "" {
|
||||||
|
req.Header.Set("Authorization", authz)
|
||||||
|
}
|
||||||
|
rr := httptest.NewRecorder()
|
||||||
|
srv.ServeHTTP(rr, req)
|
||||||
|
var resp map[string]any
|
||||||
|
require.NoError(t, json.Unmarshal(rr.Body.Bytes(), &resp))
|
||||||
|
return resp
|
||||||
|
}
|
||||||
|
|
||||||
|
func sha256hex(s string) string {
|
||||||
|
sum := sha256.Sum256([]byte(s))
|
||||||
|
return hex.EncodeToString(sum[:])
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBrainEditRoot_RecordsAnAuditEntryNamingPrincipalPathAndBothHashes(t *testing.T) {
|
||||||
|
// The gate proves a rewrite was ALLOWED. Only a record proves one HAPPENED
|
||||||
|
// — and brain-sync will commit it within 15 minutes authored `brain-sync`,
|
||||||
|
// so git history cannot tell it apart from routine note traffic.
|
||||||
|
srv, dir, sink, _ := govServer(t)
|
||||||
|
|
||||||
|
resp := authCall(t, srv, "Bearer "+capStaticTok, "brain_edit_root", map[string]any{
|
||||||
|
"path": "CLAUDE.md", "old_str": "# rules", "new_str": "# rewritten",
|
||||||
|
})
|
||||||
|
require.NotContains(t, resp, "error", "%v", resp["error"])
|
||||||
|
|
||||||
|
require.Len(t, sink.entries, 1, "exactly one audit record per governance rewrite")
|
||||||
|
e := sink.entries[0]
|
||||||
|
assert.Equal(t, "local-cli", e.Principal)
|
||||||
|
assert.Contains(t, e.Items, "CLAUDE.md", "the record must name what was rewritten")
|
||||||
|
|
||||||
|
joined := strings.Join(e.SecurityEvents, " ")
|
||||||
|
assert.Contains(t, joined, sha256hex("# rules\n"), "before-hash must be recorded")
|
||||||
|
assert.Contains(t, joined, sha256hex("# rewritten\n"), "after-hash must be recorded")
|
||||||
|
assert.NotZero(t, e.Timestamp)
|
||||||
|
|
||||||
|
got, err := os.ReadFile(filepath.Join(dir, "CLAUDE.md"))
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, "# rewritten\n", string(got))
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBrainEditRoot_NotifiesWithPathAndHashesButNeverContent(t *testing.T) {
|
||||||
|
srv, _, _, notifier := govServer(t)
|
||||||
|
|
||||||
|
resp := authCall(t, srv, "Bearer "+capStaticTok, "brain_edit_root", map[string]any{
|
||||||
|
"path": "CLAUDE.md", "old_str": "# rules", "new_str": "# rewritten",
|
||||||
|
})
|
||||||
|
require.NotContains(t, resp, "error", "%v", resp["error"])
|
||||||
|
|
||||||
|
require.Len(t, notifier.msgs, 1)
|
||||||
|
msg := notifier.msgs[0]
|
||||||
|
assert.Contains(t, msg, "CLAUDE.md")
|
||||||
|
assert.Contains(t, msg, "local-cli")
|
||||||
|
assert.Contains(t, msg, sha256hex("# rules\n"))
|
||||||
|
assert.Contains(t, msg, sha256hex("# rewritten\n"))
|
||||||
|
|
||||||
|
// The notification travels to a phone through a third-party topic. A hash
|
||||||
|
// identifies a rewrite; the bytes would leak whatever was written.
|
||||||
|
assert.NotContains(t, msg, "# rewritten", "no file content in the notification body")
|
||||||
|
assert.NotContains(t, msg, "# rules")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBrainEditRoot_RefusedProducesNoRecordNoNotificationNoWrite(t *testing.T) {
|
||||||
|
srv, dir, sink, notifier := govServer(t)
|
||||||
|
|
||||||
|
// No Authorization header → no static principal → the gate stays shut.
|
||||||
|
resp := authCall(t, srv, "", "brain_edit_root", map[string]any{
|
||||||
|
"path": "CLAUDE.md", "old_str": "# rules", "new_str": "# rewritten",
|
||||||
|
})
|
||||||
|
require.Contains(t, resp, "error")
|
||||||
|
|
||||||
|
assert.Empty(t, sink.entries, "a refusal is not a rewrite — recording one would cry wolf")
|
||||||
|
assert.Empty(t, notifier.msgs)
|
||||||
|
got, err := os.ReadFile(filepath.Join(dir, "CLAUDE.md"))
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, "# rules\n", string(got))
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBrainEditRoot_FailedEditProducesNoRecordNoNotification(t *testing.T) {
|
||||||
|
// Authorised but no match: the door opened and nothing went through it.
|
||||||
|
srv, _, sink, notifier := govServer(t)
|
||||||
|
|
||||||
|
resp := authCall(t, srv, "Bearer "+capStaticTok, "brain_edit_root", map[string]any{
|
||||||
|
"path": "CLAUDE.md", "old_str": "absent-from-the-file", "new_str": "x",
|
||||||
|
})
|
||||||
|
require.Contains(t, resp, "error")
|
||||||
|
|
||||||
|
assert.Empty(t, sink.entries)
|
||||||
|
assert.Empty(t, notifier.msgs)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBrainEdit_OnOrdinaryNotesIsNotAudited(t *testing.T) {
|
||||||
|
// The decided asymmetry (#97): note edits are covered by git history, which
|
||||||
|
// carries the same before/after bytes with a diff. Governance files get the
|
||||||
|
// extra trail because they change what later agents obey, not because they
|
||||||
|
// are harder to reconstruct.
|
||||||
|
srv, _, sink, notifier := govServer(t)
|
||||||
|
|
||||||
|
resp := authCall(t, srv, "Bearer "+capStaticTok, "brain_edit", map[string]any{
|
||||||
|
"path": "wiki/homelab/facts/n.md", "old_str": "beta", "new_str": "BETA",
|
||||||
|
})
|
||||||
|
require.NotContains(t, resp, "error", "%v", resp["error"])
|
||||||
|
|
||||||
|
assert.Empty(t, sink.entries, "note edits must not fill the governance audit trail")
|
||||||
|
assert.Empty(t, notifier.msgs)
|
||||||
|
}
|
||||||
@@ -0,0 +1,282 @@
|
|||||||
|
package notes
|
||||||
|
|
||||||
|
import (
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"syscall"
|
||||||
|
"unicode/utf8"
|
||||||
|
)
|
||||||
|
|
||||||
|
// maxNoteBytes caps what Read will hand back. Generous for knowledge/ and
|
||||||
|
// wiki/ (the largest note today is far under it) and small enough that a
|
||||||
|
// pathological file cannot flood a caller's context.
|
||||||
|
const maxNoteBytes = 256 << 10
|
||||||
|
|
||||||
|
// Errors callers distinguish.
|
||||||
|
var (
|
||||||
|
ErrNoMatch = errors.New("notes: old_str not found")
|
||||||
|
ErrAmbiguous = errors.New("notes: old_str matches more than once")
|
||||||
|
ErrNotUTF8 = errors.New("notes: content is not valid UTF-8")
|
||||||
|
ErrGitBusy = errors.New("notes: a git operation is in progress")
|
||||||
|
ErrTooLarge = errors.New("notes: file exceeds the readable size limit")
|
||||||
|
ErrDegenerate = errors.New("notes: edit is degenerate")
|
||||||
|
ErrInConflict = errors.New("notes: file contains unresolved conflict markers")
|
||||||
|
)
|
||||||
|
|
||||||
|
// pathLocks serialises edits per file within this process. The flock below
|
||||||
|
// covers the cross-process case (two pods share the hostPath during a rolling
|
||||||
|
// update, and brain-fix/brain-enrich write the same tree).
|
||||||
|
var pathLocks sync.Map // string -> *sync.Mutex
|
||||||
|
|
||||||
|
// Result is what a successful edit reports.
|
||||||
|
type Result struct {
|
||||||
|
Path string `json:"path"`
|
||||||
|
// PrevContentHash is the sha256 of the file as it was before this edit.
|
||||||
|
// Paired with ContentHash it identifies a rewrite in an audit trail or an
|
||||||
|
// alert without carrying any of the bytes (hyperguild#97).
|
||||||
|
PrevContentHash string `json:"prev_content_hash"`
|
||||||
|
ContentHash string `json:"content_hash"`
|
||||||
|
BytesBefore int `json:"bytes_before"`
|
||||||
|
BytesAfter int `json:"bytes_after"`
|
||||||
|
ReindexError string `json:"reindex_error,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Read returns a note's bytes verbatim.
|
||||||
|
//
|
||||||
|
// No frontmatter parsing, no excerpting, no normalisation: the entire purpose
|
||||||
|
// is to give a caller an exact basis for an edit. An excerpt from brain_query
|
||||||
|
// would be truncated and normalised, and every old_str built from one would
|
||||||
|
// miss.
|
||||||
|
func Read(brainDir, path string) ([]byte, error) {
|
||||||
|
t, err := Resolve(brainDir, path, ScopeNotes)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return readContained(t)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ReadScoped is Read for a caller already holding the privileged scope.
|
||||||
|
func ReadScoped(brainDir, path string, scope Scope) ([]byte, error) {
|
||||||
|
t, err := Resolve(brainDir, path, scope)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return readContained(t)
|
||||||
|
}
|
||||||
|
|
||||||
|
func readContained(t Target) ([]byte, error) {
|
||||||
|
fi, err := os.Stat(t.Abs)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("notes: stat %s: %w", t.Rel, err)
|
||||||
|
}
|
||||||
|
if fi.Size() > maxNoteBytes {
|
||||||
|
return nil, fmt.Errorf("%w: %s is %d bytes (limit %d)", ErrTooLarge, t.Rel, fi.Size(), maxNoteBytes)
|
||||||
|
}
|
||||||
|
b, err := os.ReadFile(t.Abs)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("notes: read %s: %w", t.Rel, err)
|
||||||
|
}
|
||||||
|
// The MCP response marshals this into a JSON string, and encoding/json
|
||||||
|
// substitutes U+FFFD for invalid UTF-8 — silently returning different
|
||||||
|
// bytes than the file holds, which defeats "verbatim" entirely.
|
||||||
|
if !utf8.Valid(b) {
|
||||||
|
return nil, fmt.Errorf("%w: %s", ErrNotUTF8, t.Rel)
|
||||||
|
}
|
||||||
|
return b, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Edit replaces exactly one occurrence of oldStr with newStr, in place.
|
||||||
|
//
|
||||||
|
// Fails loudly and writes nothing on: zero matches, more than one match
|
||||||
|
// (counted with overlap), degenerate arguments, invalid UTF-8, a git operation
|
||||||
|
// in flight, or unresolved conflict markers in the target.
|
||||||
|
func Edit(brainDir, path, oldStr, newStr string, scope Scope) (Result, error) {
|
||||||
|
t, err := Resolve(brainDir, path, scope)
|
||||||
|
if err != nil {
|
||||||
|
return Result{}, err
|
||||||
|
}
|
||||||
|
if oldStr == "" {
|
||||||
|
return Result{}, fmt.Errorf("%w: old_str is empty", ErrDegenerate)
|
||||||
|
}
|
||||||
|
if oldStr == newStr {
|
||||||
|
return Result{}, fmt.Errorf("%w: old_str equals new_str", ErrDegenerate)
|
||||||
|
}
|
||||||
|
if !utf8.ValidString(newStr) {
|
||||||
|
return Result{}, fmt.Errorf("%w: new_str", ErrNotUTF8)
|
||||||
|
}
|
||||||
|
// brain#2 guard: refuse to write to a tree the sync does not commit from.
|
||||||
|
if err := assertSyncSource(brainDir); err != nil {
|
||||||
|
return Result{}, err
|
||||||
|
}
|
||||||
|
if err := assertGitIdle(brainDir); err != nil {
|
||||||
|
return Result{}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
mu := lockFor(t.Abs)
|
||||||
|
mu.Lock()
|
||||||
|
defer mu.Unlock()
|
||||||
|
|
||||||
|
unlock, err := flockFile(t.Abs)
|
||||||
|
if err != nil {
|
||||||
|
return Result{}, err
|
||||||
|
}
|
||||||
|
defer unlock()
|
||||||
|
|
||||||
|
// Read INSIDE the lock: the match count must be evaluated against the
|
||||||
|
// bytes we are about to overwrite, or a concurrent edit is silently lost
|
||||||
|
// while both callers are told they succeeded.
|
||||||
|
before, err := readContained(t)
|
||||||
|
if err != nil {
|
||||||
|
return Result{}, err
|
||||||
|
}
|
||||||
|
if hasConflictMarkers(before) {
|
||||||
|
return Result{}, fmt.Errorf("%w: %s", ErrInConflict, t.Rel)
|
||||||
|
}
|
||||||
|
|
||||||
|
switch n := countOverlapping(string(before), oldStr); {
|
||||||
|
case n == 0:
|
||||||
|
return Result{}, fmt.Errorf("%w in %s (0 matches)%s", ErrNoMatch, t.Rel, crlfHint(string(before), oldStr))
|
||||||
|
case n > 1:
|
||||||
|
return Result{}, fmt.Errorf("%w in %s (%d matches) — extend old_str until it is unique",
|
||||||
|
ErrAmbiguous, t.Rel, n)
|
||||||
|
}
|
||||||
|
|
||||||
|
after := strings.Replace(string(before), oldStr, newStr, 1)
|
||||||
|
if err := writeAtomic(t.Abs, []byte(after)); err != nil {
|
||||||
|
return Result{}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
sum := sha256.Sum256([]byte(after))
|
||||||
|
prev := sha256.Sum256(before)
|
||||||
|
return Result{
|
||||||
|
Path: t.Rel,
|
||||||
|
PrevContentHash: hex.EncodeToString(prev[:]),
|
||||||
|
ContentHash: hex.EncodeToString(sum[:]),
|
||||||
|
BytesBefore: len(before),
|
||||||
|
BytesAfter: len(after),
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// countOverlapping counts occurrences allowing overlap. bytes.Count does not:
|
||||||
|
// "aa" occurs once in "aaa" by its reckoning, so a Count-based uniqueness
|
||||||
|
// check accepts an ambiguous edit as unique.
|
||||||
|
func countOverlapping(s, sub string) int {
|
||||||
|
var n, from int
|
||||||
|
for {
|
||||||
|
i := strings.Index(s[from:], sub)
|
||||||
|
if i < 0 {
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
n++
|
||||||
|
from += i + 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// crlfHint reports whether the caller's old_str would have matched if line
|
||||||
|
// endings agreed. It never normalises — that would break byte-identity — but
|
||||||
|
// an opaque "no match" is unfixable from the caller's side, and a stripped \r
|
||||||
|
// is the likeliest cause.
|
||||||
|
func crlfHint(content, oldStr string) string {
|
||||||
|
if strings.Contains(content, "\r\n") && !strings.Contains(oldStr, "\r\n") {
|
||||||
|
if strings.Contains(strings.ReplaceAll(content, "\r\n", "\n"), oldStr) {
|
||||||
|
return " — the file uses CRLF line endings and old_str does not; include the \\r"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
func hasConflictMarkers(b []byte) bool {
|
||||||
|
s := string(b)
|
||||||
|
return strings.Contains(s, "\n<<<<<<< ") || strings.HasPrefix(s, "<<<<<<< ")
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertGitIdle refuses while brain-sync (or brain-fix, or a human) has a git
|
||||||
|
// operation in flight. An edit landing between `git merge` and its abort is
|
||||||
|
// either hard-reset away or committed as a wrongly resolved conflict — and the
|
||||||
|
// caller was told it succeeded.
|
||||||
|
func assertGitIdle(brainDir string) error {
|
||||||
|
for _, marker := range []string{"MERGE_HEAD", "REBASE_HEAD", "CHERRY_PICK_HEAD", "index.lock"} {
|
||||||
|
if _, err := os.Stat(filepath.Join(brainDir, ".git", marker)); err == nil {
|
||||||
|
return fmt.Errorf("%w (.git/%s present) — retry after the sync settles", ErrGitBusy, marker)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func lockFor(abs string) *sync.Mutex {
|
||||||
|
v, _ := pathLocks.LoadOrStore(abs, &sync.Mutex{})
|
||||||
|
return v.(*sync.Mutex)
|
||||||
|
}
|
||||||
|
|
||||||
|
// flockFile takes an advisory lock covering other processes sharing the
|
||||||
|
// hostPath — during a RollingUpdate two ingestion pods do.
|
||||||
|
func flockFile(abs string) (func(), error) {
|
||||||
|
f, err := os.OpenFile(abs+".lock", os.O_CREATE|os.O_RDWR, 0o644)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("notes: lock %s: %w", filepath.Base(abs), err)
|
||||||
|
}
|
||||||
|
if err := syscall.Flock(int(f.Fd()), syscall.LOCK_EX); err != nil {
|
||||||
|
_ = f.Close()
|
||||||
|
return nil, fmt.Errorf("notes: flock %s: %w", filepath.Base(abs), err)
|
||||||
|
}
|
||||||
|
return func() {
|
||||||
|
_ = syscall.Flock(int(f.Fd()), syscall.LOCK_UN)
|
||||||
|
_ = f.Close()
|
||||||
|
_ = os.Remove(abs + ".lock")
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// writeAtomic replaces the file via a same-directory temp + rename, then
|
||||||
|
// fsyncs the directory so the rename survives a node crash.
|
||||||
|
//
|
||||||
|
// Same directory because a rename across the hostPath boundary fails with
|
||||||
|
// EXDEV; the ".tmp" suffix because the tree's .gitignore already covers it, so
|
||||||
|
// a leaked temp is not committed by `git add -A`.
|
||||||
|
func writeAtomic(abs string, content []byte) error {
|
||||||
|
dir := filepath.Dir(abs)
|
||||||
|
tmp, err := os.CreateTemp(dir, filepath.Base(abs)+".*.tmp")
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("notes: temp file: %w", err)
|
||||||
|
}
|
||||||
|
tmpName := tmp.Name()
|
||||||
|
defer func() { _ = os.Remove(tmpName) }() // no-op once renamed
|
||||||
|
|
||||||
|
if _, err := tmp.Write(content); err != nil {
|
||||||
|
_ = tmp.Close()
|
||||||
|
return fmt.Errorf("notes: write temp: %w", err)
|
||||||
|
}
|
||||||
|
if err := tmp.Sync(); err != nil {
|
||||||
|
_ = tmp.Close()
|
||||||
|
return fmt.Errorf("notes: sync temp: %w", err)
|
||||||
|
}
|
||||||
|
if err := tmp.Close(); err != nil {
|
||||||
|
return fmt.Errorf("notes: close temp: %w", err)
|
||||||
|
}
|
||||||
|
// CreateTemp makes 0600; every other writer of this tree uses 0644 and
|
||||||
|
// other uids read it.
|
||||||
|
if err := os.Chmod(tmpName, 0o644); err != nil {
|
||||||
|
return fmt.Errorf("notes: chmod temp: %w", err)
|
||||||
|
}
|
||||||
|
if err := os.Rename(tmpName, abs); err != nil {
|
||||||
|
return fmt.Errorf("notes: rename: %w", err)
|
||||||
|
}
|
||||||
|
return fsyncDir(dir)
|
||||||
|
}
|
||||||
|
|
||||||
|
func fsyncDir(dir string) error {
|
||||||
|
d, err := os.Open(dir)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("notes: open dir: %w", err)
|
||||||
|
}
|
||||||
|
defer func() { _ = d.Close() }()
|
||||||
|
if err := d.Sync(); err != nil {
|
||||||
|
return fmt.Errorf("notes: fsync dir: %w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,256 @@
|
|||||||
|
package notes
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func seed(t *testing.T, dir, rel, content string) string {
|
||||||
|
t.Helper()
|
||||||
|
abs := filepath.Join(dir, rel)
|
||||||
|
if err := os.MkdirAll(filepath.Dir(abs), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(abs, []byte(content), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return abs
|
||||||
|
}
|
||||||
|
|
||||||
|
func read(t *testing.T, abs string) string {
|
||||||
|
t.Helper()
|
||||||
|
b, err := os.ReadFile(abs)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return string(b)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRead_ReturnsVerbatimBytes(t *testing.T) {
|
||||||
|
dir := newBrain(t)
|
||||||
|
const body = "---\ntitle: 'x'\n---\n\nbody with double spaces\r\nand a CRLF line\nno trailing newline"
|
||||||
|
seed(t, dir, "knowledge/verbatim.md", body)
|
||||||
|
|
||||||
|
got, err := Read(dir, "knowledge/verbatim.md")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Read: %v", err)
|
||||||
|
}
|
||||||
|
if string(got) != body {
|
||||||
|
t.Errorf("Read returned %q, want the file byte-for-byte", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRead_RefusesNonUTF8RatherThanCorruptingIt(t *testing.T) {
|
||||||
|
// The response is marshalled into a JSON string, and encoding/json
|
||||||
|
// replaces invalid UTF-8 with U+FFFD. Returning it would mean every
|
||||||
|
// old_str the caller derives from that region can never match, and
|
||||||
|
// pasting it back would write the corruption to disk permanently.
|
||||||
|
dir := newBrain(t)
|
||||||
|
seed(t, dir, "knowledge/bad.md", "ok \xff\xfe broken\n")
|
||||||
|
|
||||||
|
if _, err := Read(dir, "knowledge/bad.md"); err == nil {
|
||||||
|
t.Fatal("Read accepted invalid UTF-8; it must refuse rather than silently substitute")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEdit_ReplacesExactlyOnceAndLeavesEveryOtherByteAlone(t *testing.T) {
|
||||||
|
dir := newBrain(t)
|
||||||
|
const body = "---\ntitle: 'x'\n---\n\nalpha\nbeta\ngamma\n"
|
||||||
|
abs := seed(t, dir, "knowledge/n.md", body)
|
||||||
|
|
||||||
|
res, err := Edit(dir, "knowledge/n.md", "beta", "BETA", ScopeNotes)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Edit: %v", err)
|
||||||
|
}
|
||||||
|
want := strings.Replace(body, "beta", "BETA", 1)
|
||||||
|
if got := read(t, abs); got != want {
|
||||||
|
t.Errorf("file = %q, want %q", got, want)
|
||||||
|
}
|
||||||
|
if res.ContentHash == "" {
|
||||||
|
t.Error("result should carry a content hash for read-after-write confirmation")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEdit_RefusesAmbiguousAndAbsentMatchesWithoutWriting(t *testing.T) {
|
||||||
|
dir := newBrain(t)
|
||||||
|
const body = "alpha\nalpha\n"
|
||||||
|
abs := seed(t, dir, "knowledge/n.md", body)
|
||||||
|
|
||||||
|
cases := []struct{ name, old, want string }{
|
||||||
|
{"two matches", "alpha", "2"},
|
||||||
|
{"no match", "delta", "0"},
|
||||||
|
}
|
||||||
|
for _, tc := range cases {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
_, err := Edit(dir, "knowledge/n.md", tc.old, "x", ScopeNotes)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("want a loud refusal")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), tc.want) {
|
||||||
|
t.Errorf("error must state the match count, got %v", err)
|
||||||
|
}
|
||||||
|
if got := read(t, abs); got != body {
|
||||||
|
t.Errorf("file changed on a refused edit: %q", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEdit_CountsOverlappingMatches(t *testing.T) {
|
||||||
|
// bytes.Count counts NON-overlapping occurrences: "aa" in "aaa" is 1, so
|
||||||
|
// a Count-based uniqueness check calls an ambiguous edit unique.
|
||||||
|
dir := newBrain(t)
|
||||||
|
seed(t, dir, "knowledge/n.md", "aaa\n")
|
||||||
|
|
||||||
|
if _, err := Edit(dir, "knowledge/n.md", "aa", "b", ScopeNotes); err == nil {
|
||||||
|
t.Fatal("overlapping matches must be reported as ambiguous")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEdit_RejectsDegenerateArguments(t *testing.T) {
|
||||||
|
dir := newBrain(t)
|
||||||
|
seed(t, dir, "knowledge/n.md", "alpha\n")
|
||||||
|
|
||||||
|
if _, err := Edit(dir, "knowledge/n.md", "", "x", ScopeNotes); err == nil {
|
||||||
|
t.Error("empty old_str must be rejected")
|
||||||
|
}
|
||||||
|
if _, err := Edit(dir, "knowledge/n.md", "alpha", "alpha", ScopeNotes); err == nil {
|
||||||
|
t.Error("a no-op edit must be rejected rather than burning a sync cycle")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEdit_DeletionIsLegal(t *testing.T) {
|
||||||
|
dir := newBrain(t)
|
||||||
|
abs := seed(t, dir, "knowledge/n.md", "keep\ndrop\n")
|
||||||
|
|
||||||
|
if _, err := Edit(dir, "knowledge/n.md", "drop\n", "", ScopeNotes); err != nil {
|
||||||
|
t.Fatalf("Edit: %v", err)
|
||||||
|
}
|
||||||
|
if got := read(t, abs); got != "keep\n" {
|
||||||
|
t.Errorf("file = %q, want %q", got, "keep\n")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEdit_ReportsWhenOnlyLineEndingsDiffer(t *testing.T) {
|
||||||
|
// The likeliest real failure: the caller's old_str lost its \r. Never
|
||||||
|
// normalise silently — that breaks byte-identity — but say so, because an
|
||||||
|
// opaque "no match" is unfixable from the caller's side.
|
||||||
|
dir := newBrain(t)
|
||||||
|
seed(t, dir, "knowledge/n.md", "alpha\r\nbeta\r\n")
|
||||||
|
|
||||||
|
_, err := Edit(dir, "knowledge/n.md", "alpha\nbeta", "x", ScopeNotes)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("want a refusal")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "line ending") {
|
||||||
|
t.Errorf("error should point at CRLF, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEdit_RefusesWhileGitIsMidOperation(t *testing.T) {
|
||||||
|
// brain-sync runs `git merge -X ours origin/main`, and aborts on conflict
|
||||||
|
// with a hard reset. An edit landing inside that window is either erased
|
||||||
|
// or committed as a wrongly-resolved conflict, and the caller was told it
|
||||||
|
// succeeded.
|
||||||
|
dir := newBrain(t)
|
||||||
|
seed(t, dir, "knowledge/n.md", "alpha\n")
|
||||||
|
|
||||||
|
for _, marker := range []string{"MERGE_HEAD", "index.lock", "REBASE_HEAD"} {
|
||||||
|
t.Run(marker, func(t *testing.T) {
|
||||||
|
p := filepath.Join(dir, ".git", marker)
|
||||||
|
if err := os.WriteFile(p, []byte("x"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer func() { _ = os.Remove(p) }()
|
||||||
|
|
||||||
|
if _, err := Edit(dir, "knowledge/n.md", "alpha", "beta", ScopeNotes); err == nil {
|
||||||
|
t.Errorf("edit accepted while %s exists", marker)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEdit_RefusesAFileHoldingConflictMarkers(t *testing.T) {
|
||||||
|
dir := newBrain(t)
|
||||||
|
seed(t, dir, "knowledge/n.md", "<<<<<<< HEAD\nours\n=======\ntheirs\n>>>>>>> origin/main\n")
|
||||||
|
|
||||||
|
if _, err := Edit(dir, "knowledge/n.md", "ours", "x", ScopeNotes); err == nil {
|
||||||
|
t.Fatal("editing one side of an unresolved conflict must be refused")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEdit_ConcurrentEditsBothApply(t *testing.T) {
|
||||||
|
// Read-modify-write with no lock loses one update AND reports success for
|
||||||
|
// both. old_str uniqueness does not protect against it: uniqueness is
|
||||||
|
// evaluated against the stale read.
|
||||||
|
dir := newBrain(t)
|
||||||
|
abs := seed(t, dir, "knowledge/n.md", "alpha\nbeta\n")
|
||||||
|
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
errs := make([]error, 2)
|
||||||
|
for i, pair := range [][2]string{{"alpha", "ALPHA"}, {"beta", "BETA"}} {
|
||||||
|
wg.Add(1)
|
||||||
|
go func(i int, old, new string) {
|
||||||
|
defer wg.Done()
|
||||||
|
_, errs[i] = Edit(dir, "knowledge/n.md", old, new, ScopeNotes)
|
||||||
|
}(i, pair[0], pair[1])
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
|
||||||
|
for i, err := range errs {
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("edit %d failed: %v", i, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if got := read(t, abs); got != "ALPHA\nBETA\n" {
|
||||||
|
t.Errorf("file = %q, want both edits applied", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEdit_LeavesNoTemporaryFileBehind(t *testing.T) {
|
||||||
|
dir := newBrain(t)
|
||||||
|
seed(t, dir, "knowledge/n.md", "alpha\n")
|
||||||
|
|
||||||
|
if _, err := Edit(dir, "knowledge/n.md", "alpha", "beta", ScopeNotes); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
entries, err := os.ReadDir(filepath.Join(dir, "knowledge"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for _, e := range entries {
|
||||||
|
if e.Name() != "note.md" && e.Name() != "n.md" {
|
||||||
|
t.Errorf("stray file left in the tree: %q (git add -A would commit it)", e.Name())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEdit_PreservesReadableMode(t *testing.T) {
|
||||||
|
// Other uids write this tree (brain-fix, brain-enrich). os.CreateTemp
|
||||||
|
// yields 0600 and rename preserves it, which would break them silently.
|
||||||
|
dir := newBrain(t)
|
||||||
|
abs := seed(t, dir, "knowledge/n.md", "alpha\n")
|
||||||
|
|
||||||
|
if _, err := Edit(dir, "knowledge/n.md", "alpha", "beta", ScopeNotes); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
fi, err := os.Stat(abs)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if fi.Mode().Perm() != 0o644 {
|
||||||
|
t.Errorf("mode = %v, want 0644", fi.Mode().Perm())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEdit_RefusesNonUTF8Replacement(t *testing.T) {
|
||||||
|
dir := newBrain(t)
|
||||||
|
seed(t, dir, "knowledge/n.md", "alpha\n")
|
||||||
|
|
||||||
|
if _, err := Edit(dir, "knowledge/n.md", "alpha", "bad \xff", ScopeNotes); err == nil {
|
||||||
|
t.Fatal("new_str with invalid UTF-8 must be refused, not written")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,201 @@
|
|||||||
|
// Package notes provides containment-checked read and surgical edit of brain
|
||||||
|
// notes on the live tree that brain-sync commits from.
|
||||||
|
//
|
||||||
|
// Everything here is defensive for one reason: brain content is ingested from
|
||||||
|
// the outside world, that content reaches a model's context via brain_query /
|
||||||
|
// brain_context, and these primitives are callable by that model. A path bug
|
||||||
|
// is therefore not a path bug — it is an outside author choosing a file.
|
||||||
|
package notes
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Scope is which part of the tree a caller is allowed to touch.
|
||||||
|
type Scope int
|
||||||
|
|
||||||
|
const (
|
||||||
|
// ScopeNotes is the model-facing scope: markdown under knowledge/ and
|
||||||
|
// wiki/ only.
|
||||||
|
ScopeNotes Scope = iota
|
||||||
|
// ScopeRoot admits the named governance files and NOTHING else. It exists so
|
||||||
|
// there is an MCP path to CLAUDE.md at all — the absence of one is what
|
||||||
|
// pushed a session to edit via the gitea REST API on 2026-06-16, which
|
||||||
|
// never reached the live tree and wedged brain-sync for hours. It is NOT
|
||||||
|
// reachable from the model-facing toolset; see tools_notes.go.
|
||||||
|
ScopeRoot
|
||||||
|
)
|
||||||
|
|
||||||
|
// rootFiles is an exact-name allowlist, deliberately not a "root *.md" glob.
|
||||||
|
//
|
||||||
|
// Two reasons. log.md is gitignored, so an edit to it applies to the live tree
|
||||||
|
// and never reaches gitea — the exact shape of brain#2, admitted by a glob. And
|
||||||
|
// a glob auto-admits whatever governance file appears next (AGENTS.md, a
|
||||||
|
// bootstrap doc) with nobody reviewing that decision.
|
||||||
|
var rootFiles = map[string]bool{"CLAUDE.md": true}
|
||||||
|
|
||||||
|
// noteDirs are the two trees that hold notes.
|
||||||
|
var noteDirs = []string{"knowledge/", "wiki/"}
|
||||||
|
|
||||||
|
// deniedPrefixes are refused in every scope, independent of the allowlist.
|
||||||
|
//
|
||||||
|
// A static backstop rather than a consequence of the allowlist being right:
|
||||||
|
// .git/hooks/post-commit is executed by brain-sync inside a pod holding the
|
||||||
|
// Gitea push credentials, and archive/ + sessions/ hold verbatim session
|
||||||
|
// transcripts that BM25 never indexes and so nobody thinks of as reachable.
|
||||||
|
var deniedPrefixes = []string{".git/", "archive/", "sessions/", "raw/", "training-data/", ".obsidian/"}
|
||||||
|
|
||||||
|
// Errors are sentinels so callers can distinguish a refusal from an I/O fault.
|
||||||
|
var (
|
||||||
|
ErrOutsideBrain = errors.New("notes: path escapes the brain dir")
|
||||||
|
ErrNotAllowed = errors.New("notes: path is not editable through this tool")
|
||||||
|
ErrNotSyncSource = errors.New("notes: brain dir is not the sync source")
|
||||||
|
)
|
||||||
|
|
||||||
|
// Target is a resolved, contained path.
|
||||||
|
type Target struct {
|
||||||
|
Rel string // brain-relative, cleaned
|
||||||
|
Abs string // absolute, symlink-checked
|
||||||
|
}
|
||||||
|
|
||||||
|
// Resolve canonicalises path, then checks it against the scope's allowlist,
|
||||||
|
// then re-checks containment after symlink evaluation.
|
||||||
|
//
|
||||||
|
// Order matters and is the point: a prefix test on the raw argument accepts
|
||||||
|
// "knowledge/../.git/hooks/post-commit", because Clean only collapses it
|
||||||
|
// afterwards.
|
||||||
|
func Resolve(brainDir, path string, scope Scope) (Target, error) {
|
||||||
|
// Note: the sync-source assertion lives on the WRITE path (Edit), not
|
||||||
|
// here. brain#2 was an edit landing where the sync could not commit it; a
|
||||||
|
// read from such a tree is merely a read.
|
||||||
|
|
||||||
|
// 1. Canonicalise. Rooting at "/" makes Clean collapse every ".." that
|
||||||
|
// would otherwise escape, and ToSlash normalises separators first.
|
||||||
|
clean := strings.TrimPrefix(filepath.Clean("/"+filepath.ToSlash(path)), "/")
|
||||||
|
if clean == "" || clean == "." {
|
||||||
|
return Target{}, fmt.Errorf("%w: empty path", ErrNotAllowed)
|
||||||
|
}
|
||||||
|
|
||||||
|
// 2. Static denials, before any allowlist reasoning.
|
||||||
|
for _, d := range deniedPrefixes {
|
||||||
|
if strings.HasPrefix(clean+"/", d) {
|
||||||
|
return Target{}, fmt.Errorf("%w: %q is not note content", ErrNotAllowed, clean)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if strings.HasPrefix(filepath.Base(clean), ".") {
|
||||||
|
return Target{}, fmt.Errorf("%w: dotfiles are not editable", ErrNotAllowed)
|
||||||
|
}
|
||||||
|
|
||||||
|
// 3. Scope allowlist.
|
||||||
|
if err := allow(clean, scope); err != nil {
|
||||||
|
return Target{}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// 4. Containment, lexically.
|
||||||
|
abs := filepath.Join(brainDir, filepath.FromSlash(clean))
|
||||||
|
if !within(brainDir, abs) {
|
||||||
|
return Target{}, fmt.Errorf("%w: %q", ErrOutsideBrain, path)
|
||||||
|
}
|
||||||
|
|
||||||
|
// 5. Containment again, after resolving symlinks. The tree is Syncthing
|
||||||
|
// managed, so a link can appear without anyone adding one on purpose;
|
||||||
|
// a symlinked note or directory would otherwise read and write outside
|
||||||
|
// the brain entirely.
|
||||||
|
if err := assertNoSymlinkEscape(brainDir, abs); err != nil {
|
||||||
|
return Target{}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
return Target{Rel: clean, Abs: abs}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// allow enforces the scope's allowlist. The two scopes are DISJOINT, not
|
||||||
|
// nested: the privileged root scope reaches governance files and nothing else,
|
||||||
|
// so the tool that carries the higher privilege carries the smaller reach.
|
||||||
|
func allow(clean string, scope Scope) error {
|
||||||
|
if scope == ScopeRoot {
|
||||||
|
if !rootFiles[clean] {
|
||||||
|
return fmt.Errorf("%w: %q is not a governance file", ErrNotAllowed, clean)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if !isNotePath(clean) {
|
||||||
|
return fmt.Errorf("%w: %q is outside knowledge/ and wiki/", ErrNotAllowed, clean)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func isNotePath(clean string) bool {
|
||||||
|
var inDir bool
|
||||||
|
for _, d := range noteDirs {
|
||||||
|
if strings.HasPrefix(clean, d) {
|
||||||
|
inDir = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !inDir || filepath.Ext(clean) != ".md" {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
// _index.md is generated per wing, and brain-sync's conflict resolver
|
||||||
|
// deliberately resolves it in origin's favour — an edit here is reverted
|
||||||
|
// by design, so accepting one would be a lie.
|
||||||
|
return filepath.Base(clean) != "_index.md"
|
||||||
|
}
|
||||||
|
|
||||||
|
func within(root, abs string) bool {
|
||||||
|
rel, err := filepath.Rel(root, abs)
|
||||||
|
if err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return rel != ".." && !strings.HasPrefix(rel, ".."+string(filepath.Separator))
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertNoSymlinkEscape resolves the deepest existing ancestor of abs and
|
||||||
|
// requires it to still be inside brainDir. The file itself need not exist
|
||||||
|
// (a create is legal); its directory chain must be real and contained.
|
||||||
|
func assertNoSymlinkEscape(brainDir, abs string) error {
|
||||||
|
if fi, err := os.Lstat(abs); err == nil && fi.Mode()&os.ModeSymlink != 0 {
|
||||||
|
return fmt.Errorf("%w: %q is a symlink", ErrNotAllowed, abs)
|
||||||
|
}
|
||||||
|
|
||||||
|
realRoot, err := filepath.EvalSymlinks(brainDir)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("notes: resolve brain dir: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
dir := filepath.Dir(abs)
|
||||||
|
for {
|
||||||
|
resolved, err := filepath.EvalSymlinks(dir)
|
||||||
|
if err == nil {
|
||||||
|
if !within(realRoot, resolved) {
|
||||||
|
return fmt.Errorf("%w: %q resolves outside the brain dir", ErrOutsideBrain, dir)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if !os.IsNotExist(err) {
|
||||||
|
return fmt.Errorf("notes: resolve %q: %w", dir, err)
|
||||||
|
}
|
||||||
|
parent := filepath.Dir(dir)
|
||||||
|
if parent == dir {
|
||||||
|
return fmt.Errorf("%w: no existing ancestor for %q", ErrOutsideBrain, abs)
|
||||||
|
}
|
||||||
|
dir = parent
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertSyncSource refuses a brain dir that is not a git working tree.
|
||||||
|
//
|
||||||
|
// brain#2 was an edit landing on a directory the sync does not commit from.
|
||||||
|
// That cannot be prevented by review — a wrong INGEST_BRAIN_DIR in a future
|
||||||
|
// manifest reintroduces it while every unit test still passes — so it is
|
||||||
|
// asserted at runtime, the same check brain-sync itself makes before running.
|
||||||
|
func assertSyncSource(brainDir string) error {
|
||||||
|
if fi, err := os.Stat(filepath.Join(brainDir, ".git")); err != nil || !fi.IsDir() {
|
||||||
|
return fmt.Errorf("%w: %q has no .git, so an edit there would never be committed",
|
||||||
|
ErrNotSyncSource, brainDir)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,146 @@
|
|||||||
|
package notes
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// newBrain lays out a tree shaped like the live one: a git dir, note dirs,
|
||||||
|
// and the non-note areas that must never be reachable.
|
||||||
|
func newBrain(t *testing.T) string {
|
||||||
|
t.Helper()
|
||||||
|
dir := t.TempDir()
|
||||||
|
for _, d := range []string{
|
||||||
|
".git/hooks", "knowledge", "wiki/homelab/facts", "archive/claude-sessions/koala",
|
||||||
|
"sessions", "raw", ".obsidian",
|
||||||
|
} {
|
||||||
|
if err := os.MkdirAll(filepath.Join(dir, d), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
write := func(p, s string) {
|
||||||
|
t.Helper()
|
||||||
|
if err := os.WriteFile(filepath.Join(dir, p), []byte(s), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
write("CLAUDE.md", "# instructions\n")
|
||||||
|
write("log.md", "ignored scratch\n")
|
||||||
|
write("knowledge/note.md", "# note\n")
|
||||||
|
write("wiki/homelab/facts/f.md", "# fact\n")
|
||||||
|
write("wiki/homelab/_index.md", "# index\n")
|
||||||
|
write("archive/claude-sessions/koala/session-1.md", "verbatim transcript\n")
|
||||||
|
write("sessions/s.jsonl", "{}\n")
|
||||||
|
write(".git/hooks/post-commit", "#!/bin/sh\n")
|
||||||
|
return dir
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolve_AcceptsNotePaths(t *testing.T) {
|
||||||
|
dir := newBrain(t)
|
||||||
|
for _, p := range []string{"knowledge/note.md", "wiki/homelab/facts/f.md"} {
|
||||||
|
got, err := Resolve(dir, p, ScopeNotes)
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("Resolve(%q): %v", p, err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if !strings.HasPrefix(got.Abs, dir) {
|
||||||
|
t.Errorf("Resolve(%q).Abs = %q, want inside %q", p, got.Abs, dir)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolve_RefusesEscapesAndNonNoteAreas(t *testing.T) {
|
||||||
|
dir := newBrain(t)
|
||||||
|
|
||||||
|
// Canonicalisation must happen BEFORE the allowlist check. A prefix test
|
||||||
|
// on the raw argument accepts every one of these: "knowledge/..%2f" style
|
||||||
|
// inputs collapse to .git, and .git/hooks/post-commit is executed by
|
||||||
|
// brain-sync inside a pod holding the Gitea push credentials.
|
||||||
|
refuse := []string{
|
||||||
|
"knowledge/../.git/hooks/post-commit",
|
||||||
|
"wiki/../.git/config",
|
||||||
|
"./knowledge/../.git/config",
|
||||||
|
"knowledge//../.git/config",
|
||||||
|
"../../etc/passwd",
|
||||||
|
"/etc/passwd",
|
||||||
|
".git/config",
|
||||||
|
"archive/claude-sessions/koala/session-1.md", // verbatim transcripts
|
||||||
|
"sessions/s.jsonl",
|
||||||
|
"raw/x.md",
|
||||||
|
".obsidian/workspace.json",
|
||||||
|
"log.md", // gitignored: an edit here would never sync (brain#2)
|
||||||
|
"wiki/homelab/_index.md", // generated; brain-sync resolves it in origin's favour
|
||||||
|
"knowledge/note.txt", // not markdown
|
||||||
|
"CLAUDE.md", // root file — not reachable in the notes scope
|
||||||
|
}
|
||||||
|
for _, p := range refuse {
|
||||||
|
if _, err := Resolve(dir, p, ScopeNotes); err == nil {
|
||||||
|
t.Errorf("Resolve(%q) = nil error, want refusal", p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolve_RootScopeAdmitsOnlyNamedFiles(t *testing.T) {
|
||||||
|
dir := newBrain(t)
|
||||||
|
|
||||||
|
if _, err := Resolve(dir, "CLAUDE.md", ScopeRoot); err != nil {
|
||||||
|
t.Errorf("Resolve(CLAUDE.md, ScopeRoot): %v", err)
|
||||||
|
}
|
||||||
|
// log.md is the reason this is an exact-name list and not "root *.md":
|
||||||
|
// it is gitignored, so an edit would apply locally and never reach gitea.
|
||||||
|
for _, p := range []string{"log.md", "AGENTS.md", "knowledge/note.md", ".git/config"} {
|
||||||
|
if _, err := Resolve(dir, p, ScopeRoot); err == nil {
|
||||||
|
t.Errorf("Resolve(%q, ScopeRoot) = nil error, want refusal", p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolve_RefusesSymlinkedTargets(t *testing.T) {
|
||||||
|
dir := newBrain(t)
|
||||||
|
outside := filepath.Join(t.TempDir(), "secret.md")
|
||||||
|
if err := os.WriteFile(outside, []byte("secret\n"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
// The tree is Syncthing-managed, so a symlink can appear without anyone
|
||||||
|
// adding one deliberately. Containment must survive it.
|
||||||
|
if err := os.Symlink(outside, filepath.Join(dir, "knowledge/link.md")); err != nil {
|
||||||
|
t.Skipf("symlinks unavailable: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := Resolve(dir, "knowledge/link.md", ScopeNotes); err == nil {
|
||||||
|
t.Error("Resolve followed a symlink out of the brain dir")
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := os.Symlink(filepath.Dir(outside), filepath.Join(dir, "wiki/escape")); err != nil {
|
||||||
|
t.Skipf("symlinks unavailable: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := Resolve(dir, "wiki/escape/secret.md", ScopeNotes); err == nil {
|
||||||
|
t.Error("Resolve followed a symlinked directory out of the brain dir")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolve_RefusesWhenTheTreeIsNotTheSyncSource(t *testing.T) {
|
||||||
|
// brain#2 was "wrote to a directory the sync does not commit from". No
|
||||||
|
// code review prevents that recurring — a wrong INGEST_BRAIN_DIR in a
|
||||||
|
// future manifest reintroduces it and every unit test still passes. So
|
||||||
|
// assert the property at runtime, the way brain-sync itself does.
|
||||||
|
dir := t.TempDir()
|
||||||
|
if err := os.MkdirAll(filepath.Join(dir, "knowledge"), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(filepath.Join(dir, "knowledge/note.md"), []byte("x"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
// Reads are fine anywhere; the guard belongs to writes.
|
||||||
|
if _, err := Resolve(dir, "knowledge/note.md", ScopeNotes); err != nil {
|
||||||
|
t.Fatalf("Resolve should not require .git for a read: %v", err)
|
||||||
|
}
|
||||||
|
_, err := Edit(dir, "knowledge/note.md", "x", "y", ScopeNotes)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Edit accepted a tree with no .git — the write would never be committed")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "sync source") {
|
||||||
|
t.Errorf("error should name the cause, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -76,6 +76,15 @@ func buildFrontmatter(rp RawPage, date string) string {
|
|||||||
}
|
}
|
||||||
fmt.Fprintf(&sb, "date_ingested: %s\n", date)
|
fmt.Fprintf(&sb, "date_ingested: %s\n", date)
|
||||||
fmt.Fprintf(&sb, "last_updated: %s\n", date)
|
fmt.Fprintf(&sb, "last_updated: %s\n", date)
|
||||||
|
if rp.Source != "" {
|
||||||
|
fmt.Fprintf(&sb, "source: %s\n", yamlScalar(rp.Source))
|
||||||
|
}
|
||||||
|
if rp.Author != "" {
|
||||||
|
fmt.Fprintf(&sb, "author: %s\n", yamlScalar(rp.Author))
|
||||||
|
}
|
||||||
|
if rp.Published != "" {
|
||||||
|
fmt.Fprintf(&sb, "published: %s\n", yamlScalar(rp.Published))
|
||||||
|
}
|
||||||
case "concept":
|
case "concept":
|
||||||
if rp.Domain != "" {
|
if rp.Domain != "" {
|
||||||
fmt.Fprintf(&sb, "domain: %s\n", yamlScalar(rp.Domain))
|
fmt.Fprintf(&sb, "domain: %s\n", yamlScalar(rp.Domain))
|
||||||
|
|||||||
@@ -154,6 +154,52 @@ func TestBuildPages_EntityNoSubtype(t *testing.T) {
|
|||||||
assert.Contains(t, pages[0].Content, "title: 'Basecamp'")
|
assert.Contains(t, pages[0].Content, "title: 'Basecamp'")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestBuildPages_SourcePageCarriesSourceAuthorPublished(t *testing.T) {
|
||||||
|
raw := []RawPage{
|
||||||
|
{
|
||||||
|
Title: "Ornith",
|
||||||
|
Type: "source",
|
||||||
|
Subtype: "article",
|
||||||
|
Content: "## Summary\n\nAn agentic coding model.\n",
|
||||||
|
Source: "https://example.com/ornith",
|
||||||
|
Author: "Jane Doe",
|
||||||
|
Published: "2026-07-20",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
pages, warnings := BuildPages(raw, "ornith", "2026-07-26")
|
||||||
|
require.Len(t, pages, 1)
|
||||||
|
assert.Empty(t, warnings)
|
||||||
|
|
||||||
|
p := pages[0]
|
||||||
|
assert.Contains(t, p.Content, "source: 'https://example.com/ornith'")
|
||||||
|
assert.Contains(t, p.Content, "author: 'Jane Doe'")
|
||||||
|
assert.Contains(t, p.Content, "published: '2026-07-20'")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildPages_SourcePageOmitsSourceAuthorPublishedWhenEmpty(t *testing.T) {
|
||||||
|
raw := []RawPage{
|
||||||
|
{Title: "Shape Up", Type: "source", Subtype: "book", Content: "## Summary\n\nA book.\n"},
|
||||||
|
}
|
||||||
|
pages, _ := BuildPages(raw, "shape-up", "2026-04-23")
|
||||||
|
require.Len(t, pages, 1)
|
||||||
|
assert.NotContains(t, pages[0].Content, "source:")
|
||||||
|
assert.NotContains(t, pages[0].Content, "author:")
|
||||||
|
assert.NotContains(t, pages[0].Content, "published:")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildPages_ConceptPageIgnoresSourceAuthorPublished(t *testing.T) {
|
||||||
|
// source/author/published are source-note-only metadata; a concept page
|
||||||
|
// shouldn't carry them even if somehow set on the RawPage.
|
||||||
|
raw := []RawPage{
|
||||||
|
{Title: "Betting", Type: "concept", Content: "## Definition\n\nFoo.\n", Source: "x", Author: "y", Published: "z"},
|
||||||
|
}
|
||||||
|
pages, _ := BuildPages(raw, "src", "2026-04-23")
|
||||||
|
require.Len(t, pages, 1)
|
||||||
|
assert.NotContains(t, pages[0].Content, "source:")
|
||||||
|
assert.NotContains(t, pages[0].Content, "author:")
|
||||||
|
assert.NotContains(t, pages[0].Content, "published:")
|
||||||
|
}
|
||||||
|
|
||||||
func TestBuildPages_EmptyTitleSkippedWithWarning(t *testing.T) {
|
func TestBuildPages_EmptyTitleSkippedWithWarning(t *testing.T) {
|
||||||
raw := []RawPage{
|
raw := []RawPage{
|
||||||
{Title: "", Type: "concept", Content: "## Definition\n\nFoo.\n"},
|
{Title: "", Type: "concept", Content: "## Definition\n\nFoo.\n"},
|
||||||
|
|||||||
@@ -51,6 +51,21 @@ func buildTitleMap(pages []wiki.Page, inventory map[wiki.PageType][]wiki.Entry)
|
|||||||
return m
|
return m
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// pathStylePrefixes are known root prefixes the LLM extraction step
|
||||||
|
// sometimes bakes into a wikilink target instead of emitting a clean
|
||||||
|
// wing/hall/slug path (or bare title). Stripping them repairs the link
|
||||||
|
// in place — see hyperguild#87.
|
||||||
|
var pathStylePrefixes = []string{"wing:", "wiki/"}
|
||||||
|
|
||||||
|
func stripPathStylePrefix(displayName string) (string, bool) {
|
||||||
|
for _, prefix := range pathStylePrefixes {
|
||||||
|
if stripped, ok := strings.CutPrefix(displayName, prefix); ok {
|
||||||
|
return stripped, true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return displayName, false
|
||||||
|
}
|
||||||
|
|
||||||
func canonicalizeContent(content string, titleToSlug map[string]string) (string, []string) {
|
func canonicalizeContent(content string, titleToSlug map[string]string) (string, []string) {
|
||||||
var warnings []string
|
var warnings []string
|
||||||
result := plainLinkRE.ReplaceAllStringFunc(content, func(match string) string {
|
result := plainLinkRE.ReplaceAllStringFunc(content, func(match string) string {
|
||||||
@@ -59,12 +74,17 @@ func canonicalizeContent(content string, titleToSlug map[string]string) (string,
|
|||||||
return match
|
return match
|
||||||
}
|
}
|
||||||
displayName := sub[1]
|
displayName := sub[1]
|
||||||
slug, ok := titleToSlug[strings.ToLower(displayName)]
|
|
||||||
if !ok {
|
if slug, ok := titleToSlug[strings.ToLower(displayName)]; ok {
|
||||||
warnings = append(warnings, fmt.Sprintf("unknown wikilink: [[%s]]", displayName))
|
return "[[" + slug + "|" + displayName + "]]"
|
||||||
return match
|
|
||||||
}
|
}
|
||||||
return "[[" + slug + "|" + displayName + "]]"
|
|
||||||
|
if stripped, hadPrefix := stripPathStylePrefix(displayName); hadPrefix {
|
||||||
|
return "[[" + stripped + "]]"
|
||||||
|
}
|
||||||
|
|
||||||
|
warnings = append(warnings, fmt.Sprintf("unknown wikilink: [[%s]]", displayName))
|
||||||
|
return match
|
||||||
})
|
})
|
||||||
return result, warnings
|
return result, warnings
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -102,6 +102,53 @@ func TestCanonicalizeLinks_CurrentBatchPagesResolved(t *testing.T) {
|
|||||||
assert.Contains(t, got[0].Content, "[[betting|Betting]]")
|
assert.Contains(t, got[0].Content, "[[betting|Betting]]")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestCanonicalizeLinks_StripsWingColonPrefix(t *testing.T) {
|
||||||
|
pages := []wiki.Page{
|
||||||
|
{
|
||||||
|
Path: "wiki/homelab/failures/act-runner-host-mode-container-needs-node-and-libatomic.md",
|
||||||
|
Content: "---\ntitle: 'act_runner host-mode'\n---\n\nSee [[wing:homelab/failures/rootless-buildah-act-runner-run-containers-denied]].\n",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
got, warnings := CanonicalizeLinks(pages, map[wiki.PageType][]wiki.Entry{})
|
||||||
|
require.Len(t, got, 1)
|
||||||
|
assert.Empty(t, warnings)
|
||||||
|
assert.Contains(t, got[0].Content, "[[homelab/failures/rootless-buildah-act-runner-run-containers-denied]]")
|
||||||
|
assert.NotContains(t, got[0].Content, "wing:")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCanonicalizeLinks_StripsWikiSlashPrefix(t *testing.T) {
|
||||||
|
pages := []wiki.Page{
|
||||||
|
{
|
||||||
|
Path: "wiki/agentsquad/hypotheses/council-consolidation-standalone-deliberation-service.md",
|
||||||
|
Content: "---\ntitle: 'council consolidation'\n---\n\nSee [[wiki/agentsquad/decisions/autoresearch-council-sibling-pipe]].\n",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
got, warnings := CanonicalizeLinks(pages, map[wiki.PageType][]wiki.Entry{})
|
||||||
|
require.Len(t, got, 1)
|
||||||
|
assert.Empty(t, warnings)
|
||||||
|
assert.Contains(t, got[0].Content, "[[agentsquad/decisions/autoresearch-council-sibling-pipe]]")
|
||||||
|
assert.NotContains(t, got[0].Content, "wiki/agentsquad/decisions/autoresearch-council-sibling-pipe]]\n\n") // no leftover wiki/ prefix
|
||||||
|
assert.NotContains(t, got[0].Content, "[[wiki/")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCanonicalizeLinks_TitleLookupStillTakesPriorityOverPrefixStrip(t *testing.T) {
|
||||||
|
// A plain link that resolves via the title map must still use the
|
||||||
|
// normal slug|Display form, not fall through to prefix-strip repair.
|
||||||
|
pages := []wiki.Page{
|
||||||
|
{
|
||||||
|
Path: "wiki/sources/shape-up.md",
|
||||||
|
Content: "---\ntitle: 'Shape Up'\n---\n\nSee [[Betting]].\n",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
inventory := map[wiki.PageType][]wiki.Entry{
|
||||||
|
wiki.PageTypeConcept: {{Slug: "betting", Title: "Betting"}},
|
||||||
|
}
|
||||||
|
got, warnings := CanonicalizeLinks(pages, inventory)
|
||||||
|
require.Len(t, got, 1)
|
||||||
|
assert.Empty(t, warnings)
|
||||||
|
assert.Contains(t, got[0].Content, "[[betting|Betting]]")
|
||||||
|
}
|
||||||
|
|
||||||
func TestCanonicalizeLinks_MultipleLinksInOnePage(t *testing.T) {
|
func TestCanonicalizeLinks_MultipleLinksInOnePage(t *testing.T) {
|
||||||
pages := []wiki.Page{
|
pages := []wiki.Page{
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -15,6 +15,14 @@ type RawPage struct {
|
|||||||
Subtype string `json:"subtype"` // entity: person|company|tool|model|framework|technology; source: article|pdf|book|video|note|project
|
Subtype string `json:"subtype"` // entity: person|company|tool|model|framework|technology; source: article|pdf|book|video|note|project
|
||||||
Domain string `json:"domain"`
|
Domain string `json:"domain"`
|
||||||
Content string `json:"content"` // Markdown body only — no frontmatter
|
Content string `json:"content"` // Markdown body only — no frontmatter
|
||||||
|
|
||||||
|
// Source, Author, Published are deterministic passthrough from the raw
|
||||||
|
// ingested content's own frontmatter (see parseContentFrontmatter) — never
|
||||||
|
// set by the LLM. json:"-" keeps them immune to same-named keys the LLM
|
||||||
|
// might emit. Only meaningful for Type == "source".
|
||||||
|
Source string `json:"-"`
|
||||||
|
Author string `json:"-"`
|
||||||
|
Published string `json:"-"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// ParseRawPages parses LLM output as a JSON array of RawPage objects.
|
// ParseRawPages parses LLM output as a JSON array of RawPage objects.
|
||||||
@@ -98,6 +106,60 @@ func repairJSON(s string) string {
|
|||||||
return b.String()
|
return b.String()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// sourceMeta is source/author/published pulled from the raw ingested
|
||||||
|
// content's own frontmatter — deterministic passthrough, never LLM output.
|
||||||
|
type sourceMeta struct {
|
||||||
|
Source string
|
||||||
|
Author string
|
||||||
|
Published string
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseContentFrontmatter extracts source/author/published from a leading
|
||||||
|
// "---\n...\n---" YAML block in raw ingested content. Only these three flat
|
||||||
|
// scalar keys are recognised; anything else in the block is ignored. Returns
|
||||||
|
// a zero-value sourceMeta if content has no frontmatter block.
|
||||||
|
func parseContentFrontmatter(content string) sourceMeta {
|
||||||
|
var meta sourceMeta
|
||||||
|
if !strings.HasPrefix(content, "---\n") && !strings.HasPrefix(content, "---\r\n") {
|
||||||
|
return meta
|
||||||
|
}
|
||||||
|
|
||||||
|
lines := strings.Split(content, "\n")
|
||||||
|
for _, line := range lines[1:] {
|
||||||
|
if strings.TrimSpace(line) == "---" {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
key, val, ok := strings.Cut(line, ":")
|
||||||
|
if !ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
key = strings.TrimSpace(key)
|
||||||
|
val = strings.Trim(strings.TrimSpace(val), `"'`)
|
||||||
|
switch key {
|
||||||
|
case "source":
|
||||||
|
meta.Source = val
|
||||||
|
case "author":
|
||||||
|
meta.Author = val
|
||||||
|
case "published":
|
||||||
|
meta.Published = val
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return meta
|
||||||
|
}
|
||||||
|
|
||||||
|
// applySourceMeta deterministically overwrites Source/Author/Published on
|
||||||
|
// every "source"-type page with meta — the LLM never controls these fields.
|
||||||
|
func applySourceMeta(pages []RawPage, meta sourceMeta) {
|
||||||
|
for i := range pages {
|
||||||
|
if pages[i].Type != "source" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
pages[i].Source = meta.Source
|
||||||
|
pages[i].Author = meta.Author
|
||||||
|
pages[i].Published = meta.Published
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func stripFences(s string) string {
|
func stripFences(s string) string {
|
||||||
for _, prefix := range []string{"```json\n", "```json\r\n", "```\n", "```\r\n"} {
|
for _, prefix := range []string{"```json\n", "```json\r\n", "```\n", "```\r\n"} {
|
||||||
if strings.HasPrefix(s, prefix) {
|
if strings.HasPrefix(s, prefix) {
|
||||||
|
|||||||
@@ -59,6 +59,8 @@ func Run(ctx context.Context, cfg Config, brainDir, content, source string, dryR
|
|||||||
allWarnings = append(allWarnings, warnings...)
|
allWarnings = append(allWarnings, warnings...)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
applySourceMeta(allRaw, parseContentFrontmatter(content))
|
||||||
|
|
||||||
return buildAndWrite(allRaw, sourceSlug, date, brainDir, source, inventory, allWarnings, dryRun)
|
return buildAndWrite(allRaw, sourceSlug, date, brainDir, source, inventory, allWarnings, dryRun)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -130,6 +130,40 @@ func TestRun_MergesDuplicatePaths(t *testing.T) {
|
|||||||
assert.Contains(t, string(content), "[[Baz]]")
|
assert.Contains(t, string(content), "[[Baz]]")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRun_ThreadsSourceAuthorPublishedFromContentFrontmatter(t *testing.T) {
|
||||||
|
brainDir := t.TempDir()
|
||||||
|
for _, sub := range []string{"wiki/concepts", "wiki/entities", "wiki/sources"} {
|
||||||
|
require.NoError(t, os.MkdirAll(filepath.Join(brainDir, sub), 0o755))
|
||||||
|
}
|
||||||
|
|
||||||
|
llmResponse := mustJSON([]RawPage{{
|
||||||
|
Title: "Ornith",
|
||||||
|
Type: "source",
|
||||||
|
Subtype: "article",
|
||||||
|
Content: "## Summary\n\nAn agentic coding model.\n",
|
||||||
|
}})
|
||||||
|
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
_ = json.NewEncoder(w).Encode(map[string]any{
|
||||||
|
"choices": []map[string]any{{"message": map[string]any{"content": llmResponse}}},
|
||||||
|
})
|
||||||
|
}))
|
||||||
|
defer srv.Close()
|
||||||
|
|
||||||
|
cfg := Config{Complete: llm.New(srv.URL, "", "m", 30*time.Second).Complete}
|
||||||
|
rawContent := "---\nsource: https://example.com/ornith\nauthor: Jane Doe\npublished: 2026-07-20\n---\n\nAn agentic coding model that runs on your laptop.\n"
|
||||||
|
|
||||||
|
result, err := Run(context.Background(), cfg, brainDir, rawContent, "ornith", false)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, result.Pages, 1)
|
||||||
|
|
||||||
|
content, err := os.ReadFile(filepath.Join(brainDir, "wiki", "sources", "ornith.md"))
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Contains(t, string(content), "source: 'https://example.com/ornith'")
|
||||||
|
assert.Contains(t, string(content), "author: 'Jane Doe'")
|
||||||
|
assert.Contains(t, string(content), "published: '2026-07-20'")
|
||||||
|
}
|
||||||
|
|
||||||
func mustJSON(v any) string {
|
func mustJSON(v any) string {
|
||||||
b, err := json.Marshal(v)
|
b, err := json.Marshal(v)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
|
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
|
||||||
|
"github.com/mathiasbq/hyperguild/ingestion/internal/textutil"
|
||||||
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
|
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -427,8 +428,5 @@ func excerpt(content string, maxLen int) string {
|
|||||||
if len(parts) == 3 {
|
if len(parts) == 3 {
|
||||||
body = strings.TrimSpace(parts[2])
|
body = strings.TrimSpace(parts[2])
|
||||||
}
|
}
|
||||||
if len(body) > maxLen {
|
return textutil.Truncate(body, maxLen)
|
||||||
return body[:maxLen] + "…"
|
|
||||||
}
|
|
||||||
return body
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,75 @@
|
|||||||
|
// Package startupretry retries a startup dependency that may not be serving
|
||||||
|
// yet, instead of exiting on its first failure.
|
||||||
|
//
|
||||||
|
// It exists for the cold-boot ordering race (#83): on a cluster restart the
|
||||||
|
// ingestion pod can reach OIDC discovery before dex is answering, gets a 502
|
||||||
|
// from ingress, and dies — after having already connected postgres, backfilled
|
||||||
|
// the graph and ingested session batches, all of which the restart redoes.
|
||||||
|
// Nothing is misconfigured in that scenario; the pod simply started first.
|
||||||
|
//
|
||||||
|
// The budget is bounded on purpose. Retrying forever would turn a genuine
|
||||||
|
// misconfiguration (wrong issuer URL, bad audience) into a process that looks
|
||||||
|
// healthy and never serves, which is harder to diagnose than a crash loop.
|
||||||
|
package startupretry
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Policy bounds the retry: at most Attempts calls, with the delay doubling
|
||||||
|
// from Base and capped at Max.
|
||||||
|
type Policy struct {
|
||||||
|
Attempts int // total calls, including the first; < 1 means 1
|
||||||
|
Base time.Duration // delay before the second attempt
|
||||||
|
Max time.Duration // ceiling for the doubling delay
|
||||||
|
}
|
||||||
|
|
||||||
|
// Do calls fn until it succeeds, the attempt budget is spent, or ctx ends.
|
||||||
|
//
|
||||||
|
// The returned error wraps the last failure, so callers keep the diagnosis
|
||||||
|
// they would have had without the retry.
|
||||||
|
func Do[T any](ctx context.Context, p Policy, fn func(context.Context) (T, error)) (T, error) {
|
||||||
|
var zero T
|
||||||
|
|
||||||
|
attempts := p.Attempts
|
||||||
|
if attempts < 1 {
|
||||||
|
attempts = 1
|
||||||
|
}
|
||||||
|
|
||||||
|
delay := p.Base
|
||||||
|
var lastErr error
|
||||||
|
|
||||||
|
for i := range attempts {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return zero, fmt.Errorf("startupretry: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
v, err := fn(ctx)
|
||||||
|
if err == nil {
|
||||||
|
return v, nil
|
||||||
|
}
|
||||||
|
lastErr = err
|
||||||
|
|
||||||
|
if i == attempts-1 {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
if delay <= 0 {
|
||||||
|
delay = time.Second
|
||||||
|
}
|
||||||
|
if p.Max > 0 && delay > p.Max {
|
||||||
|
delay = p.Max
|
||||||
|
}
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return zero, fmt.Errorf("startupretry: %w", ctx.Err())
|
||||||
|
case <-time.After(delay):
|
||||||
|
}
|
||||||
|
delay *= 2
|
||||||
|
}
|
||||||
|
|
||||||
|
return zero, fmt.Errorf("startupretry: gave up after %d attempts: %w", attempts, lastErr)
|
||||||
|
}
|
||||||
@@ -0,0 +1,144 @@
|
|||||||
|
// ingestion/internal/startupretry/startupretry_test.go
|
||||||
|
package startupretry
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func fastPolicy(attempts int) Policy {
|
||||||
|
return Policy{Attempts: attempts, Base: time.Millisecond, Max: 2 * time.Millisecond}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDo_ReturnsImmediatelyOnFirstSuccess(t *testing.T) {
|
||||||
|
calls := 0
|
||||||
|
got, err := Do(context.Background(), fastPolicy(5), func(context.Context) (string, error) {
|
||||||
|
calls++
|
||||||
|
return "ready", nil
|
||||||
|
})
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Do: %v", err)
|
||||||
|
}
|
||||||
|
if got != "ready" {
|
||||||
|
t.Errorf("got %q, want %q", got, "ready")
|
||||||
|
}
|
||||||
|
if calls != 1 {
|
||||||
|
t.Errorf("calls = %d, want 1", calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDo_SurvivesTransientFailuresThenSucceeds(t *testing.T) {
|
||||||
|
// The observed cold-boot race: dex is not serving yet, ingress answers
|
||||||
|
// 502 for the first few attempts, then discovery works.
|
||||||
|
calls := 0
|
||||||
|
got, err := Do(context.Background(), fastPolicy(5), func(context.Context) (string, error) {
|
||||||
|
calls++
|
||||||
|
if calls < 4 {
|
||||||
|
return "", errors.New("oidc discovery: status 502")
|
||||||
|
}
|
||||||
|
return "ready", nil
|
||||||
|
})
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Do returned %v — a transient 502 must not be fatal", err)
|
||||||
|
}
|
||||||
|
if got != "ready" {
|
||||||
|
t.Errorf("got %q, want %q", got, "ready")
|
||||||
|
}
|
||||||
|
if calls != 4 {
|
||||||
|
t.Errorf("calls = %d, want 4", calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDo_GivesUpAfterTheAttemptBudgetAndReportsTheLastError(t *testing.T) {
|
||||||
|
// A genuinely broken issuer must still fail — retrying forever would turn
|
||||||
|
// a config error into a silent hang, which is worse than crash-looping.
|
||||||
|
last := errors.New("oidc discovery: status 404")
|
||||||
|
calls := 0
|
||||||
|
|
||||||
|
_, err := Do(context.Background(), fastPolicy(3), func(context.Context) (string, error) {
|
||||||
|
calls++
|
||||||
|
return "", last
|
||||||
|
})
|
||||||
|
|
||||||
|
if !errors.Is(err, last) {
|
||||||
|
t.Fatalf("err = %v, want it to wrap the last failure", err)
|
||||||
|
}
|
||||||
|
if calls != 3 {
|
||||||
|
t.Errorf("calls = %d, want 3 (the attempt budget)", calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDo_StopsPromptlyWhenTheContextIsCancelled(t *testing.T) {
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
calls := 0
|
||||||
|
|
||||||
|
done := make(chan struct{})
|
||||||
|
var err error
|
||||||
|
go func() {
|
||||||
|
defer close(done)
|
||||||
|
_, err = Do(ctx, Policy{Attempts: 100, Base: 50 * time.Millisecond, Max: time.Second},
|
||||||
|
func(context.Context) (string, error) {
|
||||||
|
calls++
|
||||||
|
if calls == 1 {
|
||||||
|
cancel()
|
||||||
|
}
|
||||||
|
return "", errors.New("still down")
|
||||||
|
})
|
||||||
|
}()
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-done:
|
||||||
|
case <-time.After(2 * time.Second):
|
||||||
|
t.Fatal("Do did not return after context cancellation")
|
||||||
|
}
|
||||||
|
if !errors.Is(err, context.Canceled) {
|
||||||
|
t.Errorf("err = %v, want context.Canceled", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDo_BacksOffAndIsCappedByMax(t *testing.T) {
|
||||||
|
var delays []time.Duration
|
||||||
|
p := Policy{Attempts: 5, Base: 10 * time.Millisecond, Max: 25 * time.Millisecond}
|
||||||
|
|
||||||
|
last := time.Now()
|
||||||
|
_, _ = Do(context.Background(), p, func(context.Context) (string, error) {
|
||||||
|
now := time.Now()
|
||||||
|
delays = append(delays, now.Sub(last))
|
||||||
|
last = now
|
||||||
|
return "", errors.New("down")
|
||||||
|
})
|
||||||
|
|
||||||
|
// delays[0] is the first call (no wait); the rest are the sleeps.
|
||||||
|
if len(delays) != 5 {
|
||||||
|
t.Fatalf("attempts = %d, want 5", len(delays))
|
||||||
|
}
|
||||||
|
if delays[1] >= delays[2] {
|
||||||
|
t.Errorf("backoff is not growing: %v then %v", delays[1], delays[2])
|
||||||
|
}
|
||||||
|
for i, d := range delays[1:] {
|
||||||
|
if d > 2*p.Max {
|
||||||
|
t.Errorf("delay %d = %v, exceeds the Max cap %v with slack", i+1, d, p.Max)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPolicy_ZeroValueStillMakesOneAttempt(t *testing.T) {
|
||||||
|
// A caller that forgets to configure the policy must not silently get a
|
||||||
|
// no-op that returns a nil validator and a nil error.
|
||||||
|
calls := 0
|
||||||
|
_, err := Do(context.Background(), Policy{}, func(context.Context) (string, error) {
|
||||||
|
calls++
|
||||||
|
return "", errors.New("down")
|
||||||
|
})
|
||||||
|
|
||||||
|
if calls != 1 {
|
||||||
|
t.Errorf("calls = %d, want 1", calls)
|
||||||
|
}
|
||||||
|
if err == nil {
|
||||||
|
t.Error("err = nil, want the failure surfaced")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
// Package textutil holds small text helpers shared across the ingestion
|
||||||
|
// service.
|
||||||
|
package textutil
|
||||||
|
|
||||||
|
import "unicode/utf8"
|
||||||
|
|
||||||
|
// Ellipsis marks a value that was cut short.
|
||||||
|
const Ellipsis = "…"
|
||||||
|
|
||||||
|
// Truncate shortens s to at most limit CHARACTERS, appending an ellipsis when
|
||||||
|
// anything was dropped.
|
||||||
|
//
|
||||||
|
// It exists because four call sites did `s[:limit] + "…"` (#93). Both halves
|
||||||
|
// of that are byte operations: len() counts bytes and a slice cuts bytes, so
|
||||||
|
// on Swedish text — where å ä ö are two bytes and an em dash is three — the
|
||||||
|
// cut lands mid-character and leaves an orphaned lead byte. The result is
|
||||||
|
// invalid UTF-8.
|
||||||
|
//
|
||||||
|
// That failed loudly in exactly one place, wiki/index.md, which is written
|
||||||
|
// raw to disk and then rejected by every strict decoder. The other three fed
|
||||||
|
// MCP responses, where encoding/json silently substitutes U+FFFD — so those
|
||||||
|
// had been quietly eating the last character of truncated excerpts with no
|
||||||
|
// error anywhere.
|
||||||
|
//
|
||||||
|
// The marker is appended only when content was actually dropped: an ellipsis
|
||||||
|
// on complete text is a lie about the content.
|
||||||
|
func Truncate(s string, limit int) string {
|
||||||
|
if limit <= 0 {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
if utf8.RuneCountInString(s) <= limit {
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
// Walk runes rather than converting to []rune: the input can be a whole
|
||||||
|
// note body, and this runs on the read path.
|
||||||
|
count := 0
|
||||||
|
for i := range s {
|
||||||
|
if count == limit {
|
||||||
|
return s[:i] + Ellipsis
|
||||||
|
}
|
||||||
|
count++
|
||||||
|
}
|
||||||
|
return s + Ellipsis
|
||||||
|
}
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
// ingestion/internal/textutil/truncate_test.go
|
||||||
|
package textutil_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
|
"github.com/mathiasbq/hyperguild/ingestion/internal/textutil"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestTruncate_ShortStringsAreUntouched(t *testing.T) {
|
||||||
|
for _, s := range []string{"", "kort", "räv", "a — b"} {
|
||||||
|
if got := textutil.Truncate(s, 100); got != s {
|
||||||
|
t.Errorf("Truncate(%q, 100) = %q, want it unchanged", s, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTruncate_NeverCutsInsideACharacter(t *testing.T) {
|
||||||
|
// The property, not an example: whatever the limit, the result must be
|
||||||
|
// valid UTF-8. Byte-slicing Swedish text (å ä ö) or an em dash produces
|
||||||
|
// an orphaned lead byte, which is what corrupted wiki/index.md (#93).
|
||||||
|
s := strings.Repeat("räv — ål — ökäd ", 20)
|
||||||
|
|
||||||
|
for limit := 0; limit <= len(s)+5; limit++ {
|
||||||
|
got := textutil.Truncate(s, limit)
|
||||||
|
if !utf8.ValidString(got) {
|
||||||
|
t.Fatalf("Truncate(s, %d) produced invalid UTF-8: %q", limit, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTruncate_CountsCharactersNotBytes(t *testing.T) {
|
||||||
|
// "räv" is 4 bytes, 3 runes. A byte-based limit of 3 would cut inside the
|
||||||
|
// "ä"; a rune-based one returns it whole.
|
||||||
|
if got := textutil.Truncate("räv", 3); got != "räv" {
|
||||||
|
t.Errorf("Truncate(\"räv\", 3) = %q, want the whole word", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
got := textutil.Truncate("räven springer", 5)
|
||||||
|
if utf8.RuneCountInString(got) != 6 { // 5 runes + the ellipsis
|
||||||
|
t.Errorf("Truncate(...) = %q, want 5 runes plus the marker", got)
|
||||||
|
}
|
||||||
|
if !strings.HasSuffix(got, "…") {
|
||||||
|
t.Errorf("Truncate(...) = %q, want it marked as truncated", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTruncate_MarksOnlyWhenSomethingWasDropped(t *testing.T) {
|
||||||
|
// An ellipsis on text that was not cut is a lie about the content.
|
||||||
|
if got := textutil.Truncate("räv", 10); strings.HasSuffix(got, "…") {
|
||||||
|
t.Errorf("Truncate(%q, 10) = %q, want no truncation marker", "räv", got)
|
||||||
|
}
|
||||||
|
if got := textutil.Truncate("räv", 3); strings.HasSuffix(got, "…") {
|
||||||
|
t.Errorf("exact-length input must not be marked: %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTruncate_NonPositiveLimitYieldsNothingRatherThanPanicking(t *testing.T) {
|
||||||
|
for _, limit := range []int{0, -1, -100} {
|
||||||
|
if got := textutil.Truncate("räven", limit); got != "" {
|
||||||
|
t.Errorf("Truncate(_, %d) = %q, want empty", limit, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -74,6 +74,13 @@ func processDir(ctx context.Context, cfg Config, date string) []error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// AutoTunnel's own fuzzy-match human-review queue, not source
|
||||||
|
// material to extract (hyperguild#88) — same exclusion as
|
||||||
|
// api.ListPending already applies when listing raw/ for promotion.
|
||||||
|
if strings.HasPrefix(d.Name(), "tunnel-candidates-") {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
// Skip files that have already been processed or permanently failed.
|
// Skip files that have already been processed or permanently failed.
|
||||||
if _, err := os.Stat(path + ".processed"); err == nil {
|
if _, err := os.Stat(path + ".processed"); err == nil {
|
||||||
return nil
|
return nil
|
||||||
|
|||||||
@@ -229,3 +229,49 @@ func TestProcessDir_SkipsSubdirs(t *testing.T) {
|
|||||||
_, err = os.Stat(failedFile)
|
_, err = os.Stat(failedFile)
|
||||||
assert.NoError(t, err, "failed subdir file should be untouched")
|
assert.NoError(t, err, "failed subdir file should be untouched")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestProcessDir_SkipsTunnelCandidateFiles guards against hyperguild#88:
|
||||||
|
// AutoTunnel's own human-review queue (brain/raw/tunnel-candidates-*.md)
|
||||||
|
// must never be re-ingested as if it were external source material — doing
|
||||||
|
// so fed the raw "(term: X)" log entries back through the LLM extraction
|
||||||
|
// pipeline, which then legitimately (from its own perspective) surfaced
|
||||||
|
// [[X]] as a wikilink for any term that happened to match a real page
|
||||||
|
// title, however generic (e.g. "mission").
|
||||||
|
func TestProcessDir_SkipsTunnelCandidateFiles(t *testing.T) {
|
||||||
|
brainDir := setupBrainDir(t)
|
||||||
|
|
||||||
|
tunnelFile := filepath.Join(brainDir, "raw", "tunnel-candidates-2026-07-19.md")
|
||||||
|
require.NoError(t, os.WriteFile(tunnelFile, []byte(
|
||||||
|
"# Tunnel candidates 2026-07-19\n\n- `wiki/homelab/decisions/foo.md` ↔ `wiki/telos/decisions/mission.md` (term: \"mission\")\n",
|
||||||
|
), 0o644))
|
||||||
|
|
||||||
|
var completeCalls int
|
||||||
|
completeFn := func(ctx context.Context, system, user string) (string, error) {
|
||||||
|
completeCalls++
|
||||||
|
raw := pipeline.RawPage{Title: "Should not be written", Type: "source", Subtype: "article", Content: "## Summary\n\nx.\n"}
|
||||||
|
b, _ := json.Marshal([]pipeline.RawPage{raw})
|
||||||
|
return string(b), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg := Config{
|
||||||
|
BrainDir: brainDir,
|
||||||
|
Interval: time.Hour, // not used; we call processDir directly
|
||||||
|
Pipeline: pipeline.Config{
|
||||||
|
Complete: completeFn,
|
||||||
|
ChunkSize: 0,
|
||||||
|
Schema: "# Schema\nThree page types.",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
date := time.Now().UTC().Format("2006-01-02")
|
||||||
|
errs := processDir(context.Background(), cfg, date)
|
||||||
|
assert.Empty(t, errs)
|
||||||
|
|
||||||
|
assert.Zero(t, completeCalls, "tunnel-candidates file must never reach the LLM extraction pipeline")
|
||||||
|
|
||||||
|
// File must be left alone in raw/ — not moved to processed/, no marker written.
|
||||||
|
_, err := os.Stat(tunnelFile + ".processed")
|
||||||
|
assert.True(t, os.IsNotExist(err), "tunnel-candidates file should not get a .processed marker")
|
||||||
|
_, err = os.Stat(filepath.Join(brainDir, "raw", "processed", date, "tunnel-candidates-2026-07-19.md"))
|
||||||
|
assert.True(t, os.IsNotExist(err), "tunnel-candidates file should not be copied to processed/")
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,109 @@
|
|||||||
|
// Package webhook triggers an on-demand brain-sync Job when Gitea pushes to
|
||||||
|
// mathias/brain, instead of waiting for the next 15-minute CronJob poll.
|
||||||
|
package webhook
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/hmac"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
|
||||||
|
batchv1 "k8s.io/api/batch/v1"
|
||||||
|
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||||
|
"k8s.io/client-go/kubernetes"
|
||||||
|
)
|
||||||
|
|
||||||
|
// VerifySignature checks a Gitea webhook's X-Gitea-Signature header: a
|
||||||
|
// hex-encoded HMAC-SHA256 of the raw request body, keyed by the shared
|
||||||
|
// webhook secret. Constant-time compare — timing must not leak how much of
|
||||||
|
// the signature matched.
|
||||||
|
func VerifySignature(payload []byte, signatureHeader, secret string) bool {
|
||||||
|
if signatureHeader == "" {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
mac := hmac.New(sha256.New, []byte(secret))
|
||||||
|
mac.Write(payload)
|
||||||
|
expected := hex.EncodeToString(mac.Sum(nil))
|
||||||
|
return hmac.Equal([]byte(expected), []byte(signatureHeader))
|
||||||
|
}
|
||||||
|
|
||||||
|
// pushEvent is the subset of Gitea's push webhook payload this handler needs.
|
||||||
|
type pushEvent struct {
|
||||||
|
Ref string `json:"ref"`
|
||||||
|
Repo struct {
|
||||||
|
FullName string `json:"full_name"`
|
||||||
|
} `json:"repository"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// TriggerJobFromCronJob reads the named CronJob's job template and creates a
|
||||||
|
// new, uniquely-named Job from it — the same thing `kubectl create job
|
||||||
|
// --from=cronjob/<name>` does. Reuses the CronJob's already-tested script
|
||||||
|
// rather than re-implementing sync logic here.
|
||||||
|
func TriggerJobFromCronJob(ctx context.Context, cs kubernetes.Interface, namespace, cronJobName string) (string, error) {
|
||||||
|
cj, err := cs.BatchV1().CronJobs(namespace).Get(ctx, cronJobName, metav1.GetOptions{})
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("get cronjob %s/%s: %w", namespace, cronJobName, err)
|
||||||
|
}
|
||||||
|
job := &batchv1.Job{
|
||||||
|
ObjectMeta: metav1.ObjectMeta{
|
||||||
|
GenerateName: cronJobName + "-webhook-",
|
||||||
|
Namespace: namespace,
|
||||||
|
Annotations: map[string]string{
|
||||||
|
"triggered-by": "brain-webhook",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
Spec: cj.Spec.JobTemplate.Spec,
|
||||||
|
}
|
||||||
|
created, err := cs.BatchV1().Jobs(namespace).Create(ctx, job, metav1.CreateOptions{})
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("create job from cronjob %s/%s: %w", namespace, cronJobName, err)
|
||||||
|
}
|
||||||
|
return created.Name, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Handler is the HTTP handler for Gitea's push webhook on mathias/brain.
|
||||||
|
type Handler struct {
|
||||||
|
Secret string
|
||||||
|
Clientset kubernetes.Interface
|
||||||
|
Namespace string // e.g. "brain"
|
||||||
|
CronJobName string // e.g. "brain-sync"
|
||||||
|
WatchRepo string // e.g. "mathias/brain"
|
||||||
|
Logger *slog.Logger
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||||
|
body, err := io.ReadAll(r.Body)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "bad body", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
sig := r.Header.Get("X-Gitea-Signature")
|
||||||
|
if !VerifySignature(body, sig, h.Secret) {
|
||||||
|
http.Error(w, "bad signature", http.StatusUnauthorized)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var ev pushEvent
|
||||||
|
if err := json.Unmarshal(body, &ev); err != nil {
|
||||||
|
http.Error(w, "bad payload", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if ev.Repo.FullName != h.WatchRepo || ev.Ref != "refs/heads/main" {
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
_, _ = fmt.Fprintf(w, "ignored: repo=%s ref=%s", ev.Repo.FullName, ev.Ref)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
jobName, err := TriggerJobFromCronJob(r.Context(), h.Clientset, h.Namespace, h.CronJobName)
|
||||||
|
if err != nil {
|
||||||
|
h.Logger.Error("webhook: trigger job failed", "err", err)
|
||||||
|
http.Error(w, "trigger failed", http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
h.Logger.Info("webhook: triggered brain-sync job", "job", jobName)
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
_, _ = fmt.Fprintf(w, "triggered %s", jobName)
|
||||||
|
}
|
||||||
@@ -0,0 +1,222 @@
|
|||||||
|
package webhook
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"crypto/hmac"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"encoding/json"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
batchv1 "k8s.io/api/batch/v1"
|
||||||
|
corev1 "k8s.io/api/core/v1"
|
||||||
|
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||||
|
"k8s.io/apimachinery/pkg/runtime"
|
||||||
|
"k8s.io/client-go/kubernetes/fake"
|
||||||
|
k8stesting "k8s.io/client-go/testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// newFakeClientset simulates the real API server's GenerateName expansion,
|
||||||
|
// which the plain fake clientset tracker does not do on its own -- without
|
||||||
|
// this, every created Job keeps an empty Name (a fake-clientset limitation,
|
||||||
|
// not real API server behavior).
|
||||||
|
func newFakeClientset(objects ...runtime.Object) *fake.Clientset {
|
||||||
|
cs := fake.NewSimpleClientset(objects...)
|
||||||
|
cs.PrependReactor("create", "jobs", func(action k8stesting.Action) (bool, runtime.Object, error) {
|
||||||
|
createAction := action.(k8stesting.CreateAction)
|
||||||
|
job, ok := createAction.GetObject().(*batchv1.Job)
|
||||||
|
if ok && job.Name == "" && job.GenerateName != "" {
|
||||||
|
job.Name = job.GenerateName + "test0001"
|
||||||
|
}
|
||||||
|
return false, nil, nil // not "handled" -- let the default reactor store it
|
||||||
|
})
|
||||||
|
return cs
|
||||||
|
}
|
||||||
|
|
||||||
|
func sign(t *testing.T, payload []byte, secret string) string {
|
||||||
|
t.Helper()
|
||||||
|
mac := hmac.New(sha256.New, []byte(secret))
|
||||||
|
mac.Write(payload)
|
||||||
|
return hex.EncodeToString(mac.Sum(nil))
|
||||||
|
}
|
||||||
|
|
||||||
|
func testLogger() *slog.Logger {
|
||||||
|
return slog.New(slog.NewTextHandler(os.Stderr, nil))
|
||||||
|
}
|
||||||
|
|
||||||
|
func fakeCronJob(namespace, name string) *batchv1.CronJob {
|
||||||
|
return &batchv1.CronJob{
|
||||||
|
ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: namespace},
|
||||||
|
Spec: batchv1.CronJobSpec{
|
||||||
|
JobTemplate: batchv1.JobTemplateSpec{
|
||||||
|
Spec: batchv1.JobSpec{
|
||||||
|
Template: corev1.PodTemplateSpec{
|
||||||
|
Spec: corev1.PodSpec{
|
||||||
|
Containers: []corev1.Container{
|
||||||
|
{Name: "sync", Image: "alpine/git:v2.47.2"},
|
||||||
|
},
|
||||||
|
RestartPolicy: corev1.RestartPolicyNever,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// --------------------------------------------------------------------------- #
|
||||||
|
// VerifySignature
|
||||||
|
// --------------------------------------------------------------------------- #
|
||||||
|
func TestVerifySignature_AcceptsCorrectHMAC(t *testing.T) {
|
||||||
|
payload := []byte(`{"ref":"refs/heads/main"}`)
|
||||||
|
secret := "s3cret"
|
||||||
|
sig := sign(t, payload, secret)
|
||||||
|
assert.True(t, VerifySignature(payload, sig, secret))
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVerifySignature_RejectsWrongSecret(t *testing.T) {
|
||||||
|
payload := []byte(`{"ref":"refs/heads/main"}`)
|
||||||
|
sig := sign(t, payload, "right-secret")
|
||||||
|
assert.False(t, VerifySignature(payload, sig, "wrong-secret"))
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVerifySignature_RejectsTamperedPayload(t *testing.T) {
|
||||||
|
secret := "s3cret"
|
||||||
|
sig := sign(t, []byte(`{"ref":"refs/heads/main"}`), secret)
|
||||||
|
assert.False(t, VerifySignature([]byte(`{"ref":"refs/heads/evil"}`), sig, secret))
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVerifySignature_RejectsEmptySignature(t *testing.T) {
|
||||||
|
assert.False(t, VerifySignature([]byte("payload"), "", "secret"))
|
||||||
|
}
|
||||||
|
|
||||||
|
// --------------------------------------------------------------------------- #
|
||||||
|
// TriggerJobFromCronJob
|
||||||
|
// --------------------------------------------------------------------------- #
|
||||||
|
func TestTriggerJobFromCronJob_CreatesJobMatchingTemplate(t *testing.T) {
|
||||||
|
cs := newFakeClientset(fakeCronJob("brain", "brain-sync"))
|
||||||
|
|
||||||
|
jobName, err := TriggerJobFromCronJob(context.Background(), cs, "brain", "brain-sync")
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.NotEmpty(t, jobName)
|
||||||
|
|
||||||
|
jobs, err := cs.BatchV1().Jobs("brain").List(context.Background(), metav1.ListOptions{})
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, jobs.Items, 1)
|
||||||
|
assert.Equal(t, "alpine/git:v2.47.2", jobs.Items[0].Spec.Template.Spec.Containers[0].Image)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTriggerJobFromCronJob_ErrorsWhenCronJobMissing(t *testing.T) {
|
||||||
|
cs := fake.NewSimpleClientset()
|
||||||
|
_, err := TriggerJobFromCronJob(context.Background(), cs, "brain", "brain-sync")
|
||||||
|
assert.Error(t, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// --------------------------------------------------------------------------- #
|
||||||
|
// Handler.ServeHTTP
|
||||||
|
// --------------------------------------------------------------------------- #
|
||||||
|
func newTestHandler(cs *fake.Clientset) *Handler {
|
||||||
|
return &Handler{
|
||||||
|
Secret: "s3cret",
|
||||||
|
Clientset: cs,
|
||||||
|
Namespace: "brain",
|
||||||
|
CronJobName: "brain-sync",
|
||||||
|
WatchRepo: "mathias/brain",
|
||||||
|
Logger: testLogger(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func pushPayload(t *testing.T, repo, ref string) []byte {
|
||||||
|
t.Helper()
|
||||||
|
body := map[string]any{
|
||||||
|
"ref": ref,
|
||||||
|
"repository": map[string]any{
|
||||||
|
"full_name": repo,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
b, err := json.Marshal(body)
|
||||||
|
require.NoError(t, err)
|
||||||
|
return b
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandler_RejectsMissingSignature(t *testing.T) {
|
||||||
|
cs := fake.NewSimpleClientset(fakeCronJob("brain", "brain-sync"))
|
||||||
|
h := newTestHandler(cs)
|
||||||
|
payload := pushPayload(t, "mathias/brain", "refs/heads/main")
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/webhooks/brain-sync", bytes.NewReader(payload))
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
assert.Equal(t, http.StatusUnauthorized, rec.Code)
|
||||||
|
jobs, _ := cs.BatchV1().Jobs("brain").List(context.Background(), metav1.ListOptions{})
|
||||||
|
assert.Empty(t, jobs.Items, "must not trigger a job on an unsigned request")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandler_RejectsWrongSignature(t *testing.T) {
|
||||||
|
cs := fake.NewSimpleClientset(fakeCronJob("brain", "brain-sync"))
|
||||||
|
h := newTestHandler(cs)
|
||||||
|
payload := pushPayload(t, "mathias/brain", "refs/heads/main")
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/webhooks/brain-sync", bytes.NewReader(payload))
|
||||||
|
req.Header.Set("X-Gitea-Signature", sign(t, payload, "not-the-real-secret"))
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
assert.Equal(t, http.StatusUnauthorized, rec.Code)
|
||||||
|
jobs, _ := cs.BatchV1().Jobs("brain").List(context.Background(), metav1.ListOptions{})
|
||||||
|
assert.Empty(t, jobs.Items)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandler_IgnoresOtherRepos(t *testing.T) {
|
||||||
|
cs := fake.NewSimpleClientset(fakeCronJob("brain", "brain-sync"))
|
||||||
|
h := newTestHandler(cs)
|
||||||
|
payload := pushPayload(t, "mathias/some-other-repo", "refs/heads/main")
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/webhooks/brain-sync", bytes.NewReader(payload))
|
||||||
|
req.Header.Set("X-Gitea-Signature", sign(t, payload, "s3cret"))
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
assert.Equal(t, http.StatusOK, rec.Code)
|
||||||
|
jobs, _ := cs.BatchV1().Jobs("brain").List(context.Background(), metav1.ListOptions{})
|
||||||
|
assert.Empty(t, jobs.Items, "must not trigger for a push to an unrelated repo")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandler_IgnoresNonMainBranch(t *testing.T) {
|
||||||
|
cs := fake.NewSimpleClientset(fakeCronJob("brain", "brain-sync"))
|
||||||
|
h := newTestHandler(cs)
|
||||||
|
payload := pushPayload(t, "mathias/brain", "refs/heads/some-feature-branch")
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/webhooks/brain-sync", bytes.NewReader(payload))
|
||||||
|
req.Header.Set("X-Gitea-Signature", sign(t, payload, "s3cret"))
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
assert.Equal(t, http.StatusOK, rec.Code)
|
||||||
|
jobs, _ := cs.BatchV1().Jobs("brain").List(context.Background(), metav1.ListOptions{})
|
||||||
|
assert.Empty(t, jobs.Items, "must not trigger for a push to a non-main branch")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandler_TriggersJobOnValidMainPush(t *testing.T) {
|
||||||
|
cs := fake.NewSimpleClientset(fakeCronJob("brain", "brain-sync"))
|
||||||
|
h := newTestHandler(cs)
|
||||||
|
payload := pushPayload(t, "mathias/brain", "refs/heads/main")
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/webhooks/brain-sync", bytes.NewReader(payload))
|
||||||
|
req.Header.Set("X-Gitea-Signature", sign(t, payload, "s3cret"))
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
assert.Equal(t, http.StatusOK, rec.Code)
|
||||||
|
jobs, err := cs.BatchV1().Jobs("brain").List(context.Background(), metav1.ListOptions{})
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Len(t, jobs.Items, 1, "a valid push to main on the watched repo must trigger exactly one job")
|
||||||
|
}
|
||||||
@@ -6,6 +6,8 @@ import (
|
|||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
|
"github.com/mathiasbq/hyperguild/ingestion/internal/textutil"
|
||||||
)
|
)
|
||||||
|
|
||||||
// RebuildIndex writes brain/wiki/index.md from the current wiki contents.
|
// RebuildIndex writes brain/wiki/index.md from the current wiki contents.
|
||||||
@@ -62,10 +64,7 @@ func pageFirstSentence(brainDir string, e Entry) string {
|
|||||||
if line == "" || strings.HasPrefix(line, "#") {
|
if line == "" || strings.HasPrefix(line, "#") {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if len(line) > 100 {
|
return textutil.Truncate(line, 100)
|
||||||
return line[:100] + "…"
|
|
||||||
}
|
|
||||||
return line
|
|
||||||
}
|
}
|
||||||
return ""
|
return ""
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -13,7 +13,7 @@ import (
|
|||||||
"github.com/lestrrat-go/jwx/v2/jwa"
|
"github.com/lestrrat-go/jwx/v2/jwa"
|
||||||
"github.com/lestrrat-go/jwx/v2/jwk"
|
"github.com/lestrrat-go/jwx/v2/jwk"
|
||||||
"github.com/lestrrat-go/jwx/v2/jwt"
|
"github.com/lestrrat-go/jwx/v2/jwt"
|
||||||
"github.com/mathiasbq/supervisor/internal/auth"
|
"git.d-ma.be/mathias/hyperguild/internal/auth"
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ import (
|
|||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/mathiasbq/supervisor/internal/auth"
|
"git.d-ma.be/mathias/hyperguild/internal/auth"
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ import (
|
|||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/mathiasbq/supervisor/internal/brain"
|
"git.d-ma.be/mathias/hyperguild/internal/brain"
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -3,7 +3,7 @@ package config_test
|
|||||||
import (
|
import (
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/mathiasbq/supervisor/internal/config"
|
"git.d-ma.be/mathias/hyperguild/internal/config"
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ import (
|
|||||||
"path/filepath"
|
"path/filepath"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/mathiasbq/supervisor/internal/config"
|
"git.d-ma.be/mathias/hyperguild/internal/config"
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -3,7 +3,7 @@ package config_test
|
|||||||
import (
|
import (
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/mathiasbq/supervisor/internal/config"
|
"git.d-ma.be/mathias/hyperguild/internal/config"
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
iexec "github.com/mathiasbq/supervisor/internal/exec"
|
iexec "git.d-ma.be/mathias/hyperguild/internal/exec"
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ import (
|
|||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/mathiasbq/supervisor/internal/githubclient"
|
"git.d-ma.be/mathias/hyperguild/internal/githubclient"
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -8,8 +8,8 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"github.com/mathiasbq/supervisor/internal/auth"
|
"git.d-ma.be/mathias/hyperguild/internal/auth"
|
||||||
"github.com/mathiasbq/supervisor/internal/registry"
|
"git.d-ma.be/mathias/hyperguild/internal/registry"
|
||||||
)
|
)
|
||||||
|
|
||||||
type request struct {
|
type request struct {
|
||||||
|
|||||||
@@ -8,8 +8,8 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/mathiasbq/supervisor/internal/mcp"
|
"git.d-ma.be/mathias/hyperguild/internal/mcp"
|
||||||
"github.com/mathiasbq/supervisor/internal/registry"
|
"git.d-ma.be/mathias/hyperguild/internal/registry"
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ import (
|
|||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/mathiasbq/supervisor/internal/mcpclient"
|
"git.d-ma.be/mathias/hyperguild/internal/mcpclient"
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ import (
|
|||||||
"encoding/json"
|
"encoding/json"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/mathiasbq/supervisor/internal/registry"
|
"git.d-ma.be/mathias/hyperguild/internal/registry"
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -1,21 +0,0 @@
|
|||||||
package routing
|
|
||||||
|
|
||||||
import (
|
|
||||||
"crypto/sha256"
|
|
||||||
"encoding/binary"
|
|
||||||
)
|
|
||||||
|
|
||||||
// CanonicalHash returns a deterministic 64-bit hash of (system, user).
|
|
||||||
// Used to make sample-band routing decisions reproducible: identical input
|
|
||||||
// strings produce the same hash on every call, independent of process state.
|
|
||||||
//
|
|
||||||
// Inputs are joined with a 0x00 byte separator before hashing — distinguishes
|
|
||||||
// (system="ab", user="cd") from (system="abcd", user="").
|
|
||||||
func CanonicalHash(system, user string) uint64 {
|
|
||||||
h := sha256.New()
|
|
||||||
h.Write([]byte(system))
|
|
||||||
h.Write([]byte{0})
|
|
||||||
h.Write([]byte(user))
|
|
||||||
sum := h.Sum(nil)
|
|
||||||
return binary.BigEndian.Uint64(sum[:8])
|
|
||||||
}
|
|
||||||
@@ -1,46 +0,0 @@
|
|||||||
package routing_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/mathiasbq/supervisor/internal/routing"
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestCanonicalHashDeterministic(t *testing.T) {
|
|
||||||
a := routing.CanonicalHash("system one", "user one")
|
|
||||||
b := routing.CanonicalHash("system one", "user one")
|
|
||||||
assert.Equal(t, a, b, "same inputs must produce same hash")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestCanonicalHashDistinguishesInputs(t *testing.T) {
|
|
||||||
cases := [][2]string{
|
|
||||||
{"sys", "user"},
|
|
||||||
{"sys", "user2"},
|
|
||||||
{"sys2", "user"},
|
|
||||||
{"", "system\x00user"}, // separator collision attempt
|
|
||||||
{"system\x00user", ""},
|
|
||||||
}
|
|
||||||
seen := make(map[uint64]bool)
|
|
||||||
for _, c := range cases {
|
|
||||||
h := routing.CanonicalHash(c[0], c[1])
|
|
||||||
assert.False(t, seen[h], "collision on %v", c)
|
|
||||||
seen[h] = true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestCanonicalHashLowBitDistribution(t *testing.T) {
|
|
||||||
// Sanity check: across 1000 distinct inputs, low-bit split is roughly even.
|
|
||||||
zeros, ones := 0, 0
|
|
||||||
for i := 0; i < 1000; i++ {
|
|
||||||
h := routing.CanonicalHash("sys", string(rune('a'+(i%26)))+string(rune(i)))
|
|
||||||
if h&1 == 0 {
|
|
||||||
zeros++
|
|
||||||
} else {
|
|
||||||
ones++
|
|
||||||
}
|
|
||||||
}
|
|
||||||
// Allow ±15% deviation from 500/500. Tighter would be flaky on real data.
|
|
||||||
assert.InDelta(t, 500, zeros, 150)
|
|
||||||
assert.InDelta(t, 500, ones, 150)
|
|
||||||
}
|
|
||||||
@@ -1,92 +0,0 @@
|
|||||||
package routing
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"context"
|
|
||||||
"encoding/json"
|
|
||||||
"fmt"
|
|
||||||
"net/http"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
// LogEntry describes a single routing decision to log via the brain MCP.
|
|
||||||
type LogEntry struct {
|
|
||||||
SessionID string
|
|
||||||
Skill string // the original skill the call routed (e.g., "review")
|
|
||||||
Decision string // "local" or "thinking" or "thinking_fallback"
|
|
||||||
Message string // free-form, e.g. "model=qwen35, pass_rate=0.94"
|
|
||||||
ProjectRoot string
|
|
||||||
DurationMs int64
|
|
||||||
Failed bool // true → final_status: "fail"; false → "pass"
|
|
||||||
}
|
|
||||||
|
|
||||||
// Logger posts session_log entries to a brain MCP at BrainURL + /mcp.
|
|
||||||
type Logger struct {
|
|
||||||
BrainURL string
|
|
||||||
Token string // bearer for the (auth-gated) ingestion /mcp; empty = no header
|
|
||||||
HTTP *http.Client
|
|
||||||
}
|
|
||||||
|
|
||||||
// NewLogger creates a Logger with a 2-second HTTP timeout. token authenticates
|
|
||||||
// to the bearer-gated ingestion /mcp; an empty token sends no Authorization
|
|
||||||
// header (and silently 401s against a gated server — see brain
|
|
||||||
// mcpclient-empty-token-silent-401-envfrom-missing-key).
|
|
||||||
func NewLogger(brainURL, token string) *Logger {
|
|
||||||
return &Logger{
|
|
||||||
BrainURL: brainURL,
|
|
||||||
Token: token,
|
|
||||||
HTTP: &http.Client{Timeout: 2 * time.Second},
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// LogDecision posts a session_log MCP call. Errors are returned but the caller
|
|
||||||
// MUST NOT block real work on them — logging is best-effort.
|
|
||||||
func (l *Logger) LogDecision(ctx context.Context, e LogEntry) error {
|
|
||||||
// A completed routed call is a pass (liveness); only an execution error is a
|
|
||||||
// fail. There is no "skip" for routing — the prior default-to-"skip" meant a
|
|
||||||
// successful call never counted toward pass_rate, so the gate was unreachable.
|
|
||||||
status := "pass"
|
|
||||||
if e.Failed {
|
|
||||||
status = "fail"
|
|
||||||
}
|
|
||||||
payload := map[string]any{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 1,
|
|
||||||
"method": "tools/call",
|
|
||||||
"params": map[string]any{
|
|
||||||
"name": "session_log",
|
|
||||||
"arguments": map[string]any{
|
|
||||||
"session_id": e.SessionID,
|
|
||||||
// The real skill, so /pass-rate?skill=review|debug sees these
|
|
||||||
// records; routing decisions stay groupable via session_id "_routing".
|
|
||||||
"skill": e.Skill,
|
|
||||||
"phase": "decide",
|
|
||||||
"final_status": status,
|
|
||||||
"message": fmt.Sprintf("%s: %s — %s", e.Skill, e.Decision, e.Message),
|
|
||||||
"duration_ms": e.DurationMs,
|
|
||||||
"project_root": e.ProjectRoot,
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
body, err := json.Marshal(payload)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("log: marshal: %w", err)
|
|
||||||
}
|
|
||||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, l.BrainURL+"/mcp", bytes.NewReader(body))
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("log: build request: %w", err)
|
|
||||||
}
|
|
||||||
req.Header.Set("Content-Type", "application/json")
|
|
||||||
if l.Token != "" {
|
|
||||||
req.Header.Set("Authorization", "Bearer "+l.Token)
|
|
||||||
}
|
|
||||||
resp, err := l.HTTP.Do(req)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("log: request: %w", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = resp.Body.Close() }()
|
|
||||||
if resp.StatusCode != http.StatusOK {
|
|
||||||
return fmt.Errorf("log: server returned status %d", resp.StatusCode)
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
@@ -1,106 +0,0 @@
|
|||||||
package routing_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"encoding/json"
|
|
||||||
"io"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/mathiasbq/supervisor/internal/routing"
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestLoggerLogDecision(t *testing.T) {
|
|
||||||
var captured map[string]any
|
|
||||||
var authHeader string
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
||||||
assert.Equal(t, http.MethodPost, r.Method)
|
|
||||||
assert.Equal(t, "/mcp", r.URL.Path)
|
|
||||||
authHeader = r.Header.Get("Authorization")
|
|
||||||
body, _ := io.ReadAll(r.Body)
|
|
||||||
require.NoError(t, json.Unmarshal(body, &captured))
|
|
||||||
_ = json.NewEncoder(w).Encode(map[string]any{"jsonrpc": "2.0", "id": 1, "result": map[string]any{"content": []map[string]any{{"type": "text", "text": "ok"}}}})
|
|
||||||
}))
|
|
||||||
defer srv.Close()
|
|
||||||
|
|
||||||
l := routing.NewLogger(srv.URL, "test-token")
|
|
||||||
err := l.LogDecision(context.Background(), routing.LogEntry{
|
|
||||||
SessionID: "sess-1",
|
|
||||||
Skill: "review",
|
|
||||||
Decision: "local",
|
|
||||||
Message: "model=qwen36, pass_rate=0.94",
|
|
||||||
ProjectRoot: "/home/x/proj",
|
|
||||||
DurationMs: 1234,
|
|
||||||
Failed: false,
|
|
||||||
})
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
// Bug C fix: the POST authenticates to the bearer-gated ingestion /mcp.
|
|
||||||
assert.Equal(t, "Bearer test-token", authHeader)
|
|
||||||
|
|
||||||
params := captured["params"].(map[string]any)
|
|
||||||
assert.Equal(t, "tools/call", captured["method"])
|
|
||||||
assert.Equal(t, "session_log", params["name"])
|
|
||||||
|
|
||||||
args := params["arguments"].(map[string]any)
|
|
||||||
// Bug B fix: the record carries the real skill so /pass-rate?skill=review sees it.
|
|
||||||
assert.Equal(t, "review", args["skill"])
|
|
||||||
assert.Equal(t, "decide", args["phase"])
|
|
||||||
// Bug A fix: a successful routed call logs "pass", not "skip".
|
|
||||||
assert.Equal(t, "pass", args["final_status"])
|
|
||||||
assert.Contains(t, args["message"].(string), "review: local")
|
|
||||||
// session grouping is preserved via session_id.
|
|
||||||
assert.Equal(t, "sess-1", args["session_id"])
|
|
||||||
assert.Equal(t, "/home/x/proj", args["project_root"])
|
|
||||||
assert.Equal(t, float64(1234), args["duration_ms"])
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestLoggerLogFailure(t *testing.T) {
|
|
||||||
var captured map[string]any
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
||||||
body, _ := io.ReadAll(r.Body)
|
|
||||||
_ = json.Unmarshal(body, &captured)
|
|
||||||
_ = json.NewEncoder(w).Encode(map[string]any{"jsonrpc": "2.0", "id": 1, "result": map[string]any{}})
|
|
||||||
}))
|
|
||||||
defer srv.Close()
|
|
||||||
|
|
||||||
l := routing.NewLogger(srv.URL, "test-token")
|
|
||||||
err := l.LogDecision(context.Background(), routing.LogEntry{
|
|
||||||
SessionID: "s", Skill: "debug", Decision: "local", Message: "litellm down", Failed: true,
|
|
||||||
})
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
args := captured["params"].(map[string]any)["arguments"].(map[string]any)
|
|
||||||
assert.Equal(t, "debug", args["skill"])
|
|
||||||
assert.Equal(t, "fail", args["final_status"])
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestLoggerOmitsAuthWhenTokenEmpty(t *testing.T) {
|
|
||||||
var authHeader string
|
|
||||||
hasAuth := false
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
||||||
authHeader = r.Header.Get("Authorization")
|
|
||||||
_, hasAuth = r.Header["Authorization"]
|
|
||||||
_ = json.NewEncoder(w).Encode(map[string]any{"jsonrpc": "2.0", "id": 1, "result": map[string]any{}})
|
|
||||||
}))
|
|
||||||
defer srv.Close()
|
|
||||||
|
|
||||||
l := routing.NewLogger(srv.URL, "")
|
|
||||||
require.NoError(t, l.LogDecision(context.Background(), routing.LogEntry{Skill: "review", SessionID: "_routing", Decision: "local"}))
|
|
||||||
assert.False(t, hasAuth, "no Authorization header should be set when token is empty")
|
|
||||||
assert.Equal(t, "", authHeader)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestLoggerSurfacesUpstreamError(t *testing.T) {
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
|
||||||
http.Error(w, "down", http.StatusBadGateway)
|
|
||||||
}))
|
|
||||||
defer srv.Close()
|
|
||||||
|
|
||||||
l := routing.NewLogger(srv.URL, "test-token")
|
|
||||||
err := l.LogDecision(context.Background(), routing.LogEntry{Skill: "x", SessionID: "y", Decision: "local"})
|
|
||||||
require.Error(t, err)
|
|
||||||
}
|
|
||||||
@@ -1,85 +0,0 @@
|
|||||||
package routing
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"encoding/json"
|
|
||||||
"fmt"
|
|
||||||
"net/http"
|
|
||||||
"net/url"
|
|
||||||
"sync"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Fetcher reads /pass-rate from the brain pod with a per-skill TTL cache.
|
|
||||||
type Fetcher struct {
|
|
||||||
BaseURL string
|
|
||||||
Window string
|
|
||||||
TTL time.Duration
|
|
||||||
HTTP *http.Client
|
|
||||||
|
|
||||||
mu sync.Mutex
|
|
||||||
cache map[string]cachedRate
|
|
||||||
}
|
|
||||||
|
|
||||||
type cachedRate struct {
|
|
||||||
value *float64
|
|
||||||
at time.Time
|
|
||||||
}
|
|
||||||
|
|
||||||
type passRateResponse struct {
|
|
||||||
PassRate *float64 `json:"pass_rate"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// NewFetcher returns a Fetcher that calls baseURL + /pass-rate with the
|
|
||||||
// given window string. If ttl is zero, defaults to 60 seconds. The HTTP
|
|
||||||
// client uses a 1-second total timeout.
|
|
||||||
func NewFetcher(baseURL, window string, ttl time.Duration) *Fetcher {
|
|
||||||
if ttl == 0 {
|
|
||||||
ttl = 60 * time.Second
|
|
||||||
}
|
|
||||||
return &Fetcher{
|
|
||||||
BaseURL: baseURL,
|
|
||||||
Window: window,
|
|
||||||
TTL: ttl,
|
|
||||||
HTTP: &http.Client{Timeout: time.Second},
|
|
||||||
cache: make(map[string]cachedRate),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Get returns the pass rate for the named skill, or nil if no data exists,
|
|
||||||
// or an error if the brain is unreachable. Caches successful results.
|
|
||||||
func (f *Fetcher) Get(ctx context.Context, skill string) (*float64, error) {
|
|
||||||
f.mu.Lock()
|
|
||||||
if c, ok := f.cache[skill]; ok && time.Since(c.at) < f.TTL {
|
|
||||||
v := c.value
|
|
||||||
f.mu.Unlock()
|
|
||||||
return v, nil
|
|
||||||
}
|
|
||||||
f.mu.Unlock()
|
|
||||||
|
|
||||||
u := fmt.Sprintf("%s/pass-rate?skill=%s&window=%s",
|
|
||||||
f.BaseURL, url.QueryEscape(skill), url.QueryEscape(f.Window))
|
|
||||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("passrate: build request: %w", err)
|
|
||||||
}
|
|
||||||
resp, err := f.HTTP.Do(req)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("passrate: request: %w", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = resp.Body.Close() }()
|
|
||||||
if resp.StatusCode != http.StatusOK {
|
|
||||||
return nil, fmt.Errorf("passrate: server returned status %d", resp.StatusCode)
|
|
||||||
}
|
|
||||||
|
|
||||||
var body passRateResponse
|
|
||||||
if err := json.NewDecoder(resp.Body).Decode(&body); err != nil {
|
|
||||||
return nil, fmt.Errorf("passrate: decode: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
f.mu.Lock()
|
|
||||||
f.cache[skill] = cachedRate{value: body.PassRate, at: time.Now()}
|
|
||||||
f.mu.Unlock()
|
|
||||||
|
|
||||||
return body.PassRate, nil
|
|
||||||
}
|
|
||||||
@@ -1,94 +0,0 @@
|
|||||||
package routing_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"encoding/json"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"sync/atomic"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/mathiasbq/supervisor/internal/routing"
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestFetcherGetReturnsPassRate(t *testing.T) {
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
||||||
assert.Equal(t, http.MethodGet, r.Method)
|
|
||||||
assert.Equal(t, "/pass-rate", r.URL.Path)
|
|
||||||
assert.Equal(t, "tdd", r.URL.Query().Get("skill"))
|
|
||||||
assert.Equal(t, "7d", r.URL.Query().Get("window"))
|
|
||||||
w.Header().Set("Content-Type", "application/json")
|
|
||||||
_ = json.NewEncoder(w).Encode(map[string]any{"skill": "tdd", "pass_rate": 0.94})
|
|
||||||
}))
|
|
||||||
defer srv.Close()
|
|
||||||
|
|
||||||
f := routing.NewFetcher(srv.URL, "7d", time.Minute)
|
|
||||||
pr, err := f.Get(context.Background(), "tdd")
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.NotNil(t, pr)
|
|
||||||
assert.InDelta(t, 0.94, *pr, 1e-9)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFetcherGetReturnsNilWhenNoData(t *testing.T) {
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
||||||
_ = json.NewEncoder(w).Encode(map[string]any{"skill": "novel", "pass_rate": nil})
|
|
||||||
}))
|
|
||||||
defer srv.Close()
|
|
||||||
|
|
||||||
f := routing.NewFetcher(srv.URL, "7d", time.Minute)
|
|
||||||
pr, err := f.Get(context.Background(), "novel")
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Nil(t, pr)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFetcherCachesWithinTTL(t *testing.T) {
|
|
||||||
var calls int32
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
|
||||||
atomic.AddInt32(&calls, 1)
|
|
||||||
_ = json.NewEncoder(w).Encode(map[string]any{"pass_rate": 0.5})
|
|
||||||
}))
|
|
||||||
defer srv.Close()
|
|
||||||
|
|
||||||
f := routing.NewFetcher(srv.URL, "7d", time.Minute)
|
|
||||||
for i := 0; i < 5; i++ {
|
|
||||||
_, err := f.Get(context.Background(), "tdd")
|
|
||||||
require.NoError(t, err)
|
|
||||||
}
|
|
||||||
assert.Equal(t, int32(1), atomic.LoadInt32(&calls), "should hit upstream once and serve four times from cache")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFetcherFetchesAgainAfterTTLExpires(t *testing.T) {
|
|
||||||
var calls int32
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
|
||||||
atomic.AddInt32(&calls, 1)
|
|
||||||
_ = json.NewEncoder(w).Encode(map[string]any{"pass_rate": 0.5})
|
|
||||||
}))
|
|
||||||
defer srv.Close()
|
|
||||||
|
|
||||||
// Tight TTL so the test stays fast.
|
|
||||||
f := routing.NewFetcher(srv.URL, "7d", 5*time.Millisecond)
|
|
||||||
_, err := f.Get(context.Background(), "tdd")
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Equal(t, int32(1), atomic.LoadInt32(&calls))
|
|
||||||
|
|
||||||
// Sleep past TTL, then a second Get should hit upstream again.
|
|
||||||
time.Sleep(15 * time.Millisecond)
|
|
||||||
_, err = f.Get(context.Background(), "tdd")
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Equal(t, int32(2), atomic.LoadInt32(&calls), "expected fresh upstream call after TTL expiry")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFetcherSurfacesUpstreamError(t *testing.T) {
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
|
||||||
http.Error(w, "boom", http.StatusInternalServerError)
|
|
||||||
}))
|
|
||||||
defer srv.Close()
|
|
||||||
|
|
||||||
f := routing.NewFetcher(srv.URL, "7d", time.Minute)
|
|
||||||
pr, err := f.Get(context.Background(), "tdd")
|
|
||||||
require.Error(t, err)
|
|
||||||
assert.Nil(t, pr)
|
|
||||||
}
|
|
||||||
@@ -1,47 +0,0 @@
|
|||||||
package routing
|
|
||||||
|
|
||||||
// Decision is the route picked for a single skill call.
|
|
||||||
type Decision int
|
|
||||||
|
|
||||||
const (
|
|
||||||
DecideLocal Decision = iota
|
|
||||||
DecideClaude
|
|
||||||
)
|
|
||||||
|
|
||||||
func (d Decision) String() string {
|
|
||||||
if d == DecideLocal {
|
|
||||||
return "local"
|
|
||||||
}
|
|
||||||
return "claude"
|
|
||||||
}
|
|
||||||
|
|
||||||
// Policy holds the floor/ceil thresholds for routing decisions.
|
|
||||||
//
|
|
||||||
// Rules (in order):
|
|
||||||
//
|
|
||||||
// 1. passRate == nil → DecideLocal (default-to-local for cost-routable skills)
|
|
||||||
// 2. *passRate >= Floor → DecideLocal (trust local)
|
|
||||||
// 3. *passRate < Ceil → DecideClaude (don't trust local)
|
|
||||||
// 4. otherwise (sample band) → requestHash low bit picks: 0=local, 1=claude
|
|
||||||
type Policy struct {
|
|
||||||
Floor float64
|
|
||||||
Ceil float64
|
|
||||||
}
|
|
||||||
|
|
||||||
// Decide returns the routing decision for a single call.
|
|
||||||
// requestHash is consulted only when passRate is in the sample band [Ceil, Floor).
|
|
||||||
func (p Policy) Decide(passRate *float64, requestHash uint64) Decision {
|
|
||||||
if passRate == nil {
|
|
||||||
return DecideLocal
|
|
||||||
}
|
|
||||||
if *passRate >= p.Floor {
|
|
||||||
return DecideLocal
|
|
||||||
}
|
|
||||||
if *passRate < p.Ceil {
|
|
||||||
return DecideClaude
|
|
||||||
}
|
|
||||||
if requestHash&1 == 0 {
|
|
||||||
return DecideLocal
|
|
||||||
}
|
|
||||||
return DecideClaude
|
|
||||||
}
|
|
||||||
@@ -1,36 +0,0 @@
|
|||||||
package routing_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/mathiasbq/supervisor/internal/routing"
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
)
|
|
||||||
|
|
||||||
func ptr(f float64) *float64 { return &f }
|
|
||||||
|
|
||||||
func TestPolicyDecide(t *testing.T) {
|
|
||||||
p := routing.Policy{Floor: 0.9, Ceil: 0.7}
|
|
||||||
|
|
||||||
cases := []struct {
|
|
||||||
name string
|
|
||||||
passRate *float64
|
|
||||||
hash uint64
|
|
||||||
want routing.Decision
|
|
||||||
}{
|
|
||||||
{"null pass rate → local", nil, 0, routing.DecideLocal},
|
|
||||||
{"null pass rate, hash irrelevant → local", nil, 0xDEADBEEF, routing.DecideLocal},
|
|
||||||
{"at floor → local", ptr(0.9), 0, routing.DecideLocal},
|
|
||||||
{"above floor → local", ptr(0.95), 0, routing.DecideLocal},
|
|
||||||
{"below ceil → claude", ptr(0.5), 0, routing.DecideClaude},
|
|
||||||
{"at ceil → sample-band even-hash → local", ptr(0.7), 0, routing.DecideLocal},
|
|
||||||
{"sample band, even hash → local", ptr(0.8), 2, routing.DecideLocal},
|
|
||||||
{"sample band, odd hash → claude", ptr(0.8), 3, routing.DecideClaude},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tc := range cases {
|
|
||||||
t.Run(tc.name, func(t *testing.T) {
|
|
||||||
assert.Equal(t, tc.want, p.Decide(tc.passRate, tc.hash))
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,84 +0,0 @@
|
|||||||
package routing
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"fmt"
|
|
||||||
"log/slog"
|
|
||||||
)
|
|
||||||
|
|
||||||
// CompleteFunc matches the signature used by every skill package's Config.
|
|
||||||
type CompleteFunc func(ctx context.Context, model, system, user string) (string, int64, error)
|
|
||||||
|
|
||||||
// RunInput captures the per-call inputs the dispatch wrapper needs.
|
|
||||||
type RunInput struct {
|
|
||||||
Skill string
|
|
||||||
System string
|
|
||||||
User string
|
|
||||||
SessionID string
|
|
||||||
ProjectRoot string
|
|
||||||
}
|
|
||||||
|
|
||||||
// Router composes a pass-rate fetcher, a decision policy, a session logger,
|
|
||||||
// and a LiteLLM client. Skill packages receive Router.Run as their CompleteFunc.
|
|
||||||
type Router struct {
|
|
||||||
Fetcher *Fetcher
|
|
||||||
Logger *Logger
|
|
||||||
Policy Policy
|
|
||||||
FastModel string
|
|
||||||
ThinkingModel string
|
|
||||||
Complete CompleteFunc
|
|
||||||
}
|
|
||||||
|
|
||||||
// Run executes one skill call: decides local vs claude, calls LiteLLM, logs the
|
|
||||||
// decision. On local-side error, falls open by retrying once on the Claude model.
|
|
||||||
func (r *Router) Run(ctx context.Context, in RunInput) (string, int64, error) {
|
|
||||||
pr, ferr := r.Fetcher.Get(ctx, in.Skill)
|
|
||||||
if ferr != nil {
|
|
||||||
slog.Warn("router: pass-rate unreachable, defaulting to local", "skill", in.Skill, "err", ferr)
|
|
||||||
pr = nil
|
|
||||||
}
|
|
||||||
hash := CanonicalHash(in.System, in.User)
|
|
||||||
decision := r.Policy.Decide(pr, hash)
|
|
||||||
|
|
||||||
model := r.ThinkingModel
|
|
||||||
if decision == DecideLocal {
|
|
||||||
model = r.FastModel
|
|
||||||
}
|
|
||||||
|
|
||||||
out, ms, err := r.Complete(ctx, model, in.System, in.User)
|
|
||||||
if lerr := r.Logger.LogDecision(ctx, LogEntry{
|
|
||||||
SessionID: in.SessionID,
|
|
||||||
Skill: in.Skill,
|
|
||||||
Decision: decision.String(),
|
|
||||||
Message: fmt.Sprintf("model=%s, pass_rate=%s", model, formatPassRate(pr)),
|
|
||||||
ProjectRoot: in.ProjectRoot,
|
|
||||||
DurationMs: ms,
|
|
||||||
Failed: err != nil,
|
|
||||||
}); lerr != nil {
|
|
||||||
slog.Warn("router: log decision failed", "skill", in.Skill, "err", lerr)
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil && decision == DecideLocal {
|
|
||||||
slog.Warn("router: fast failed, falling open to thinking model", "skill", in.Skill, "err", err)
|
|
||||||
out, ms, err = r.Complete(ctx, r.ThinkingModel, in.System, in.User)
|
|
||||||
if lerr := r.Logger.LogDecision(ctx, LogEntry{
|
|
||||||
SessionID: in.SessionID,
|
|
||||||
Skill: in.Skill,
|
|
||||||
Decision: "thinking_fallback",
|
|
||||||
Message: fmt.Sprintf("model=%s, after-fast-error", r.ThinkingModel),
|
|
||||||
ProjectRoot: in.ProjectRoot,
|
|
||||||
DurationMs: ms,
|
|
||||||
Failed: err != nil,
|
|
||||||
}); lerr != nil {
|
|
||||||
slog.Warn("router: log decision failed", "skill", in.Skill, "err", lerr)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return out, ms, err
|
|
||||||
}
|
|
||||||
|
|
||||||
func formatPassRate(pr *float64) string {
|
|
||||||
if pr == nil {
|
|
||||||
return "null"
|
|
||||||
}
|
|
||||||
return fmt.Sprintf("%.2f", *pr)
|
|
||||||
}
|
|
||||||
@@ -1,136 +0,0 @@
|
|||||||
package routing_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"sync"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/mathiasbq/supervisor/internal/routing"
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
)
|
|
||||||
|
|
||||||
type fakeLLM struct {
|
|
||||||
mu sync.Mutex
|
|
||||||
calls []struct{ Model, System, User string }
|
|
||||||
resp string
|
|
||||||
err error
|
|
||||||
errOn string // if non-empty, only the named model errors
|
|
||||||
}
|
|
||||||
|
|
||||||
func (f *fakeLLM) Complete(_ context.Context, model, system, user string) (string, int64, error) {
|
|
||||||
f.mu.Lock()
|
|
||||||
defer f.mu.Unlock()
|
|
||||||
f.calls = append(f.calls, struct{ Model, System, User string }{model, system, user})
|
|
||||||
if f.errOn == model {
|
|
||||||
return "", 0, f.err
|
|
||||||
}
|
|
||||||
if f.err != nil && f.errOn == "" {
|
|
||||||
return "", 0, f.err
|
|
||||||
}
|
|
||||||
return f.resp, 100, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func newRouter(t *testing.T, llm *fakeLLM, passRate float64) (*routing.Router, *httptest.Server, *httptest.Server) {
|
|
||||||
t.Helper()
|
|
||||||
brain := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
||||||
switch r.URL.Path {
|
|
||||||
case "/pass-rate":
|
|
||||||
_ = json.NewEncoder(w).Encode(map[string]any{"pass_rate": passRate})
|
|
||||||
case "/mcp":
|
|
||||||
_ = json.NewEncoder(w).Encode(map[string]any{"jsonrpc": "2.0", "id": 1, "result": map[string]any{}})
|
|
||||||
}
|
|
||||||
}))
|
|
||||||
t.Cleanup(brain.Close)
|
|
||||||
|
|
||||||
r := &routing.Router{
|
|
||||||
Fetcher: routing.NewFetcher(brain.URL, "7d", time.Minute),
|
|
||||||
Logger: routing.NewLogger(brain.URL, ""),
|
|
||||||
Policy: routing.Policy{Floor: 0.9, Ceil: 0.7},
|
|
||||||
FastModel: "koala/qwen35-9b-fast",
|
|
||||||
ThinkingModel: "iguana/gemma4-26b",
|
|
||||||
Complete: llm.Complete,
|
|
||||||
}
|
|
||||||
return r, brain, brain
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRouterRoutesLocalAtHighPassRate(t *testing.T) {
|
|
||||||
llm := &fakeLLM{resp: "ok"}
|
|
||||||
r, _, _ := newRouter(t, llm, 0.95)
|
|
||||||
|
|
||||||
out, _, err := r.Run(context.Background(), routing.RunInput{
|
|
||||||
Skill: "review", System: "sys", User: "user", SessionID: "s1", ProjectRoot: "/p",
|
|
||||||
})
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Equal(t, "ok", out)
|
|
||||||
|
|
||||||
llm.mu.Lock()
|
|
||||||
defer llm.mu.Unlock()
|
|
||||||
require.Len(t, llm.calls, 1)
|
|
||||||
assert.Equal(t, "koala/qwen35-9b-fast", llm.calls[0].Model)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRouterRoutesThinkingAtLowPassRate(t *testing.T) {
|
|
||||||
llm := &fakeLLM{resp: "ok"}
|
|
||||||
r, _, _ := newRouter(t, llm, 0.3)
|
|
||||||
|
|
||||||
_, _, err := r.Run(context.Background(), routing.RunInput{
|
|
||||||
Skill: "review", System: "sys", User: "user", SessionID: "s2",
|
|
||||||
})
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
llm.mu.Lock()
|
|
||||||
defer llm.mu.Unlock()
|
|
||||||
require.Len(t, llm.calls, 1)
|
|
||||||
assert.Equal(t, "iguana/gemma4-26b", llm.calls[0].Model)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRouterFailsOpenFastErrorToThinking(t *testing.T) {
|
|
||||||
llm := &fakeLLM{resp: "ok-after-fallback", err: errors.New("fast boom"), errOn: "koala/qwen35-9b-fast"}
|
|
||||||
r, _, _ := newRouter(t, llm, 0.95) // would route fast
|
|
||||||
|
|
||||||
out, _, err := r.Run(context.Background(), routing.RunInput{
|
|
||||||
Skill: "review", System: "sys", User: "user", SessionID: "s3",
|
|
||||||
})
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Equal(t, "ok-after-fallback", out)
|
|
||||||
|
|
||||||
llm.mu.Lock()
|
|
||||||
defer llm.mu.Unlock()
|
|
||||||
require.Len(t, llm.calls, 2)
|
|
||||||
assert.Equal(t, "koala/qwen35-9b-fast", llm.calls[0].Model)
|
|
||||||
assert.Equal(t, "iguana/gemma4-26b", llm.calls[1].Model)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRouterDefaultsToFastWhenBrainUnreachable(t *testing.T) {
|
|
||||||
// Brain returns 500 → fetcher errors → router treats pass rate as nil → fast.
|
|
||||||
brain := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
|
||||||
http.Error(w, "down", http.StatusInternalServerError)
|
|
||||||
}))
|
|
||||||
defer brain.Close()
|
|
||||||
|
|
||||||
llm := &fakeLLM{resp: "ok"}
|
|
||||||
r := &routing.Router{
|
|
||||||
Fetcher: routing.NewFetcher(brain.URL, "7d", time.Minute),
|
|
||||||
Logger: routing.NewLogger(brain.URL, ""),
|
|
||||||
Policy: routing.Policy{Floor: 0.9, Ceil: 0.7},
|
|
||||||
FastModel: "koala/qwen35-9b-fast",
|
|
||||||
ThinkingModel: "iguana/gemma4-26b",
|
|
||||||
Complete: llm.Complete,
|
|
||||||
}
|
|
||||||
|
|
||||||
_, _, err := r.Run(context.Background(), routing.RunInput{
|
|
||||||
Skill: "review", System: "sys", User: "user", SessionID: "s4",
|
|
||||||
})
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
llm.mu.Lock()
|
|
||||||
defer llm.mu.Unlock()
|
|
||||||
require.Len(t, llm.calls, 1)
|
|
||||||
assert.Equal(t, "koala/qwen35-9b-fast", llm.calls[0].Model)
|
|
||||||
}
|
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user