Compare commits

...
Author SHA1 Message Date
mathiasandClaude Opus 5.5 cda18e6db8 fix(ingestion): an unlistable brain root is a scan error, not clean
CI / Lint / Test / Vet (pull_request) Successful in 44s
CI / Mirror to GitHub (pull_request) Skipped
PR #159 round 2: with the brain root at mode 0100 (enterable, not
listable) WalkDir failed at the root, Scan discarded the error and
reported clean, and startup passed while search, which goes straight
into wiki/, served a planted symlink. Scan now returns the root's walk
error, so startup refuses on anything but a missing root. Also pins
that a cleared set is reported again when it reappears.

Refs #157

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 17:14:46 +02:00
mathiasandClaude Opus 5.5 956ad8b9fb fix(ingestion): close the PR #159 review findings on the symlink tripwire
CI / Lint / Test / Vet (pull_request) Successful in 35s
CI / Mirror to GitHub (pull_request) Skipped
- Startup passed on ANY scan error: WalkDir stopped at the first
  unreadable dir and Scan discarded what it had found, while search
  kept walking and served a planted symlink. Scan now records
  unreadable dirs and keeps going; startup refuses on symlinks or
  unreadable dirs and only lets a missing brain dir pass.
- Watch went silent while a scan error persisted, and marked a set as
  reported before Notify, so an ntfy outage lost the alert. It now
  alerts on unreadable dirs and scan errors too, and records a set as
  reported only after Notify succeeds (retried each tick).
- Only the root .git is skipped; a nested .git is scanned.
- A non-positive BRAIN_SYMLINK_SCAN_INTERVAL fell into time.NewTicker
  and crashlooped the server; it now falls back to the default.
- Findings are quoted in alerts, so a newline in a name cannot merge
  two sets; alertingWarnings names the tripwire when ntfy is unset.
- Test names match what they assert (two-set dedupe, symlinked entries
  inside a symlinked root, nested .git).

Flipped: startup ignoring unreadable dirs, .git skipped at any depth,
and no retry each turn a test red. Watch tests stable over -count=10
and under -race.

Refs #157

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 17:11:45 +02:00
mathiasandClaude Opus 5.5 796d226497 feat(ingestion): symlink tripwire for the Syncthing-shared brain dir
CI / Lint / Test / Vet (pull_request) Successful in 34s
CI / Mirror to GitHub (pull_request) Skipped
The brain dir is a Syncthing sendreceive folder and Syncthing syncs
symlinks; search, vectorstore, context and graph readers follow one out
of the brain and would index or return what it points to (reproduced
with search.Query). Decision (#157): a tripwire now, brain-wide os.Root
readers next.

- symlinkguard.Scan lists every symlink inside the brain dir (skipping
  .git; the brain dir itself may be a mount symlink).
- At startup the server refuses to start while any exists, naming them.
- symlinkguard.Watch rescans every BRAIN_SYMLINK_SCAN_INTERVAL seconds
  (default 300) and sends one ntfy alert per new set of symlinks.

The live brain dir had 0 symlinks on 2026-10-03, checked before this
change, so deploying it does not trip.

Refs #157

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 17:04:41 +02:00
mathias cfe3f24e25 fix(ingestion): authenticate the REST API; confine ingest paths to brain/raw/ (#149) (#152)
CI / Lint / Test / Vet (push) Successful in 34s
CI / Mirror to GitHub (push) Successful in 4s
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-03 14:38:07 +00:00
mathiasandClaude Opus 5.5 a6da93332d feat(brain): audit-log every tools/call; server-owned write provenance (#147)
CI / Lint / Test / Vet (push) Successful in 39s
CI / Mirror to GitHub (push) Successful in 4s
Closes #144 (audit line per tools/call) and #145 (origin/written_by stamped on every model-reachable write, surfaced in brain_query/brain_get).

Passed pr-gatekeeper after four rounds of adversarial review, verified by execution: #147

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-01 21:06:36 +00:00
mathiasandClaude Opus 5.5 f21be1aef6 chore(context): re-sync adapters after root brain-scoping correction
CI / Lint / Test / Vet (push) Successful in 43s
CI / Mirror to GitHub (push) Successful in 4s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 20:42:10 +02:00
48 changed files with 2323 additions and 83 deletions
+5 -1
View File
@@ -64,7 +64,11 @@ the routing pod; brain tools moved to the brain MCP.
The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`, The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`,
`/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300 `/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300
for shell scripts and non-MCP clients. for shell scripts and non-MCP clients, **with the same bearer token as `/mcp`**
(`BRAIN_MCP_TOKEN`; hyperguild#149). It is not served on the public host
(infra#517): use the tailnet (`koala:8443`) or in-cluster
`ingestion.supervisor.svc:3300`. `/ingest-path` and `brain_ingest` `path`
accept only paths inside `brain/raw/`.
`brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai `brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai
gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title, gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title,
+13 -3
View File
@@ -304,10 +304,16 @@ entries that age well are about *why*, *how to avoid*, and *what to do when*.
|---------|-------|-------| |---------|-------|-------|
| **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool | | **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool |
| **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same | | **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same |
| **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` | | **Anything HTTP-only (curl, scripts)** | Tailnet only: `POST https://koala.tail30472e.ts.net:8443/query` with `{"query":"..."}`, sending `BRAIN_MCP_TOKEN` (not yet enforced on REST, hyperguild#149). The public `brain-mcp.d-ma.be` serves `/mcp` only (infra#517). | `POST .../write` with `{"content":"...","wing":"...","hall":"..."}` |
| **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files | | **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files |
- **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`. - **Scoping — the brain holds no confidential client content.** Nothing on the read path
filters by client: every token holder can read every wing, and `wing` on `brain_query`
is an optional argument, not an access control. Client confidentiality is enforced only
at write time (`capture`'s I1 gate refuses internal/confidential). So never `brain_write`
client material; generalise it first or keep it in the client's own repo. (hyperguild#144,
decided 2026-10-01. This line previously claimed client projects were filtered; they
were not.)
- **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as - **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as
fallback. Both are configurable in the `supervisor/ingestion-deployment.yaml` fallback. Both are configurable in the `supervisor/ingestion-deployment.yaml`
on the koala k3s cluster; don't hardcode local-only model names into the on the koala k3s cluster; don't hardcode local-only model names into the
@@ -487,7 +493,11 @@ the routing pod; brain tools moved to the brain MCP.
The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`, The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`,
`/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300 `/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300
for shell scripts and non-MCP clients. for shell scripts and non-MCP clients, **with the same bearer token as `/mcp`**
(`BRAIN_MCP_TOKEN`; hyperguild#149). It is not served on the public host
(infra#517): use the tailnet (`koala:8443`) or in-cluster
`ingestion.supervisor.svc:3300`. `/ingest-path` and `brain_ingest` `path`
accept only paths inside `brain/raw/`.
`brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai `brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai
gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title, gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title,
+13 -3
View File
@@ -299,10 +299,16 @@ entries that age well are about *why*, *how to avoid*, and *what to do when*.
|---------|-------|-------| |---------|-------|-------|
| **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool | | **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool |
| **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same | | **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same |
| **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` | | **Anything HTTP-only (curl, scripts)** | Tailnet only: `POST https://koala.tail30472e.ts.net:8443/query` with `{"query":"..."}`, sending `BRAIN_MCP_TOKEN` (not yet enforced on REST, hyperguild#149). The public `brain-mcp.d-ma.be` serves `/mcp` only (infra#517). | `POST .../write` with `{"content":"...","wing":"...","hall":"..."}` |
| **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files | | **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files |
- **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`. - **Scoping — the brain holds no confidential client content.** Nothing on the read path
filters by client: every token holder can read every wing, and `wing` on `brain_query`
is an optional argument, not an access control. Client confidentiality is enforced only
at write time (`capture`'s I1 gate refuses internal/confidential). So never `brain_write`
client material; generalise it first or keep it in the client's own repo. (hyperguild#144,
decided 2026-10-01. This line previously claimed client projects were filtered; they
were not.)
- **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as - **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as
fallback. Both are configurable in the `supervisor/ingestion-deployment.yaml` fallback. Both are configurable in the `supervisor/ingestion-deployment.yaml`
on the koala k3s cluster; don't hardcode local-only model names into the on the koala k3s cluster; don't hardcode local-only model names into the
@@ -482,7 +488,11 @@ the routing pod; brain tools moved to the brain MCP.
The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`, The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`,
`/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300 `/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300
for shell scripts and non-MCP clients. for shell scripts and non-MCP clients, **with the same bearer token as `/mcp`**
(`BRAIN_MCP_TOKEN`; hyperguild#149). It is not served on the public host
(infra#517): use the tailnet (`koala:8443`) or in-cluster
`ingestion.supervisor.svc:3300`. `/ingest-path` and `brain_ingest` `path`
accept only paths inside `brain/raw/`.
`brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai `brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai
gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title, gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title,
+5 -1
View File
@@ -64,7 +64,11 @@ the routing pod; brain tools moved to the brain MCP.
The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`, The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`,
`/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300 `/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300
for shell scripts and non-MCP clients. for shell scripts and non-MCP clients, **with the same bearer token as `/mcp`**
(`BRAIN_MCP_TOKEN`; hyperguild#149). It is not served on the public host
(infra#517): use the tailnet (`koala:8443`) or in-cluster
`ingestion.supervisor.svc:3300`. `/ingest-path` and `brain_ingest` `path`
accept only paths inside `brain/raw/`.
`brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai `brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai
gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title, gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title,
+15 -5
View File
@@ -20,6 +20,7 @@ const defaultBrainURL = "http://koala:30330"
// shell-friendly client; MCP framing is unnecessary. // shell-friendly client; MCP framing is unnecessary.
type brainClient struct { type brainClient struct {
baseURL string baseURL string
token string // BRAIN_MCP_TOKEN; the REST API requires it since hyperguild#149
http *http.Client http *http.Client
} }
@@ -30,6 +31,7 @@ func newBrainClient() *brainClient {
} }
return &brainClient{ return &brainClient{
baseURL: u, baseURL: u,
token: os.Getenv("BRAIN_MCP_TOKEN"),
http: &http.Client{Timeout: 5 * time.Second}, http: &http.Client{Timeout: 5 * time.Second},
} }
} }
@@ -62,8 +64,7 @@ func (c *brainClient) Query(ctx context.Context, topic string, limit int) (*Quer
return nil, fmt.Errorf("build request: %w", err) return nil, fmt.Errorf("build request: %w", err)
} }
req.Header.Set("Content-Type", "application/json") req.Header.Set("Content-Type", "application/json")
resp, err := c.do(req)
resp, err := c.http.Do(req)
if err != nil { if err != nil {
return nil, fmt.Errorf("brain POST /query: %w", err) return nil, fmt.Errorf("brain POST /query: %w", err)
} }
@@ -105,8 +106,7 @@ func (c *brainClient) Write(ctx context.Context, wing, hall, filename string, co
return nil, fmt.Errorf("build request: %w", err) return nil, fmt.Errorf("build request: %w", err)
} }
req.Header.Set("Content-Type", "application/json") req.Header.Set("Content-Type", "application/json")
resp, err := c.do(req)
resp, err := c.http.Do(req)
if err != nil { if err != nil {
return nil, fmt.Errorf("brain POST /write: %w", err) return nil, fmt.Errorf("brain POST /write: %w", err)
} }
@@ -143,7 +143,7 @@ func (c *brainClient) PassRate(ctx context.Context, skill, window string) (*Pass
if err != nil { if err != nil {
return nil, fmt.Errorf("build request: %w", err) return nil, fmt.Errorf("build request: %w", err)
} }
resp, err := c.http.Do(req) resp, err := c.do(req)
if err != nil { if err != nil {
return nil, fmt.Errorf("brain GET /pass-rate: %w", err) return nil, fmt.Errorf("brain GET /pass-rate: %w", err)
} }
@@ -158,3 +158,13 @@ func (c *brainClient) PassRate(ctx context.Context, skill, window string) (*Pass
} }
return &out, nil return &out, nil
} }
// do sends every brain request, adding the bearer token when one is
// configured. All methods go through here so none can forget it (PassRate
// did, PR #152 review).
func (c *brainClient) do(req *http.Request) (*http.Response, error) {
if c.token != "" {
req.Header.Set("Authorization", "Bearer "+c.token)
}
return c.http.Do(req)
}
+57
View File
@@ -0,0 +1,57 @@
package main
import (
"context"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// hyperguild#149: the ingestion REST API now requires a bearer token; the CLI
// sends BRAIN_MCP_TOKEN, or it gets 401 on every brain call.
func TestBrainClientSendsBrainMCPTokenOnQueryWriteAndPassRate(t *testing.T) {
var got []string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
got = append(got, r.URL.Path+" "+r.Header.Get("Authorization"))
switch r.URL.Path {
case "/write":
_, _ = w.Write([]byte(`{"path":"wiki/x/facts/y.md"}`))
return
case "/pass-rate":
_, _ = w.Write([]byte(`{}`))
return
}
_, _ = w.Write([]byte(`{"results":[]}`))
}))
defer srv.Close()
t.Setenv("BRAIN_URL", srv.URL)
t.Setenv("BRAIN_MCP_TOKEN", "cli-test-token")
c := newBrainClient()
_, err := c.Query(context.Background(), "x", 1)
require.NoError(t, err)
_, err = c.Write(context.Background(), "x", "facts", "y", strings.NewReader("# y"))
require.NoError(t, err)
// PR #152 review: PassRate sent no token and would 401 in production.
_, _ = c.PassRate(context.Background(), "tdd", "7d")
assert.Equal(t, []string{"/query Bearer cli-test-token", "/write Bearer cli-test-token",
"/pass-rate Bearer cli-test-token"}, got)
}
func TestBrainClientSendsNoAuthorizationWhenTokenUnset(t *testing.T) {
var auth string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
auth = r.Header.Get("Authorization")
_, _ = w.Write([]byte(`{"results":[]}`))
}))
defer srv.Close()
t.Setenv("BRAIN_URL", srv.URL)
t.Setenv("BRAIN_MCP_TOKEN", "")
_, err := newBrainClient().Query(context.Background(), "x", 1)
require.NoError(t, err)
assert.Empty(t, auth)
}
@@ -24,7 +24,7 @@ Only a `tools/call` to the routing pod records a pass/fail.
|---------|-----|------| |---------|-----|------|
| Routing MCP (local, Tailscale) | `http://koala:30310/mcp` | Bearer `ROUTING_MCP_TOKEN` | | Routing MCP (local, Tailscale) | `http://koala:30310/mcp` | Bearer `ROUTING_MCP_TOKEN` |
| Routing MCP (remote) | `https://routing-mcp.d-ma.be/mcp` | OAuth via `auth.d-ma.be` (audience `claude-ai`) | | Routing MCP (remote) | `https://routing-mcp.d-ma.be/mcp` | OAuth via `auth.d-ma.be` (audience `claude-ai`) |
| Pass-rate readout | `http://koala:30330/pass-rate?skill=<name>` | none (read-only) | | Pass-rate readout | `http://koala:30330/pass-rate?skill=<name>` | Bearer `BRAIN_MCP_TOKEN` (since hyperguild#149) |
Tools advertised: **`review`**, **`debug`** (the two the #35 gate measures), Tools advertised: **`review`**, **`debug`** (the two the #35 gate measures),
plus `session_log`, `retrospective`, `trainer`. plus `session_log`, `retrospective`, `trainer`.
+28 -20
View File
@@ -33,6 +33,7 @@ import (
"github.com/mathiasbq/hyperguild/ingestion/internal/reranker" "github.com/mathiasbq/hyperguild/ingestion/internal/reranker"
"github.com/mathiasbq/hyperguild/ingestion/internal/search" "github.com/mathiasbq/hyperguild/ingestion/internal/search"
"github.com/mathiasbq/hyperguild/ingestion/internal/startupretry" "github.com/mathiasbq/hyperguild/ingestion/internal/startupretry"
"github.com/mathiasbq/hyperguild/ingestion/internal/symlinkguard"
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore" "github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
"github.com/mathiasbq/hyperguild/ingestion/internal/watcher" "github.com/mathiasbq/hyperguild/ingestion/internal/watcher"
"github.com/mathiasbq/hyperguild/ingestion/internal/webhook" "github.com/mathiasbq/hyperguild/ingestion/internal/webhook"
@@ -173,7 +174,8 @@ func alertingWarnings(getenv func(string) string) []string {
var out []string var out []string
if getenv("BRAIN_NTFY_URL") == "" { if getenv("BRAIN_NTFY_URL") == "" {
out = append(out, "alerting DISABLED: BRAIN_NTFY_URL unset — "+ out = append(out, "alerting DISABLED: BRAIN_NTFY_URL unset — "+
"brain_edit_root governance rewrites and capture audit degradation will not notify") "brain_edit_root governance rewrites, capture audit degradation and the "+
"brain symlink tripwire (hyperguild#157) will not notify")
} }
return out return out
} }
@@ -239,6 +241,10 @@ func main() {
} }
brainDir := envOr("INGEST_BRAIN_DIR", "../brain") brainDir := envOr("INGEST_BRAIN_DIR", "../brain")
if err := startupSymlinkCheck(brainDir); err != nil {
logger.Error("brain symlink tripwire", "err", err)
os.Exit(1)
}
port := envOr("INGEST_PORT", "3300") port := envOr("INGEST_PORT", "3300")
llmURL := envOr("INGEST_LLM_URL", "http://iguana:4000/v1") llmURL := envOr("INGEST_LLM_URL", "http://iguana:4000/v1")
@@ -360,6 +366,8 @@ func main() {
} }
ctx := context.Background() ctx := context.Background()
// hyperguild#157: report a symlink that appears in the brain while running.
go symlinkguard.Watch(ctx, brainDir, symlinkScanInterval(os.Getenv), buildNotifier())
// Built before the watcher, the graph backfill and session ingestion, and // Built before the watcher, the graph backfill and session ingestion, and
// retried rather than fatal (#83). Two separate reasons: // retried rather than fatal (#83). Two separate reasons:
@@ -503,20 +511,10 @@ func main() {
logger.Info("embed sync started", "interval_s", embedSyncInterval) logger.Info("embed sync started", "interval_s", embedSyncInterval)
} }
mux := http.NewServeMux() // Public routes only; buildMux mounts the authenticated REST API (#149).
mux.HandleFunc("POST /query", h.Query) public := map[string]http.Handler{}
mux.HandleFunc("POST /write", h.Write)
mux.HandleFunc("POST /index", h.Index)
mux.HandleFunc("POST /ingest", h.Ingest)
mux.HandleFunc("POST /ingest-path", h.IngestPath)
mux.HandleFunc("POST /ingest-raw", h.IngestRaw)
mux.HandleFunc("POST /backfill-refs", h.BackfillRefs)
mux.HandleFunc("GET /pending", h.Pending)
mux.HandleFunc("POST /promote", h.Promote)
mux.HandleFunc("POST /backfill-embeddings", h.BackfillEmbeddings)
mux.HandleFunc("GET /pass-rate", h.PassRate)
if webhookHandler != nil { if webhookHandler != nil {
mux.Handle("POST /webhooks/brain-sync", webhookHandler) public["POST /webhooks/brain-sync"] = webhookHandler
} }
// Resource-metadata URL is only emitted on 401 when Dex OAuth is // Resource-metadata URL is only emitted on 401 when Dex OAuth is
// configured. Static-Bearer-only deployments leave this empty so // configured. Static-Bearer-only deployments leave this empty so
@@ -524,14 +522,14 @@ func main() {
var resourceMetadataURL string var resourceMetadataURL string
if dexURL := os.Getenv("DEX_ISSUER_URL"); dexURL != "" { if dexURL := os.Getenv("DEX_ISSUER_URL"); dexURL != "" {
resourceURL := os.Getenv("MCP_RESOURCE_URL") resourceURL := os.Getenv("MCP_RESOURCE_URL")
mux.HandleFunc("GET /.well-known/oauth-protected-resource", public["GET /.well-known/oauth-protected-resource"] = http.HandlerFunc(
chassisauth.ProtectedResourceHandler(resourceURL, dexURL)) chassisauth.ProtectedResourceHandler(resourceURL, dexURL))
if resourceURL != "" { if resourceURL != "" {
resourceMetadataURL = strings.TrimRight(resourceURL, "/") + "/.well-known/oauth-protected-resource" resourceMetadataURL = strings.TrimRight(resourceURL, "/") + "/.well-known/oauth-protected-resource"
} }
} }
mux.Handle("/mcp", chassisauth.BearerMiddleware(mcpToken, jwtValidator, "brain", resourceMetadataURL, mcpSrv)) public["/mcp"] = chassisauth.BearerMiddleware(mcpToken, jwtValidator, "brain", resourceMetadataURL, mcpSrv)
// POST /capture (#53/#54): the uniform capture REST door. Needs a ticket // POST /capture (#53/#54): the uniform capture REST door. Needs a ticket
// tracker to file action items, so it only mounts when Gitea is // tracker to file action items, so it only mounts when Gitea is
@@ -553,7 +551,7 @@ func main() {
sovereign := splitList(os.Getenv("BRAIN_CAPTURE_SOVEREIGN_PRINCIPALS")) sovereign := splitList(os.Getenv("BRAIN_CAPTURE_SOVEREIGN_PRINCIPALS"))
resolver := capturehttp.NewOriginResolver(sovereign) resolver := capturehttp.NewOriginResolver(sovereign)
captureH := capturehttp.New(captureSvc, jwtValidator, mcpToken, "local-cli", resolver) captureH := capturehttp.New(captureSvc, jwtValidator, mcpToken, "local-cli", resolver)
mux.Handle("POST /capture", captureH) public["POST /capture"] = captureH
// Same use-case behind the MCP `capture` tool (#55 relay) so MCP-native // Same use-case behind the MCP `capture` tool (#55 relay) so MCP-native
// harnesses (claude.ai, Crush, Pi, LLM Council) reach capture through // harnesses (claude.ai, Crush, Pi, LLM Council) reach capture through
// the existing /mcp OAuth connector. mcpSrv is already wrapped above; // the existing /mcp OAuth connector. mcpSrv is already wrapped above;
@@ -583,9 +581,9 @@ func main() {
logger.Error("OAUTH_CLIENT_ID/SECRET set but MCP_RESOURCE_URL is empty; cannot derive issuer") logger.Error("OAUTH_CLIENT_ID/SECRET set but MCP_RESOURCE_URL is empty; cannot derive issuer")
os.Exit(1) os.Exit(1)
} }
mux.HandleFunc("GET /.well-known/oauth-authorization-server", public["GET /.well-known/oauth-authorization-server"] = http.HandlerFunc(
oauth.MetadataHandler(issuer)) oauth.MetadataHandler(issuer))
mux.HandleFunc("POST /oauth/token", oauth.TokenHandler(oauth.TokenConfig{ public["POST /oauth/token"] = http.HandlerFunc(oauth.TokenHandler(oauth.TokenConfig{
ClientID: oauthID, ClientID: oauthID,
ClientSecret: oauthSecret, ClientSecret: oauthSecret,
AccessToken: mcpToken, AccessToken: mcpToken,
@@ -604,7 +602,17 @@ func main() {
// real request latency. /metrics itself is excluded from its own // real request latency. /metrics itself is excluded from its own
// observation by registering it on the outer mux (post-wrap). // observation by registering it on the outer mux (post-wrap).
reg := metrics.New() reg := metrics.New()
mux.HandleFunc("GET /metrics", reg.Handler()) public["GET /metrics"] = http.HandlerFunc(reg.Handler())
// The whole router, built in one place so the test of buildMux is a test
// of production wiring (PR #152 review). The REST API's credential is the
// same as /mcp's.
mux, err := buildMux(restAuth{token: mcpToken, validator: jwtValidator, realm: "brain",
resourceMetadataURL: resourceMetadataURL}, h, public)
if err != nil {
logger.Error("build router", "err", err)
os.Exit(1)
}
logger.Info("metrics endpoint registered", "path", "/metrics") logger.Info("metrics endpoint registered", "path", "/metrics")
addr := ":" + port addr := ":" + port
+77
View File
@@ -0,0 +1,77 @@
package main
import (
"errors"
"fmt"
"net/http"
chassisauth "git.d-ma.be/mathias/mcp-chassis/auth"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
)
// publicRoutes is the complete set of patterns allowed on the outer mux,
// outside the REST API's auth (hyperguild#149). Each authenticates itself or is
// meant to be open: /mcp (bearer/JWT), /capture (bearer/JWT), the webhook
// (HMAC), /oauth/token (client secret), OAuth metadata and /metrics (open).
// A REST pattern added here would beat the authenticated catch-all by
// specificity and ship open, which is why buildMux refuses anything else.
var publicRoutes = map[string]bool{
"/mcp": true,
"POST /capture": true,
"POST /webhooks/brain-sync": true,
"GET /.well-known/oauth-protected-resource": true,
"GET /.well-known/oauth-authorization-server": true,
"POST /oauth/token": true,
"GET /metrics": true,
}
// restAuth is the credential the REST API checks: the same as /mcp's.
type restAuth struct {
token string
validator *chassisauth.JWTValidator
realm string
resourceMetadataURL string
}
// buildMux is the server's whole router. The REST API is mounted at "/"
// behind bearer auth that buildMux constructs itself, so a caller cannot pass
// a no-op; public handlers are registered as given, and only for patterns on
// the publicRoutes allowlist. It returns an http.Handler, not the
// *ServeMux, so nothing can register a route on the result afterwards and
// bypass the allowlist.
func buildMux(auth restAuth, h *api.Handler, public map[string]http.Handler) (http.Handler, error) {
if auth.token == "" {
return nil, errors.New("buildMux: REST auth needs a token")
}
mux := http.NewServeMux()
for pattern, handler := range public {
if !publicRoutes[pattern] {
return nil, fmt.Errorf("buildMux: %q is not an allowed public route", pattern)
}
mux.Handle(pattern, handler)
}
mountREST(mux, h, func(next http.Handler) http.Handler {
return chassisauth.BearerMiddleware(auth.token, auth.validator, auth.realm, auth.resourceMetadataURL, next)
})
return mux, nil
}
// mountREST registers the REST API on its own mux and mounts it at "/" behind
// auth: anything not registered as public lands here and is authenticated,
// including routes added later.
func mountREST(mux *http.ServeMux, h *api.Handler, auth func(http.Handler) http.Handler) {
rest := http.NewServeMux()
rest.HandleFunc("POST /query", h.Query)
rest.HandleFunc("POST /write", h.Write)
rest.HandleFunc("POST /index", h.Index)
rest.HandleFunc("POST /ingest", h.Ingest)
rest.HandleFunc("POST /ingest-path", h.IngestPath)
rest.HandleFunc("POST /ingest-raw", h.IngestRaw)
rest.HandleFunc("POST /backfill-refs", h.BackfillRefs)
rest.HandleFunc("GET /pending", h.Pending)
rest.HandleFunc("POST /promote", h.Promote)
rest.HandleFunc("POST /backfill-embeddings", h.BackfillEmbeddings)
rest.HandleFunc("GET /pass-rate", h.PassRate)
mux.Handle("/", auth(rest))
}
+106
View File
@@ -0,0 +1,106 @@
package main
import (
"log/slog"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
)
const routesTok = "routes-test-token"
func okHandler(body string) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _, _ = w.Write([]byte(body)) })
}
// testMux builds the router exactly as main does: buildMux with the full
// public set main registers (stub handlers) and real bearer config.
// hyperguild#149: the REST API is authenticated as a whole; the PR #152
// review showed a test of a COPY of the router let main.go regress unseen.
func testMux(t *testing.T) http.Handler {
t.Helper()
h := api.NewHandler(t.TempDir(), slog.New(slog.NewTextHandler(os.Stderr, nil)), pipeline.Config{})
public := map[string]http.Handler{}
for p := range publicRoutes {
public[p] = okHandler("public " + p)
}
mux, err := buildMux(restAuth{token: routesTok, realm: "brain"}, h, public)
require.NoError(t, err)
return mux
}
func do(t *testing.T, h http.Handler, method, path, authz string) (int, string) {
t.Helper()
req := httptest.NewRequest(method, path, strings.NewReader(`{"query":"x"}`))
if authz != "" {
req.Header.Set("Authorization", authz)
}
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
return rec.Code, rec.Body.String()
}
func TestBuildMuxRefusesRESTRoutesWithoutAValidToken(t *testing.T) {
r := testMux(t)
for _, rt := range []struct{ method, path string }{
{"POST", "/query"}, {"POST", "/write"}, {"POST", "/index"}, {"POST", "/ingest"},
{"POST", "/ingest-path"}, {"POST", "/ingest-raw"}, {"POST", "/backfill-refs"},
{"GET", "/pending"}, {"POST", "/promote"}, {"POST", "/backfill-embeddings"}, {"GET", "/pass-rate"},
} {
code, _ := do(t, r, rt.method, rt.path, "")
assert.Equal(t, http.StatusUnauthorized, code, "%s %s without a token", rt.method, rt.path)
code, _ = do(t, r, rt.method, rt.path, "Bearer wrong")
assert.Equal(t, http.StatusUnauthorized, code, "%s %s with a wrong token", rt.method, rt.path)
}
}
func TestBuildMuxServesRESTRoutesToAValidToken(t *testing.T) {
r := testMux(t)
code, _ := do(t, r, "GET", "/pending", "Bearer "+routesTok)
assert.Equal(t, http.StatusOK, code)
}
func TestBuildMuxAuthenticatesAnUnregisteredPath(t *testing.T) {
code, _ := do(t, testMux(t), "POST", "/some-future-route", "")
assert.Equal(t, http.StatusUnauthorized, code)
}
// buildMux registers public handlers as given, without adding auth: each
// public route owns its own (bearer on /mcp, HMAC on the webhook, client
// secret on /oauth/token) or is meant to be open (/metrics, OAuth metadata).
func TestBuildMuxDoesNotWrapPublicRoutes(t *testing.T) {
r := testMux(t)
for p := range publicRoutes {
method, path, ok := strings.Cut(p, " ")
if !ok {
method, path = "GET", p
}
code, body := do(t, r, method, path, "")
assert.Equal(t, http.StatusOK, code, p)
assert.Equal(t, "public "+p, body, p)
}
}
// The allowlist is the control: a REST pattern registered as public would win
// over the authenticated catch-all (more specific pattern), so it is refused.
func TestBuildMuxRefusesAPublicPatternOffTheAllowlist(t *testing.T) {
h := api.NewHandler(t.TempDir(), slog.New(slog.NewTextHandler(os.Stderr, nil)), pipeline.Config{})
for _, p := range []string{"POST /query", "GET /pending", "/", "POST /metrics"} {
_, err := buildMux(restAuth{token: routesTok, realm: "brain"}, h, map[string]http.Handler{p: okHandler("x")})
assert.Error(t, err, "public pattern %q must be refused", p)
}
}
func TestBuildMuxRefusesAnEmptyToken(t *testing.T) {
h := api.NewHandler(t.TempDir(), slog.New(slog.NewTextHandler(os.Stderr, nil)), pipeline.Config{})
_, err := buildMux(restAuth{token: "", realm: "brain"}, h, nil)
assert.Error(t, err)
}
+39
View File
@@ -0,0 +1,39 @@
package main
import (
"errors"
"fmt"
"os"
"strconv"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/symlinkguard"
)
// startupSymlinkCheck is the hyperguild#157 tripwire at boot: a symlink in the
// brain dir would let brain readers serve a file from outside the brain, so
// the server does not start until it is removed. A directory the scan cannot
// read could hide one, so it refuses on that too. A missing brain dir (first
// boot) is the only case it lets pass.
func startupSymlinkCheck(brainDir string) error {
r, err := symlinkguard.Scan(brainDir)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
return nil
}
return fmt.Errorf("refusing to start: cannot scan the brain dir %s (hyperguild#157): %w", brainDir, err)
}
if !r.Clean() {
return fmt.Errorf("refusing to start: brain dir %s has %s (hyperguild#157); fix them first", brainDir, r)
}
return nil
}
// symlinkScanInterval reads BRAIN_SYMLINK_SCAN_INTERVAL (seconds), falling
// back to the default on an unset, non-numeric or non-positive value.
func symlinkScanInterval(getenv func(string) string) time.Duration {
if s, err := strconv.Atoi(getenv("BRAIN_SYMLINK_SCAN_INTERVAL")); err == nil && s > 0 {
return time.Duration(s) * time.Second
}
return symlinkguard.DefaultInterval
}
@@ -0,0 +1,65 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// hyperguild#157: the server refuses to start while a symlink exists in the
// brain dir, or while any part of it cannot be scanned (which could hide
// one); a missing brain dir is the only thing it lets pass.
func TestStartupSymlinkCheckRefusesABrainWithASymlink(t *testing.T) {
brain := t.TempDir()
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, "planted")))
err := startupSymlinkCheck(brain)
require.Error(t, err)
assert.Contains(t, err.Error(), "planted")
}
func TestStartupSymlinkCheckRefusesABrainWithAnUnreadableDir(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root reads mode-000 directories")
}
brain := t.TempDir()
locked := filepath.Join(brain, "aaa")
require.NoError(t, os.MkdirAll(locked, 0o755))
require.NoError(t, os.Chmod(locked, 0o000))
t.Cleanup(func() { _ = os.Chmod(locked, 0o755) })
err := startupSymlinkCheck(brain)
require.Error(t, err)
assert.Contains(t, err.Error(), "aaa")
}
func TestStartupSymlinkCheckPassesACleanOrMissingBrain(t *testing.T) {
brain := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(brain, "n.md"), []byte("x"), 0o644))
assert.NoError(t, startupSymlinkCheck(brain))
assert.NoError(t, startupSymlinkCheck(filepath.Join(brain, "missing")))
}
func TestSymlinkScanIntervalFallsBackOnNonPositive(t *testing.T) {
for _, v := range []string{"0", "-5", "", "junk"} {
got := symlinkScanInterval(func(string) string { return v })
assert.Positive(t, got, "env %q", v)
}
}
func TestAlertingWarningsNamesTheSymlinkTripwire(t *testing.T) {
w := strings.Join(alertingWarnings(func(string) string { return "" }), " ")
assert.Contains(t, w, "symlink tripwire")
}
func TestStartupSymlinkCheckRefusesAnUnlistableBrainRoot(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root lists mode-0100 directories")
}
brain := t.TempDir()
require.NoError(t, os.Chmod(brain, 0o100))
t.Cleanup(func() { _ = os.Chmod(brain, 0o755) })
assert.Error(t, startupSymlinkCheck(brain))
}
+45 -7
View File
@@ -118,6 +118,11 @@ type WriteNoteOptions struct {
Wing string Wing string
Hall string Hall string
SourceType string // "internal" marks a first-party observation (e.g. claudewatcher) that needs no external citation SourceType string // "internal" marks a first-party observation (e.g. claudewatcher) that needs no external citation
// Origin/WrittenBy are server-owned provenance (hyperguild#145): any
// origin/written_by in Content's own frontmatter is discarded, so a
// caller cannot claim a more trusted origin than it authenticated as.
Origin string
WrittenBy string
} }
// WriteNote writes a markdown note into the brain. Returns the path // WriteNote writes a markdown note into the brain. Returns the path
@@ -169,9 +174,6 @@ func writeHallNote(brainDir string, opts WriteNoteOptions) (string, error) {
return "", fmt.Errorf("claim envelope: %w", err) return "", fmt.Errorf("claim envelope: %w", err)
} }
if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
return "", fmt.Errorf("create hall dir: %w", err)
}
emitted := make(map[string]bool, 6) emitted := make(map[string]bool, 6)
var fm strings.Builder var fm strings.Builder
@@ -219,7 +221,17 @@ func writeHallNote(brainDir string, opts WriteNoteOptions) (string, error) {
} }
fm.WriteString("---\n") fm.WriteString("---\n")
if err := os.WriteFile(dest, []byte(fm.String()+body), 0o644); err != nil { final, err := StampProvenance(fm.String()+body, opts.Origin, opts.WrittenBy)
if err != nil {
return "", err
}
// The directory is created only once the note is known to be writable:
// a refused write leaves nothing behind, not even an empty hall dir
// (brain#21; provenance refusal joins the envelope refusal here).
if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
return "", fmt.Errorf("create hall dir: %w", err)
}
if err := os.WriteFile(dest, []byte(final), 0o644); err != nil {
return "", fmt.Errorf("write: %w", err) return "", fmt.Errorf("write: %w", err)
} }
rel, _ := filepath.Rel(brainDir, dest) rel, _ := filepath.Rel(brainDir, dest)
@@ -295,7 +307,16 @@ func (h *Handler) Write(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, "invalid JSON") writeError(w, http.StatusBadRequest, "invalid JSON")
return return
} }
relPath, err := WriteNote(h.brainDir, WriteNoteOptions(req)) // Field by field, not a struct conversion: provenance is server-owned and
// this door carries no principal, so it stamps origin: unknown
// (hyperguild#145). A JSON body must never be able to set it.
relPath, err := WriteNote(h.brainDir, WriteNoteOptions{
Content: req.Content,
Filename: req.Filename,
Wing: req.Wing,
Hall: req.Hall,
SourceType: req.SourceType,
})
if err != nil { if err != nil {
h.logger.Error("write failed", "err", err) h.logger.Error("write failed", "err", err)
writeError(w, http.StatusBadRequest, err.Error()) writeError(w, http.StatusBadRequest, err.Error())
@@ -426,6 +447,12 @@ func (h *Handler) IngestPath(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, "path is required") writeError(w, http.StatusBadRequest, "path is required")
return return
} }
resolved, err := ResolveIngestPath(h.brainDir, req.Path)
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
req.Path = resolved
info, err := os.Stat(req.Path) info, err := os.Stat(req.Path)
if err != nil { if err != nil {
@@ -448,6 +475,12 @@ func (h *Handler) IngestPath(w http.ResponseWriter, r *http.Request) {
if !isSupportedExtension(ext) { if !isSupportedExtension(ext) {
return nil return nil
} }
// Each walked file is re-resolved: a symlinked file inside raw/
// would otherwise lead extract.Text out of the inbox (#149).
if _, rerr := ResolveIngestPath(h.brainDir, path); rerr != nil {
allWarnings = append(allWarnings, fmt.Sprintf("skipped %s: %v", path, rerr))
return nil
}
content, readErr := extract.Text(path) content, readErr := extract.Text(path)
if readErr != nil { if readErr != nil {
allWarnings = append(allWarnings, fmt.Sprintf("extract %s: %v", path, readErr)) allWarnings = append(allWarnings, fmt.Sprintf("extract %s: %v", path, readErr))
@@ -529,7 +562,8 @@ func (h *Handler) IngestRaw(w http.ResponseWriter, r *http.Request) {
return return
} }
result, err := pipeline.RunRaw(h.brainDir, req.Source, req.Pages, req.DryRun) // No principal on this door: pages are stamped origin: unknown (#145).
result, err := pipeline.RunRaw(r.Context(), h.brainDir, req.Source, req.Pages, req.DryRun)
if err != nil { if err != nil {
h.logger.Error("ingest-raw failed", "source", req.Source, "err", err) h.logger.Error("ingest-raw failed", "source", req.Source, "err", err)
writeError(w, http.StatusInternalServerError, "ingest error") writeError(w, http.StatusInternalServerError, "ingest error")
@@ -585,7 +619,11 @@ func (h *Handler) Promote(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, "invalid JSON") writeError(w, http.StatusBadRequest, "invalid JSON")
return return
} }
rel, err := PromoteNote(h.brainDir, PromoteOptions(req)) // Field by field: this door carries no principal, so the promoted note is
// stamped origin: unknown and the body cannot set provenance (#145).
rel, err := PromoteNote(h.brainDir, PromoteOptions{
Filename: req.Filename, Wing: req.Wing, Hall: req.Hall, Slug: req.Slug,
})
if err != nil { if err != nil {
writeError(w, http.StatusBadRequest, err.Error()) writeError(w, http.StatusBadRequest, err.Error())
return return
+6 -4
View File
@@ -361,10 +361,11 @@ func TestIngestPath_MissingPath(t *testing.T) {
} }
func TestIngestPath_File(t *testing.T) { func TestIngestPath_File(t *testing.T) {
_, h := setup(t) brainDir, h := setup(t)
// Create a temp file with content // Create a temp file with content
dir := t.TempDir() dir := filepath.Join(brainDir, "raw", "fixture") // ingest paths must live in brain/raw/ (#149)
require.NoError(t, os.MkdirAll(dir, 0o755))
f := filepath.Join(dir, "doc.md") f := filepath.Join(dir, "doc.md")
require.NoError(t, os.WriteFile(f, []byte("# Hello\nThis is markdown content."), 0o644)) require.NoError(t, os.WriteFile(f, []byte("# Hello\nThis is markdown content."), 0o644))
@@ -468,10 +469,11 @@ func TestIngestRaw_DryRun(t *testing.T) {
} }
func TestIngestPath_Directory(t *testing.T) { func TestIngestPath_Directory(t *testing.T) {
_, h := setup(t) brainDir, h := setup(t)
// Create a temp dir with one .md file // Create a temp dir with one .md file
dir := t.TempDir() dir := filepath.Join(brainDir, "raw", "fixture") // ingest paths must live in brain/raw/ (#149)
require.NoError(t, os.MkdirAll(dir, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "notes.md"), []byte("# Notes\nSome notes."), 0o644)) require.NoError(t, os.WriteFile(filepath.Join(dir, "notes.md"), []byte("# Notes\nSome notes."), 0o644))
body, _ := json.Marshal(map[string]any{ body, _ := json.Marshal(map[string]any{
@@ -40,9 +40,10 @@ func TestIngestPath_DocxSupportedWhenDocmarkConfigured(t *testing.T) {
t.Setenv("DOCMARK_URL", srv.URL+"/mcp") t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
t.Setenv("DOCMARK_BEARER_TOKEN", "tok") t.Setenv("DOCMARK_BEARER_TOKEN", "tok")
_, h := setup(t) brainDir, h := setup(t)
dir := t.TempDir() dir := filepath.Join(brainDir, "raw", "fixture") // ingest paths must live in brain/raw/ (#149)
require.NoError(t, os.MkdirAll(dir, 0o755))
f := filepath.Join(dir, "doc.docx") f := filepath.Join(dir, "doc.docx")
require.NoError(t, os.WriteFile(f, []byte("fake docx"), 0o644)) require.NoError(t, os.WriteFile(f, []byte("fake docx"), 0o644))
+38
View File
@@ -0,0 +1,38 @@
package api
import (
"errors"
"path/filepath"
"strings"
)
// ErrOutsideIngestRoot is the one refusal for every bad ingest path. It never
// carries the underlying error: "no such file" versus "permission denied"
// would let a caller probe which server files exist (PR #152 review).
var ErrOutsideIngestRoot = errors.New("ingest path must be an existing file or directory inside brain/raw/")
// ResolveIngestPath confines a caller-supplied ingest path to brain/raw/, the
// inbox the watcher already owns (hyperguild#149). Relative paths resolve
// against that root; symlinks are evaluated before the containment check, so
// a link inside raw/ that points elsewhere is refused. The path must exist.
func ResolveIngestPath(brainDir, p string) (string, error) {
if strings.TrimSpace(p) == "" {
return "", ErrOutsideIngestRoot
}
root, err := filepath.EvalSymlinks(filepath.Join(brainDir, "raw"))
if err != nil {
return "", ErrOutsideIngestRoot
}
if !filepath.IsAbs(p) {
p = filepath.Join(root, p)
}
real, err := filepath.EvalSymlinks(filepath.Clean(p))
if err != nil {
return "", ErrOutsideIngestRoot
}
rel, err := filepath.Rel(root, real)
if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) || filepath.IsAbs(rel) {
return "", ErrOutsideIngestRoot
}
return real, nil
}
@@ -0,0 +1,88 @@
package api_test
import (
"bytes"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
)
// hyperguild#149: /ingest-path and MCP brain_ingest read a caller-chosen
// server path, so any file the pod could read (the ServiceAccount token,
// /etc/passwd) could be pulled into the brain and read back out. Ingest paths
// now resolve inside brain/raw/ — the inbox the watcher already owns — with
// symlinks evaluated, or are refused.
func TestResolveIngestPathConfinesToRawInbox(t *testing.T) {
brain := t.TempDir()
raw := filepath.Join(brain, "raw")
require.NoError(t, os.MkdirAll(filepath.Join(raw, "sub"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(raw, "sub", "ok.md"), []byte("x"), 0o644))
outside := filepath.Join(t.TempDir(), "secret.txt")
require.NoError(t, os.WriteFile(outside, []byte("s"), 0o644))
require.NoError(t, os.Symlink(outside, filepath.Join(raw, "link.md")))
require.NoError(t, os.Symlink(filepath.Dir(outside), filepath.Join(raw, "dirlink")))
for _, ok := range []string{filepath.Join(raw, "sub", "ok.md"), "sub/ok.md", raw, "sub"} {
_, err := api.ResolveIngestPath(brain, ok)
assert.NoError(t, err, ok)
}
for _, bad := range []string{
"/etc/hostname", outside, "../secret.txt", filepath.Join(raw, "..", "wiki"),
filepath.Join(raw, "link.md"), filepath.Join(raw, "dirlink", "secret.txt"),
raw + "-evil/x.md", "",
} {
_, err := api.ResolveIngestPath(brain, bad)
assert.Error(t, err, "must refuse %q", bad)
}
}
func TestIngestPathRefusesAPathOutsideTheInbox(t *testing.T) {
_, h := setup(t)
body, _ := json.Marshal(map[string]any{"path": "/etc/hostname"})
rec := httptest.NewRecorder()
h.IngestPath(rec, httptest.NewRequest(http.MethodPost, "/ingest-path", bytes.NewReader(body)))
assert.Equal(t, http.StatusBadRequest, rec.Code)
assert.Contains(t, rec.Body.String(), "raw")
}
// A directory ingest must not follow a symlinked FILE inside raw/ out of it.
func TestIngestPathDirectoryWalkSkipsSymlinksLeavingTheInbox(t *testing.T) {
dir, h := setup(t)
raw := filepath.Join(dir, "raw", "batch")
require.NoError(t, os.MkdirAll(raw, 0o755))
outside := filepath.Join(t.TempDir(), "secret.md")
require.NoError(t, os.WriteFile(outside, []byte("# secret\n\nSENTINEL-149-OUTSIDE"), 0o644))
require.NoError(t, os.Symlink(outside, filepath.Join(raw, "leak.md")))
body, _ := json.Marshal(map[string]any{"path": "batch", "source": "walk"})
rec := httptest.NewRecorder()
h.IngestPath(rec, httptest.NewRequest(http.MethodPost, "/ingest-path", bytes.NewReader(body)))
assert.Contains(t, rec.Body.String(), "skipped")
assert.NotContains(t, rec.Body.String(), "SENTINEL-149-OUTSIDE")
}
// PR #152 review: the raw EvalSymlinks error told an authenticated caller
// whether any server file existed ("no such file") or was unreadable
// ("permission denied"), and leaked the brain directory. Every refusal now
// reads the same.
func TestResolveIngestPathRefusalsAreIndistinguishable(t *testing.T) {
brain := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(brain, "raw"), 0o755))
var msgs []string
for _, p := range []string{"/etc/hostname", "/etc/definitely-not-here-149", "../x", "nope/missing.md", ""} {
_, err := api.ResolveIngestPath(brain, p)
require.Error(t, err, p)
msgs = append(msgs, err.Error())
assert.NotContains(t, err.Error(), brain, "refusal for %q leaks the brain dir", p)
}
for _, m := range msgs[1:] {
assert.Equal(t, msgs[0], m)
}
}
+9 -1
View File
@@ -72,6 +72,10 @@ type PromoteOptions struct {
Wing string Wing string
Hall string Hall string
Slug string // optional; defaults to Filename minus date prefix + .md Slug string // optional; defaults to Filename minus date prefix + .md
// Origin/WrittenBy re-attribute the promoted note to the promoter: the raw
// note's own frontmatter claims are not provenance (hyperguild#145).
Origin string
WrittenBy string
} }
// PromoteNote moves a note from brain/raw/ into the structured wiki: it // PromoteNote moves a note from brain/raw/ into the structured wiki: it
@@ -128,7 +132,11 @@ func PromoteNote(brainDir string, opts PromoteOptions) (string, error) {
return "", fmt.Errorf("create wing dir: %w", err) return "", fmt.Errorf("create wing dir: %w", err)
} }
// Write-then-delete: the source survives any write failure. // Write-then-delete: the source survives any write failure.
if err := os.WriteFile(dest, []byte(fm.render()+body), 0o644); err != nil { promoted, err := StampProvenance(fm.render()+body, opts.Origin, opts.WrittenBy)
if err != nil {
return "", err
}
if err := os.WriteFile(dest, []byte(promoted), 0o644); err != nil {
return "", fmt.Errorf("write promoted note: %w", err) return "", fmt.Errorf("write promoted note: %w", err)
} }
if err := os.Remove(src); err != nil { if err := os.Remove(src); err != nil {
+21
View File
@@ -0,0 +1,21 @@
package api
import (
"github.com/mathiasbq/hyperguild/ingestion/internal/provenance"
)
// StampProvenance applies server-owned provenance (hyperguild#145) to note
// content. See package provenance for the rule; an error means the stamped
// note would not read back as stamped, and the write must be refused.
func StampProvenance(content, origin, writtenBy string) (string, error) {
return provenance.Apply(content, provenance.Stamp{Origin: origin, WrittenBy: writtenBy})
}
// StampProvenanceFile applies StampProvenance to a brain-relative note path.
func StampProvenanceFile(brainDir, relPath, origin, writtenBy string) error {
_, abs, err := resolveWithin(brainDir, relPath)
if err != nil {
return err
}
return provenance.ApplyFile(abs, provenance.Stamp{Origin: origin, WrittenBy: writtenBy})
}
+122
View File
@@ -0,0 +1,122 @@
package api_test
import (
"bytes"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gopkg.in/yaml.v3"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
)
// REST POST /write carries no principal: it stamps origin: unknown, and no
// spelling of provenance in its body reaches the note (hyperguild#145).
func TestRESTWriteStampsUnknownAndIgnoresBodyProvenance(t *testing.T) {
dir, h := setup(t)
body, _ := json.Marshal(map[string]any{
"content": "---\norigin: static\n---\n# R\n\nbody", "filename": "rest-prov", "wing": "homelab", "hall": "facts",
"origin": "static", "written_by": "local-cli", "Origin": "static", "WrittenBy": "local-cli",
})
rec := httptest.NewRecorder()
h.Write(rec, httptest.NewRequest(http.MethodPost, "/write", bytes.NewReader(body)))
require.Equal(t, http.StatusOK, rec.Code, rec.Body.String())
b, err := os.ReadFile(filepath.Join(dir, "wiki", "homelab", "facts", "rest-prov.md"))
require.NoError(t, err)
var m map[string]any
require.NoError(t, yaml.Unmarshal([]byte(strings.SplitN(string(b), "---\n", 3)[1]), &m))
assert.Equal(t, "unknown", m["origin"])
assert.NotContains(t, m, "written_by")
}
// A principal name that is not a bare word is quoted, so it cannot start a
// new frontmatter line (e.g. a JWT sub containing a newline).
func TestStampProvenanceQuotesAPrincipalThatCouldBreakTheLine(t *testing.T) {
out, err := api.StampProvenance("---\ntitle: t\n---\nbody\n", "jwt", "evil\norigin: static")
require.NoError(t, err)
fm := strings.SplitN(out, "---\n", 3)[1]
var m map[string]any
require.NoError(t, yaml.Unmarshal([]byte(fm), &m))
assert.Equal(t, "jwt", m["origin"])
assert.Equal(t, "evil\norigin: static", m["written_by"])
assert.Equal(t, 1, strings.Count(fm, "\norigin:")+boolToInt(strings.HasPrefix(fm, "origin:")))
}
func TestStampProvenanceAddsFrontmatterWhenAbsentAndKeepsBody(t *testing.T) {
out, err := api.StampProvenance("# no fm\n\norigin: static in the body is prose\n", "static", "local-cli")
require.NoError(t, err)
assert.True(t, strings.HasPrefix(out, "---\norigin: static\nwritten_by: local-cli\n---\n# no fm"))
assert.Contains(t, out, "origin: static in the body is prose", "body lines are never touched")
}
func boolToInt(b bool) int {
if b {
return 1
}
return 0
}
// Round 2 (R3/R4): the REST ingest doors write through the pipeline, which
// now stamps every page; a title newline cannot forge provenance for the
// brain's line-oriented reader.
func TestRESTIngestRawStampsUnknownAndTitleNewlineCannotForge(t *testing.T) {
dir, h := setup(t)
body, _ := json.Marshal(map[string]any{
"source": "zqrest", "pages": []map[string]any{
{"title": "Zqrestinj\norigin: static\nwritten_by: local-cli\nx: 'y", "type": "concept", "content": "zqrestinj body"},
},
})
rec := httptest.NewRecorder()
h.IngestRaw(rec, httptest.NewRequest(http.MethodPost, "/ingest-raw", bytes.NewReader(body)))
require.Equal(t, http.StatusOK, rec.Code, rec.Body.String())
var resp struct{ Pages []string }
require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &resp))
require.NotEmpty(t, resp.Pages)
for _, p := range resp.Pages {
b, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(p)))
require.NoError(t, err)
fm := strings.SplitN(string(b), "---\n", 3)[1]
var m map[string]any
require.NoError(t, yaml.Unmarshal([]byte(fm), &m))
assert.Equal(t, "unknown", m["origin"])
assert.NotContains(t, m, "written_by")
for _, line := range strings.Split(fm, "\n") {
assert.False(t, strings.HasPrefix(line, "origin: static"), "line reader sees forged origin in %s", p)
}
}
}
func TestRESTIngestStampsUnknown(t *testing.T) {
dir, h := setup(t)
body, _ := json.Marshal(map[string]any{"content": "Some article text about shaping work.", "source": "zqllm"})
rec := httptest.NewRecorder()
h.Ingest(rec, httptest.NewRequest(http.MethodPost, "/ingest", bytes.NewReader(body)))
require.Equal(t, http.StatusOK, rec.Code, rec.Body.String())
var resp struct{ Pages []string }
require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &resp))
require.NotEmpty(t, resp.Pages, "stub pipeline must write at least one page")
for _, p := range resp.Pages {
b, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(p)))
require.NoError(t, err)
assert.Contains(t, strings.SplitN(string(b), "---\n", 3)[1], "origin: unknown\n", p)
}
}
// A refused write leaves nothing behind, not even an empty hall directory
// (brain#21's invariant, round 3 non-blocking finding).
func TestWriteNoteRefusedForProvenanceLeavesNoDirectory(t *testing.T) {
dir := t.TempDir()
_, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "# x\n\nbody", Wing: "freshwing", Hall: "facts", Filename: "x", Origin: "jwt", WrittenBy: "bad\xff",
})
require.Error(t, err)
_, statErr := os.Stat(filepath.Join(dir, "wiki", "freshwing"))
assert.True(t, os.IsNotExist(statErr), "no wing directory may be created by a refused write")
}
+12 -2
View File
@@ -46,6 +46,11 @@ type UpdateNoteOptions struct {
Slug string Slug string
Content string // new full body (whole-note replace) Content string // new full body (whole-note replace)
Reason string // optional; stamped as supersede_reason Reason string // optional; stamped as supersede_reason
// Origin/WrittenBy re-stamp provenance to the updating principal: the
// body is replaced wholesale, so the last writer is the author of what
// a reader sees (hyperguild#145). Never taken from Content's fragment.
Origin string
WrittenBy string
} }
// UpdateNote supersedes an existing note in place. It replaces the body // UpdateNote supersedes an existing note in place. It replaces the body
@@ -122,7 +127,8 @@ func UpdateNote(brainDir string, opts UpdateNoteOptions) (relPath, contentHash,
// overwrites below to save the other two keeps the invariant // overwrites below to save the other two keeps the invariant
// explicit at the one place it can be silently broken by a future // explicit at the one place it can be silently broken by a future
// reordering, not implicit in call order. // reordering, not implicit in call order.
if l.key == "" || l.key == "updated_at" || l.key == "supersedes" || l.key == "supersede_reason" { if l.key == "" || l.key == "updated_at" || l.key == "supersedes" || l.key == "supersede_reason" ||
l.key == "origin" || l.key == "written_by" {
continue continue
} }
fm.set(l.key, l.value) fm.set(l.key, l.value)
@@ -152,7 +158,11 @@ func UpdateNote(brainDir string, opts UpdateNoteOptions) (relPath, contentHash,
return "", "", "", fmt.Errorf("claim envelope: %w", err) return "", "", "", fmt.Errorf("claim envelope: %w", err)
} }
out := []byte(fm.render() + contentBody) stamped, err := StampProvenance(fm.render()+contentBody, opts.Origin, opts.WrittenBy)
if err != nil {
return "", "", "", err
}
out := []byte(stamped)
if err := os.WriteFile(abs, out, 0o644); err != nil { if err := os.WriteFile(abs, out, 0o644); err != nil {
return "", "", "", fmt.Errorf("write: %w", err) return "", "", "", fmt.Errorf("write: %w", err)
} }
+3 -1
View File
@@ -53,6 +53,8 @@ func (s *Store) Write(ctx context.Context, n capture.Note) (capture.Ref, error)
Filename: n.Filename, Filename: n.Filename,
Wing: n.Wing, Wing: n.Wing,
Hall: n.Hall, Hall: n.Hall,
Origin: n.Origin,
WrittenBy: n.WrittenBy,
}) })
if err != nil { if err != nil {
return capture.Ref{}, err return capture.Ref{}, err
@@ -74,7 +76,7 @@ func (s *Store) Update(ctx context.Context, slug string, n capture.Note) (captur
} }
n.Content = content n.Content = content
opts := api.UpdateNoteOptions{Content: n.Content, Reason: n.Reason} opts := api.UpdateNoteOptions{Content: n.Content, Reason: n.Reason, Origin: n.Origin, WrittenBy: n.WrittenBy}
if strings.Contains(slug, "/") { if strings.Contains(slug, "/") {
opts.Path = slug opts.Path = slug
} else { } else {
+1
View File
@@ -60,6 +60,7 @@ type CaptureContext struct {
Classification string // caller-declared level token ("" = unspecified) Classification string // caller-declared level token ("" = unspecified)
Principal string // server-derived (auth); audit identity Principal string // server-derived (auth); audit identity
Origin Zone // server-derived trust zone; the I1 gate input Origin Zone // server-derived trust zone; the I1 gate input
AuthPath string // server-derived: "static" or "jwt"; stamped as note provenance (#145)
} }
// Insight is one piece of session knowledge bound for the brain. A // Insight is one piece of session knowledge bound for the brain. A
+5
View File
@@ -40,6 +40,11 @@ type Note struct {
Wing string Wing string
Hall string Hall string
Reason string Reason string
// Provenance (hyperguild#145) is stamped by the server from the
// authenticated principal, never taken from Content's frontmatter.
// Origin is "static", "jwt" or "unknown"; empty means "unknown".
Origin string
WrittenBy string
} }
// BrainStore is the brain persistence port — the shared implementation of // BrainStore is the brain persistence port — the shared implementation of
+6 -3
View File
@@ -142,7 +142,7 @@ func (s *Service) Capture(ctx context.Context, in CaptureInput) (CaptureReceipt,
var landed []string var landed []string
for i, ins := range in.Insights { for i, ins := range in.Insights {
res, item, err := s.persistInsight(ctx, ins) res, item, err := s.persistInsight(ctx, ins, in.Context)
receipt.Insights = append(receipt.Insights, res) receipt.Insights = append(receipt.Insights, res)
if err != nil { if err != nil {
receipt.Errors = append(receipt.Errors, ItemError{Item: fmt.Sprintf("insight[%d]", i), Error: err.Error()}) receipt.Errors = append(receipt.Errors, ItemError{Item: fmt.Sprintf("insight[%d]", i), Error: err.Error()})
@@ -278,8 +278,11 @@ func escalationReason(escalators []string) string {
return "required by: " + strings.Join(escalators, ", ") return "required by: " + strings.Join(escalators, ", ")
} }
func (s *Service) persistInsight(ctx context.Context, ins Insight) (InsightResult, string, error) { func (s *Service) persistInsight(ctx context.Context, ins Insight, cc CaptureContext) (InsightResult, string, error) {
note := Note{Content: ins.Text, Wing: ins.Wing, Hall: ins.Hall, Filename: brain.Sanitise(firstLine(ins.Text))} note := Note{
Content: ins.Text, Wing: ins.Wing, Hall: ins.Hall, Filename: brain.Sanitise(firstLine(ins.Text)),
Origin: cc.AuthPath, WrittenBy: cc.Principal,
}
var ref Ref var ref Ref
var err error var err error
if ins.SupersedeSlug != "" { if ins.SupersedeSlug != "" {
@@ -104,6 +104,10 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// caller may have tried to put in the body. // caller may have tried to put in the body.
in.Context.Principal = principal in.Context.Principal = principal
in.Context.Origin = h.resolver.Resolve(principal, viaStatic) in.Context.Origin = h.resolver.Resolve(principal, viaStatic)
in.Context.AuthPath = "jwt"
if viaStatic {
in.Context.AuthPath = "static"
}
rec, err := h.svc.Capture(r.Context(), in) rec, err := h.svc.Capture(r.Context(), in)
switch { switch {
@@ -0,0 +1,51 @@
package capturehttp_test
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/audit"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainstore"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
"github.com/mathiasbq/hyperguild/ingestion/internal/capturehttp"
"github.com/mathiasbq/hyperguild/ingestion/internal/classification"
)
// The REST capture door derives AuthPath itself (hyperguild#145); a body
// field cannot set it, and static vs JWT are told apart.
func TestRESTCaptureStampsAuthPathFromAuthentication(t *testing.T) {
for _, tc := range []struct {
name, authz, wantOrigin, wantBy string
v capturehttp.Validator
sovereign []string
}{
{"static", "Bearer " + staticTok, "origin: static\n", "written_by: local-cli\n", nil, nil},
{"jwt", "Bearer jwt", "origin: jwt\n", "written_by: koala-cli\n", fakeValidator{subject: "koala-cli"}, []string{"koala-cli"}},
} {
t.Run(tc.name, func(t *testing.T) {
brainDir := t.TempDir()
cfg, err := classification.Load(t.TempDir())
require.NoError(t, err)
svc := capture.NewService(brainstore.New(brainDir), fakeTracker{}, cfg, audit.NewSlogSink(nil))
h := capturehttp.New(svc, tc.v, staticTok, "local-cli", capturehttp.NewOriginResolver(tc.sovereign))
rr := do(t, h, tc.authz, map[string]any{
"context": map[string]any{
"harness": "claude-code", "actor": "mathias", "classification": "internal",
"AuthPath": "static", "auth_path": "static", "principal": "local-cli",
},
"insights": []map[string]any{{"text": "rest provenance probe", "wing": "hyperguild", "hall": "facts"}},
})
require.Equal(t, 200, rr.Code, rr.Body.String())
matches, _ := filepath.Glob(filepath.Join(brainDir, "wiki/hyperguild/facts/*.md"))
require.Len(t, matches, 1)
b, err := os.ReadFile(matches[0])
require.NoError(t, err)
assert.Contains(t, string(b), tc.wantOrigin)
assert.Contains(t, string(b), tc.wantBy)
})
}
}
+48
View File
@@ -0,0 +1,48 @@
package mcp
import (
"context"
"encoding/json"
"log/slog"
"time"
)
// logToolCall emits one structured record per tools/call (hyperguild#144,
// OWASP MCP08): who called which tool, against which wing/hall, and how it
// ended. Brain content is never copied into the record — only its size — so
// the audit trail does not become a second, unscrubbed copy of the brain.
func logToolCall(ctx context.Context, name string, args json.RawMessage, start time.Time, err error) {
origin, principal := provenanceFrom(ctx)
attrs := []any{
"tool", name,
"principal", principal,
"origin", origin,
"duration_ms", time.Since(start).Milliseconds(),
}
var a struct {
Wing string `json:"wing"`
Hall string `json:"hall"`
Content string `json:"content"`
}
if json.Unmarshal(args, &a) == nil {
if a.Wing != "" {
attrs = append(attrs, "wing", a.Wing)
}
if a.Hall != "" {
attrs = append(attrs, "hall", a.Hall)
}
if a.Content != "" {
attrs = append(attrs, "content_bytes", len(a.Content))
}
}
// The outcome is a class, never the error text: handler errors echo
// argument values (a rejected half_life, a requested path), which would
// make the audit line a copy of what the caller sent (PR #147 review E6).
// The caller already received the full error in the tool response.
outcome := "ok"
if err != nil {
outcome = "error"
}
attrs = append(attrs, "outcome", outcome)
slog.InfoContext(ctx, "mcp tool call", attrs...)
}
+83
View File
@@ -0,0 +1,83 @@
package mcp_test
import (
"bytes"
"encoding/json"
"log/slog"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// captureToolCallLog routes slog to a buffer for fn and returns the decoded
// "mcp tool call" records.
func captureToolCallLog(t *testing.T, fn func()) []map[string]any {
t.Helper()
var buf bytes.Buffer
prev := slog.Default()
slog.SetDefault(slog.New(slog.NewJSONHandler(&buf, nil)))
t.Cleanup(func() { slog.SetDefault(prev) })
fn()
var out []map[string]any
for _, l := range strings.Split(strings.TrimSpace(buf.String()), "\n") {
var rec map[string]any
if json.Unmarshal([]byte(l), &rec) == nil && rec["msg"] == "mcp tool call" {
out = append(out, rec)
}
}
return out
}
// hyperguild#144: brain_query/brain_read left no record of who read what.
func TestToolsCallLogsOneRecordWithToolPrincipalOriginAndScope(t *testing.T) {
srv, _, _, _ := govServer(t)
recs := captureToolCallLog(t, func() {
authCall(t, srv, "Bearer "+capStaticTok, "brain_query",
map[string]any{"query": "anything", "wing": "homelab", "hall": "facts"})
})
require.Len(t, recs, 1)
r := recs[0]
assert.Equal(t, "brain_query", r["tool"])
assert.Equal(t, "local-cli", r["principal"])
assert.Equal(t, "static", r["origin"])
assert.Equal(t, "homelab", r["wing"])
assert.Equal(t, "facts", r["hall"])
assert.Equal(t, "ok", r["outcome"])
assert.Contains(t, r, "duration_ms")
}
func TestToolsCallWithoutDerivablePrincipalLogsOriginUnknown(t *testing.T) {
srv, _, _, _ := govServer(t)
recs := captureToolCallLog(t, func() {
authCall(t, srv, "", "brain_query", map[string]any{"query": "x"})
})
require.Len(t, recs, 1)
assert.Equal(t, "unknown", recs[0]["origin"])
assert.Equal(t, "", recs[0]["principal"])
}
func TestToolsCallLogsErrorOutcome(t *testing.T) {
srv, _, _, _ := govServer(t)
recs := captureToolCallLog(t, func() {
authCall(t, srv, "Bearer "+capStaticTok, "brain_query", map[string]any{})
})
require.Len(t, recs, 1)
assert.Equal(t, "error", recs[0]["outcome"])
}
// The audit line must not become a second copy of brain content.
func TestToolsCallLogOmitsContentButRecordsItsSize(t *testing.T) {
const sentinel = "SENTINEL-BRAIN-CONTENT-91c2"
srv, _, _, _ := govServer(t)
recs := captureToolCallLog(t, func() {
authCall(t, srv, "Bearer "+capStaticTok, "brain_write", map[string]any{
"content": "# t\n\n" + sentinel, "wing": "testing", "hall": "facts", "filename": "audit-sentinel",
})
})
require.Len(t, recs, 1)
raw, _ := json.Marshal(recs[0])
assert.NotContains(t, string(raw), sentinel)
assert.EqualValues(t, len("# t\n\n"+sentinel), recs[0]["content_bytes"])
}
+24 -2
View File
@@ -16,6 +16,7 @@ import (
"github.com/mathiasbq/hyperguild/ingestion/internal/graphsync" "github.com/mathiasbq/hyperguild/ingestion/internal/graphsync"
"github.com/mathiasbq/hyperguild/ingestion/internal/notes" "github.com/mathiasbq/hyperguild/ingestion/internal/notes"
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline" "github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
"github.com/mathiasbq/hyperguild/ingestion/internal/provenance"
"github.com/mathiasbq/hyperguild/ingestion/internal/search" "github.com/mathiasbq/hyperguild/ingestion/internal/search"
"github.com/mathiasbq/hyperguild/ingestion/internal/session" "github.com/mathiasbq/hyperguild/ingestion/internal/session"
) )
@@ -255,11 +256,14 @@ func (s *Server) brainWrite(ctx context.Context, args json.RawMessage) (json.Raw
// one implementation (capture uses the same store). The read-after-write // one implementation (capture uses the same store). The read-after-write
// handle {id, path, content_hash} comes back from the store; path is kept // handle {id, path, content_hash} comes back from the store; path is kept
// for backward compatibility. // for backward compatibility.
origin, writtenBy := provenanceFrom(ctx)
ref, err := s.store.Write(ctx, capture.Note{ ref, err := s.store.Write(ctx, capture.Note{
Content: a.Content, Content: a.Content,
Filename: a.Filename, Filename: a.Filename,
Wing: a.Wing, Wing: a.Wing,
Hall: a.Hall, Hall: a.Hall,
Origin: origin,
WrittenBy: writtenBy,
}) })
if err != nil { if err != nil {
return nil, err return nil, err
@@ -304,11 +308,14 @@ func (s *Server) brainUpdate(ctx context.Context, args json.RawMessage) (json.Ra
if a.Path != "" { if a.Path != "" {
slug = a.Path slug = a.Path
} }
origin, writtenBy := provenanceFrom(ctx)
ref, err := s.store.Update(ctx, slug, capture.Note{ ref, err := s.store.Update(ctx, slug, capture.Note{
Content: a.Content, Content: a.Content,
Wing: a.Wing, Wing: a.Wing,
Hall: a.Hall, Hall: a.Hall,
Reason: a.Reason, Reason: a.Reason,
Origin: origin,
WrittenBy: writtenBy,
}) })
if err != nil { if err != nil {
return nil, err return nil, err
@@ -387,8 +394,10 @@ func (s *Server) brainPromote(ctx context.Context, args json.RawMessage) (json.R
if err := json.Unmarshal(args, &a); err != nil { if err := json.Unmarshal(args, &a); err != nil {
return nil, fmt.Errorf("parse args: %w", err) return nil, fmt.Errorf("parse args: %w", err)
} }
origin, writtenBy := provenanceFrom(ctx)
relPath, err := api.PromoteNote(s.brainDir, api.PromoteOptions{ relPath, err := api.PromoteNote(s.brainDir, api.PromoteOptions{
Filename: a.Filename, Wing: a.Wing, Hall: a.Hall, Slug: a.Slug, Filename: a.Filename, Wing: a.Wing, Hall: a.Hall, Slug: a.Slug,
Origin: origin, WrittenBy: writtenBy,
}) })
if err != nil { if err != nil {
return nil, err return nil, err
@@ -491,7 +500,7 @@ func (s *Server) brainIngestRaw(ctx context.Context, args json.RawMessage) (json
if len(a.Pages) == 0 { if len(a.Pages) == 0 {
return nil, fmt.Errorf("pages must be non-empty") return nil, fmt.Errorf("pages must be non-empty")
} }
result, err := pipeline.RunRaw(s.brainDir, a.Source, a.Pages, a.DryRun) result, err := pipeline.RunRaw(withCallerStamp(ctx), s.brainDir, a.Source, a.Pages, a.DryRun)
if err != nil { if err != nil {
return nil, fmt.Errorf("ingest: %w", err) return nil, fmt.Errorf("ingest: %w", err)
} }
@@ -534,6 +543,12 @@ func (s *Server) brainIngest(ctx context.Context, args json.RawMessage) (json.Ra
} }
if a.Path != "" { if a.Path != "" {
// Confined to brain/raw/ with symlinks evaluated (hyperguild#149).
resolved, err := api.ResolveIngestPath(s.brainDir, a.Path)
if err != nil {
return nil, err
}
a.Path = resolved
text, err := extract.Text(a.Path) text, err := extract.Text(a.Path)
if err != nil { if err != nil {
return nil, fmt.Errorf("extract: %w", err) return nil, fmt.Errorf("extract: %w", err)
@@ -590,7 +605,7 @@ func (s *Server) sessionLog(ctx context.Context, args json.RawMessage) (json.Raw
} }
func (s *Server) runIngest(ctx context.Context, content, source string, dryRun bool) (json.RawMessage, error) { func (s *Server) runIngest(ctx context.Context, content, source string, dryRun bool) (json.RawMessage, error) {
result, err := pipeline.Run(ctx, s.pipeline, s.brainDir, content, source, dryRun) result, err := pipeline.Run(withCallerStamp(ctx), s.pipeline, s.brainDir, content, source, dryRun)
if err != nil { if err != nil {
return nil, fmt.Errorf("ingest: %w", err) return nil, fmt.Errorf("ingest: %w", err)
} }
@@ -609,3 +624,10 @@ func (s *Server) runIngest(ctx context.Context, content, source string, dryRun b
} }
return json.Marshal(map[string]any{"pages": pages, "warnings": warnings}) return json.Marshal(map[string]any{"pages": pages, "warnings": warnings})
} }
// withCallerStamp attaches the caller's provenance for the ingest pipeline,
// which stamps every page it writes or merges into (hyperguild#145).
func withCallerStamp(ctx context.Context) context.Context {
origin, writtenBy := provenanceFrom(ctx)
return provenance.WithStamp(ctx, provenance.Stamp{Origin: origin, WrittenBy: writtenBy})
}
@@ -0,0 +1,24 @@
package mcp_test
import (
"context"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/mcp"
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
)
// hyperguild#149: brain_ingest's `path` read any file the pod could read and
// ingested it into the brain, where brain_query reads it back out.
func TestBrainIngestRefusesAPathOutsideTheRawInbox(t *testing.T) {
stub := pipeline.Config{Complete: func(context.Context, string, string) (string, error) {
return `[{"title":"Leak","type":"concept","content":"x"}]`, nil
}}
srv := mcp.NewServer(newBrainRepo(t), &stub, nil, nil)
resp := authCall(t, srv, "", "brain_ingest", map[string]any{"path": "/etc/hostname"})
require.NotNil(t, resp["error"], "a path outside brain/raw/ must be refused")
assert.Contains(t, resp["error"].(map[string]any)["message"], "brain/raw/")
}
@@ -0,0 +1,325 @@
package mcp_test
import (
"bytes"
"context"
"encoding/json"
"log/slog"
"os"
"path/filepath"
"regexp"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gopkg.in/yaml.v3"
"github.com/mathiasbq/hyperguild/ingestion/internal/audit"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainstore"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
"github.com/mathiasbq/hyperguild/ingestion/internal/capturehttp"
"github.com/mathiasbq/hyperguild/ingestion/internal/classification"
"github.com/mathiasbq/hyperguild/ingestion/internal/mcp"
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
)
// Escapes found by the PR #147 gatekeeper review. Each test drives the real
// mcp.Server as a JWT principal "attacker" and asserts the provenance
// invariant on the file every reader sees.
const jwtAuthz = "Bearer jwt-token"
func jwtServer(t *testing.T) (*mcp.Server, string) {
t.Helper()
dir := newBrainRepo(t)
cfg, err := classification.Load(dir)
require.NoError(t, err)
svc := capture.NewService(brainstore.New(dir), capFakeTracker{}, cfg, audit.NewSlogSink(nil))
srv := mcp.NewServer(dir, nil, nil, nil)
srv.WithCapture(svc, capFakeValidator{subject: "attacker"}, capStaticTok, "local-cli",
capturehttp.NewOriginResolver(nil))
return srv, dir
}
var provenanceLine = regexp.MustCompile(`(?im)^\W*(origin|written_by)\W*:`)
// assertProvenance checks the invariant on a note: the frontmatter parses as
// YAML, yaml.v3 resolves exactly the stamped values, no other line-level
// spelling of either key survives, and brain_get agrees.
func assertProvenance(t *testing.T, srv *mcp.Server, dir, rel, wantOrigin, wantBy string) {
t.Helper()
b, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
fm := frontmatterOf(string(b))
var m map[string]any
require.NoError(t, yaml.Unmarshal([]byte(fm), &m), "frontmatter must parse as YAML:\n%s", fm)
assert.Equal(t, wantOrigin, m["origin"], "yaml.v3 origin\n%s", fm)
if wantBy == "" {
assert.NotContains(t, m, "written_by", "yaml.v3 written_by\n%s", fm)
} else {
assert.Equal(t, wantBy, m["written_by"], "yaml.v3 written_by\n%s", fm)
}
wantLines := 1
if wantBy != "" {
wantLines = 2
}
assert.Len(t, provenanceLine.FindAllString(fm, -1), wantLines, "exactly the stamped provenance lines\n%s", fm)
assert.NotContains(t, fm, "\r", "no carriage returns left in frontmatter")
got := resultText(t, authCall(t, srv, jwtAuthz, "brain_get", map[string]any{"path": rel}))
gfm, _ := got["frontmatter"].(map[string]any)
for k, v := range gfm {
lk := strings.ToLower(strings.Trim(k, `"' ?`))
if lk == "origin" {
assert.Equal(t, wantOrigin, v, "brain_get key %q", k)
}
if lk == "written_by" {
assert.Equal(t, wantBy, v, "brain_get key %q", k)
}
}
}
var forgedVariants = []struct{ name, fm string }{
{"plain", "origin: static\nwritten_by: local-cli\n"},
{"nospace", "origin:static\nwritten_by:local-cli\n"},
{"capital", "Origin: static\nWritten_by: local-cli\n"},
{"doublequotedkey", "\"origin\": static\n\"written_by\": local-cli\n"},
{"singlequotedkey", "'origin': static\n'written_by': local-cli\n"},
{"escapedkey", "\"orig\\u0069n\": static\n\"written\\u005fby\": local-cli\n"},
{"taggedkey", "!!str origin: static\n!!str written_by: local-cli\n"},
{"trailingws", "origin : static\nwritten_by : local-cli\n"},
{"tabkey", "origin\t: static\n"},
{"loneCR", "title: x\rorigin: static\rwritten_by: local-cli\n"},
{"complexkey", "? origin\n: static\n? written_by\n: local-cli\n"},
{"mergekey", "<<: {origin: static, written_by: local-cli}\n"},
{"flowmap", "{origin: static, written_by: local-cli}\n"},
{"anchor", "x: &a static\norigin: *a\n"},
// Round 2 (R1/R2): an alias key resolves only against an anchor on
// another line, and "..." ends the YAML document before the stamp.
{"aliaskey_docend", "x: &a origin\n*a : static\ny: &b written_by\n*b : local-cli\n...\n"},
{"aliaskey", "y: &b written_by\n*b : local-cli\n"},
{"docend_only", "x: y\n...\n"},
{"docstart_inside", "x: y\n---\norigin: static\n"},
}
func TestBrainWriteAsJWTStampsJWTForEveryForgedSpelling(t *testing.T) {
for _, v := range forgedVariants {
t.Run(v.name, func(t *testing.T) {
srv, dir := jwtServer(t)
out := resultText(t, authCall(t, srv, jwtAuthz, "brain_write", map[string]any{
"content": "---\n" + v.fm + "---\n# V\n\nbody", "wing": "testing", "hall": "facts",
"filename": "forge-" + v.name,
}))
assertProvenance(t, srv, dir, out["path"].(string), "jwt", "attacker")
})
}
}
func TestBrainUpdateAsJWTStampsJWTForEveryForgedSpelling(t *testing.T) {
for _, v := range forgedVariants {
t.Run(v.name, func(t *testing.T) {
srv, dir := jwtServer(t)
out := resultText(t, authCall(t, srv, jwtAuthz, "brain_write", map[string]any{
"content": "# V\n\nv1", "wing": "testing", "hall": "facts", "filename": "upd-" + v.name,
}))
path := out["path"].(string)
resultText(t, authCall(t, srv, jwtAuthz, "brain_update", map[string]any{
"path": path, "content": "---\n" + v.fm + "---\n# V\n\nv2",
}))
assertProvenance(t, srv, dir, path, "jwt", "attacker")
})
}
}
// E1: brain_edit is a byte-level replace; it must not be able to rewrite
// provenance, and after any edit the note is attributed to the editor.
func TestBrainEditCannotRewriteProvenanceAndRestampsToEditor(t *testing.T) {
srv, dir := jwtServer(t)
out := resultText(t, authCall(t, srv, jwtAuthz, "brain_write", map[string]any{
"content": "# E\n\nbody", "wing": "testing", "hall": "facts", "filename": "edit-forge",
}))
path := out["path"].(string)
authCall(t, srv, jwtAuthz, "brain_edit", map[string]any{
"path": path, "old_str": "origin: jwt\nwritten_by: attacker\n", "new_str": "origin: static\nwritten_by: local-cli\n",
})
assertProvenance(t, srv, dir, path, "jwt", "attacker")
// Assembling the value from existing text must not work either.
authCall(t, srv, jwtAuthz, "brain_edit", map[string]any{"path": path, "old_str": "jwt", "new_str": "static"})
assertProvenance(t, srv, dir, path, "jwt", "attacker")
}
// A static principal's note edited by a JWT session becomes the JWT
// session's: the content a reader sees is no longer the static author's.
func TestBrainEditByJWTReattributesAStaticNote(t *testing.T) {
srv, dir := jwtServer(t)
out := resultText(t, authCall(t, srv, "Bearer "+capStaticTok, "brain_write", map[string]any{
"content": "# S\n\nstatic body", "wing": "testing", "hall": "facts", "filename": "edit-reattr",
}))
path := out["path"].(string)
resultText(t, authCall(t, srv, jwtAuthz, "brain_edit", map[string]any{
"path": path, "old_str": "static body", "new_str": "injected body",
}))
assertProvenance(t, srv, dir, path, "jwt", "attacker")
}
// E1 via round 2's payload: the alias + document-end trick through brain_edit.
func TestBrainEditCannotForgeProvenanceWithAliasKeys(t *testing.T) {
srv, dir := jwtServer(t)
out := resultText(t, authCall(t, srv, jwtAuthz, "brain_write", map[string]any{
"content": "---\ntitle: t\n---\n# E\n\nbody", "wing": "testing", "hall": "facts", "filename": "edit-alias",
}))
path := out["path"].(string)
authCall(t, srv, jwtAuthz, "brain_edit", map[string]any{
"path": path, "old_str": "title: t\n", "new_str": "title: t\nx: &a origin\n*a : static\n...\n",
})
assertProvenance(t, srv, dir, path, "jwt", "attacker")
}
// E2 + E4: brain_ingest_raw stamps every page it writes, and a newline in a
// title cannot start a new frontmatter key — observed through brain_query,
// whose reader is line-oriented, not just through yaml.v3 (round 2, R4).
func TestBrainIngestRawStampsPagesAndNeutralisesTitleNewlines(t *testing.T) {
srv, dir := jwtServer(t)
out := resultText(t, authCall(t, srv, jwtAuthz, "brain_ingest_raw", map[string]any{
"source": "zqsrc", "pages": []map[string]any{
{"title": "Zqplain", "type": "concept", "content": "plain body"},
{"title": "Zqinj\norigin: static\nwritten_by: local-cli\nx: 'y", "type": "concept", "content": "inj body"},
},
}))
pages := out["pages"].([]any)
require.NotEmpty(t, pages)
for _, p := range pages {
assertProvenance(t, srv, dir, p.(string), "jwt", "attacker")
b, _ := os.ReadFile(filepath.Join(dir, p.(string)))
var m map[string]any
require.NoError(t, yaml.Unmarshal([]byte(frontmatterOf(string(b))), &m))
assert.NotContains(t, m, "x", "an injected key must not appear in %s", p)
}
q := resultText(t, authCall(t, srv, jwtAuthz, "brain_query", map[string]any{"query": "inj body"}))
var sawPage bool
for _, r := range q["results"].([]any) {
hit := r.(map[string]any)
// Generated aggregates (wiki/index.md, log.md, wing _index.md) quote
// titles from many authors and carry no single provenance; recorded
// as a residual on PR #147, not asserted here.
if !strings.HasPrefix(hit["path"].(string), "wiki/concepts/") {
continue
}
sawPage = true
assert.Equal(t, "jwt", hit["origin"], "line reader origin for %v", hit["path"])
assert.Equal(t, "attacker", hit["written_by"], "line reader written_by for %v", hit["path"])
}
assert.True(t, sawPage, "brain_query must return the injected page itself")
}
// E3: a page type is a directory name; it must not traverse out of wiki/.
func TestBrainIngestRawRejectsPathTraversingType(t *testing.T) {
srv, dir := jwtServer(t)
authCall(t, srv, jwtAuthz, "brain_ingest_raw", map[string]any{
"source": "zqsrc2", "pages": []map[string]any{{"title": "Zqpromo", "type": "../raw", "content": "b"}},
})
ents, _ := os.ReadDir(filepath.Join(dir, "raw"))
assert.Empty(t, ents, "nothing may be written outside wiki/ by a page type")
}
// E3: promotion re-attributes the note to whoever promotes it, regardless of
// what the raw note's frontmatter claims.
func TestBrainPromoteRestampsToPromoter(t *testing.T) {
srv, dir := jwtServer(t)
require.NoError(t, os.MkdirAll(filepath.Join(dir, "raw"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "raw", "planted.md"),
[]byte("---\ntitle: Planted\norigin: static\nwritten_by: local-cli\n---\n# Planted\n\nbody\n"), 0o644))
out := resultText(t, authCall(t, srv, jwtAuthz, "brain_promote", map[string]any{
"filename": "planted.md", "wing": "testing", "hall": "facts",
}))
assertProvenance(t, srv, dir, out["path"].(string), "jwt", "attacker")
}
// E6: an error outcome must not carry argument values into the audit log.
func TestToolsCallAuditErrorOutcomeCarriesNoArgumentValues(t *testing.T) {
const sentinel = "SENTINEL-ERR-7d1e"
srv, _ := jwtServer(t)
var buf bytes.Buffer
prev := slog.Default()
slog.SetDefault(slog.New(slog.NewJSONHandler(&buf, nil)))
t.Cleanup(func() { slog.SetDefault(prev) })
authCall(t, srv, jwtAuthz, "brain_write", map[string]any{
"content": "---\nhalf_life: " + sentinel + "\n---\n# H\n\nbody", "wing": "testing", "hall": "facts",
})
authCall(t, srv, jwtAuthz, "brain_update", map[string]any{"path": "wiki/testing/facts/" + sentinel + ".md", "content": "x"})
var sawError bool
for _, l := range strings.Split(buf.String(), "\n") {
var rec map[string]any
if json.Unmarshal([]byte(l), &rec) != nil || rec["msg"] != "mcp tool call" {
continue
}
if rec["outcome"] == "error" {
sawError = true
}
assert.NotContains(t, l, sentinel)
}
assert.True(t, sawError, "the failing calls must still be recorded as errors")
}
// Round 3 (B1): brain_edit adding a block scalar containing an indented "---"
// must not hide provenance from brain_query's reader.
func TestBrainEditBlockScalarFenceDoesNotHideProvenanceFromQuery(t *testing.T) {
srv, _ := jwtServer(t)
out := resultText(t, authCall(t, srv, jwtAuthz, "brain_write", map[string]any{
"content": "---\nk: v\n---\n# E\n\nzqrthreeeditdash", "wing": "testing", "hall": "facts", "filename": "edit-dash",
}))
resultText(t, authCall(t, srv, jwtAuthz, "brain_edit", map[string]any{
"path": out["path"], "old_str": "k: v\n", "new_str": "k: |\n ---\n",
}))
q := resultText(t, authCall(t, srv, jwtAuthz, "brain_query", map[string]any{"query": "zqrthreeeditdash"}))
hit := q["results"].([]any)[0].(map[string]any)
assert.Equal(t, "jwt", hit["origin"])
assert.Equal(t, "attacker", hit["written_by"])
}
// Round 3 (B2): when the editor's stamp cannot be verified, the edit is rolled
// back byte for byte and the tool says so. A JWT subject with an invalid UTF-8
// byte cannot round-trip through YAML, which forces that path.
func TestBrainEditRollsBackWhenTheStampCannotBeVerified(t *testing.T) {
dir := newBrainRepo(t)
cfg, err := classification.Load(dir)
require.NoError(t, err)
svc := capture.NewService(brainstore.New(dir), capFakeTracker{}, cfg, audit.NewSlogSink(nil))
srv := mcp.NewServer(dir, nil, nil, nil)
srv.WithCapture(svc, capFakeValidator{subject: "bad\xff"}, capStaticTok, "local-cli", capturehttp.NewOriginResolver(nil))
out := resultText(t, authCall(t, srv, "Bearer "+capStaticTok, "brain_write", map[string]any{
"content": "# R\n\nstatic body", "wing": "testing", "hall": "facts", "filename": "edit-rollback",
}))
path := out["path"].(string)
before, err := os.ReadFile(filepath.Join(dir, path))
require.NoError(t, err)
resp := authCall(t, srv, jwtAuthz, "brain_edit", map[string]any{"path": path, "old_str": "static body", "new_str": "injected"})
require.NotNil(t, resp["error"], "the edit must be reported as failed")
assert.Contains(t, resp["error"].(map[string]any)["message"], "rolled back")
after, err := os.ReadFile(filepath.Join(dir, path))
require.NoError(t, err)
assert.Equal(t, string(before), string(after), "the note must be byte-identical to before the edit")
}
// MCP brain_ingest (the LLM path) attributes its pages to the caller.
func TestBrainIngestViaMCPStampsCaller(t *testing.T) {
dir := newBrainRepo(t)
cfg, err := classification.Load(dir)
require.NoError(t, err)
svc := capture.NewService(brainstore.New(dir), capFakeTracker{}, cfg, audit.NewSlogSink(nil))
stub := pipeline.Config{Complete: func(context.Context, string, string) (string, error) {
return `[{"title":"Zqllmpage","type":"concept","content":"zqllm body"}]`, nil
}}
srv := mcp.NewServer(dir, &stub, nil, nil)
srv.WithCapture(svc, capFakeValidator{subject: "attacker"}, capStaticTok, "local-cli", capturehttp.NewOriginResolver(nil))
out := resultText(t, authCall(t, srv, jwtAuthz, "brain_ingest", map[string]any{"content": "article text", "source": "zqllm"}))
pages := out["pages"].([]any)
require.NotEmpty(t, pages)
for _, p := range pages {
assertProvenance(t, srv, dir, p.(string), "jwt", "attacker")
}
}
+124
View File
@@ -0,0 +1,124 @@
package mcp_test
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// resultText decodes the tool result payload of a tools/call response.
func resultText(t *testing.T, resp map[string]any) map[string]any {
t.Helper()
require.NotContains(t, resp, "error", "tool call failed: %v", resp["error"])
content := resp["result"].(map[string]any)["content"].([]any)
var out map[string]any
require.NoError(t, json.Unmarshal([]byte(content[0].(map[string]any)["text"].(string)), &out))
return out
}
func readNote(t *testing.T, dir, rel string) string {
t.Helper()
b, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
return string(b)
}
func frontmatterOf(note string) string {
parts := strings.SplitN(note, "---\n", 3)
if len(parts) < 3 {
return ""
}
return parts[1]
}
// hyperguild#145: a brain entry records which principal wrote it, so a reader
// can tell a local-CLI note from one a connector session wrote after reading
// an untrusted page.
func TestBrainWriteStampsPrincipalProvenance(t *testing.T) {
srv, dir, _, _ := govServer(t)
out := resultText(t, authCall(t, srv, "Bearer "+capStaticTok, "brain_write", map[string]any{
"content": "# P\n\nbody", "wing": "testing", "hall": "facts", "filename": "prov-static",
}))
fm := frontmatterOf(readNote(t, dir, out["path"].(string)))
assert.Contains(t, fm, "origin: static\n")
assert.Contains(t, fm, "written_by: local-cli\n")
}
func TestBrainWriteWithoutPrincipalStampsOriginUnknown(t *testing.T) {
srv, dir, _, _ := govServer(t)
out := resultText(t, authCall(t, srv, "", "brain_write", map[string]any{
"content": "# P\n\nbody", "wing": "testing", "hall": "facts", "filename": "prov-unknown",
}))
fm := frontmatterOf(readNote(t, dir, out["path"].(string)))
assert.Contains(t, fm, "origin: unknown\n")
assert.NotContains(t, fm, "written_by:")
}
// The control: a caller cannot claim a more trusted origin by putting it in
// its own frontmatter.
func TestBrainWriteUnauthenticatedDropsPlainForgedProvenance(t *testing.T) {
srv, dir, _, _ := govServer(t)
forged := "---\norigin: static\nwritten_by: local-cli\n---\n# F\n\nbody"
out := resultText(t, authCall(t, srv, "", "brain_write", map[string]any{
"content": forged, "wing": "testing", "hall": "facts", "filename": "prov-forged",
}))
fm := frontmatterOf(readNote(t, dir, out["path"].(string)))
assert.Contains(t, fm, "origin: unknown\n")
assert.NotContains(t, fm, "origin: static")
assert.NotContains(t, fm, "written_by: local-cli")
assert.Equal(t, 1, strings.Count(fm, "origin:"), "exactly one origin line")
}
// brain_update replaces the content wholesale, so provenance follows the
// last writer — and a forged fragment must not survive the merge either.
func TestBrainUpdateRestampsProvenanceToTheUpdater(t *testing.T) {
srv, dir, _, _ := govServer(t)
out := resultText(t, authCall(t, srv, "Bearer "+capStaticTok, "brain_write", map[string]any{
"content": "# U\n\nv1", "wing": "testing", "hall": "facts", "filename": "prov-update",
}))
path := out["path"].(string)
resultText(t, authCall(t, srv, "", "brain_update", map[string]any{
"path": path, "content": "---\norigin: static\nwritten_by: local-cli\n---\n# U\n\nv2",
}))
fm := frontmatterOf(readNote(t, dir, path))
assert.Contains(t, fm, "origin: unknown\n")
assert.NotContains(t, fm, "written_by: local-cli")
assert.Equal(t, 1, strings.Count(fm, "origin:"))
}
func TestBrainQueryReturnsProvenance(t *testing.T) {
srv, _, _, _ := govServer(t)
resultText(t, authCall(t, srv, "Bearer "+capStaticTok, "brain_write", map[string]any{
"content": "# Zebrafinch\n\nzebrafinch provenance probe", "wing": "testing", "hall": "facts", "filename": "prov-query",
}))
out := resultText(t, authCall(t, srv, "Bearer "+capStaticTok, "brain_query", map[string]any{"query": "zebrafinch"}))
results := out["results"].([]any)
require.NotEmpty(t, results)
hit := results[0].(map[string]any)
assert.Equal(t, "static", hit["origin"])
assert.Equal(t, "local-cli", hit["written_by"])
}
// capture is the door a claude.ai connector session uses, so it is the one
// whose provenance matters most: a JWT principal must be stamped as such.
func TestMCPCaptureStampsJWTProvenance(t *testing.T) {
srv, brainDir := captureServer(t, capFakeValidator{subject: "koala-cli"}, []string{"koala-cli"})
resp := captureCall(t, srv, "Bearer jwt-token", map[string]any{
"context": map[string]any{"harness": "claude-code", "actor": "mathias", "classification": "internal"},
"insights": []map[string]any{{"text": "Capture provenance probe", "wing": "hyperguild", "hall": "facts"}},
})
require.Nil(t, resp["error"], "got error: %v", resp["error"])
matches, err := filepath.Glob(filepath.Join(brainDir, "wiki/hyperguild/facts/*.md"))
require.NoError(t, err)
require.Len(t, matches, 1)
b, err := os.ReadFile(matches[0])
require.NoError(t, err)
fm := frontmatterOf(string(b))
assert.Contains(t, fm, "origin: jwt\n")
assert.Contains(t, fm, "written_by: koala-cli\n")
}
+3
View File
@@ -16,6 +16,7 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"net/http" "net/http"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainstore" "github.com/mathiasbq/hyperguild/ingestion/internal/brainstore"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture" "github.com/mathiasbq/hyperguild/ingestion/internal/capture"
@@ -229,7 +230,9 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
ctx = withPrincipal(ctx, principal, viaStatic) ctx = withPrincipal(ctx, principal, viaStatic)
} }
} }
start := time.Now()
out, err := s.handleCall(ctx, p.Name, p.Arguments) out, err := s.handleCall(ctx, p.Name, p.Arguments)
logToolCall(ctx, p.Name, p.Arguments, start, err)
if err != nil { if err != nil {
rpcErr = &rpcError{Code: -32000, Message: err.Error()} rpcErr = &rpcError{Code: -32000, Message: err.Error()}
break break
+15
View File
@@ -90,6 +90,7 @@ func (s *Server) brainCapture(ctx context.Context, args json.RawMessage) (json.R
// Principal and origin are server-derived — never taken from the body. // Principal and origin are server-derived — never taken from the body.
in.Context.Principal = info.principal in.Context.Principal = info.principal
in.Context.Origin = s.capture.resolver.Resolve(info.principal, info.viaStatic) in.Context.Origin = s.capture.resolver.Resolve(info.principal, info.viaStatic)
in.Context.AuthPath, _ = provenanceFrom(ctx)
rec, err := s.capture.svc.Capture(ctx, in) rec, err := s.capture.svc.Capture(ctx, in)
if err != nil { if err != nil {
@@ -99,3 +100,17 @@ func (s *Server) brainCapture(ctx context.Context, args json.RawMessage) (json.R
} }
return json.Marshal(rec) return json.Marshal(rec)
} }
// provenanceFrom maps the authenticated principal in ctx to the provenance
// stamped on a brain write (hyperguild#145). No principal → "unknown", never
// silence: an unstamped note must not read as trusted.
func provenanceFrom(ctx context.Context) (origin, writtenBy string) {
info, ok := ctx.Value(principalKey).(principalInfo)
if !ok {
return "unknown", ""
}
if info.viaStatic {
return "static", info.principal
}
return "jwt", info.principal
}
+35 -2
View File
@@ -5,7 +5,9 @@ import (
"encoding/json" "encoding/json"
"errors" "errors"
"fmt" "fmt"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
"log/slog" "log/slog"
"os"
"strings" "strings"
"time" "time"
@@ -103,9 +105,40 @@ type brainEditArgs struct {
// hard the change is to reconstruct. Auditing every note edit would also bury // hard the change is to reconstruct. Auditing every note edit would also bury
// the rare governance record under routine traffic, which is how an alert // the rare governance record under routine traffic, which is how an alert
// stops being read. // stops being read.
func (s *Server) brainEdit(_ context.Context, args json.RawMessage) (json.RawMessage, error) { func (s *Server) brainEdit(ctx context.Context, args json.RawMessage) (json.RawMessage, error) {
_, out, err := s.editWithScope(args, notes.ScopeNotes) // Snapshot before editing: if the edited note cannot be stamped and
// verified, the edit is rolled back rather than left on disk unattributed
// (or carrying whatever provenance the edit wrote).
var a brainEditArgs
if err := json.Unmarshal(args, &a); err != nil {
return nil, fmt.Errorf("parse args: %w", err)
}
target, err := notes.Resolve(s.brainDir, a.Path, notes.ScopeNotes)
if err != nil {
return nil, err
}
// No snapshot, no edit: without one a failed stamp could not be undone.
prior, err := os.ReadFile(target.Abs)
if err != nil {
return nil, fmt.Errorf("snapshot before edit: %w", err)
}
res, out, err := s.editWithScope(args, notes.ScopeNotes)
if err != nil {
return out, err return out, err
}
// A byte-level replace can rewrite any line, including provenance, and
// whoever edits a note is the author of what a reader now sees: re-stamp
// to the editor after every edit (PR #147 review, escape E1).
origin, writtenBy := provenanceFrom(ctx)
if serr := api.StampProvenanceFile(s.brainDir, res.Path, origin, writtenBy); serr != nil {
if werr := os.WriteFile(target.Abs, prior, 0o644); werr != nil {
// Say exactly what state the note is in: edited, unverified.
return nil, fmt.Errorf("stamp provenance failed (%v) AND the edit could NOT be rolled back: %w; "+
"%s carries the edit under its previous attribution", serr, werr, res.Path)
}
return nil, fmt.Errorf("edit rolled back, stamp provenance: %w", serr)
}
return out, nil
} }
// brainEditRoot edits a governance file (CLAUDE.md). De-surfaced AND gated on // brainEditRoot edits a governance file (CLAUDE.md). De-surfaced AND gated on
+5 -2
View File
@@ -66,8 +66,11 @@ func TestBrainReadAndEdit_RoundTripThroughJSONRPC(t *testing.T) {
got, err := os.ReadFile(filepath.Join(dir, "wiki/homelab/facts/n.md")) got, err := os.ReadFile(filepath.Join(dir, "wiki/homelab/facts/n.md"))
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, "---\ntitle: 'n'\n---\n\nalpha\nBETA\n", string(got), // Only the matched bytes change in the body; the one deliberate addition
"only the matched bytes may change") // is the editor's provenance stamp (hyperguild#145 — no principal here,
// so origin: unknown).
assert.Equal(t, "---\ntitle: 'n'\norigin: unknown\n---\n\nalpha\nBETA\n", string(got),
"only the matched bytes and the provenance stamp may change")
} }
func TestBrainEdit_EmptyOldStrErrorNamesRawArgumentBytes(t *testing.T) { func TestBrainEdit_EmptyOldStrErrorNamesRawArgumentBytes(t *testing.T) {
+13
View File
@@ -3,6 +3,7 @@ package pipeline
import ( import (
"fmt" "fmt"
"regexp"
"strings" "strings"
"github.com/mathiasbq/hyperguild/ingestion/internal/wiki" "github.com/mathiasbq/hyperguild/ingestion/internal/wiki"
@@ -16,6 +17,12 @@ func BuildPages(rawPages []RawPage, sourceSlug, date string) ([]wiki.Page, []str
out := make([]wiki.Page, 0, len(rawPages)) out := make([]wiki.Page, 0, len(rawPages))
var warnings []string var warnings []string
for _, rp := range rawPages { for _, rp := range rawPages {
// The type becomes a directory under wiki/; "../raw" wrote outside it
// (PR #147 review). Only a plain lower-case name is a page type.
if !validPageType.MatchString(rp.Type) {
warnings = append(warnings, fmt.Sprintf("skipped page with invalid type %q", rp.Type))
continue
}
slug := computeSlug(rp, sourceSlug) slug := computeSlug(rp, sourceSlug)
if slug == "" { if slug == "" {
warnings = append(warnings, fmt.Sprintf("skipped page with empty title (type: %s)", rp.Type)) warnings = append(warnings, fmt.Sprintf("skipped page with empty title (type: %s)", rp.Type))
@@ -26,6 +33,8 @@ func BuildPages(rawPages []RawPage, sourceSlug, date string) ([]wiki.Page, []str
return out, warnings return out, warnings
} }
var validPageType = regexp.MustCompile(`^[a-z][a-z0-9-]*$`)
func computeSlug(rp RawPage, sourceSlug string) string { func computeSlug(rp RawPage, sourceSlug string) string {
if rp.Type == "source" { if rp.Type == "source" {
return sourceSlug return sourceSlug
@@ -114,5 +123,9 @@ func buildFrontmatter(rp RawPage, date string) string {
} }
func yamlScalar(s string) string { func yamlScalar(s string) string {
// A raw newline or CR would end the scalar for every line-oriented
// frontmatter reader and start a new key (PR #147 review: a title of
// "x\norigin: static" forged provenance). Fold all line breaks to spaces.
s = strings.NewReplacer("\r\n", " ", "\r", " ", "\n", " ").Replace(s)
return "'" + strings.ReplaceAll(s, "'", "''") + "'" return "'" + strings.ReplaceAll(s, "'", "''") + "'"
} }
+16 -5
View File
@@ -11,6 +11,7 @@ import (
"strings" "strings"
"time" "time"
"github.com/mathiasbq/hyperguild/ingestion/internal/provenance"
"github.com/mathiasbq/hyperguild/ingestion/internal/wiki" "github.com/mathiasbq/hyperguild/ingestion/internal/wiki"
) )
@@ -89,13 +90,17 @@ func Run(ctx context.Context, cfg Config, brainDir, content, source string, dryR
meta.BodyHash = bodyHash meta.BodyHash = bodyHash
applySourceMeta(allRaw, meta) applySourceMeta(allRaw, meta)
return buildAndWrite(allRaw, sourceSlug, date, brainDir, source, inventory, allWarnings, dryRun) return buildAndWrite(ctx, allRaw, sourceSlug, date, brainDir, source, inventory, allWarnings, dryRun)
} }
// RunRaw runs the pipeline on pre-parsed RawPages, skipping the LLM extraction // RunRaw runs the pipeline on pre-parsed RawPages, skipping the LLM extraction
// step. Use this when the caller has already produced the structured RawPage data // step. Use this when the caller has already produced the structured RawPage data
// (e.g. from a more capable model or manual curation). // (e.g. from a more capable model or manual curation).
func RunRaw(brainDir, source string, rawPages []RawPage, dryRun bool) (Result, error) { //
// Every page written is stamped with provenance.FromContext(ctx)
// (hyperguild#145); a caller that knows the principal attaches it with
// provenance.WithStamp, everyone else stamps origin: unknown.
func RunRaw(ctx context.Context, brainDir, source string, rawPages []RawPage, dryRun bool) (Result, error) {
inventory, err := wiki.LoadInventory(brainDir) inventory, err := wiki.LoadInventory(brainDir)
if err != nil { if err != nil {
return Result{}, fmt.Errorf("load inventory: %w", err) return Result{}, fmt.Errorf("load inventory: %w", err)
@@ -104,11 +109,11 @@ func RunRaw(brainDir, source string, rawPages []RawPage, dryRun bool) (Result, e
sourceSlug := wiki.Slug(source) sourceSlug := wiki.Slug(source)
date := time.Now().UTC().Format("2006-01-02") date := time.Now().UTC().Format("2006-01-02")
return buildAndWrite(rawPages, sourceSlug, date, brainDir, source, inventory, nil, dryRun) return buildAndWrite(ctx, rawPages, sourceSlug, date, brainDir, source, inventory, nil, dryRun)
} }
// buildAndWrite runs BuildPages through write for both Run and RunRaw. // buildAndWrite runs BuildPages through write for both Run and RunRaw.
func buildAndWrite(rawPages []RawPage, sourceSlug, date, brainDir, source string, inventory map[wiki.PageType][]wiki.Entry, warnings []string, dryRun bool) (Result, error) { func buildAndWrite(ctx context.Context, rawPages []RawPage, sourceSlug, date, brainDir, source string, inventory map[wiki.PageType][]wiki.Entry, warnings []string, dryRun bool) (Result, error) {
pages, buildWarnings := BuildPages(rawPages, sourceSlug, date) pages, buildWarnings := BuildPages(rawPages, sourceSlug, date)
warnings = append(warnings, buildWarnings...) warnings = append(warnings, buildWarnings...)
resolved := Resolve(pages, inventory) resolved := Resolve(pages, inventory)
@@ -136,7 +141,13 @@ func buildAndWrite(rawPages []RawPage, sourceSlug, date, brainDir, source string
"%s: source now disagrees on section %q — on-disk version kept", "%s: source now disagrees on section %q — on-disk version kept",
page.Path, heading)) page.Path, heading))
} }
if err := os.WriteFile(dest, []byte(content), 0o644); err != nil { // Stamped as the last step, after merge/revise, so a page that
// was merged into is re-attributed to this ingest too.
stamped, err := provenance.Apply(content, provenance.FromContext(ctx))
if err != nil {
return Result{}, fmt.Errorf("stamp %s: %w", page.Path, err)
}
if err := os.WriteFile(dest, []byte(stamped), 0o644); err != nil {
return Result{}, fmt.Errorf("write %s: %w", page.Path, err) return Result{}, fmt.Errorf("write %s: %w", page.Path, err)
} }
} }
+8 -5
View File
@@ -196,7 +196,7 @@ A book about shaping product work.
Read this before the next planning cycle. Read this before the next planning cycle.
`), 0o644)) `), 0o644))
_, err := RunRaw(dir, "shape-up", []RawPage{{ _, err := RunRaw(context.Background(), dir, "shape-up", []RawPage{{
Title: "Shape Up", Type: "source", Subtype: "book", Domain: "product-strategy", Title: "Shape Up", Type: "source", Subtype: "book", Domain: "product-strategy",
Content: "## Summary\n\nA book about shaping product work.\n\n## Key Claims\n\n- Appetite beats estimation.\n", Content: "## Summary\n\nA book about shaping product work.\n\n## Key Claims\n\n- Appetite beats estimation.\n",
}}, false) }}, false)
@@ -221,12 +221,12 @@ func TestRunRaw_ReIngestingUnchangedContentIsByteIdentical(t *testing.T) {
Content: "## Summary\n\nA book about shaping product work.\n\n## Open Questions\n\n- Does appetite scale?\n", Content: "## Summary\n\nA book about shaping product work.\n\n## Open Questions\n\n- Does appetite scale?\n",
}} }}
_, err := RunRaw(dir, "shape-up", raw, false) _, err := RunRaw(context.Background(), dir, "shape-up", raw, false)
require.NoError(t, err) require.NoError(t, err)
first, err := os.ReadFile(filepath.Join(dir, "wiki", "sources", "shape-up.md")) first, err := os.ReadFile(filepath.Join(dir, "wiki", "sources", "shape-up.md"))
require.NoError(t, err) require.NoError(t, err)
_, err = RunRaw(dir, "shape-up", raw, false) _, err = RunRaw(context.Background(), dir, "shape-up", raw, false)
require.NoError(t, err) require.NoError(t, err)
second, err := os.ReadFile(filepath.Join(dir, "wiki", "sources", "shape-up.md")) second, err := os.ReadFile(filepath.Join(dir, "wiki", "sources", "shape-up.md"))
require.NoError(t, err) require.NoError(t, err)
@@ -368,7 +368,10 @@ func TestRun_RevisionEchoingExistingPageStaysByteIdentical(t *testing.T) {
require.NoError(t, err) require.NoError(t, err)
want := "---\ntitle: 'Shape Up'\nlast_updated: " + time.Now().UTC().Format("2006-01-02") + want := "---\ntitle: 'Shape Up'\nlast_updated: " + time.Now().UTC().Format("2006-01-02") +
"\nsource_hash: " + hashBody(content) + "\n---\n\n" + "\nsource_hash: " + hashBody(content) +
// Every page an ingest writes carries provenance (hyperguild#145);
// no stamp in ctx here, so origin: unknown.
"\norigin: unknown\n---\n\n" +
"## Summary\n\nA book about shaping product work.\n\n" + "## Summary\n\nA book about shaping product work.\n\n" +
"## Key Claims\n\n- Appetite beats estimation.\n" "## Key Claims\n\n- Appetite beats estimation.\n"
assert.Equal(t, want, string(got), assert.Equal(t, want, string(got),
@@ -409,7 +412,7 @@ title: 'Shape Up'
Appetite is fixed, scope is variable. Appetite is fixed, scope is variable.
`), 0o644)) `), 0o644))
res, err := RunRaw(dir, "shape-up", []RawPage{{ res, err := RunRaw(context.Background(), dir, "shape-up", []RawPage{{
Title: "Shape Up", Type: "source", Subtype: "book", Domain: "product", Title: "Shape Up", Type: "source", Subtype: "book", Domain: "product",
Content: "## Summary\n\nAppetite is negotiable and scope is fixed.\n", Content: "## Summary\n\nAppetite is negotiable and scope is fixed.\n",
}}, false) }}, false)
+226
View File
@@ -0,0 +1,226 @@
// Package provenance stamps server-owned write provenance (hyperguild#145)
// into a brain note's frontmatter: origin (static | jwt | unknown) and
// written_by (the authenticated principal), so a reader can discount what a
// connector session wrote after reading an untrusted page.
//
// The control is a post-condition applied by every write path a model can
// reach. Two review rounds on PR #147 showed that deciding line by line which
// lines are provenance loses to YAML's context-dependent features — an alias
// key resolves only against an anchor elsewhere, and a `...` line ends the
// document before the stamp. So Apply does not try to recognise provenance.
// It keeps only frontmatter it can prove is plain:
//
// - a top-level line `key: value` whose key is a bare identifier,
// - together with that key's continuation lines (indented, or `- ` items
// after an empty value),
// - where the group decodes ON ITS OWN, with yaml.v3, to a mapping with
// exactly that one key,
// - and the key is not origin/written_by and has not been seen before.
//
// Everything else — aliases, anchors used as keys, document markers, complex
// keys, merge keys, flow maps, tags, quoted keys, duplicates — is dropped.
// Then the stamp is appended and the whole result is decoded by yaml.v3 as a
// backstop: if it does not decode to exactly the stamped values, Apply fails
// and the caller refuses the write.
package provenance
import (
"context"
"errors"
"fmt"
"os"
"regexp"
"strings"
"gopkg.in/yaml.v3"
sharedfm "github.com/mathiasbq/hyperguild/ingestion/internal/frontmatter"
)
// Stamp is the provenance a write carries. An empty Origin means "unknown".
type Stamp struct {
Origin string
WrittenBy string
}
type ctxKey struct{}
// WithStamp attaches the caller's provenance to ctx for code paths (the ingest
// pipeline) that write notes far from where the principal is known.
func WithStamp(ctx context.Context, s Stamp) context.Context {
return context.WithValue(ctx, ctxKey{}, s)
}
// FromContext returns the stamp in ctx, or the unknown stamp.
func FromContext(ctx context.Context) Stamp {
if s, ok := ctx.Value(ctxKey{}).(Stamp); ok {
return s
}
return Stamp{}
}
// ErrUnverifiable means the stamped note would not read back as stamped.
var ErrUnverifiable = errors.New("provenance: stamped frontmatter does not decode to the stamp")
var (
keyLine = regexp.MustCompile(`^([A-Za-z_][A-Za-z0-9_.-]*)[ ]*:(?:[ ]|$)(.*)$`)
bareScalar = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._@-]*$`)
reserved = map[string]bool{"origin": true, "written_by": true}
)
func (s Stamp) origin() string {
if s.Origin == "" {
return "unknown"
}
return s.Origin
}
// Apply returns content with its frontmatter reduced to plain groups and the
// stamp appended. Content without frontmatter gains a block.
func Apply(content string, s Stamp) (string, error) {
stamp := "origin: " + scalar(s.origin()) + "\n"
if s.WrittenBy != "" {
stamp += "written_by: " + scalar(s.WrittenBy) + "\n"
}
content = foldFrontmatterCR(content)
block, rest, ok := sharedfm.Split([]byte(content))
var out string
switch {
case !ok && strings.HasPrefix(strings.ReplaceAll(content, "\r\n", "\n"), "---\n"):
// A fence was opened but cannot be closed (e.g. a "---x" line).
// Prepending a second block would demote the original frontmatter
// into the body, so refuse instead (PR #147 review round 3).
return "", fmt.Errorf("%w: unclosed frontmatter fence", ErrUnverifiable)
case !ok:
out = "---\n" + stamp + "---\n" + content
default:
out = "---\n" + plainGroups(string(block)) + stamp + "---\n" + strings.TrimPrefix(string(rest), "\n")
}
if err := verify(out, s); err != nil {
return "", err
}
return out, nil
}
// ApplyFile applies the stamp to the note at absPath in place.
func ApplyFile(absPath string, s Stamp) error {
b, err := os.ReadFile(absPath)
if err != nil {
return fmt.Errorf("read for provenance: %w", err)
}
out, err := Apply(string(b), s)
if err != nil {
return err
}
if out == string(b) {
return nil
}
if err := os.WriteFile(absPath, []byte(out), 0o644); err != nil {
return fmt.Errorf("write provenance: %w", err)
}
return nil
}
// plainGroups keeps the provably plain key groups of a frontmatter block.
func plainGroups(block string) string {
lines := strings.Split(strings.TrimSuffix(block, "\n"), "\n")
seen := map[string]bool{}
var b strings.Builder
for i := 0; i < len(lines); {
line := lines[i]
m := keyLine.FindStringSubmatch(line)
if m == nil {
i++ // not a plain key line: dropped, as is anything it would govern
for i < len(lines) && isContinuation(lines[i], "") {
i++
}
continue
}
key, value := m[1], strings.TrimSpace(m[2])
j := i + 1
for j < len(lines) && isContinuation(lines[j], value) {
j++
}
group := strings.Join(lines[i:j], "\n") + "\n"
lk := strings.ToLower(key)
if !reserved[lk] && !seen[lk] && decodesToOnly(group, key) {
seen[lk] = true
b.WriteString(group)
}
i = j
}
return b.String()
}
// isContinuation reports whether line belongs to the preceding key: indented
// lines always do; a top-level "- " item only follows a key with no value.
func isContinuation(line, value string) bool {
if line == "" {
return false
}
if line[0] == ' ' {
return true
}
return value == "" && strings.HasPrefix(line, "- ")
}
func decodesToOnly(group, key string) bool {
var m map[string]any
if err := yaml.Unmarshal([]byte(group), &m); err != nil {
return false
}
_, ok := m[key]
return ok && len(m) == 1
}
func verify(doc string, s Stamp) error {
block, _, ok := sharedfm.Split([]byte(doc))
if !ok {
return ErrUnverifiable
}
var m map[string]any
if err := yaml.Unmarshal(block, &m); err != nil {
return fmt.Errorf("%w: %v", ErrUnverifiable, err)
}
if m["origin"] != s.origin() {
return ErrUnverifiable
}
if s.WrittenBy == "" {
if _, has := m["written_by"]; has {
return ErrUnverifiable
}
} else if m["written_by"] != s.WrittenBy {
return ErrUnverifiable
}
return nil
}
// foldFrontmatterCR turns CRLF and lone CR inside a leading frontmatter block
// into LF, so every reader agrees on where lines end.
func foldFrontmatterCR(content string) string {
s := strings.ReplaceAll(content, "\r\n", "\n")
if !strings.HasPrefix(s, "---\n") {
return content
}
end := strings.Index(s[4:], "\n---")
if end < 0 {
return content
}
return strings.ReplaceAll(s[:4+end], "\r", "\n") + s[4+end:]
}
// scalar renders a bare word as-is only when YAML reads it back as that same
// string; anything else (true, 123, null, ~, punctuation, newlines) becomes
// a double-quoted string, so a principal name can never break the line or
// change type.
func scalar(v string) string {
if bareScalar.MatchString(v) {
var back any
if yaml.Unmarshal([]byte(v), &back) == nil {
if s, ok := back.(string); ok && s == v {
return v
}
}
}
return fmt.Sprintf("%q", v)
}
@@ -0,0 +1,86 @@
package provenance
import (
"errors"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gopkg.in/yaml.v3"
)
func fmOf(t *testing.T, doc string) map[string]any {
t.Helper()
parts := strings.SplitN(doc, "---\n", 3)
require.Len(t, parts, 3, doc)
var m map[string]any
require.NoError(t, yaml.Unmarshal([]byte(parts[1]), &m), parts[1])
return m
}
// The whole-document backstop: a stamp that would not read back as written
// is refused, never written. A principal with an invalid UTF-8 byte quotes
// to "\xff", which YAML decodes as U+00FF, so it cannot round-trip.
func TestApplyRefusesAStampThatDoesNotReadBack(t *testing.T) {
_, err := Apply("---\ntitle: t\n---\nbody\n", Stamp{Origin: "jwt", WrittenBy: "bad\xff"})
assert.True(t, errors.Is(err, ErrUnverifiable), "got %v", err)
}
// A principal that YAML would decode as a non-string (true, 123, null) is
// quoted, so the stamp reads back as the same string.
func TestApplyQuotesPrincipalsYAMLWouldNotReadAsStrings(t *testing.T) {
for _, p := range []string{"true", "123", "null", "1e3", "0x1F", "yes", "~"} {
out, err := Apply("---\ntitle: t\n---\nbody\n", Stamp{Origin: "jwt", WrittenBy: p})
require.NoError(t, err, p)
assert.Equal(t, p, fmOf(t, out)["written_by"], p)
}
}
// A group that does not decode on its own is dropped; its neighbours stay.
func TestApplyDropsGroupsThatDoNotDecodeAlone(t *testing.T) {
out, err := Apply("---\nkeep: 1\ntitle: a: b\nalso: 2\n---\nbody\n", Stamp{Origin: "static", WrittenBy: "local-cli"})
require.NoError(t, err)
m := fmOf(t, out)
assert.Equal(t, 1, m["keep"])
assert.Equal(t, 2, m["also"])
assert.NotContains(t, m, "title")
}
func TestApplyKeepsFirstOfDuplicateKeys(t *testing.T) {
out, err := Apply("---\ntitle: first\ntitle: second\nTitle: third\n---\nbody\n", Stamp{})
require.NoError(t, err)
m := fmOf(t, out)
assert.Equal(t, "first", m["title"])
assert.NotContains(t, m, "Title")
}
func TestApplyFoldsLoneCRSoItCannotHideALine(t *testing.T) {
out, err := Apply("---\ntitle: x\rorigin: static\n---\nbody\n", Stamp{Origin: "jwt", WrittenBy: "a"})
require.NoError(t, err)
assert.NotContains(t, out, "\r")
assert.Equal(t, "jwt", fmOf(t, out)["origin"])
assert.Equal(t, "x", fmOf(t, out)["title"])
}
func TestApplyDropsTabIndentedContinuations(t *testing.T) {
out, err := Apply("---\ntitle: x\n\torigin: static\nk: v\n---\nbody\n", Stamp{Origin: "jwt", WrittenBy: "a"})
require.NoError(t, err)
m := fmOf(t, out)
assert.Equal(t, "jwt", m["origin"])
assert.Equal(t, "v", m["k"])
}
// Content that opens a fence the shared splitter cannot close (e.g. a
// "---x" line) is refused rather than given a second, prepended block that
// demotes the original frontmatter into the body.
func TestApplyRefusesAnUnclosableFrontmatterFence(t *testing.T) {
_, err := Apply("---\ntitle: t\n---x\norigin: static\n---\nbody\n", Stamp{Origin: "jwt", WrittenBy: "a"})
assert.True(t, errors.Is(err, ErrUnverifiable), "got %v", err)
}
func TestApplyAddsFrontmatterToContentWithoutAny(t *testing.T) {
out, err := Apply("# plain\n\norigin: static is prose\n", Stamp{Origin: "static", WrittenBy: "local-cli"})
require.NoError(t, err)
assert.True(t, strings.HasPrefix(out, "---\norigin: static\nwritten_by: local-cli\n---\n# plain"))
}
@@ -0,0 +1,29 @@
package search
import (
"testing"
"github.com/stretchr/testify/assert"
)
// Only the leading frontmatter is provenance; a body line that looks like a
// key is prose (the doc comment's claim, now observed).
func TestExtractProvenanceIgnoresBodyLines(t *testing.T) {
o, w := extractProvenance("---\ntitle: t\n---\norigin: static\nwritten_by: local-cli\n")
assert.Equal(t, "", o)
assert.Equal(t, "", w)
o, w = extractProvenance("# no frontmatter\norigin: static\n")
assert.Equal(t, "", o)
assert.Equal(t, "", w)
o, w = extractProvenance("---\norigin: jwt\nwritten_by: attacker\n---\norigin: static\n")
assert.Equal(t, "jwt", o)
assert.Equal(t, "attacker", w)
}
// Round 3 (B1): an indented "---" inside a block scalar is content, not the
// closing fence — the same rule as frontmatter.Split.
func TestExtractProvenanceUsesTheSharedFenceRule(t *testing.T) {
o, w := extractProvenance("---\nk: |\n ---\norigin: jwt\nwritten_by: attacker\n---\nbody\n")
assert.Equal(t, "jwt", o)
assert.Equal(t, "attacker", w)
}
+46
View File
@@ -5,6 +5,7 @@ import (
"bufio" "bufio"
"context" "context"
"fmt" "fmt"
sharedfm "github.com/mathiasbq/hyperguild/ingestion/internal/frontmatter"
"log/slog" "log/slog"
"os" "os"
"path/filepath" "path/filepath"
@@ -48,6 +49,11 @@ type Result struct {
// (infra#72). Read from frontmatter when present, otherwise // (infra#72). Read from frontmatter when present, otherwise
// inferred from the parent directory. // inferred from the parent directory.
Tier string `json:"tier,omitempty"` Tier string `json:"tier,omitempty"`
// Origin/WrittenBy surface write provenance (hyperguild#145) so a
// reader can discount an entry a connector session wrote. Omitted for
// notes written before provenance existed — never fabricated.
Origin string `json:"origin,omitempty"`
WrittenBy string `json:"written_by,omitempty"`
} }
// tierWeight maps the DIKW tier to a score multiplier applied right // tierWeight maps the DIKW tier to a score multiplier applied right
@@ -146,6 +152,7 @@ func QueryContext(ctx context.Context, brainDir string, opts QueryOptions) ([]Re
rel = filepath.ToSlash(rel) rel = filepath.ToSlash(rel)
wing, hall := extractWingHall(string(content), rel) wing, hall := extractWingHall(string(content), rel)
tier := extractTier(string(content), rel) tier := extractTier(string(content), rel)
origin, writtenBy := extractProvenance(string(content))
results = append(results, Result{ results = append(results, Result{
Path: rel, Path: rel,
Title: extractTitle(string(content), d.Name()), Title: extractTitle(string(content), d.Name()),
@@ -154,6 +161,9 @@ func QueryContext(ctx context.Context, brainDir string, opts QueryOptions) ([]Re
Wing: wing, Wing: wing,
Hall: hall, Hall: hall,
Tier: tier, Tier: tier,
Origin: origin,
WrittenBy: writtenBy,
}) })
return nil return nil
}) })
@@ -272,6 +282,7 @@ func hydrate(brainDir, relPath string) (Result, error) {
} }
wing, hall := extractWingHall(string(content), relPath) wing, hall := extractWingHall(string(content), relPath)
tier := extractTier(string(content), relPath) tier := extractTier(string(content), relPath)
origin, writtenBy := extractProvenance(string(content))
return Result{ return Result{
Path: relPath, Path: relPath,
Title: extractTitle(string(content), filepath.Base(relPath)), Title: extractTitle(string(content), filepath.Base(relPath)),
@@ -279,9 +290,44 @@ func hydrate(brainDir, relPath string) (Result, error) {
Wing: wing, Wing: wing,
Hall: hall, Hall: hall,
Tier: tier, Tier: tier,
Origin: origin,
WrittenBy: writtenBy,
}, nil }, nil
} }
// extractProvenance reads origin/written_by from the leading frontmatter
// block only, located with the shared fence rule (frontmatter.Split: a "---"
// at column 0) so an indented "---" inside a block scalar is content, not a
// closing fence (PR #147 review round 3). A body line is never provenance.
func extractProvenance(content string) (origin, writtenBy string) {
block, _, ok := sharedfm.Split([]byte(content))
if !ok {
return "", ""
}
for _, line := range strings.Split(string(block), "\n") {
if line == "" || line[0] == ' ' || line[0] == '\t' {
continue
}
key, val, found := strings.Cut(line, ":")
if !found {
continue
}
v := strings.Trim(strings.TrimSpace(val), `"'`)
switch strings.TrimSpace(key) {
case "origin":
if origin == "" {
origin = v
}
case "written_by":
if writtenBy == "" {
writtenBy = v
}
}
}
return origin, writtenBy
}
// resolveRoots returns the directories to walk for the given wing/hall // resolveRoots returns the directories to walk for the given wing/hall
// filters. Validates hall against the closed vocabulary when set. // filters. Validates hall against the closed vocabulary when set.
func resolveRoots(brainDir, wing, hall string) ([]string, error) { func resolveRoots(brainDir, wing, hall string) ([]string, error) {
@@ -0,0 +1,153 @@
// Package symlinkguard is the hyperguild#157 tripwire. The brain directory is
// a Syncthing sendreceive folder and Syncthing syncs symlinks; several brain
// readers (search, vectorstore, context, graph) follow a symlink out of the
// brain and would index or return whatever it points to. Until every reader
// reads through an os.Root, a symlink inside the brain, or a directory the
// scan cannot read (which could hide one), is an incident: the server refuses
// to start on one (main), and Watch reports one that appears while it runs.
package symlinkguard
import (
"context"
"fmt"
"io/fs"
"log/slog"
"os"
"path/filepath"
"sort"
"strconv"
"strings"
"time"
)
// DefaultInterval is used when Watch is given a non-positive interval.
const DefaultInterval = 5 * time.Minute
// Notifier is the out-of-band alert sink (audit.Notifier satisfies it).
type Notifier interface {
Notify(ctx context.Context, msg string) error
}
// Result is what a scan found: symlinks, and directories it could not read
// (each of which could hide a symlink). Paths are brain-relative.
type Result struct {
Symlinks []string
Unreadable []string
}
// Clean reports whether the scan found nothing to act on.
func (r Result) Clean() bool { return len(r.Symlinks) == 0 && len(r.Unreadable) == 0 }
// String describes the findings for a log line or alert. Paths are quoted, so
// a name containing a newline cannot blur two findings together.
func (r Result) String() string {
q := func(xs []string) string {
out := make([]string, len(xs))
for i, x := range xs {
out[i] = strconv.Quote(x)
}
return strings.Join(out, ", ")
}
var parts []string
if len(r.Symlinks) > 0 {
parts = append(parts, fmt.Sprintf("%d symlink(s): %s", len(r.Symlinks), q(r.Symlinks)))
}
if len(r.Unreadable) > 0 {
parts = append(parts, fmt.Sprintf("%d unreadable dir(s): %s", len(r.Unreadable), q(r.Unreadable)))
}
return strings.Join(parts, "; ")
}
// Scan walks the brain dir and reports every symlink in it and every
// directory it could not read, continuing past both. Only the ROOT .git is
// skipped (it is brain-sync's, and no reader walks it); a nested .git is
// scanned like any other directory. brainDir itself may be a symlink (a
// volume mount); it is resolved first. The error is non-nil when the root
// cannot be resolved (wrapping os.ErrNotExist when it is missing) or cannot be
// listed.
func Scan(brainDir string) (Result, error) {
root, err := filepath.EvalSymlinks(brainDir)
if err != nil {
return Result{}, fmt.Errorf("resolve brain dir: %w", err)
}
var r Result
rootGit := filepath.Join(root, ".git")
walkErr := filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error {
rel, _ := filepath.Rel(root, p)
rel = filepath.ToSlash(rel)
if err != nil {
if p == root {
return err
}
r.Unreadable = append(r.Unreadable, rel)
if d != nil && d.IsDir() {
return filepath.SkipDir
}
return nil
}
if d.IsDir() && p == rootGit {
return filepath.SkipDir
}
if d.Type()&os.ModeSymlink != 0 {
r.Symlinks = append(r.Symlinks, rel)
}
return nil
})
sort.Strings(r.Symlinks)
sort.Strings(r.Unreadable)
if walkErr != nil {
// Only the root's own error reaches here (deeper errors are recorded
// as Unreadable). A root that cannot be listed cannot be shown clean
// (PR #159 round 2: discarding this reported clean while search
// served a planted symlink).
return r, fmt.Errorf("scan brain dir: %w", walkErr)
}
return r, nil
}
// Watch rescans every interval (DefaultInterval if non-positive) until ctx
// ends, and notifies once per distinct non-clean result. A result counts as
// reported only after Notify succeeds, so an alert lost to a notifier outage
// is retried on the next tick.
func Watch(ctx context.Context, brainDir string, interval time.Duration, n Notifier) {
if interval <= 0 {
interval = DefaultInterval
}
t := time.NewTicker(interval)
defer t.Stop()
reported := ""
for {
select {
case <-ctx.Done():
return
case <-t.C:
}
r, err := Scan(brainDir)
var key, msg string
switch {
case err != nil:
key = "error: " + err.Error()
msg = "brain symlink tripwire (hyperguild#157): cannot scan the brain dir: " + err.Error()
case r.Clean():
reported = ""
continue
default:
key = r.String()
msg = "brain symlink tripwire (hyperguild#157): " + key +
". Readers may follow symlinks out of the brain; remove them."
}
if key == reported {
continue
}
slog.Error("brain symlink tripwire", "finding", key)
if n == nil {
reported = key
continue
}
if err := n.Notify(ctx, msg); err != nil {
slog.Warn("symlink tripwire notify failed; will retry", "err", err)
continue
}
reported = key
}
}
@@ -0,0 +1,189 @@
package symlinkguard
import (
"context"
"errors"
"os"
"path/filepath"
"sync"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// hyperguild#157: the brain dir is a Syncthing sendreceive folder and
// Syncthing syncs symlinks; several readers follow them out of the brain.
// Until every reader goes through an os.Root, a symlink anywhere in the brain,
// or a directory the scan cannot read (which could hide one), is an incident.
func TestScanReportsSymlinksAtEveryDepthIncludingNestedGit(t *testing.T) {
brain := t.TempDir()
outside := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki", "w", "facts", ".git"), 0o755))
require.NoError(t, os.MkdirAll(filepath.Join(brain, ".git", "objects"), 0o755))
require.NoError(t, os.Symlink(filepath.Join(outside, "f"), filepath.Join(brain, "wiki", "w", "facts", "leak.md")))
require.NoError(t, os.Symlink(outside, filepath.Join(brain, "wiki", "dirlink")))
// PR #159 review: only the ROOT .git is skipped; a nested .git is not
// special to the brain readers, which walk into it.
require.NoError(t, os.Symlink(filepath.Join(outside, "g"), filepath.Join(brain, "wiki", "w", "facts", ".git", "leak.md")))
require.NoError(t, os.Symlink(outside, filepath.Join(brain, ".git", "objects", "alt")))
r, err := Scan(brain)
require.NoError(t, err)
assert.ElementsMatch(t, []string{"wiki/w/facts/leak.md", "wiki/dirlink", "wiki/w/facts/.git/leak.md"}, r.Symlinks)
assert.Empty(t, r.Unreadable)
}
// PR #159 review: WalkDir stopped at the first unreadable directory and the
// startup check passed, while search (which keeps walking) served a planted
// symlink. Scan now records the unreadable dir and keeps going.
func TestScanReportsUnreadableDirsAndKeepsWalkingPastThem(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root reads mode-000 directories")
}
brain := t.TempDir()
locked := filepath.Join(brain, "aaa")
require.NoError(t, os.MkdirAll(locked, 0o755))
require.NoError(t, os.Chmod(locked, 0o000))
t.Cleanup(func() { _ = os.Chmod(locked, 0o755) })
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, "zzz-planted")))
r, err := Scan(brain)
require.NoError(t, err)
assert.Equal(t, []string{"aaa"}, r.Unreadable)
assert.Equal(t, []string{"zzz-planted"}, r.Symlinks, "the walk continues past the unreadable dir")
}
func TestScanResolvesASymlinkedRootAndReportsSymlinkedEntriesInsideIt(t *testing.T) {
real := t.TempDir()
link := filepath.Join(t.TempDir(), "brain")
require.NoError(t, os.Symlink(real, link))
require.NoError(t, os.WriteFile(filepath.Join(real, "n.md"), []byte("x"), 0o644))
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(real, "inner")))
r, err := Scan(link)
require.NoError(t, err)
assert.Equal(t, []string{"inner"}, r.Symlinks)
}
func TestScanOfAMissingRootIsErrNotExist(t *testing.T) {
_, err := Scan(filepath.Join(t.TempDir(), "nope"))
assert.True(t, errors.Is(err, os.ErrNotExist), "got %v", err)
}
type recNotifier struct {
mu sync.Mutex
msgs []string
fail int // fail the first N calls
}
func (r *recNotifier) Notify(_ context.Context, m string) error {
r.mu.Lock()
defer r.mu.Unlock()
r.msgs = append(r.msgs, m)
if r.fail > 0 {
r.fail--
return errors.New("ntfy down")
}
return nil
}
func (r *recNotifier) count() int { r.mu.Lock(); defer r.mu.Unlock(); return len(r.msgs) }
func TestWatchNotifiesOncePerDistinctSetAndAgainForANewSet(t *testing.T) {
brain := t.TempDir()
n := &recNotifier{}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go Watch(ctx, brain, 10*time.Millisecond, n)
time.Sleep(50 * time.Millisecond)
assert.Zero(t, n.count(), "a clean brain raises nothing")
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, "first")))
require.Eventually(t, func() bool { return n.count() == 1 }, 2*time.Second, 10*time.Millisecond)
time.Sleep(60 * time.Millisecond)
assert.Equal(t, 1, n.count(), "the same set is reported once, not every tick")
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, "second")))
require.Eventually(t, func() bool { return n.count() == 2 }, 2*time.Second, 10*time.Millisecond)
}
// PR #159 review: last was recorded before Notify, so an alert lost to an
// ntfy outage was never retried while the symlink stayed.
func TestWatchRetriesAnAlertThatFailedToSend(t *testing.T) {
brain := t.TempDir()
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, "planted")))
n := &recNotifier{fail: 2}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go Watch(ctx, brain, 10*time.Millisecond, n)
require.Eventually(t, func() bool { return n.count() == 3 }, 2*time.Second, 10*time.Millisecond)
time.Sleep(60 * time.Millisecond)
assert.Equal(t, 3, n.count(), "after one successful send, no more")
}
func TestWatchAlertsOnAnUnreadableDir(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root reads mode-000 directories")
}
brain := t.TempDir()
locked := filepath.Join(brain, "locked")
require.NoError(t, os.MkdirAll(locked, 0o755))
require.NoError(t, os.Chmod(locked, 0o000))
t.Cleanup(func() { _ = os.Chmod(locked, 0o755) })
n := &recNotifier{}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go Watch(ctx, brain, 10*time.Millisecond, n)
require.Eventually(t, func() bool { return n.count() == 1 }, 2*time.Second, 10*time.Millisecond)
assert.Contains(t, n.msgs[0], "locked")
}
// PR #159 review: a non-positive interval panicked time.NewTicker inside the
// goroutine and crashlooped the server.
func TestWatchToleratesANonPositiveInterval(t *testing.T) {
brain := t.TempDir()
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() { Watch(ctx, brain, 0, nil); close(done) }()
cancel()
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("Watch did not return on cancel")
}
}
// PR #159 round 2: a brain root that can be entered but not listed (mode
// 0100) made WalkDir fail at the root; the error was discarded and Scan
// reported clean, while search (which goes straight into wiki/) still served
// a planted symlink. A root that cannot be listed is now an error.
func TestScanOfAnUnlistableRootIsAnError(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root lists mode-0100 directories")
}
brain := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki"), 0o755))
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, "wiki", "leak.md")))
require.NoError(t, os.Chmod(brain, 0o100))
t.Cleanup(func() { _ = os.Chmod(brain, 0o755) })
_, err := Scan(brain)
require.Error(t, err)
assert.False(t, errors.Is(err, os.ErrNotExist), "an unlistable root is not a missing root")
}
// The same set that disappears and comes back is reported again.
func TestWatchReportsASetAgainAfterItWasCleared(t *testing.T) {
brain := t.TempDir()
link := filepath.Join(brain, "planted")
n := &recNotifier{}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go Watch(ctx, brain, 10*time.Millisecond, n)
require.NoError(t, os.Symlink(t.TempDir(), link))
require.Eventually(t, func() bool { return n.count() == 1 }, 2*time.Second, 10*time.Millisecond)
require.NoError(t, os.Remove(link))
time.Sleep(60 * time.Millisecond)
require.NoError(t, os.Symlink(t.TempDir(), link))
require.Eventually(t, func() bool { return n.count() == 2 }, 2*time.Second, 10*time.Millisecond)
}
+5
View File
@@ -13,6 +13,7 @@ import (
"io" "io"
"log/slog" "log/slog"
"net/http" "net/http"
"os"
"strings" "strings"
) )
@@ -42,6 +43,10 @@ func Query(ctx context.Context, baseURL, query string, limit int) (string, error
return "", nil return "", nil
} }
req.Header.Set("Content-Type", "application/json") req.Header.Set("Content-Type", "application/json")
// The REST API requires a bearer token since hyperguild#149.
if tok := os.Getenv("BRAIN_MCP_TOKEN"); tok != "" {
req.Header.Set("Authorization", "Bearer "+tok)
}
resp, err := http.DefaultClient.Do(req) resp, err := http.DefaultClient.Do(req)
if err != nil { if err != nil {
+15
View File
@@ -65,3 +65,18 @@ func TestQueryUnavailableServerReturnsEmpty(t *testing.T) {
require.NoError(t, err) require.NoError(t, err)
assert.Empty(t, result) assert.Empty(t, result)
} }
// hyperguild#149: the REST API requires a bearer token.
func TestQuerySendsBrainMCPToken(t *testing.T) {
var auth string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
auth = r.Header.Get("Authorization")
_, _ = w.Write([]byte(`{"results":[]}`))
}))
defer srv.Close()
t.Setenv("BRAIN_MCP_TOKEN", "lib-test-token")
_, _ = brain.Query(context.Background(), srv.URL, "x", 1)
if auth != "Bearer lib-test-token" {
t.Fatalf("Authorization = %q, want the bearer token", auth)
}
}