Compare commits

...
Author SHA1 Message Date
mathiasandClaude Opus 5.5 cda18e6db8 fix(ingestion): an unlistable brain root is a scan error, not clean
CI / Lint / Test / Vet (pull_request) Successful in 44s
CI / Mirror to GitHub (pull_request) Skipped
PR #159 round 2: with the brain root at mode 0100 (enterable, not
listable) WalkDir failed at the root, Scan discarded the error and
reported clean, and startup passed while search, which goes straight
into wiki/, served a planted symlink. Scan now returns the root's walk
error, so startup refuses on anything but a missing root. Also pins
that a cleared set is reported again when it reappears.

Refs #157

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 17:14:46 +02:00
mathiasandClaude Opus 5.5 956ad8b9fb fix(ingestion): close the PR #159 review findings on the symlink tripwire
CI / Lint / Test / Vet (pull_request) Successful in 35s
CI / Mirror to GitHub (pull_request) Skipped
- Startup passed on ANY scan error: WalkDir stopped at the first
  unreadable dir and Scan discarded what it had found, while search
  kept walking and served a planted symlink. Scan now records
  unreadable dirs and keeps going; startup refuses on symlinks or
  unreadable dirs and only lets a missing brain dir pass.
- Watch went silent while a scan error persisted, and marked a set as
  reported before Notify, so an ntfy outage lost the alert. It now
  alerts on unreadable dirs and scan errors too, and records a set as
  reported only after Notify succeeds (retried each tick).
- Only the root .git is skipped; a nested .git is scanned.
- A non-positive BRAIN_SYMLINK_SCAN_INTERVAL fell into time.NewTicker
  and crashlooped the server; it now falls back to the default.
- Findings are quoted in alerts, so a newline in a name cannot merge
  two sets; alertingWarnings names the tripwire when ntfy is unset.
- Test names match what they assert (two-set dedupe, symlinked entries
  inside a symlinked root, nested .git).

Flipped: startup ignoring unreadable dirs, .git skipped at any depth,
and no retry each turn a test red. Watch tests stable over -count=10
and under -race.

Refs #157

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 17:11:45 +02:00
mathiasandClaude Opus 5.5 796d226497 feat(ingestion): symlink tripwire for the Syncthing-shared brain dir
CI / Lint / Test / Vet (pull_request) Successful in 34s
CI / Mirror to GitHub (pull_request) Skipped
The brain dir is a Syncthing sendreceive folder and Syncthing syncs
symlinks; search, vectorstore, context and graph readers follow one out
of the brain and would index or return what it points to (reproduced
with search.Query). Decision (#157): a tripwire now, brain-wide os.Root
readers next.

- symlinkguard.Scan lists every symlink inside the brain dir (skipping
  .git; the brain dir itself may be a mount symlink).
- At startup the server refuses to start while any exists, naming them.
- symlinkguard.Watch rescans every BRAIN_SYMLINK_SCAN_INTERVAL seconds
  (default 300) and sends one ntfy alert per new set of symlinks.

The live brain dir had 0 symlinks on 2026-10-03, checked before this
change, so deploying it does not trip.

Refs #157

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 17:04:41 +02:00
mathias cfe3f24e25 fix(ingestion): authenticate the REST API; confine ingest paths to brain/raw/ (#149) (#152)
CI / Lint / Test / Vet (push) Successful in 34s
CI / Mirror to GitHub (push) Successful in 4s
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-03 14:38:07 +00:00
23 changed files with 949 additions and 38 deletions
+5 -1
View File
@@ -64,7 +64,11 @@ the routing pod; brain tools moved to the brain MCP.
The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`, The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`,
`/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300 `/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300
for shell scripts and non-MCP clients. for shell scripts and non-MCP clients, **with the same bearer token as `/mcp`**
(`BRAIN_MCP_TOKEN`; hyperguild#149). It is not served on the public host
(infra#517): use the tailnet (`koala:8443`) or in-cluster
`ingestion.supervisor.svc:3300`. `/ingest-path` and `brain_ingest` `path`
accept only paths inside `brain/raw/`.
`brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai `brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai
gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title, gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title,
+6 -2
View File
@@ -304,7 +304,7 @@ entries that age well are about *why*, *how to avoid*, and *what to do when*.
|---------|-------|-------| |---------|-------|-------|
| **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool | | **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool |
| **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same | | **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same |
| **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` | | **Anything HTTP-only (curl, scripts)** | Tailnet only: `POST https://koala.tail30472e.ts.net:8443/query` with `{"query":"..."}`, sending `BRAIN_MCP_TOKEN` (not yet enforced on REST, hyperguild#149). The public `brain-mcp.d-ma.be` serves `/mcp` only (infra#517). | `POST .../write` with `{"content":"...","wing":"...","hall":"..."}` |
| **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files | | **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files |
- **Scoping — the brain holds no confidential client content.** Nothing on the read path - **Scoping — the brain holds no confidential client content.** Nothing on the read path
@@ -493,7 +493,11 @@ the routing pod; brain tools moved to the brain MCP.
The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`, The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`,
`/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300 `/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300
for shell scripts and non-MCP clients. for shell scripts and non-MCP clients, **with the same bearer token as `/mcp`**
(`BRAIN_MCP_TOKEN`; hyperguild#149). It is not served on the public host
(infra#517): use the tailnet (`koala:8443`) or in-cluster
`ingestion.supervisor.svc:3300`. `/ingest-path` and `brain_ingest` `path`
accept only paths inside `brain/raw/`.
`brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai `brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai
gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title, gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title,
+6 -2
View File
@@ -299,7 +299,7 @@ entries that age well are about *why*, *how to avoid*, and *what to do when*.
|---------|-------|-------| |---------|-------|-------|
| **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool | | **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool |
| **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same | | **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same |
| **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` | | **Anything HTTP-only (curl, scripts)** | Tailnet only: `POST https://koala.tail30472e.ts.net:8443/query` with `{"query":"..."}`, sending `BRAIN_MCP_TOKEN` (not yet enforced on REST, hyperguild#149). The public `brain-mcp.d-ma.be` serves `/mcp` only (infra#517). | `POST .../write` with `{"content":"...","wing":"...","hall":"..."}` |
| **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files | | **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files |
- **Scoping — the brain holds no confidential client content.** Nothing on the read path - **Scoping — the brain holds no confidential client content.** Nothing on the read path
@@ -488,7 +488,11 @@ the routing pod; brain tools moved to the brain MCP.
The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`, The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`,
`/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300 `/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300
for shell scripts and non-MCP clients. for shell scripts and non-MCP clients, **with the same bearer token as `/mcp`**
(`BRAIN_MCP_TOKEN`; hyperguild#149). It is not served on the public host
(infra#517): use the tailnet (`koala:8443`) or in-cluster
`ingestion.supervisor.svc:3300`. `/ingest-path` and `brain_ingest` `path`
accept only paths inside `brain/raw/`.
`brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai `brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai
gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title, gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title,
+5 -1
View File
@@ -64,7 +64,11 @@ the routing pod; brain tools moved to the brain MCP.
The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`, The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`,
`/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300 `/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300
for shell scripts and non-MCP clients. for shell scripts and non-MCP clients, **with the same bearer token as `/mcp`**
(`BRAIN_MCP_TOKEN`; hyperguild#149). It is not served on the public host
(infra#517): use the tailnet (`koala:8443`) or in-cluster
`ingestion.supervisor.svc:3300`. `/ingest-path` and `brain_ingest` `path`
accept only paths inside `brain/raw/`.
`brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai `brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai
gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title, gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title,
+15 -5
View File
@@ -20,6 +20,7 @@ const defaultBrainURL = "http://koala:30330"
// shell-friendly client; MCP framing is unnecessary. // shell-friendly client; MCP framing is unnecessary.
type brainClient struct { type brainClient struct {
baseURL string baseURL string
token string // BRAIN_MCP_TOKEN; the REST API requires it since hyperguild#149
http *http.Client http *http.Client
} }
@@ -30,6 +31,7 @@ func newBrainClient() *brainClient {
} }
return &brainClient{ return &brainClient{
baseURL: u, baseURL: u,
token: os.Getenv("BRAIN_MCP_TOKEN"),
http: &http.Client{Timeout: 5 * time.Second}, http: &http.Client{Timeout: 5 * time.Second},
} }
} }
@@ -62,8 +64,7 @@ func (c *brainClient) Query(ctx context.Context, topic string, limit int) (*Quer
return nil, fmt.Errorf("build request: %w", err) return nil, fmt.Errorf("build request: %w", err)
} }
req.Header.Set("Content-Type", "application/json") req.Header.Set("Content-Type", "application/json")
resp, err := c.do(req)
resp, err := c.http.Do(req)
if err != nil { if err != nil {
return nil, fmt.Errorf("brain POST /query: %w", err) return nil, fmt.Errorf("brain POST /query: %w", err)
} }
@@ -105,8 +106,7 @@ func (c *brainClient) Write(ctx context.Context, wing, hall, filename string, co
return nil, fmt.Errorf("build request: %w", err) return nil, fmt.Errorf("build request: %w", err)
} }
req.Header.Set("Content-Type", "application/json") req.Header.Set("Content-Type", "application/json")
resp, err := c.do(req)
resp, err := c.http.Do(req)
if err != nil { if err != nil {
return nil, fmt.Errorf("brain POST /write: %w", err) return nil, fmt.Errorf("brain POST /write: %w", err)
} }
@@ -143,7 +143,7 @@ func (c *brainClient) PassRate(ctx context.Context, skill, window string) (*Pass
if err != nil { if err != nil {
return nil, fmt.Errorf("build request: %w", err) return nil, fmt.Errorf("build request: %w", err)
} }
resp, err := c.http.Do(req) resp, err := c.do(req)
if err != nil { if err != nil {
return nil, fmt.Errorf("brain GET /pass-rate: %w", err) return nil, fmt.Errorf("brain GET /pass-rate: %w", err)
} }
@@ -158,3 +158,13 @@ func (c *brainClient) PassRate(ctx context.Context, skill, window string) (*Pass
} }
return &out, nil return &out, nil
} }
// do sends every brain request, adding the bearer token when one is
// configured. All methods go through here so none can forget it (PassRate
// did, PR #152 review).
func (c *brainClient) do(req *http.Request) (*http.Response, error) {
if c.token != "" {
req.Header.Set("Authorization", "Bearer "+c.token)
}
return c.http.Do(req)
}
+57
View File
@@ -0,0 +1,57 @@
package main
import (
"context"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// hyperguild#149: the ingestion REST API now requires a bearer token; the CLI
// sends BRAIN_MCP_TOKEN, or it gets 401 on every brain call.
func TestBrainClientSendsBrainMCPTokenOnQueryWriteAndPassRate(t *testing.T) {
var got []string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
got = append(got, r.URL.Path+" "+r.Header.Get("Authorization"))
switch r.URL.Path {
case "/write":
_, _ = w.Write([]byte(`{"path":"wiki/x/facts/y.md"}`))
return
case "/pass-rate":
_, _ = w.Write([]byte(`{}`))
return
}
_, _ = w.Write([]byte(`{"results":[]}`))
}))
defer srv.Close()
t.Setenv("BRAIN_URL", srv.URL)
t.Setenv("BRAIN_MCP_TOKEN", "cli-test-token")
c := newBrainClient()
_, err := c.Query(context.Background(), "x", 1)
require.NoError(t, err)
_, err = c.Write(context.Background(), "x", "facts", "y", strings.NewReader("# y"))
require.NoError(t, err)
// PR #152 review: PassRate sent no token and would 401 in production.
_, _ = c.PassRate(context.Background(), "tdd", "7d")
assert.Equal(t, []string{"/query Bearer cli-test-token", "/write Bearer cli-test-token",
"/pass-rate Bearer cli-test-token"}, got)
}
func TestBrainClientSendsNoAuthorizationWhenTokenUnset(t *testing.T) {
var auth string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
auth = r.Header.Get("Authorization")
_, _ = w.Write([]byte(`{"results":[]}`))
}))
defer srv.Close()
t.Setenv("BRAIN_URL", srv.URL)
t.Setenv("BRAIN_MCP_TOKEN", "")
_, err := newBrainClient().Query(context.Background(), "x", 1)
require.NoError(t, err)
assert.Empty(t, auth)
}
@@ -24,7 +24,7 @@ Only a `tools/call` to the routing pod records a pass/fail.
|---------|-----|------| |---------|-----|------|
| Routing MCP (local, Tailscale) | `http://koala:30310/mcp` | Bearer `ROUTING_MCP_TOKEN` | | Routing MCP (local, Tailscale) | `http://koala:30310/mcp` | Bearer `ROUTING_MCP_TOKEN` |
| Routing MCP (remote) | `https://routing-mcp.d-ma.be/mcp` | OAuth via `auth.d-ma.be` (audience `claude-ai`) | | Routing MCP (remote) | `https://routing-mcp.d-ma.be/mcp` | OAuth via `auth.d-ma.be` (audience `claude-ai`) |
| Pass-rate readout | `http://koala:30330/pass-rate?skill=<name>` | none (read-only) | | Pass-rate readout | `http://koala:30330/pass-rate?skill=<name>` | Bearer `BRAIN_MCP_TOKEN` (since hyperguild#149) |
Tools advertised: **`review`**, **`debug`** (the two the #35 gate measures), Tools advertised: **`review`**, **`debug`** (the two the #35 gate measures),
plus `session_log`, `retrospective`, `trainer`. plus `session_log`, `retrospective`, `trainer`.
+28 -20
View File
@@ -33,6 +33,7 @@ import (
"github.com/mathiasbq/hyperguild/ingestion/internal/reranker" "github.com/mathiasbq/hyperguild/ingestion/internal/reranker"
"github.com/mathiasbq/hyperguild/ingestion/internal/search" "github.com/mathiasbq/hyperguild/ingestion/internal/search"
"github.com/mathiasbq/hyperguild/ingestion/internal/startupretry" "github.com/mathiasbq/hyperguild/ingestion/internal/startupretry"
"github.com/mathiasbq/hyperguild/ingestion/internal/symlinkguard"
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore" "github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
"github.com/mathiasbq/hyperguild/ingestion/internal/watcher" "github.com/mathiasbq/hyperguild/ingestion/internal/watcher"
"github.com/mathiasbq/hyperguild/ingestion/internal/webhook" "github.com/mathiasbq/hyperguild/ingestion/internal/webhook"
@@ -173,7 +174,8 @@ func alertingWarnings(getenv func(string) string) []string {
var out []string var out []string
if getenv("BRAIN_NTFY_URL") == "" { if getenv("BRAIN_NTFY_URL") == "" {
out = append(out, "alerting DISABLED: BRAIN_NTFY_URL unset — "+ out = append(out, "alerting DISABLED: BRAIN_NTFY_URL unset — "+
"brain_edit_root governance rewrites and capture audit degradation will not notify") "brain_edit_root governance rewrites, capture audit degradation and the "+
"brain symlink tripwire (hyperguild#157) will not notify")
} }
return out return out
} }
@@ -239,6 +241,10 @@ func main() {
} }
brainDir := envOr("INGEST_BRAIN_DIR", "../brain") brainDir := envOr("INGEST_BRAIN_DIR", "../brain")
if err := startupSymlinkCheck(brainDir); err != nil {
logger.Error("brain symlink tripwire", "err", err)
os.Exit(1)
}
port := envOr("INGEST_PORT", "3300") port := envOr("INGEST_PORT", "3300")
llmURL := envOr("INGEST_LLM_URL", "http://iguana:4000/v1") llmURL := envOr("INGEST_LLM_URL", "http://iguana:4000/v1")
@@ -360,6 +366,8 @@ func main() {
} }
ctx := context.Background() ctx := context.Background()
// hyperguild#157: report a symlink that appears in the brain while running.
go symlinkguard.Watch(ctx, brainDir, symlinkScanInterval(os.Getenv), buildNotifier())
// Built before the watcher, the graph backfill and session ingestion, and // Built before the watcher, the graph backfill and session ingestion, and
// retried rather than fatal (#83). Two separate reasons: // retried rather than fatal (#83). Two separate reasons:
@@ -503,20 +511,10 @@ func main() {
logger.Info("embed sync started", "interval_s", embedSyncInterval) logger.Info("embed sync started", "interval_s", embedSyncInterval)
} }
mux := http.NewServeMux() // Public routes only; buildMux mounts the authenticated REST API (#149).
mux.HandleFunc("POST /query", h.Query) public := map[string]http.Handler{}
mux.HandleFunc("POST /write", h.Write)
mux.HandleFunc("POST /index", h.Index)
mux.HandleFunc("POST /ingest", h.Ingest)
mux.HandleFunc("POST /ingest-path", h.IngestPath)
mux.HandleFunc("POST /ingest-raw", h.IngestRaw)
mux.HandleFunc("POST /backfill-refs", h.BackfillRefs)
mux.HandleFunc("GET /pending", h.Pending)
mux.HandleFunc("POST /promote", h.Promote)
mux.HandleFunc("POST /backfill-embeddings", h.BackfillEmbeddings)
mux.HandleFunc("GET /pass-rate", h.PassRate)
if webhookHandler != nil { if webhookHandler != nil {
mux.Handle("POST /webhooks/brain-sync", webhookHandler) public["POST /webhooks/brain-sync"] = webhookHandler
} }
// Resource-metadata URL is only emitted on 401 when Dex OAuth is // Resource-metadata URL is only emitted on 401 when Dex OAuth is
// configured. Static-Bearer-only deployments leave this empty so // configured. Static-Bearer-only deployments leave this empty so
@@ -524,14 +522,14 @@ func main() {
var resourceMetadataURL string var resourceMetadataURL string
if dexURL := os.Getenv("DEX_ISSUER_URL"); dexURL != "" { if dexURL := os.Getenv("DEX_ISSUER_URL"); dexURL != "" {
resourceURL := os.Getenv("MCP_RESOURCE_URL") resourceURL := os.Getenv("MCP_RESOURCE_URL")
mux.HandleFunc("GET /.well-known/oauth-protected-resource", public["GET /.well-known/oauth-protected-resource"] = http.HandlerFunc(
chassisauth.ProtectedResourceHandler(resourceURL, dexURL)) chassisauth.ProtectedResourceHandler(resourceURL, dexURL))
if resourceURL != "" { if resourceURL != "" {
resourceMetadataURL = strings.TrimRight(resourceURL, "/") + "/.well-known/oauth-protected-resource" resourceMetadataURL = strings.TrimRight(resourceURL, "/") + "/.well-known/oauth-protected-resource"
} }
} }
mux.Handle("/mcp", chassisauth.BearerMiddleware(mcpToken, jwtValidator, "brain", resourceMetadataURL, mcpSrv)) public["/mcp"] = chassisauth.BearerMiddleware(mcpToken, jwtValidator, "brain", resourceMetadataURL, mcpSrv)
// POST /capture (#53/#54): the uniform capture REST door. Needs a ticket // POST /capture (#53/#54): the uniform capture REST door. Needs a ticket
// tracker to file action items, so it only mounts when Gitea is // tracker to file action items, so it only mounts when Gitea is
@@ -553,7 +551,7 @@ func main() {
sovereign := splitList(os.Getenv("BRAIN_CAPTURE_SOVEREIGN_PRINCIPALS")) sovereign := splitList(os.Getenv("BRAIN_CAPTURE_SOVEREIGN_PRINCIPALS"))
resolver := capturehttp.NewOriginResolver(sovereign) resolver := capturehttp.NewOriginResolver(sovereign)
captureH := capturehttp.New(captureSvc, jwtValidator, mcpToken, "local-cli", resolver) captureH := capturehttp.New(captureSvc, jwtValidator, mcpToken, "local-cli", resolver)
mux.Handle("POST /capture", captureH) public["POST /capture"] = captureH
// Same use-case behind the MCP `capture` tool (#55 relay) so MCP-native // Same use-case behind the MCP `capture` tool (#55 relay) so MCP-native
// harnesses (claude.ai, Crush, Pi, LLM Council) reach capture through // harnesses (claude.ai, Crush, Pi, LLM Council) reach capture through
// the existing /mcp OAuth connector. mcpSrv is already wrapped above; // the existing /mcp OAuth connector. mcpSrv is already wrapped above;
@@ -583,9 +581,9 @@ func main() {
logger.Error("OAUTH_CLIENT_ID/SECRET set but MCP_RESOURCE_URL is empty; cannot derive issuer") logger.Error("OAUTH_CLIENT_ID/SECRET set but MCP_RESOURCE_URL is empty; cannot derive issuer")
os.Exit(1) os.Exit(1)
} }
mux.HandleFunc("GET /.well-known/oauth-authorization-server", public["GET /.well-known/oauth-authorization-server"] = http.HandlerFunc(
oauth.MetadataHandler(issuer)) oauth.MetadataHandler(issuer))
mux.HandleFunc("POST /oauth/token", oauth.TokenHandler(oauth.TokenConfig{ public["POST /oauth/token"] = http.HandlerFunc(oauth.TokenHandler(oauth.TokenConfig{
ClientID: oauthID, ClientID: oauthID,
ClientSecret: oauthSecret, ClientSecret: oauthSecret,
AccessToken: mcpToken, AccessToken: mcpToken,
@@ -604,7 +602,17 @@ func main() {
// real request latency. /metrics itself is excluded from its own // real request latency. /metrics itself is excluded from its own
// observation by registering it on the outer mux (post-wrap). // observation by registering it on the outer mux (post-wrap).
reg := metrics.New() reg := metrics.New()
mux.HandleFunc("GET /metrics", reg.Handler()) public["GET /metrics"] = http.HandlerFunc(reg.Handler())
// The whole router, built in one place so the test of buildMux is a test
// of production wiring (PR #152 review). The REST API's credential is the
// same as /mcp's.
mux, err := buildMux(restAuth{token: mcpToken, validator: jwtValidator, realm: "brain",
resourceMetadataURL: resourceMetadataURL}, h, public)
if err != nil {
logger.Error("build router", "err", err)
os.Exit(1)
}
logger.Info("metrics endpoint registered", "path", "/metrics") logger.Info("metrics endpoint registered", "path", "/metrics")
addr := ":" + port addr := ":" + port
+77
View File
@@ -0,0 +1,77 @@
package main
import (
"errors"
"fmt"
"net/http"
chassisauth "git.d-ma.be/mathias/mcp-chassis/auth"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
)
// publicRoutes is the complete set of patterns allowed on the outer mux,
// outside the REST API's auth (hyperguild#149). Each authenticates itself or is
// meant to be open: /mcp (bearer/JWT), /capture (bearer/JWT), the webhook
// (HMAC), /oauth/token (client secret), OAuth metadata and /metrics (open).
// A REST pattern added here would beat the authenticated catch-all by
// specificity and ship open, which is why buildMux refuses anything else.
var publicRoutes = map[string]bool{
"/mcp": true,
"POST /capture": true,
"POST /webhooks/brain-sync": true,
"GET /.well-known/oauth-protected-resource": true,
"GET /.well-known/oauth-authorization-server": true,
"POST /oauth/token": true,
"GET /metrics": true,
}
// restAuth is the credential the REST API checks: the same as /mcp's.
type restAuth struct {
token string
validator *chassisauth.JWTValidator
realm string
resourceMetadataURL string
}
// buildMux is the server's whole router. The REST API is mounted at "/"
// behind bearer auth that buildMux constructs itself, so a caller cannot pass
// a no-op; public handlers are registered as given, and only for patterns on
// the publicRoutes allowlist. It returns an http.Handler, not the
// *ServeMux, so nothing can register a route on the result afterwards and
// bypass the allowlist.
func buildMux(auth restAuth, h *api.Handler, public map[string]http.Handler) (http.Handler, error) {
if auth.token == "" {
return nil, errors.New("buildMux: REST auth needs a token")
}
mux := http.NewServeMux()
for pattern, handler := range public {
if !publicRoutes[pattern] {
return nil, fmt.Errorf("buildMux: %q is not an allowed public route", pattern)
}
mux.Handle(pattern, handler)
}
mountREST(mux, h, func(next http.Handler) http.Handler {
return chassisauth.BearerMiddleware(auth.token, auth.validator, auth.realm, auth.resourceMetadataURL, next)
})
return mux, nil
}
// mountREST registers the REST API on its own mux and mounts it at "/" behind
// auth: anything not registered as public lands here and is authenticated,
// including routes added later.
func mountREST(mux *http.ServeMux, h *api.Handler, auth func(http.Handler) http.Handler) {
rest := http.NewServeMux()
rest.HandleFunc("POST /query", h.Query)
rest.HandleFunc("POST /write", h.Write)
rest.HandleFunc("POST /index", h.Index)
rest.HandleFunc("POST /ingest", h.Ingest)
rest.HandleFunc("POST /ingest-path", h.IngestPath)
rest.HandleFunc("POST /ingest-raw", h.IngestRaw)
rest.HandleFunc("POST /backfill-refs", h.BackfillRefs)
rest.HandleFunc("GET /pending", h.Pending)
rest.HandleFunc("POST /promote", h.Promote)
rest.HandleFunc("POST /backfill-embeddings", h.BackfillEmbeddings)
rest.HandleFunc("GET /pass-rate", h.PassRate)
mux.Handle("/", auth(rest))
}
+106
View File
@@ -0,0 +1,106 @@
package main
import (
"log/slog"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
)
const routesTok = "routes-test-token"
func okHandler(body string) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _, _ = w.Write([]byte(body)) })
}
// testMux builds the router exactly as main does: buildMux with the full
// public set main registers (stub handlers) and real bearer config.
// hyperguild#149: the REST API is authenticated as a whole; the PR #152
// review showed a test of a COPY of the router let main.go regress unseen.
func testMux(t *testing.T) http.Handler {
t.Helper()
h := api.NewHandler(t.TempDir(), slog.New(slog.NewTextHandler(os.Stderr, nil)), pipeline.Config{})
public := map[string]http.Handler{}
for p := range publicRoutes {
public[p] = okHandler("public " + p)
}
mux, err := buildMux(restAuth{token: routesTok, realm: "brain"}, h, public)
require.NoError(t, err)
return mux
}
func do(t *testing.T, h http.Handler, method, path, authz string) (int, string) {
t.Helper()
req := httptest.NewRequest(method, path, strings.NewReader(`{"query":"x"}`))
if authz != "" {
req.Header.Set("Authorization", authz)
}
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
return rec.Code, rec.Body.String()
}
func TestBuildMuxRefusesRESTRoutesWithoutAValidToken(t *testing.T) {
r := testMux(t)
for _, rt := range []struct{ method, path string }{
{"POST", "/query"}, {"POST", "/write"}, {"POST", "/index"}, {"POST", "/ingest"},
{"POST", "/ingest-path"}, {"POST", "/ingest-raw"}, {"POST", "/backfill-refs"},
{"GET", "/pending"}, {"POST", "/promote"}, {"POST", "/backfill-embeddings"}, {"GET", "/pass-rate"},
} {
code, _ := do(t, r, rt.method, rt.path, "")
assert.Equal(t, http.StatusUnauthorized, code, "%s %s without a token", rt.method, rt.path)
code, _ = do(t, r, rt.method, rt.path, "Bearer wrong")
assert.Equal(t, http.StatusUnauthorized, code, "%s %s with a wrong token", rt.method, rt.path)
}
}
func TestBuildMuxServesRESTRoutesToAValidToken(t *testing.T) {
r := testMux(t)
code, _ := do(t, r, "GET", "/pending", "Bearer "+routesTok)
assert.Equal(t, http.StatusOK, code)
}
func TestBuildMuxAuthenticatesAnUnregisteredPath(t *testing.T) {
code, _ := do(t, testMux(t), "POST", "/some-future-route", "")
assert.Equal(t, http.StatusUnauthorized, code)
}
// buildMux registers public handlers as given, without adding auth: each
// public route owns its own (bearer on /mcp, HMAC on the webhook, client
// secret on /oauth/token) or is meant to be open (/metrics, OAuth metadata).
func TestBuildMuxDoesNotWrapPublicRoutes(t *testing.T) {
r := testMux(t)
for p := range publicRoutes {
method, path, ok := strings.Cut(p, " ")
if !ok {
method, path = "GET", p
}
code, body := do(t, r, method, path, "")
assert.Equal(t, http.StatusOK, code, p)
assert.Equal(t, "public "+p, body, p)
}
}
// The allowlist is the control: a REST pattern registered as public would win
// over the authenticated catch-all (more specific pattern), so it is refused.
func TestBuildMuxRefusesAPublicPatternOffTheAllowlist(t *testing.T) {
h := api.NewHandler(t.TempDir(), slog.New(slog.NewTextHandler(os.Stderr, nil)), pipeline.Config{})
for _, p := range []string{"POST /query", "GET /pending", "/", "POST /metrics"} {
_, err := buildMux(restAuth{token: routesTok, realm: "brain"}, h, map[string]http.Handler{p: okHandler("x")})
assert.Error(t, err, "public pattern %q must be refused", p)
}
}
func TestBuildMuxRefusesAnEmptyToken(t *testing.T) {
h := api.NewHandler(t.TempDir(), slog.New(slog.NewTextHandler(os.Stderr, nil)), pipeline.Config{})
_, err := buildMux(restAuth{token: "", realm: "brain"}, h, nil)
assert.Error(t, err)
}
+39
View File
@@ -0,0 +1,39 @@
package main
import (
"errors"
"fmt"
"os"
"strconv"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/symlinkguard"
)
// startupSymlinkCheck is the hyperguild#157 tripwire at boot: a symlink in the
// brain dir would let brain readers serve a file from outside the brain, so
// the server does not start until it is removed. A directory the scan cannot
// read could hide one, so it refuses on that too. A missing brain dir (first
// boot) is the only case it lets pass.
func startupSymlinkCheck(brainDir string) error {
r, err := symlinkguard.Scan(brainDir)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
return nil
}
return fmt.Errorf("refusing to start: cannot scan the brain dir %s (hyperguild#157): %w", brainDir, err)
}
if !r.Clean() {
return fmt.Errorf("refusing to start: brain dir %s has %s (hyperguild#157); fix them first", brainDir, r)
}
return nil
}
// symlinkScanInterval reads BRAIN_SYMLINK_SCAN_INTERVAL (seconds), falling
// back to the default on an unset, non-numeric or non-positive value.
func symlinkScanInterval(getenv func(string) string) time.Duration {
if s, err := strconv.Atoi(getenv("BRAIN_SYMLINK_SCAN_INTERVAL")); err == nil && s > 0 {
return time.Duration(s) * time.Second
}
return symlinkguard.DefaultInterval
}
@@ -0,0 +1,65 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// hyperguild#157: the server refuses to start while a symlink exists in the
// brain dir, or while any part of it cannot be scanned (which could hide
// one); a missing brain dir is the only thing it lets pass.
func TestStartupSymlinkCheckRefusesABrainWithASymlink(t *testing.T) {
brain := t.TempDir()
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, "planted")))
err := startupSymlinkCheck(brain)
require.Error(t, err)
assert.Contains(t, err.Error(), "planted")
}
func TestStartupSymlinkCheckRefusesABrainWithAnUnreadableDir(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root reads mode-000 directories")
}
brain := t.TempDir()
locked := filepath.Join(brain, "aaa")
require.NoError(t, os.MkdirAll(locked, 0o755))
require.NoError(t, os.Chmod(locked, 0o000))
t.Cleanup(func() { _ = os.Chmod(locked, 0o755) })
err := startupSymlinkCheck(brain)
require.Error(t, err)
assert.Contains(t, err.Error(), "aaa")
}
func TestStartupSymlinkCheckPassesACleanOrMissingBrain(t *testing.T) {
brain := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(brain, "n.md"), []byte("x"), 0o644))
assert.NoError(t, startupSymlinkCheck(brain))
assert.NoError(t, startupSymlinkCheck(filepath.Join(brain, "missing")))
}
func TestSymlinkScanIntervalFallsBackOnNonPositive(t *testing.T) {
for _, v := range []string{"0", "-5", "", "junk"} {
got := symlinkScanInterval(func(string) string { return v })
assert.Positive(t, got, "env %q", v)
}
}
func TestAlertingWarningsNamesTheSymlinkTripwire(t *testing.T) {
w := strings.Join(alertingWarnings(func(string) string { return "" }), " ")
assert.Contains(t, w, "symlink tripwire")
}
func TestStartupSymlinkCheckRefusesAnUnlistableBrainRoot(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root lists mode-0100 directories")
}
brain := t.TempDir()
require.NoError(t, os.Chmod(brain, 0o100))
t.Cleanup(func() { _ = os.Chmod(brain, 0o755) })
assert.Error(t, startupSymlinkCheck(brain))
}
+12
View File
@@ -447,6 +447,12 @@ func (h *Handler) IngestPath(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, "path is required") writeError(w, http.StatusBadRequest, "path is required")
return return
} }
resolved, err := ResolveIngestPath(h.brainDir, req.Path)
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
req.Path = resolved
info, err := os.Stat(req.Path) info, err := os.Stat(req.Path)
if err != nil { if err != nil {
@@ -469,6 +475,12 @@ func (h *Handler) IngestPath(w http.ResponseWriter, r *http.Request) {
if !isSupportedExtension(ext) { if !isSupportedExtension(ext) {
return nil return nil
} }
// Each walked file is re-resolved: a symlinked file inside raw/
// would otherwise lead extract.Text out of the inbox (#149).
if _, rerr := ResolveIngestPath(h.brainDir, path); rerr != nil {
allWarnings = append(allWarnings, fmt.Sprintf("skipped %s: %v", path, rerr))
return nil
}
content, readErr := extract.Text(path) content, readErr := extract.Text(path)
if readErr != nil { if readErr != nil {
allWarnings = append(allWarnings, fmt.Sprintf("extract %s: %v", path, readErr)) allWarnings = append(allWarnings, fmt.Sprintf("extract %s: %v", path, readErr))
+6 -4
View File
@@ -361,10 +361,11 @@ func TestIngestPath_MissingPath(t *testing.T) {
} }
func TestIngestPath_File(t *testing.T) { func TestIngestPath_File(t *testing.T) {
_, h := setup(t) brainDir, h := setup(t)
// Create a temp file with content // Create a temp file with content
dir := t.TempDir() dir := filepath.Join(brainDir, "raw", "fixture") // ingest paths must live in brain/raw/ (#149)
require.NoError(t, os.MkdirAll(dir, 0o755))
f := filepath.Join(dir, "doc.md") f := filepath.Join(dir, "doc.md")
require.NoError(t, os.WriteFile(f, []byte("# Hello\nThis is markdown content."), 0o644)) require.NoError(t, os.WriteFile(f, []byte("# Hello\nThis is markdown content."), 0o644))
@@ -468,10 +469,11 @@ func TestIngestRaw_DryRun(t *testing.T) {
} }
func TestIngestPath_Directory(t *testing.T) { func TestIngestPath_Directory(t *testing.T) {
_, h := setup(t) brainDir, h := setup(t)
// Create a temp dir with one .md file // Create a temp dir with one .md file
dir := t.TempDir() dir := filepath.Join(brainDir, "raw", "fixture") // ingest paths must live in brain/raw/ (#149)
require.NoError(t, os.MkdirAll(dir, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "notes.md"), []byte("# Notes\nSome notes."), 0o644)) require.NoError(t, os.WriteFile(filepath.Join(dir, "notes.md"), []byte("# Notes\nSome notes."), 0o644))
body, _ := json.Marshal(map[string]any{ body, _ := json.Marshal(map[string]any{
@@ -40,9 +40,10 @@ func TestIngestPath_DocxSupportedWhenDocmarkConfigured(t *testing.T) {
t.Setenv("DOCMARK_URL", srv.URL+"/mcp") t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
t.Setenv("DOCMARK_BEARER_TOKEN", "tok") t.Setenv("DOCMARK_BEARER_TOKEN", "tok")
_, h := setup(t) brainDir, h := setup(t)
dir := t.TempDir() dir := filepath.Join(brainDir, "raw", "fixture") // ingest paths must live in brain/raw/ (#149)
require.NoError(t, os.MkdirAll(dir, 0o755))
f := filepath.Join(dir, "doc.docx") f := filepath.Join(dir, "doc.docx")
require.NoError(t, os.WriteFile(f, []byte("fake docx"), 0o644)) require.NoError(t, os.WriteFile(f, []byte("fake docx"), 0o644))
+38
View File
@@ -0,0 +1,38 @@
package api
import (
"errors"
"path/filepath"
"strings"
)
// ErrOutsideIngestRoot is the one refusal for every bad ingest path. It never
// carries the underlying error: "no such file" versus "permission denied"
// would let a caller probe which server files exist (PR #152 review).
var ErrOutsideIngestRoot = errors.New("ingest path must be an existing file or directory inside brain/raw/")
// ResolveIngestPath confines a caller-supplied ingest path to brain/raw/, the
// inbox the watcher already owns (hyperguild#149). Relative paths resolve
// against that root; symlinks are evaluated before the containment check, so
// a link inside raw/ that points elsewhere is refused. The path must exist.
func ResolveIngestPath(brainDir, p string) (string, error) {
if strings.TrimSpace(p) == "" {
return "", ErrOutsideIngestRoot
}
root, err := filepath.EvalSymlinks(filepath.Join(brainDir, "raw"))
if err != nil {
return "", ErrOutsideIngestRoot
}
if !filepath.IsAbs(p) {
p = filepath.Join(root, p)
}
real, err := filepath.EvalSymlinks(filepath.Clean(p))
if err != nil {
return "", ErrOutsideIngestRoot
}
rel, err := filepath.Rel(root, real)
if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) || filepath.IsAbs(rel) {
return "", ErrOutsideIngestRoot
}
return real, nil
}
@@ -0,0 +1,88 @@
package api_test
import (
"bytes"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
)
// hyperguild#149: /ingest-path and MCP brain_ingest read a caller-chosen
// server path, so any file the pod could read (the ServiceAccount token,
// /etc/passwd) could be pulled into the brain and read back out. Ingest paths
// now resolve inside brain/raw/ — the inbox the watcher already owns — with
// symlinks evaluated, or are refused.
func TestResolveIngestPathConfinesToRawInbox(t *testing.T) {
brain := t.TempDir()
raw := filepath.Join(brain, "raw")
require.NoError(t, os.MkdirAll(filepath.Join(raw, "sub"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(raw, "sub", "ok.md"), []byte("x"), 0o644))
outside := filepath.Join(t.TempDir(), "secret.txt")
require.NoError(t, os.WriteFile(outside, []byte("s"), 0o644))
require.NoError(t, os.Symlink(outside, filepath.Join(raw, "link.md")))
require.NoError(t, os.Symlink(filepath.Dir(outside), filepath.Join(raw, "dirlink")))
for _, ok := range []string{filepath.Join(raw, "sub", "ok.md"), "sub/ok.md", raw, "sub"} {
_, err := api.ResolveIngestPath(brain, ok)
assert.NoError(t, err, ok)
}
for _, bad := range []string{
"/etc/hostname", outside, "../secret.txt", filepath.Join(raw, "..", "wiki"),
filepath.Join(raw, "link.md"), filepath.Join(raw, "dirlink", "secret.txt"),
raw + "-evil/x.md", "",
} {
_, err := api.ResolveIngestPath(brain, bad)
assert.Error(t, err, "must refuse %q", bad)
}
}
func TestIngestPathRefusesAPathOutsideTheInbox(t *testing.T) {
_, h := setup(t)
body, _ := json.Marshal(map[string]any{"path": "/etc/hostname"})
rec := httptest.NewRecorder()
h.IngestPath(rec, httptest.NewRequest(http.MethodPost, "/ingest-path", bytes.NewReader(body)))
assert.Equal(t, http.StatusBadRequest, rec.Code)
assert.Contains(t, rec.Body.String(), "raw")
}
// A directory ingest must not follow a symlinked FILE inside raw/ out of it.
func TestIngestPathDirectoryWalkSkipsSymlinksLeavingTheInbox(t *testing.T) {
dir, h := setup(t)
raw := filepath.Join(dir, "raw", "batch")
require.NoError(t, os.MkdirAll(raw, 0o755))
outside := filepath.Join(t.TempDir(), "secret.md")
require.NoError(t, os.WriteFile(outside, []byte("# secret\n\nSENTINEL-149-OUTSIDE"), 0o644))
require.NoError(t, os.Symlink(outside, filepath.Join(raw, "leak.md")))
body, _ := json.Marshal(map[string]any{"path": "batch", "source": "walk"})
rec := httptest.NewRecorder()
h.IngestPath(rec, httptest.NewRequest(http.MethodPost, "/ingest-path", bytes.NewReader(body)))
assert.Contains(t, rec.Body.String(), "skipped")
assert.NotContains(t, rec.Body.String(), "SENTINEL-149-OUTSIDE")
}
// PR #152 review: the raw EvalSymlinks error told an authenticated caller
// whether any server file existed ("no such file") or was unreadable
// ("permission denied"), and leaked the brain directory. Every refusal now
// reads the same.
func TestResolveIngestPathRefusalsAreIndistinguishable(t *testing.T) {
brain := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(brain, "raw"), 0o755))
var msgs []string
for _, p := range []string{"/etc/hostname", "/etc/definitely-not-here-149", "../x", "nope/missing.md", ""} {
_, err := api.ResolveIngestPath(brain, p)
require.Error(t, err, p)
msgs = append(msgs, err.Error())
assert.NotContains(t, err.Error(), brain, "refusal for %q leaks the brain dir", p)
}
for _, m := range msgs[1:] {
assert.Equal(t, msgs[0], m)
}
}
+6
View File
@@ -543,6 +543,12 @@ func (s *Server) brainIngest(ctx context.Context, args json.RawMessage) (json.Ra
} }
if a.Path != "" { if a.Path != "" {
// Confined to brain/raw/ with symlinks evaluated (hyperguild#149).
resolved, err := api.ResolveIngestPath(s.brainDir, a.Path)
if err != nil {
return nil, err
}
a.Path = resolved
text, err := extract.Text(a.Path) text, err := extract.Text(a.Path)
if err != nil { if err != nil {
return nil, fmt.Errorf("extract: %w", err) return nil, fmt.Errorf("extract: %w", err)
@@ -0,0 +1,24 @@
package mcp_test
import (
"context"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/mcp"
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
)
// hyperguild#149: brain_ingest's `path` read any file the pod could read and
// ingested it into the brain, where brain_query reads it back out.
func TestBrainIngestRefusesAPathOutsideTheRawInbox(t *testing.T) {
stub := pipeline.Config{Complete: func(context.Context, string, string) (string, error) {
return `[{"title":"Leak","type":"concept","content":"x"}]`, nil
}}
srv := mcp.NewServer(newBrainRepo(t), &stub, nil, nil)
resp := authCall(t, srv, "", "brain_ingest", map[string]any{"path": "/etc/hostname"})
require.NotNil(t, resp["error"], "a path outside brain/raw/ must be refused")
assert.Contains(t, resp["error"].(map[string]any)["message"], "brain/raw/")
}
@@ -0,0 +1,153 @@
// Package symlinkguard is the hyperguild#157 tripwire. The brain directory is
// a Syncthing sendreceive folder and Syncthing syncs symlinks; several brain
// readers (search, vectorstore, context, graph) follow a symlink out of the
// brain and would index or return whatever it points to. Until every reader
// reads through an os.Root, a symlink inside the brain, or a directory the
// scan cannot read (which could hide one), is an incident: the server refuses
// to start on one (main), and Watch reports one that appears while it runs.
package symlinkguard
import (
"context"
"fmt"
"io/fs"
"log/slog"
"os"
"path/filepath"
"sort"
"strconv"
"strings"
"time"
)
// DefaultInterval is used when Watch is given a non-positive interval.
const DefaultInterval = 5 * time.Minute
// Notifier is the out-of-band alert sink (audit.Notifier satisfies it).
type Notifier interface {
Notify(ctx context.Context, msg string) error
}
// Result is what a scan found: symlinks, and directories it could not read
// (each of which could hide a symlink). Paths are brain-relative.
type Result struct {
Symlinks []string
Unreadable []string
}
// Clean reports whether the scan found nothing to act on.
func (r Result) Clean() bool { return len(r.Symlinks) == 0 && len(r.Unreadable) == 0 }
// String describes the findings for a log line or alert. Paths are quoted, so
// a name containing a newline cannot blur two findings together.
func (r Result) String() string {
q := func(xs []string) string {
out := make([]string, len(xs))
for i, x := range xs {
out[i] = strconv.Quote(x)
}
return strings.Join(out, ", ")
}
var parts []string
if len(r.Symlinks) > 0 {
parts = append(parts, fmt.Sprintf("%d symlink(s): %s", len(r.Symlinks), q(r.Symlinks)))
}
if len(r.Unreadable) > 0 {
parts = append(parts, fmt.Sprintf("%d unreadable dir(s): %s", len(r.Unreadable), q(r.Unreadable)))
}
return strings.Join(parts, "; ")
}
// Scan walks the brain dir and reports every symlink in it and every
// directory it could not read, continuing past both. Only the ROOT .git is
// skipped (it is brain-sync's, and no reader walks it); a nested .git is
// scanned like any other directory. brainDir itself may be a symlink (a
// volume mount); it is resolved first. The error is non-nil when the root
// cannot be resolved (wrapping os.ErrNotExist when it is missing) or cannot be
// listed.
func Scan(brainDir string) (Result, error) {
root, err := filepath.EvalSymlinks(brainDir)
if err != nil {
return Result{}, fmt.Errorf("resolve brain dir: %w", err)
}
var r Result
rootGit := filepath.Join(root, ".git")
walkErr := filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error {
rel, _ := filepath.Rel(root, p)
rel = filepath.ToSlash(rel)
if err != nil {
if p == root {
return err
}
r.Unreadable = append(r.Unreadable, rel)
if d != nil && d.IsDir() {
return filepath.SkipDir
}
return nil
}
if d.IsDir() && p == rootGit {
return filepath.SkipDir
}
if d.Type()&os.ModeSymlink != 0 {
r.Symlinks = append(r.Symlinks, rel)
}
return nil
})
sort.Strings(r.Symlinks)
sort.Strings(r.Unreadable)
if walkErr != nil {
// Only the root's own error reaches here (deeper errors are recorded
// as Unreadable). A root that cannot be listed cannot be shown clean
// (PR #159 round 2: discarding this reported clean while search
// served a planted symlink).
return r, fmt.Errorf("scan brain dir: %w", walkErr)
}
return r, nil
}
// Watch rescans every interval (DefaultInterval if non-positive) until ctx
// ends, and notifies once per distinct non-clean result. A result counts as
// reported only after Notify succeeds, so an alert lost to a notifier outage
// is retried on the next tick.
func Watch(ctx context.Context, brainDir string, interval time.Duration, n Notifier) {
if interval <= 0 {
interval = DefaultInterval
}
t := time.NewTicker(interval)
defer t.Stop()
reported := ""
for {
select {
case <-ctx.Done():
return
case <-t.C:
}
r, err := Scan(brainDir)
var key, msg string
switch {
case err != nil:
key = "error: " + err.Error()
msg = "brain symlink tripwire (hyperguild#157): cannot scan the brain dir: " + err.Error()
case r.Clean():
reported = ""
continue
default:
key = r.String()
msg = "brain symlink tripwire (hyperguild#157): " + key +
". Readers may follow symlinks out of the brain; remove them."
}
if key == reported {
continue
}
slog.Error("brain symlink tripwire", "finding", key)
if n == nil {
reported = key
continue
}
if err := n.Notify(ctx, msg); err != nil {
slog.Warn("symlink tripwire notify failed; will retry", "err", err)
continue
}
reported = key
}
}
@@ -0,0 +1,189 @@
package symlinkguard
import (
"context"
"errors"
"os"
"path/filepath"
"sync"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// hyperguild#157: the brain dir is a Syncthing sendreceive folder and
// Syncthing syncs symlinks; several readers follow them out of the brain.
// Until every reader goes through an os.Root, a symlink anywhere in the brain,
// or a directory the scan cannot read (which could hide one), is an incident.
func TestScanReportsSymlinksAtEveryDepthIncludingNestedGit(t *testing.T) {
brain := t.TempDir()
outside := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki", "w", "facts", ".git"), 0o755))
require.NoError(t, os.MkdirAll(filepath.Join(brain, ".git", "objects"), 0o755))
require.NoError(t, os.Symlink(filepath.Join(outside, "f"), filepath.Join(brain, "wiki", "w", "facts", "leak.md")))
require.NoError(t, os.Symlink(outside, filepath.Join(brain, "wiki", "dirlink")))
// PR #159 review: only the ROOT .git is skipped; a nested .git is not
// special to the brain readers, which walk into it.
require.NoError(t, os.Symlink(filepath.Join(outside, "g"), filepath.Join(brain, "wiki", "w", "facts", ".git", "leak.md")))
require.NoError(t, os.Symlink(outside, filepath.Join(brain, ".git", "objects", "alt")))
r, err := Scan(brain)
require.NoError(t, err)
assert.ElementsMatch(t, []string{"wiki/w/facts/leak.md", "wiki/dirlink", "wiki/w/facts/.git/leak.md"}, r.Symlinks)
assert.Empty(t, r.Unreadable)
}
// PR #159 review: WalkDir stopped at the first unreadable directory and the
// startup check passed, while search (which keeps walking) served a planted
// symlink. Scan now records the unreadable dir and keeps going.
func TestScanReportsUnreadableDirsAndKeepsWalkingPastThem(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root reads mode-000 directories")
}
brain := t.TempDir()
locked := filepath.Join(brain, "aaa")
require.NoError(t, os.MkdirAll(locked, 0o755))
require.NoError(t, os.Chmod(locked, 0o000))
t.Cleanup(func() { _ = os.Chmod(locked, 0o755) })
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, "zzz-planted")))
r, err := Scan(brain)
require.NoError(t, err)
assert.Equal(t, []string{"aaa"}, r.Unreadable)
assert.Equal(t, []string{"zzz-planted"}, r.Symlinks, "the walk continues past the unreadable dir")
}
func TestScanResolvesASymlinkedRootAndReportsSymlinkedEntriesInsideIt(t *testing.T) {
real := t.TempDir()
link := filepath.Join(t.TempDir(), "brain")
require.NoError(t, os.Symlink(real, link))
require.NoError(t, os.WriteFile(filepath.Join(real, "n.md"), []byte("x"), 0o644))
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(real, "inner")))
r, err := Scan(link)
require.NoError(t, err)
assert.Equal(t, []string{"inner"}, r.Symlinks)
}
func TestScanOfAMissingRootIsErrNotExist(t *testing.T) {
_, err := Scan(filepath.Join(t.TempDir(), "nope"))
assert.True(t, errors.Is(err, os.ErrNotExist), "got %v", err)
}
type recNotifier struct {
mu sync.Mutex
msgs []string
fail int // fail the first N calls
}
func (r *recNotifier) Notify(_ context.Context, m string) error {
r.mu.Lock()
defer r.mu.Unlock()
r.msgs = append(r.msgs, m)
if r.fail > 0 {
r.fail--
return errors.New("ntfy down")
}
return nil
}
func (r *recNotifier) count() int { r.mu.Lock(); defer r.mu.Unlock(); return len(r.msgs) }
func TestWatchNotifiesOncePerDistinctSetAndAgainForANewSet(t *testing.T) {
brain := t.TempDir()
n := &recNotifier{}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go Watch(ctx, brain, 10*time.Millisecond, n)
time.Sleep(50 * time.Millisecond)
assert.Zero(t, n.count(), "a clean brain raises nothing")
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, "first")))
require.Eventually(t, func() bool { return n.count() == 1 }, 2*time.Second, 10*time.Millisecond)
time.Sleep(60 * time.Millisecond)
assert.Equal(t, 1, n.count(), "the same set is reported once, not every tick")
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, "second")))
require.Eventually(t, func() bool { return n.count() == 2 }, 2*time.Second, 10*time.Millisecond)
}
// PR #159 review: last was recorded before Notify, so an alert lost to an
// ntfy outage was never retried while the symlink stayed.
func TestWatchRetriesAnAlertThatFailedToSend(t *testing.T) {
brain := t.TempDir()
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, "planted")))
n := &recNotifier{fail: 2}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go Watch(ctx, brain, 10*time.Millisecond, n)
require.Eventually(t, func() bool { return n.count() == 3 }, 2*time.Second, 10*time.Millisecond)
time.Sleep(60 * time.Millisecond)
assert.Equal(t, 3, n.count(), "after one successful send, no more")
}
func TestWatchAlertsOnAnUnreadableDir(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root reads mode-000 directories")
}
brain := t.TempDir()
locked := filepath.Join(brain, "locked")
require.NoError(t, os.MkdirAll(locked, 0o755))
require.NoError(t, os.Chmod(locked, 0o000))
t.Cleanup(func() { _ = os.Chmod(locked, 0o755) })
n := &recNotifier{}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go Watch(ctx, brain, 10*time.Millisecond, n)
require.Eventually(t, func() bool { return n.count() == 1 }, 2*time.Second, 10*time.Millisecond)
assert.Contains(t, n.msgs[0], "locked")
}
// PR #159 review: a non-positive interval panicked time.NewTicker inside the
// goroutine and crashlooped the server.
func TestWatchToleratesANonPositiveInterval(t *testing.T) {
brain := t.TempDir()
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() { Watch(ctx, brain, 0, nil); close(done) }()
cancel()
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("Watch did not return on cancel")
}
}
// PR #159 round 2: a brain root that can be entered but not listed (mode
// 0100) made WalkDir fail at the root; the error was discarded and Scan
// reported clean, while search (which goes straight into wiki/) still served
// a planted symlink. A root that cannot be listed is now an error.
func TestScanOfAnUnlistableRootIsAnError(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root lists mode-0100 directories")
}
brain := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki"), 0o755))
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, "wiki", "leak.md")))
require.NoError(t, os.Chmod(brain, 0o100))
t.Cleanup(func() { _ = os.Chmod(brain, 0o755) })
_, err := Scan(brain)
require.Error(t, err)
assert.False(t, errors.Is(err, os.ErrNotExist), "an unlistable root is not a missing root")
}
// The same set that disappears and comes back is reported again.
func TestWatchReportsASetAgainAfterItWasCleared(t *testing.T) {
brain := t.TempDir()
link := filepath.Join(brain, "planted")
n := &recNotifier{}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go Watch(ctx, brain, 10*time.Millisecond, n)
require.NoError(t, os.Symlink(t.TempDir(), link))
require.Eventually(t, func() bool { return n.count() == 1 }, 2*time.Second, 10*time.Millisecond)
require.NoError(t, os.Remove(link))
time.Sleep(60 * time.Millisecond)
require.NoError(t, os.Symlink(t.TempDir(), link))
require.Eventually(t, func() bool { return n.count() == 2 }, 2*time.Second, 10*time.Millisecond)
}
+5
View File
@@ -13,6 +13,7 @@ import (
"io" "io"
"log/slog" "log/slog"
"net/http" "net/http"
"os"
"strings" "strings"
) )
@@ -42,6 +43,10 @@ func Query(ctx context.Context, baseURL, query string, limit int) (string, error
return "", nil return "", nil
} }
req.Header.Set("Content-Type", "application/json") req.Header.Set("Content-Type", "application/json")
// The REST API requires a bearer token since hyperguild#149.
if tok := os.Getenv("BRAIN_MCP_TOKEN"); tok != "" {
req.Header.Set("Authorization", "Bearer "+tok)
}
resp, err := http.DefaultClient.Do(req) resp, err := http.DefaultClient.Do(req)
if err != nil { if err != nil {
+15
View File
@@ -65,3 +65,18 @@ func TestQueryUnavailableServerReturnsEmpty(t *testing.T) {
require.NoError(t, err) require.NoError(t, err)
assert.Empty(t, result) assert.Empty(t, result)
} }
// hyperguild#149: the REST API requires a bearer token.
func TestQuerySendsBrainMCPToken(t *testing.T) {
var auth string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
auth = r.Header.Get("Authorization")
_, _ = w.Write([]byte(`{"results":[]}`))
}))
defer srv.Close()
t.Setenv("BRAIN_MCP_TOKEN", "lib-test-token")
_, _ = brain.Query(context.Background(), srv.URL, "x", 1)
if auth != "Bearer lib-test-token" {
t.Fatalf("Authorization = %q, want the bearer token", auth)
}
}