Compare commits

...
Author SHA1 Message Date
mathiasandClaude Opus 5.5 8fdf7284aa fix(linkcheck): skip a linked tree that is not a real directory (#157)
CI / Lint / Test / Vet (pull_request) Successful in 56s
CI / Mirror to GitHub (pull_request) Skipped
Review round 1 on PR #175:

- entryIndex returned WalkRegular's walk-root ErrNotRegular as an error,
  and StripUnresolvedLinks runs before every brain_write: a symlinked,
  dangling or regular-file knowledge/ (or wiki/) made every write fail
  and lose the learning. main's filepath.WalkDir skipped such a root. It
  is now skipped with a warning; other walk errors still abort, as on
  main. New table test over the three shapes.
- The symlinked-hall assertion passed with no index written at all. It
  now indexes a real hall next to the symlinked one and asserts no
  error, the index exists and lists the real note; a new test does the
  same for BuildAllWingIndexes with a symlinked wing next to a real one.
- The test comment no longer says this PR fixed the FIFO hang on every
  brain_write: that path is tunnel.DetectTunnels (slice 5b).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 07:17:43 +02:00
mathiasandClaude Opus 5.5 6e3c51c952 fix(brain): wing indexes and the link check read through brainfs (#157, slice 5a)
CI / Lint / Test / Vet (pull_request) Successful in 56s
CI / Mirror to GitHub (pull_request) Skipped
BuildWingIndex/BuildAllWingIndexes listed wiki/<wing>/<hall>/ with
os.ReadDir and opened each note by path: a symlinked note put the
target's title into _index.md (110k leaks in the swap race), a symlinked
wing was followed and its _index.md written into the target, and a FIFO
hung the build (it runs on every brain_write).

- New brainfs.ReadDirDirs: real subdirectories only, every component of
  the dir a real directory. Wings and halls are listed with it and
  ReadDirRegular; titles are read with ReadRegular
  (readTitleAndCreatedIn). A symlinked wing is an error; a symlinked hall
  is skipped.
- linkcheck entryIndex walks with WalkRegular: a symlinked .md name no
  longer makes a [[link]] resolve.
- readTitleAndCreated (path-based) stays for tunnel.DetectTunnels until
  PR #170 lands; that is slice 5b.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 07:09:03 +02:00
mathiasandClaude Opus 5.5 b6ef11600b fix(deps): bump x/net and x/text past called vulns
CI / Lint / Test / Vet (push) Successful in 1m45s
CI / Mirror to GitHub (push) Successful in 4s
govulncheck 2026-10-04 found called vulns in ingestion (brain-mcp, public),
reached via audit.LokiCentral.Push: golang.org/x/net v0.26.0
(GO-2026-4918 HTTP/2 infinite loop, GO-2026-5026) and golang.org/x/text
v0.29.0 (GO-2026-5970). x/net -> v0.55.0, x/text -> v0.39.0; tidy also
moved x/sync, x/sys, x/term. govulncheck ./... now: 0 called findings
(19 in required modules, none reachable).

Refs: govulncheck 2026-10-04

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 06:59:31 +02:00
mathiasandClaude Opus 5.5 432e09d122 test(ingestion): OIDC discovery 502s then 200 against a real issuer (#83)
The retry landed in 9a232ad but was tested only through startupretry with
fake funcs; nothing exercised chassisauth.NewJWTValidator's real 502 path.
Extracts the main() closure into buildJWTValidator (behaviour unchanged)
and tests it against an httptest issuer: 3x502 then 200 succeeds with 4
discoveries; a persistent 502 stops at exactly the 8-attempt budget with
the last error; cancelling ctx mid-backoff returns context.Canceled.
Flip-verified: Attempts=1 turns all three red.

Lane: risk/medium issue, committed to main because the behaviour change
already shipped in 9a232ad; this is a test plus a pure extraction.

Closes #83

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 06:57:45 +02:00
mathiasandClaude Opus 5.5 77e32de33c fix(graph,wiki): read the brain through brainfs (#157, slice 4) (#168)
CI / Lint / Test / Vet (push) Successful in 52s
CI / Mirror to GitHub (push) Successful in 4s
graphsync, wiki inventory/index read through brainfs; brainfs refuses an in-brain file swapped in behind a symlink (os.SameFile). Gatekeeper round 2 PASS on d90ce66, CI run 9180 green.

Refs #157

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-05 04:53:34 +00:00
mathiasandClaude Opus 5.5 57d0fb0fae fix(vectorstore): walk and read the brain through brainfs (#157, slice 3) (#166)
CI / Lint / Test / Vet (push) Successful in 48s
CI / Mirror to GitHub (push) Successful in 3s
vectorstore.Sync walks and reads via brainfs (regular files only, handle-checked), so symlinked notes are no longer embedded and their old chunks are purged. brainfs.WalkRegular requires every component of its root to be a real directory (fixes in-brain root aliasing in Sync and search Wing/Wing+Hall). Walk errors, including a symlinked scan root, abort Sync before the delete pass. brainfs gains Lstat. Directory-swap race recorded as accepted residual (#167).

Refs #157. Adversarial review: BLOCKED, BLOCKED, PASS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-04 06:35:17 +00:00
mathiasandClaude Opus 5.5 27a01dd2af fix(brain): context, brain_get and brain_read read through brainfs (#157, slice 2) (#165)
CI / Lint / Test / Vet (push) Successful in 46s
CI / Mirror to GitHub (push) Successful in 4s
readExcerpt, ReadNote and ReadScoped now read through brainfs (os.Root, regular files only, type checked on the opened handle), so a symlink swapped in after a check is not followed and a FIFO does not hang the reader. ReadRegularMax bounds the read itself with io.LimitReader(max+1). notes.Target carries the brain root.

Refs #157. Adversarial review round 2: PASS (round 1 BLOCKED: unbounded read, rule-9 test names).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-04 05:59:55 +00:00
26 changed files with 1420 additions and 160 deletions
+25
View File
@@ -0,0 +1,25 @@
package main
import (
"context"
"log/slog"
chassisauth "git.d-ma.be/mathias/mcp-chassis/auth"
"github.com/mathiasbq/hyperguild/ingestion/internal/startupretry"
)
// buildJWTValidator runs OIDC discovery under policy, so a 502 from an auth
// upstream that is not serving yet on a cold boot is retried instead of
// exiting (#83). It honours ctx between attempts.
func buildJWTValidator(
ctx context.Context, logger *slog.Logger, issuer, audience string, policy startupretry.Policy,
) (*chassisauth.JWTValidator, error) {
return startupretry.Do(ctx, policy, func(ctx context.Context) (*chassisauth.JWTValidator, error) {
v, err := chassisauth.NewJWTValidator(ctx, issuer, audience)
if err != nil {
logger.Warn("oidc discovery not ready, retrying", "err", err)
}
return v, err
})
}
@@ -0,0 +1,86 @@
package main
import (
"context"
"io"
"log/slog"
"net/http"
"net/http/httptest"
"sync/atomic"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/startupretry"
)
// fakeIssuer answers OIDC discovery with 502 for the first fail requests, as
// ingress does while dex is not yet serving on a cold boot (#83), then with a
// real discovery document and an empty JWKS.
func fakeIssuer(t *testing.T, fail int32) (*httptest.Server, *atomic.Int32) {
t.Helper()
var discoveries atomic.Int32
mux := http.NewServeMux()
var srv *httptest.Server
mux.HandleFunc("/.well-known/openid-configuration", func(w http.ResponseWriter, _ *http.Request) {
if discoveries.Add(1) <= fail {
w.WriteHeader(http.StatusBadGateway)
return
}
w.Header().Set("Content-Type", "application/json")
_, _ = io.WriteString(w, `{"issuer":"`+srv.URL+`","jwks_uri":"`+srv.URL+`/keys"}`)
})
mux.HandleFunc("/keys", func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = io.WriteString(w, `{"keys":[]}`)
})
srv = httptest.NewServer(mux)
t.Cleanup(srv.Close)
return srv, &discoveries
}
var fastPolicy = startupretry.Policy{Attempts: 8, Base: time.Millisecond, Max: 5 * time.Millisecond}
func TestBuildJWTValidatorRetriesDiscovery502sThenSucceeds(t *testing.T) {
srv, discoveries := fakeIssuer(t, 3)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
v, err := buildJWTValidator(ctx, slog.New(slog.DiscardHandler), srv.URL, "brain", fastPolicy)
require.NoError(t, err)
assert.NotNil(t, v)
assert.Equal(t, int32(4), discoveries.Load(), "three 502s, then the successful discovery")
}
func TestBuildJWTValidatorGivesUpAfterTheBudgetWithTheLastError(t *testing.T) {
srv, discoveries := fakeIssuer(t, 1000)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
_, err := buildJWTValidator(ctx, slog.New(slog.DiscardHandler), srv.URL, "brain", fastPolicy)
require.Error(t, err)
assert.Contains(t, err.Error(), "status 502")
assert.Equal(t, int32(8), discoveries.Load(), "bounded: exactly the attempt budget")
}
func TestBuildJWTValidatorStopsRetryingWhenTheContextIsCancelled(t *testing.T) {
srv, _ := fakeIssuer(t, 1000)
ctx, cancel := context.WithCancel(context.Background())
slow := startupretry.Policy{Attempts: 8, Base: time.Hour, Max: time.Hour}
done := make(chan error, 1)
go func() {
_, err := buildJWTValidator(ctx, slog.New(slog.DiscardHandler), srv.URL, "brain", slow)
done <- err
}()
time.Sleep(50 * time.Millisecond)
cancel()
select {
case err := <-done:
assert.ErrorIs(t, err, context.Canceled)
case <-time.After(3 * time.Second):
t.Fatal("did not return after the context was cancelled")
}
}
+3 -9
View File
@@ -381,15 +381,9 @@ func main() {
//
// The budget stays bounded: retrying forever would turn a misconfiguration
// into a process that looks alive and never serves.
jwtValidator, err := startupretry.Do(ctx,
startupretry.Policy{Attempts: 8, Base: time.Second, Max: 30 * time.Second},
func(ctx context.Context) (*chassisauth.JWTValidator, error) {
v, err := chassisauth.NewJWTValidator(ctx, os.Getenv("DEX_ISSUER_URL"), os.Getenv("MCP_AUDIENCE"))
if err != nil {
logger.Warn("oidc discovery not ready, retrying", "err", err)
}
return v, err
})
jwtValidator, err := buildJWTValidator(ctx, logger,
os.Getenv("DEX_ISSUER_URL"), os.Getenv("MCP_AUDIENCE"),
startupretry.Policy{Attempts: 8, Base: time.Second, Max: 30 * time.Second})
if err != nil {
logger.Error("build jwt validator", "err", err)
os.Exit(1)
+6 -6
View File
@@ -34,9 +34,9 @@ require (
github.com/rogpeppe/go-internal v1.15.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/x448/float16 v0.8.4 // indirect
golang.org/x/net v0.26.0 // indirect
golang.org/x/net v0.55.0 // indirect
golang.org/x/oauth2 v0.21.0 // indirect
golang.org/x/term v0.28.0 // indirect
golang.org/x/term v0.43.0 // indirect
golang.org/x/time v0.3.0 // indirect
google.golang.org/protobuf v1.34.2 // indirect
gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
@@ -66,9 +66,9 @@ require (
github.com/lestrrat-go/option v1.0.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/segmentio/asm v1.2.0 // indirect
golang.org/x/crypto v0.32.0 // indirect
golang.org/x/sync v0.17.0 // indirect
golang.org/x/sys v0.31.0 // indirect
golang.org/x/text v0.29.0 // indirect
golang.org/x/crypto v0.51.0 // indirect
golang.org/x/sync v0.21.0 // indirect
golang.org/x/sys v0.45.0 // indirect
golang.org/x/text v0.39.0 // indirect
gopkg.in/yaml.v3 v3.0.1
)
+14 -14
View File
@@ -117,42 +117,42 @@ github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9dec
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.32.0 h1:euUpcYgM8WcP71gNpTqQCn6rC2t6ULUPiOzfWaXVVfc=
golang.org/x/crypto v0.32.0/go.mod h1:ZnnJkOaASj8g0AjIduWNlq2NRxL0PlBrbKVyZ6V/Ugc=
golang.org/x/crypto v0.51.0 h1:IBPXwPfKxY7cWQZ38ZCIRPI50YLeevDLlLnyC5wRGTI=
golang.org/x/crypto v0.51.0/go.mod h1:8AdwkbraGNABw2kOX6YFPs3WM22XqI4EXEd8g+x7Oc8=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.26.0 h1:soB7SVo0PWrY4vPW/+ay0jKDNScG2X9wFeYlXIvJsOQ=
golang.org/x/net v0.26.0/go.mod h1:5YKkiSynbBIh3p6iOc/vibscux0x38BZDkn8sCUPxHE=
golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8=
golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww=
golang.org/x/oauth2 v0.21.0 h1:tsimM75w1tF/uws5rbeHzIWxEqElMehnc+iW793zsZs=
golang.org/x/oauth2 v0.21.0/go.mod h1:XYTD2NtWslqkgxebSiOHnXEap4TF09sJSc7H1sXbhtI=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.17.0 h1:l60nONMj9l5drqw6jlhIELNv9I0A4OFgRsG9k2oT9Ug=
golang.org/x/sync v0.17.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik=
golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/term v0.28.0 h1:/Ts8HFuMR2E6IP/jlo7QVLZHggjKQbhu/7H0LJFr3Gg=
golang.org/x/term v0.28.0/go.mod h1:Sw/lC2IAUZ92udQNf3WodGtn4k/XoLyZoh8v/8uiwek=
golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/term v0.43.0 h1:S4RLU2sB31O/NCl+zFN9Aru9A/Cq2aqKpTZJ6B+DwT4=
golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.29.0 h1:1neNs90w9YzJ9BocxfsQNHKuAT4pkghyXc4nhZ6sJvk=
golang.org/x/text v0.29.0/go.mod h1:7MhJOA9CD2qZyOKYazxdYMF85OwPdEr9jTtBpO7ydH4=
golang.org/x/text v0.39.0 h1:UbZz4pLOvn600D6Oh6GGEI6VAmndrEBLv8/6BEXzyus=
golang.org/x/text v0.39.0/go.mod h1:3UwRclnC2g0TU9x8PZiyfOajCd1zaUNHF9cvqcQZ+ZM=
golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4=
golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.36.0 h1:kWS0uv/zsvHEle1LbV5LE8QujrxB3wfQyxHfhOk0Qkg=
golang.org/x/tools v0.36.0/go.mod h1:WBDiHKJK8YgLHlcQPYQzNCkUxUypCaa5ZegCVutKm+s=
golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
@@ -0,0 +1,43 @@
package api_test
import (
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs/brainfstest"
)
// hyperguild#157 slice 2: ReadNote (brain_get, read-after-write) read by
// path; it now reads through brainfs.
func TestReadNoteRefusesSymlinksAndSurvivesSwapRaces(t *testing.T) {
brain := t.TempDir()
facts := filepath.Join(brain, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
require.NoError(t, os.Symlink(brainfstest.Outside(t), filepath.Join(facts, "leak.md")))
_, body, _, err := api.ReadNote(brain, "wiki/w/facts/leak.md")
assert.Error(t, err)
assert.NotContains(t, body, brainfstest.Sentinel)
p := filepath.Join(facts, "r.md")
require.NoError(t, os.WriteFile(p, []byte("---\nx: y\n---\ninside"), 0o644))
fm, body, _, err := api.ReadNote(brain, "wiki/w/facts/r.md")
require.NoError(t, err)
assert.Equal(t, "y", fm["x"])
assert.Contains(t, body, "inside")
read := func() string {
fm, body, _, _ := api.ReadNote(brain, "wiki/w/facts/r.md")
s := body
for k, v := range fm {
s += k + v
}
return s
}
brainfstest.AssertNoLeak(t, p, "---\nx: y\n---\ninside", 1500*time.Millisecond, read)
brainfstest.AssertNoHang(t, p, "---\nx: y\n---\ninside", 1500*time.Millisecond, 6*time.Second, func() { _ = read() })
}
+9 -2
View File
@@ -4,6 +4,7 @@ import (
"crypto/sha256"
"encoding/hex"
"fmt"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
"os"
"path/filepath"
"strings"
@@ -174,11 +175,17 @@ func UpdateNote(brainDir string, opts UpdateNoteOptions) (relPath, contentHash,
// write primitive behind brain_get: the hash it returns equals the hash
// brain_write / brain_update returned for the same bytes.
func ReadNote(brainDir, relPath string) (fm map[string]string, body, contentHash string, err error) {
_, abs, err := resolveWithin(brainDir, relPath)
rel, _, err := resolveWithin(brainDir, relPath)
if err != nil {
return nil, "", "", err
}
raw, err := os.ReadFile(abs)
// Through the brain root, regular files only (hyperguild#157).
bfs, err := brainfs.Open(brainDir)
if err != nil {
return nil, "", "", fmt.Errorf("read note: %w", err)
}
defer func() { _ = bfs.Close() }()
raw, err := bfs.ReadRegular(rel)
if err != nil {
if os.IsNotExist(err) {
return nil, "", "", fmt.Errorf("note %q does not exist", relPath)
+67 -37
View File
@@ -2,12 +2,18 @@ package brain
import (
"bufio"
"bytes"
"errors"
"fmt"
"io"
"io/fs"
"os"
"path/filepath"
"sort"
"strings"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
)
// noteEntry is one row in a Wing _index.md.
@@ -27,13 +33,18 @@ func BuildWingIndex(brainDir, wing string) error {
return fmt.Errorf("invalid wing %q", wing)
}
wingDir := filepath.Join(brainDir, "wiki", w)
if _, err := os.Stat(wingDir); os.IsNotExist(err) {
// Listed and read through the brain root (hyperguild#157): a wing or
// hall that is a symlink is not followed, and notes are regular files.
bfs, err := brainfs.Open(brainDir)
if err != nil {
return fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
if _, err := bfs.Lstat("wiki/" + w); errors.Is(err, fs.ErrNotExist) {
return nil
} else if err != nil {
return fmt.Errorf("stat wing: %w", err)
}
entries, err := collectWingEntries(wingDir)
entries, err := collectWingEntries(bfs, "wiki/"+w)
if err != nil {
return err
}
@@ -58,51 +69,47 @@ func BuildWingIndex(brainDir, wing string) error {
// BuildAllWingIndexes regenerates _index.md for every wing under brain/wiki/.
func BuildAllWingIndexes(brainDir string) error {
wikiDir := filepath.Join(brainDir, "wiki")
ents, err := os.ReadDir(wikiDir)
if os.IsNotExist(err) {
bfs, err := brainfs.Open(brainDir)
if err != nil {
return fmt.Errorf("open brain: %w", err)
}
wings, err := bfs.ReadDirDirs("wiki")
_ = bfs.Close()
if errors.Is(err, fs.ErrNotExist) {
return nil
}
if err != nil {
return fmt.Errorf("read wiki: %w", err)
}
for _, e := range ents {
if !e.IsDir() {
continue
}
if err := BuildWingIndex(brainDir, e.Name()); err != nil {
return fmt.Errorf("index %s: %w", e.Name(), err)
for _, w := range wings {
if err := BuildWingIndex(brainDir, w); err != nil {
return fmt.Errorf("index %s: %w", w, err)
}
}
return nil
}
func collectWingEntries(wingDir string) ([]noteEntry, error) {
func collectWingEntries(bfs *brainfs.Brain, wingRel string) ([]noteEntry, error) {
var out []noteEntry
ents, err := os.ReadDir(wingDir)
halls, err := bfs.ReadDirDirs(wingRel)
if err != nil {
return nil, fmt.Errorf("read wing: %w", err)
}
for _, hallEnt := range ents {
if !hallEnt.IsDir() {
continue
}
hall := hallEnt.Name()
for _, hall := range halls {
if !IsValidHall(hall) {
continue
}
hallDir := filepath.Join(wingDir, hall)
notes, err := os.ReadDir(hallDir)
hallRel := wingRel + "/" + hall
names, err := bfs.ReadDirRegular(hallRel)
if err != nil {
return nil, fmt.Errorf("read hall %s: %w", hall, err)
}
for _, n := range notes {
if n.IsDir() || !strings.HasSuffix(n.Name(), ".md") || n.Name() == "_index.md" {
for _, name := range names {
if !strings.HasSuffix(name, ".md") || name == "_index.md" {
continue
}
slug := strings.TrimSuffix(n.Name(), ".md")
full := filepath.Join(hallDir, n.Name())
title, created := readTitleAndCreated(full, slug)
slug := strings.TrimSuffix(name, ".md")
title, created := readTitleAndCreatedIn(bfs, hallRel+"/"+name, slug)
out = append(out, noteEntry{Hall: hall, Slug: slug, Title: title, Created: created})
}
}
@@ -117,9 +124,40 @@ func readTitleAndCreated(path, slug string) (string, string) {
return slug, ""
}
defer func() { _ = f.Close() }()
title, created := parseTitleAndCreated(f)
if created == "" {
if info, err := os.Stat(path); err == nil {
created = info.ModTime().UTC().Format("2006-01-02")
}
}
return titleOrSlug(title, slug), created
}
title, created := "", ""
scanner := bufio.NewScanner(f)
// readTitleAndCreatedIn is readTitleAndCreated through the brain root,
// regular files only (hyperguild#157).
func readTitleAndCreatedIn(bfs *brainfs.Brain, rel, slug string) (string, string) {
content, err := bfs.ReadRegular(rel)
if err != nil {
return slug, ""
}
title, created := parseTitleAndCreated(bytes.NewReader(content))
if created == "" {
if info, err := bfs.Lstat(rel); err == nil {
created = info.ModTime().UTC().Format("2006-01-02")
}
}
return titleOrSlug(title, slug), created
}
func titleOrSlug(title, slug string) string {
if title == "" {
return strings.ReplaceAll(slug, "-", " ")
}
return title
}
func parseTitleAndCreated(r io.Reader) (title, created string) {
scanner := bufio.NewScanner(r)
inFrontmatter := false
for scanner.Scan() {
line := scanner.Text()
@@ -149,13 +187,5 @@ func readTitleAndCreated(path, slug string) (string, string) {
}
}
}
if title == "" {
title = strings.ReplaceAll(slug, "-", " ")
}
if created == "" {
if info, err := os.Stat(path); err == nil {
created = info.ModTime().UTC().Format("2006-01-02")
}
}
return title, created
}
+132
View File
@@ -0,0 +1,132 @@
package brain_test
import (
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs/brainfstest"
)
func readIndex(t *testing.T, brainDir, wing string) string {
t.Helper()
b, _ := os.ReadFile(filepath.Join(brainDir, "wiki", wing, "_index.md"))
return string(b)
}
// hyperguild#157 slice 5: BuildWingIndex listed wiki/<wing>/<hall>/ with
// os.ReadDir and opened each note by path, so a symlinked note put the
// target's title into _index.md, and a FIFO hung the index build.
func TestBuildWingIndexRefusesSymlinkedNotesAndSurvivesSwapRaces(t *testing.T) {
b := t.TempDir()
facts := filepath.Join(b, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(facts, "ok.md"), []byte("---\ntitle: zqinside\n---\n"), 0o644))
require.NoError(t, os.Symlink(brainfstest.Outside(t), filepath.Join(facts, "leak.md")))
inBrain := brainfstest.WriteSentinelAt(t, filepath.Join(b, "raw", "s.md"))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "..", "raw", "s.md"), filepath.Join(facts, "inlink.md")))
require.NoError(t, brain.BuildWingIndex(b, "w"))
idx := readIndex(t, b, "w")
assert.Contains(t, idx, "zqinside", "positive control")
assert.NotContains(t, idx, brainfstest.Sentinel)
assert.NotContains(t, idx, "leak")
assert.NotContains(t, idx, "inlink")
p := filepath.Join(facts, "r.md")
read := func() string {
_ = brain.BuildWingIndex(b, "w")
return readIndex(t, b, "w")
}
brainfstest.AssertNoLeak(t, p, "---\ntitle: zqinside\n---\n", 1500*time.Millisecond, read)
brainfstest.AssertNoLeakFrom(t, p, "---\ntitle: zqinside\n---\n", inBrain, 1500*time.Millisecond, read)
brainfstest.AssertNoHang(t, p, "---\ntitle: zqinside\n---\n", 1500*time.Millisecond, 6*time.Second, func() { _ = read() })
}
// A symlinked hall is skipped while a real hall next to it is indexed; a
// symlinked wing is refused and nothing is written into its target.
func TestBuildWingIndexSkipsASymlinkedHallAndRefusesASymlinkedWing(t *testing.T) {
b := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(b, "raw", "facts"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(b, "raw", "facts", "r.md"), []byte("---\ntitle: zqrawonly\n---\n"), 0o644))
require.NoError(t, os.MkdirAll(filepath.Join(b, "wiki", "w", "decisions"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(b, "wiki", "w", "decisions", "d.md"), []byte("---\ntitle: zqrealhall\n---\n"), 0o644))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "raw", "facts"), filepath.Join(b, "wiki", "w", "facts")))
require.NoError(t, os.Symlink(filepath.Join("..", "raw"), filepath.Join(b, "wiki", "alias")))
require.NoError(t, brain.BuildWingIndex(b, "w"))
idx, err := os.ReadFile(filepath.Join(b, "wiki", "w", "_index.md"))
require.NoError(t, err, "the wing index is written")
assert.Contains(t, string(idx), "zqrealhall", "the real hall is indexed")
assert.NotContains(t, string(idx), "zqrawonly", "the symlinked hall is not")
assert.Error(t, brain.BuildWingIndex(b, "alias"))
_, err = os.Stat(filepath.Join(b, "raw", "_index.md"))
assert.True(t, os.IsNotExist(err), "an index was written into the symlinked wing's target")
}
// BuildAllWingIndexes indexes the real wings and skips a symlinked one.
func TestBuildAllWingIndexesSkipsASymlinkedWing(t *testing.T) {
b := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(b, "wiki", "w", "facts"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(b, "wiki", "w", "facts", "n.md"), []byte("---\ntitle: zqrealwing\n---\n"), 0o644))
require.NoError(t, os.MkdirAll(filepath.Join(b, "raw", "facts"), 0o755))
require.NoError(t, os.Symlink(filepath.Join("..", "raw"), filepath.Join(b, "wiki", "alias")))
require.NoError(t, brain.BuildAllWingIndexes(b))
idx, err := os.ReadFile(filepath.Join(b, "wiki", "w", "_index.md"))
require.NoError(t, err)
assert.Contains(t, string(idx), "zqrealwing")
_, err = os.Stat(filepath.Join(b, "raw", "_index.md"))
assert.True(t, os.IsNotExist(err))
}
// A link to a symlinked note does not resolve: only regular files count as
// entries, so StripUnresolvedLinks unwraps it.
func TestStripUnresolvedLinksDoesNotCountASymlinkedNoteAsAnEntry(t *testing.T) {
b := t.TempDir()
facts := filepath.Join(b, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(facts, "real.md"), []byte("x"), 0o644))
brainfstest.WriteSentinelAt(t, filepath.Join(b, "raw", "s.md"))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "..", "raw", "s.md"), filepath.Join(facts, "ghost.md")))
out, unlinked, err := brain.StripUnresolvedLinks(b, "see [[real]] and [[ghost]]")
require.NoError(t, err)
assert.Contains(t, out, "[[real]]", "positive control")
assert.NotContains(t, out, "[[ghost]]")
assert.Contains(t, unlinked, "ghost")
}
// A linked tree (knowledge/ or wiki/) that is not a real directory is
// skipped, not an error: StripUnresolvedLinks runs before every brain_write,
// and an error there loses the learning.
func TestStripUnresolvedLinksSkipsALinkedTreeThatIsNotARealDirectory(t *testing.T) {
for name, mk := range map[string]func(t *testing.T, b string){
"symlink to an in-brain dir": func(t *testing.T, b string) {
require.NoError(t, os.MkdirAll(filepath.Join(b, "raw", "k"), 0o755))
require.NoError(t, os.Symlink(filepath.Join("raw", "k"), filepath.Join(b, "knowledge")))
},
"dangling symlink": func(t *testing.T, b string) {
require.NoError(t, os.Symlink("nowhere", filepath.Join(b, "knowledge")))
},
"regular file": func(t *testing.T, b string) {
require.NoError(t, os.WriteFile(filepath.Join(b, "knowledge"), []byte("x"), 0o644))
},
} {
t.Run(name, func(t *testing.T) {
b := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(b, "wiki", "w", "facts"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(b, "wiki", "w", "facts", "real.md"), []byte("x"), 0o644))
mk(t, b)
out, _, err := brain.StripUnresolvedLinks(b, "learning [[real]] and [[gone]]")
require.NoError(t, err)
assert.Contains(t, out, "[[real]]", "links into wiki/ still resolve")
assert.NotContains(t, out, "[[gone]]")
})
}
}
+32 -17
View File
@@ -1,12 +1,16 @@
package brain
import (
"errors"
"fmt"
"os"
"path/filepath"
"io/fs"
"log/slog"
"path"
"regexp"
"sort"
"strings"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
)
// wikilinkRE matches [[target]] and [[target|Display Text]].
@@ -98,29 +102,40 @@ var linkedTrees = []string{"knowledge", "wiki"}
// one the audit will also resolve.
func entryIndex(brainDir string) (map[string]struct{}, error) {
index := map[string]struct{}{}
// Walked through the brain root (hyperguild#157): only regular files
// count as entries, so a symlinked .md name cannot make a link resolve.
bfs, err := brainfs.Open(brainDir)
if err != nil {
return nil, fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
for _, dir := range linkedTrees {
root := filepath.Join(brainDir, dir)
if _, err := os.Stat(root); os.IsNotExist(err) {
if _, err := bfs.Lstat(dir); errors.Is(err, fs.ErrNotExist) {
continue
} else if err != nil {
return nil, fmt.Errorf("stat %s: %w", dir, err)
}
err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() || !strings.HasSuffix(path, ".md") {
var walkErr error
err := bfs.WalkRegular(dir, func(rel string) error {
if !strings.HasSuffix(rel, ".md") {
return nil
}
rel, err := filepath.Rel(brainDir, path)
if err != nil {
return err
}
rel = filepath.ToSlash(rel)
index[strings.TrimSuffix(filepath.Base(rel), ".md")] = struct{}{}
index[strings.TrimSuffix(path.Base(rel), ".md")] = struct{}{}
index[strings.TrimPrefix(strings.TrimSuffix(rel, ".md"), "wiki/")] = struct{}{}
return nil
}, func(rel string, err error) {
// A linked tree that is not a real directory (a symlink, a
// file) is skipped, as filepath.WalkDir did: this runs before
// every brain_write, and an error here loses the learning.
if rel == dir && errors.Is(err, brainfs.ErrNotRegular) {
slog.Warn("link check: skipping a linked tree that is not a real directory", "dir", dir)
return
}
if walkErr == nil {
walkErr = fmt.Errorf("%s: %w", rel, err)
}
})
if err == nil {
err = walkErr
}
if err != nil {
return nil, fmt.Errorf("walk %s: %w", dir, err)
}
+145 -7
View File
@@ -13,13 +13,18 @@ import (
"io"
"io/fs"
"os"
"path"
"path/filepath"
"strings"
"syscall"
)
// ErrNotRegular is returned for an entry that is not a regular file.
var ErrNotRegular = errors.New("not a regular file")
// ErrTooLarge is returned by ReadRegularMax for a file over the limit.
var ErrTooLarge = errors.New("file too large")
// Brain is an open brain directory.
type Brain struct {
root *os.Root
@@ -50,6 +55,76 @@ func (b *Brain) Rel(path string) (string, error) {
return filepath.ToSlash(rel), nil
}
// Lstat stats a brain-relative path through the root without following a
// final symlink.
func (b *Brain) Lstat(rel string) (fs.FileInfo, error) {
return b.root.Lstat(filepath.ToSlash(rel))
}
// realDir returns ErrNotRegular unless every component of rel is a real
// directory. A swap after the check is not caught (accepted residual,
// hyperguild#157).
func (b *Brain) realDir(rel string) error {
if rel == "." {
return nil
}
prefix := ""
for _, part := range strings.Split(rel, "/") {
prefix = path.Join(prefix, part)
fi, err := b.root.Lstat(prefix)
if err != nil {
return err
}
if !fi.IsDir() {
return fmt.Errorf("%s: walk root component %s is not a directory: %w", rel, prefix, ErrNotRegular)
}
}
return nil
}
// ReadDirRegular lists the regular files directly in a brain-relative
// directory, by name. Every component of relDir must be a real directory
// (ErrNotRegular otherwise); symlinks, FIFOs, devices and subdirectories in
// it are left out.
func (b *Brain) ReadDirRegular(relDir string) ([]string, error) {
relDir = filepath.ToSlash(relDir)
if err := b.realDir(relDir); err != nil {
return nil, err
}
ents, err := fs.ReadDir(b.root.FS(), relDir)
if err != nil {
return nil, err
}
var names []string
for _, e := range ents {
if e.Type().IsRegular() {
names = append(names, e.Name())
}
}
return names, nil
}
// ReadDirDirs lists the real subdirectories directly in a brain-relative
// directory, by name. Every component of relDir must be a real directory
// (ErrNotRegular otherwise); symlinks to directories are left out.
func (b *Brain) ReadDirDirs(relDir string) ([]string, error) {
relDir = filepath.ToSlash(relDir)
if err := b.realDir(relDir); err != nil {
return nil, err
}
ents, err := fs.ReadDir(b.root.FS(), relDir)
if err != nil {
return nil, err
}
var names []string
for _, e := range ents {
if e.Type().IsDir() {
names = append(names, e.Name())
}
}
return names, nil
}
// ReadRegular reads a brain-relative path, refusing anything that is not a
// regular file, both before the open and on the opened handle.
func (b *Brain) ReadRegular(rel string) ([]byte, error) {
@@ -60,7 +135,7 @@ func (b *Brain) ReadRegular(rel string) ([]byte, error) {
if !fi.Mode().IsRegular() {
return nil, fmt.Errorf("%s: %w", rel, ErrNotRegular)
}
return b.readOpened(rel)
return b.readOpenedMax(rel, -1, fi)
}
// readOpened opens rel non-blocking and reads it only if the OPENED handle is
@@ -74,7 +149,26 @@ func (b *Brain) ReadRegular(rel string) ([]byte, error) {
// cannot create one; fs.protected_hardlinks=1 and the pod's secrets are on a
// different filesystem from the brain volume, so a local caller cannot link
// them in. Refusing Nlink > 1 is the next step if either changes.
func (b *Brain) readOpened(rel string) ([]byte, error) {
func (b *Brain) readOpened(rel string) ([]byte, error) { return b.readOpenedMax(rel, -1, nil) }
// ReadRegularMax is ReadRegular with a size limit, checked on the opened
// handle before reading (max < 0 means no limit).
func (b *Brain) ReadRegularMax(rel string, max int64) ([]byte, error) {
fi, err := b.root.Lstat(rel)
if err != nil {
return nil, err
}
if !fi.Mode().IsRegular() {
return nil, fmt.Errorf("%s: %w", rel, ErrNotRegular)
}
return b.readOpenedMax(rel, max, fi)
}
// readOpenedMax opens rel and reads it if the opened handle is a regular file
// and, when lfi is set, the same file the caller Lstat-ed: os.Root follows an
// in-root final symlink even with O_NOFOLLOW, so a symlink to another brain
// file swapped in between the Lstat and the open is refused only here.
func (b *Brain) readOpenedMax(rel string, max int64, lfi fs.FileInfo) ([]byte, error) {
f, err := b.root.OpenFile(rel, os.O_RDONLY|syscall.O_NONBLOCK, 0)
if err != nil {
return nil, err
@@ -87,15 +181,59 @@ func (b *Brain) readOpened(rel string) ([]byte, error) {
if !hfi.Mode().IsRegular() {
return nil, fmt.Errorf("%s: %w", rel, ErrNotRegular)
}
return io.ReadAll(f)
if lfi != nil && !os.SameFile(lfi, hfi) {
return nil, fmt.Errorf("%s: changed while opening: %w", rel, ErrNotRegular)
}
if max >= 0 && hfi.Size() > max {
return nil, fmt.Errorf("%s is %d bytes (limit %d): %w", rel, hfi.Size(), max, ErrTooLarge)
}
var r io.Reader = f
if wrapAfterStat != nil {
r = wrapAfterStat(r)
}
if max < 0 {
return io.ReadAll(r)
}
// The handle's size can grow after the check; bound the read itself.
data, err := io.ReadAll(io.LimitReader(r, max+1))
if err != nil {
return nil, err
}
if int64(len(data)) > max {
return nil, fmt.Errorf("%s grew past %d bytes while read: %w", rel, max, ErrTooLarge)
}
return data, nil
}
// wrapAfterStat is a test seam: it runs between the size check and the read,
// and wraps the reader the read consumes. A package global: brainfs tests
// must not use t.Parallel.
var wrapAfterStat func(io.Reader) io.Reader
// WalkRegular walks a brain-relative directory through the root and calls fn
// for each regular file (directories are descended; symlinks, FIFOs and
// devices are skipped, never followed). An entry error is passed to onErr and
// the walk continues; a nil onErr ignores it.
// for each regular file. Directories are descended; symlinks, FIFOs and
// devices found by the walk are skipped. A walk root with any component that
// is not a real directory goes to onErr as ErrNotRegular and nothing is
// yielded. An entry error is passed to onErr and the walk continues; a nil
// onErr ignores it.
//
// Accepted residual (hyperguild#157): a directory swapped for a symlink
// between the listing and the read is followed. os.Root still keeps the read
// inside the brain, but not inside relDir.
func (b *Brain) WalkRegular(relDir string, fn func(rel string) error, onErr func(rel string, err error)) error {
return fs.WalkDir(b.root.FS(), filepath.ToSlash(relDir), func(p string, d fs.DirEntry, err error) error {
relDir = filepath.ToSlash(relDir)
// fs.WalkDir stats its root with a follow, and os.Root allows a symlink
// that stays inside the brain: knowledge -> . would alias the whole
// brain. Require every component of the root to be a real directory
// (Lstat checks only the last one).
err := b.realDir(relDir)
if err != nil {
if onErr != nil {
onErr(relDir, err)
}
return nil
}
return fs.WalkDir(b.root.FS(), relDir, func(p string, d fs.DirEntry, err error) error {
if err != nil {
if onErr != nil {
onErr(p, err)
+190
View File
@@ -2,8 +2,12 @@ package brainfs
import (
"errors"
"io"
"io/fs"
"os"
"path/filepath"
"sync"
"sync/atomic"
"syscall"
"testing"
"time"
@@ -90,3 +94,189 @@ func TestWalkRegularYieldsOnlyRegularFiles(t *testing.T) {
require.NoError(t, b.WalkRegular(".", func(rel string) error { got = append(got, rel); return nil }, nil))
assert.Equal(t, []string{"a/n.md"}, got)
}
func TestReadRegularMaxRefusesAFileOverTheLimit(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "big.md"), make([]byte, 100), 0o644))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
_, err = b.ReadRegularMax("big.md", 99)
assert.True(t, errors.Is(err, ErrTooLarge), "got %v", err)
got, err := b.ReadRegularMax("big.md", 100)
require.NoError(t, err)
assert.Len(t, got, 100)
}
func TestReadRegularMaxRefusesAndStopsReadingAFileThatGrowsPastTheLimit(t *testing.T) {
dir := t.TempDir()
p := filepath.Join(dir, "grow.md")
require.NoError(t, os.WriteFile(p, make([]byte, 10), 0o644))
consumed := &countingReader{}
wrapAfterStat = func(r io.Reader) io.Reader {
f, err := os.OpenFile(p, os.O_WRONLY|os.O_APPEND, 0)
require.NoError(t, err)
_, err = f.Write(make([]byte, 1000))
require.NoError(t, err)
require.NoError(t, f.Close())
consumed.r = r
return consumed
}
t.Cleanup(func() { wrapAfterStat = nil })
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
got, err := b.ReadRegularMax("grow.md", 100)
assert.True(t, errors.Is(err, ErrTooLarge), "got %v", err)
assert.Nil(t, got)
assert.LessOrEqual(t, consumed.n, 101, "read past max+1 bytes")
}
type countingReader struct {
r io.Reader
n int
}
func (c *countingReader) Read(p []byte) (int, error) {
n, err := c.r.Read(p)
c.n += n
return n, err
}
func TestLstatDoesNotFollowSymlinksAndRefusesEscapes(t *testing.T) {
dir := t.TempDir()
outside := filepath.Join(t.TempDir(), "o.md")
require.NoError(t, os.WriteFile(outside, []byte("o"), 0o644))
require.NoError(t, os.Symlink(outside, filepath.Join(dir, "l.md")))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
fi, err := b.Lstat("l.md")
require.NoError(t, err)
assert.Equal(t, fs.ModeSymlink, fi.Mode().Type())
_, err = b.Lstat("../" + filepath.Base(filepath.Dir(outside)) + "/o.md")
assert.Error(t, err)
}
func TestWalkRegularReportsASymlinkedWalkRootAndYieldsNothing(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "real"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "real", "n.md"), []byte("n"), 0o644))
require.NoError(t, os.Symlink("real", filepath.Join(dir, "alias")))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
var got []string
var errs []error
require.NoError(t, b.WalkRegular("alias", func(rel string) error { got = append(got, rel); return nil },
func(_ string, err error) { errs = append(errs, err) }))
assert.Empty(t, got)
require.Len(t, errs, 1)
assert.ErrorIs(t, errs[0], ErrNotRegular)
}
func TestWalkRegularReportsASymlinkedIntermediateComponentOfTheWalkRoot(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "real", "sub"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "real", "sub", "n.md"), []byte("n"), 0o644))
require.NoError(t, os.Symlink("real", filepath.Join(dir, "alias")))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
var got []string
var errs []error
require.NoError(t, b.WalkRegular("alias/sub", func(rel string) error { got = append(got, rel); return nil },
func(_ string, err error) { errs = append(errs, err) }))
assert.Empty(t, got)
require.Len(t, errs, 1)
assert.ErrorIs(t, errs[0], ErrNotRegular)
}
func TestReadDirRegularListsOnlyRegularFilesAndRefusesSymlinkedComponents(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "a", "b", "sub"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "a", "b", "n.md"), []byte("n"), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(dir, "other.md"), []byte("o"), 0o644))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "other.md"), filepath.Join(dir, "a", "b", "l.md")))
require.NoError(t, syscall.Mkfifo(filepath.Join(dir, "a", "b", "f.md"), 0o644))
require.NoError(t, os.Symlink("a", filepath.Join(dir, "alias")))
require.NoError(t, os.MkdirAll(filepath.Join(dir, "x"), 0o755))
require.NoError(t, os.Symlink(filepath.Join("..", "a"), filepath.Join(dir, "x", "mid")))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
names, err := b.ReadDirRegular("a/b")
require.NoError(t, err)
assert.Equal(t, []string{"n.md"}, names)
for _, rel := range []string{"alias/b", "x/mid/b"} {
_, err := b.ReadDirRegular(rel)
assert.ErrorIs(t, err, ErrNotRegular, rel)
}
}
// A final component swapped for a symlink to another regular file in the
// brain between the Lstat and the open: os.Root follows in-root links, so
// only comparing the two (os.SameFile) refuses it.
func TestReadRegularNeverReturnsAnInBrainFileSwappedInBehindASymlink(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "s.md"), []byte("zqsamefilesentinel"), 0o644))
p := filepath.Join(dir, "r.md")
require.NoError(t, os.WriteFile(p, []byte("inside"), 0o644))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
var stop atomic.Bool
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
tmp := p + ".swap"
for i := 0; !stop.Load(); i++ {
_ = os.Remove(tmp)
if i%2 == 0 {
_ = os.WriteFile(tmp, []byte("inside"), 0o644)
} else {
_ = os.Symlink("s.md", tmp)
}
_ = os.Rename(tmp, p)
}
}()
leaks, ok := 0, 0
for deadline := time.Now().Add(1500 * time.Millisecond); time.Now().Before(deadline); {
got, err := b.ReadRegular("r.md")
if err == nil {
ok++
}
if string(got) == "zqsamefilesentinel" {
leaks++
}
got, _ = b.ReadRegularMax("r.md", 1<<20)
if string(got) == "zqsamefilesentinel" {
leaks++
}
}
stop.Store(true)
wg.Wait()
assert.Zero(t, leaks)
assert.Positive(t, ok, "positive control: the regular file is read")
}
func TestReadDirDirsListsOnlyRealSubdirsAndRefusesSymlinkedComponents(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "a", "real"), 0o755))
require.NoError(t, os.MkdirAll(filepath.Join(dir, "other"), 0o755))
require.NoError(t, os.Symlink(filepath.Join("..", "other"), filepath.Join(dir, "a", "linked")))
require.NoError(t, os.WriteFile(filepath.Join(dir, "a", "f.md"), []byte("f"), 0o644))
require.NoError(t, os.Symlink("a", filepath.Join(dir, "alias")))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
names, err := b.ReadDirDirs("a")
require.NoError(t, err)
assert.Equal(t, []string{"real"}, names)
_, err = b.ReadDirDirs("alias")
assert.ErrorIs(t, err, ErrNotRegular)
}
@@ -0,0 +1,141 @@
// Package brainfstest is the shared harness for hyperguild#157's swap-race
// tests: it keeps swapping a regular note, a symlink to an outside file, or a
// FIFO (with a writer held open) into a path while a reader under test loops.
// It observes two things: no sentinel byte from the outside file reaches the
// reader's output, and the reader returns within a timeout. It does not cover
// hard links, other paths into the reader, or races it does not generate.
package brainfstest
import (
"fmt"
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"syscall"
"testing"
"time"
)
// Sentinel marks the outside file's contents.
const Sentinel = "zqbrainfssentinelonefiftyseven"
// outsideContent carries Sentinel in every field a reader might extract: the
// frontmatter title and aliases, a heading, a wikilink and the body.
const outsideContent = "---\ntitle: " + Sentinel + "\naliases:\n - " + Sentinel + "\n---\n# " + Sentinel +
"\n\n" + Sentinel + " [[" + Sentinel + "]]\n"
// Outside writes a file outside any brain dir containing Sentinel.
func Outside(t *testing.T) string {
t.Helper()
p := filepath.Join(t.TempDir(), "secret.md")
if err := os.WriteFile(p, []byte(outsideContent), 0o644); err != nil {
t.Fatal(err)
}
return p
}
var swapSeq atomic.Int64
// StartSwap renames, in a loop, a regular note (content inside) and either
// a symlink to outside or a FIFO into path, until the test ends. FIFOs get a
// writer held open until the swapper exits, so a reader that skips the
// handle check parks forever instead of reading EOF. The swapper is stopped
// and awaited before t.TempDir cleanup.
func StartSwap(t *testing.T, path, inside, outside string, fifo bool) {
t.Helper()
var stop atomic.Bool
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
// A name of its own: swappers on one path share a test, and one
// opening another's FIFO for write would block forever.
tmp := fmt.Sprintf("%s.swap%d", path, swapSeq.Add(1))
// A relative link: os.Root refuses every absolute link on its own, so
// an absolute in-brain target would test nothing.
link := outside
if rel, err := filepath.Rel(filepath.Dir(path), outside); err == nil {
link = rel
}
var writers []*os.File
defer func() {
for _, w := range writers {
_ = w.Close()
}
}()
for i := 0; !stop.Load(); i++ {
_ = os.Remove(tmp)
switch {
case i%2 == 0:
_ = os.WriteFile(tmp, []byte(inside), 0o644)
case fifo:
if err := syscall.Mkfifo(tmp, 0o644); err == nil {
if w, err := os.OpenFile(tmp, os.O_RDWR, 0); err == nil {
writers = append(writers, w)
}
}
default:
_ = os.Symlink(link, tmp)
}
_ = os.Rename(tmp, path)
}
}()
t.Cleanup(func() { stop.Store(true); wg.Wait() })
}
// WriteSentinelAt writes a file carrying Sentinel at p (creating its
// directory), for an in-brain symlink target.
func WriteSentinelAt(t *testing.T, p string) string {
t.Helper()
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte(outsideContent), 0o644); err != nil {
t.Fatal(err)
}
return p
}
// AssertNoLeak loops read for d while a symlink swapper runs on path, and
// fails if any output contains Sentinel. The symlink points outside the
// brain; os.Root refuses that on its own, so pair it with AssertNoLeakFrom
// and an in-brain target.
func AssertNoLeak(t *testing.T, path, inside string, d time.Duration, read func() string) {
t.Helper()
AssertNoLeakFrom(t, path, inside, Outside(t), d, read)
}
// AssertNoLeakFrom is AssertNoLeak with the symlink pointing at target, e.g.
// a file elsewhere in the brain written with WriteSentinelAt.
func AssertNoLeakFrom(t *testing.T, path, inside, target string, d time.Duration, read func() string) {
t.Helper()
StartSwap(t, path, inside, target, false)
leaks, iters := 0, 0
for deadline := time.Now().Add(d); time.Now().Before(deadline); iters++ {
leaks += strings.Count(read(), Sentinel)
}
if leaks > 0 {
t.Fatalf("outside file leaked %d times in %d iterations", leaks, iters)
}
}
// AssertNoHang loops read for d while a FIFO swapper (held writers) runs on
// path, and fails if a read does not return within timeout.
func AssertNoHang(t *testing.T, path, inside string, d, timeout time.Duration, read func()) {
t.Helper()
StartSwap(t, path, inside, Outside(t), true)
done := make(chan struct{})
go func() {
for deadline := time.Now().Add(d); time.Now().Before(deadline); {
read()
}
close(done)
}()
select {
case <-done:
case <-time.After(timeout):
t.Fatal("reader hung on a swapped-in FIFO with a writer held open")
}
}
+43 -21
View File
@@ -12,10 +12,14 @@ package graphsync
import (
"context"
"errors"
"fmt"
"os"
"io/fs"
"path"
"path/filepath"
"strings"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
"github.com/mathiasbq/hyperguild/ingestion/internal/graph"
"github.com/mathiasbq/hyperguild/ingestion/internal/graphstore"
)
@@ -46,8 +50,25 @@ func IndexDoc(ctx context.Context, store Store, brainDir, relPath string) error
if relPath == "" {
return nil
}
abs := filepath.Join(brainDir, filepath.FromSlash(relPath))
content, err := os.ReadFile(abs)
// Callers pass caller-supplied paths (brain_tunnel, brain_update): index
// only a clean path under wiki/ or knowledge/.
if path.Clean(relPath) != relPath || !filepath.IsLocal(relPath) ||
(!strings.HasPrefix(relPath, "wiki/") && !strings.HasPrefix(relPath, "knowledge/")) {
return fmt.Errorf("index %q: not a clean wiki/ or knowledge/ path", relPath)
}
bfs, err := brainfs.Open(brainDir)
if err != nil {
return fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
return indexDoc(ctx, store, bfs, relPath)
}
// indexDoc reads relPath through the brain root, regular files only
// (hyperguild#157): the brain is a Syncthing folder, and os.ReadFile
// followed a synced symlink into the graph and blocked on a FIFO.
func indexDoc(ctx context.Context, store Store, bfs *brainfs.Brain, relPath string) error {
content, err := bfs.ReadRegular(relPath)
if err != nil {
return fmt.Errorf("read %q: %w", relPath, err)
}
@@ -77,33 +98,34 @@ func BackfillFromBrainDir(ctx context.Context, store Store, brainDir string) (in
if store == nil {
return 0, nil
}
roots := []string{"wiki", "knowledge"}
for _, root := range roots {
base := filepath.Join(brainDir, root)
if _, err := os.Stat(base); os.IsNotExist(err) {
bfs, err := brainfs.Open(brainDir)
if err != nil {
return 0, fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
for _, root := range []string{"wiki", "knowledge"} {
if _, err := bfs.Lstat(root); errors.Is(err, fs.ErrNotExist) {
continue
}
err := filepath.WalkDir(base, func(path string, d os.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if d.IsDir() {
// Any walk error, including a root that is a symlink, aborts.
var walkErr error
err := bfs.WalkRegular(root, func(rel string) error {
if path.Ext(rel) != ".md" {
return nil
}
if filepath.Ext(path) != ".md" {
return nil
}
rel, relErr := filepath.Rel(brainDir, path)
if relErr != nil {
return fmt.Errorf("rel %q: %w", path, relErr)
}
rel = filepath.ToSlash(rel)
if err := IndexDoc(ctx, store, brainDir, rel); err != nil {
if err := indexDoc(ctx, store, bfs, rel); err != nil {
return fmt.Errorf("index %q: %w", rel, err)
}
indexed++
return nil
}, func(rel string, err error) {
if walkErr == nil {
walkErr = fmt.Errorf("%s: %w", rel, err)
}
})
if err == nil {
err = walkErr
}
if err != nil {
return indexed, fmt.Errorf("walk %s: %w", root, err)
}
@@ -0,0 +1,92 @@
package graphsync
import (
"context"
"fmt"
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs/brainfstest"
)
func (f *fakeStore) dump() string {
f.mu.Lock()
defer f.mu.Unlock()
return fmt.Sprintf("%+v %+v", f.upserts, f.replaces)
}
// hyperguild#157 slice 4: IndexDoc and BackfillFromBrainDir read notes with
// os.ReadFile, so a symlinked note put the target's title and wikilinks into
// the graph, and a FIFO hung the read.
func TestIndexDocRefusesSymlinksAndSurvivesSwapRaces(t *testing.T) {
brain := t.TempDir()
facts := filepath.Join(brain, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(facts, "ok.md"), []byte("---\ntitle: zqinside\n---\nbody [[other]]\n"), 0o644))
require.NoError(t, os.Symlink(brainfstest.Outside(t), filepath.Join(facts, "leak.md")))
store := newFakeStore()
require.NoError(t, IndexDoc(context.Background(), store, brain, "wiki/w/facts/ok.md"))
assert.Contains(t, store.dump(), "zqinside", "positive control: a regular note is indexed")
assert.Error(t, IndexDoc(context.Background(), store, brain, "wiki/w/facts/leak.md"))
// A link to a file elsewhere in the brain: os.Root allows it, only the
// regular-file check refuses it.
inBrain := brainfstest.WriteSentinelAt(t, filepath.Join(brain, "raw", "s.md"))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "..", "raw", "s.md"), filepath.Join(facts, "inlink.md")))
assert.Error(t, IndexDoc(context.Background(), store, brain, "wiki/w/facts/inlink.md"))
assert.NotContains(t, store.dump(), brainfstest.Sentinel)
p := filepath.Join(facts, "r.md")
read := func() string {
s := newFakeStore()
_ = IndexDoc(context.Background(), s, brain, "wiki/w/facts/r.md")
return s.dump()
}
brainfstest.AssertNoLeak(t, p, "---\ntitle: zqinside\n---\n", 1500*time.Millisecond, read)
brainfstest.AssertNoLeakFrom(t, p, "---\ntitle: zqinside\n---\n", inBrain, 1500*time.Millisecond, read)
brainfstest.AssertNoHang(t, p, "---\ntitle: zqinside\n---\n", 1500*time.Millisecond, 6*time.Second, func() { _ = read() })
}
func TestBackfillSkipsSymlinkedNotesAndAbortsOnASymlinkedRoot(t *testing.T) {
brain := t.TempDir()
facts := filepath.Join(brain, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(facts, "ok.md"), []byte("---\ntitle: zqinside\n---\n"), 0o644))
require.NoError(t, os.Symlink(brainfstest.Outside(t), filepath.Join(facts, "leak.md")))
require.NoError(t, os.MkdirAll(filepath.Join(brain, "raw"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(brain, "raw", "r.md"), []byte("---\ntitle: zqrawonly\n---\n"), 0o644))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "..", "raw", "r.md"), filepath.Join(facts, "inlink.md")))
require.NoError(t, os.Symlink(".", filepath.Join(brain, "knowledge")))
store := newFakeStore()
_, err := BackfillFromBrainDir(context.Background(), store, brain)
assert.Error(t, err, "a symlinked root aborts the backfill")
assert.NotContains(t, store.dump(), brainfstest.Sentinel)
assert.NotContains(t, store.dump(), "zqrawonly")
require.NoError(t, os.Remove(filepath.Join(brain, "knowledge")))
store = newFakeStore()
n, err := BackfillFromBrainDir(context.Background(), store, brain)
require.NoError(t, err)
assert.Equal(t, 1, n, "only the regular note is indexed")
assert.Contains(t, store.dump(), "zqinside")
assert.NotContains(t, store.dump(), brainfstest.Sentinel)
}
// IndexDoc gets caller-supplied paths (brain_tunnel, brain_update): only a
// clean path under wiki/ or knowledge/ is indexed.
func TestIndexDocRefusesPathsOutsideWikiAndKnowledgeAndUncleanPaths(t *testing.T) {
brain := t.TempDir()
brainfstest.WriteSentinelAt(t, filepath.Join(brain, "raw", "secret.md"))
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki", "w"), 0o755))
for _, p := range []string{"raw/secret.md", "wiki/w/../../raw/secret.md", "./raw/secret.md", "/raw/secret.md"} {
store := newFakeStore()
assert.Error(t, IndexDoc(context.Background(), store, brain, p), p)
assert.NotContains(t, store.dump(), brainfstest.Sentinel, p)
}
}
@@ -0,0 +1,34 @@
package mcp
import (
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs/brainfstest"
)
// hyperguild#157 slice 2: readExcerpt (brain_context) read by path, so a
// symlink in the Syncthing-shared brain put an outside file in context, and
// a FIFO hung it. It now reads through brainfs.
func TestReadExcerptRefusesSymlinksAndSurvivesSwapRaces(t *testing.T) {
brain := t.TempDir()
facts := filepath.Join(brain, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
require.NoError(t, os.Symlink(brainfstest.Outside(t), filepath.Join(facts, "leak.md")))
assert.NotContains(t, readExcerpt(brain, "wiki/w/facts/leak.md", 500), brainfstest.Sentinel)
p := filepath.Join(facts, "r.md")
require.NoError(t, os.WriteFile(p, []byte("---\nx: y\n---\ninside"), 0o644))
assert.Equal(t, "inside", readExcerpt(brain, "wiki/w/facts/r.md", 500))
brainfstest.AssertNoLeak(t, p, "---\nx: y\n---\ninside", 1500*time.Millisecond, func() string {
return readExcerpt(brain, "wiki/w/facts/r.md", 500)
})
brainfstest.AssertNoHang(t, p, "---\nx: y\n---\ninside", 1500*time.Millisecond, 6*time.Second, func() {
_ = readExcerpt(brain, "wiki/w/facts/r.md", 500)
})
}
+8 -3
View File
@@ -4,7 +4,7 @@ import (
"context"
"encoding/json"
"fmt"
"os"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
"path/filepath"
"sort"
"strings"
@@ -181,8 +181,13 @@ func readExcerpt(brainDir, relPath string, maxLen int) string {
if relPath == "" {
return ""
}
full := filepath.Join(brainDir, filepath.FromSlash(relPath))
content, err := os.ReadFile(full)
// Through the brain root, regular files only (hyperguild#157).
bfs, err := brainfs.Open(brainDir)
if err != nil {
return ""
}
defer func() { _ = bfs.Close() }()
content, err := bfs.ReadRegular(filepath.ToSlash(relPath))
if err != nil {
return ""
}
+10 -5
View File
@@ -5,6 +5,7 @@ import (
"encoding/hex"
"errors"
"fmt"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
"os"
"path/filepath"
"strings"
@@ -71,14 +72,18 @@ func ReadScoped(brainDir, path string, scope Scope) ([]byte, error) {
}
func readContained(t Target) ([]byte, error) {
fi, err := os.Stat(t.Abs)
// Through the brain root, regular files only, size checked on the opened
// handle (hyperguild#157): Resolve's Lstat could be raced by a swap, and
// os.ReadFile blocked on a FIFO.
bfs, err := brainfs.Open(t.Brain)
if err != nil {
return nil, fmt.Errorf("notes: stat %s: %w", t.Rel, err)
return nil, fmt.Errorf("notes: %w", err)
}
if fi.Size() > maxNoteBytes {
return nil, fmt.Errorf("%w: %s is %d bytes (limit %d)", ErrTooLarge, t.Rel, fi.Size(), maxNoteBytes)
defer func() { _ = bfs.Close() }()
b, err := bfs.ReadRegularMax(t.Rel, maxNoteBytes)
if errors.Is(err, brainfs.ErrTooLarge) {
return nil, fmt.Errorf("%w: %s (limit %d)", ErrTooLarge, t.Rel, maxNoteBytes)
}
b, err := os.ReadFile(t.Abs)
if err != nil {
return nil, fmt.Errorf("notes: read %s: %w", t.Rel, err)
}
+4 -3
View File
@@ -58,8 +58,9 @@ var (
// Target is a resolved, contained path.
type Target struct {
Rel string // brain-relative, cleaned
Abs string // absolute, symlink-checked
Rel string // brain-relative, cleaned
Abs string // absolute, symlink-checked
Brain string // the brain root Rel is relative to
}
// Resolve canonicalises path, then checks it against the scope's allowlist,
@@ -109,7 +110,7 @@ func Resolve(brainDir, path string, scope Scope) (Target, error) {
return Target{}, err
}
return Target{Rel: clean, Abs: abs}, nil
return Target{Rel: clean, Abs: abs, Brain: brainDir}, nil
}
// allow enforces the scope's allowlist. The two scopes are DISJOINT, not
@@ -0,0 +1,40 @@
package notes
import (
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs/brainfstest"
)
// hyperguild#157 slice 2: ReadScoped (brain_read) Lstat-checked the final
// component, then reopened by name with os.ReadFile: a swap after the check
// was followed, and a FIFO blocked the read. It now reads through brainfs.
func TestReadScopedRefusesSymlinksAndSurvivesSwapRaces(t *testing.T) {
brain := t.TempDir()
facts := filepath.Join(brain, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
p := filepath.Join(facts, "r.md")
read := func() string {
b, _ := ReadScoped(brain, "wiki/w/facts/r.md", ScopeNotes)
return string(b)
}
require.NoError(t, os.WriteFile(p, []byte("inside"), 0o644))
assert.Equal(t, "inside", read())
brainfstest.AssertNoLeak(t, p, "inside", 1500*time.Millisecond, read)
brainfstest.AssertNoHang(t, p, "inside", 1500*time.Millisecond, 6*time.Second, func() { _ = read() })
}
func TestReadScopedRefusesANoteOverMaxNoteBytes(t *testing.T) {
brain := t.TempDir()
facts := filepath.Join(brain, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(facts, "big.md"), make([]byte, maxNoteBytes+1), 0o644))
_, err := ReadScoped(brain, "wiki/w/facts/big.md", ScopeNotes)
assert.ErrorIs(t, err, ErrTooLarge)
}
@@ -61,6 +61,31 @@ func TestQueryDoesNotFollowASymlinkedDirectory(t *testing.T) {
assert.Zero(t, leaked(rs))
}
// A search root that is a symlink staying inside the brain (wiki/<w> -> ..)
// was followed by fs.WalkDir, aliasing raw/ and root files into results
// (PR #166 review). The root is now Lstat-checked and skipped.
func TestQueryDoesNotFollowASymlinkedWingRootInsideTheBrain(t *testing.T) {
brain, _, _ := searchBrain(t)
require.NoError(t, os.MkdirAll(filepath.Join(brain, "raw"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(brain, "raw", "r.md"), []byte("# r\n\nzqrawonly"), 0o644))
require.NoError(t, os.Symlink("..", filepath.Join(brain, "wiki", "alias")))
rs, err := Query(brain, QueryOptions{Query: "zqrawonly", Wing: "alias", Limit: 10})
require.NoError(t, err)
assert.Empty(t, rs)
}
// With a hall the search root is wiki/<wing>/<hall>; a symlinked <wing> with
// a real <hall> directory under its target was followed (PR #166 round 2).
func TestQueryDoesNotFollowASymlinkedWingUnderAWingAndHallRoot(t *testing.T) {
brain, _, _ := searchBrain(t)
require.NoError(t, os.MkdirAll(filepath.Join(brain, "raw", "facts"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(brain, "raw", "facts", "r.md"), []byte("# r\n\nzqrawhall"), 0o644))
require.NoError(t, os.Symlink(filepath.Join("..", "raw"), filepath.Join(brain, "wiki", "alias")))
rs, err := Query(brain, QueryOptions{Query: "zqrawhall", Wing: "alias", Hall: "facts", Limit: 10})
require.NoError(t, err)
assert.Empty(t, rs)
}
// swapInto keeps renaming a regular note, and either a symlink to outside or
// a FIFO, into path. Each FIFO has a writer held open (as any local process
// can): a FIFO with no writer reads EOF at once, which would let a reader
+32 -17
View File
@@ -2,12 +2,15 @@ package vectorstore
import (
"context"
"errors"
"fmt"
"io/fs"
"log/slog"
"os"
"path/filepath"
"path"
"strings"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
)
// Embedder produces dense vectors. The embed package's Client satisfies
@@ -90,31 +93,38 @@ func Sync(ctx context.Context, brainDir string, store Store, embedder Embedder)
}
seenParents := make(map[string]struct{})
// Through the brain root, regular files only (hyperguild#157): the brain
// is a Syncthing folder, and os.ReadFile followed a synced symlink into
// the embeddings and blocked on a FIFO.
bfs, err := brainfs.Open(brainDir)
if err != nil {
return res, fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
for _, sub := range scanDirs {
root := filepath.Join(brainDir, sub)
if _, err := os.Stat(root); os.IsNotExist(err) {
if _, err := bfs.Lstat(sub); errors.Is(err, fs.ErrNotExist) {
continue
}
err = filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
if err != nil {
return err
// Any other walk error, including a scan root that is a symlink,
// aborts the run below.
var walkErr error
onErr := func(rel string, err error) {
if walkErr == nil {
walkErr = fmt.Errorf("%s: %w", rel, err)
}
if d.IsDir() || !strings.HasSuffix(path, ".md") || d.Name() == "_index.md" {
}
err = bfs.WalkRegular(sub, func(relSlash string) error {
if !strings.HasSuffix(relSlash, ".md") || path.Base(relSlash) == "_index.md" {
return nil
}
rel, err := filepath.Rel(brainDir, path)
if err != nil {
return err
}
relSlash := filepath.ToSlash(rel)
seenParents[relSlash] = struct{}{}
if ps, ok := parents[relSlash]; ok {
// File already has chunks in the store. Re-embed only when
// the file has been edited since the oldest chunk was
// written. Tolerate clock skew with a sub-second grace.
info, statErr := d.Info()
info, statErr := bfs.Lstat(relSlash)
if statErr != nil {
res.Errors = append(res.Errors, fmt.Errorf("stat %s: %w", relSlash, statErr))
return nil
@@ -132,7 +142,7 @@ func Sync(ctx context.Context, brainDir string, store Store, embedder Embedder)
}
}
content, readErr := os.ReadFile(path)
content, readErr := bfs.ReadRegular(relSlash)
if readErr != nil {
res.Errors = append(res.Errors, fmt.Errorf("read %s: %w", relSlash, readErr))
return nil
@@ -151,7 +161,12 @@ func Sync(ctx context.Context, brainDir string, store Store, embedder Embedder)
res.Added++
}
return nil
})
}, onErr)
if err == nil {
err = walkErr
}
// Abort before the delete pass: an incomplete walk would leave
// seenParents short and delete chunks of files that still exist.
if err != nil {
return res, fmt.Errorf("walk %s: %w", sub, err)
}
@@ -0,0 +1,126 @@
package vectorstore_test
import (
"context"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs/brainfstest"
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
)
// recordingEmbedder keeps every text it was asked to embed.
type recordingEmbedder struct {
mu sync.Mutex
texts []string
}
func (e *recordingEmbedder) Embed(_ context.Context, text string) ([]float32, error) {
e.mu.Lock()
defer e.mu.Unlock()
e.texts = append(e.texts, text)
return []float32{1}, nil
}
func (e *recordingEmbedder) all() string {
e.mu.Lock()
defer e.mu.Unlock()
return strings.Join(e.texts, "\n")
}
// hyperguild#157 slice 3: Sync walked wiki/ and knowledge/ and read each .md
// with os.ReadFile, so a symlinked note had its target embedded and served by
// semantic search, and a FIFO stalled the sync.
func TestSyncRefusesSymlinkedNotesAndSurvivesSwapRaces(t *testing.T) {
brain := t.TempDir()
writeNote(t, brain, "wiki/w/facts/ok.md", "inside note")
require.NoError(t, os.Symlink(brainfstest.Outside(t), filepath.Join(brain, "wiki", "w", "facts", "leak.md")))
emb := &recordingEmbedder{}
store := &stubStore{}
_, err := vectorstore.Sync(context.Background(), brain, store, emb)
require.NoError(t, err)
assert.Contains(t, emb.all(), "inside note", "positive control: the regular note is embedded")
assert.NotContains(t, emb.all(), brainfstest.Sentinel)
for p := range store.upserts {
assert.NotContains(t, p, "leak.md")
}
p := filepath.Join(brain, "knowledge", "r.md")
require.NoError(t, os.MkdirAll(filepath.Dir(p), 0o755))
sync := func() string {
e := &recordingEmbedder{}
_, _ = vectorstore.Sync(context.Background(), brain, &stubStore{}, e)
return e.all()
}
brainfstest.AssertNoLeak(t, p, "inside", 1500*time.Millisecond, sync)
brainfstest.AssertNoHang(t, p, "inside", 1500*time.Millisecond, 6*time.Second, func() { _ = sync() })
}
// Chunks embedded from a symlinked note before this fix are deleted: the
// note is no longer seen, so the delete pass drops its rows.
func TestSyncDeletesChunksOfASymlinkedNote(t *testing.T) {
brain := t.TempDir()
writeNote(t, brain, "wiki/w/facts/ok.md", "inside note")
require.NoError(t, os.Symlink(brainfstest.Outside(t), filepath.Join(brain, "wiki", "w", "facts", "leak.md")))
store := &stubStore{known: map[string]time.Time{
"wiki/w/facts/leak.md#0001": {},
"wiki/w/facts/ok.md#0001": {},
}}
_, err := vectorstore.Sync(context.Background(), brain, store, &recordingEmbedder{})
require.NoError(t, err)
assert.Equal(t, []string{"wiki/w/facts/leak.md#0001"}, store.deletes)
}
// A walk error aborts Sync before the delete pass, so chunks of files the
// walk could not see are kept.
func TestSyncAbortsBeforeDeletingWhenASubdirIsUnreadable(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root ignores directory permissions")
}
brain := t.TempDir()
writeNote(t, brain, "wiki/w/facts/ok.md", "inside note")
locked := filepath.Join(brain, "wiki", "w")
require.NoError(t, os.Chmod(locked, 0o000))
t.Cleanup(func() { _ = os.Chmod(locked, 0o755) })
store := &stubStore{known: map[string]time.Time{"wiki/w/facts/ok.md#0001": {}}}
_, err := vectorstore.Sync(context.Background(), brain, store, &recordingEmbedder{})
assert.Error(t, err)
assert.Empty(t, store.deletes)
}
// A scan root that is a symlink inside the brain (knowledge -> .) was
// followed by fs.WalkDir, aliasing raw/, sessions/ and root files into the
// index. Sync now aborts on it, before embedding or deleting anything.
func TestSyncAbortsOnASymlinkedScanRoot(t *testing.T) {
brain := t.TempDir()
writeNote(t, brain, "wiki/w/a.md", "wiki note")
writeNote(t, brain, "sessions/s1.md", "session note")
require.NoError(t, os.Symlink(".", filepath.Join(brain, "knowledge")))
emb := &recordingEmbedder{}
store := &stubStore{known: map[string]time.Time{"wiki/w/a.md#0001": {}}}
_, err := vectorstore.Sync(context.Background(), brain, store, emb)
assert.Error(t, err)
assert.NotContains(t, emb.all(), "session note")
for p := range store.upserts {
assert.False(t, strings.HasPrefix(p, "knowledge/"), "upserted %s", p)
}
assert.Empty(t, store.deletes)
}
// A missing scan root is skipped; the other root still syncs.
func TestSyncSkipsAMissingScanRoot(t *testing.T) {
brain := t.TempDir()
writeNote(t, brain, "wiki/w/a.md", "wiki note")
emb := &recordingEmbedder{}
_, err := vectorstore.Sync(context.Background(), brain, &stubStore{}, emb)
require.NoError(t, err)
assert.Contains(t, emb.all(), "wiki note")
}
+11 -4
View File
@@ -7,6 +7,7 @@ import (
"path/filepath"
"strings"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
"github.com/mathiasbq/hyperguild/ingestion/internal/textutil"
)
@@ -16,6 +17,11 @@ func RebuildIndex(brainDir, date string) error {
if err != nil {
return fmt.Errorf("load inventory: %w", err)
}
bfs, err := brainfs.Open(brainDir)
if err != nil {
return fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
total := len(inv[PageTypeConcept]) + len(inv[PageTypeEntity]) + len(inv[PageTypeSource])
var sb strings.Builder
@@ -34,7 +40,7 @@ func RebuildIndex(brainDir, date string) error {
label := strings.ToUpper(string(pt)[:1]) + string(pt)[1:]
fmt.Fprintf(&sb, "## %s\n\n", label)
for _, e := range entries {
summary := pageFirstSentence(brainDir, e)
summary := pageFirstSentence(bfs, e)
if summary != "" {
fmt.Fprintf(&sb, "- [[%s|%s]] — %s\n", e.Slug, e.Title, summary)
} else {
@@ -48,9 +54,10 @@ func RebuildIndex(brainDir, date string) error {
return os.WriteFile(dest, []byte(sb.String()), 0o644)
}
func pageFirstSentence(brainDir string, e Entry) string {
path := filepath.Join(brainDir, "wiki", string(e.Type), e.Slug+".md")
content, err := os.ReadFile(path)
// pageFirstSentence reads the page through the brain root, regular files
// only (hyperguild#157).
func pageFirstSentence(bfs *brainfs.Brain, e Entry) string {
content, err := bfs.ReadRegular("wiki/" + string(e.Type) + "/" + e.Slug + ".md")
if err != nil {
return ""
}
+23 -15
View File
@@ -3,36 +3,45 @@ package wiki
import (
"bufio"
"bytes"
"errors"
"fmt"
"os"
"path/filepath"
"io/fs"
"strings"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
)
// LoadInventory walks brain/wiki/ and returns all pages grouped by type.
// Missing subdirectories are silently skipped.
// Missing subdirectories are silently skipped. Pages are listed and read
// through the brain root, regular files only (hyperguild#157); a type
// directory that is a symlink is an error.
func LoadInventory(brainDir string) (map[PageType][]Entry, error) {
result := map[PageType][]Entry{
PageTypeConcept: {},
PageTypeEntity: {},
PageTypeSource: {},
}
bfs, err := brainfs.Open(brainDir)
if err != nil {
return nil, fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
for pt := range result {
dir := filepath.Join(brainDir, "wiki", string(pt))
entries, err := os.ReadDir(dir)
if os.IsNotExist(err) {
dir := "wiki/" + string(pt)
names, err := bfs.ReadDirRegular(dir)
if errors.Is(err, fs.ErrNotExist) {
continue
}
if err != nil {
return nil, fmt.Errorf("read dir %s: %w", dir, err)
}
for _, e := range entries {
if e.IsDir() || !strings.HasSuffix(e.Name(), ".md") {
for _, name := range names {
if !strings.HasSuffix(name, ".md") {
continue
}
slug := strings.TrimSuffix(e.Name(), ".md")
path := filepath.Join(dir, e.Name())
title, aliases := readFrontmatter(path, slug)
slug := strings.TrimSuffix(name, ".md")
title, aliases := readFrontmatter(bfs, dir+"/"+name, slug)
result[pt] = append(result[pt], Entry{Slug: slug, Title: title, Aliases: aliases, Type: pt})
}
}
@@ -41,15 +50,14 @@ func LoadInventory(brainDir string) (map[PageType][]Entry, error) {
// readFrontmatter extracts title and aliases from YAML frontmatter.
// Falls back to slug for title and empty aliases on any error.
func readFrontmatter(path, fallbackSlug string) (title string, aliases []string) {
func readFrontmatter(bfs *brainfs.Brain, rel, fallbackSlug string) (title string, aliases []string) {
title = fallbackSlug
f, err := os.Open(path)
content, err := bfs.ReadRegular(rel)
if err != nil {
return
}
defer f.Close() //nolint:errcheck
scanner := bufio.NewScanner(f)
scanner := bufio.NewScanner(bytes.NewReader(content))
inFM := false
inAliases := false
for scanner.Scan() {
+79
View File
@@ -0,0 +1,79 @@
package wiki
import (
"fmt"
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs/brainfstest"
)
// hyperguild#157 slice 4: LoadInventory listed wiki/<type>/ with os.ReadDir
// and opened each .md by path, so a symlinked page put the target's title and
// aliases into the inventory (and the LLM prompts built from it), and a FIFO
// hung it. RebuildIndex read first sentences the same way.
func TestLoadInventoryRefusesSymlinksAndSurvivesSwapRaces(t *testing.T) {
brain := t.TempDir()
dir := filepath.Join(brain, "wiki", "concepts")
require.NoError(t, os.MkdirAll(dir, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "ok.md"), []byte("---\ntitle: zqinside\n---\n"), 0o644))
require.NoError(t, os.Symlink(brainfstest.Outside(t), filepath.Join(dir, "leak.md")))
inBrain := brainfstest.WriteSentinelAt(t, filepath.Join(brain, "raw", "s.md"))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "raw", "s.md"), filepath.Join(dir, "inlink.md")))
inv, err := LoadInventory(brain)
require.NoError(t, err)
got := fmt.Sprintf("%+v", inv)
assert.Contains(t, got, "zqinside", "positive control")
assert.NotContains(t, got, brainfstest.Sentinel)
assert.NotContains(t, got, "leak", "a symlinked page is not listed")
assert.NotContains(t, got, "inlink", "a page linked to another brain file is not listed")
p := filepath.Join(dir, "r.md")
read := func() string {
inv, _ := LoadInventory(brain)
return fmt.Sprintf("%+v", inv)
}
brainfstest.AssertNoLeak(t, p, "---\ntitle: zqinside\n---\n", 1500*time.Millisecond, read)
brainfstest.AssertNoLeakFrom(t, p, "---\ntitle: zqinside\n---\n", inBrain, 1500*time.Millisecond, read)
brainfstest.AssertNoHang(t, p, "---\ntitle: zqinside\n---\n", 1500*time.Millisecond, 6*time.Second, func() { _ = read() })
}
func TestLoadInventoryReturnsErrNotRegularForASymlinkedTypeDir(t *testing.T) {
brain := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki"), 0o755))
require.NoError(t, os.MkdirAll(filepath.Join(brain, "raw"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(brain, "raw", "r.md"), []byte("---\ntitle: zqrawonly\n---\n"), 0o644))
require.NoError(t, os.Symlink(filepath.Join("..", "raw"), filepath.Join(brain, "wiki", "concepts")))
inv, err := LoadInventory(brain)
require.ErrorIs(t, err, brainfs.ErrNotRegular)
assert.NotContains(t, fmt.Sprintf("%+v", inv), "zqrawonly")
}
func TestRebuildIndexDoesNotReadAPageSwappedForASymlink(t *testing.T) {
brain := t.TempDir()
dir := filepath.Join(brain, "wiki", "concepts")
require.NoError(t, os.MkdirAll(dir, 0o755))
p := filepath.Join(dir, "r.md")
require.NoError(t, os.WriteFile(p, []byte("---\ntitle: R\n---\nzqinside sentence.\n"), 0o644))
require.NoError(t, RebuildIndex(brain, "2026-10-04"))
idx, err := os.ReadFile(filepath.Join(brain, "wiki", "index.md"))
require.NoError(t, err)
require.Contains(t, string(idx), "zqinside", "positive control: the summary is read")
read := func() string {
_ = RebuildIndex(brain, "2026-10-04")
b, _ := os.ReadFile(filepath.Join(brain, "wiki", "index.md"))
return string(b)
}
brainfstest.AssertNoLeak(t, p, "---\ntitle: R\n---\nzqinside sentence.\n", 1500*time.Millisecond, read)
inBrain := brainfstest.WriteSentinelAt(t, filepath.Join(brain, "raw", "s.md"))
brainfstest.AssertNoLeakFrom(t, p, "---\ntitle: R\n---\nzqinside sentence.\n", inBrain, 1500*time.Millisecond, read)
brainfstest.AssertNoHang(t, p, "---\ntitle: R\n---\nzqinside sentence.\n", 1500*time.Millisecond, 6*time.Second, func() { _ = read() })
}