Compare commits

..
Author SHA1 Message Date
mathiasandClaude Opus 5.5 cce4dc3093 Merge origin/main into fix/tunnel-traversal (conflict-only)
CI / Lint / Test / Vet (pull_request) Successful in 1m7s
CI / Mirror to GitHub (pull_request) Skipped
Conflict in ingestion/internal/brainfs/brainfs_test.go only: both sides
appended a test after TestReadRegularNeverReturnsAnInBrainFileSwappedInBehindASymlink
(this branch: TestWriteRegularRefusesFinalSymlinksFIFOsEscapesAndMissingFilesAndRewritesRegularFiles;
main: TestReadDirDirsListsOnlyRealSubdirsAndRefusesSymlinkedComponents).
Kept both, unchanged. No production code touched by the resolution.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:37:53 +02:00
mathiasandClaude Opus 5.5 0bd0fbf03e test(brainfstest): swap-race asserts fail when nothing was tested
CI / Lint / Test / Vet (push) Successful in 55s
CI / Mirror to GitHub (push) Successful in 4s
AssertNoLeakFrom and the search Query race test passed vacuously when
every symlink swap failed or every read errored. Both now count
successful symlink swaps and reads that returned the inside note and
fail if either is zero; the Query race reader no longer swallows
Query's error.

Verified by flipping: a swapper whose os.Symlink always fails turns
notes, wiki and search race tests red; Query on a missing brain dir
turns the search race test red.

Closes #177

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 09:32:25 +02:00
mathiasandClaude Opus 5.5 ef39435252 fix(brain): wing indexes and the link check read through brainfs (#157, slice 5a) (#175)
CI / Lint / Test / Vet (push) Successful in 55s
CI / Mirror to GitHub (push) Successful in 5s
BuildWingIndex/BuildAllWingIndexes list wings and halls with the new brainfs.ReadDirDirs and ReadDirRegular and read titles with ReadRegular: a symlinked note no longer reaches _index.md, a symlinked wing is refused (nothing written into its target), a symlinked hall is skipped. linkcheck entryIndex walks with WalkRegular, so a symlinked .md cannot make a link resolve, and a linked tree that is not a real directory is skipped with a warning (brain_write still lands).

Refs #157. Adversarial review: BLOCKED, PASS. Follow-up: #178.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-05 07:26:12 +00:00
mathiasandClaude Opus 5.5 b6ef11600b fix(deps): bump x/net and x/text past called vulns
CI / Lint / Test / Vet (push) Successful in 1m45s
CI / Mirror to GitHub (push) Successful in 4s
govulncheck 2026-10-04 found called vulns in ingestion (brain-mcp, public),
reached via audit.LokiCentral.Push: golang.org/x/net v0.26.0
(GO-2026-4918 HTTP/2 infinite loop, GO-2026-5026) and golang.org/x/text
v0.29.0 (GO-2026-5970). x/net -> v0.55.0, x/text -> v0.39.0; tidy also
moved x/sync, x/sys, x/term. govulncheck ./... now: 0 called findings
(19 in required modules, none reachable).

Refs: govulncheck 2026-10-04

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 06:59:31 +02:00
mathiasandClaude Opus 5.5 432e09d122 test(ingestion): OIDC discovery 502s then 200 against a real issuer (#83)
The retry landed in 9a232ad but was tested only through startupretry with
fake funcs; nothing exercised chassisauth.NewJWTValidator's real 502 path.
Extracts the main() closure into buildJWTValidator (behaviour unchanged)
and tests it against an httptest issuer: 3x502 then 200 succeeds with 4
discoveries; a persistent 502 stops at exactly the 8-attempt budget with
the last error; cancelling ctx mid-backoff returns context.Canceled.
Flip-verified: Attempts=1 turns all three red.

Lane: risk/medium issue, committed to main because the behaviour change
already shipped in 9a232ad; this is a test plus a pure extraction.

Closes #83

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 06:57:45 +02:00
mathiasandClaude Opus 5.5 6325c2b414 test(session): pin the id '/' ban and the FIFO O_NONBLOCK guard (#169)
CI / Lint / Test / Vet (pull_request) Successful in 56s
CI / Mirror to GitHub (pull_request) Skipped
PR #170 round-2 review: both guards stayed green when removed. Adds an
ErrInvalidID sentinel, a plain-prefix-then-'..' escape case asserted on
the sentinel, and a no-reader FIFO case with a timeout. Flip-verified:
allowing '/' in the id tail and dropping O_NONBLOCK each turn the suite red.

Refs #169

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 06:54:23 +02:00
mathiasandClaude Opus 5.5 b3d782b25e fix(update,session): brain_update and session_log confine caller paths (#169)
Review round 1 on PR #170 found two more handlers with the #169 class of
bug (verified by execution):

- brain_update resolved its caller path with resolveWithin, a lexical
  containment check: any brain file could be superseded
  (classification.yaml, raw/, sessions/), and a final symlink was
  written through. UpdateNote now resolves like brain_edit
  (notes.Resolve, ScopeNotes: knowledge/ and wiki/ .md notes), and reads
  and rewrites through brainfs (ReadRegular, WriteRegular).
- session_log joined the caller's session_id onto the sessions dir and
  opened it O_CREATE|O_APPEND: an id with path segments created or
  appended a .jsonl file anywhere the pod could write. The id must now
  be a plain name (^[A-Za-z0-9_][A-Za-z0-9._-]{0,127}$, which every live
  session file matches), and the open adds O_NOFOLLOW|O_NONBLOCK. Read
  checks the id too.
- Tunnel tests: the .txt case now names an existing file (it failed on
  "missing" before), the symlink test adds an in-brain target, and the
  FIFO comment no longer claims the handle check is the only guard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 06:54:23 +02:00
mathiasandClaude Opus 5.5 e548b97ba4 fix(tunnel): brain_tunnel cannot write outside the wiki or through symlinks
brain_tunnel passes caller-supplied source and target paths to
WriteTunnel, which checked only the first segments ("wiki/<wing>/...")
and joined them onto the brain dir. "wiki/b/x/../../../../tmp/f.md"
resolved outside the brain, and appendSeeAlso read and rewrote that file
(verified by execution): any MCP token holder could append to any
existing file the pod user can write, inside the brain (raw/, sessions/,
classification.yaml) or outside it.

- checkNotePath: only a clean, local wiki/.../<slug>.md path, with no
  ".", ".." or empty segments.
- Both notes are read and rewritten through brainfs. New
  brainfs.WriteRegular: never creates; Lstat through the root, open with
  O_NOFOLLOW|O_NONBLOCK, and require the opened handle to be the same
  regular file (os.SameFile). os.Root.OpenFile follows an in-root final
  symlink even with O_NOFOLLOW (verified on go1.27), so the flag alone
  does not refuse it.
- The source is no longer rewritten when the target is refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 06:54:23 +02:00
mathiasandClaude Opus 5.5 77e32de33c fix(graph,wiki): read the brain through brainfs (#157, slice 4) (#168)
CI / Lint / Test / Vet (push) Successful in 52s
CI / Mirror to GitHub (push) Successful in 4s
graphsync, wiki inventory/index read through brainfs; brainfs refuses an in-brain file swapped in behind a symlink (os.SameFile). Gatekeeper round 2 PASS on d90ce66, CI run 9180 green.

Refs #157

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-05 04:53:34 +00:00
19 changed files with 1075 additions and 129 deletions
+25
View File
@@ -0,0 +1,25 @@
package main
import (
"context"
"log/slog"
chassisauth "git.d-ma.be/mathias/mcp-chassis/auth"
"github.com/mathiasbq/hyperguild/ingestion/internal/startupretry"
)
// buildJWTValidator runs OIDC discovery under policy, so a 502 from an auth
// upstream that is not serving yet on a cold boot is retried instead of
// exiting (#83). It honours ctx between attempts.
func buildJWTValidator(
ctx context.Context, logger *slog.Logger, issuer, audience string, policy startupretry.Policy,
) (*chassisauth.JWTValidator, error) {
return startupretry.Do(ctx, policy, func(ctx context.Context) (*chassisauth.JWTValidator, error) {
v, err := chassisauth.NewJWTValidator(ctx, issuer, audience)
if err != nil {
logger.Warn("oidc discovery not ready, retrying", "err", err)
}
return v, err
})
}
@@ -0,0 +1,86 @@
package main
import (
"context"
"io"
"log/slog"
"net/http"
"net/http/httptest"
"sync/atomic"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/startupretry"
)
// fakeIssuer answers OIDC discovery with 502 for the first fail requests, as
// ingress does while dex is not yet serving on a cold boot (#83), then with a
// real discovery document and an empty JWKS.
func fakeIssuer(t *testing.T, fail int32) (*httptest.Server, *atomic.Int32) {
t.Helper()
var discoveries atomic.Int32
mux := http.NewServeMux()
var srv *httptest.Server
mux.HandleFunc("/.well-known/openid-configuration", func(w http.ResponseWriter, _ *http.Request) {
if discoveries.Add(1) <= fail {
w.WriteHeader(http.StatusBadGateway)
return
}
w.Header().Set("Content-Type", "application/json")
_, _ = io.WriteString(w, `{"issuer":"`+srv.URL+`","jwks_uri":"`+srv.URL+`/keys"}`)
})
mux.HandleFunc("/keys", func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = io.WriteString(w, `{"keys":[]}`)
})
srv = httptest.NewServer(mux)
t.Cleanup(srv.Close)
return srv, &discoveries
}
var fastPolicy = startupretry.Policy{Attempts: 8, Base: time.Millisecond, Max: 5 * time.Millisecond}
func TestBuildJWTValidatorRetriesDiscovery502sThenSucceeds(t *testing.T) {
srv, discoveries := fakeIssuer(t, 3)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
v, err := buildJWTValidator(ctx, slog.New(slog.DiscardHandler), srv.URL, "brain", fastPolicy)
require.NoError(t, err)
assert.NotNil(t, v)
assert.Equal(t, int32(4), discoveries.Load(), "three 502s, then the successful discovery")
}
func TestBuildJWTValidatorGivesUpAfterTheBudgetWithTheLastError(t *testing.T) {
srv, discoveries := fakeIssuer(t, 1000)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
_, err := buildJWTValidator(ctx, slog.New(slog.DiscardHandler), srv.URL, "brain", fastPolicy)
require.Error(t, err)
assert.Contains(t, err.Error(), "status 502")
assert.Equal(t, int32(8), discoveries.Load(), "bounded: exactly the attempt budget")
}
func TestBuildJWTValidatorStopsRetryingWhenTheContextIsCancelled(t *testing.T) {
srv, _ := fakeIssuer(t, 1000)
ctx, cancel := context.WithCancel(context.Background())
slow := startupretry.Policy{Attempts: 8, Base: time.Hour, Max: time.Hour}
done := make(chan error, 1)
go func() {
_, err := buildJWTValidator(ctx, slog.New(slog.DiscardHandler), srv.URL, "brain", slow)
done <- err
}()
time.Sleep(50 * time.Millisecond)
cancel()
select {
case err := <-done:
assert.ErrorIs(t, err, context.Canceled)
case <-time.After(3 * time.Second):
t.Fatal("did not return after the context was cancelled")
}
}
+3 -9
View File
@@ -381,15 +381,9 @@ func main() {
// //
// The budget stays bounded: retrying forever would turn a misconfiguration // The budget stays bounded: retrying forever would turn a misconfiguration
// into a process that looks alive and never serves. // into a process that looks alive and never serves.
jwtValidator, err := startupretry.Do(ctx, jwtValidator, err := buildJWTValidator(ctx, logger,
startupretry.Policy{Attempts: 8, Base: time.Second, Max: 30 * time.Second}, os.Getenv("DEX_ISSUER_URL"), os.Getenv("MCP_AUDIENCE"),
func(ctx context.Context) (*chassisauth.JWTValidator, error) { startupretry.Policy{Attempts: 8, Base: time.Second, Max: 30 * time.Second})
v, err := chassisauth.NewJWTValidator(ctx, os.Getenv("DEX_ISSUER_URL"), os.Getenv("MCP_AUDIENCE"))
if err != nil {
logger.Warn("oidc discovery not ready, retrying", "err", err)
}
return v, err
})
if err != nil { if err != nil {
logger.Error("build jwt validator", "err", err) logger.Error("build jwt validator", "err", err)
os.Exit(1) os.Exit(1)
+6 -6
View File
@@ -34,9 +34,9 @@ require (
github.com/rogpeppe/go-internal v1.15.0 // indirect github.com/rogpeppe/go-internal v1.15.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect github.com/spf13/pflag v1.0.5 // indirect
github.com/x448/float16 v0.8.4 // indirect github.com/x448/float16 v0.8.4 // indirect
golang.org/x/net v0.26.0 // indirect golang.org/x/net v0.55.0 // indirect
golang.org/x/oauth2 v0.21.0 // indirect golang.org/x/oauth2 v0.21.0 // indirect
golang.org/x/term v0.28.0 // indirect golang.org/x/term v0.43.0 // indirect
golang.org/x/time v0.3.0 // indirect golang.org/x/time v0.3.0 // indirect
google.golang.org/protobuf v1.34.2 // indirect google.golang.org/protobuf v1.34.2 // indirect
gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
@@ -66,9 +66,9 @@ require (
github.com/lestrrat-go/option v1.0.1 // indirect github.com/lestrrat-go/option v1.0.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/segmentio/asm v1.2.0 // indirect github.com/segmentio/asm v1.2.0 // indirect
golang.org/x/crypto v0.32.0 // indirect golang.org/x/crypto v0.51.0 // indirect
golang.org/x/sync v0.17.0 // indirect golang.org/x/sync v0.21.0 // indirect
golang.org/x/sys v0.31.0 // indirect golang.org/x/sys v0.45.0 // indirect
golang.org/x/text v0.29.0 // indirect golang.org/x/text v0.39.0 // indirect
gopkg.in/yaml.v3 v3.0.1 gopkg.in/yaml.v3 v3.0.1
) )
+14 -14
View File
@@ -117,42 +117,42 @@ github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9dec
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.32.0 h1:euUpcYgM8WcP71gNpTqQCn6rC2t6ULUPiOzfWaXVVfc= golang.org/x/crypto v0.51.0 h1:IBPXwPfKxY7cWQZ38ZCIRPI50YLeevDLlLnyC5wRGTI=
golang.org/x/crypto v0.32.0/go.mod h1:ZnnJkOaASj8g0AjIduWNlq2NRxL0PlBrbKVyZ6V/Ugc= golang.org/x/crypto v0.51.0/go.mod h1:8AdwkbraGNABw2kOX6YFPs3WM22XqI4EXEd8g+x7Oc8=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.26.0 h1:soB7SVo0PWrY4vPW/+ay0jKDNScG2X9wFeYlXIvJsOQ= golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8=
golang.org/x/net v0.26.0/go.mod h1:5YKkiSynbBIh3p6iOc/vibscux0x38BZDkn8sCUPxHE= golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww=
golang.org/x/oauth2 v0.21.0 h1:tsimM75w1tF/uws5rbeHzIWxEqElMehnc+iW793zsZs= golang.org/x/oauth2 v0.21.0 h1:tsimM75w1tF/uws5rbeHzIWxEqElMehnc+iW793zsZs=
golang.org/x/oauth2 v0.21.0/go.mod h1:XYTD2NtWslqkgxebSiOHnXEap4TF09sJSc7H1sXbhtI= golang.org/x/oauth2 v0.21.0/go.mod h1:XYTD2NtWslqkgxebSiOHnXEap4TF09sJSc7H1sXbhtI=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.17.0 h1:l60nONMj9l5drqw6jlhIELNv9I0A4OFgRsG9k2oT9Ug= golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
golang.org/x/sync v0.17.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI= golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik= golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k= golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/term v0.28.0 h1:/Ts8HFuMR2E6IP/jlo7QVLZHggjKQbhu/7H0LJFr3Gg= golang.org/x/term v0.43.0 h1:S4RLU2sB31O/NCl+zFN9Aru9A/Cq2aqKpTZJ6B+DwT4=
golang.org/x/term v0.28.0/go.mod h1:Sw/lC2IAUZ92udQNf3WodGtn4k/XoLyZoh8v/8uiwek= golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.29.0 h1:1neNs90w9YzJ9BocxfsQNHKuAT4pkghyXc4nhZ6sJvk= golang.org/x/text v0.39.0 h1:UbZz4pLOvn600D6Oh6GGEI6VAmndrEBLv8/6BEXzyus=
golang.org/x/text v0.29.0/go.mod h1:7MhJOA9CD2qZyOKYazxdYMF85OwPdEr9jTtBpO7ydH4= golang.org/x/text v0.39.0/go.mod h1:3UwRclnC2g0TU9x8PZiyfOajCd1zaUNHF9cvqcQZ+ZM=
golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4= golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4=
golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.36.0 h1:kWS0uv/zsvHEle1LbV5LE8QujrxB3wfQyxHfhOk0Qkg= golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
golang.org/x/tools v0.36.0/go.mod h1:WBDiHKJK8YgLHlcQPYQzNCkUxUypCaa5ZegCVutKm+s= golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+18 -6
View File
@@ -3,15 +3,18 @@ package api
import ( import (
"crypto/sha256" "crypto/sha256"
"encoding/hex" "encoding/hex"
"errors"
"fmt" "fmt"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs" "io/fs"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
"time" "time"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain" "github.com/mathiasbq/hyperguild/ingestion/internal/brain"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
"github.com/mathiasbq/hyperguild/ingestion/internal/claim" "github.com/mathiasbq/hyperguild/ingestion/internal/claim"
"github.com/mathiasbq/hyperguild/ingestion/internal/notes"
) )
// ContentHash returns the lowercase hex sha256 of b. It is the note's // ContentHash returns the lowercase hex sha256 of b. It is the note's
@@ -91,14 +94,23 @@ func UpdateNote(brainDir string, opts UpdateNoteOptions) (relPath, contentHash,
rel = filepath.ToSlash(rel) rel = filepath.ToSlash(rel)
} }
rel, abs, err := resolveWithin(brainDir, rel) // The path is caller-supplied (brain_update): resolve it as brain_edit
// does, knowledge/ and wiki/ notes only, and read and rewrite it through
// the brain root, never through a symlink (hyperguild#169).
t, err := notes.Resolve(brainDir, rel, notes.ScopeNotes)
if err != nil { if err != nil {
return "", "", "", err return "", "", "", err
} }
rel = t.Rel
prior, err := os.ReadFile(abs) bfs, err := brainfs.Open(brainDir)
if err != nil { if err != nil {
if os.IsNotExist(err) { return "", "", "", fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
prior, err := bfs.ReadRegular(rel)
if err != nil {
if errors.Is(err, fs.ErrNotExist) {
return "", "", "", fmt.Errorf("note %q does not exist: use brain_write to create", rel) return "", "", "", fmt.Errorf("note %q does not exist: use brain_write to create", rel)
} }
return "", "", "", fmt.Errorf("read target: %w", err) return "", "", "", fmt.Errorf("read target: %w", err)
@@ -164,7 +176,7 @@ func UpdateNote(brainDir string, opts UpdateNoteOptions) (relPath, contentHash,
return "", "", "", err return "", "", "", err
} }
out := []byte(stamped) out := []byte(stamped)
if err := os.WriteFile(abs, out, 0o644); err != nil { if err := bfs.WriteRegular(rel, out); err != nil {
return "", "", "", fmt.Errorf("write: %w", err) return "", "", "", fmt.Errorf("write: %w", err)
} }
return rel, ContentHash(out), priorHash, nil return rel, ContentHash(out), priorHash, nil
+112
View File
@@ -0,0 +1,112 @@
package api_test
import (
"os"
"path/filepath"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
)
// brain_update passed the caller's path through resolveWithin, a lexical
// containment check only: any file in the brain could be superseded
// (classification.yaml, raw/, sessions/), and a final symlink was written
// through. It now resolves like brain_edit: knowledge/ and wiki/ notes only,
// read and rewritten through the brain root.
func TestUpdateNoteRefusesPathsOutsideKnowledgeAndWikiNotes(t *testing.T) {
brain := t.TempDir()
files := map[string]string{
"classification.yaml": "levels: {}\n",
"raw/r.md": "raw\n",
"sessions/s.jsonl": "{}\n",
"wiki/w/facts/ok.md": "---\ntitle: ok\n---\nold\n",
"knowledge/k.md": "---\ntitle: k\n---\nold\n",
"wiki/w/facts/x.txt": "txt\n",
}
for rel, body := range files {
p := filepath.Join(brain, filepath.FromSlash(rel))
require.NoError(t, os.MkdirAll(filepath.Dir(p), 0o755))
require.NoError(t, os.WriteFile(p, []byte(body), 0o644))
}
for _, rel := range []string{"classification.yaml", "raw/r.md", "sessions/s.jsonl", "wiki/w/facts/x.txt", "wiki/w/../../raw/r.md"} {
_, _, _, err := api.UpdateNote(brain, api.UpdateNoteOptions{Path: rel, Content: "replaced"})
assert.Error(t, err, rel)
}
for rel, body := range files {
if rel == "wiki/w/facts/ok.md" || rel == "knowledge/k.md" {
continue
}
got, err := os.ReadFile(filepath.Join(brain, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.Equal(t, body, string(got), rel)
}
for _, rel := range []string{"wiki/w/facts/ok.md", "knowledge/k.md"} {
_, _, _, err := api.UpdateNote(brain, api.UpdateNoteOptions{Path: rel, Content: "new body"})
require.NoError(t, err, "positive control: %s", rel)
got, _ := os.ReadFile(filepath.Join(brain, filepath.FromSlash(rel)))
assert.Contains(t, string(got), "new body")
}
}
func TestUpdateNoteDoesNotWriteThroughASymlinkedNote(t *testing.T) {
brain := t.TempDir()
facts := filepath.Join(brain, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
victim := filepath.Join(brain, "raw", "victim.md")
require.NoError(t, os.MkdirAll(filepath.Dir(victim), 0o755))
require.NoError(t, os.WriteFile(victim, []byte("untouched\n"), 0o644))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "..", "raw", "victim.md"), filepath.Join(facts, "l.md")))
_, _, _, err := api.UpdateNote(brain, api.UpdateNoteOptions{Path: "wiki/w/facts/l.md", Content: "replaced"})
assert.Error(t, err)
got, _ := os.ReadFile(victim)
assert.Equal(t, "untouched\n", string(got))
}
// The note is swapped between a regular file and an in-brain symlink to a
// raw/ file while UpdateNote runs: the read and the write each refuse a
// symlink, so the raw/ file is never rewritten.
func TestUpdateNoteNeverRewritesASymlinkSwappedInForTheNote(t *testing.T) {
brain := t.TempDir()
facts := filepath.Join(brain, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
victim := filepath.Join(brain, "raw", "victim.md")
require.NoError(t, os.MkdirAll(filepath.Dir(victim), 0o755))
require.NoError(t, os.WriteFile(victim, []byte("untouched\n"), 0o644))
note := filepath.Join(facts, "n.md")
body := "---\ntitle: n\n---\nold\n"
require.NoError(t, os.WriteFile(note, []byte(body), 0o644))
var stop atomic.Bool
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
tmp := note + ".swap"
for i := 0; !stop.Load(); i++ {
_ = os.Remove(tmp)
if i%2 == 0 {
_ = os.WriteFile(tmp, []byte(body), 0o644)
} else {
_ = os.Symlink(filepath.Join("..", "..", "..", "raw", "victim.md"), tmp)
}
_ = os.Rename(tmp, note)
}
}()
ok := 0
for deadline := time.Now().Add(1500 * time.Millisecond); time.Now().Before(deadline); {
if _, _, _, err := api.UpdateNote(brain, api.UpdateNoteOptions{Path: "wiki/w/facts/n.md", Content: "new"}); err == nil {
ok++
}
}
stop.Store(true)
wg.Wait()
got, _ := os.ReadFile(victim)
assert.Equal(t, "untouched\n", string(got))
assert.Positive(t, ok, "positive control: some updates reach the regular note")
}
+67 -37
View File
@@ -2,12 +2,18 @@ package brain
import ( import (
"bufio" "bufio"
"bytes"
"errors"
"fmt" "fmt"
"io"
"io/fs"
"os" "os"
"path/filepath" "path/filepath"
"sort" "sort"
"strings" "strings"
"time" "time"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
) )
// noteEntry is one row in a Wing _index.md. // noteEntry is one row in a Wing _index.md.
@@ -27,13 +33,18 @@ func BuildWingIndex(brainDir, wing string) error {
return fmt.Errorf("invalid wing %q", wing) return fmt.Errorf("invalid wing %q", wing)
} }
wingDir := filepath.Join(brainDir, "wiki", w) wingDir := filepath.Join(brainDir, "wiki", w)
if _, err := os.Stat(wingDir); os.IsNotExist(err) { // Listed and read through the brain root (hyperguild#157): a wing or
// hall that is a symlink is not followed, and notes are regular files.
bfs, err := brainfs.Open(brainDir)
if err != nil {
return fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
if _, err := bfs.Lstat("wiki/" + w); errors.Is(err, fs.ErrNotExist) {
return nil return nil
} else if err != nil {
return fmt.Errorf("stat wing: %w", err)
} }
entries, err := collectWingEntries(wingDir) entries, err := collectWingEntries(bfs, "wiki/"+w)
if err != nil { if err != nil {
return err return err
} }
@@ -58,51 +69,47 @@ func BuildWingIndex(brainDir, wing string) error {
// BuildAllWingIndexes regenerates _index.md for every wing under brain/wiki/. // BuildAllWingIndexes regenerates _index.md for every wing under brain/wiki/.
func BuildAllWingIndexes(brainDir string) error { func BuildAllWingIndexes(brainDir string) error {
wikiDir := filepath.Join(brainDir, "wiki") bfs, err := brainfs.Open(brainDir)
ents, err := os.ReadDir(wikiDir) if err != nil {
if os.IsNotExist(err) { return fmt.Errorf("open brain: %w", err)
}
wings, err := bfs.ReadDirDirs("wiki")
_ = bfs.Close()
if errors.Is(err, fs.ErrNotExist) {
return nil return nil
} }
if err != nil { if err != nil {
return fmt.Errorf("read wiki: %w", err) return fmt.Errorf("read wiki: %w", err)
} }
for _, e := range ents { for _, w := range wings {
if !e.IsDir() { if err := BuildWingIndex(brainDir, w); err != nil {
continue return fmt.Errorf("index %s: %w", w, err)
}
if err := BuildWingIndex(brainDir, e.Name()); err != nil {
return fmt.Errorf("index %s: %w", e.Name(), err)
} }
} }
return nil return nil
} }
func collectWingEntries(wingDir string) ([]noteEntry, error) { func collectWingEntries(bfs *brainfs.Brain, wingRel string) ([]noteEntry, error) {
var out []noteEntry var out []noteEntry
ents, err := os.ReadDir(wingDir) halls, err := bfs.ReadDirDirs(wingRel)
if err != nil { if err != nil {
return nil, fmt.Errorf("read wing: %w", err) return nil, fmt.Errorf("read wing: %w", err)
} }
for _, hallEnt := range ents { for _, hall := range halls {
if !hallEnt.IsDir() {
continue
}
hall := hallEnt.Name()
if !IsValidHall(hall) { if !IsValidHall(hall) {
continue continue
} }
hallDir := filepath.Join(wingDir, hall) hallRel := wingRel + "/" + hall
notes, err := os.ReadDir(hallDir) names, err := bfs.ReadDirRegular(hallRel)
if err != nil { if err != nil {
return nil, fmt.Errorf("read hall %s: %w", hall, err) return nil, fmt.Errorf("read hall %s: %w", hall, err)
} }
for _, n := range notes { for _, name := range names {
if n.IsDir() || !strings.HasSuffix(n.Name(), ".md") || n.Name() == "_index.md" { if !strings.HasSuffix(name, ".md") || name == "_index.md" {
continue continue
} }
slug := strings.TrimSuffix(n.Name(), ".md") slug := strings.TrimSuffix(name, ".md")
full := filepath.Join(hallDir, n.Name()) title, created := readTitleAndCreatedIn(bfs, hallRel+"/"+name, slug)
title, created := readTitleAndCreated(full, slug)
out = append(out, noteEntry{Hall: hall, Slug: slug, Title: title, Created: created}) out = append(out, noteEntry{Hall: hall, Slug: slug, Title: title, Created: created})
} }
} }
@@ -117,9 +124,40 @@ func readTitleAndCreated(path, slug string) (string, string) {
return slug, "" return slug, ""
} }
defer func() { _ = f.Close() }() defer func() { _ = f.Close() }()
title, created := parseTitleAndCreated(f)
if created == "" {
if info, err := os.Stat(path); err == nil {
created = info.ModTime().UTC().Format("2006-01-02")
}
}
return titleOrSlug(title, slug), created
}
title, created := "", "" // readTitleAndCreatedIn is readTitleAndCreated through the brain root,
scanner := bufio.NewScanner(f) // regular files only (hyperguild#157).
func readTitleAndCreatedIn(bfs *brainfs.Brain, rel, slug string) (string, string) {
content, err := bfs.ReadRegular(rel)
if err != nil {
return slug, ""
}
title, created := parseTitleAndCreated(bytes.NewReader(content))
if created == "" {
if info, err := bfs.Lstat(rel); err == nil {
created = info.ModTime().UTC().Format("2006-01-02")
}
}
return titleOrSlug(title, slug), created
}
func titleOrSlug(title, slug string) string {
if title == "" {
return strings.ReplaceAll(slug, "-", " ")
}
return title
}
func parseTitleAndCreated(r io.Reader) (title, created string) {
scanner := bufio.NewScanner(r)
inFrontmatter := false inFrontmatter := false
for scanner.Scan() { for scanner.Scan() {
line := scanner.Text() line := scanner.Text()
@@ -149,13 +187,5 @@ func readTitleAndCreated(path, slug string) (string, string) {
} }
} }
} }
if title == "" {
title = strings.ReplaceAll(slug, "-", " ")
}
if created == "" {
if info, err := os.Stat(path); err == nil {
created = info.ModTime().UTC().Format("2006-01-02")
}
}
return title, created return title, created
} }
+132
View File
@@ -0,0 +1,132 @@
package brain_test
import (
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs/brainfstest"
)
func readIndex(t *testing.T, brainDir, wing string) string {
t.Helper()
b, _ := os.ReadFile(filepath.Join(brainDir, "wiki", wing, "_index.md"))
return string(b)
}
// hyperguild#157 slice 5: BuildWingIndex listed wiki/<wing>/<hall>/ with
// os.ReadDir and opened each note by path, so a symlinked note put the
// target's title into _index.md, and a FIFO hung the index build.
func TestBuildWingIndexRefusesSymlinkedNotesAndSurvivesSwapRaces(t *testing.T) {
b := t.TempDir()
facts := filepath.Join(b, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(facts, "ok.md"), []byte("---\ntitle: zqinside\n---\n"), 0o644))
require.NoError(t, os.Symlink(brainfstest.Outside(t), filepath.Join(facts, "leak.md")))
inBrain := brainfstest.WriteSentinelAt(t, filepath.Join(b, "raw", "s.md"))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "..", "raw", "s.md"), filepath.Join(facts, "inlink.md")))
require.NoError(t, brain.BuildWingIndex(b, "w"))
idx := readIndex(t, b, "w")
assert.Contains(t, idx, "zqinside", "positive control")
assert.NotContains(t, idx, brainfstest.Sentinel)
assert.NotContains(t, idx, "leak")
assert.NotContains(t, idx, "inlink")
p := filepath.Join(facts, "r.md")
read := func() string {
_ = brain.BuildWingIndex(b, "w")
return readIndex(t, b, "w")
}
brainfstest.AssertNoLeak(t, p, "---\ntitle: zqinside\n---\n", 1500*time.Millisecond, read)
brainfstest.AssertNoLeakFrom(t, p, "---\ntitle: zqinside\n---\n", inBrain, 1500*time.Millisecond, read)
brainfstest.AssertNoHang(t, p, "---\ntitle: zqinside\n---\n", 1500*time.Millisecond, 6*time.Second, func() { _ = read() })
}
// A symlinked hall is skipped while a real hall next to it is indexed; a
// symlinked wing is refused and nothing is written into its target.
func TestBuildWingIndexSkipsASymlinkedHallAndRefusesASymlinkedWing(t *testing.T) {
b := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(b, "raw", "facts"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(b, "raw", "facts", "r.md"), []byte("---\ntitle: zqrawonly\n---\n"), 0o644))
require.NoError(t, os.MkdirAll(filepath.Join(b, "wiki", "w", "decisions"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(b, "wiki", "w", "decisions", "d.md"), []byte("---\ntitle: zqrealhall\n---\n"), 0o644))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "raw", "facts"), filepath.Join(b, "wiki", "w", "facts")))
require.NoError(t, os.Symlink(filepath.Join("..", "raw"), filepath.Join(b, "wiki", "alias")))
require.NoError(t, brain.BuildWingIndex(b, "w"))
idx, err := os.ReadFile(filepath.Join(b, "wiki", "w", "_index.md"))
require.NoError(t, err, "the wing index is written")
assert.Contains(t, string(idx), "zqrealhall", "the real hall is indexed")
assert.NotContains(t, string(idx), "zqrawonly", "the symlinked hall is not")
assert.Error(t, brain.BuildWingIndex(b, "alias"))
_, err = os.Stat(filepath.Join(b, "raw", "_index.md"))
assert.True(t, os.IsNotExist(err), "an index was written into the symlinked wing's target")
}
// BuildAllWingIndexes indexes the real wings and skips a symlinked one.
func TestBuildAllWingIndexesSkipsASymlinkedWing(t *testing.T) {
b := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(b, "wiki", "w", "facts"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(b, "wiki", "w", "facts", "n.md"), []byte("---\ntitle: zqrealwing\n---\n"), 0o644))
require.NoError(t, os.MkdirAll(filepath.Join(b, "raw", "facts"), 0o755))
require.NoError(t, os.Symlink(filepath.Join("..", "raw"), filepath.Join(b, "wiki", "alias")))
require.NoError(t, brain.BuildAllWingIndexes(b))
idx, err := os.ReadFile(filepath.Join(b, "wiki", "w", "_index.md"))
require.NoError(t, err)
assert.Contains(t, string(idx), "zqrealwing")
_, err = os.Stat(filepath.Join(b, "raw", "_index.md"))
assert.True(t, os.IsNotExist(err))
}
// A link to a symlinked note does not resolve: only regular files count as
// entries, so StripUnresolvedLinks unwraps it.
func TestStripUnresolvedLinksDoesNotCountASymlinkedNoteAsAnEntry(t *testing.T) {
b := t.TempDir()
facts := filepath.Join(b, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(facts, "real.md"), []byte("x"), 0o644))
brainfstest.WriteSentinelAt(t, filepath.Join(b, "raw", "s.md"))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "..", "raw", "s.md"), filepath.Join(facts, "ghost.md")))
out, unlinked, err := brain.StripUnresolvedLinks(b, "see [[real]] and [[ghost]]")
require.NoError(t, err)
assert.Contains(t, out, "[[real]]", "positive control")
assert.NotContains(t, out, "[[ghost]]")
assert.Contains(t, unlinked, "ghost")
}
// A linked tree (knowledge/ or wiki/) that is not a real directory is
// skipped, not an error: StripUnresolvedLinks runs before every brain_write,
// and an error there loses the learning.
func TestStripUnresolvedLinksSkipsALinkedTreeThatIsNotARealDirectory(t *testing.T) {
for name, mk := range map[string]func(t *testing.T, b string){
"symlink to an in-brain dir": func(t *testing.T, b string) {
require.NoError(t, os.MkdirAll(filepath.Join(b, "raw", "k"), 0o755))
require.NoError(t, os.Symlink(filepath.Join("raw", "k"), filepath.Join(b, "knowledge")))
},
"dangling symlink": func(t *testing.T, b string) {
require.NoError(t, os.Symlink("nowhere", filepath.Join(b, "knowledge")))
},
"regular file": func(t *testing.T, b string) {
require.NoError(t, os.WriteFile(filepath.Join(b, "knowledge"), []byte("x"), 0o644))
},
} {
t.Run(name, func(t *testing.T) {
b := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(b, "wiki", "w", "facts"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(b, "wiki", "w", "facts", "real.md"), []byte("x"), 0o644))
mk(t, b)
out, _, err := brain.StripUnresolvedLinks(b, "learning [[real]] and [[gone]]")
require.NoError(t, err)
assert.Contains(t, out, "[[real]]", "links into wiki/ still resolve")
assert.NotContains(t, out, "[[gone]]")
})
}
}
+32 -17
View File
@@ -1,12 +1,16 @@
package brain package brain
import ( import (
"errors"
"fmt" "fmt"
"os" "io/fs"
"path/filepath" "log/slog"
"path"
"regexp" "regexp"
"sort" "sort"
"strings" "strings"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
) )
// wikilinkRE matches [[target]] and [[target|Display Text]]. // wikilinkRE matches [[target]] and [[target|Display Text]].
@@ -98,29 +102,40 @@ var linkedTrees = []string{"knowledge", "wiki"}
// one the audit will also resolve. // one the audit will also resolve.
func entryIndex(brainDir string) (map[string]struct{}, error) { func entryIndex(brainDir string) (map[string]struct{}, error) {
index := map[string]struct{}{} index := map[string]struct{}{}
// Walked through the brain root (hyperguild#157): only regular files
// count as entries, so a symlinked .md name cannot make a link resolve.
bfs, err := brainfs.Open(brainDir)
if err != nil {
return nil, fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
for _, dir := range linkedTrees { for _, dir := range linkedTrees {
root := filepath.Join(brainDir, dir) if _, err := bfs.Lstat(dir); errors.Is(err, fs.ErrNotExist) {
if _, err := os.Stat(root); os.IsNotExist(err) {
continue continue
} else if err != nil {
return nil, fmt.Errorf("stat %s: %w", dir, err)
} }
err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error { var walkErr error
if err != nil { err := bfs.WalkRegular(dir, func(rel string) error {
return err if !strings.HasSuffix(rel, ".md") {
}
if d.IsDir() || !strings.HasSuffix(path, ".md") {
return nil return nil
} }
rel, err := filepath.Rel(brainDir, path) index[strings.TrimSuffix(path.Base(rel), ".md")] = struct{}{}
if err != nil {
return err
}
rel = filepath.ToSlash(rel)
index[strings.TrimSuffix(filepath.Base(rel), ".md")] = struct{}{}
index[strings.TrimPrefix(strings.TrimSuffix(rel, ".md"), "wiki/")] = struct{}{} index[strings.TrimPrefix(strings.TrimSuffix(rel, ".md"), "wiki/")] = struct{}{}
return nil return nil
}, func(rel string, err error) {
// A linked tree that is not a real directory (a symlink, a
// file) is skipped, as filepath.WalkDir did: this runs before
// every brain_write, and an error here loses the learning.
if rel == dir && errors.Is(err, brainfs.ErrNotRegular) {
slog.Warn("link check: skipping a linked tree that is not a real directory", "dir", dir)
return
}
if walkErr == nil {
walkErr = fmt.Errorf("%s: %w", rel, err)
}
}) })
if err == nil {
err = walkErr
}
if err != nil { if err != nil {
return nil, fmt.Errorf("walk %s: %w", dir, err) return nil, fmt.Errorf("walk %s: %w", dir, err)
} }
+54 -20
View File
@@ -4,9 +4,12 @@ import (
"bufio" "bufio"
"fmt" "fmt"
"os" "os"
"path"
"path/filepath" "path/filepath"
"strings" "strings"
"time" "time"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
) )
// seeAlsoHeader is the markdown heading used to group cross-wing links. // seeAlsoHeader is the markdown heading used to group cross-wing links.
@@ -197,6 +200,12 @@ func fileExists(p string) bool {
// section headers. Rejects same-wing pairs (a tunnel is by definition // section headers. Rejects same-wing pairs (a tunnel is by definition
// cross-wing) and missing notes. // cross-wing) and missing notes.
func WriteTunnel(brainDir, sourcePath, targetPath string) error { func WriteTunnel(brainDir, sourcePath, targetPath string) error {
if err := checkNotePath(sourcePath); err != nil {
return fmt.Errorf("source: %w", err)
}
if err := checkNotePath(targetPath); err != nil {
return fmt.Errorf("target: %w", err)
}
srcWing, err := wingOf(sourcePath) srcWing, err := wingOf(sourcePath)
if err != nil { if err != nil {
return fmt.Errorf("source: %w", err) return fmt.Errorf("source: %w", err)
@@ -209,20 +218,50 @@ func WriteTunnel(brainDir, sourcePath, targetPath string) error {
return fmt.Errorf("tunnel must cross wings; got both in %q", srcWing) return fmt.Errorf("tunnel must cross wings; got both in %q", srcWing)
} }
srcFull := filepath.Join(brainDir, filepath.FromSlash(sourcePath)) // Both notes are read and rewritten through the brain root, regular
tgtFull := filepath.Join(brainDir, filepath.FromSlash(targetPath)) // files only: brain_tunnel passes caller-supplied paths, and the brain is
if _, err := os.Stat(srcFull); err != nil { // a Syncthing folder that can hold symlinks.
bfs, err := brainfs.Open(brainDir)
if err != nil {
return fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
srcContent, err := bfs.ReadRegular(sourcePath)
if err != nil {
return fmt.Errorf("source note: %w", err) return fmt.Errorf("source note: %w", err)
} }
if _, err := os.Stat(tgtFull); err != nil { tgtContent, err := bfs.ReadRegular(targetPath)
if err != nil {
return fmt.Errorf("target note: %w", err) return fmt.Errorf("target note: %w", err)
} }
if err := appendSeeAlso(srcFull, wikilinkOf(targetPath)); err != nil { if out, changed, err := withSeeAlso(srcContent, wikilinkOf(targetPath)); err != nil {
return fmt.Errorf("update source: %w", err) return fmt.Errorf("update source: %w", err)
} else if changed {
if err := bfs.WriteRegular(sourcePath, out); err != nil {
return fmt.Errorf("update source: %w", err)
}
} }
if err := appendSeeAlso(tgtFull, wikilinkOf(sourcePath)); err != nil { if out, changed, err := withSeeAlso(tgtContent, wikilinkOf(sourcePath)); err != nil {
return fmt.Errorf("update target: %w", err) return fmt.Errorf("update target: %w", err)
} else if changed {
if err := bfs.WriteRegular(targetPath, out); err != nil {
return fmt.Errorf("update target: %w", err)
}
}
return nil
}
// checkNotePath accepts only a clean, local wiki/<wing>/<hall>/.../<slug>.md
// path: no "." or ".." segments, no empty segments, no leading slash.
func checkNotePath(p string) error {
if p == "" || path.Clean(p) != p || !filepath.IsLocal(p) || !strings.HasSuffix(p, ".md") {
return fmt.Errorf("not a clean wiki note path: %q", p)
}
for _, part := range strings.Split(p, "/") {
if part == "" || part == "." || part == ".." {
return fmt.Errorf("not a clean wiki note path: %q", p)
}
} }
return nil return nil
} }
@@ -248,16 +287,13 @@ func wingOf(relPath string) (string, error) {
return parts[1], nil return parts[1], nil
} }
// appendSeeAlso inserts `- [[link]]` under the file's See also section, // withSeeAlso returns content with `- [[link]]` under its See also section,
// creating the section if absent. No-op when the link is already present. // creating the section if absent. changed is false when the link is already
func appendSeeAlso(filePath, link string) error { // present.
content, err := os.ReadFile(filePath) func withSeeAlso(content []byte, link string) (out []byte, changed bool, err error) {
if err != nil {
return err
}
wikilink := "[[" + link + "]]" wikilink := "[[" + link + "]]"
if strings.Contains(string(content), wikilink) { if strings.Contains(string(content), wikilink) {
return nil return content, false, nil
} }
bullet := "- " + wikilink bullet := "- " + wikilink
@@ -266,8 +302,7 @@ func appendSeeAlso(filePath, link string) error {
// No section yet — append a fresh one. Always emit a trailing // No section yet — append a fresh one. Always emit a trailing
// newline so subsequent appends don't merge into the previous line. // newline so subsequent appends don't merge into the previous line.
trimmed := strings.TrimRight(string(content), "\n") trimmed := strings.TrimRight(string(content), "\n")
out := trimmed + "\n\n" + seeAlsoHeader + "\n\n" + bullet + "\n" return []byte(trimmed + "\n\n" + seeAlsoHeader + "\n\n" + bullet + "\n"), true, nil
return os.WriteFile(filePath, []byte(out), 0o644)
} }
// Section exists — splice the bullet in just before the next `## ` // Section exists — splice the bullet in just before the next `## `
@@ -293,12 +328,11 @@ func appendSeeAlso(filePath, link string) error {
b.WriteByte('\n') b.WriteByte('\n')
} }
if err := scanner.Err(); err != nil { if err := scanner.Err(); err != nil {
return err return nil, false, err
} }
if !inserted { if !inserted {
// section was the last thing in the file — just append bullet // section was the last thing in the file — just append bullet
out := strings.TrimRight(b.String(), "\n") + "\n" + bullet + "\n" return []byte(strings.TrimRight(b.String(), "\n") + "\n" + bullet + "\n"), true, nil
return os.WriteFile(filePath, []byte(out), 0o644)
} }
return os.WriteFile(filePath, []byte(b.String()), 0o644) return []byte(b.String()), true, nil
} }
@@ -0,0 +1,141 @@
package brain_test
import (
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
)
// brain_tunnel passes caller-supplied source and target paths to WriteTunnel,
// which checked only "wiki/<wing>/..." and then joined them onto the brain
// dir: ".." segments resolved outside the brain and appendSeeAlso rewrote the
// file there.
func TestWriteTunnelRefusesATargetThatTraversesOutOfTheBrain(t *testing.T) {
brain0 := t.TempDir()
src := filepath.Join(brain0, "wiki", "a", "facts", "s.md")
require.NoError(t, os.MkdirAll(filepath.Dir(src), 0o755))
require.NoError(t, os.WriteFile(src, []byte("---\ntitle: S\n---\nbody\n"), 0o644))
victim := filepath.Join(t.TempDir(), "victim.md")
require.NoError(t, os.WriteFile(victim, []byte("untouched\n"), 0o644))
up := strings.Repeat("../", strings.Count(filepath.Clean(brain0), "/")+3)
target := "wiki/b/x/" + up + strings.TrimPrefix(victim, "/")
err := brain.WriteTunnel(brain0, "wiki/a/facts/s.md", target)
assert.Error(t, err)
got, rerr := os.ReadFile(victim)
require.NoError(t, rerr)
assert.Equal(t, "untouched\n", string(got), "a file outside the brain was modified")
}
func TestWriteTunnelRefusesUncleanOrNonWikiPaths(t *testing.T) {
brain0 := t.TempDir()
for _, p := range []string{"wiki/a/facts/s.md", "wiki/b/facts/t.md", "wiki/b/facts/t.txt", "raw/r.md"} {
require.NoError(t, os.MkdirAll(filepath.Dir(filepath.Join(brain0, p)), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(brain0, p), []byte("x\n"), 0o644))
}
for _, target := range []string{
"wiki/b/facts/../../../raw/r.md",
"wiki/b/./facts/t.md",
"wiki/b//facts/t.md",
"/wiki/b/facts/t.md",
"wiki/b/facts/t.txt",
"wiki/b/facts/",
} {
assert.Error(t, brain.WriteTunnel(brain0, "wiki/a/facts/s.md", target), target)
}
for _, p := range []string{"raw/r.md", "wiki/b/facts/t.txt"} {
got, err := os.ReadFile(filepath.Join(brain0, p))
require.NoError(t, err)
assert.Equal(t, "x\n", string(got), p)
}
require.NoError(t, brain.WriteTunnel(brain0, "wiki/a/facts/s.md", "wiki/b/facts/t.md"), "positive control")
got, err := os.ReadFile(filepath.Join(brain0, "wiki", "b", "facts", "t.md"))
require.NoError(t, err)
assert.Contains(t, string(got), "[[a/facts/s]]")
}
func TestWriteTunnelDoesNotWriteThroughASymlinkedNote(t *testing.T) {
brain0 := t.TempDir()
src := filepath.Join(brain0, "wiki", "a", "facts", "s.md")
require.NoError(t, os.MkdirAll(filepath.Dir(src), 0o755))
require.NoError(t, os.WriteFile(src, []byte("x\n"), 0o644))
tdir := filepath.Join(brain0, "wiki", "b", "facts")
require.NoError(t, os.MkdirAll(tdir, 0o755))
victim := filepath.Join(t.TempDir(), "victim.md")
require.NoError(t, os.WriteFile(victim, []byte("untouched\n"), 0o644))
require.NoError(t, os.Symlink(victim, filepath.Join(tdir, "t.md")))
assert.Error(t, brain.WriteTunnel(brain0, "wiki/a/facts/s.md", "wiki/b/facts/t.md"))
got, err := os.ReadFile(victim)
require.NoError(t, err)
assert.Equal(t, "untouched\n", string(got))
// The same with a link to a file elsewhere in the brain, which os.Root
// would follow.
inVictim := filepath.Join(brain0, "raw", "v.md")
require.NoError(t, os.MkdirAll(filepath.Dir(inVictim), 0o755))
require.NoError(t, os.WriteFile(inVictim, []byte("untouched\n"), 0o644))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "..", "raw", "v.md"), filepath.Join(tdir, "u.md")))
assert.Error(t, brain.WriteTunnel(brain0, "wiki/a/facts/s.md", "wiki/b/facts/u.md"))
got, err = os.ReadFile(inVictim)
require.NoError(t, err)
assert.Equal(t, "untouched\n", string(got))
got, err = os.ReadFile(src)
require.NoError(t, err)
assert.Equal(t, "x\n", string(got), "the source is not touched when the target is refused")
}
// The target is swapped between a regular note and a symlink to an in-brain
// file outside wiki/ while WriteTunnel runs: the read and the write each
// refuse a symlink, so the other file is never rewritten.
func TestWriteTunnelNeverRewritesASymlinkSwappedInForTheTarget(t *testing.T) {
brain0 := t.TempDir()
src := filepath.Join(brain0, "wiki", "a", "facts", "s.md")
require.NoError(t, os.MkdirAll(filepath.Dir(src), 0o755))
require.NoError(t, os.WriteFile(src, []byte("x\n"), 0o644))
tdir := filepath.Join(brain0, "wiki", "b", "facts")
require.NoError(t, os.MkdirAll(tdir, 0o755))
victim := filepath.Join(brain0, "raw", "victim.md")
require.NoError(t, os.MkdirAll(filepath.Dir(victim), 0o755))
require.NoError(t, os.WriteFile(victim, []byte("untouched\n"), 0o644))
tgt := filepath.Join(tdir, "t.md")
require.NoError(t, os.WriteFile(tgt, []byte("t\n"), 0o644))
var stop atomic.Bool
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
tmp := tgt + ".swap"
for i := 0; !stop.Load(); i++ {
_ = os.Remove(tmp)
if i%2 == 0 {
_ = os.WriteFile(tmp, []byte("t\n"), 0o644)
} else {
_ = os.Symlink(filepath.Join("..", "..", "..", "raw", "victim.md"), tmp)
}
_ = os.Rename(tmp, tgt)
}
}()
ok := 0
for deadline := time.Now().Add(1500 * time.Millisecond); time.Now().Before(deadline); {
if brain.WriteTunnel(brain0, "wiki/a/facts/s.md", "wiki/b/facts/t.md") == nil {
ok++
}
_ = os.WriteFile(src, []byte("x\n"), 0o644)
}
stop.Store(true)
wg.Wait()
got, err := os.ReadFile(victim)
require.NoError(t, err)
assert.Equal(t, "untouched\n", string(got), "the swapped-in symlink's target was rewritten")
assert.Positive(t, ok, "positive control: some tunnels were written to the regular target")
}
+59
View File
@@ -104,6 +104,27 @@ func (b *Brain) ReadDirRegular(relDir string) ([]string, error) {
return names, nil return names, nil
} }
// ReadDirDirs lists the real subdirectories directly in a brain-relative
// directory, by name. Every component of relDir must be a real directory
// (ErrNotRegular otherwise); symlinks to directories are left out.
func (b *Brain) ReadDirDirs(relDir string) ([]string, error) {
relDir = filepath.ToSlash(relDir)
if err := b.realDir(relDir); err != nil {
return nil, err
}
ents, err := fs.ReadDir(b.root.FS(), relDir)
if err != nil {
return nil, err
}
var names []string
for _, e := range ents {
if e.Type().IsDir() {
names = append(names, e.Name())
}
}
return names, nil
}
// ReadRegular reads a brain-relative path, refusing anything that is not a // ReadRegular reads a brain-relative path, refusing anything that is not a
// regular file, both before the open and on the opened handle. // regular file, both before the open and on the opened handle.
func (b *Brain) ReadRegular(rel string) ([]byte, error) { func (b *Brain) ReadRegular(rel string) ([]byte, error) {
@@ -228,3 +249,41 @@ func (b *Brain) WalkRegular(relDir string, fn func(rel string) error, onErr func
return fn(p) return fn(p)
}) })
} }
// WriteRegular replaces the contents of an existing regular file at a
// brain-relative path. It never creates a file and never writes through a
// final symlink: os.Root.OpenFile follows an in-root final symlink even with
// O_NOFOLLOW (verified on go1.27), so the path is Lstat-ed through the root
// first and the opened handle must be that same file (os.SameFile), which
// also catches a swap between the two. O_NONBLOCK keeps a FIFO from blocking
// the open. Intermediate in-brain symlinks are still followed (os.Root keeps
// the write inside the brain; hyperguild#167).
func (b *Brain) WriteRegular(rel string, data []byte) error {
rel = filepath.ToSlash(rel)
lfi, err := b.root.Lstat(rel)
if err != nil {
return err
}
if !lfi.Mode().IsRegular() {
return fmt.Errorf("%s: %w", rel, ErrNotRegular)
}
f, err := b.root.OpenFile(rel, os.O_WRONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0)
if err != nil {
return err
}
defer func() { _ = f.Close() }()
fi, err := f.Stat()
if err != nil {
return err
}
if !fi.Mode().IsRegular() || !os.SameFile(lfi, fi) {
return fmt.Errorf("%s: changed while opening: %w", rel, ErrNotRegular)
}
if err := f.Truncate(0); err != nil {
return err
}
if _, err := f.Write(data); err != nil {
return err
}
return f.Close()
}
@@ -262,3 +262,67 @@ func TestReadRegularNeverReturnsAnInBrainFileSwappedInBehindASymlink(t *testing.
assert.Zero(t, leaks) assert.Zero(t, leaks)
assert.Positive(t, ok, "positive control: the regular file is read") assert.Positive(t, ok, "positive control: the regular file is read")
} }
func TestWriteRegularRefusesFinalSymlinksFIFOsEscapesAndMissingFilesAndRewritesRegularFiles(t *testing.T) {
dir := t.TempDir()
outside := filepath.Join(t.TempDir(), "o.md")
require.NoError(t, os.WriteFile(outside, []byte("untouched"), 0o644))
require.NoError(t, os.Symlink(outside, filepath.Join(dir, "l.md")))
require.NoError(t, syscall.Mkfifo(filepath.Join(dir, "f.md"), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(dir, "n.md"), []byte("old content"), 0o644))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
assert.Error(t, b.WriteRegular("l.md", []byte("x")))
got, _ := os.ReadFile(outside)
assert.Equal(t, "untouched", string(got))
// A symlink to another file inside the brain: os.Root would allow it,
// only O_NOFOLLOW refuses it.
require.NoError(t, os.WriteFile(filepath.Join(dir, "in.md"), []byte("inside"), 0o644))
require.NoError(t, os.Symlink("in.md", filepath.Join(dir, "il.md")))
assert.Error(t, b.WriteRegular("il.md", []byte("x")))
got, _ = os.ReadFile(filepath.Join(dir, "in.md"))
assert.Equal(t, "inside", string(got))
// A reader held open, so a non-blocking write open would succeed. The
// Lstat type check refuses the FIFO first; the handle check and
// O_NONBLOCK matter only if it is swapped in after the Lstat.
r, err := os.OpenFile(filepath.Join(dir, "f.md"), os.O_RDONLY|syscall.O_NONBLOCK, 0)
require.NoError(t, err)
defer func() { _ = r.Close() }()
done := make(chan error, 1)
go func() { done <- b.WriteRegular("f.md", []byte("x")) }()
select {
case err := <-done:
assert.Error(t, err)
case <-time.After(3 * time.Second):
t.Fatal("WriteRegular hung on a FIFO")
}
assert.Error(t, b.WriteRegular("../"+filepath.Base(filepath.Dir(outside))+"/o.md", []byte("x")))
assert.Error(t, b.WriteRegular("missing.md", []byte("x")), "does not create files")
require.NoError(t, b.WriteRegular("n.md", []byte("new")))
got, _ = os.ReadFile(filepath.Join(dir, "n.md"))
assert.Equal(t, "new", string(got), "truncated and rewritten")
}
func TestReadDirDirsListsOnlyRealSubdirsAndRefusesSymlinkedComponents(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "a", "real"), 0o755))
require.NoError(t, os.MkdirAll(filepath.Join(dir, "other"), 0o755))
require.NoError(t, os.Symlink(filepath.Join("..", "other"), filepath.Join(dir, "a", "linked")))
require.NoError(t, os.WriteFile(filepath.Join(dir, "a", "f.md"), []byte("f"), 0o644))
require.NoError(t, os.Symlink("a", filepath.Join(dir, "alias")))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
names, err := b.ReadDirDirs("a")
require.NoError(t, err)
assert.Equal(t, []string{"real"}, names)
_, err = b.ReadDirDirs("alias")
assert.ErrorIs(t, err, ErrNotRegular)
}
@@ -27,7 +27,7 @@ const outsideContent = "---\ntitle: " + Sentinel + "\naliases:\n - " + Sentinel
"\n\n" + Sentinel + " [[" + Sentinel + "]]\n" "\n\n" + Sentinel + " [[" + Sentinel + "]]\n"
// Outside writes a file outside any brain dir containing Sentinel. // Outside writes a file outside any brain dir containing Sentinel.
func Outside(t *testing.T) string { func Outside(t testing.TB) string {
t.Helper() t.Helper()
p := filepath.Join(t.TempDir(), "secret.md") p := filepath.Join(t.TempDir(), "secret.md")
if err := os.WriteFile(p, []byte(outsideContent), 0o644); err != nil { if err := os.WriteFile(p, []byte(outsideContent), 0o644); err != nil {
@@ -42,9 +42,12 @@ var swapSeq atomic.Int64
// a symlink to outside or a FIFO into path, until the test ends. FIFOs get a // a symlink to outside or a FIFO into path, until the test ends. FIFOs get a
// writer held open until the swapper exits, so a reader that skips the // writer held open until the swapper exits, so a reader that skips the
// handle check parks forever instead of reading EOF. The swapper is stopped // handle check parks forever instead of reading EOF. The swapper is stopped
// and awaited before t.TempDir cleanup. // and awaited before t.TempDir cleanup. It returns the number of symlinks
func StartSwap(t *testing.T, path, inside, outside string, fifo bool) { // or FIFOs actually renamed into path so far, so a caller can tell a race
// that ran from one whose every swap failed (hyperguild#177).
func StartSwap(t testing.TB, path, inside, outside string, fifo bool) *atomic.Int64 {
t.Helper() t.Helper()
var swaps atomic.Int64
var stop atomic.Bool var stop atomic.Bool
var wg sync.WaitGroup var wg sync.WaitGroup
wg.Add(1) wg.Add(1)
@@ -67,22 +70,26 @@ func StartSwap(t *testing.T, path, inside, outside string, fifo bool) {
}() }()
for i := 0; !stop.Load(); i++ { for i := 0; !stop.Load(); i++ {
_ = os.Remove(tmp) _ = os.Remove(tmp)
var made error
switch { switch {
case i%2 == 0: case i%2 == 0:
_ = os.WriteFile(tmp, []byte(inside), 0o644) _ = os.WriteFile(tmp, []byte(inside), 0o644)
case fifo: case fifo:
if err := syscall.Mkfifo(tmp, 0o644); err == nil { if made = syscall.Mkfifo(tmp, 0o644); made == nil {
if w, err := os.OpenFile(tmp, os.O_RDWR, 0); err == nil { if w, err := os.OpenFile(tmp, os.O_RDWR, 0); err == nil {
writers = append(writers, w) writers = append(writers, w)
} }
} }
default: default:
_ = os.Symlink(link, tmp) made = os.Symlink(link, tmp)
}
if err := os.Rename(tmp, path); err == nil && made == nil && i%2 == 1 {
swaps.Add(1)
} }
_ = os.Rename(tmp, path)
} }
}() }()
t.Cleanup(func() { stop.Store(true); wg.Wait() }) t.Cleanup(func() { stop.Store(true); wg.Wait() })
return &swaps
} }
// WriteSentinelAt writes a file carrying Sentinel at p (creating its // WriteSentinelAt writes a file carrying Sentinel at p (creating its
@@ -107,18 +114,41 @@ func AssertNoLeak(t *testing.T, path, inside string, d time.Duration, read func(
AssertNoLeakFrom(t, path, inside, Outside(t), d, read) AssertNoLeakFrom(t, path, inside, Outside(t), d, read)
} }
// InsideMarker must appear in the inside note's content and in read's
// output whenever read saw that note: the floor below counts such reads.
const InsideMarker = "inside"
// AssertNoLeakFrom is AssertNoLeak with the symlink pointing at target, e.g. // AssertNoLeakFrom is AssertNoLeak with the symlink pointing at target, e.g.
// a file elsewhere in the brain written with WriteSentinelAt. // a file elsewhere in the brain written with WriteSentinelAt.
func AssertNoLeakFrom(t *testing.T, path, inside, target string, d time.Duration, read func() string) { //
// Beyond "no Sentinel in any output" it sets two floors, because without
// them the assertion passes when nothing was tested (hyperguild#177): at
// least one symlink must have been swapped into path, and at least one read
// must have returned InsideMarker (a reader that always errors returns
// nothing, and so never leaks).
func AssertNoLeakFrom(t testing.TB, path, inside, target string, d time.Duration, read func() string) {
t.Helper() t.Helper()
StartSwap(t, path, inside, target, false) if !strings.Contains(inside, InsideMarker) {
leaks, iters := 0, 0 t.Fatalf("inside content must contain %q", InsideMarker)
}
swaps := StartSwap(t, path, inside, target, false)
leaks, iters, insideReads := 0, 0, 0
for deadline := time.Now().Add(d); time.Now().Before(deadline); iters++ { for deadline := time.Now().Add(d); time.Now().Before(deadline); iters++ {
leaks += strings.Count(read(), Sentinel) out := read()
leaks += strings.Count(out, Sentinel)
if strings.Contains(out, InsideMarker) {
insideReads++
}
} }
if leaks > 0 { if leaks > 0 {
t.Fatalf("outside file leaked %d times in %d iterations", leaks, iters) t.Fatalf("outside file leaked %d times in %d iterations", leaks, iters)
} }
if n := swaps.Load(); n == 0 {
t.Fatalf("no symlink was swapped into %s in %d iterations: the race did not run", path, iters)
}
if insideReads == 0 {
t.Fatalf("no read in %d iterations returned the inside note (%q): the reader read nothing", iters, InsideMarker)
}
} }
// AssertNoHang loops read for d while a FIFO swapper (held writers) runs on // AssertNoHang loops read for d while a FIFO swapper (held writers) runs on
@@ -0,0 +1,79 @@
package brainfstest
import (
"fmt"
"os"
"path/filepath"
"runtime"
"syscall"
"testing"
"time"
)
// recordingTB stands in for the test under AssertNoLeakFrom so a fatal
// there can be observed instead of failing this test. Fatalf ends the
// calling goroutine the way testing.T does.
type recordingTB struct {
testing.TB
failed bool
msg string
}
func (r *recordingTB) Fatalf(format string, args ...any) {
r.failed, r.msg = true, fmt.Sprintf(format, args...)
runtime.Goexit()
}
func runAssert(t *testing.T, path, target string, read func() string) *recordingTB {
t.Helper()
r := &recordingTB{TB: t}
done := make(chan struct{})
go func() {
defer close(done)
AssertNoLeakFrom(r, path, "inside", target, 200*time.Millisecond, read)
}()
<-done
return r
}
// hyperguild#177: with every swap failing, no leak can occur, so the leak
// assertion passed without testing anything.
func TestAssertNoLeakFromFailsWhenNoSymlinkSwapSucceeds(t *testing.T) {
path := filepath.Join(t.TempDir(), "missing-dir", "r.md")
r := runAssert(t, path, Outside(t), func() string { return "inside" })
if !r.failed {
t.Fatal("AssertNoLeakFrom passed with zero successful symlink swaps")
}
}
// hyperguild#177: a reader that always errors returns nothing, so it never
// leaks either.
func TestAssertNoLeakFromFailsWhenNoReadReturnsTheInsideNote(t *testing.T) {
path := filepath.Join(t.TempDir(), "r.md")
r := runAssert(t, path, Outside(t), func() string { return "" })
if !r.failed {
t.Fatal("AssertNoLeakFrom passed with zero reads returning the inside note")
}
}
// Positive control: a reader that refuses symlinks and reads regular files
// passes, so the floors do not fail a working reader.
func TestAssertNoLeakFromPassesAReaderThatRefusesSymlinks(t *testing.T) {
path := filepath.Join(t.TempDir(), "r.md")
read := func() string {
f, err := os.OpenFile(path, os.O_RDONLY|noFollow, 0)
if err != nil {
return ""
}
defer func() { _ = f.Close() }()
b := make([]byte, 64)
n, _ := f.Read(b)
return string(b[:n])
}
r := runAssert(t, path, Outside(t), read)
if r.failed {
t.Fatalf("a symlink-refusing reader failed: %s", r.msg)
}
}
const noFollow = syscall.O_NOFOLLOW
+26 -9
View File
@@ -93,16 +93,20 @@ func TestQueryDoesNotFollowASymlinkedWingUnderAWingAndHallRoot(t *testing.T) {
// bypass of readOpened hung Query after 5 iterations). // bypass of readOpened hung Query after 5 iterations).
// startSwap runs swapInto until the test ends and waits for it before // startSwap runs swapInto until the test ends and waits for it before
// t.TempDir's cleanup (hyperguild#162). // t.TempDir's cleanup (hyperguild#162).
func startSwap(t *testing.T, path, outside string, fifo bool) { // It returns the count of symlinks or FIFOs actually renamed into path
// (hyperguild#177: a swapper whose every swap fails tests nothing).
func startSwap(t *testing.T, path, outside string, fifo bool) *atomic.Int64 {
t.Helper() t.Helper()
var stop atomic.Bool var stop atomic.Bool
var swaps atomic.Int64
var wg sync.WaitGroup var wg sync.WaitGroup
wg.Add(1) wg.Add(1)
go func() { defer wg.Done(); swapInto(path, outside, fifo, &stop) }() go func() { defer wg.Done(); swapInto(path, outside, fifo, &stop, &swaps) }()
t.Cleanup(func() { stop.Store(true); wg.Wait() }) t.Cleanup(func() { stop.Store(true); wg.Wait() })
return &swaps
} }
func swapInto(path, outside string, fifo bool, stop *atomic.Bool) { func swapInto(path, outside string, fifo bool, stop *atomic.Bool, swaps *atomic.Int64) {
tmp := path + ".swap" tmp := path + ".swap"
// Writers stay open until the swapper exits, so a reader parked on a // Writers stay open until the swapper exits, so a reader parked on a
// FIFO stays parked (closing one would hand the reader EOF and hide the // FIFO stays parked (closing one would hand the reader EOF and hide the
@@ -115,33 +119,46 @@ func swapInto(path, outside string, fifo bool, stop *atomic.Bool) {
}() }()
for i := 0; !stop.Load(); i++ { for i := 0; !stop.Load(); i++ {
_ = os.Remove(tmp) _ = os.Remove(tmp)
var made error
switch { switch {
case i%2 == 0: case i%2 == 0:
_ = os.WriteFile(tmp, []byte("# r\n\nzqinside "+"zqracenote"), 0o644) _ = os.WriteFile(tmp, []byte("# r\n\nzqinside "+"zqracenote"), 0o644)
case fifo: case fifo:
if err := syscall.Mkfifo(tmp, 0o644); err == nil { if made = syscall.Mkfifo(tmp, 0o644); made == nil {
if w, err := os.OpenFile(tmp, os.O_RDWR, 0); err == nil { if w, err := os.OpenFile(tmp, os.O_RDWR, 0); err == nil {
writers = append(writers, w) writers = append(writers, w)
} }
} }
default: default:
_ = os.Symlink(outside, tmp) made = os.Symlink(outside, tmp)
}
if err := os.Rename(tmp, path); err == nil && made == nil && i%2 == 1 {
swaps.Add(1)
} }
_ = os.Rename(tmp, path)
} }
} }
func TestQueryLeaksNothingUnderASymlinkSwapRace(t *testing.T) { func TestQueryLeaksNothingUnderASymlinkSwapRace(t *testing.T) {
brain, facts, outside := searchBrain(t) brain, facts, outside := searchBrain(t)
startSwap(t, filepath.Join(facts, "r.md"), outside, false) swaps := startSwap(t, filepath.Join(facts, "r.md"), outside, false)
leaks, iters := 0, 0 leaks, iters, insideReads := 0, 0, 0
for deadline := time.Now().Add(1500 * time.Millisecond); time.Now().Before(deadline); iters++ { for deadline := time.Now().Add(1500 * time.Millisecond); time.Now().Before(deadline); iters++ {
rs, _ := Query(brain, QueryOptions{Query: sentinel157, Limit: 10}) rs, err := Query(brain, QueryOptions{Query: sentinel157, Limit: 10})
require.NoError(t, err)
leaks += leaked(rs) leaks += leaked(rs)
// Positive control (hyperguild#177): a Query that always errors
// returns nothing and so never leaks.
rs, err = Query(brain, QueryOptions{Query: "zqracenote", Limit: 10})
require.NoError(t, err)
if len(rs) > 0 {
insideReads++
}
} }
if leaks > 0 { if leaks > 0 {
t.Fatalf("outside file leaked %d times in %d iterations", leaks, iters) t.Fatalf("outside file leaked %d times in %d iterations", leaks, iters)
} }
require.Positive(t, swaps.Load(), "no symlink was swapped in: the race did not run")
require.Positive(t, insideReads, "no Query returned the inside note")
} }
func TestQueryDoesNotHangUnderAFIFOSwapRace(t *testing.T) { func TestQueryDoesNotHangUnderAFIFOSwapRace(t *testing.T) {
+92
View File
@@ -0,0 +1,92 @@
package session
import (
"errors"
"os"
"path/filepath"
"syscall"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// session_log passed the caller's session_id straight into
// filepath.Join(sessionsDir, id+".jsonl") and opened it O_CREATE|O_APPEND:
// path segments in the id created or appended files anywhere the pod could
// write. The id is now a plain name, and the open does not follow a final
// symlink.
func TestAppendRefusesSessionIDsThatAreNotPlainNames(t *testing.T) {
root := t.TempDir()
dir := filepath.Join(root, "sessions")
for _, id := range []string{"../x", "a/b", "/abs", ".", "..", ".hidden", "", "a\\b", "a\x00b"} {
assert.Error(t, Append(dir, id, Entry{SessionID: id}), "%q", id)
}
ents, _ := os.ReadDir(root)
for _, e := range ents {
assert.Equal(t, "sessions", e.Name(), "nothing created beside the sessions dir")
}
for _, id := range []string{"1557b2e1-bf0d-5c4f-bbe3-0754d087e356", "_routing", "2026-05-05-infra.plan", "0"} {
require.NoError(t, Append(dir, id, Entry{SessionID: id}), "positive control: %q", id)
}
}
func TestAppendDoesNotWriteThroughASymlinkedLog(t *testing.T) {
root := t.TempDir()
dir := filepath.Join(root, "sessions")
require.NoError(t, os.MkdirAll(dir, 0o755))
victim := filepath.Join(root, "victim.md")
require.NoError(t, os.WriteFile(victim, []byte("untouched\n"), 0o644))
require.NoError(t, os.Symlink(filepath.Join("..", "victim.md"), filepath.Join(dir, "s.jsonl")))
assert.Error(t, Append(dir, "s", Entry{SessionID: "s"}))
got, _ := os.ReadFile(victim)
assert.Equal(t, "untouched\n", string(got))
}
func TestReadRefusesSessionIDsThatAreNotPlainNames(t *testing.T) {
root := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(root, "x.jsonl"), []byte(`{"session_id":"x"}`+"\n"), 0o644))
_, err := Read(filepath.Join(root, "sessions"), "../x")
assert.Error(t, err)
}
// A name that starts plainly and then climbs out with ".." is exactly what the
// ban on "/" in the id's tail stops: filepath.Join cleans "a/../../x" to a
// path beside the sessions dir without needing sessions/a to exist. The
// assertion is on the refusal itself (ErrInvalidID), not on any error.
func TestAppendRefusesAnIDThatClimbsOutAfterAPlainPrefix(t *testing.T) {
root := t.TempDir()
dir := filepath.Join(root, "sessions")
require.NoError(t, os.MkdirAll(dir, 0o755))
for _, id := range []string{"a/../../escape", "a/../../../escape", "a/../b"} {
err := Append(dir, id, Entry{SessionID: id})
assert.ErrorIs(t, err, ErrInvalidID, "%q", id)
}
_, err := os.Lstat(filepath.Join(root, "escape.jsonl"))
assert.True(t, errors.Is(err, os.ErrNotExist), "nothing created beside the sessions dir")
_, err = os.Lstat(filepath.Join(dir, "b.jsonl"))
assert.True(t, errors.Is(err, os.ErrNotExist), "no in-dir alias created either")
}
// Append does no Lstat type check, so O_NONBLOCK is the only thing that keeps
// a FIFO named <id>.jsonl with no reader from blocking session_log forever.
func TestAppendReturnsPromptlyOnAFIFOLogWithNoReader(t *testing.T) {
dir := filepath.Join(t.TempDir(), "sessions")
require.NoError(t, os.MkdirAll(dir, 0o755))
fifo := filepath.Join(dir, "s.jsonl")
require.NoError(t, syscall.Mkfifo(fifo, 0o644))
done := make(chan error, 1)
go func() { done <- Append(dir, "s", Entry{SessionID: "s"}) }()
select {
case err := <-done:
assert.Error(t, err, "a FIFO with no reader is refused, not written")
case <-time.After(3 * time.Second):
// Unblock the stuck open so the goroutine and TempDir cleanup finish.
if r, err := os.OpenFile(fifo, os.O_RDONLY|syscall.O_NONBLOCK, 0); err == nil {
defer r.Close() //nolint:errcheck
}
<-done
t.Fatal("Append blocked on a FIFO log with no reader")
}
}
+25 -1
View File
@@ -9,6 +9,8 @@ import (
"io/fs" "io/fs"
"os" "os"
"path/filepath" "path/filepath"
"regexp"
"syscall"
"time" "time"
) )
@@ -42,13 +44,32 @@ type Attempt struct {
RunnerOutput string `json:"runner_output,omitempty"` RunnerOutput string `json:"runner_output,omitempty"`
} }
// idPattern is a plain file name: session_log takes the id from the caller,
// and a path segment in it created or appended files anywhere the pod could
// write (hyperguild#169).
var idPattern = regexp.MustCompile(`^[A-Za-z0-9_][A-Za-z0-9._-]{0,127}$`)
// ErrInvalidID is returned when a session id is not a plain file name.
var ErrInvalidID = errors.New("session id is not a plain name")
func checkID(sessionID string) error {
if !idPattern.MatchString(sessionID) {
return fmt.Errorf("%w: %q", ErrInvalidID, sessionID)
}
return nil
}
// Append writes entry as a single JSON line to sessionsDir/{sessionID}.jsonl. // Append writes entry as a single JSON line to sessionsDir/{sessionID}.jsonl.
// The open does not follow a final symlink.
func Append(sessionsDir, sessionID string, entry Entry) error { func Append(sessionsDir, sessionID string, entry Entry) error {
if err := checkID(sessionID); err != nil {
return err
}
if err := os.MkdirAll(sessionsDir, 0o755); err != nil { if err := os.MkdirAll(sessionsDir, 0o755); err != nil {
return fmt.Errorf("create sessions dir: %w", err) return fmt.Errorf("create sessions dir: %w", err)
} }
path := filepath.Join(sessionsDir, sessionID+".jsonl") path := filepath.Join(sessionsDir, sessionID+".jsonl")
f, err := os.OpenFile(path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o644) f, err := os.OpenFile(path, os.O_CREATE|os.O_APPEND|os.O_WRONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0o644)
if err != nil { if err != nil {
return fmt.Errorf("open session log: %w", err) return fmt.Errorf("open session log: %w", err)
} }
@@ -70,6 +91,9 @@ func Append(sessionsDir, sessionID string, entry Entry) error {
// Read returns all entries for sessionID. Returns empty slice if no log exists. // Read returns all entries for sessionID. Returns empty slice if no log exists.
func Read(sessionsDir, sessionID string) ([]Entry, error) { func Read(sessionsDir, sessionID string) ([]Entry, error) {
if err := checkID(sessionID); err != nil {
return nil, err
}
path := filepath.Join(sessionsDir, sessionID+".jsonl") path := filepath.Join(sessionsDir, sessionID+".jsonl")
f, err := os.Open(path) f, err := os.Open(path)
if errors.Is(err, fs.ErrNotExist) { if errors.Is(err, fs.ErrNotExist) {