Compare commits

..
Author SHA1 Message Date
mathiasandClaude Opus 5.5 ac84ddf5a9 test: pin CreateRegular directly and promote/reject with in-brain targets (#188)
CI / Lint / Test / Vet (pull_request) Successful in 1m3s
CI / Mirror to GitHub (pull_request) Skipped
Review round 2 on PR #189: CreateRegular (promote's and reject's only
write path) had no test; replacing mkdirReal with root.MkdirAll, dropping
cleanRel or calling PutRegular all stayed green, and the promote/reject
test planted only outside targets, which os.Root refuses on its own.

- TestCreateRegularCreatesOnlyNewFilesThroughRealDirectories: in-brain
  symlinked parent, unclean path, existing file and dangling link
  (fs.ErrExist, nothing changed), positive control with missing parents.
- TestPromoteAndRejectRefuseAnInBrainSymlinkedDestinationDirectory.

All three flips are red now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 23:19:02 +02:00
mathiasandClaude Opus 5.5 db18785bfc test: pin the edit symlink refusal, AppendRegular's clean check and Remove (#188)
CI / Lint / Test / Vet (pull_request) Failing after 1m6s
CI / Mirror to GitHub (pull_request) Skipped
Review round 1 on PR #189 (three tests stayed green with their guard
removed):
- TestEditRefusesANoteBehindASymlinkedDirectory had no .git, so Edit
  refused "not the sync source" before any path check. It now has one,
  plus a positive control.
- AppendRegular's unclean-path case was refused by a final symlink, not
  cleanRel. New test where ".." after a symlinked dir resolves to a real
  file, for AppendRegular and PutRegular.
- brainfs.Remove had no test; new one (refuses behind a symlinked dir,
  removes a final link but never its target, positive control).

Also: promote/reject create with O_EXCL (new brainfs.CreateRegular), so a
note that appears after the collision check is never overwritten;
gofmt fixes in provenance.go and wiki/log.go; dead wingDir and
putBrainFile removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 22:57:26 +02:00
mathiasandClaude Opus 5.5 138d6e2545 fix: generated and promoted writes go through the brain root (#188, #158)
CI / Lint / Test / Vet (pull_request) Successful in 1m17s
CI / Mirror to GitHub (pull_request) Skipped
Writers that created or rewrote brain files by path followed symlinks:
- promote/reject: os.MkdirAll + os.WriteFile of the destination, then
  os.Remove of the raw source. Now PutRegular + brainfs.Remove; the source
  is kept when the write is refused.
- StampProvenanceFile (brain_edit's re-stamp): provenance.ApplyFile read
  and rewrote by path. Now ReadRegular + WriteRegular; ApplyFile, now
  unused, is removed.
- BuildWingIndex (_index.md) and wiki.RebuildIndex (wiki/index.md): now
  PutRegular.
- wiki.AppendLog (log.md) and AutoTunnel's fuzzy-candidate log
  (raw/tunnel-candidates-*.md): now AppendRegular; tunnel's fileExists,
  now unused, is removed.
- New brainfs.Remove: clean path, real parent, removes a final symlink
  itself, never its target.

brain_edit's own atomic rename was already refused behind a symlinked
directory by #187's read check; a guard test pins it.

Pipeline writes, BackfillRefs, the watcher's raw/ markers and the session
archive in cmd/server follow in a separate PR (#158).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 22:39:41 +02:00
mathiasandClaude Opus 5.5 41b7105b31 fix(brainfs,api): PutRegular/AppendRegular; brain_write creates through the brain root (#188)
brain_write created a new note with os.MkdirAll + os.WriteFile by path,
so a symlinked wing or hall directory redirected the note elsewhere in
the brain or outside it.

New brainfs primitives for creating writes:
- PutRegular: create or replace. Missing parents are created one
  component at a time through the root and every existing component
  must be a real directory; an existing file is replaced only via
  WriteRegular's checks; a new one is created with O_EXCL and confirmed
  (Lstat + SameFile) before writing.
- AppendRegular: create-or-append with the same parent rule, writing
  only after confirming the path is the regular file it opened.
- RealParent: the clean-path + real-directory check, for writers that do
  their own atomic rename.

writeHallNote now uses PutRegular.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 22:35:19 +02:00
13 changed files with 54 additions and 412 deletions
+4 -9
View File
@@ -17,7 +17,6 @@ import (
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
"github.com/mathiasbq/hyperguild/ingestion/internal/audit"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
"github.com/mathiasbq/hyperguild/ingestion/internal/capturehttp"
"github.com/mathiasbq/hyperguild/ingestion/internal/classification"
@@ -90,15 +89,11 @@ func (s *claudeSink) Ingest(ctx context.Context, b claudewatcher.Batch) error {
sb.WriteString("\n\n")
}
slug := "session-" + b.Host + "-" + b.SessionID
dest := "archive/claude-sessions/" + b.Host + "/" + slug + ".md"
// Written through the brain root, never through a symlinked directory
// or file (hyperguild#158).
bfs, err := brainfs.Open(s.brainDir)
if err != nil {
return fmt.Errorf("open brain: %w", err)
dest := filepath.Join(s.brainDir, "archive", "claude-sessions", b.Host, slug+".md")
if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
return fmt.Errorf("create claude-sessions archive dir: %w", err)
}
defer func() { _ = bfs.Close() }()
if err := bfs.PutRegular(dest, []byte(sb.String())); err != nil {
if err := os.WriteFile(dest, []byte(sb.String()), 0o644); err != nil {
return fmt.Errorf("write claude session archive: %w", err)
}
s.logger.Debug("claude session archived (non-indexed)", "path", dest)
@@ -4,7 +4,6 @@ import (
"os"
"path/filepath"
"strings"
"syscall"
"testing"
"github.com/stretchr/testify/assert"
@@ -64,13 +63,3 @@ func TestStartupSymlinkCheckRefusesAnUnlistableBrainRoot(t *testing.T) {
t.Cleanup(func() { _ = os.Chmod(brain, 0o755) })
assert.Error(t, startupSymlinkCheck(brain))
}
// hyperguild#179: a FIFO in the brain is refused at startup with a clear
// message, instead of hanging or exiting later in a reader.
func TestStartupSymlinkCheckRefusesABrainWithAFIFO(t *testing.T) {
brain := t.TempDir()
require.NoError(t, syscall.Mkfifo(filepath.Join(brain, "classification.yaml"), 0o644))
err := startupSymlinkCheck(brain)
require.Error(t, err)
assert.Contains(t, err.Error(), "classification.yaml")
}
@@ -1,34 +0,0 @@
package api_test
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
)
// hyperguild#178: an unreadable directory under wiki/ made the link check
// fail and brain_write refuse the note, losing the learning. The note is now
// written with its links as given.
func TestWriteNoteKeepsTheLearningWhenTheLinkCheckCannotIndex(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root ignores directory permissions")
}
brain := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki", "w", "facts"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(brain, "wiki", "w", "facts", "real.md"), []byte("x"), 0o644))
locked := filepath.Join(brain, "wiki", "other", "locked")
require.NoError(t, os.MkdirAll(locked, 0o755))
require.NoError(t, os.Chmod(locked, 0o000))
t.Cleanup(func() { _ = os.Chmod(locked, 0o755) })
rel, err := api.WriteNote(brain, api.WriteNoteOptions{Wing: "w", Hall: "facts", Filename: "n", Content: "learning [[real]] and [[gone]]"})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(brain, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.Contains(t, string(got), "[[gone]]", "links kept as given when the index is partial")
}
+1 -6
View File
@@ -47,12 +47,7 @@ var (
func StripUnresolvedLinks(brainDir, content string) (string, []string, error) {
index, err := entryIndex(brainDir)
if err != nil {
// A partial index cannot tell an invented link from one into the
// part it could not read, so unwrap nothing, and do not refuse the
// write either: this runs before every brain_write, and a refused
// write is a lost learning (hyperguild#178).
slog.Warn("link check skipped: cannot index the brain", "err", err)
return content, nil, nil
return "", nil, err
}
if len(index) == 0 {
return content, nil, nil
@@ -1,41 +0,0 @@
package brain_test
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
)
// hyperguild#178: StripUnresolvedLinks runs before every brain_write and
// brain_update. An unreadable directory under knowledge/ or wiki/ made it
// error, the write was refused and the learning lost. When the link index
// cannot be built, the content is now returned unchanged (no link is
// unwrapped, none is invented) and no error stops the write.
func TestStripUnresolvedLinksKeepsContentWhenTheIndexCannotBeBuilt(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root ignores directory permissions")
}
b := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(b, "wiki", "w", "facts"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(b, "wiki", "w", "facts", "real.md"), []byte("x"), 0o644))
locked := filepath.Join(b, "wiki", "other", "locked")
require.NoError(t, os.MkdirAll(locked, 0o755))
require.NoError(t, os.Chmod(locked, 0o000))
t.Cleanup(func() { _ = os.Chmod(locked, 0o755) })
in := "learning [[real]] and [[gone]]"
out, unlinked, err := brain.StripUnresolvedLinks(b, in)
require.NoError(t, err)
assert.Equal(t, in, out, "content unchanged: no link unwrapped on a partial index")
assert.Empty(t, unlinked)
require.NoError(t, os.Chmod(locked, 0o755))
out, _, err = brain.StripUnresolvedLinks(b, in)
require.NoError(t, err)
assert.NotContains(t, out, "[[gone]]", "positive control: with a readable tree the strip runs")
}
-21
View File
@@ -243,27 +243,6 @@ func (b *Brain) RealParent(rel string) error {
return b.realDir(path.Dir(rel))
}
// PutFile opens the brain dir as a root and calls PutRegular, for one-off
// writers that hold no Brain.
func PutFile(brainDir, rel string, data []byte) error {
b, err := Open(brainDir)
if err != nil {
return err
}
defer func() { _ = b.Close() }()
return b.PutRegular(rel, data)
}
// AppendFile opens the brain dir as a root and calls AppendRegular.
func AppendFile(brainDir, rel string, data []byte) error {
b, err := Open(brainDir)
if err != nil {
return err
}
defer func() { _ = b.Close() }()
return b.AppendRegular(rel, data)
}
// mkdirReal makes sure every component of rel exists as a real directory,
// creating missing ones through the root; an existing component that is
// not a real directory is ErrNotRegular.
+4 -1
View File
@@ -7,6 +7,8 @@ import (
"fmt"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
"io/fs"
"os"
"path/filepath"
"strings"
"github.com/mathiasbq/hyperguild/ingestion/internal/wiki"
@@ -88,7 +90,8 @@ func BackfillRefs(ctx context.Context, brainDir string) (int, error) {
}
for relPath, page := range pending {
if err := bfs.PutRegular(relPath, []byte(page.Content)); err != nil {
dest := filepath.Join(brainDir, filepath.FromSlash(relPath))
if err := os.WriteFile(dest, []byte(page.Content), 0o644); err != nil {
return 0, fmt.Errorf("write %s: %w", relPath, err)
}
}
+7 -14
View File
@@ -9,6 +9,8 @@ import (
"fmt"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
"io/fs"
"os"
"path/filepath"
"strings"
"time"
@@ -126,6 +128,10 @@ func buildAndWrite(ctx context.Context, rawPages []RawPage, sourceSlug, date, br
var written []string
for _, page := range merged {
if !dryRun {
dest := filepath.Join(brainDir, filepath.FromSlash(page.Path))
if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
return Result{}, fmt.Errorf("mkdir for %s: %w", page.Path, err)
}
content, conflicts, err := reviseOrCreate(brainDir, page)
if err != nil {
return Result{}, err
@@ -144,9 +150,7 @@ func buildAndWrite(ctx context.Context, rawPages []RawPage, sourceSlug, date, br
if err != nil {
return Result{}, fmt.Errorf("stamp %s: %w", page.Path, err)
}
// Written through the brain root, never through a symlinked
// directory or page (hyperguild#158).
if err := writeBrainFile(brainDir, page.Path, []byte(stamped)); err != nil {
if err := os.WriteFile(dest, []byte(stamped), 0o644); err != nil {
return Result{}, fmt.Errorf("write %s: %w", page.Path, err)
}
}
@@ -197,17 +201,6 @@ func readSourcePage(brainDir, sourceSlug string) string {
return string(raw)
}
// writeBrainFile creates or replaces one brain-relative file through the
// brain root (hyperguild#158).
func writeBrainFile(brainDir, rel string, data []byte) error {
bfs, err := brainfs.Open(brainDir)
if err != nil {
return err
}
defer func() { _ = bfs.Close() }()
return bfs.PutRegular(rel, data)
}
// readBrainFile reads one brain-relative file through the brain root,
// regular files only (hyperguild#157).
func readBrainFile(brainDir, rel string) ([]byte, error) {
-25
View File
@@ -165,28 +165,3 @@ func TestBackfillRefsDoesNotCopyAPageSwappedInBehindASymlink(t *testing.T) {
require.Contains(t, read(), "[[s|S]]", "positive control: the regular page gets the source ref")
brainfstest.AssertNoLeakFrom(t, page, inside, target, 3*time.Second, read)
}
// hyperguild#158: BackfillRefs read a regular concept page, a symlink could
// be swapped in, and os.WriteFile then followed it and overwrote the link's
// target. The target is not restored between rounds here, so any
// write-through shows.
func TestBackfillRefsNeverWritesThroughAPageSwappedForASymlink(t *testing.T) {
brain := t.TempDir()
mkdirs(t, brain, "wiki/sources", "wiki/concepts", "raw")
require.NoError(t, os.WriteFile(filepath.Join(brain, "wiki", "sources", "s.md"), []byte("---\ntitle: S\n---\n[[c|C]]\n"), 0o644))
page := filepath.Join(brain, "wiki", "concepts", "c.md")
target := filepath.Join(brain, "raw", "secret.md")
require.NoError(t, os.WriteFile(target, []byte("untouched"), 0o644))
inside := "---\ntitle: C\n---\nbody\n"
require.NoError(t, os.WriteFile(page, []byte(inside), 0o644))
brainfstest.StartSwap(t, page, inside, target, false)
ok := 0
for deadline := time.Now().Add(2 * time.Second); time.Now().Before(deadline); {
if _, err := BackfillRefs(context.Background(), brain); err == nil {
ok++
}
}
got, _ := os.ReadFile(target)
assert.Equal(t, "untouched", string(got), "BackfillRefs wrote through a swapped-in symlink")
assert.Positive(t, ok, "positive control: some backfills ran")
}
@@ -1,104 +0,0 @@
package symlinkguard
import (
"context"
"net"
"os"
"path/filepath"
"syscall"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// hyperguild#179: the tripwire looked only for symlinks, so a FIFO (or
// socket) in the brain passed it. A FIFO at classification.yaml then made
// startup exit; at a note path it could hang readers that skip the handle
// check.
func TestScanReportsFIFOsAndSocketsAsSpecialFiles(t *testing.T) {
brain := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki", "w"), 0o755))
require.NoError(t, syscall.Mkfifo(filepath.Join(brain, "classification.yaml"), 0o644))
require.NoError(t, syscall.Mkfifo(filepath.Join(brain, "wiki", "w", "f.md"), 0o644))
l, err := net.Listen("unix", filepath.Join(brain, "s.sock"))
require.NoError(t, err)
defer func() { _ = l.Close() }()
require.NoError(t, os.WriteFile(filepath.Join(brain, "wiki", "w", "n.md"), []byte("n"), 0o644))
r, err := Scan(brain)
require.NoError(t, err)
assert.Equal(t, []string{"classification.yaml", "s.sock", "wiki/w/f.md"}, r.Special)
assert.False(t, r.Clean())
assert.Contains(t, r.String(), "special file")
}
// Watch removes a FIFO as it removes symlinks, and leaves a regular file
// (which Scan never reports) alone.
func TestWatchRemovesAFIFOAndLeavesAnUnreportedRegularFile(t *testing.T) {
brain := t.TempDir()
fifo := filepath.Join(brain, "f.md")
require.NoError(t, syscall.Mkfifo(fifo, 0o644))
regular := filepath.Join(brain, "n.md")
require.NoError(t, os.WriteFile(regular, []byte("n"), 0o644))
n := &recNotifier{}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go Watch(ctx, brain, 10*time.Millisecond, n)
require.Eventually(t, func() bool {
_, err := os.Lstat(fifo)
return os.IsNotExist(err)
}, 2*time.Second, 10*time.Millisecond, "the FIFO is removed")
require.Eventually(t, func() bool { return n.count() >= 1 }, 2*time.Second, 10*time.Millisecond)
b, err := os.ReadFile(regular)
require.NoError(t, err)
assert.Equal(t, "n", string(b))
n.mu.Lock()
defer n.mu.Unlock()
assert.Contains(t, n.msgs[0], "f.md")
}
// removeSpecial re-checks the type: a path that is a regular file or a
// directory at the re-check is not removed. (A swap between the re-check and
// the unlink can still remove one; Linux has no conditional unlink. The same
// window exists in removeSymlinks.)
func TestRemoveSpecialRefusesRegularFilesAndDirectories(t *testing.T) {
brain := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(brain, "n.md"), []byte("n"), 0o644))
require.NoError(t, os.MkdirAll(filepath.Join(brain, "d"), 0o755))
require.NoError(t, syscall.Mkfifo(filepath.Join(brain, "f.md"), 0o644))
removed, failed := removeSpecial(brain, []string{"n.md", "d", "f.md"})
assert.Equal(t, []string{"f.md"}, removed)
assert.Equal(t, []string{"n.md", "d"}, failed)
_, err := os.Stat(filepath.Join(brain, "n.md"))
assert.NoError(t, err)
}
// A special file Watch cannot remove (its directory is read-only) stays
// reported: the persistent-finding alert names it.
func TestWatchReportsASpecialFileItCannotRemove(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root ignores directory permissions")
}
brain := t.TempDir()
ro := filepath.Join(brain, "ro")
require.NoError(t, os.MkdirAll(ro, 0o755))
require.NoError(t, syscall.Mkfifo(filepath.Join(ro, "f.md"), 0o644))
require.NoError(t, os.Chmod(ro, 0o555))
t.Cleanup(func() { _ = os.Chmod(ro, 0o755) })
n := &recNotifier{}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go Watch(ctx, brain, 10*time.Millisecond, n)
require.Eventually(t, func() bool { return n.count() >= 1 }, 2*time.Second, 10*time.Millisecond)
_, err := os.Lstat(filepath.Join(ro, "f.md"))
require.NoError(t, err, "the FIFO could not be removed")
n.mu.Lock()
defer n.mu.Unlock()
assert.Contains(t, n.msgs[0], "ro/f.md")
assert.Contains(t, n.msgs[0], "special file")
for _, m := range n.msgs {
assert.NotContains(t, m, "removed", "a file it could not remove was reported as removed")
}
}
@@ -46,16 +46,10 @@ type Notifier interface {
type Result struct {
Symlinks []string
Unreadable []string
// Special holds FIFOs, sockets and devices (hyperguild#179): a reader
// that opens one by path can hang, and a FIFO at classification.yaml
// stops startup.
Special []string
}
// Clean reports whether the scan found nothing to act on.
func (r Result) Clean() bool {
return len(r.Symlinks) == 0 && len(r.Unreadable) == 0 && len(r.Special) == 0
}
func (r Result) Clean() bool { return len(r.Symlinks) == 0 && len(r.Unreadable) == 0 }
// String describes the findings for a log line or alert. Paths are quoted, so
// a name containing a newline cannot blur two findings together.
@@ -65,9 +59,6 @@ func (r Result) String() string {
if len(r.Symlinks) > 0 {
parts = append(parts, fmt.Sprintf("%d symlink(s): %s", len(r.Symlinks), q(r.Symlinks)))
}
if len(r.Special) > 0 {
parts = append(parts, fmt.Sprintf("%d special file(s) (FIFO, socket or device; delete them): %s", len(r.Special), q(r.Special)))
}
if len(r.Unreadable) > 0 {
parts = append(parts, fmt.Sprintf("%d unreadable dir(s): %s", len(r.Unreadable), q(r.Unreadable)))
}
@@ -122,16 +113,12 @@ func Scan(brainDir string) (Result, error) {
if d.IsDir() && p == rootGit {
return filepath.SkipDir
}
switch {
case d.Type()&os.ModeSymlink != 0:
if d.Type()&os.ModeSymlink != 0 {
r.Symlinks = append(r.Symlinks, rel)
case !d.IsDir() && !d.Type().IsRegular():
r.Special = append(r.Special, rel)
}
return nil
})
sort.Strings(r.Symlinks)
sort.Strings(r.Special)
sort.Strings(r.Unreadable)
if walkErr != nil {
// Only the root's own error reaches here (deeper errors are recorded
@@ -195,20 +182,14 @@ func Watch(ctx context.Context, brainDir string, interval time.Duration, n Notif
q.push(fmt.Sprintf("brain symlink tripwire (hyperguild#157): removed %d symlink(s) from the brain dir: %s",
len(removed), Result{}.quoted(removed)))
}
sRemoved, sFailed := removeSpecial(brainDir, r.Special)
if len(sRemoved) > 0 {
slog.Error("brain symlink tripwire removed special files", "count", len(sRemoved))
q.push(fmt.Sprintf("brain symlink tripwire (hyperguild#179): removed %d special file(s) (FIFO, socket or device) from the brain dir: %s",
len(sRemoved), Result{}.quoted(sRemoved)))
}
key = Result{Symlinks: failed, Unreadable: r.Unreadable, Special: sFailed}.String()
key = Result{Symlinks: failed, Unreadable: r.Unreadable}.String()
}
if key != reported {
reported = key
if key != "" {
slog.Error("brain symlink tripwire", "finding", key)
q.push("brain symlink tripwire (hyperguild#157, #179): " + key +
". Symlinks can lead readers out of the brain and special files can hang them; remove them.")
q.push("brain symlink tripwire (hyperguild#157): " + key +
". Readers may follow symlinks out of the brain; fix them.")
}
}
}
@@ -341,40 +322,6 @@ func permanent(err error) bool {
return errors.As(err, &p) && p.Permanent()
}
// removeSpecial unlinks each brain-relative FIFO, socket or device through
// an os.Root on the brain dir, after re-checking that it is still one: a
// path that is a regular file, a directory or a symlink at the re-check is
// left alone and reported as failed. The re-check narrows, but cannot close,
// the window before the unlink (Linux has no conditional unlink); the same
// window exists in removeSymlinks, and only a local writer can use it.
func removeSpecial(brainDir string, rels []string) (removed, failed []string) {
if len(rels) == 0 {
return nil, nil
}
resolved, err := filepath.EvalSymlinks(brainDir)
if err != nil {
return nil, rels
}
root, err := os.OpenRoot(resolved)
if err != nil {
return nil, rels
}
defer func() { _ = root.Close() }()
for _, rel := range rels {
fi, err := root.Lstat(rel)
if err != nil || fi.IsDir() || fi.Mode().IsRegular() || fi.Mode()&os.ModeSymlink != 0 {
failed = append(failed, rel)
continue
}
if err := root.Remove(rel); err != nil {
failed = append(failed, rel)
continue
}
removed = append(removed, rel)
}
return removed, failed
}
// removeSymlinks unlinks each brain-relative symlink through an os.Root on
// the brain dir, after re-checking it is still a symlink. Removing a symlink
// never touches its target. It returns what was removed and what was not.
+33 -22
View File
@@ -12,7 +12,6 @@ import (
"time"
"unicode"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
"github.com/mathiasbq/hyperguild/ingestion/internal/extract"
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
)
@@ -146,12 +145,16 @@ func processFile(ctx context.Context, cfg Config, root *os.Root, rawDir, path, d
_, runErr := pipeline.Run(ctx, cfg.Pipeline, cfg.BrainDir, content, source, false)
if runErr != nil {
// Copy to failed/ and leave a .failed marker so we don't retry. Both
// written through the brain root (hyperguild#158).
if cpErr := copyFromRoot(root, rel, cfg.BrainDir, "raw/failed/"+filename); cpErr != nil {
// Copy to failed/ and leave a .failed marker so we don't retry.
failedDir := filepath.Join(cfg.BrainDir, "raw", "failed")
if mkErr := os.MkdirAll(failedDir, 0o755); mkErr != nil {
return fmt.Errorf("mkdir failed dir: %w", mkErr)
}
dest := filepath.Join(failedDir, filename)
if cpErr := copyFromRoot(root, rel, dest); cpErr != nil {
return fmt.Errorf("copy to failed: %w", cpErr)
}
if mkErr := putBrainFile(cfg.BrainDir, "raw/"+filepath.ToSlash(rel)+".failed", []byte(runErr.Error())); mkErr != nil {
if mkErr := os.WriteFile(path+".failed", []byte(runErr.Error()), 0o644); mkErr != nil {
slog.Error("watcher: failed to write .failed marker", "error", mkErr)
}
@@ -165,18 +168,21 @@ func processFile(ctx context.Context, cfg Config, root *os.Root, rawDir, path, d
}
// Copy to processed/YYYY-MM-DD/ and leave a .processed marker so we don't re-ingest.
// Written through the brain root (hyperguild#158).
dest := "raw/processed/" + date + "/" + filename
if _, err := os.Lstat(filepath.Join(cfg.BrainDir, filepath.FromSlash(dest))); err == nil {
processedDir := filepath.Join(cfg.BrainDir, "raw", "processed", date)
if err := os.MkdirAll(processedDir, 0o755); err != nil {
return fmt.Errorf("mkdir processed dir: %w", err)
}
dest := filepath.Join(processedDir, filename)
if _, err := os.Stat(dest); err == nil {
// Archive copy already exists; append timestamp to avoid overwriting.
ext := filepath.Ext(filename)
base := strings.TrimSuffix(filename, ext)
dest = "raw/processed/" + date + "/" + base + "-" + time.Now().UTC().Format("150405") + ext
dest = filepath.Join(processedDir, base+"-"+time.Now().UTC().Format("150405")+ext)
}
if err := copyFromRoot(root, rel, cfg.BrainDir, dest); err != nil {
if err := copyFromRoot(root, rel, dest); err != nil {
return fmt.Errorf("copy to processed: %w", err)
}
if err := putBrainFile(cfg.BrainDir, "raw/"+filepath.ToSlash(rel)+".processed", []byte(date)); err != nil {
if err := os.WriteFile(path+".processed", []byte(date), 0o644); err != nil {
slog.Error("watcher: failed to write .processed marker", "error", err)
}
@@ -208,22 +214,23 @@ func appendWatcherLog(brainDir, filename string, runErr error, date string) erro
entry := fmt.Sprintf("## %s — watcher error\n\n- **File:** %s\n- **Error:** %s\n\n",
date, filename, runErr.Error())
if err := brainfs.AppendFile(brainDir, "log.md", []byte(entry)); err != nil {
logPath := filepath.Join(brainDir, "log.md")
f, err := os.OpenFile(logPath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
if err != nil {
return fmt.Errorf("open log: %w", err)
}
if _, err = f.WriteString(entry); err != nil {
f.Close() //nolint:errcheck
return fmt.Errorf("write log: %w", err)
}
return nil
}
// putBrainFile creates or replaces a brain-relative file through the brain
// root, never through a symlinked directory or file (hyperguild#158).
func putBrainFile(brainDir, rel string, data []byte) error {
return brainfs.PutFile(brainDir, rel, data)
return f.Close()
}
// copyFromRoot archives a raw/ file read through the root, refusing anything
// that is not a regular file, so a swapped-in symlink cannot copy an outside
// file into raw/processed/ or raw/failed/ (hyperguild#153).
func copyFromRoot(root *os.Root, rel, brainDir, dstRel string) error {
func copyFromRoot(root *os.Root, rel, dst string) error {
fi, err := root.Lstat(rel)
if err != nil {
return err
@@ -236,9 +243,13 @@ func copyFromRoot(root *os.Root, rel, brainDir, dstRel string) error {
return err
}
defer func() { _ = in.Close() }()
data, err := io.ReadAll(in)
out, err := os.Create(dst)
if err != nil {
return err
}
return putBrainFile(brainDir, dstRel, data)
if _, err := io.Copy(out, in); err != nil {
_ = out.Close()
return err
}
return out.Close()
}
@@ -1,66 +0,0 @@
package watcher
import (
"context"
"encoding/json"
"errors"
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
)
func okPipeline() pipeline.Config {
return pipeline.Config{Complete: func(context.Context, string, string) (string, error) {
b, _ := json.Marshal([]pipeline.RawPage{{Title: "X", Type: "source", Subtype: "article", Content: "## S\n\nx\n"}})
return string(b), nil
}, Schema: "# Schema"}
}
func failingPipeline() pipeline.Config {
return pipeline.Config{Complete: func(context.Context, string, string) (string, error) {
return "", errors.New("model down")
}, Schema: "# Schema"}
}
// hyperguild#158: the watcher wrote the archive copy under raw/processed or
// raw/failed, and its log.md entry, by path, so a symlinked archive
// directory or log redirected the write elsewhere in the brain or outside
// it. (A pre-planted marker symlink makes the watcher skip the note, so the
// marker write itself is reachable only by a race.)
func TestProcessDirDoesNotWriteThroughASymlinkedArchiveDirOrLog(t *testing.T) {
for name, pcfg := range map[string]pipeline.Config{"processed": okPipeline(), "failed": failingPipeline()} {
t.Run(name, func(t *testing.T) {
brainDir := setupBrainDir(t)
outside := t.TempDir()
victim := filepath.Join(brainDir, "wiki", "concepts", "v.md")
require.NoError(t, os.WriteFile(victim, []byte("untouched"), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(brainDir, "raw", "n.md"), []byte("note"), 0o644))
require.NoError(t, os.Symlink(outside, filepath.Join(brainDir, "raw", name)))
if name == "failed" {
require.NoError(t, os.Symlink(filepath.Join("wiki", "concepts", "v.md"), filepath.Join(brainDir, "log.md")))
}
_ = processDir(context.Background(), Config{BrainDir: brainDir, Interval: time.Hour, Pipeline: pcfg}, "2026-10-05")
got, _ := os.ReadFile(victim)
assert.Equal(t, "untouched", string(got), "the log was written through a symlink")
ents, _ := os.ReadDir(outside)
assert.Empty(t, ents, "the archive copy was written outside the brain")
})
}
}
func TestProcessDirWritesMarkersAndArchiveInRealDirectories(t *testing.T) {
brainDir := setupBrainDir(t)
require.NoError(t, os.WriteFile(filepath.Join(brainDir, "raw", "n.md"), []byte("note"), 0o644))
errs := processDir(context.Background(), Config{BrainDir: brainDir, Interval: time.Hour, Pipeline: okPipeline()}, "2026-10-05")
assert.Empty(t, errs)
_, err := os.Stat(filepath.Join(brainDir, "raw", "n.md.processed"))
assert.NoError(t, err, "positive control: marker written")
_, err = os.Stat(filepath.Join(brainDir, "raw", "processed", "2026-10-05", "n.md"))
assert.NoError(t, err, "positive control: archive copy written")
}