PR #156 round 3: a readRawNote rewrite to os.ReadFile or Root.ReadFile
passed the lint and brought back the escape (19k leaks in 4s) and the
FIFO hang. Listing forbidden APIs is whack-a-mole, so the control now
rests on behaviour:
- TestRawReadersLeakNothingUnderASymlinkSwapRace and
TestRawReadersDoNotHangUnderAFIFOSwapRace swap a regular note, a
symlink to an outside file, or a FIFO into raw/r.md while ListPending
and ReadIngestFile loop for 1.5s; any leaked byte or a hang fails.
- TestProcessDir_DoesNotHangUnderAFIFOSwapRace does the same for the
unattended watcher (read and archive copy).
Flipped: os.ReadFile -> leak + hang tests red; Root.ReadFile -> hang
test red; copyFromRoot back to Root.Open -> watcher test red.
The lint stays as a second layer: it also forbids ReadFile, matches the
selector (so a stored method value is caught), refuses a dot-import of
extract, and says it only covers the three files it parses.
Refs #153
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Reverting the ingest call sites to extract.Text(path) stayed green;
a test now fails if anything outside internal/extract calls
extract.Text( (only TextIn, through an os.Root, may be used).
- The watcher symlink test passed for a watcher that ingested nothing,
or stopped at the first symlink (SkipAll: one synced link halting all
ingestion). It now plants a real note after the symlink and asserts
exactly one ingest, and the symlinked-raw refusal has its own test.
- A FIFO swapped in between Lstat and Open hung open(2) forever.
extract.OpenRegular opens non-blocking and checks the type on the
handle; readRawNote, TextIn and the watcher's archive copy use it.
- The symlinked-root test covers pending, promote, reject, resolve and
ReadIngestFile, and is named for exactly those.
- The hard-link residual is documented where it is accepted.
Out of scope, filed: #157 (symlinks elsewhere in the Syncthing-shared
brain dir reach search and other readers), #158 (writes under raw/
follow symlinks).
Refs #153
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
raw/ is writable via Syncthing (a sendreceive folder shared with
flamingo, symlinks synced) and by the brain CronJobs, and several
readers followed symlinks out of it:
- /pending and brain_pending showed a symlink target's contents;
brain_promote would copy it into wiki/ (then brain-sync to Gitea);
brain_reject would copy it into raw/rejected/.
- The watcher, which needs no API call at all, ingested symlinked files
(WalkDir yields them; extract.Text followed them). Not in the issue;
found while fixing it.
- Ingest checked a path then reopened it by name (TOCTOU).
OpenRawRoot opens brain/raw as an os.Root and refuses a raw/ that is
itself a symlink. ListPending, PromoteNote and RejectNote read through
it and refuse anything not a regular file. extract.TextIn reads through
a root (temp copy for pdf/docmark); /ingest-path, brain_ingest and the
watcher use it, and the watcher archives via the root too, so a file
swapped for a symlink after a check still cannot lead out.
Refs #153
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds brain_reject (RejectNote + MCP tool) and fixes the watcher to skip raw/rejected/, preventing declined notes from being silently re-ingested and republished.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
Root cause was neither of the two mechanisms the issue named
(CanonicalizeLinks over-canonicalizing, or a schema/prompt bias) —
CanonicalizeLinks correctly resolved [[Mission]] because
wiki/concepts/mission.md genuinely exists; the extraction prompt has
no "mission" bias either. The real bug: watcher.processDir walks
brain/raw/ and feeds every .md/.txt/.pdf file to pipeline.Run as
source material to extract, with no exclusion for
tunnel-candidates-*.md — AutoTunnel's own fuzzy-match human-review
queue (logFuzzyCandidates writes it into that same raw/ directory).
The watcher's next poll re-ingested the raw candidate log itself, and
the LLM naturally surfaced "mission" as a wikilink because the log
literally lists `(term: "mission")` entries in its content — a
generic-word title match DetectTunnels correctly flagged as fuzzy
(not auto-tunneled) but which still leaked downstream once treated as
real source material.
api.ListPending already excludes tunnel-candidates-* when listing
raw/ for promotion; processDir gets the same exclusion.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
- New extract package: Text() dispatcher for .md/.txt passthrough and
PDF extraction via pdftotext subprocess
- wiki.Entry gains Aliases []string, loaded from YAML frontmatter
- Fuzzy entity resolution in pipeline: normalizes titles (lowercase,
strip articles, collapse hyphens) and matches proposed pages against
existing inventory slugs and aliases to prevent proliferation
- Watcher and API handler now use extract.Text() instead of os.ReadFile
- Dockerfile: apk add poppler-utils in Alpine runtime stage
Files dropped into brain/raw/ are now copied to processed/ or failed/ rather
than moved. A .processed or .failed marker is written next to the original so
the watcher skips it on subsequent polls without deleting it. This keeps
Syncthing-synced Obsidian vaults intact after ingestion.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Polls brain/raw/ on a configurable ticker, derives human-readable source
names from filenames, runs the pipeline, and moves files to
processed/YYYY-MM-DD/ on success or failed/ on error with a log.md entry.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
mathiasandClaude Opus 5.5