Review of PR #133 found a serious bug: the file watcher's processDir
walks brain/raw/ with filepath.WalkDir (recurses by default) and only
skipped processed/ and failed/ by name. brain_reject moves a declined
note to raw/rejected/, which the watcher had no reason to know about --
so on its next tick (30s default), it picked the note back up, ran it
through the full LLM extraction pipeline, and published it into the
wiki. A human's explicit "not worth promoting" decision was silently
reversed by a background job 30 seconds later. Reproduced directly:
flip-testing the fix back out shows the watcher's own log line, "file
processed file=declined-file.md", proving this wasn't hypothetical.
Fixed by adding "rejected" to the same subdirectory skip-list
processed/failed already use. Checked the other two consumers of raw/
content for the same class of bug and found neither is affected:
graphsync.BackfillFromBrainDir only walks wiki/ and knowledge/; search's
resolveRoots only ever returns knowledge/ and wiki/. Neither touches
raw/ at all, rejected or not.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root cause was neither of the two mechanisms the issue named
(CanonicalizeLinks over-canonicalizing, or a schema/prompt bias) —
CanonicalizeLinks correctly resolved [[Mission]] because
wiki/concepts/mission.md genuinely exists; the extraction prompt has
no "mission" bias either. The real bug: watcher.processDir walks
brain/raw/ and feeds every .md/.txt/.pdf file to pipeline.Run as
source material to extract, with no exclusion for
tunnel-candidates-*.md — AutoTunnel's own fuzzy-match human-review
queue (logFuzzyCandidates writes it into that same raw/ directory).
The watcher's next poll re-ingested the raw candidate log itself, and
the LLM naturally surfaced "mission" as a wikilink because the log
literally lists `(term: "mission")` entries in its content — a
generic-word title match DetectTunnels correctly flagged as fuzzy
(not auto-tunneled) but which still leaked downstream once treated as
real source material.
api.ListPending already excludes tunnel-candidates-* when listing
raw/ for promotion; processDir gets the same exclusion.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
- New extract package: Text() dispatcher for .md/.txt passthrough and
PDF extraction via pdftotext subprocess
- wiki.Entry gains Aliases []string, loaded from YAML frontmatter
- Fuzzy entity resolution in pipeline: normalizes titles (lowercase,
strip articles, collapse hyphens) and matches proposed pages against
existing inventory slugs and aliases to prevent proliferation
- Watcher and API handler now use extract.Text() instead of os.ReadFile
- Dockerfile: apk add poppler-utils in Alpine runtime stage
Files dropped into brain/raw/ are now copied to processed/ or failed/ rather
than moved. A .processed or .failed marker is written next to the original so
the watcher skips it on subsequent polls without deleting it. This keeps
Syncthing-synced Obsidian vaults intact after ingestion.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Polls brain/raw/ on a configurable ticker, derives human-readable source
names from filenames, runs the pipeline, and moves files to
processed/YYYY-MM-DD/ on success or failed/ on error with a log.md entry.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
mathiasandClaude Sonnet 5