- brain_reject on a parked pdf/txt moves the bytes; it no longer prepends YAML
- raw/ notes are read through a cap (readCapped); an endless reader test fails
instead of exhausting memory
- a byte-order mark no longer hides a note's wing: key
- INGEST_WING_ROUTING is parsed strictly; a typo stops the server rather than
leaving routing silently off
- tests for the reserved-wing check in every disguise (Sources, sources., ...)
Mutation-checked.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- brain_promote/brain_reject take an optional queue (inbox | needs-review); a
name present in both is refused instead of silently acting on the top-level
note; brain_pending labels every entry with its queue
- parked pdf/txt files are listed (they were invisible); only markdown can be
promoted, the others are moved or rejected
- promote refuses sources/concepts/entities/wiki as a wing
- frontmatterWing reads only a top-level key inside a properly fenced block
- docs/raw-inbox.md: the contract, the three env vars, the known limits
Each behaviour has a test that fails without it (mutation-checked).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- a marker that blocks a note is warned about once, not silently forever
- a marker dated more than 10 minutes ahead (clock skew, a peer) is not
believed and is treated as stale
- every stale marker is removed, not only the last; removal errors are logged
Each has a test that fails without it.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
From the independent review of #210:
- F1: processed/ and failed/ copies are created O_EXCL with -2,-3 suffixes via
createUnique; same-named notes from different folders no longer replace one
another's only surviving copy
- F2: extract.TextInMax / readAllMax cap the bytes actually read (a note can
grow after the Lstat); extract.TextIn gets a 64 MiB default cap
- F3: a regular marker older than its note describes an earlier note of the
same name; it no longer hides the new one (warn + remove); non-regular
markers still count as done (no writes through links)
- F4: the planted-marker-link test is renamed to what it observes
Each has a test that fails without it; the read-cap test uses an endless reader
that errors at 8 MiB so a regression fails instead of exhausting memory.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A clip dropped in raw/ with no other action lands in web-clip, its entity and
concept are the shared pages, raw/ is empty and nothing waits for a human;
client-marked text and a typo'd wing name park; a deliberate subfolder places.
Real pipeline and file layout, model stubbed.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
MaxNoteBytes (default 64 MiB) is checked on the Lstat size before extraction;
an oversize note stays in raw/, is reported once and not retried until it
changes. The #199 case (marker name over NAME_MAX) is covered by the
remember-on-any-failure retire path and now has a test that fails without it.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Reconciled with v0.25.0 semantics:
- markers are only the fallback when the original cannot be removed, and are
now written through the brain root; the refused set (size+mtime keyed) replaces
the path-only 'stuck' set
- #195's tests updated for inbox zero (archive written, original retired, no
marker), and the planted-marker-link test keeps its point: nothing is ever
written through the link, and the note is processed once
- the 'cannot retire' test now counts archive copies; it counted LLM calls,
which pipeline.Run's source-hash check hides (mutation: removing the
remember step left it green)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
From the second independent review of #207:
- parkForReview now skips the delete when the note changed since it was read
(an LLM call sits in that window; Syncthing can deliver a newer version)
- parked copies are created via brainfs (real dirs only, O_EXCL, no symlink
following), so a planted raw/needs-review symlink cannot redirect or hide a
note; this also removes the need for the guard allowlist entry added in
1e14c56, which is reverted
- a '----' line no longer closes the proposal block
Each has a test that failed first and a mutant that fails it.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
From the independent review of #207:
- parked notes are created O_EXCL with -2,-3 suffixes (same-named notes from
different subfolders were overwritten and their originals deleted)
- proposed_wing is emitted only if a plain wing name; a note's own proposal_*
keys are dropped so they cannot outrank the resolver's (first-wins reads)
- CRLF notes: wing: key honoured, one valid frontmatter block on parking
- KnownWings skips wiki dirs that are not canonical lower-case names
- adds the wingmigrate dry-run/apply command
Each fix has a test that failed first and a mutant that fails it.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
With Config.Resolver set, an explicit subfolder/frontmatter wing or an agreeing
classifier+scorer routes the note into that wing; anything else is parked with
its proposal in frontmatter and never extracted. needs-review/ is skipped by
the walk like rejected/. Nil Resolver keeps the retired layout.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The archive holds the content read at the start; if Syncthing replaced the
note during the (minutes-long) LLM pass, removing the original would drop the
newer content. Compare size+mtime before removing; leave it for the next poll.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
When both Remove and the marker write fail (read-only raw/), remember the
path in memory so later polls skip it instead of writing another archive
copy every tick. Adds a test that a failed archive copy leaves the original
in raw/, mutation-checked against removing the early return.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
After a note is archived into raw/processed/ or raw/failed/, remove the
original from raw/ so raw/ only holds unprocessed input. The .processed /
.failed marker is now only a fallback when the removal itself fails.
Existing markers are still honoured.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Review round 1 on PR #195:
- A refused marker or archive write left no marker, so the note was
reprocessed every interval (archive copies piling up; an LLM call per
round on the failure path). The marker check now uses Lstat (any entry
at the marker path counts, so a dangling marker link skips the note
instead of looping), and a note whose archive copy or marker write is
refused is remembered (path, size, mtime) and not retried until it
changes.
- cleanRel no longer refuses a backslash: it is a legal filename byte on
Linux, not a traversal, and a raw note named with one was never
archived or marked.
- Tests now reach every converted writer: the log.md append (real
raw/failed so the log is reached, symlinked log.md), a dangling marker
link, the refused-retry loop with a positive control, a backslash name,
a buildAndWrite swap race, and the claude-session archive with in-brain
and outside symlinked archive dirs. The stale lint allowlist entry for
appendWatcherLog is removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The remaining writers created or rewrote brain files by path:
- pipeline buildAndWrite: os.MkdirAll + os.WriteFile of every page.
- BackfillRefs: os.WriteFile of each concept page; a page swapped for a
symlink after the read was written through (the race test overwrote an
in-brain target before this change).
- watcher: the raw/processed and raw/failed archive copies (os.Create),
the .processed/.failed markers and the log.md entry (os.OpenFile).
- cmd/server: the claude-session archive.
All now go through brainfs PutRegular/AppendRegular (new package-level
PutFile/AppendFile for one-off writers: the watcher's call-site lint bans
.Open calls in that file so raw/ reads stay on OpenRegular).
Stacked on #189 (needs PutRegular/AppendRegular).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds brain_reject (RejectNote + MCP tool) and fixes the watcher to skip raw/rejected/, preventing declined notes from being silently re-ingested and republished.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
Root cause was neither of the two mechanisms the issue named
(CanonicalizeLinks over-canonicalizing, or a schema/prompt bias) —
CanonicalizeLinks correctly resolved [[Mission]] because
wiki/concepts/mission.md genuinely exists; the extraction prompt has
no "mission" bias either. The real bug: watcher.processDir walks
brain/raw/ and feeds every .md/.txt/.pdf file to pipeline.Run as
source material to extract, with no exclusion for
tunnel-candidates-*.md — AutoTunnel's own fuzzy-match human-review
queue (logFuzzyCandidates writes it into that same raw/ directory).
The watcher's next poll re-ingested the raw candidate log itself, and
the LLM naturally surfaced "mission" as a wikilink because the log
literally lists `(term: "mission")` entries in its content — a
generic-word title match DetectTunnels correctly flagged as fuzzy
(not auto-tunneled) but which still leaked downstream once treated as
real source material.
api.ListPending already excludes tunnel-candidates-* when listing
raw/ for promotion; processDir gets the same exclusion.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
- New extract package: Text() dispatcher for .md/.txt passthrough and
PDF extraction via pdftotext subprocess
- wiki.Entry gains Aliases []string, loaded from YAML frontmatter
- Fuzzy entity resolution in pipeline: normalizes titles (lowercase,
strip articles, collapse hyphens) and matches proposed pages against
existing inventory slugs and aliases to prevent proliferation
- Watcher and API handler now use extract.Text() instead of os.ReadFile
- Dockerfile: apk add poppler-utils in Alpine runtime stage
Files dropped into brain/raw/ are now copied to processed/ or failed/ rather
than moved. A .processed or .failed marker is written next to the original so
the watcher skips it on subsequent polls without deleting it. This keeps
Syncthing-synced Obsidian vaults intact after ingestion.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Polls brain/raw/ on a configurable ticker, derives human-readable source
names from filenames, runs the pipeline, and moves files to
processed/YYYY-MM-DD/ on success or failed/ on error with a log.md entry.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
mathiasandClaude Sonnet 5.5