The source page carries source_hash, which the drift check trusts. Written first,
a batch that failed midway left a hash behind, so re-ingesting the same content
was skipped and the rest of the batch was lost for good. In a wing it is now
written last. An over-cap note in brain_pending says promote/reject cannot act
on it and to delete or move it by hand. Mutation-checked.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- in a wing, an ingest that writes no source page is an error: the note goes to
raw/failed with the warnings in log.md instead of leaving the inbox as
'processed' (model returned [], only invalid types, only clashing entities)
- the wing's source path is deterministic, so Resolve no longer redirects a
source onto another note's page because the model gave both the same title
('Home'); that overwrote the hash and the text
- a batch tie between concept and entity prefers the page already on disk, so
its update is not lost; the wing's own source page keeps its slug even when a
foreign page shares it
- brain_pending lists an unreadable (over-cap) note with the reason instead of
dropping it; the client filter's 4 KiB match-span limit is documented
Each has a test that fails without it (mutation-checked).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- brain_reject on a parked pdf/txt moves the bytes; it no longer prepends YAML
- raw/ notes are read through a cap (readCapped); an endless reader test fails
instead of exhausting memory
- a byte-order mark no longer hides a note's wing: key
- INGEST_WING_ROUTING is parsed strictly; a typo stops the server rather than
leaving routing silently off
- tests for the reserved-wing check in every disguise (Sources, sources., ...)
Mutation-checked.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Views of each chunk: as is; NFKD + marks/format/fillers dropped + Cyrillic/Greek
lookalikes mapped + dashes normalised, with whitespace collapsed (acme corp,
\s) or removed (acme-\nclient); and a soft hyphen read as a dash. Chunks of 1 MiB
with a 4 KiB overlap bound the folding's allocation (it was ~14x the input: 919
MiB for a 64 MiB note). Documented limits: leetspeak, punctuation splits,
encodings and homoglyphs outside a small table. Mutation-checked.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
dropSlugClashes only compared against what was on disk, so a clip named like an
entity the model extracted wrote both pages and neither could be updated again.
In-batch winners: source over entity over concept. Against the brain, only
entities/concepts are skipped; the source's slug was already chosen free or its
own by resolveSourceSlug, so an existing bad pair no longer locks the source out.
Mutation-checked.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- brain_promote/brain_reject take an optional queue (inbox | needs-review); a
name present in both is refused instead of silently acting on the top-level
note; brain_pending labels every entry with its queue
- parked pdf/txt files are listed (they were invisible); only markdown can be
promoted, the others are moved or rejected
- promote refuses sources/concepts/entities/wiki as a wing
- frontmatterWing reads only a top-level key inside a properly fenced block
- docs/raw-inbox.md: the contract, the three env vars, the known limits
Each behaviour has a test that fails without it (mutation-checked).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- client pattern is matched against the text and against an NFKC-folded,
format-character-free, dash-normalised, whitespace-free form
(non-breaking hyphen, zero-width characters, soft hyphen, line splits,
fullwidth letters). x/text/unicode/norm was already in the module graph as an
indirect dependency; this makes it direct, no new download.
- folder/frontmatter wing names must be ASCII (strings.ToLower folds the Kelvin
sign to 'k') and must not be sources/concepts/entities/wiki; such names are
never proposed as a wing, and a reserved default is a startup error
Mutation-checked.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- a source whose slug is taken by anything but this wing's own page gets a
wing-qualified, stable slug (talk-homelab) instead of failing the ingest
- an extracted entity/concept whose slug belongs to a different page is skipped
with a warning, so one page can no longer lock a source out of re-ingest
- in a wing only source/concept/entity are valid page types; the model (and so
the clip) can no longer choose wiki/<type>/ directories
Mutation-checked; stable-slug choice pinned directly because title resolution
masks it at the Run level.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Videos and articles move to the default wing (wing/hall stamped, body and
links byte-preserved); auto-tunnel review logs are deleted; notes, projects and
untyped pages stay. Plan is read-only; Apply archives every page it touches to
archive/wing-migration-<date>/ (outside raw/) before changing anything,
creates destinations O_EXCL and removes sources only after; Rollback restores
byte for byte. concepts/ and entities/ are the shared layer and are untouched.
Dry run against the live brain: 16 move, 24 delete, 15 left.
Mutation-checked: junk guard, type filter, slug collision, no-overwrite,
archive-before-change, action gate.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
FindSlug returned the first hit, so a wing's own page of a name masked another
page of the same name (the retired dirs are searched before the wings, and wings
alphabetically). FindSlugs returns every match and callers refuse any that is
not their own. Found by the legacy-move planner's own test.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Removes the LLM classifier, the substring 'BM25' scorer, the agreement gate,
the backtest and the old migrate/wingmigrate tooling: measured at 2 of 55
pages placed (#208), and the migration was built for the per-wing facts layout
this branch replaces. wing now holds the decision types, the Router and
KnownWings. INGEST_WING_ROUTING=1 builds the Router (default wing
INGEST_DEFAULT_WING=web-clip, client pattern from INGEST_CLIENT_BLOCK or
CLAUDE_INGEST_CLIENT_BLOCK); a bad default or pattern is fatal at startup.
Watcher tests rewritten for the Router, including a deliberate subfolder into
any existing wing, typo'd folders parked, forged-key and edit-while-placing
cases.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- a marker that blocks a note is warned about once, not silently forever
- a marker dated more than 10 minutes ahead (clock skew, a peer) is not
believed and is treated as stale
- every stale marker is removed, not only the last; removal errors are logged
Each has a test that fails without it.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
brain_pending lists a parked note under its bare filename, but promote/reject
only read raw/<name>, so the documented approval path did not exist. They now
fall back to raw/needs-review/ (top level wins on a name clash; a symlinked
directory is refused) and promote drops the resolver's proposal keys.
Mutation-checked.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
From the independent review of #210:
- F1: processed/ and failed/ copies are created O_EXCL with -2,-3 suffixes via
createUnique; same-named notes from different folders no longer replace one
another's only surviving copy
- F2: extract.TextInMax / readAllMax cap the bytes actually read (a note can
grow after the Lstat); extract.TextIn gets a 64 MiB default cap
- F3: a regular marker older than its note describes an earlier note of the
same name; it no longer hides the new one (warn + remove); non-regular
markers still count as done (no writes through links)
- F4: the planted-marker-link test is renamed to what it observes
Each has a test that fails without it; the read-cap test uses an endless reader
that errors at 8 MiB so a regression fails instead of exhausting memory.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A clip dropped in raw/ with no other action lands in web-clip, its entity and
concept are the shared pages, raw/ is empty and nothing waits for a human;
client-marked text and a typo'd wing name park; a deliberate subfolder places.
Real pipeline and file layout, model stubbed.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
With Config.Wing a source goes to wiki/<wing>/sources/ (wing/hall frontmatter);
concepts and entities stay in the shared wiki/concepts and wiki/entities, so an
entity cited from two wings is ONE page with both sources (the per-wing facts/
layout duplicated it). A source slug already used anywhere but its own wing's
sources/ is refused (slug is the graph identity). wiki.FindSlug added.
Mutation-checked: collision guard, shared inventory, per-wing source path.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
No model. A subfolder/frontmatter wing must be a plain name of an existing wing
and then places; conflicting signals and client-marker text park; everything
else goes to the default wing. Wings are read from disk per decision.
Mutation-checked.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
MaxNoteBytes (default 64 MiB) is checked on the Lstat size before extraction;
an oversize note stays in raw/, is reported once and not retried until it
changes. The #199 case (marker name over NAME_MAX) is covered by the
remember-on-any-failure retire path and now has a test that fails without it.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Reconciled with v0.25.0 semantics:
- markers are only the fallback when the original cannot be removed, and are
now written through the brain root; the refused set (size+mtime keyed) replaces
the path-only 'stuck' set
- #195's tests updated for inbox zero (archive written, original retired, no
marker), and the planted-marker-link test keeps its point: nothing is ever
written through the link, and the note is processed once
- the 'cannot retire' test now counts archive copies; it counted LLM calls,
which pipeline.Run's source-hash check hides (mutation: removing the
remember step left it green)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
From the second independent review of #207:
- parkForReview now skips the delete when the note changed since it was read
(an LLM call sits in that window; Syncthing can deliver a newer version)
- parked copies are created via brainfs (real dirs only, O_EXCL, no symlink
following), so a planted raw/needs-review symlink cannot redirect or hide a
note; this also removes the need for the guard allowlist entry added in
1e14c56, which is reverted
- a '----' line no longer closes the proposal block
Each has a test that failed first and a mutant that fails it.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The previous commit was pushed with this guard red: the gate was run with ';'
instead of '&&'. Allowlisting is the guard's own sanctioned mechanism and is
called out for review.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
From the independent review of #207:
- parked notes are created O_EXCL with -2,-3 suffixes (same-named notes from
different subfolders were overwritten and their originals deleted)
- proposed_wing is emitted only if a plain wing name; a note's own proposal_*
keys are dropped so they cannot outrank the resolver's (first-wins reads)
- CRLF notes: wing: key honoured, one valid frontmatter block on parking
- KnownWings skips wiki dirs that are not canonical lower-case names
- adds the wingmigrate dry-run/apply command
Each fix has a test that failed first and a mutant that fails it.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sources are placed by the wing Resolver; concepts/entities inherit the strict
majority wing of the sources that cite them. Collisions, junk (auto-tunnel
logs) and anything unplaceable are Exceptions left in place. Apply writes the
destination O_EXCL then removes the source, only for Auto items.
Mutation-checked (junk guard, majority, slug identity, status gate,
no-overwrite, destination-exists).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The first version fed the text through a path that never carried it into the
file, so it passed under the mutant. A classifier error does carry it; the
test now fails with %q replaced by %s.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Approval is moving the note into raw/<wing>/ (the watcher ingests it there);
a dedicated approve tool is tracked separately. Proposal reason is written
Go-quoted so LLM text cannot start a frontmatter key; pinned by a
mutation-checked test.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Off by default (retired layout unchanged). On: eligibility from
classification.yaml, local ingest LLM as classifier, BM25 as cross-check,
INGEST_WING_THRESHOLD (default 0.9, rejected outside [0,1]). Build failure
is fatal when explicitly enabled.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
With Config.Resolver set, an explicit subfolder/frontmatter wing or an agreeing
classifier+scorer routes the note into that wing; anything else is parked with
its proposal in frontmatter and never extracted. needs-review/ is skipped by
the walk like rejected/. Nil Resolver keeps the retired layout.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sources -> wiki/<wing>/sources, concepts/entities -> wiki/<wing>/facts with
kind: frontmatter; inventory, title resolution and back-references are
confined to the target wing. Empty Wing keeps the retired layout unchanged.
Mutation-checked: widening the inventory beyond the wing fails the
cross-wing back-reference test; a non-plain wing name is rejected.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A first backtest showed BM25 summed over a wing's top hits favours big wings
(0/4 agreement). Best-page scoring is volume-robust; pinned by a test that
failed before the change.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Explicit subfolder/frontmatter wins; otherwise a classifier and an
independent scorer must agree above a threshold on a known, eligible wing.
Anything else is an Exception for a human. Never creates a wing, never
assigns an ineligible (client/untagged) wing. Mutation-checked: dropping the
eligibility, known-wing, scorer, threshold or conflict check fails a test.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The archive holds the content read at the start; if Syncthing replaced the
note during the (minutes-long) LLM pass, removing the original would drop the
newer content. Compare size+mtime before removing; leave it for the next poll.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
When both Remove and the marker write fail (read-only raw/), remember the
path in memory so later polls skip it instead of writing another archive
copy every tick. Adds a test that a failed archive copy leaves the original
in raw/, mutation-checked against removing the early return.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
After a note is archived into raw/processed/ or raw/failed/, remove the
original from raw/ so raw/ only holds unprocessed input. The .processed /
.failed marker is now only a fallback when the removal itself fails.
Existing markers are still honoured.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Review round 2 on PR #195: the marker writes had no test, so reverting
them to by-path writes, or the refusal to log-only, stayed green. The
link is planted from inside the fake model call, between the walk's
Lstat and the marker write, for both markers and both an outside and
an in-brain relative target; red on all four flips. The marker-entry
skip is now checked for .failed too.
Also pins cleanRel's acceptance of a backslash as a filename byte in
brainfs itself, and stops the buildAndWrite control claiming writes it
does not count.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An authenticated REST request to an unregistered path reached the inner
mux, which left Pattern empty, and metrics.Middleware fell back to the
raw URL path: any token holder could create one Prometheus series per
unique path. The fallback is now the fixed label "unmatched"; real routes
keep their pattern. Tests: buildMux with an authenticated random path,
and the middleware unit test; the timing test now routes through a
ServeMux as production does.
Fixes#185.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 1 on PR #195:
- A refused marker or archive write left no marker, so the note was
reprocessed every interval (archive copies piling up; an LLM call per
round on the failure path). The marker check now uses Lstat (any entry
at the marker path counts, so a dangling marker link skips the note
instead of looping), and a note whose archive copy or marker write is
refused is remembered (path, size, mtime) and not retried until it
changes.
- cleanRel no longer refuses a backslash: it is a legal filename byte on
Linux, not a traversal, and a raw note named with one was never
archived or marked.
- Tests now reach every converted writer: the log.md append (real
raw/failed so the log is reached, symlinked log.md), a dangling marker
link, the refused-retry loop with a positive control, a backslash name,
a buildAndWrite swap race, and the claude-session archive with in-brain
and outside symlinked archive dirs. The stale lint allowlist entry for
appendWatcherLog is removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three Watch tests flaked under -race and CPU load (3/8 of 8 parallel
x10 runs on main):
- QueueIsBounded slept 100ms before looking for the summary; it now
waits for the summary, then checks the bound.
- DeliversEveryQueuedAlert asserted FIFO order, but the outage ends
asynchronously, possibly mid-flush, so a later item can send first. It
now asserts each removal is delivered exactly once (order is not what
it is named for).
- AccountsForEveryRemoval slept 50ms after the first summary, counted
only "removed 1" and took the last summary's count. It now sums every
"removed N" and every suppression count and waits until they account
for all plants.
Stress: 0/8 failures in three rounds of 8 parallel x -test.count=10 under
-race. Flips still red: unbounded queue, dropped retries, suppressions
not counted.
Fixes#193.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every alert was titled "brain-capture audit degraded", including the
symlink tripwire's and governance-edit alerts. NtfyNotifier.WithTitle
returns a titled copy (nil stays nil); buildNotifier(getenv, title) is
called per source: capture audit (unchanged title), "brain symlink
tripwire", "brain governance edit". Tests assert the Title header.
Fixes#163.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The remaining writers created or rewrote brain files by path:
- pipeline buildAndWrite: os.MkdirAll + os.WriteFile of every page.
- BackfillRefs: os.WriteFile of each concept page; a page swapped for a
symlink after the read was written through (the race test overwrote an
in-brain target before this change).
- watcher: the raw/processed and raw/failed archive copies (os.Create),
the .processed/.failed markers and the log.md entry (os.OpenFile).
- cmd/server: the claude-session archive.
All now go through brainfs PutRegular/AppendRegular (new package-level
PutFile/AppendFile for one-off writers: the watcher's call-site lint bans
.Open calls in that file so raw/ reads stay on OpenRegular).
Stacked on #189 (needs PutRegular/AppendRegular).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New brainfs PutRegular (create or replace), CreateRegular (create only), AppendRegular, Remove and RealParent: missing parents created one component at a time through the root, every existing component a real directory, new files created with O_EXCL and confirmed before writing. brain_write, promote/reject (O_EXCL create, source kept on refusal), StampProvenanceFile, BuildWingIndex, RebuildIndex, AppendLog and the tunnel candidate log use them; provenance.ApplyFile removed.
Fixes#188. Refs #158. Adversarial review: BLOCKED, BLOCKED, PASS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
StripUnresolvedLinks logs a warning and returns the content unchanged when the link index cannot be built (e.g. an unreadable subdirectory), instead of failing every brain_write/brain_update and losing the learning. Nothing is unwrapped on a partial index.
Fixes#178.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
Scan reports special files (not regular, directory or symlink) in Result.Special, so the startup check refuses with a message naming them; Watch removes them through an os.Root after re-checking the type, and reports any it cannot remove (never as removed).
Fixes#179. Adversarial review: BLOCKED, BLOCKED, PASS. Flake on main tracked in #193.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
ReadRegular, ReadRegularMax and WriteRegular require a clean, local path and a real directory at every level above the file, so an in-brain directory symlink (or a ".." after one) can no longer redirect brain_update, brain_tunnel, brain_read/get or the other readers elsewhere in the brain. session.Append opens through an os.Root on the brain dir, refuses a non-directory sessions dir, and writes only after confirming the log path is the regular file it opened.
Fixes#173. Adversarial review: BLOCKED, PASS. Follow-up: #188 (brain_write create path).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
