Ingest now stamps source_hash — sha256 of the ingested body, frontmatter
excluded — onto the source page, and compares it before doing any work. An
unchanged source returns Skipped with no LLM call at all.
This is cost and signal, not correctness: #103 already made a re-ingest
byte-identical, so a repeated run was harmless but not free — a full
extraction over every chunk to produce a result that merges to nothing.
Building this first would have put idempotency in the guard rather than in
the write, which is the wrong dependency direction.
Body-only hashing on purpose. A re-crawl that stamps a new fetch date has
not changed the source, and treating that as drift would re-extract the
whole corpus every time a crawler ran. A missing or unreadable hash always
means "extract": a missed skip costs one extraction, a wrong skip loses an
update silently, and silence is the failure mode this repo keeps paying
for.
The skip is reported (Result.Skipped plus a warning naming the hash), not
silent — a run that does nothing must say so, or "the pipeline is fine" and
"the pipeline stopped working" look identical (#107).
Storing the hash required settling who owns which frontmatter field. #103
made the on-disk block win wholesale, which protects human-set metadata
from a regenerating writer but froze fields the generator legitimately
owns. wiki.Merge now refreshes an explicit generatorOwned allowlist —
last_updated and source_hash — and keeps everything else on disk. An
allowlist rather than a carve-out, so a field nobody has thought about
stays human-owned by default. That also retires the last_updated
limitation #103 shipped with.
Refs #105, #103, #107, brain#21.
Compile-on-ingest, first half. buildAndWrite did an unconditional
os.WriteFile, so re-ingesting a source replaced its page wholesale and
discarded everything accumulated since — including human edits.
hyperguild#43 recorded the symptom as "re-ingest overwrites wholesale";
the framing that ingest "appends" understates it.
Now the write reads what is on disk and merges the fresh extraction into
it, asymmetrically in the existing page's favour: on-disk frontmatter and
on-disk sections win, extraction-only sections are added, bullet lists
union. An extraction can add to a note but never silently rewrite what a
human put there, which is what makes it safe on every ingest rather than
behind a review gate.
Two supporting fixes in wiki.Merge, both required for this to be safe to
run repeatedly:
- append sections (Open Questions, Updates, Evolving Notes) concatenated
wholesale, which is correct exactly once. Re-ingesting an unchanged
source would grow them every run until the page was unreadable. Now only
genuinely new lines are appended. Line-level rather than block-level,
because the common case is a re-ingest carrying one new line alongside
several it already has, and a block-level check would duplicate the old
ones every time.
- rebuildContent kept whatever blank lines a section was parsed with and
added its own, so every merge pass grew the page by a blank line —
invisible in a diff viewer, fatal to byte-stability. Section bodies are
now normalised. Re-ingesting unchanged content is byte-identical, so
brain-sync does not commit noise that hides real edits.
The drift guard that would otherwise catch repeated ingests (hashing the
source body, B3) does not exist yet, so idempotency holds here rather than
upstream.
Known limitation, deliberate: on-disk frontmatter wins, so last_updated
keeps its first-ingest date on a revised page. The alternative is this
function editing frontmatter it does not own, and no lint reads that field.
Refs hyperguild#43.
Four call sites did `s[:limit] + "…"`. Both halves are byte operations:
len() counts bytes and a slice cuts bytes. On Swedish text — å ä ö are
two bytes, an em dash three — the cut lands mid-character and leaves an
orphaned lead byte, producing invalid UTF-8.
It failed loudly in exactly one place: wiki/index.md is written raw to
disk, where every strict decoder then rejects the whole file. The other
three (brain_context excerpts x2, brain_query excerpts) feed MCP
responses, where encoding/json silently substitutes U+FFFD — so those had
been quietly eating the last character of truncated excerpts, with no
error anywhere and nothing to notice.
textutil.Truncate counts runes and appends the marker only when content
was actually dropped; an ellipsis on complete text is a lie about the
content. The test is a property, not an example: it walks the limit
across every offset of a multi-byte string and asserts utf8.Valid at each
one, so a future off-by-one cannot pass by picking a lucky boundary.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeAp5LTscnz2W7Ubt4eJ6m
mathias
