Compile-on-ingest, first half. buildAndWrite did an unconditional
os.WriteFile, so re-ingesting a source replaced its page wholesale and
discarded everything accumulated since — including human edits.
hyperguild#43 recorded the symptom as "re-ingest overwrites wholesale";
the framing that ingest "appends" understates it.
Now the write reads what is on disk and merges the fresh extraction into
it, asymmetrically in the existing page's favour: on-disk frontmatter and
on-disk sections win, extraction-only sections are added, bullet lists
union. An extraction can add to a note but never silently rewrite what a
human put there, which is what makes it safe on every ingest rather than
behind a review gate.
Two supporting fixes in wiki.Merge, both required for this to be safe to
run repeatedly:
- append sections (Open Questions, Updates, Evolving Notes) concatenated
wholesale, which is correct exactly once. Re-ingesting an unchanged
source would grow them every run until the page was unreadable. Now only
genuinely new lines are appended. Line-level rather than block-level,
because the common case is a re-ingest carrying one new line alongside
several it already has, and a block-level check would duplicate the old
ones every time.
- rebuildContent kept whatever blank lines a section was parsed with and
added its own, so every merge pass grew the page by a blank line —
invisible in a diff viewer, fatal to byte-stability. Section bodies are
now normalised. Re-ingesting unchanged content is byte-identical, so
brain-sync does not commit noise that hides real edits.
The drift guard that would otherwise catch repeated ingests (hashing the
source body, B3) does not exist yet, so idempotency holds here rather than
upstream.
Known limitation, deliberate: on-disk frontmatter wins, so last_updated
keeps its first-ingest date on a revised page. The alternative is this
function editing frontmatter it does not own, and no lint reads that field.
Refs hyperguild#43.
Four call sites did `s[:limit] + "…"`. Both halves are byte operations:
len() counts bytes and a slice cuts bytes. On Swedish text — å ä ö are
two bytes, an em dash three — the cut lands mid-character and leaves an
orphaned lead byte, producing invalid UTF-8.
It failed loudly in exactly one place: wiki/index.md is written raw to
disk, where every strict decoder then rejects the whole file. The other
three (brain_context excerpts x2, brain_query excerpts) feed MCP
responses, where encoding/json silently substitutes U+FFFD — so those had
been quietly eating the last character of truncated excerpts, with no
error anywhere and nothing to notice.
textutil.Truncate counts runes and appends the marker only when content
was actually dropped; an ellipsis on complete text is a lie about the
content. The test is a property, not an example: it walks the limit
across every offset of a multi-byte string and asserts utf8.Valid at each
one, so a future off-by-one cannot pass by picking a lucky boundary.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeAp5LTscnz2W7Ubt4eJ6m
mathias
