The asymmetry that makes unattended ingest safe (#103) also discarded a
real signal. When a fresh extraction produced a different version of a
section already on disk, the on-disk version won and the model's version
went in the bin without a word — so "the source is unchanged" and "the
source changed its mind and we ignored it" were indistinguishable.
wiki.MergeReport now returns the headings where the incoming version
disagreed and lost, and the ingest run reports them as warnings, which
AppendLog already writes into log.md.
This is the simplified answer to contradiction flagging at ingest. The
issue proposed calling brain-gardener's Judge from the ingest path, which
needs either a shared Go module across two repos or an HTTP service that
does not exist — and puts an LLM call on the write path, where a timeout
must neither fail the ingest nor silently mark nothing as contested.
None of that is necessary to get the signal. The merge already computes
the disagreement; it was throwing it away. Surfacing it costs nothing,
needs no model, no cue-word matcher, no cross-repo dependency, and is
deterministic.
Deliberately textual disagreement, not semantic contradiction. Deciding
whether a disagreement is a genuine contradiction stays with the
Judge-gated enrich pass that already runs hourly and is approval-gated by
design (brain-gardener#175). Detection here, adjudication there.
Merge is kept as a thin wrapper over MergeReport so existing callers are
untouched.
Refs #104, #103, brain-gardener#175, brain#21.
Ingest now stamps source_hash — sha256 of the ingested body, frontmatter
excluded — onto the source page, and compares it before doing any work. An
unchanged source returns Skipped with no LLM call at all.
This is cost and signal, not correctness: #103 already made a re-ingest
byte-identical, so a repeated run was harmless but not free — a full
extraction over every chunk to produce a result that merges to nothing.
Building this first would have put idempotency in the guard rather than in
the write, which is the wrong dependency direction.
Body-only hashing on purpose. A re-crawl that stamps a new fetch date has
not changed the source, and treating that as drift would re-extract the
whole corpus every time a crawler ran. A missing or unreadable hash always
means "extract": a missed skip costs one extraction, a wrong skip loses an
update silently, and silence is the failure mode this repo keeps paying
for.
The skip is reported (Result.Skipped plus a warning naming the hash), not
silent — a run that does nothing must say so, or "the pipeline is fine" and
"the pipeline stopped working" look identical (#107).
Storing the hash required settling who owns which frontmatter field. #103
made the on-disk block win wholesale, which protects human-set metadata
from a regenerating writer but froze fields the generator legitimately
owns. wiki.Merge now refreshes an explicit generatorOwned allowlist —
last_updated and source_hash — and keeps everything else on disk. An
allowlist rather than a carve-out, so a field nobody has thought about
stays human-owned by default. That also retires the last_updated
limitation #103 shipped with.
Refs #105, #103, #107, brain#21.
Compile-on-ingest, first half. buildAndWrite did an unconditional
os.WriteFile, so re-ingesting a source replaced its page wholesale and
discarded everything accumulated since — including human edits.
hyperguild#43 recorded the symptom as "re-ingest overwrites wholesale";
the framing that ingest "appends" understates it.
Now the write reads what is on disk and merges the fresh extraction into
it, asymmetrically in the existing page's favour: on-disk frontmatter and
on-disk sections win, extraction-only sections are added, bullet lists
union. An extraction can add to a note but never silently rewrite what a
human put there, which is what makes it safe on every ingest rather than
behind a review gate.
Two supporting fixes in wiki.Merge, both required for this to be safe to
run repeatedly:
- append sections (Open Questions, Updates, Evolving Notes) concatenated
wholesale, which is correct exactly once. Re-ingesting an unchanged
source would grow them every run until the page was unreadable. Now only
genuinely new lines are appended. Line-level rather than block-level,
because the common case is a re-ingest carrying one new line alongside
several it already has, and a block-level check would duplicate the old
ones every time.
- rebuildContent kept whatever blank lines a section was parsed with and
added its own, so every merge pass grew the page by a blank line —
invisible in a diff viewer, fatal to byte-stability. Section bodies are
now normalised. Re-ingesting unchanged content is byte-identical, so
brain-sync does not commit noise that hides real edits.
The drift guard that would otherwise catch repeated ingests (hashing the
source body, B3) does not exist yet, so idempotency holds here rather than
upstream.
Known limitation, deliberate: on-disk frontmatter wins, so last_updated
keeps its first-ingest date on a revised page. The alternative is this
function editing frontmatter it does not own, and no lint reads that field.
Refs hyperguild#43.
pipeline.Run parses source/author/published from the raw content's own
frontmatter and applies them deterministically to source-type RawPages
after LLM extraction — never relies on the LLM to copy them through.
buildFrontmatter now emits all three (source pages only) when present.
Backfill of the 46 already-affected wiki/sources/*.md notes is a
separate concern per the issue, not done here.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
Allows callers to provide pre-structured RawPage data directly, bypassing the
LLM extraction step. The pipeline still handles slug computation, frontmatter,
link canonicalization, source back-references, and dedup — only the extraction
is skipped. Useful when a more capable model or manual curation produces the
structured data.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Strips slug authority from the LLM. The new RawPage type carries only
{title, type, subtype, domain, content} — no paths or frontmatter.
Pipeline will derive slugs deterministically (Task 4).
pipeline.go gets a temporary bridge stub (TODO task4) to keep the
package compiling between tasks.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CLAUDE.md has a specific meaning in the Claude Code ecosystem (agent
instructions). The wiki schema for the ingestion pipeline should live
in schema.md to avoid confusion.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds prompt.go (BuildPrompt + systemPrompt) and pipeline.go (Run, Config,
Result, mergeAll) that wire chunking, LLM calls, parse, merge, index rebuild,
and log append into a single ingestion pipeline. Includes integration tests
covering write, dry-run, and duplicate-path merge scenarios.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
mathias
