Ingest now stamps source_hash — sha256 of the ingested body, frontmatter
excluded — onto the source page, and compares it before doing any work. An
unchanged source returns Skipped with no LLM call at all.
This is cost and signal, not correctness: #103 already made a re-ingest
byte-identical, so a repeated run was harmless but not free — a full
extraction over every chunk to produce a result that merges to nothing.
Building this first would have put idempotency in the guard rather than in
the write, which is the wrong dependency direction.
Body-only hashing on purpose. A re-crawl that stamps a new fetch date has
not changed the source, and treating that as drift would re-extract the
whole corpus every time a crawler ran. A missing or unreadable hash always
means "extract": a missed skip costs one extraction, a wrong skip loses an
update silently, and silence is the failure mode this repo keeps paying
for.
The skip is reported (Result.Skipped plus a warning naming the hash), not
silent — a run that does nothing must say so, or "the pipeline is fine" and
"the pipeline stopped working" look identical (#107).
Storing the hash required settling who owns which frontmatter field. #103
made the on-disk block win wholesale, which protects human-set metadata
from a regenerating writer but froze fields the generator legitimately
owns. wiki.Merge now refreshes an explicit generatorOwned allowlist —
last_updated and source_hash — and keeps everything else on disk. An
allowlist rather than a carve-out, so a field nobody has thought about
stays human-owned by default. That also retires the last_updated
limitation #103 shipped with.
Refs #105, #103, #107, brain#21.
Compile-on-ingest, first half. buildAndWrite did an unconditional
os.WriteFile, so re-ingesting a source replaced its page wholesale and
discarded everything accumulated since — including human edits.
hyperguild#43 recorded the symptom as "re-ingest overwrites wholesale";
the framing that ingest "appends" understates it.
Now the write reads what is on disk and merges the fresh extraction into
it, asymmetrically in the existing page's favour: on-disk frontmatter and
on-disk sections win, extraction-only sections are added, bullet lists
union. An extraction can add to a note but never silently rewrite what a
human put there, which is what makes it safe on every ingest rather than
behind a review gate.
Two supporting fixes in wiki.Merge, both required for this to be safe to
run repeatedly:
- append sections (Open Questions, Updates, Evolving Notes) concatenated
wholesale, which is correct exactly once. Re-ingesting an unchanged
source would grow them every run until the page was unreadable. Now only
genuinely new lines are appended. Line-level rather than block-level,
because the common case is a re-ingest carrying one new line alongside
several it already has, and a block-level check would duplicate the old
ones every time.
- rebuildContent kept whatever blank lines a section was parsed with and
added its own, so every merge pass grew the page by a blank line —
invisible in a diff viewer, fatal to byte-stability. Section bodies are
now normalised. Re-ingesting unchanged content is byte-identical, so
brain-sync does not commit noise that hides real edits.
The drift guard that would otherwise catch repeated ingests (hashing the
source body, B3) does not exist yet, so idempotency holds here rather than
upstream.
Known limitation, deliberate: on-disk frontmatter wins, so last_updated
keeps its first-ingest date on a revised page. The alternative is this
function editing frontmatter it does not own, and no lint reads that field.
Refs hyperguild#43.
plainLinkRE matches any [[...]] without a pipe, including path-prefixed
forms like [[wing:homelab/failures/foo]] or [[wiki/agentsquad/decisions/bar]]
that the LLM extraction step occasionally emits. Title-lookup against
titleToSlug always fails for these (they're paths, not titles), so they
were silently left broken.
Before falling to the unknown-wikilink warning, strip a known wing:/wiki/
root prefix and emit the clean wing/hall/slug path directly — matching
the brain-graph path-style link convention.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
pipeline.Run parses source/author/published from the raw content's own
frontmatter and applies them deterministically to source-type RawPages
after LLM extraction — never relies on the LLM to copy them through.
buildFrontmatter now emits all three (source pages only) when present.
Backfill of the 46 already-affected wiki/sources/*.md notes is a
separate concern per the issue, not done here.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
Allows callers to provide pre-structured RawPage data directly, bypassing the
LLM extraction step. The pipeline still handles slug computation, frontmatter,
link canonicalization, source back-references, and dedup — only the extraction
is skipped. Useful when a more capable model or manual curation produces the
structured data.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Change prompt to reflect new output format: title, type, subtype, domain, content
- Remove slug/path generation responsibility from LLM — pipeline now handles it
- Wikilinks change from [[slug|Display Name]] to [[Display Name]] only
- LLM no longer includes frontmatter or paths in output
docs(schema): update LLM output format and wikilink convention for Level 3
- Specify JSON schema: title, type, subtype, domain, content fields
- Remove frontmatter requirements from schema output (handled by pipeline)
- Simplify wikilink format to [[Display Name]] — no slug or pipe
- Pipeline now responsible for slug generation and frontmatter construction
These changes shift slug/frontmatter generation from LLM to pipeline,
reducing cognitive load on the model and improving control over output.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Strips slug authority from the LLM. The new RawPage type carries only
{title, type, subtype, domain, content} — no paths or frontmatter.
Pipeline will derive slugs deterministically (Task 4).
pipeline.go gets a temporary bridge stub (TODO task4) to keep the
package compiling between tasks.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CLAUDE.md has a specific meaning in the Claude Code ecosystem (agent
instructions). The wiki schema for the ingestion pipeline should live
in schema.md to avoid confusion.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds prompt.go (BuildPrompt + systemPrompt) and pipeline.go (Run, Config,
Result, mergeAll) that wire chunking, LLM calls, parse, merge, index rebuild,
and log append into a single ingestion pipeline. Includes integration tests
covering write, dry-run, and duplicate-path merge scenarios.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
mathias
