4 Commits

Include renames

Author	SHA1	Message	Date
mathiasandClaude Sonnet 5	41a9112860	fix(wiki): append generator-owned frontmatter fields a page never had, not just replace ... Review of PR #126 found refreshOwned only replaced a generator-owned line that already existed in a's frontmatter -- it never appended one a didn't have. Every page ingested before this feature shipped has no source_hash line at all, so on its first re-ingest after deploy, the fresh hash was silently dropped: storedSourceHash reads "" forever for that page, the hash never matches, and the skip-on-unchanged feature this whole PR/issue exists to build never engages for the entire pre-existing corpus, permanently, with nothing telling anyone. Fixed by tracking the closing frontmatter fence's position while scanning a's lines, then inserting any generator-owned fields b supplied that a didn't have just before that fence -- sorted for a deterministic, byte-stable result, since map iteration order isn't. Flip-tested: forcing the old drop-silently return turns the new test red; restoring the fix turns it green. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-23 06:56:41 +02:00
mathias	a5a328fea5	feat(pipeline): skip extraction when the source has not changed (#105) ... Ingest now stamps source_hash — sha256 of the ingested body, frontmatter excluded — onto the source page, and compares it before doing any work. An unchanged source returns Skipped with no LLM call at all. This is cost and signal, not correctness: #103 already made a re-ingest byte-identical, so a repeated run was harmless but not free — a full extraction over every chunk to produce a result that merges to nothing. Building this first would have put idempotency in the guard rather than in the write, which is the wrong dependency direction. Body-only hashing on purpose. A re-crawl that stamps a new fetch date has not changed the source, and treating that as drift would re-extract the whole corpus every time a crawler ran. A missing or unreadable hash always means "extract": a missed skip costs one extraction, a wrong skip loses an update silently, and silence is the failure mode this repo keeps paying for. The skip is reported (Result.Skipped plus a warning naming the hash), not silent — a run that does nothing must say so, or "the pipeline is fine" and "the pipeline stopped working" look identical (#107). Storing the hash required settling who owns which frontmatter field. #103 made the on-disk block win wholesale, which protects human-set metadata from a regenerating writer but froze fields the generator legitimately owns. wiki.Merge now refreshes an explicit generatorOwned allowlist — last_updated and source_hash — and keeps everything else on disk. An allowlist rather than a carve-out, so a field nobody has thought about stays human-owned by default. That also retires the last_updated limitation #103 shipped with. Refs #105, #103, #107, brain#21.	2026-09-22 23:45:25 +02:00
mathias	f450a6c84f	feat(pipeline): ingest revises the existing page instead of clobbering it (#103) ... fix(wiki): give the preamble its own field instead of overloading heading=="" (#103) Round 3 review: verified structurally that the preamble is no longer in the sections slice or the idx map, so it cannot collide with a degenerate `## ` heading. Flip-tested by reverting to the heading=="" model and confirming TestMerge_DegenerateEmptyHeadingDoesNotCollideWithPreamble goes red. Adversarial pass (multiple degenerate headings, frontmatter-only page, unicode around the ## boundary, 3x self-merge idempotency with preamble+degenerate) all held. CI green on 09aeaa9 (run 7820). Co-authored-by: Mathias <mthbqv@gmail.com>	2026-09-22 21:43:04 +00:00
mathias	ae5a4d04f0	feat(ingestion): add wiki page merge logic	2026-04-22 22:28:55 +02:00