feat(pipeline): ingest revises the existing page instead of clobbering it
Compile-on-ingest, first half. buildAndWrite did an unconditional os.WriteFile, so re-ingesting a source replaced its page wholesale and discarded everything accumulated since — including human edits. hyperguild#43 recorded the symptom as "re-ingest overwrites wholesale"; the framing that ingest "appends" understates it. Now the write reads what is on disk and merges the fresh extraction into it, asymmetrically in the existing page's favour: on-disk frontmatter and on-disk sections win, extraction-only sections are added, bullet lists union. An extraction can add to a note but never silently rewrite what a human put there, which is what makes it safe on every ingest rather than behind a review gate. Two supporting fixes in wiki.Merge, both required for this to be safe to run repeatedly: - append sections (Open Questions, Updates, Evolving Notes) concatenated wholesale, which is correct exactly once. Re-ingesting an unchanged source would grow them every run until the page was unreadable. Now only genuinely new lines are appended. Line-level rather than block-level, because the common case is a re-ingest carrying one new line alongside several it already has, and a block-level check would duplicate the old ones every time. - rebuildContent kept whatever blank lines a section was parsed with and added its own, so every merge pass grew the page by a blank line — invisible in a diff viewer, fatal to byte-stability. Section bodies are now normalised. Re-ingesting unchanged content is byte-identical, so brain-sync does not commit noise that hides real edits. The drift guard that would otherwise catch repeated ingests (hashing the source body, B3) does not exist yet, so idempotency holds here rather than upstream. Known limitation, deliberate: on-disk frontmatter wins, so last_updated keeps its first-ingest date on a revised page. The alternative is this function editing frontmatter it does not own, and no lint reads that field. Refs hyperguild#43.
This commit is contained in:
@@ -76,6 +76,33 @@ that prompted the question is answered by naming it here instead.
|
||||
Full reasoning, including the reversal trigger: brain
|
||||
`wiki/hyperguild/decisions/brain-mcp-stays-in-ingestion-fix-discoverability-by-name`.
|
||||
|
||||
## Ingest revises, it does not overwrite
|
||||
|
||||
Re-ingesting a source **merges into the page already on disk** rather than
|
||||
replacing it. Previously the write was unconditional, so a second ingest of the
|
||||
same source discarded everything accumulated since — including hand-written
|
||||
sections.
|
||||
|
||||
The merge is deliberately asymmetric in the existing page's favour:
|
||||
|
||||
| On the page | In the fresh extraction | Result |
|
||||
|---|---|---|
|
||||
| a section | the same section | **the on-disk version wins** |
|
||||
| — | a new section | added |
|
||||
| a bullet list (`Sources`, `Key Claims`, …) | more bullets | unioned, no duplicates |
|
||||
| an append section (`Open Questions`, `Updates`, …) | more lines | only genuinely new lines appended |
|
||||
| frontmatter | frontmatter | **the on-disk frontmatter wins** |
|
||||
|
||||
So an extraction can add to a note but never silently rewrite what a human put
|
||||
there, which is what makes it safe to run on every ingest rather than behind a
|
||||
review gate.
|
||||
|
||||
Re-ingesting unchanged content is **byte-identical** — no diff, so brain-sync
|
||||
does not commit noise that hides real edits.
|
||||
|
||||
Known limitation: because the on-disk frontmatter wins, `last_updated` keeps its
|
||||
first-ingest date on a revised page.
|
||||
|
||||
## Phase 1 tools (available now)
|
||||
|
||||
| Tool | What it does |
|
||||
|
||||
Reference in New Issue
Block a user