feat(pipeline): skip extraction when the source has not changed (#105)

Conversation 1 Commits 2 Files Changed 7

+466 -9

2 Commits

Author	SHA1	Message	Date
mathias	ce365eee44	feat(pipeline): skip extraction when the source has not changed (#105 ) Ingest now stamps source_hash — sha256 of the ingested body, frontmatter excluded — onto the source page, and compares it before doing any work. An unchanged source returns Skipped with no LLM call at all. This is cost and signal, not correctness: #103 already made a re-ingest byte-identical, so a repeated run was harmless but not free — a full extraction over every chunk to produce a result that merges to nothing. Building this first would have put idempotency in the guard rather than in the write, which is the wrong dependency direction. Body-only hashing on purpose. A re-crawl that stamps a new fetch date has not changed the source, and treating that as drift would re-extract the whole corpus every time a crawler ran. A missing or unreadable hash always means "extract": a missed skip costs one extraction, a wrong skip loses an update silently, and silence is the failure mode this repo keeps paying for. The skip is reported (Result.Skipped plus a warning naming the hash), not silent — a run that does nothing must say so, or "the pipeline is fine" and "the pipeline stopped working" look identical (#107). Storing the hash required settling who owns which frontmatter field. #103 made the on-disk block win wholesale, which protects human-set metadata from a regenerating writer but froze fields the generator legitimately owns. wiki.Merge now refreshes an explicit generatorOwned allowlist — last_updated and source_hash — and keeps everything else on disk. An allowlist rather than a carve-out, so a field nobody has thought about stays human-owned by default. That also retires the last_updated limitation #103 shipped with. Refs #105, #103, #107, brain#21.	2026-08-22 16:15:56 +02:00
mathias	60a8801a1d	feat(pipeline): ingest revises the existing page instead of clobbering it CI / Lint / Test / Vet (pull_request) Successful in 38s Details CI / Mirror to GitHub (pull_request) Skipped Details Compile-on-ingest, first half. buildAndWrite did an unconditional os.WriteFile, so re-ingesting a source replaced its page wholesale and discarded everything accumulated since — including human edits. hyperguild#43 recorded the symptom as "re-ingest overwrites wholesale"; the framing that ingest "appends" understates it. Now the write reads what is on disk and merges the fresh extraction into it, asymmetrically in the existing page's favour: on-disk frontmatter and on-disk sections win, extraction-only sections are added, bullet lists union. An extraction can add to a note but never silently rewrite what a human put there, which is what makes it safe on every ingest rather than behind a review gate. Two supporting fixes in wiki.Merge, both required for this to be safe to run repeatedly: - append sections (Open Questions, Updates, Evolving Notes) concatenated wholesale, which is correct exactly once. Re-ingesting an unchanged source would grow them every run until the page was unreadable. Now only genuinely new lines are appended. Line-level rather than block-level, because the common case is a re-ingest carrying one new line alongside several it already has, and a block-level check would duplicate the old ones every time. - rebuildContent kept whatever blank lines a section was parsed with and added its own, so every merge pass grew the page by a blank line — invisible in a diff viewer, fatal to byte-stability. Section bodies are now normalised. Re-ingesting unchanged content is byte-identical, so brain-sync does not commit noise that hides real edits. The drift guard that would otherwise catch repeated ingests (hashing the source body, B3) does not exist yet, so idempotency holds here rather than upstream. Known limitation, deliberate: on-disk frontmatter wins, so last_updated keeps its first-ingest date on a revised page. The alternative is this function editing frontmatter it does not own, and no lint reads that field. Refs hyperguild#43.	2026-08-22 13:19:14 +02:00

Author

SHA1

Message

Date

mathias

ce365eee44

feat(pipeline): skip extraction when the source has not changed (#105 )

Ingest now stamps source_hash — sha256 of the ingested body, frontmatter
excluded — onto the source page, and compares it before doing any work. An
unchanged source returns Skipped with no LLM call at all.

This is cost and signal, not correctness: #103 already made a re-ingest
byte-identical, so a repeated run was harmless but not free — a full
extraction over every chunk to produce a result that merges to nothing.
Building this first would have put idempotency in the guard rather than in
the write, which is the wrong dependency direction.

Body-only hashing on purpose. A re-crawl that stamps a new fetch date has
not changed the source, and treating that as drift would re-extract the
whole corpus every time a crawler ran. A missing or unreadable hash always
means "extract": a missed skip costs one extraction, a wrong skip loses an
update silently, and silence is the failure mode this repo keeps paying
for.

The skip is reported (Result.Skipped plus a warning naming the hash), not
silent — a run that does nothing must say so, or "the pipeline is fine" and
"the pipeline stopped working" look identical (#107).

Storing the hash required settling who owns which frontmatter field. #103
made the on-disk block win wholesale, which protects human-set metadata
from a regenerating writer but froze fields the generator legitimately
owns. wiki.Merge now refreshes an explicit generatorOwned allowlist —
last_updated and source_hash — and keeps everything else on disk. An
allowlist rather than a carve-out, so a field nobody has thought about
stays human-owned by default. That also retires the last_updated
limitation #103 shipped with.

Refs #105, #103, #107, brain#21.

2026-08-22 16:15:56 +02:00

mathias

60a8801a1d

feat(pipeline): ingest revises the existing page instead of clobbering it

CI / Lint / Test / Vet (pull_request) Successful in 38s

Details

CI / Mirror to GitHub (pull_request) Skipped

Details

Compile-on-ingest, first half. buildAndWrite did an unconditional
os.WriteFile, so re-ingesting a source replaced its page wholesale and
discarded everything accumulated since — including human edits.
hyperguild#43 recorded the symptom as "re-ingest overwrites wholesale";
the framing that ingest "appends" understates it.

Now the write reads what is on disk and merges the fresh extraction into
it, asymmetrically in the existing page's favour: on-disk frontmatter and
on-disk sections win, extraction-only sections are added, bullet lists
union. An extraction can add to a note but never silently rewrite what a
human put there, which is what makes it safe on every ingest rather than
behind a review gate.

Two supporting fixes in wiki.Merge, both required for this to be safe to
run repeatedly:

- append sections (Open Questions, Updates, Evolving Notes) concatenated
  wholesale, which is correct exactly once. Re-ingesting an unchanged
  source would grow them every run until the page was unreadable. Now only
  genuinely new lines are appended. Line-level rather than block-level,
  because the common case is a re-ingest carrying one new line alongside
  several it already has, and a block-level check would duplicate the old
  ones every time.

- rebuildContent kept whatever blank lines a section was parsed with and
  added its own, so every merge pass grew the page by a blank line —
  invisible in a diff viewer, fatal to byte-stability. Section bodies are
  now normalised. Re-ingesting unchanged content is byte-identical, so
  brain-sync does not commit noise that hides real edits.

The drift guard that would otherwise catch repeated ingests (hashing the
source body, B3) does not exist yet, so idempotency holds here rather than
upstream.

Known limitation, deliberate: on-disk frontmatter wins, so last_updated
keeps its first-ingest date on a revised page. The alternative is this
function editing frontmatter it does not own, and no lint reads that field.

Refs hyperguild#43.

2026-08-22 13:19:14 +02:00

feat(pipeline): skip extraction when the source has not changed (#105) #109