feat(pipeline): skip extraction when the source has not changed (#105) (#126)
CI / Lint / Test / Vet (push) Successful in 40s
CI / Mirror to GitHub (push) Successful in 4s

Co-authored-by: Mathias <mthbqv@gmail.com>
This commit was merged in pull request #126.
This commit is contained in:
2026-09-23 05:09:30 +00:00
committed by mathias
parent f450a6c84f
commit b340f86a45
7 changed files with 457 additions and 8 deletions
+15 -2
View File
@@ -100,8 +100,21 @@ review gate.
Re-ingesting unchanged content is **byte-identical** — no diff, so brain-sync
does not commit noise that hides real edits.
Known limitation: because the on-disk frontmatter wins, `last_updated` keeps its
first-ingest date on a revised page.
Two fields are **generator-owned** and do get refreshed on a revise:
`last_updated` and `source_hash`. Everything else on disk wins. The boundary is
an explicit allowlist, so a field nobody has thought about stays human-owned.
## Re-ingesting an unchanged source is skipped
Ingest stores `source_hash` — sha256 of the ingested **body**, frontmatter
excluded — on the source page. A later ingest of the same source compares it
first and skips extraction entirely when it matches, reporting the skip rather
than doing it silently.
Body-only is deliberate: a re-crawl that stamps a new fetch date has not changed
the source, and treating that as drift would re-extract the corpus every time a
crawler ran. A missing or unreadable hash always means "extract" — a missed skip
costs one extraction, a wrong skip loses an update.
## Phase 1 tools (available now)