feat(pipeline): skip extraction when the source has not changed (#105) #109

2026-08-22T14:16:21Z

mathias commented

2026-08-22 14:16:21 +00:00

Closes #105. Stacked on #103 — base is agent/b1-compile-on-ingest, not main, because the hash lives in frontmatter that only survives a re-ingest because of #103's merge. Review #103 first; this retargets to main automatically once that lands.

What

Ingest stamps source_hash — sha256 of the ingested body, frontmatter excluded — onto the source page, and checks it before doing any work. An unchanged source returns early with zero LLM calls.

Cost and signal, not correctness

Worth being precise, because the plan had this the other way round. #103 already made a re-ingest byte-identical, so repeating one was harmless — just not free: a full extraction over every chunk to produce a result that merges to nothing.

Building this first would have put idempotency in the guard rather than in the write. The write has to be safe on its own; this is an optimisation on top.

The second half is signal: "this source changed since we ingested it" is information the corpus doesn't have today, and it's the natural input to half_life (brain#21) — a volatile note whose source has changed is the highest-value re-verification candidate there is.

Three deliberate choices

Body-only hashing. A re-crawl that stamps a new fetch date hasn't changed the source. Hashing the whole file would re-extract the corpus every time a crawler ran. Tested: TestRun_FrontmatterOnlyChangeIsNotDrift.

Absence means extract. No page, no field, unreadable file → extract. A missed skip costs one extraction; a wrong skip loses an update silently, and silence is the failure mode this repo keeps paying for (#100).

The skip is reported. Result.Skipped plus a warning naming the hash. A run that does nothing must say so, or "the pipeline is fine" and "the pipeline stopped working" look identical — the same argument as #107.

The frontmatter-ownership question, settled

#105 asked where the hash should live given #103 made the on-disk block win wholesale. Answer: keep it in frontmatter (survives a corpus move, human-visible) and make ownership explicit.

wiki.Merge now refreshes a named generatorOwned set — last_updated, source_hash — and keeps everything else on disk. An allowlist, not a carve-out: a field nobody has thought about stays human-owned by default, which is the safe direction.

That also retires the last_updated limitation #103 shipped with, and it's tested against the fields that matter — a human-fixed title and the claim envelope (half_life, claim_provenance) both survive regeneration.

Tests

Red-first. TestRun_SkipsExtractionWhenTheSourceIsUnchanged and TestRun_FrontmatterOnlyChangeIsNotDrift failed by extracting twice; TestRun_ReExtractsWhenTheSourceBodyChanged passed from the start (today it always extracts) and guards against over-skipping. TestMerge_GeneratorOwnedFrontmatterFieldsAreRefreshed failed on frozen fields.

Call-counting stub rather than an HTTP fixture, so "did it call the model" is asserted directly.

task check exit 0, full suite green.

Note on the rebase

I first branched this off main and the helper insertion silently no-opped, because main doesn't have #103's function that my anchor targeted. Caught by the build, not by review. Rebased onto the B1 branch and resolved the append-vs-append test conflicts by keeping both sides.

Risk: LOW — the new early return only fires on an exact hash match, and every unknown state falls through to today's behaviour.

Closes #105. **Stacked on #103** — base is `agent/b1-compile-on-ingest`, not `main`, because the hash lives in frontmatter that only survives a re-ingest because of #103's merge. Review #103 first; this retargets to `main` automatically once that lands. ## What Ingest stamps `source_hash` — sha256 of the ingested **body**, frontmatter excluded — onto the source page, and checks it before doing any work. An unchanged source returns early with **zero LLM calls**. ## Cost and signal, not correctness Worth being precise, because the plan had this the other way round. #103 already made a re-ingest byte-identical, so repeating one was harmless — just not free: a full extraction over every chunk to produce a result that merges to nothing. Building this *first* would have put idempotency in the guard rather than in the write. The write has to be safe on its own; this is an optimisation on top. The second half is signal: *"this source changed since we ingested it"* is information the corpus doesn't have today, and it's the natural input to `half_life` (brain#21) — a `volatile` note whose source has changed is the highest-value re-verification candidate there is. ## Three deliberate choices **Body-only hashing.** A re-crawl that stamps a new fetch date hasn't changed the source. Hashing the whole file would re-extract the corpus every time a crawler ran. Tested: `TestRun_FrontmatterOnlyChangeIsNotDrift`. **Absence means extract.** No page, no field, unreadable file → extract. A missed skip costs one extraction; a wrong skip loses an update silently, and silence is the failure mode this repo keeps paying for (#100). **The skip is reported.** `Result.Skipped` plus a warning naming the hash. A run that does nothing must say so, or "the pipeline is fine" and "the pipeline stopped working" look identical — the same argument as #107. ## The frontmatter-ownership question, settled #105 asked where the hash should live given #103 made the on-disk block win wholesale. Answer: keep it in frontmatter (survives a corpus move, human-visible) and make ownership **explicit**. `wiki.Merge` now refreshes a named `generatorOwned` set — `last_updated`, `source_hash` — and keeps everything else on disk. An **allowlist, not a carve-out**: a field nobody has thought about stays human-owned by default, which is the safe direction. That also retires the `last_updated` limitation #103 shipped with, and it's tested against the fields that matter — a human-fixed `title` and the claim envelope (`half_life`, `claim_provenance`) both survive regeneration. ## Tests Red-first. `TestRun_SkipsExtractionWhenTheSourceIsUnchanged` and `TestRun_FrontmatterOnlyChangeIsNotDrift` failed by extracting twice; `TestRun_ReExtractsWhenTheSourceBodyChanged` passed from the start (today it always extracts) and guards against over-skipping. `TestMerge_GeneratorOwnedFrontmatterFieldsAreRefreshed` failed on frozen fields. Call-counting stub rather than an HTTP fixture, so "did it call the model" is asserted directly. `task check` exit 0, full suite green. ## Note on the rebase I first branched this off `main` and the helper insertion silently no-opped, because `main` doesn't have #103's function that my anchor targeted. Caught by the build, not by review. Rebased onto the B1 branch and resolved the append-vs-append test conflicts by keeping both sides. **Risk:** LOW — the new early return only fires on an exact hash match, and every unknown state falls through to today's behaviour.

mathias referenced this pull request

2026-08-22 20:52:54 +00:00

feat(pipeline): report when a re-ingest disagrees with the page on disk (#104) #110

mathias changed target branch from agent/b1-compile-on-ingest to main

2026-09-22 21:44:16 +00:00

mathias added 2 commits 2026-09-22 21:44:16 +00:00

feat(pipeline): ingest revises the existing page instead of clobbering it

CI / Lint / Test / Vet (pull_request) Successful in 38s

Details

CI / Mirror to GitHub (pull_request) Skipped

Details

60a8801a1d

Compile-on-ingest, first half. buildAndWrite did an unconditional
os.WriteFile, so re-ingesting a source replaced its page wholesale and
discarded everything accumulated since — including human edits.
hyperguild#43 recorded the symptom as "re-ingest overwrites wholesale";
the framing that ingest "appends" understates it.

Now the write reads what is on disk and merges the fresh extraction into
it, asymmetrically in the existing page's favour: on-disk frontmatter and
on-disk sections win, extraction-only sections are added, bullet lists
union. An extraction can add to a note but never silently rewrite what a
human put there, which is what makes it safe on every ingest rather than
behind a review gate.

Two supporting fixes in wiki.Merge, both required for this to be safe to
run repeatedly:

- append sections (Open Questions, Updates, Evolving Notes) concatenated
  wholesale, which is correct exactly once. Re-ingesting an unchanged
  source would grow them every run until the page was unreadable. Now only
  genuinely new lines are appended. Line-level rather than block-level,
  because the common case is a re-ingest carrying one new line alongside
  several it already has, and a block-level check would duplicate the old
  ones every time.

- rebuildContent kept whatever blank lines a section was parsed with and
  added its own, so every merge pass grew the page by a blank line —
  invisible in a diff viewer, fatal to byte-stability. Section bodies are
  now normalised. Re-ingesting unchanged content is byte-identical, so
  brain-sync does not commit noise that hides real edits.

The drift guard that would otherwise catch repeated ingests (hashing the
source body, B3) does not exist yet, so idempotency holds here rather than
upstream.

Known limitation, deliberate: on-disk frontmatter wins, so last_updated
keeps its first-ingest date on a revised page. The alternative is this
function editing frontmatter it does not own, and no lint reads that field.

Refs hyperguild#43.

feat(pipeline): skip extraction when the source has not changed (#105 ) ce365eee44

Ingest now stamps source_hash — sha256 of the ingested body, frontmatter
excluded — onto the source page, and compares it before doing any work. An
unchanged source returns Skipped with no LLM call at all.

This is cost and signal, not correctness: #103 already made a re-ingest
byte-identical, so a repeated run was harmless but not free — a full
extraction over every chunk to produce a result that merges to nothing.
Building this first would have put idempotency in the guard rather than in
the write, which is the wrong dependency direction.

Body-only hashing on purpose. A re-crawl that stamps a new fetch date has
not changed the source, and treating that as drift would re-extract the
whole corpus every time a crawler ran. A missing or unreadable hash always
means "extract": a missed skip costs one extraction, a wrong skip loses an
update silently, and silence is the failure mode this repo keeps paying
for.

The skip is reported (Result.Skipped plus a warning naming the hash), not
silent — a run that does nothing must say so, or "the pipeline is fine" and
"the pipeline stopped working" look identical (#107).

Storing the hash required settling who owns which frontmatter field. #103
made the on-disk block win wholesale, which protects human-set metadata
from a regenerating writer but froze fields the generator legitimately
owns. wiki.Merge now refreshes an explicit generatorOwned allowlist —
last_updated and source_hash — and keeps everything else on disk. An
allowlist rather than a carve-out, so a field nobody has thought about
stays human-owned by default. That also retires the last_updated
limitation #103 shipped with.

Refs #105, #103, #107, brain#21.

mathias referenced this pull request

2026-09-23 04:50:49 +00:00

feat(pipeline): skip extraction when the source has not changed (#105) #126

mathias commented

2026-09-23 04:50:53 +00:00

Superseded by #126: same commit, rebased onto current main (this branch's base, agent/b1-compile-on-ingest, merged as #103 with two review-driven fixes on top that this branch predates). Closing rather than force-pushing a rewrite over this branch.

Created via git-mcp on behalf of @mathias

Superseded by #126: same commit, rebased onto current `main` (this branch's base, `agent/b1-compile-on-ingest`, merged as #103 with two review-driven fixes on top that this branch predates). Closing rather than force-pushing a rewrite over this branch. --- _Created via git-mcp on behalf of @mathias_

mathias closed this pull request

2026-09-23 04:51:00 +00:00

mathias referenced this pull request

2026-09-23 04:51:58 +00:00

feat(pipeline): report when a re-ingest disagrees with the page on disk (#104) #127

mathias referenced this pull request

2026-09-23 04:52:10 +00:00

feat(pipeline): report when a re-ingest disagrees with the page on disk (#104) #110