feat(pipeline): report when a re-ingest disagrees with the page on disk (#104)

Conversation 1 Commits 2 Files Changed 7

+389 -14

2 Commits

Author	SHA1	Message	Date
mathias	9345a0a867	feat(pipeline): report when a re-ingest disagrees with the page on disk (#104 ) CI / Lint / Test / Vet (pull_request) Successful in 34s Details CI / Mirror to GitHub (pull_request) Skipped Details The asymmetry that makes unattended ingest safe (#103) also discarded a real signal. When a fresh extraction produced a different version of a section already on disk, the on-disk version won and the model's version went in the bin without a word — so "the source is unchanged" and "the source changed its mind and we ignored it" were indistinguishable. wiki.MergeReport now returns the headings where the incoming version disagreed and lost, and the ingest run reports them as warnings, which AppendLog already writes into log.md. This is the simplified answer to contradiction flagging at ingest. The issue proposed calling brain-gardener's Judge from the ingest path, which needs either a shared Go module across two repos or an HTTP service that does not exist — and puts an LLM call on the write path, where a timeout must neither fail the ingest nor silently mark nothing as contested. None of that is necessary to get the signal. The merge already computes the disagreement; it was throwing it away. Surfacing it costs nothing, needs no model, no cue-word matcher, no cross-repo dependency, and is deterministic. Deliberately textual disagreement, not semantic contradiction. Deciding whether a disagreement is a genuine contradiction stays with the Judge-gated enrich pass that already runs hourly and is approval-gated by design (brain-gardener#175). Detection here, adjudication there. Merge is kept as a thin wrapper over MergeReport so existing callers are untouched. Refs #104, #103, brain-gardener#175, brain#21.	2026-09-23 06:51:49 +02:00
mathias	a5a328fea5	feat(pipeline): skip extraction when the source has not changed (#105 ) CI / Lint / Test / Vet (pull_request) Successful in 34s Details CI / Mirror to GitHub (pull_request) Skipped Details Ingest now stamps source_hash — sha256 of the ingested body, frontmatter excluded — onto the source page, and compares it before doing any work. An unchanged source returns Skipped with no LLM call at all. This is cost and signal, not correctness: #103 already made a re-ingest byte-identical, so a repeated run was harmless but not free — a full extraction over every chunk to produce a result that merges to nothing. Building this first would have put idempotency in the guard rather than in the write, which is the wrong dependency direction. Body-only hashing on purpose. A re-crawl that stamps a new fetch date has not changed the source, and treating that as drift would re-extract the whole corpus every time a crawler ran. A missing or unreadable hash always means "extract": a missed skip costs one extraction, a wrong skip loses an update silently, and silence is the failure mode this repo keeps paying for. The skip is reported (Result.Skipped plus a warning naming the hash), not silent — a run that does nothing must say so, or "the pipeline is fine" and "the pipeline stopped working" look identical (#107). Storing the hash required settling who owns which frontmatter field. #103 made the on-disk block win wholesale, which protects human-set metadata from a regenerating writer but froze fields the generator legitimately owns. wiki.Merge now refreshes an explicit generatorOwned allowlist — last_updated and source_hash — and keeps everything else on disk. An allowlist rather than a carve-out, so a field nobody has thought about stays human-owned by default. That also retires the last_updated limitation #103 shipped with. Refs #105, #103, #107, brain#21.	2026-09-22 23:45:25 +02:00

Author

SHA1

Message

Date

mathias

9345a0a867

feat(pipeline): report when a re-ingest disagrees with the page on disk (#104 )

CI / Lint / Test / Vet (pull_request) Successful in 34s

Details

CI / Mirror to GitHub (pull_request) Skipped

Details

The asymmetry that makes unattended ingest safe (#103) also discarded a
real signal. When a fresh extraction produced a different version of a
section already on disk, the on-disk version won and the model's version
went in the bin without a word — so "the source is unchanged" and "the
source changed its mind and we ignored it" were indistinguishable.

wiki.MergeReport now returns the headings where the incoming version
disagreed and lost, and the ingest run reports them as warnings, which
AppendLog already writes into log.md.

This is the simplified answer to contradiction flagging at ingest. The
issue proposed calling brain-gardener's Judge from the ingest path, which
needs either a shared Go module across two repos or an HTTP service that
does not exist — and puts an LLM call on the write path, where a timeout
must neither fail the ingest nor silently mark nothing as contested.

None of that is necessary to get the signal. The merge already computes
the disagreement; it was throwing it away. Surfacing it costs nothing,
needs no model, no cue-word matcher, no cross-repo dependency, and is
deterministic.

Deliberately textual disagreement, not semantic contradiction. Deciding
whether a disagreement is a genuine contradiction stays with the
Judge-gated enrich pass that already runs hourly and is approval-gated by
design (brain-gardener#175). Detection here, adjudication there.

Merge is kept as a thin wrapper over MergeReport so existing callers are
untouched.

Refs #104, #103, brain-gardener#175, brain#21.

2026-09-23 06:51:49 +02:00

mathias

a5a328fea5

feat(pipeline): skip extraction when the source has not changed (#105 )

CI / Lint / Test / Vet (pull_request) Successful in 34s

Details

CI / Mirror to GitHub (pull_request) Skipped

Details

Ingest now stamps source_hash — sha256 of the ingested body, frontmatter
excluded — onto the source page, and compares it before doing any work. An
unchanged source returns Skipped with no LLM call at all.

This is cost and signal, not correctness: #103 already made a re-ingest
byte-identical, so a repeated run was harmless but not free — a full
extraction over every chunk to produce a result that merges to nothing.
Building this first would have put idempotency in the guard rather than in
the write, which is the wrong dependency direction.

Body-only hashing on purpose. A re-crawl that stamps a new fetch date has
not changed the source, and treating that as drift would re-extract the
whole corpus every time a crawler ran. A missing or unreadable hash always
means "extract": a missed skip costs one extraction, a wrong skip loses an
update silently, and silence is the failure mode this repo keeps paying
for.

The skip is reported (Result.Skipped plus a warning naming the hash), not
silent — a run that does nothing must say so, or "the pipeline is fine" and
"the pipeline stopped working" look identical (#107).

Storing the hash required settling who owns which frontmatter field. #103
made the on-disk block win wholesale, which protects human-set metadata
from a regenerating writer but froze fields the generator legitimately
owns. wiki.Merge now refreshes an explicit generatorOwned allowlist —
last_updated and source_hash — and keeps everything else on disk. An
allowlist rather than a carve-out, so a field nobody has thought about
stays human-owned by default. That also retires the last_updated
limitation #103 shipped with.

Refs #105, #103, #107, brain#21.

2026-09-22 23:45:25 +02:00

feat(pipeline): report when a re-ingest disagrees with the page on disk (#104) #127