feat(pipeline): report when a re-ingest disagrees with the page on disk (#104) #127

2026-09-23T04:51:58Z

mathias commented

2026-09-23 04:51:58 +00:00

Replaces #110, which was stacked on #109 (now closed/replaced by #126). Same single commit, rebased onto #126's branch instead of force-pushing over the old stale one (same reasoning as #126 itself).

One real conflict in ingestion/internal/wiki/merge.go: #110 refactors Merge into a thin wrapper around a new MergeReport(a, b) (Page, []string) (reports section headings where b disagreed with a and lost, rather than discarding silently), while #103's later fix threads preambleA through as a separate parseSections/rebuildContent argument. Resolved by keeping both: MergeReport now returns preambleA through to rebuildContent alongside the refreshOwned frontmatter and the existing conflicts slice.

Full module go build ./... && go vet ./... && go test ./... -race -count=1 green after resolution.

Depends on #126 (base branch) merging first — same stacking order as before, just rebuilt on the surviving commit.

Closes #104.

Created via git-mcp on behalf of @mathias

Replaces #110, which was stacked on #109 (now closed/replaced by #126). Same single commit, rebased onto #126's branch instead of force-pushing over the old stale one (same reasoning as #126 itself). One real conflict in `ingestion/internal/wiki/merge.go`: #110 refactors `Merge` into a thin wrapper around a new `MergeReport(a, b) (Page, []string)` (reports section headings where `b` disagreed with `a` and lost, rather than discarding silently), while #103's later fix threads `preambleA` through as a separate `parseSections`/`rebuildContent` argument. Resolved by keeping both: `MergeReport` now returns `preambleA` through to `rebuildContent` alongside the `refreshOwned` frontmatter and the existing `conflicts` slice. Full module `go build ./... && go vet ./... && go test ./... -race -count=1` green after resolution. Depends on #126 (base branch) merging first — same stacking order as before, just rebuilt on the surviving commit. Closes #104. --- _Created via git-mcp on behalf of @mathias_

mathias changed target branch from main to fix/105-source-drift-on-main

2026-09-23 04:52:07 +00:00

mathias referenced this pull request

2026-09-23 04:52:10 +00:00

feat(pipeline): report when a re-ingest disagrees with the page on disk (#104) #110

mathias referenced this pull request

2026-09-23 04:54:26 +00:00

Source drift: hash the ingested body, skip an unchanged re-ingest, flag a changed one #105

mathias referenced this pull request

2026-09-23 05:05:30 +00:00

Source drift: hash the ingested body, skip an unchanged re-ingest, flag a changed one #105

mathias referenced this pull request

2026-09-23 05:09:50 +00:00

Source drift: hash the ingested body, skip an unchanged re-ingest, flag a changed one #105

mathias changed target branch from fix/105-source-drift-on-main to main

2026-09-23 05:10:56 +00:00

mathias added 2 commits 2026-09-23 05:10:56 +00:00

feat(pipeline): skip extraction when the source has not changed (#105 )

CI / Lint / Test / Vet (pull_request) Successful in 34s

Details

CI / Mirror to GitHub (pull_request) Skipped

Details

a5a328fea5

Ingest now stamps source_hash — sha256 of the ingested body, frontmatter
excluded — onto the source page, and compares it before doing any work. An
unchanged source returns Skipped with no LLM call at all.

This is cost and signal, not correctness: #103 already made a re-ingest
byte-identical, so a repeated run was harmless but not free — a full
extraction over every chunk to produce a result that merges to nothing.
Building this first would have put idempotency in the guard rather than in
the write, which is the wrong dependency direction.

Body-only hashing on purpose. A re-crawl that stamps a new fetch date has
not changed the source, and treating that as drift would re-extract the
whole corpus every time a crawler ran. A missing or unreadable hash always
means "extract": a missed skip costs one extraction, a wrong skip loses an
update silently, and silence is the failure mode this repo keeps paying
for.

The skip is reported (Result.Skipped plus a warning naming the hash), not
silent — a run that does nothing must say so, or "the pipeline is fine" and
"the pipeline stopped working" look identical (#107).

Storing the hash required settling who owns which frontmatter field. #103
made the on-disk block win wholesale, which protects human-set metadata
from a regenerating writer but froze fields the generator legitimately
owns. wiki.Merge now refreshes an explicit generatorOwned allowlist —
last_updated and source_hash — and keeps everything else on disk. An
allowlist rather than a carve-out, so a field nobody has thought about
stays human-owned by default. That also retires the last_updated
limitation #103 shipped with.

Refs #105, #103, #107, brain#21.

feat(pipeline): report when a re-ingest disagrees with the page on disk (#104 )

CI / Lint / Test / Vet (pull_request) Successful in 34s

Details

CI / Mirror to GitHub (pull_request) Skipped

Details

9345a0a867

The asymmetry that makes unattended ingest safe (#103) also discarded a
real signal. When a fresh extraction produced a different version of a
section already on disk, the on-disk version won and the model's version
went in the bin without a word — so "the source is unchanged" and "the
source changed its mind and we ignored it" were indistinguishable.

wiki.MergeReport now returns the headings where the incoming version
disagreed and lost, and the ingest run reports them as warnings, which
AppendLog already writes into log.md.

This is the simplified answer to contradiction flagging at ingest. The
issue proposed calling brain-gardener's Judge from the ingest path, which
needs either a shared Go module across two repos or an HTTP service that
does not exist — and puts an LLM call on the write path, where a timeout
must neither fail the ingest nor silently mark nothing as contested.

None of that is necessary to get the signal. The merge already computes
the disagreement; it was throwing it away. Surfacing it costs nothing,
needs no model, no cue-word matcher, no cross-repo dependency, and is
deterministic.

Deliberately textual disagreement, not semantic contradiction. Deciding
whether a disagreement is a genuine contradiction stays with the
Judge-gated enrich pass that already runs hourly and is approval-gated by
design (brain-gardener#175). Detection here, adjudication there.

Merge is kept as a thin wrapper over MergeReport so existing callers are
untouched.

Refs #104, #103, brain-gardener#175, brain#21.

mathias referenced this pull request

2026-09-23 05:12:04 +00:00

feat(pipeline): report when a re-ingest disagrees with the page on disk (#104) #129

mathias commented

2026-09-23 05:12:08 +00:00

Superseded by #129: same commit, rebased onto main now that #126 (this branch's base) merged via squash. Same non-force-push pattern as #126/#109 earlier in this sprint.

Created via git-mcp on behalf of @mathias

Superseded by #129: same commit, rebased onto main now that #126 (this branch's base) merged via squash. Same non-force-push pattern as #126/#109 earlier in this sprint. --- _Created via git-mcp on behalf of @mathias_

mathias closed this pull request

2026-09-23 05:12:09 +00:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mathias/hyperguild#127