feat(pipeline): skip extraction when the source has not changed (#105)
Ingest now stamps source_hash — sha256 of the ingested body, frontmatter excluded — onto the source page, and compares it before doing any work. An unchanged source returns Skipped with no LLM call at all. This is cost and signal, not correctness: #103 already made a re-ingest byte-identical, so a repeated run was harmless but not free — a full extraction over every chunk to produce a result that merges to nothing. Building this first would have put idempotency in the guard rather than in the write, which is the wrong dependency direction. Body-only hashing on purpose. A re-crawl that stamps a new fetch date has not changed the source, and treating that as drift would re-extract the whole corpus every time a crawler ran. A missing or unreadable hash always means "extract": a missed skip costs one extraction, a wrong skip loses an update silently, and silence is the failure mode this repo keeps paying for. The skip is reported (Result.Skipped plus a warning naming the hash), not silent — a run that does nothing must say so, or "the pipeline is fine" and "the pipeline stopped working" look identical (#107). Storing the hash required settling who owns which frontmatter field. #103 made the on-disk block win wholesale, which protects human-set metadata from a regenerating writer but froze fields the generator legitimately owns. wiki.Merge now refreshes an explicit generatorOwned allowlist — last_updated and source_hash — and keeps everything else on disk. An allowlist rather than a carve-out, so a field nobody has thought about stays human-owned by default. That also retires the last_updated limitation #103 shipped with. Refs #105, #103, #107, brain#21.
This commit is contained in:
@@ -38,7 +38,7 @@ type section struct {
|
||||
// not present in a are appended.
|
||||
func Merge(a, b Page) Page {
|
||||
fmA, preambleA, secsA := parseSections(a.Content)
|
||||
_, _, secsB := parseSections(b.Content)
|
||||
fmB, _, secsB := parseSections(b.Content)
|
||||
|
||||
idx := make(map[string]int, len(secsA))
|
||||
for i, s := range secsA {
|
||||
@@ -61,7 +61,56 @@ func Merge(a, b Page) Page {
|
||||
}
|
||||
}
|
||||
|
||||
return Page{Path: a.Path, Content: rebuildContent(fmA, preambleA, secsA)}
|
||||
return Page{Path: a.Path, Content: rebuildContent(refreshOwned(fmA, fmB), preambleA, secsA)}
|
||||
}
|
||||
|
||||
// generatorOwned names the frontmatter fields the ingest pipeline may refresh
|
||||
// on a page it is revising. Everything else on disk wins.
|
||||
//
|
||||
// #103 made the on-disk frontmatter win wholesale, which is what stops a
|
||||
// regenerating writer from overwriting human-set metadata — a title someone
|
||||
// fixed, or the claim-property envelope (brain#21). The cost was that fields
|
||||
// the generator legitimately owns froze at their first-ingest values: when it
|
||||
// last ran, and what it last saw.
|
||||
//
|
||||
// An explicit allowlist rather than a carve-out: the ownership boundary is
|
||||
// nameable, and a field nobody has thought about stays human-owned by default,
|
||||
// which is the safe direction.
|
||||
var generatorOwned = map[string]bool{
|
||||
"last_updated": true,
|
||||
"source_hash": true,
|
||||
}
|
||||
|
||||
// refreshOwned returns a's frontmatter with the generator-owned fields taken
|
||||
// from b where b supplies them. Fields absent from b keep a's value, so a
|
||||
// generator that stops emitting a field cannot blank it.
|
||||
func refreshOwned(a, b string) string {
|
||||
bVals := make(map[string]string)
|
||||
for _, line := range strings.Split(b, "\n") {
|
||||
key, value, found := strings.Cut(line, ":")
|
||||
if !found || !generatorOwned[strings.TrimSpace(key)] {
|
||||
continue
|
||||
}
|
||||
bVals[strings.TrimSpace(key)] = strings.TrimSpace(value)
|
||||
}
|
||||
if len(bVals) == 0 {
|
||||
return a
|
||||
}
|
||||
|
||||
out := make([]string, 0, len(strings.Split(a, "\n")))
|
||||
for _, line := range strings.Split(a, "\n") {
|
||||
key, _, found := strings.Cut(line, ":")
|
||||
k := strings.TrimSpace(key)
|
||||
if found && generatorOwned[k] {
|
||||
if v, ok := bVals[k]; ok {
|
||||
out = append(out, k+": "+v)
|
||||
delete(bVals, k)
|
||||
continue
|
||||
}
|
||||
}
|
||||
out = append(out, line)
|
||||
}
|
||||
return strings.Join(out, "\n")
|
||||
}
|
||||
|
||||
// parseSections splits markdown into its frontmatter, its preamble (content
|
||||
|
||||
Reference in New Issue
Block a user