5 Commits

Include renames

Author	SHA1	Message	Date
mathias	a5a328fea5	feat(pipeline): skip extraction when the source has not changed (#105) ... Ingest now stamps source_hash — sha256 of the ingested body, frontmatter excluded — onto the source page, and compares it before doing any work. An unchanged source returns Skipped with no LLM call at all. This is cost and signal, not correctness: #103 already made a re-ingest byte-identical, so a repeated run was harmless but not free — a full extraction over every chunk to produce a result that merges to nothing. Building this first would have put idempotency in the guard rather than in the write, which is the wrong dependency direction. Body-only hashing on purpose. A re-crawl that stamps a new fetch date has not changed the source, and treating that as drift would re-extract the whole corpus every time a crawler ran. A missing or unreadable hash always means "extract": a missed skip costs one extraction, a wrong skip loses an update silently, and silence is the failure mode this repo keeps paying for. The skip is reported (Result.Skipped plus a warning naming the hash), not silent — a run that does nothing must say so, or "the pipeline is fine" and "the pipeline stopped working" look identical (#107). Storing the hash required settling who owns which frontmatter field. #103 made the on-disk block win wholesale, which protects human-set metadata from a regenerating writer but froze fields the generator legitimately owns. wiki.Merge now refreshes an explicit generatorOwned allowlist — last_updated and source_hash — and keeps everything else on disk. An allowlist rather than a carve-out, so a field nobody has thought about stays human-owned by default. That also retires the last_updated limitation #103 shipped with. Refs #105, #103, #107, brain#21.	2026-09-22 23:45:25 +02:00
mathiasandClaude Sonnet 5	6ad275b505	fix(ingest): thread source/author/published frontmatter through extraction (#85) ... pipeline.Run parses source/author/published from the raw content's own frontmatter and applies them deterministically to source-type RawPages after LLM extraction — never relies on the LLM to copy them through. buildFrontmatter now emits all three (source pages only) when present. Backfill of the 46 already-affected wiki/sources/*.md notes is a separate concern per the issue, not done here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A	2026-07-26 23:42:33 +02:00
mathiasandClaude Sonnet 4.6	3e9a648115	fix(pipeline): repair invalid JSON escape sequences from LLM output before parsing ... Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-23 22:04:27 +02:00
mathiasandClaude Sonnet 4.6	26b5636b43	feat(pipeline): replace ParsePages with ParseRawPages + RawPage type ... Strips slug authority from the LLM. The new RawPage type carries only {title, type, subtype, domain, content} — no paths or frontmatter. Pipeline will derive slugs deterministically (Task 4). pipeline.go gets a temporary bridge stub (TODO task4) to keep the package compiling between tasks. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-23 18:41:33 +02:00
mathias	9b11719481	feat(ingestion): add content chunking and LLM JSON output parser	2026-04-22 22:37:14 +02:00