mathias/hyperguild
1
0
Fork 0
Code Issues 30 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

Extraction pipeline drops source/author/published frontmatter (46/46 wiki/sources notes affected) #85

New Issue
Closed
opened 2026-07-26 21:12:07 +00:00 by mathias · 1 comment
mathias commented 2026-07-26 21:12:07 +00:00
Owner
Copy Link
Copy Source

Problem

Every wiki/sources/*.md note produced by ingestion/internal/pipeline.Run is missing source:, author:, published: frontmatter — even though the raw clipping passed in as content has them. Checked all 46 existing source notes: 0 have a source: field. Confirmed on a fresh test article ingested today (2026-07-26): meet-ornith-the-agentic-coding-model-that-runs-entirely-on-your-laptop.md's raw copy has source:/author:/published:, the generated wiki page has none of it (only title/type/domain/date_ingested/last_updated/aliases).

Result: no wiki source note is traceable back to its origin URL without manually opening the raw archive.

Where it lives

pipeline.Run (pipeline.go) → BuildPrompt(schema, source, chunk, inventory) → LLM Complete → ParseRawPages(output) → BuildPages(rawPages, sourceSlug, date). The source param passed into Run is just used as a slug basis (wiki.Slug(source)), not threaded through as structured metadata — and the extraction prompt apparently isn't told to carry source/author/published from the input's own frontmatter into its output.

Scope

Parse source/author/published out of the raw content's frontmatter before/alongside the LLM extraction call, and have BuildPages inject them into the generated wiki/sources/*.md frontmatter unchanged (don't rely on the LLM to faithfully copy them through — treat as deterministic passthrough, not an extraction task).

Acceptance criteria

  • Ingesting raw content with source:/author:/published: in its frontmatter produces a wiki source note carrying all three fields verbatim.
  • Regression test: fixture raw content with these fields → assert present in generated frontmatter.
  • Existing 46 files are a separate backfill concern, not in scope here (raw copies are on koala at /var/lib/supervisor/brain/raw/processed/**, cross-referenceable by title if a backfill is wanted later).
## Problem Every `wiki/sources/*.md` note produced by `ingestion/internal/pipeline.Run` is missing `source:`, `author:`, `published:` frontmatter — even though the raw clipping passed in as `content` has them. Checked all 46 existing source notes: **0 have a `source:` field.** Confirmed on a fresh test article ingested today (2026-07-26): `meet-ornith-the-agentic-coding-model-that-runs-entirely-on-your-laptop.md`'s raw copy has `source:`/`author:`/`published:`, the generated wiki page has none of it (only `title/type/domain/date_ingested/last_updated/aliases`). Result: no wiki source note is traceable back to its origin URL without manually opening the raw archive. ## Where it lives `pipeline.Run` (`pipeline.go`) → `BuildPrompt(schema, source, chunk, inventory)` → LLM `Complete` → `ParseRawPages(output)` → `BuildPages(rawPages, sourceSlug, date)`. The `source` param passed into `Run` is just used as a slug basis (`wiki.Slug(source)`), not threaded through as structured metadata — and the extraction prompt apparently isn't told to carry `source`/`author`/`published` from the input's own frontmatter into its output. ## Scope Parse `source`/`author`/`published` out of the raw content's frontmatter before/alongside the LLM extraction call, and have `BuildPages` inject them into the generated `wiki/sources/*.md` frontmatter unchanged (don't rely on the LLM to faithfully copy them through — treat as deterministic passthrough, not an extraction task). ## Acceptance criteria - Ingesting raw content with `source:`/`author:`/`published:` in its frontmatter produces a wiki source note carrying all three fields verbatim. - Regression test: fixture raw content with these fields → assert present in generated frontmatter. - Existing 46 files are a separate backfill concern, not in scope here (raw copies are on koala at `/var/lib/supervisor/brain/raw/processed/**`, cross-referenceable by title if a backfill is wanted later).
mathias commented 2026-07-26 21:42:41 +00:00
Author
Owner
Copy Link
Copy Source

Fixed in 6ad275b. pipeline.Run now parses source/author/published out of the raw content's own frontmatter (parseContentFrontmatter) and applies them deterministically to source-type RawPages post-extraction (applySourceMeta) — never relies on the LLM to copy them through (fields are json:"-" on RawPage, immune to LLM-controlled JSON). buildFrontmatter emits all three, source pages only, when present. Regression tests added at both the BuildPages and Run level. Full go test ./... green.

Backfill of the 46 existing notes left out of scope per the issue — raw copies are on koala at /var/lib/supervisor/brain/raw/processed/** if you want that done separately.

Fixed in 6ad275b. `pipeline.Run` now parses `source`/`author`/`published` out of the raw content's own frontmatter (`parseContentFrontmatter`) and applies them deterministically to source-type `RawPage`s post-extraction (`applySourceMeta`) — never relies on the LLM to copy them through (fields are `json:"-"` on `RawPage`, immune to LLM-controlled JSON). `buildFrontmatter` emits all three, source pages only, when present. Regression tests added at both the `BuildPages` and `Run` level. Full `go test ./...` green. Backfill of the 46 existing notes left out of scope per the issue — raw copies are on koala at `/var/lib/supervisor/brain/raw/processed/**` if you want that done separately.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agent/ready
Meets the agent-ready contract; authorizes unattended execution
 blocked/client
Waiting on the client
 blocked/supplier
Waiting on a supplier
 blocked/upstream
Waiting on an upstream project or vendor
 risk/high
Irreversible, security-relevant, or control plane
 risk/low
Reversible, local, test-covered
 risk/medium
Touches shared state, deploys or data
 size/l
Too big - split it
 size/m
Under two days
 size/s
Under half a day
 size/xs
Under an hour
 status:dispatch
status:dispatched
status:review
status:reviewing
type/chore
Maintenance, no behaviour change
 type/decision
Human decision required; never agent/ready
 type/docs
Documentation only
 type/feat
New capability
 type/fix
Corrects broken behaviour
 type/ops
Infrastructure, deploy, runtime
 type/research
Investigate; output is a finding, not code
 type/test
Test coverage only
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
renovate-bot mathias
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/hyperguild#85