feat(ingestion): place raw notes in a wing without human typing (#205 S1–S4, S6) #207

2026-10-20T04:12:43Z

mathias commented

2026-10-20 04:12:43 +00:00

Refs #205 (steps 1–4 and 6; step 5 migration is a separate PR). Risk: medium — changes where ingestion writes and adds an automatic decision path. Off by default: nothing changes until INGEST_WING_RESOLVE=1 is set on the ingestion deployment.

What

internal/wing: Resolver — explicit raw/<wing>/ subfolder or wing: frontmatter wins; otherwise a local-LLM classifier and an independent BM25 scorer must agree above a threshold on a wing that is known and eligible (classification.yaml: only wings classifying below Confidential; an untagged wing fails safe to confidential). Never creates a wing; never auto-assigns a client/untagged wing; anything else is an Exception.
pipeline.Config.Wing: writes wiki/<wing>/sources/ and wiki/<wing>/facts/ (concept/entity told apart by kind:), with wing/hall frontmatter. Inventory, title resolution and back-references are confined to the target wing. Empty Wing = retired layout, unchanged.
Watcher: with a Resolver, unplaceable notes are parked in raw/needs-review/ with the proposal in frontmatter; never extracted. needs-review/ is skipped by the walk.
brain_pending lists parked notes with queue, proposed_wing, proposal_reason, proposal_confidence. Approve = move the note into raw/<wing>/.
cmd/wingbacktest: measures the policy against pages whose wing a human already chose.

Evidence

Tests first at every step; go vet, go test -race ./... (0 FAIL), golangci-lint 0 issues.
Mutation-checked safety rules (each mutant fails a test): eligibility, known-wing, scorer agreement, threshold boundary, subfolder/frontmatter conflict, scorer self-exclusion (backtest leakage), inventory scoped to the target wing (cross-wing back-reference), non-plain wing name rejected, needs-review/ skipped, exception never ingested, frontmatter forgery via model text (%q), re-synced unchanged note not re-extracted.
Two of my own tests were wrong first time and were fixed after a mutant survived (copy-failure test earlier; frontmatter-forgery test here).
Backtest: see comment below (running against the local model on koala; numbers decide the default threshold).

Known limits / not in this PR

Only hyperguild and homelab are Internal in classification.yaml today, so with the flag on only those wings can be auto-assigned. Tagging more wings is a sovereignty decision for Mathias (the file says so).
Known wings are read at startup; a newly created wing is unknown until restart (fails safe: parked).
PDFs: parked as-is, proposal only in log.md.
No approve tool yet; approval is a file move.
Migration of legacy pages (#205 step 5) and the 12 old raw files: separate PR.

Review ask

Verify by execution: (1) no path writes into an ineligible or unknown wing, including via the model's answer, an explicit folder name, or ../-style names; (2) a model-supplied string cannot forge a frontmatter key in a parked note; (3) with the flag off the watcher behaves as before; (4) cross-wing isolation of inventory/back-references.

🤖 Generated with Claude Code

Created via git-mcp on behalf of @mathias

Refs #205 (steps 1–4 and 6; step 5 migration is a separate PR). Risk: **medium** — changes where ingestion writes and adds an automatic decision path. **Off by default**: nothing changes until `INGEST_WING_RESOLVE=1` is set on the ingestion deployment. ## What - `internal/wing`: `Resolver` — explicit `raw/<wing>/` subfolder or `wing:` frontmatter wins; otherwise a local-LLM classifier and an independent BM25 scorer must agree above a threshold on a wing that is **known** and **eligible** (`classification.yaml`: only wings classifying below Confidential; an untagged wing fails safe to confidential). Never creates a wing; never auto-assigns a client/untagged wing; anything else is an Exception. - `pipeline.Config.Wing`: writes `wiki/<wing>/sources/` and `wiki/<wing>/facts/` (concept/entity told apart by `kind:`), with `wing`/`hall` frontmatter. Inventory, title resolution and back-references are confined to the target wing. Empty Wing = retired layout, unchanged. - Watcher: with a Resolver, unplaceable notes are parked in `raw/needs-review/` with the proposal in frontmatter; never extracted. `needs-review/` is skipped by the walk. - `brain_pending` lists parked notes with `queue`, `proposed_wing`, `proposal_reason`, `proposal_confidence`. Approve = move the note into `raw/<wing>/`. - `cmd/wingbacktest`: measures the policy against pages whose wing a human already chose. ## Evidence - Tests first at every step; `go vet`, `go test -race ./...` (0 FAIL), `golangci-lint` 0 issues. - Mutation-checked safety rules (each mutant fails a test): eligibility, known-wing, scorer agreement, threshold boundary, subfolder/frontmatter conflict, scorer self-exclusion (backtest leakage), inventory scoped to the target wing (cross-wing back-reference), non-plain wing name rejected, `needs-review/` skipped, exception never ingested, frontmatter forgery via model text (`%q`), re-synced unchanged note not re-extracted. - Two of my own tests were wrong first time and were fixed after a mutant survived (copy-failure test earlier; frontmatter-forgery test here). - **Backtest: see comment below** (running against the local model on koala; numbers decide the default threshold). ## Known limits / not in this PR - Only `hyperguild` and `homelab` are Internal in `classification.yaml` today, so with the flag on only those wings can be auto-assigned. Tagging more wings is a sovereignty decision for Mathias (the file says so). - `Known` wings are read at startup; a newly created wing is unknown until restart (fails safe: parked). - PDFs: parked as-is, proposal only in log.md. - No approve tool yet; approval is a file move. - Migration of legacy pages (#205 step 5) and the 12 old raw files: separate PR. ## Review ask Verify by execution: (1) no path writes into an ineligible or unknown wing, including via the model's answer, an explicit folder name, or `../`-style names; (2) a model-supplied string cannot forge a frontmatter key in a parked note; (3) with the flag off the watcher behaves as before; (4) cross-wing isolation of inventory/back-references. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- _Created via git-mcp on behalf of @mathias_

mathias added 11 commits 2026-10-20 04:12:43 +00:00

feat(wing): resolver that places a raw note in a wing without human typing (#205 S1) 5bbf47064d

Explicit subfolder/frontmatter wins; otherwise a classifier and an
independent scorer must agree above a threshold on a known, eligible wing.
Anything else is an Exception for a human. Never creates a wing, never
assigns an ineligible (client/untagged) wing. Mutation-checked: dropping the
eligibility, known-wing, scorer, threshold or conflict check fails a test.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(wing): LLM classifier with tolerant, fail-closed reply parsing (#205 S1) 85d8ecb5c7

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(wing): BM25 scorer as the independent cross-check (#205 S1) c06509afe4

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(wing): backtest metric (coverage/precision at a threshold) and shared TopWing (#205 S1) 365130459f

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(wing): score a wing by its best page, not summed hits; add wingbacktest command (#205 S1) 62de8ac2e6

A first backtest showed BM25 summed over a wing's top hits favours big wings
(0/4 agreement). Best-page scoring is volume-robust; pinned by a test that
failed before the change.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(pipeline): write into wiki/<wing>/{sources,facts}/ when Config.Wing is set (#205 S2) ad793f320d

Sources -> wiki/<wing>/sources, concepts/entities -> wiki/<wing>/facts with
kind: frontmatter; inventory, title resolution and back-references are
confined to the target wing. Empty Wing keeps the retired layout unchanged.
Mutation-checked: widening the inventory beyond the wing fails the
cross-wing back-reference test; a non-plain wing name is rejected.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(watcher): resolve each note's wing; park the unplaceable in raw/needs-review/ (#205 S3) 2c6220782d

With Config.Resolver set, an explicit subfolder/frontmatter wing or an agreeing
classifier+scorer routes the note into that wing; anything else is parked with
its proposal in frontmatter and never extracted. needs-review/ is skipped by
the walk like rejected/. Nil Resolver keeps the retired layout.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(server): INGEST_WING_RESOLVE=1 wires the wing resolver into the watcher (#205 S3) 20d874a23d

Off by default (retired layout unchanged). On: eligibility from
classification.yaml, local ingest LLM as classifier, BM25 as cross-check,
INGEST_WING_THRESHOLD (default 0.9, rejected outside [0,1]). Build failure
is fatal when explicitly enabled.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(api): brain_pending lists notes parked in raw/needs-review/ with the wing proposal (#205 S4) b9657973ff

Approval is moving the note into raw/<wing>/ (the watcher ingests it there);
a dedicated approve tool is tracked separately. Proposal reason is written
Go-quoted so LLM text cannot start a frontmatter key; pinned by a
mutation-checked test.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

test(watcher): make the frontmatter-forgery test reach the code it names (#205 S4) 96c63906c2

The first version fed the text through a path that never carried it into the
file, so it passed under the mutant. A classifier error does carry it; the
test now fails with %q replaced by %s.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

test(watcher): a re-synced unchanged note is not extracted twice; a changed one is (#205 S6)

CI / Lint / Test / Vet (pull_request) Successful in 1m4s

Details

CI / Mirror to GitHub (pull_request) Skipped

Details

b4b36c9bfd

Mutation-checked against the source-hash drift check.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

mathias referenced this issue from a commit

2026-10-20 04:16:50 +00:00

fix(watcher): never overwrite a parked note; a proposal cannot be forged or pre-empted; CRLF notes; wing dirs must be plain (#205)

mathias added 2 commits 2026-10-20 04:16:50 +00:00

feat(migrate): plan and apply the move out of the retired wiki layout (#205 S5) dd4a825f0a

Sources are placed by the wing Resolver; concepts/entities inherit the strict
majority wing of the sources that cite them. Collisions, junk (auto-tunnel
logs) and anything unplaceable are Exceptions left in place. Apply writes the
destination O_EXCL then removes the source, only for Auto items.
Mutation-checked (junk guard, majority, slug identity, status gate,
no-overwrite, destination-exists).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(watcher): never overwrite a parked note; a proposal cannot be forged or pre-empted; CRLF notes; wing dirs must be plain (#205 )

CI / Lint / Test / Vet (pull_request) Failing after 1m2s

Details

CI / Mirror to GitHub (pull_request) Skipped

Details

006a891423

From the independent review of #207:
- parked notes are created O_EXCL with -2,-3 suffixes (same-named notes from
  different subfolders were overwritten and their originals deleted)
- proposed_wing is emitted only if a plain wing name; a note's own proposal_*
  keys are dropped so they cannot outrank the resolver's (first-wins reads)
- CRLF notes: wing: key honoured, one valid frontmatter block on parking
- KnownWings skips wiki dirs that are not canonical lower-case names
- adds the wingmigrate dry-run/apply command
Each fix has a test that failed first and a mutant that fails it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

mathias added 1 commit 2026-10-20 04:17:33 +00:00

test(extract): allow reserveParkedName in the raw-reader guard (creates in needs-review/, reads nothing) (#205 )

CI / Lint / Test / Vet (pull_request) Successful in 1m1s

Details

CI / Mirror to GitHub (pull_request) Skipped

Details

1e14c56a49

The previous commit was pushed with this guard red: the gate was run with ';'
instead of '&&'. Allowlisting is the guard's own sanctioned mechanism and is
called out for review.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

mathias commented

2026-10-20 04:17:37 +00:00

Independent review pass (pr-gatekeeper agent, verified by execution; reviewer is an agent, not four-eyes). First verdict: BLOCK. Resolution:

Confirmed and fixed (each has a test that failed first and a mutant that fails it):

Silent data loss when parking same-named notes (u1/x.md..u4/x.md: two overwritten, originals deleted). Parked copies are now created O_EXCL with -2, -3 suffixes. TestProcessDir_SameNamedNotesAreAllParkedNoneOverwritten.
Frontmatter forgery in parked notes / brain_pending via proposed_wing (model reply or folder name with newline), and a note's own proposal_* keys outranking the resolver's (first-wins reads). proposed_wing is emitted only for a plain wing name (else it is quoted into the reason); the note's own proposal_* keys are dropped. TestProcessDir_AHostileWingCannotForgeTheProposal, ..._ANotesOwnProposalKeysDoNotOutrankTheResolvers.
CRLF notes: wing: is honoured; one valid frontmatter block on parking.
Known wings: a wiki dir that is not a canonical lower-case name is not a candidate (the pipeline would have created a different wing).

Acknowledged, not changed (stated so the reviewer's finding is not silently dropped):

Flag-off is not byte-identical to main: raw/needs-review/ is skipped by the walk unconditionally, and brain_pending lists it regardless of the flag. Deliberate: a note parked while the flag was on must not be ingested into the retired layout if the flag is later turned off. Nothing else differs with Resolver == nil.
stuck skip set is in memory, so after a restart a note whose original cannot be removed is parked once more (production initialises it in Start; only direct processDir use with a nil set re-parks every poll).
The candidate list sent to the classifier contains wing names including client wings; page text goes only to INGEST_LLM_URL, which is the local model in the deployment (koala/qwen35-9b-fast, sovereignty: strict). Not enforced in code.

My own slip, disclosed: the fix commit was pushed with internal/extract's raw-reader guard red because I chained the gate with ;. The next commit adds reserveParkedName to that guard's own allowlist (it creates a file in needs-review/, reads nothing from raw/). Please look at that allowlist change specifically. Full gate now: go vet ./... and go test -race ./... exit 0, 0 FAIL lines, golangci-lint 0 issues.

Also on this branch since the review: internal/migrate + cmd/wingmigrate (#205 S5, dry-run by default, -apply only performs Auto items; -what-if-eligible is dry-run only). Not yet run against the live brain; the plan needs the backtest and Mathias's classification.yaml tags first.

Created via git-mcp on behalf of @mathias

**Independent review pass (pr-gatekeeper agent, verified by execution; reviewer is an agent, not four-eyes). First verdict: BLOCK.** Resolution: Confirmed and fixed (each has a test that failed first and a mutant that fails it): 1. **Silent data loss when parking same-named notes** (u1/x.md..u4/x.md: two overwritten, originals deleted). Parked copies are now created O_EXCL with `-2`, `-3` suffixes. `TestProcessDir_SameNamedNotesAreAllParkedNoneOverwritten`. 2. **Frontmatter forgery in parked notes / brain_pending** via `proposed_wing` (model reply or folder name with newline), and a note's own `proposal_*` keys outranking the resolver's (first-wins reads). `proposed_wing` is emitted only for a plain wing name (else it is quoted into the reason); the note's own `proposal_*` keys are dropped. `TestProcessDir_AHostileWingCannotForgeTheProposal`, `..._ANotesOwnProposalKeysDoNotOutrankTheResolvers`. 3. **CRLF notes**: `wing:` is honoured; one valid frontmatter block on parking. 4. **Known wings**: a wiki dir that is not a canonical lower-case name is not a candidate (the pipeline would have created a different wing). Acknowledged, not changed (stated so the reviewer's finding is not silently dropped): - **Flag-off is not byte-identical to main:** `raw/needs-review/` is skipped by the walk unconditionally, and `brain_pending` lists it regardless of the flag. Deliberate: a note parked while the flag was on must not be ingested into the retired layout if the flag is later turned off. Nothing else differs with `Resolver == nil`. - **`stuck` skip set is in memory**, so after a restart a note whose original cannot be removed is parked once more (production initialises it in `Start`; only direct `processDir` use with a nil set re-parks every poll). - The candidate list sent to the classifier contains wing **names** including client wings; page text goes only to `INGEST_LLM_URL`, which is the local model in the deployment (koala/qwen35-9b-fast, sovereignty: strict). Not enforced in code. **My own slip, disclosed:** the fix commit was pushed with `internal/extract`'s raw-reader guard red because I chained the gate with `;`. The next commit adds `reserveParkedName` to that guard's own allowlist (it creates a file in needs-review/, reads nothing from raw/). **Please look at that allowlist change specifically.** Full gate now: `go vet ./...` and `go test -race ./...` exit 0, 0 FAIL lines, golangci-lint 0 issues. Also on this branch since the review: `internal/migrate` + `cmd/wingmigrate` (#205 S5, dry-run by default, `-apply` only performs Auto items; `-what-if-eligible` is dry-run only). Not yet run against the live brain; the plan needs the backtest and Mathias's `classification.yaml` tags first. --- _Created via git-mcp on behalf of @mathias_

mathias referenced this issue from a commit

2026-10-20 04:21:26 +00:00

fix(watcher): park through the brain root, keep a note edited mid-resolve, honour only real fences (#205)

mathias added 1 commit 2026-10-20 04:21:26 +00:00

fix(watcher): park through the brain root, keep a note edited mid-resolve, honour only real fences (#205 )

CI / Lint / Test / Vet (pull_request) Successful in 1m7s

Details

CI / Mirror to GitHub (pull_request) Skipped

Details

bd33a0a4e8

From the second independent review of #207:
- parkForReview now skips the delete when the note changed since it was read
  (an LLM call sits in that window; Syncthing can deliver a newer version)
- parked copies are created via brainfs (real dirs only, O_EXCL, no symlink
  following), so a planted raw/needs-review symlink cannot redirect or hide a
  note; this also removes the need for the guard allowlist entry added in
  1e14c56, which is reverted
- a '----' line no longer closes the proposal block
Each has a test that failed first and a mutant that fails it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

mathias commented

2026-10-20 04:22:57 +00:00

Independent review pass 3 (pr-gatekeeper agent, by execution; agent, not four-eyes): nothing blocks at bd33a0a.

Fixed and re-probed by the reviewer:

Note edited while the resolver ran (in-place, rename-replace, same-size): the newer version stays in raw/, the stale copy is parked, next poll parks the new one as n-2.md; 5 polls, no duplicate storm, no loss.
raw/needs-review as a symlink (outside the brain, and to wiki/client-x), as a regular file: refused every poll, nothing written through it, note stays in raw/. Missing: created as a real directory. Symlinked raw/ itself: refused.
----, ---x, EOF-without-newline fences handled; CRLF normalised; a note's own proposed_wing dropped.
204 same-named notes in 204 subfolders: all parked, none overwritten.
go vet ./... and go test -race -count=1 ./... exit 0, 0 FAIL.

Residual, accepted and documented: changedSince is size+mtime (a same-size rewrite with a restored mtime is not detected; the new bytes then survive only in the parked copy); appendWatcherLog, the retire marker fallback and copyFromRoot are path-based but pre-existing and not the park path; a planted needs-review symlink produces a persistent error log (loud, intended).

Not verified by the reviewer: PDF parking through these paths, the TOCTOU between changedSince and Remove, real Syncthing behaviour.

The guard allowlist entry from 1e14c56 has been reverted (parking no longer opens files by path).

Created via git-mcp on behalf of @mathias

**Independent review pass 3 (pr-gatekeeper agent, by execution; agent, not four-eyes): nothing blocks at bd33a0a.** Fixed and re-probed by the reviewer: - Note edited while the resolver ran (in-place, rename-replace, same-size): the newer version stays in raw/, the stale copy is parked, next poll parks the new one as `n-2.md`; 5 polls, no duplicate storm, no loss. - `raw/needs-review` as a symlink (outside the brain, and to `wiki/client-x`), as a regular file: refused every poll, nothing written through it, note stays in raw/. Missing: created as a real directory. Symlinked `raw/` itself: refused. - `----`, `---x`, EOF-without-newline fences handled; CRLF normalised; a note's own `proposed_wing` dropped. - 204 same-named notes in 204 subfolders: all parked, none overwritten. - `go vet ./...` and `go test -race -count=1 ./...` exit 0, 0 FAIL. Residual, accepted and documented: `changedSince` is size+mtime (a same-size rewrite with a restored mtime is not detected; the new bytes then survive only in the parked copy); `appendWatcherLog`, the `retire` marker fallback and `copyFromRoot` are path-based but pre-existing and not the park path; a planted needs-review symlink produces a persistent error log (loud, intended). Not verified by the reviewer: PDF parking through these paths, the TOCTOU between `changedSince` and Remove, real Syncthing behaviour. The guard allowlist entry from 1e14c56 has been reverted (parking no longer opens files by path). --- _Created via git-mcp on behalf of @mathias_

All checks were successful

CI / Lint / Test / Vet (pull_request) Successful in 1m7s

Required

Details

CI / Mirror to GitHub (pull_request) Skipped

Details

You are not authorized to merge this pull request.

This pull request can be merged automatically.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin wt/wing-layout:wt/wing-layout

git checkout wt/wing-layout

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mathias/hyperguild#207