mathias/hyperguild
1
0
Fork 0
Code Issues 32 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

raw/ inbox: route auto-ingest to wiki/<wing>/sources/ (wing from raw/<wing>/ subfolder) #205

New Issue
Open
opened 2026-10-19 18:59:06 +00:00 by mathias · 2 comments
mathias commented 2026-10-19 18:59:06 +00:00
Owner
Copy Link
Copy Source

Follow-up to the inbox-zero change (watcher removes archived originals from raw/).

Problem

ingestion/internal/pipeline/build.go buildPage writes auto-ingested pages to the retired legacy dirs wiki/sources|concepts|entities/. brain CLAUDE.md: those are archive-only; new sources go to wiki/<wing>/sources/. Result: no wing/hall, no per-wing _index.md, found by BM25 only.

Decided (2026-10-19, Mathias)

Option A: dump and forget. Wing comes from the subfolder the file is dropped in, raw/<wing>/file.md. brain_pending then shows only what is unprocessed or failed/needs a wing.

To do (each its own commit, test first)

  1. Watcher walks raw/<wing>/, passes wing into pipeline.Run.
  2. buildPage: source -> wiki/<wing>/sources/<slug>.md with wing/hall frontmatter. Decide where concept/entity pages go (facts? same wing) — and how wiki.LoadInventory/Resolve (keyed by PageType) stay correct across wings.
  3. Files dropped at top level of raw/ with no wing: leave in raw/ (visible in brain_pending) rather than guess.
  4. One-off migration: the ~8 pages auto-written to legacy dirs since Sep 2026 (e.g. wiki/entities/jev.md, wiki/concepts/dark-factory.md, wiki/sources/jev-will-10x-*.md) and the leftover *.processed markers + originals still sitting in raw/.
  5. Dedupe on re-sync: a file reappearing via Syncthing is skipped by the source-hash drift check (pipeline.Run Skipped) — add a test that pins this.

Source: session 2026-10-19, watcher.go processFile / promote.go ListPending.

Created via git-mcp on behalf of @mathias

Follow-up to the inbox-zero change (watcher removes archived originals from raw/). ## Problem `ingestion/internal/pipeline/build.go` `buildPage` writes auto-ingested pages to the retired legacy dirs `wiki/sources|concepts|entities/`. brain CLAUDE.md: those are archive-only; new sources go to `wiki/<wing>/sources/`. Result: no wing/hall, no per-wing `_index.md`, found by BM25 only. ## Decided (2026-10-19, Mathias) Option A: dump and forget. Wing comes from the subfolder the file is dropped in, `raw/<wing>/file.md`. `brain_pending` then shows only what is unprocessed or failed/needs a wing. ## To do (each its own commit, test first) 1. Watcher walks `raw/<wing>/`, passes wing into `pipeline.Run`. 2. `buildPage`: source -> `wiki/<wing>/sources/<slug>.md` with `wing`/`hall` frontmatter. Decide where concept/entity pages go (`facts`? same wing) — and how `wiki.LoadInventory`/`Resolve` (keyed by PageType) stay correct across wings. 3. Files dropped at top level of raw/ with no wing: leave in raw/ (visible in `brain_pending`) rather than guess. 4. One-off migration: the ~8 pages auto-written to legacy dirs since Sep 2026 (e.g. wiki/entities/jev.md, wiki/concepts/dark-factory.md, wiki/sources/jev-will-10x-*.md) and the leftover `*.processed` markers + originals still sitting in raw/. 5. Dedupe on re-sync: a file reappearing via Syncthing is skipped by the source-hash drift check (`pipeline.Run` Skipped) — add a test that pins this. Source: session 2026-10-19, watcher.go processFile / promote.go ListPending. --- _Created via git-mcp on behalf of @mathias_
mathias commented 2026-10-19 19:09:13 +00:00
Author
Owner
Copy Link
Copy Source

Deferred from the PR #206 review (independent pass, verified by execution):

  1. changedSince misses a same-size rewrite with restored mtime (new bytes then live only in raw/processed/<date>/; wiki stays stale). Hardening: os.SameFile catches new inode; hashing the ingested bytes vs the archive copy closes it fully.
  2. stuck skip set is keyed by path only: after a user fixes permissions AND edits the note, new content is ignored until restart. Key on (path, size, mtime).
  3. A note kept by changedSince and re-ingested next poll leaves a duplicate archive copy (doc-HHMMSS.md). Wiki page is idempotent (no duplicate).
  4. Check-then-act window between changedSince and Remove (small; archive copy exists, loss partial).
    None blocks #206. Step 5 of this issue (dedupe on Syncthing re-sync) still stands.

Created via git-mcp on behalf of @mathias

Deferred from the PR #206 review (independent pass, verified by execution): 1. `changedSince` misses a same-size rewrite with restored mtime (new bytes then live only in `raw/processed/<date>/`; wiki stays stale). Hardening: `os.SameFile` catches new inode; hashing the ingested bytes vs the archive copy closes it fully. 2. `stuck` skip set is keyed by path only: after a user fixes permissions AND edits the note, new content is ignored until restart. Key on (path, size, mtime). 3. A note kept by `changedSince` and re-ingested next poll leaves a duplicate archive copy (`doc-HHMMSS.md`). Wiki page is idempotent (no duplicate). 4. Check-then-act window between `changedSince` and `Remove` (small; archive copy exists, loss partial). None blocks #206. Step 5 of this issue (dedupe on Syncthing re-sync) still stands. --- _Created via git-mcp on behalf of @mathias_
mathias added the risk/medium label 2026-10-19 19:19:57 +00:00
mathias commented 2026-10-19 19:42:51 +00:00
Author
Owner
Copy Link
Copy Source

Sprint plan + decisions (2026-10-19, Mathias)

Decided: (1) Clips: Dark Factory + 10 Levels -> applied-ai-radar with an unverified-vendor-claims banner; Best Jev use cases folded in; Pi Durable kept as plain source; Zazen / Paperclip / RoboNuggets-Jev rejected. Mathias delegated the 6 older ones to the agent. (2) Wing naming must work without the human typing it; human approves exceptions only. (4) Standing merge approval for every PR in this sprint (risk/medium, CI green on the head commit, independent pr-gatekeeper pass with no blocker). Does not extend beyond this sprint.

Wing resolution (no typing)

Layered, deterministic first:

  1. Explicit signal wins: raw/<wing>/ subfolder or frontmatter wing: — but only if <wing> is an existing wing.
  2. Else classify with the local ingest model (never cloud; client rule): input = note + each candidate wing's purpose (CLAUDE.md wing table + top titles of its _index.md); output = wing, confidence, reason. Cross-check with BM25 of the note against each wing's index. Agree + confident -> auto. Disagree or low confidence -> exception.
  3. Reading-material default (clips/articles with no tracked-project link): applied-ai-radar.
  4. Safety: auto-assign only to an allowlist of non-client wings (those tagged internal/public in classification.yaml; unlisted fails safe to confidential = never auto). Never auto-create a wing; a proposed new wing is an exception.
  5. Exceptions land in raw/needs-review/ with proposed wing + confidence + reason in frontmatter; brain_pending lists them; approving = existing brain_promote. Every auto-assignment is logged (reason + confidence) to log.md for spot-checks.
  6. Threshold comes from a backtest, not a guess: existing wiki pages already carry human-decided wings; strip the wing, classify, compare. Pick the confidence cut-off for a stated precision.

Steps (each its own commit, test first; PRs as needed)

  • S1 pure wing resolver + allowlist + backtest harness (report in the PR).
  • S2 pipeline layout wiki/<wing>/sources|facts/, inventory across wings (step 2).
  • S3 watcher: signals -> resolver -> pipeline, or needs-review/ (steps 1,3).
  • S4 brain_pending shows needs-review/ with proposals.
  • S5 migrate ALL legacy wiki/concepts|entities|sources pages (~141) with the same resolver: dry-run report first, auto-apply above threshold, exceptions in one batch for Mathias; reversible per-commit; wikilinks are slug-based so they survive (verify with brain graph before/after). Plus clear the 12 legacy raw originals + .processed markers.
  • S6 pin test: a file re-appearing via Syncthing is skipped by the drift check (step 5).
  • Then clips disposition, brain note, tag, worktree cleanup.

Created via git-mcp on behalf of @mathias

## Sprint plan + decisions (2026-10-19, Mathias) **Decided:** (1) Clips: Dark Factory + 10 Levels -> `applied-ai-radar` with an unverified-vendor-claims banner; Best Jev use cases folded in; Pi Durable kept as plain source; Zazen / Paperclip / RoboNuggets-Jev rejected. Mathias delegated the 6 older ones to the agent. (2) Wing naming must work **without the human typing it**; human approves exceptions only. (4) **Standing merge approval for every PR in this sprint** (risk/medium, CI green on the head commit, independent pr-gatekeeper pass with no blocker). Does not extend beyond this sprint. ## Wing resolution (no typing) Layered, deterministic first: 1. Explicit signal wins: `raw/<wing>/` subfolder or frontmatter `wing:` — but only if `<wing>` is an existing wing. 2. Else classify with the **local** ingest model (never cloud; client rule): input = note + each candidate wing's purpose (CLAUDE.md wing table + top titles of its `_index.md`); output = wing, confidence, reason. Cross-check with BM25 of the note against each wing's index. Agree + confident -> auto. Disagree or low confidence -> exception. 3. Reading-material default (clips/articles with no tracked-project link): `applied-ai-radar`. 4. **Safety:** auto-assign only to an allowlist of non-client wings (those tagged `internal`/`public` in classification.yaml; unlisted fails safe to confidential = never auto). Never auto-create a wing; a proposed new wing is an exception. 5. Exceptions land in `raw/needs-review/` with proposed wing + confidence + reason in frontmatter; `brain_pending` lists them; approving = existing `brain_promote`. Every auto-assignment is logged (reason + confidence) to log.md for spot-checks. 6. **Threshold comes from a backtest, not a guess:** existing wiki pages already carry human-decided wings; strip the wing, classify, compare. Pick the confidence cut-off for a stated precision. ## Steps (each its own commit, test first; PRs as needed) - S1 pure `wing` resolver + allowlist + backtest harness (report in the PR). - S2 pipeline layout `wiki/<wing>/sources|facts/`, inventory across wings (step 2). - S3 watcher: signals -> resolver -> pipeline, or `needs-review/` (steps 1,3). - S4 `brain_pending` shows `needs-review/` with proposals. - S5 migrate ALL legacy `wiki/concepts|entities|sources` pages (~141) with the same resolver: dry-run report first, auto-apply above threshold, exceptions in one batch for Mathias; reversible per-commit; wikilinks are slug-based so they survive (verify with brain graph before/after). Plus clear the 12 legacy raw originals + `.processed` markers. - S6 pin test: a file re-appearing via Syncthing is skipped by the drift check (step 5). - Then clips disposition, brain note, tag, worktree cleanup. --- _Created via git-mcp on behalf of @mathias_
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agent/ready
Meets the agent-ready contract; authorizes unattended execution
 blocked/client
Waiting on the client
 blocked/supplier
Waiting on a supplier
 blocked/upstream
Waiting on an upstream project or vendor
 risk/high
Irreversible, security-relevant, or control plane
 risk/low
Reversible, local, test-covered
 risk/medium
Touches shared state, deploys or data
 size/l
Too big - split it
 size/m
Under two days
 size/s
Under half a day
 size/xs
Under an hour
 status:dispatch
status:dispatched
status:review
status:reviewing
type/chore
Maintenance, no behaviour change
 type/decision
Human decision required; never agent/ready
 type/docs
Documentation only
 type/feat
New capability
 type/fix
Corrects broken behaviour
 type/ops
Infrastructure, deploy, runtime
 type/research
Investigate; output is a finding, not code
 type/test
Test coverage only
No labels risk/medium
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
renovate-bot mathias
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/hyperguild#205