Sprint: clips land in web-clip with raw/ at zero — acceptance first (supersedes #205 steps 1–5) #209

Open
opened 2026-10-20 17:49:41 +00:00 by mathias · 1 comment
Owner

Goal (observable): drop a clip into raw/ -> it appears as a page under wiki/web-clip/sources/, shared concepts/entities are updated once (no per-wing duplicates), raw/ is empty again, nothing client-related is ever auto-placed. Falsified if >20% of web-clip items are moved by hand in two weeks.

Pattern: Sweep Sprint — read-only research per issue first (read bodies AND comments), then sequential one-issue-per-commit, test first, each PR independently gatekept. Context: #205 (plan), #208 (defects + direction), skills#56.

Needs Mathias before start

A. Amend brain CLAUDE.md: wiki/concepts + wiki/entities become the shared cross-wing layer (currently "retired").
B. Delete the 25 tunnel-candidates-* pages from wiki/sources (git keeps them).
C. Leave the 15 legacy notes/project pages untouched (list exceptions).
D. Remove the dead classifier/scorer now vs keep dormant 2 weeks (agent recommends remove).
E. Merge approval for this sprint's PRs (previous standing approval ended with the last sprint).

Sprint backlog (order = dependency)

  1. Acceptance test first (size/s): end-to-end "drop clip -> web-clip, raw empty". Red before anything else.
  2. #158 (risk/medium, size/m): writes under raw/ (processed/, failed/, rejected/, .processed markers) follow symlinks. Inbox zero (v0.25.0) added more such writes; must precede enabling anything. Tests for all four cases.
  3. Pipeline: shared concepts/entities, per-wing sources (size/m, #208 item 3): inventory = wing sources + global concepts/entities; remove per-wing entity duplication.
  4. Routing without a classifier (size/m, #208 items 1,4): explicit subfolder always places unless the wing is on a client deny-list; no signal -> web-clip; decouple from classification.yaml; client content check before extraction (reuse the CLAUDE_INGEST_CLIENT_BLOCK mechanism from claudewatcher — currently NOT applied to raw/).
  5. Approval path (size/s, #208 item 1): brain_promote/brain_reject read raw/needs-review/.
  6. Watcher hardening on the same files (one issue per commit): #198 (ReadAll, no size cap), #199 (marker-write failure only logged), changedSince size+mtime limit, stuck key by (path,size,mtime), duplicate archive copy after a changed note, empty-frontmatter note, park-failure error spam.
  7. Legacy migration, sources only (size/m): manifest + rollback, dry-run in repo; apply as a commit to the brain repo with watcher/sync paused (host step for Mathias). 15 videos/articles -> web-clip; 25 junk deleted (B); 118 concepts/entities untouched; 15 notes untouched (C).
  8. Remove dead code (D) + rename/delete the misnamed BM25Scorer; keep wingbacktest only if still useful.
  9. Close-out: Google Ads clip out of rejected/ + generic link to the trucker Google Ads incident note (no client detail in brain); 6 older clip dispositions; brain notes (see below); tag; clean 3 worktrees; amend brain CLAUDE.md (A).

Explicitly NOT in this sprint

  • #203 brain_query ranking is substring counting (root cause of the useless scorer). Real fix = proper BM25 (IDF + length norm) for the whole brain; changes production search ranking, needs a retrieval eval first. Next sprint; pairs with signals#95.
  • #192 / #196 / #197 / #167 brain_edit path I/O, edit locks, brainfs fd-walk (risk/high write-path hardening) -> own sprint.
  • skills#56 holdouts (needs CAD read-scope check), signals#95 (comment + eval after #203), analysis actions: assessor-loop ADR-0001 check, agentsquad cost-per-lap.
  • infra: koala syncthing.service (user unit) failed since 2026-10-06 while two syncthing processes run — investigate, host-side.
  • Backlog triage: #36, #40, #41, #63, #64, #71, #80, #113, #114, #122 are 6+ weeks old; close/park decision for Mathias.

Brain notes to write at close (generalised, no tooling-specific rationale)

  • A self-reported model confidence that is saturated carries no information; measure calibration on the real population before building a gate on it.
  • Run the end-to-end acceptance scenario before hardening the parts; three gate passes polished code that could not reach its goal.
  • Test/type names must match what they observe (BM25Scorer that counts substrings; a test whose input never reached the code it named).
  • "Rejected" in an inbox is a decision with an author; verify who decided before using it as a preference signal.

Created via git-mcp on behalf of @mathias

**Goal (observable):** drop a clip into `raw/` -> it appears as a page under `wiki/web-clip/sources/`, shared concepts/entities are updated once (no per-wing duplicates), `raw/` is empty again, nothing client-related is ever auto-placed. Falsified if >20% of web-clip items are moved by hand in two weeks. Pattern: Sweep Sprint — read-only research per issue first (read bodies AND comments), then sequential one-issue-per-commit, test first, each PR independently gatekept. Context: #205 (plan), #208 (defects + direction), skills#56. ## Needs Mathias before start A. Amend brain CLAUDE.md: `wiki/concepts` + `wiki/entities` become the shared cross-wing layer (currently "retired"). B. Delete the 25 `tunnel-candidates-*` pages from wiki/sources (git keeps them). C. Leave the 15 legacy notes/project pages untouched (list exceptions). D. Remove the dead classifier/scorer now vs keep dormant 2 weeks (agent recommends remove). E. Merge approval for this sprint's PRs (previous standing approval ended with the last sprint). ## Sprint backlog (order = dependency) 1. **Acceptance test first** (size/s): end-to-end "drop clip -> web-clip, raw empty". Red before anything else. 2. **#158** (risk/medium, size/m): writes under raw/ (processed/, failed/, rejected/, .processed markers) follow symlinks. Inbox zero (v0.25.0) added more such writes; must precede enabling anything. Tests for all four cases. 3. **Pipeline: shared concepts/entities, per-wing sources** (size/m, #208 item 3): inventory = wing sources + global concepts/entities; remove per-wing entity duplication. 4. **Routing without a classifier** (size/m, #208 items 1,4): explicit subfolder always places unless the wing is on a client deny-list; no signal -> `web-clip`; decouple from classification.yaml; client content check before extraction (reuse the CLAUDE_INGEST_CLIENT_BLOCK mechanism from claudewatcher — currently NOT applied to raw/). 5. **Approval path** (size/s, #208 item 1): `brain_promote`/`brain_reject` read `raw/needs-review/`. 6. **Watcher hardening on the same files** (one issue per commit): #198 (ReadAll, no size cap), #199 (marker-write failure only logged), changedSince size+mtime limit, `stuck` key by (path,size,mtime), duplicate archive copy after a changed note, empty-frontmatter note, park-failure error spam. 7. **Legacy migration, sources only** (size/m): manifest + rollback, dry-run in repo; apply as a commit to the brain repo with watcher/sync paused (host step for Mathias). 15 videos/articles -> web-clip; 25 junk deleted (B); 118 concepts/entities untouched; 15 notes untouched (C). 8. **Remove dead code (D)** + rename/delete the misnamed BM25Scorer; keep wingbacktest only if still useful. 9. **Close-out:** Google Ads clip out of `rejected/` + generic link to the trucker Google Ads incident note (no client detail in brain); 6 older clip dispositions; brain notes (see below); tag; clean 3 worktrees; amend brain CLAUDE.md (A). ## Explicitly NOT in this sprint - **#203** brain_query ranking is substring counting (root cause of the useless scorer). Real fix = proper BM25 (IDF + length norm) for the whole brain; changes production search ranking, needs a retrieval eval first. Next sprint; pairs with signals#95. - **#192 / #196 / #197 / #167** brain_edit path I/O, edit locks, brainfs fd-walk (risk/high write-path hardening) -> own sprint. - skills#56 holdouts (needs CAD read-scope check), signals#95 (comment + eval after #203), analysis actions: assessor-loop ADR-0001 check, agentsquad cost-per-lap. - infra: koala `syncthing.service` (user unit) failed since 2026-10-06 while two syncthing processes run — investigate, host-side. - Backlog triage: #36, #40, #41, #63, #64, #71, #80, #113, #114, #122 are 6+ weeks old; close/park decision for Mathias. ## Brain notes to write at close (generalised, no tooling-specific rationale) - A self-reported model confidence that is saturated carries no information; measure calibration on the real population before building a gate on it. - Run the end-to-end acceptance scenario before hardening the parts; three gate passes polished code that could not reach its goal. - Test/type names must match what they observe (BM25Scorer that counts substrings; a test whose input never reached the code it named). - "Rejected" in an inbox is a decision with an author; verify who decided before using it as a preference signal. --- _Created via git-mcp on behalf of @mathias_
mathias added the risk/mediumsize/ltype/feat labels 2026-10-20 17:49:41 +00:00
Author
Owner

Decisions (Mathias, 2026-10-20, "kör med standardvalen"): A yes (concepts/entities = shared cross-wing layer; brain CLAUDE.md to be amended at close-out). B yes (delete the 25 tunnel-candidates-* pages, via a reviewed commit, git keeps them). C yes (the 15 legacy notes/project pages stay untouched). D yes (remove the dead classifier/scorer). E yes — merge approval for this sprint's PRs (risk/medium, green CI on the head commit, independent pr-gatekeeper pass with no blocker). Does not extend beyond this sprint.

Routing design settled before coding (so PRs don't relitigate it): wing.Router, no LLM. (1) subfolder/frontmatter wing -> must be a plain name of an EXISTING wing (typo never creates one), else parked; conflicting signals parked. A deliberate subfolder is not gated by classification.yaml or a client list (extraction runs on the local model). (2) No signal -> content checked against the client-marker regex; a match parks, otherwise web-clip (the only wing created automatically). (3) Wings are read fresh from disk on each decision (fixes the read-once Known list). Satisfies the existing Resolve shape, so the watcher change is small.

PR plan: PR-A watcher write safety (#158, #198, #199 + residuals). PR-B acceptance test + shared concepts/entities + Router + promote reads needs-review + dead code removal. PR-C source migration (manifest, dry-run). PR-D close-out.


Created via git-mcp on behalf of @mathias

**Decisions (Mathias, 2026-10-20, "kör med standardvalen"):** A yes (concepts/entities = shared cross-wing layer; brain CLAUDE.md to be amended at close-out). B yes (delete the 25 `tunnel-candidates-*` pages, via a reviewed commit, git keeps them). C yes (the 15 legacy notes/project pages stay untouched). D yes (remove the dead classifier/scorer). E yes — merge approval for **this sprint's PRs** (risk/medium, green CI on the head commit, independent pr-gatekeeper pass with no blocker). Does not extend beyond this sprint. **Routing design settled before coding** (so PRs don't relitigate it): `wing.Router`, no LLM. (1) subfolder/frontmatter wing -> must be a plain name of an EXISTING wing (typo never creates one), else parked; conflicting signals parked. A deliberate subfolder is not gated by classification.yaml or a client list (extraction runs on the local model). (2) No signal -> content checked against the client-marker regex; a match parks, otherwise `web-clip` (the only wing created automatically). (3) Wings are read fresh from disk on each decision (fixes the read-once Known list). Satisfies the existing `Resolve` shape, so the watcher change is small. **PR plan:** PR-A watcher write safety (#158, #198, #199 + residuals). PR-B acceptance test + shared concepts/entities + Router + promote reads needs-review + dead code removal. PR-C source migration (manifest, dry-run). PR-D close-out. --- _Created via git-mcp on behalf of @mathias_
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/hyperguild#209