feat(legacymove): clear wiki/sources and raw/ leftovers of the retired layout, with archive and rollback (#209) #213

2026-10-20T19:37:56Z

mathias commented

2026-10-20 19:37:56 +00:00

Sprint #209, PR-C. Refs #209 #205. Risk: medium by label, but it is a tool that DELETES live data when run with -apply; the dry run is the default. Mathias will see the dry-run table before any apply.

What

cmd/legacymove + internal/legacymove:

Plan: in wiki/sources/, type: video|article pages move to wiki/<wing>/sources/ (default web-clip, wing/hall stamped, body and links byte-preserved); tunnel-candidates-* (AutoTunnel review logs that were ingested as "sources") are deleted; notes/projects/untyped pages are left. A page whose slug is used elsewhere is skipped.
PlanRaw: in raw/, an original is deleted with its .processed marker only if it HAS the marker AND a byte-identical copy (SHA-256) exists under raw/processed/; a marker with no note is deleted; an unmarked note, a note whose archive copy differs, and tunnel-candidates-* (AutoTunnel's own queue) are never touched.
Apply: archives every page it touches into archive/wing-migration-<date>/originals/ (path flattened, so raw/x.md and wiki/sources/x.md cannot collide) and a manifest.json rewritten after every step; creates destinations O_EXCL; removes the source only after. Rollback restores byte for byte and refuses to overwrite.
The archive is outside raw/, so the watcher never ingests it.
concepts/ and entities/ are the shared layer and are not touched.

Evidence

Live dry run (read only, 2026-10-20): 84 entries = 16 move, 53 delete (24 tunnel logs + 29 raw leftovers), 15 left.
Tests first; go vet, go test -race ./... exit 0, 0 FAIL, lint 0. Mutation-checked: junk guard, type filter, slug collision, no-overwrite, archive-before-change, action gate, identical-copy check, marker requirement, tunnel exemption, orphan-marker rule, archive-key flattening. Two of my mutants failed to compile (unused variables) and were redone as valid ones.

Review ask (this deletes files on a live, Syncthing-shared, git-synced tree)

By execution: (1) can PlanRaw ever select a note that is still unprocessed or whose archive copy is not byte-identical, including a hash collision-free path (hardlinks, symlinks in raw/ or raw/processed/, a symlinked processed dir, a file replaced between Plan and Apply)? (2) Apply against a changing tree (the watcher running): a file that changed between Plan and Apply — is the archived copy the bytes that were deleted? (it reads, archives, then removes: is there a window where the removed file differs from the archived one?); (3) Rollback after a partial Apply (manifest written per step), rollback twice, rollback when a destination was edited; (4) path handling: names with newlines/unicode/.., archive key collisions after flattening (a/b__c vs a__b/c); (5) the 16 moves: any link or graph consequence (slug unchanged); (6) anything that deletes outside raw/ and wiki/sources/.

🤖 Generated with Claude Code

Created via git-mcp on behalf of @mathias

Sprint #209, PR-C. Refs #209 #205. Risk: **medium by label, but it is a tool that DELETES live data when run with `-apply`**; the dry run is the default. Mathias will see the dry-run table before any apply. ## What `cmd/legacymove` + `internal/legacymove`: - `Plan`: in `wiki/sources/`, `type: video|article` pages move to `wiki/<wing>/sources/` (default `web-clip`, wing/hall stamped, body and links byte-preserved); `tunnel-candidates-*` (AutoTunnel review logs that were ingested as "sources") are deleted; notes/projects/untyped pages are left. A page whose slug is used elsewhere is skipped. - `PlanRaw`: in `raw/`, an original is deleted with its `.processed` marker only if it HAS the marker AND a byte-identical copy (SHA-256) exists under `raw/processed/`; a marker with no note is deleted; an unmarked note, a note whose archive copy differs, and `tunnel-candidates-*` (AutoTunnel's own queue) are never touched. - `Apply`: archives every page it touches into `archive/wing-migration-<date>/originals/` (path flattened, so `raw/x.md` and `wiki/sources/x.md` cannot collide) and a `manifest.json` rewritten after every step; creates destinations O_EXCL; removes the source only after. `Rollback` restores byte for byte and refuses to overwrite. - The archive is outside `raw/`, so the watcher never ingests it. - `concepts/` and `entities/` are the shared layer and are not touched. ## Evidence - Live dry run (read only, 2026-10-20): 84 entries = 16 move, 53 delete (24 tunnel logs + 29 raw leftovers), 15 left. - Tests first; `go vet`, `go test -race ./...` exit 0, 0 FAIL, lint 0. Mutation-checked: junk guard, type filter, slug collision, no-overwrite, archive-before-change, action gate, identical-copy check, marker requirement, tunnel exemption, orphan-marker rule, archive-key flattening. Two of my mutants failed to compile (unused variables) and were redone as valid ones. ## Review ask (this deletes files on a live, Syncthing-shared, git-synced tree) By execution: (1) can `PlanRaw` ever select a note that is still unprocessed or whose archive copy is not byte-identical, including a hash collision-free path (hardlinks, symlinks in `raw/` or `raw/processed/`, a symlinked processed dir, a file replaced between Plan and Apply)? (2) `Apply` against a changing tree (the watcher running): a file that changed between Plan and Apply — is the archived copy the bytes that were deleted? (it reads, archives, then removes: is there a window where the removed file differs from the archived one?); (3) `Rollback` after a partial Apply (manifest written per step), rollback twice, rollback when a destination was edited; (4) path handling: names with newlines/unicode/`..`, archive key collisions after flattening (`a/b__c` vs `a__b/c`); (5) the 16 moves: any link or graph consequence (slug unchanged); (6) anything that deletes outside `raw/` and `wiki/sources/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- _Created via git-mcp on behalf of @mathias_

mathias added 4 commits 2026-10-20 19:37:56 +00:00

feat(legacymove): clear the retired wiki/sources layout with an archive and a rollback (#209 ) e5f8947f3b

Videos and articles move to the default wing (wing/hall stamped, body and
links byte-preserved); auto-tunnel review logs are deleted; notes, projects and
untyped pages stay. Plan is read-only; Apply archives every page it touches to
archive/wing-migration-<date>/ (outside raw/) before changing anything,
creates destinations O_EXCL and removes sources only after; Rollback restores
byte for byte. concepts/ and entities/ are the shared layer and are untouched.
Dry run against the live brain: 16 move, 24 delete, 15 left.
Mutation-checked: junk guard, type filter, slug collision, no-overwrite,
archive-before-change, action gate.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/wt/web-clip' into wt/legacy-move e14577d814

feat(legacymove): also clear raw/ leftovers from before inbox zero (#209 ) 2b16f2cc2f

PlanRaw: an original that was archived but never removed (it has a .processed
marker AND a byte-identical copy under raw/processed/) is deleted with its
marker; a marker without a note is deleted; an unmarked note, a note without an
identical archive copy, and AutoTunnel's tunnel-candidates-* review queue are
never touched. The archive key flattens the whole path so raw/x.md and
wiki/sources/x.md cannot overwrite each other. Live dry run: 84 entries, 16
move, 53 delete (24 junk + 29 raw leftovers), 15 left.
Mutation-checked.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into wt/legacy-move

CI / Lint / Test / Vet (pull_request) Successful in 1m5s

Details

CI / Mirror to GitHub (pull_request) Skipped

Details

f2abee9b21

mathias referenced this issue from a commit

2026-10-20 19:42:46 +00:00

fix(legacymove): second run never reuses an archive; Apply re-checks the planned bytes; rollback checks first and is repeatable (review of #213)

mathias added 1 commit 2026-10-20 19:42:46 +00:00

fix(legacymove): second run never reuses an archive; Apply re-checks the planned bytes; rollback checks first and is repeatable (review of #213 )

CI / Lint / Test / Vet (pull_request) Successful in 1m3s

Details

CI / Mirror to GitHub (pull_request) Skipped

Details

b66b45b8fd

From the independent review:
- the archive dir is unique per run (-2, -3 ...): a second Apply the same day
  replaced the first run's archived original and rewrote its manifest
- every Move/Delete carries the SHA-256 it was planned on; Apply skips and
  reports (Result.Changed) a page whose bytes changed since planning, so a note
  dropped again after PlanRaw is not deleted unprocessed
- Rollback verifies all targets before changing anything, recognises pages
  already restored (so it can be run again), and refuses to delete a moved copy
  that was edited since the move
- names whose flattened archive key exceeds NAME_MAX are skipped at planning
  time instead of wedging the run
- a page that already has wing/hall keys, or whose type sits in a nested/block
  scalar line or an unterminated block, is not moved
Each has a test that fails without it (mutation-checked). Residual: a swap
between Apply's read and its remove; the archived copy is still the bytes read.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

mathias commented

2026-10-20 19:42:49 +00:00

Independent review pass 1 (pr-gatekeeper agent, by execution): BLOCK. Do not run -apply on the live brain until the next pass. Findings and resolution at the head above (tests first, mutation-checked):

HIGH a second Apply on the same UTC day replaced the first run's archived original (PutRegular) and rewrote its manifest from empty -> a deleted page with no surviving copy. Fixed: archive dir unique per run (-2, -3).
HIGH (conditional) PlanRaw's invariant (identical copy exists) was not enforced at Apply: a note re-dropped between plan and apply was archived and deleted as unprocessed. Fixed: every Move/Delete carries the SHA-256 it was planned on; Apply skips and reports changed pages (Result.Changed).
MEDIUM Rollback was not resumable (stopped at the first already-restored page), deleted an edited moved copy, and could half-roll-back. Fixed: all targets verified first, nothing changed on a conflict, already-restored pages recognised by bytes, an edited moved copy is never deleted (rollback refuses).
MEDIUM a long name's flattened archive key exceeded NAME_MAX and wedged every rerun (0 such names on the live brain today). Fixed: skipped at planning.
LOW a page that already had wing:/hall: would get duplicate keys; type: inside a nested/block-scalar line or an unterminated block classified a page. Fixed: such pages are not moved.

Accepted/documented: the archive write is O_EXCL in intent but the dir is always fresh, so a mutant that uses PutRegular for it is not caught by a test (defence in depth only); a swap between Apply's read and its remove (the archived copy is still the bytes read); mtime/mode of restored pages are new; restore order note-then-marker leaves a restored pair "held" with a one-time warning, which matches its pre-move state.
Held up under the reviewer's probes: false-positive PlanRaw cases (1-byte difference, symlinked archive dir/file, copy only under raw/failed/) are all skipped; nothing outside raw/ and wiki/sources/ can be deleted; symlinks and .. refused; -apply off by default; the live dry run is read-only.

Gate: go vet ./..., go test -race ./... exit 0, 0 FAIL, lint 0. A second pass follows before any apply; you will see the dry-run table first.

Created via git-mcp on behalf of @mathias

**Independent review pass 1 (pr-gatekeeper agent, by execution): BLOCK.** Do not run `-apply` on the live brain until the next pass. Findings and resolution at the head above (tests first, mutation-checked): 1. **HIGH** a second Apply on the same UTC day replaced the first run's archived original (PutRegular) and rewrote its manifest from empty -> a deleted page with no surviving copy. Fixed: archive dir unique per run (`-2`, `-3`). 2. **HIGH (conditional)** PlanRaw's invariant (identical copy exists) was not enforced at Apply: a note re-dropped between plan and apply was archived and deleted as unprocessed. Fixed: every Move/Delete carries the SHA-256 it was planned on; Apply skips and reports changed pages (`Result.Changed`). 3. **MEDIUM** Rollback was not resumable (stopped at the first already-restored page), deleted an edited moved copy, and could half-roll-back. Fixed: all targets verified first, nothing changed on a conflict, already-restored pages recognised by bytes, an edited moved copy is never deleted (rollback refuses). 4. **MEDIUM** a long name's flattened archive key exceeded NAME_MAX and wedged every rerun (0 such names on the live brain today). Fixed: skipped at planning. 6. **LOW** a page that already had `wing:`/`hall:` would get duplicate keys; `type:` inside a nested/block-scalar line or an unterminated block classified a page. Fixed: such pages are not moved. - Accepted/documented: the archive write is O_EXCL in intent but the dir is always fresh, so a mutant that uses PutRegular for it is not caught by a test (defence in depth only); a swap between Apply's read and its remove (the archived copy is still the bytes read); mtime/mode of restored pages are new; restore order note-then-marker leaves a restored pair "held" with a one-time warning, which matches its pre-move state. - Held up under the reviewer's probes: false-positive `PlanRaw` cases (1-byte difference, symlinked archive dir/file, copy only under `raw/failed/`) are all skipped; nothing outside `raw/` and `wiki/sources/` can be deleted; symlinks and `..` refused; `-apply` off by default; the live dry run is read-only. Gate: `go vet ./...`, `go test -race ./...` exit 0, 0 FAIL, lint 0. A second pass follows before any apply; you will see the dry-run table first. --- _Created via git-mcp on behalf of @mathias_

mathias commented

2026-10-20 19:49:01 +00:00

Independent review pass 2 (pr-gatekeeper agent, by execution): not blocked, clear for a dry run and for apply at b66b45b (the reviewer found nothing that violates: delete only after a byte-for-byte archive, never overwrite, rollback restores exactly and never deletes user data).

Re-verified by execution: each pass-1 repro is fixed and flipping the fix turns a test red (SHA check, name-length guard, wing/hall key skip, edited-moved-copy rollback, target-exists preflight; the archive-dir uniqueness mutants did not compile or hung, but the behaviour was confirmed: an existing archive/wing-migration-2026-10-20 gave -2). Attacks that held: A->B->A bytes (moved, correct), empty SHA (untouched, in Changed), plan for another dir (Changed or error, nothing touched), duplicate delete (error, manifest consistent, rollback restores), a late destination (source kept, destination untouched, no manifest entry), symlinked archive/ (refused, nothing written outside). archive/ is not read by search, vector sync, graph sync or brain_get/read (denied prefix); the watcher walks only raw/. Performance: 1000 notes plan in 83 ms, apply 2000 deletes 199 ms.

Low items left (not changed): (1) the manifest is rewritten with a truncating write, so a kill mid-write could leave a short manifest.json (originals are still archived, so nothing is lost but one-command rollback would be); (2) a failed first item leaves an archive with originals and no manifest; (3) a page edited between the SHA read and the remove is deleted, with the archive holding the planned bytes (milliseconds, documented); (4) one failed item aborts the run (re-plan and re-run is safe, fresh archive each time); (5) rollback trusts the manifest, bounded to deleting a file byte-identical to a stamped archived original. Not verified: concurrent Apply runs, live-brain contents beyond the dry run, -rollback ../x.

Gate: go vet ./..., go test -race ./... exit 0, 0 FAIL, lint 0.

Created via git-mcp on behalf of @mathias

**Independent review pass 2 (pr-gatekeeper agent, by execution): not blocked, clear for a dry run and for apply** at b66b45b (the reviewer found nothing that violates: delete only after a byte-for-byte archive, never overwrite, rollback restores exactly and never deletes user data). Re-verified by execution: each pass-1 repro is fixed and flipping the fix turns a test red (SHA check, name-length guard, wing/hall key skip, edited-moved-copy rollback, target-exists preflight; the archive-dir uniqueness mutants did not compile or hung, but the behaviour was confirmed: an existing `archive/wing-migration-2026-10-20` gave `-2`). Attacks that held: A->B->A bytes (moved, correct), empty SHA (untouched, in Changed), plan for another dir (Changed or error, nothing touched), duplicate delete (error, manifest consistent, rollback restores), a late destination (source kept, destination untouched, no manifest entry), symlinked `archive/` (refused, nothing written outside). `archive/` is not read by search, vector sync, graph sync or brain_get/read (denied prefix); the watcher walks only `raw/`. Performance: 1000 notes plan in 83 ms, apply 2000 deletes 199 ms. Low items left (not changed): (1) the manifest is rewritten with a truncating write, so a kill mid-write could leave a short `manifest.json` (originals are still archived, so nothing is lost but one-command rollback would be); (2) a failed first item leaves an archive with originals and no manifest; (3) a page edited between the SHA read and the remove is deleted, with the archive holding the planned bytes (milliseconds, documented); (4) one failed item aborts the run (re-plan and re-run is safe, fresh archive each time); (5) rollback trusts the manifest, bounded to deleting a file byte-identical to a stamped archived original. Not verified: concurrent Apply runs, live-brain contents beyond the dry run, `-rollback ../x`. Gate: `go vet ./...`, `go test -race ./...` exit 0, 0 FAIL, lint 0. --- _Created via git-mcp on behalf of @mathias_

All checks were successful

CI / Lint / Test / Vet (pull_request) Successful in 1m3s

Required

Details

CI / Mirror to GitHub (pull_request) Skipped

Details

You are not authorized to merge this pull request.

This pull request can be merged automatically.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin wt/legacy-move:wt/legacy-move

git checkout wt/legacy-move

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mathias/hyperguild#213