feat(watcher): inbox zero — archived raw/ originals are removed, not copied #206

2026-10-19T19:02:56Z

mathias commented

2026-10-19 19:02:56 +00:00

Refs #205 (step 0). Risk: medium — the watcher now deletes files in a Syncthing-shared folder (raw/), after archiving them.

What

After a note is archived into raw/processed/<date>/ or raw/failed/, processFile removes the original from raw/ through the os.Root (no symlink escape). raw/ then holds only unprocessed input, so brain_pending / the pending nudge stop listing already-ingested notes. The .processed/.failed marker is written only as a fallback if the removal fails (so a stuck file is not re-ingested every 30s). Existing markers are still honoured.

Evidence

Tests first: TestStart_ProcessesFile and TestStart_MovesToFailedOnError were changed to assert original removed + no marker; run red (4 assertion failures), then green.
go vet ./..., go test -race ./... (0 FAIL), golangci-lint on the package: 0 issues.

Known limits (not in this PR)

Re-sync of a removed file from an offline Syncthing peer would re-ingest it; pipeline.Run's source-hash drift check skips an unchanged source. Pinning test is step 5 of #205.
Retired-schema output paths (wiki/sources|concepts|entities) are #205 steps 1–4.
The ~12 legacy originals and .processed markers already in raw/ stay until #205 step 4.

Review ask

Verify by execution: removal cannot follow a symlink out of raw/; a failed archive copy never removes the original.

🤖 Generated with Claude Code

Created via git-mcp on behalf of @mathias

Refs #205 (step 0). Risk: **medium** — the watcher now deletes files in a Syncthing-shared folder (raw/), after archiving them. ## What After a note is archived into `raw/processed/<date>/` or `raw/failed/`, `processFile` removes the original from `raw/` through the `os.Root` (no symlink escape). `raw/` then holds only unprocessed input, so `brain_pending` / the pending nudge stop listing already-ingested notes. The `.processed`/`.failed` marker is written only as a fallback if the removal fails (so a stuck file is not re-ingested every 30s). Existing markers are still honoured. ## Evidence - Tests first: `TestStart_ProcessesFile` and `TestStart_MovesToFailedOnError` were changed to assert original removed + no marker; run red (4 assertion failures), then green. - `go vet ./...`, `go test -race ./...` (0 FAIL), `golangci-lint` on the package: 0 issues. ## Known limits (not in this PR) - Re-sync of a removed file from an offline Syncthing peer would re-ingest it; `pipeline.Run`'s source-hash drift check skips an unchanged source. Pinning test is step 5 of #205. - Retired-schema output paths (`wiki/sources|concepts|entities`) are #205 steps 1–4. - The ~12 legacy originals and `.processed` markers already in raw/ stay until #205 step 4. ## Review ask Verify by execution: removal cannot follow a symlink out of raw/; a failed archive copy never removes the original. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- _Created via git-mcp on behalf of @mathias_

mathias added 1 commit 2026-10-19 19:02:56 +00:00

feat(watcher): inbox zero — archived raw/ originals are removed, not copied

CI / Lint / Test / Vet (pull_request) Successful in 2m27s

Details

CI / Mirror to GitHub (pull_request) Skipped

Details

6bf61d75c0

After a note is archived into raw/processed/ or raw/failed/, remove the
original from raw/ so raw/ only holds unprocessed input. The .processed /
.failed marker is now only a fallback when the removal itself fails.
Existing markers are still honoured.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

mathias added 1 commit 2026-10-19 19:06:08 +00:00

fix(watcher): don't re-ingest a note that cannot be retired; pin archive-failure keeps original

CI / Lint / Test / Vet (pull_request) Successful in 1m3s

Details

CI / Mirror to GitHub (pull_request) Skipped

Details

87ae782527

When both Remove and the marker write fail (read-only raw/), remember the
path in memory so later polls skip it instead of writing another archive
copy every tick. Adds a test that a failed archive copy leaves the original
in raw/, mutation-checked against removing the early return.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

mathias commented

2026-10-19 19:06:09 +00:00

Independent review pass (pr-gatekeeper agent, verified by execution; reviewer is an agent, not four-eyes). Verdict at first look: not mergeable yet, CI pending. Findings:

Symlink swap: pass. A symlinked note / symlinked subdir pointing outside raw/ is never ingested, copied or removed; Remove goes through os.Root.
Archive copy failure keeps the original: pass in code, but was untested. Now pinned by TestProcessDir_KeepsTheOriginalWhenTheArchiveCopyFails (mutation-checked: removing the early return fails it). The first draft of that test hit the mkdir branch instead of the copy branch and survived the mutant — fixed.
Existing markers honoured: pass.
Hole (confirmed by test, fixed in the follow-up commit): with a read-only raw/, Remove and the marker write both fail, so the note was re-ingested every poll, writing a new archive copy each time. Fixed with an in-memory skip set; TestProcessDir_DoesNotReingestANoteItCannotRetire.
Open, not fixed here: small race — if Syncthing replaces the note between archive and Remove, the newer content is deleted after the older was archived. To be tracked as a follow-up on #205.
Not verified: sticky-bit dir case (Remove fails, marker succeeds), Syncthing propagation itself.

Merge only on green CI.

Created via git-mcp on behalf of @mathias

**Independent review pass (pr-gatekeeper agent, verified by execution; reviewer is an agent, not four-eyes).** Verdict at first look: **not mergeable yet, CI pending**. Findings: - Symlink swap: pass. A symlinked note / symlinked subdir pointing outside raw/ is never ingested, copied or removed; Remove goes through os.Root. - Archive copy failure keeps the original: pass in code, but **was untested**. Now pinned by `TestProcessDir_KeepsTheOriginalWhenTheArchiveCopyFails` (mutation-checked: removing the early return fails it). The first draft of that test hit the mkdir branch instead of the copy branch and survived the mutant — fixed. - Existing markers honoured: pass. - **Hole (confirmed by test, fixed in the follow-up commit):** with a read-only raw/, Remove and the marker write both fail, so the note was re-ingested every poll, writing a new archive copy each time. Fixed with an in-memory skip set; `TestProcessDir_DoesNotReingestANoteItCannotRetire`. - Open, not fixed here: small race — if Syncthing replaces the note between archive and Remove, the newer content is deleted after the older was archived. To be tracked as a follow-up on #205. - Not verified: sticky-bit dir case (Remove fails, marker succeeds), Syncthing propagation itself. Merge only on green CI. --- _Created via git-mcp on behalf of @mathias_

mathias added 1 commit 2026-10-19 19:06:49 +00:00

fix(watcher): keep a note that changed while it was being ingested

CI / Lint / Test / Vet (pull_request) Successful in 1m3s

Details

CI / Mirror to GitHub (pull_request) Skipped

Details

97ec9e7ce8

The archive holds the content read at the start; if Syncthing replaced the
note during the (minutes-long) LLM pass, removing the original would drop the
newer content. Compare size+mtime before removing; leave it for the next poll.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

mathias referenced this pull request

2026-10-19 19:09:13 +00:00

raw/ inbox: route auto-ingest to wiki/<wing>/sources/ (wing from raw/<wing>/ subfolder) #205

mathias added 1 commit 2026-10-19 19:09:14 +00:00

docs(watcher): state what changedSince can and cannot detect

CI / Lint / Test / Vet (pull_request) Successful in 1m4s

Details

CI / Mirror to GitHub (pull_request) Skipped

Details

e6e14102ee

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

mathias merged commit 2212ceedda into main

2026-10-19 19:22:58 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mathias/hyperguild#206