symlinkguard: Watch alert-queue tests flake under -race and CPU load #193

New Issue

2026-10-05T21:01:06Z

mathias commented

2026-10-05 21:01:06 +00:00

Source: found during the pr-gatekeeper review of PR #190 (round 3), 2026-10-05.

On origin/main I compiled the symlinkguard tests with -race (go test -race -c ./internal/symlinkguard/). Running 8 copies of the binary in parallel, each with -test.count=10, 3 of the 8 failed:

TestWatchAccountsForEveryRemovalDuringAnOutage (2)
TestWatchQueueIsBoundedDuringAnOutage (1)

The PR #190 head fails the same way under the same load: TestWatchAccountsForEveryRemovalDuringAnOutage and TestWatchDeliversEveryQueuedAlertAfterAShortOutage, 1 each. It also failed once in a plain sequential go test -race -count=5 run. The new #179 tests never failed.

These are timing-dependent Watch/deliver tests: 10ms intervals with wall-clock waits. Under CPU contention they flake. The danger is a red gate that people learn to re-run. It can also go the other way: a test loosened to stop flaking stops proving what it is named for.

Done when

The Watch alert-queue tests pass 8 parallel x -test.count=10 under -race with zero failures. This must come from a deterministic clock or synchronisation, not longer timeouts.
A flip of each test's mechanism still turns it red.

**Source:** found during the pr-gatekeeper review of PR #190 (round 3), 2026-10-05. On origin/main I compiled the symlinkguard tests with `-race` (`go test -race -c ./internal/symlinkguard/`). Running 8 copies of the binary in parallel, each with `-test.count=10`, 3 of the 8 failed: - `TestWatchAccountsForEveryRemovalDuringAnOutage` (2) - `TestWatchQueueIsBoundedDuringAnOutage` (1) The PR #190 head fails the same way under the same load: `TestWatchAccountsForEveryRemovalDuringAnOutage` and `TestWatchDeliversEveryQueuedAlertAfterAShortOutage`, 1 each. It also failed once in a plain sequential `go test -race -count=5` run. The new #179 tests never failed. These are timing-dependent Watch/deliver tests: 10ms intervals with wall-clock waits. Under CPU contention they flake. The danger is a red gate that people learn to re-run. It can also go the other way: a test loosened to stop flaking stops proving what it is named for. ## Done when - The Watch alert-queue tests pass 8 parallel x `-test.count=10` under `-race` with zero failures. This must come from a deterministic clock or synchronisation, not longer timeouts. - A flip of each test's mechanism still turns it red.

mathias referenced this issue

2026-10-05 21:01:19 +00:00

fix(symlinkguard): report and remove FIFOs, sockets and devices too (#179) #190

mathias referenced this issue

2026-10-05 21:01:20 +00:00

symlinkguard ignores FIFOs and other special files: a planted FIFO classification.yaml crash-loops the pod #179

mathias referenced this issue

2026-10-05 21:01:58 +00:00

fix: creating and generated writes go through the brain root (#188, #158 part A) #189

mathias referenced this issue

2026-10-05 21:01:59 +00:00

brain_write follows a symlinked wing directory, including one that points outside the brain #188