feat(agents): ship pr-gatekeeper from the repo, not one host's ~/.claude (#51)
CI / Check / Test (push) Successful in 4s
CI / Tag release (push) Successful in 1s

pr-gatekeeper blocked dispatch#75 five times. Every block was a real,
executable escape — twelve pathspec bypasses, arbitrary command
execution via `git -c alias`, `--upload-pack='rm …'`, a write primitive
through `--output=`, `${IFS}` word-splitting past the policy, and a path
denylist that had been a no-op in the live flow since it was written.
task check was green on all five.

The agent definition existed in no repository. A code search across the
estate found only references to it — comments in dispatch source, a
local-dev sprint doc — while the file itself lived at
~/.claude/agents/pr-gatekeeper.md on koala alone: unversioned, on one
host, one rm from gone. local-dev AGENT.md rule 8 now requires routing
control code through it, so doctrine depended on an artifact with no
history.

PORTED, PLUS WHAT THE FIVE ROUNDS ACTUALLY TAUGHT

The file already had flip-testing. Three habits that did the finding were
implied at best, and are now explicit (§4b):

  - Verify by EXECUTION, not by reading the implementation. Every finding
    came from running the command in a scratch repo and looking at the
    filesystem. The matcher said one thing and the filesystem another,
    five times.
  - Read the SURVIVORS. After an allow/deny sweep, read what is still
    permitted, one at a time. This caught `git commit -F /etc/passwd` —
    exfiltration wearing a message flag — which neither the review nor
    the differential found, because both implementations agreed and both
    were wrong. Agreement is not correctness.
  - Grade a tool on its EXIT CODE, never by grepping output. A crashed
    linter greps as clean.

Plus rule 9 in §4: a test whose name claims more than it proves is worse
than no test, because the name wins in the reader's head.

AND THE INCIDENT THIS AGENT CAUSED (§4c)

On 2026-09-19 a probe of this agent's own making printed /proc/$PPID/environ
and `ps e` raw, and a `git credential fill` consulted the host's real
store. Seven live credentials reached the transcript and need rotation
(infra#445). The finding was correct; the method was not. Versioning the
file without that lesson would have shipped the incident along with the
agent. So: a probe that reads an environment prints a count, never the
buffer; invented sentinels only; isolated HOME for anything touching
git's credential subsystem; prove a primitive and stop.

DOCTOR

New `shadow` check: a real file in ~/.claude/agents with the same name as
a shipped agent wins over the shipped one and nothing says so. Same
silent-drift shape as the existing `outside` and `unmanaged` checks — it
works, so nothing complains, while serving content nobody updates. A
symlink is managed wiring and is not flagged.

The host-local copy on koala has been moved aside, so the shipped file is
what runs once the plugin updates.

TESTS

14 assertions, each named for what it observes rather than what it
guarantees — a grep proves a phrase is present, not that a procedure is
followed, and naming it otherwise would be the defect rule 9 exists for.
Flip-tested both directions: removing the doctor check, and altering the
probe-safety phrase, each turn the suite red.

`agents` added to task check's root-entry allowlist. Leaving it out made
check_tolerates_unverifiable_test.sh fail, which is #23's stray-entry
guard working exactly as designed — an undeclared root entry loads in no
harness and errors in none either.

Closes #51

Bump-Type: minor
This commit is contained in:
2026-09-28 23:30:37 +02:00
parent e2a93c1e99
commit c25175e4b3
5 changed files with 359 additions and 2 deletions
+108
View File
@@ -0,0 +1,108 @@
#!/bin/sh
# Tests for the agents/ directory the plugin ships, and doctor's shadow check.
#
# skills#51: pr-gatekeeper blocked dispatch#75 five times, every block a real
# escape verified by execution — and the agent definition lived only at
# ~/.claude/agents/pr-gatekeeper.md on one host. Unversioned, unreviewable, one
# rm from gone. Doctrine now depends on it (local-dev AGENT.md rule 8), so the
# file has to be in the repo.
#
# These assertions observe FILE CONTENT and doctor's OUTPUT. They cannot and do
# not claim the agent reviews well — naming them otherwise would be the exact
# defect rule 9 exists for.
set -u
ROOT=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
AGENT="$ROOT/agents/pr-gatekeeper.md"
DOCTOR="$ROOT/bin/doctor"
pass=0
fail=0
WORK=${TMPDIR:-/tmp}/agents_test.$$
ok() { pass=$((pass + 1)); printf 'ok: %s\n' "$1"; }
no() { fail=$((fail + 1)); printf 'FAIL: %s\n' "$1"; }
# --- the file is in the repo ------------------------------------------------
if [ -f "$AGENT" ]; then
ok 'agents/pr-gatekeeper.md exists in the checkout'
else
no 'agents/pr-gatekeeper.md exists in the checkout'
printf '\n%s passed, %s failed\n' "$pass" "$fail"
exit 1
fi
# --- frontmatter is present and well-formed ---------------------------------
head -1 "$AGENT" | grep -q '^---$' \
&& ok 'the file opens with a frontmatter fence' \
|| no 'the file opens with a frontmatter fence'
for field in name description tools; do
if sed -n '2,/^---$/p' "$AGENT" | grep -q "^$field:"; then
ok "frontmatter declares $field"
else
no "frontmatter declares $field"
fi
done
fm_name=$(sed -n '2,/^---$/p' "$AGENT" | sed -n 's/^name: *//p' | head -1)
if [ "$fm_name" = "pr-gatekeeper" ]; then
ok 'the frontmatter name equals the filename stem'
else
no "the frontmatter name equals the filename stem (got '$fm_name')"
fi
# --- the practices that actually found the defects --------------------------
#
# Each string below is traceable to a specific round of dispatch#75 or to the
# incident the agent itself caused. A grep is a weak check — it proves the
# phrase is present, not that the procedure is followed — so the test is named
# for the phrase, not for the behaviour.
check_phrase() {
if grep -qi -- "$1" "$AGENT"; then
ok "the file names: $2"
else
no "the file names: $2"
fi
}
check_phrase 'verify by execution' 'verify by execution, not by reading the implementation'
check_phrase 'flip' 'flip-testing'
check_phrase 'read the survivors' 'read the survivors of a differential'
check_phrase 'exit code' 'grade a linter on its exit code'
check_phrase 'never the buffer' 'a probe prints a count, never the buffer'
check_phrase 'claims more than' 'a test name must not claim more than it proves'
# --- doctor flags a host-local copy that shadows the shipped one ------------
setup() {
rm -rf "$WORK"
mkdir -p "$WORK/home/.claude/agents" "$WORK/ck/skills/x" "$WORK/ck/agents"
printf 'x\n' > "$WORK/ck/skills/x/SKILL.md"
printf -- '---\nname: pr-gatekeeper\n---\n' > "$WORK/ck/agents/pr-gatekeeper.md"
}
setup
printf -- '---\nname: pr-gatekeeper\n---\n' > "$WORK/home/.claude/agents/pr-gatekeeper.md"
out=$(HOME="$WORK/home" DOCTOR_CHECKOUT="$WORK/ck" sh "$DOCTOR" 2>&1)
if printf '%s' "$out" | grep -q 'pr-gatekeeper'; then
ok 'doctor reports a host-local agent that shadows a shipped one'
else
no "doctor reports a host-local agent that shadows a shipped one (got: $out)"
fi
setup
out=$(HOME="$WORK/home" DOCTOR_CHECKOUT="$WORK/ck" sh "$DOCTOR" 2>&1)
if printf '%s' "$out" | grep -q 'pr-gatekeeper'; then
no "doctor is silent when no host-local copy exists (got: $out)"
else
ok 'doctor is silent when no host-local copy exists'
fi
rm -rf "$WORK"
printf '\n%s passed, %s failed\n' "$pass" "$fail"
[ "$fail" -eq 0 ]