mathiasandClaude Opus 5
8359d61e33
Closes #44. That issue overstated the risk and this corrects it. The hook is three lines and echoes a string -- no bash-isms, so it cannot fail the way secret-guard appeared to. The real exposure is narrower and duller: it can stop working silently, and its entire job is to be visible. THE ASSERTION THAT MATTERS IS REGISTRATION. A hook file that exists but is absent from hooks.json does nothing at all and looks identical on disk to one that works. So the test derives the command from hooks.json and runs it exactly as registered, substituting CLAUDE_PLUGIN_ROOT the way the harness does, rather than inventing its own invocation. That choice is the lesson from #44's sibling. secret_guard_test.sh piped its hook into `sh` while hooks.json runs it with `bash`; on Debian /bin/sh is dash, which has no `set -o pipefail`, so the suite failed 20/20 in CI against a hook that was FINE in production. The test was wrong, not the hook -- and the earlier commit message for that fix said the shebang decides the interpreter, which hooks.json shows is not true. Deriving the invocation removes the whole class rather than fixing one instance of it. No red phase to report: the hook was already correct, so this is regression pinning, not TDD. Teeth proven by mutation against a throwaway copy instead: - hook runs but prints nothing -> 6 failures - hook removed from hooks.json -> caught, suite stops early - ONE ritual step dropped, rule 0 still named -> caught (9 passed, 1 failed) The third is the one worth having. Losing the whole line is obvious; quietly losing "failing test" from the ritual while the banner still reads correctly is the drift that would actually happen. Bump-Type: patch Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
104 lines
3.8 KiB
Bash
104 lines
3.8 KiB
Bash
#!/bin/sh
|
|
# Tests for hooks/pre-task-ritual.sh — the UserPromptSubmit hook that surfaces
|
|
# rule 0 on every prompt (#44).
|
|
#
|
|
# WHY THIS EXISTS, AND WHAT IT DELIBERATELY DOES NOT CLAIM
|
|
# This hook is three lines and echoes a string. It has no bash-isms, so it
|
|
# cannot fail the way secret-guard appeared to; that risk was overstated when
|
|
# this test was requested. What it CAN do is stop working silently, and its
|
|
# whole job is to be visible -- if the text stops appearing, rule 0 stops being
|
|
# surfaced on every prompt and nothing announces that.
|
|
#
|
|
# THE ASSERTION THAT MATTERS IS REGISTRATION.
|
|
# A hook file that exists but is not in hooks.json does absolutely nothing, and
|
|
# looks identical on disk to one that works. So this derives the command from
|
|
# hooks.json rather than assuming one: the registration is the source of truth
|
|
# for how the hook is invoked, and a test that invokes it its own way is
|
|
# testing something the harness never runs.
|
|
#
|
|
# That distinction is not hypothetical. secret_guard_test.sh piped its hook into
|
|
# `sh` while hooks.json runs it with `bash`; on Debian /bin/sh is dash, which
|
|
# has no `set -o pipefail`, and the suite failed 20/20 in CI against a hook that
|
|
# was fine in production. The test was wrong, not the hook. Deriving the
|
|
# invocation removes the opportunity for that class of mistake entirely.
|
|
|
|
set -u
|
|
|
|
ROOT=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
|
|
HOOK="$ROOT/hooks/pre-task-ritual.sh"
|
|
REG="$ROOT/hooks/hooks.json"
|
|
|
|
pass=0
|
|
fail=0
|
|
ok() { pass=$((pass + 1)); printf ' ok %s\n' "$1"; }
|
|
no() { fail=$((fail + 1)); printf ' FAIL %s\n' "$1"; }
|
|
|
|
if ! command -v jq >/dev/null 2>&1; then
|
|
printf ' SKIP: jq not installed -- cannot read the hook registration\n'
|
|
exit 0
|
|
fi
|
|
|
|
[ -f "$HOOK" ] || { printf ' FAIL %s does not exist\n' "$HOOK"; exit 1; }
|
|
ok 'hook file exists'
|
|
|
|
[ -f "$REG" ] || { printf ' FAIL %s does not exist\n' "$REG"; exit 1; }
|
|
|
|
# 1. Registered, and on the right event. UserPromptSubmit is what makes it fire
|
|
# on every prompt; registered under any other event it would still "exist"
|
|
# and still do nothing useful.
|
|
CMD=$(jq -r '
|
|
.hooks.UserPromptSubmit[]?.hooks[]?
|
|
| select(.command | test("pre-task-ritual"))
|
|
| .command
|
|
' "$REG" 2>/dev/null | head -1)
|
|
|
|
if [ -n "$CMD" ]; then
|
|
ok 'registered on UserPromptSubmit in hooks.json'
|
|
else
|
|
no 'not registered on UserPromptSubmit -- the file would exist and never run'
|
|
printf ' %d passed, %d failed\n' "$pass" "$fail"
|
|
exit 1
|
|
fi
|
|
|
|
# 2. Run it EXACTLY as registered, with the harness variable substituted the way
|
|
# the harness substitutes it.
|
|
OUT=$(CLAUDE_PLUGIN_ROOT="$ROOT" sh -c "$CMD" 2>&1)
|
|
rc=$?
|
|
|
|
[ "$rc" -eq 0 ] && ok 'exits 0 as registered' \
|
|
|| no "exited $rc -- a non-zero UserPromptSubmit hook can block the prompt"
|
|
|
|
# 3. The content is the point. A hook that runs and says nothing is the same as
|
|
# no hook, and that is the failure this pins.
|
|
if [ -n "$OUT" ]; then
|
|
ok 'produces output'
|
|
else
|
|
no 'produced no output -- rule 0 would silently stop being surfaced'
|
|
fi
|
|
|
|
case "$OUT" in
|
|
*"Rule 0"*) ok 'names rule 0' ;;
|
|
*) no "does not name rule 0: '$OUT'" ;;
|
|
esac
|
|
|
|
# Each of the four steps, individually. Naming rule 0 while having quietly lost
|
|
# a step is the plausible drift, not losing the whole line.
|
|
for step in brain_query skill 'failing test' 'success criterion'; do
|
|
case "$OUT" in
|
|
*"$step"*) ok "states the '$step' step" ;;
|
|
*) no "lost the '$step' step from the ritual" ;;
|
|
esac
|
|
done
|
|
|
|
# 4. The interpreter named in the registration must actually exist, or the hook
|
|
# fails on that host with a message nobody reads.
|
|
INTERP=${CMD%% *}
|
|
if command -v "$INTERP" >/dev/null 2>&1; then
|
|
ok "registered interpreter '$INTERP' is available"
|
|
else
|
|
no "registered interpreter '$INTERP' not found on this host"
|
|
fi
|
|
|
|
printf ' %d passed, %d failed\n' "$pass" "$fail"
|
|
[ "$fail" -eq 0 ]
|