Compare commits

...
4 Commits
Author SHA1 Message Date
mathiasandClaude Opus 5 4edf42ce9f fix(test): run the secret guard via its own shebang, not through sh
CI / Check / Test (push) Successful in 3s
CI / Tag release (push) Successful in 0s
The suite piped hook input into `sh "$GUARD"`, which overrides the hook's
`#!/usr/bin/env bash` and tests an interpreter production never uses. Claude
Code executes the hook as a program.

On Debian /bin/sh is dash, which has no `set -o pipefail`, so the guard died
on line 23 before reading a byte of input:

    secret-guard.sh: 23: set: Illegal option -o pipefail
    secret_guard: 0 passed, 20 failed

while passing 20/20 on a host where /bin/sh happens to be bash. A security
gate that tests green and cannot start is the worst of both, and the test was
the only thing standing between those two states.

Found because baked-v16 added jq to the runner: this suite needs jq to build
its hook JSON, so it had been silently doing nothing in CI and started running
for the first time. Same shape as installing zip and immediately learning the
gitea backup check had never run -- a missing tool does not announce that it
is suppressing a whole suite.

Also asserts the hook is executable, since direct invocation depends on the
exec bit, and that assertion is what stops this regressing to `sh "$GUARD"`.

Checked the rest of the suite: every other `sh "$SUT"` targets a POSIX script,
which is correct. hooks/pre-task-ritual.sh is the other bash-shebang hook and
has no test at all -- filed separately rather than fixed here.

Bump-Type: patch

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 22:23:06 +02:00
mathiasandClaude Opus 5 2f53346b8c docs(check): say plainly that CI can never verify the MCP env wiring
CI / Check / Test (push) Failing after 2s
CI / Tag release (push) Skipped
infra#368 added jq to the gitea-runner image on the stated grounds that
check-mcp-env would then verify in CI instead of skipping. Verified after
rolling baked-v16: jq IS present now, the `SKIP: jq not installed` branch is
gone -- and the check still reports unverified, because CI has no ~/.op-env.

The env-file branch was always the binding one. It holds live credentials, so
it is not on the runner and must not be, which means this check can only ever
verify on a host that has one. Putting ~/.op-env on the runner to turn the
NOTE green would be trading a real secret for a green tick.

So the message now says so, at the point someone reads it. The risk otherwise
is not the skip -- it is a future reader seeing a passing pipeline and
concluding the MCP wiring is covered. It is covered on developer hosts only.

jq was still worth adding: it is a genuine dependency of this checker, the
runner's tool-contract.txt now asserts it, and a future image built without it
is a red pod at roll time rather than a silently degraded check.

Bump-Type: patch

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 22:21:10 +02:00
mathias dcc0677270 chore(release): plugin.json version 0.19.2
Skip-Release: true
2026-09-07 18:14:53 +00:00
mathiasandClaude Opus 5 9fd5a054b3 fix(check): tolerate an unverifiable check instead of dying on it
CI / Check / Test (push) Failing after 1s
CI / Tag release (push) Skipped
bin/check-mcp-env exits 2 for "could not run" -- no jq, or no env-file --
and the gate was written to report that as a NOTE rather than fail:

    "$DIR/check-mcp-env" "$DIR/.mcp.json" "$HOME/.op-env"
    case $? in 0) ;; 2) echo "NOTE: ..." ;; *) fail=1 ;; esac

That tolerance never executed. Task runs each cmds block under `sh -e`, so
a bare command exiting non-zero aborts the shell before `case $?` is
reached. main went red on the very commit that added the check (run #57),
with `task: Failed to run task "check": exit status 2`.

It passed locally because this host has both jq and ~/.op-env, so the
check exited 0 and the untested branch never ran. CI has neither. A
tolerated exit code that is only tolerated where it never fires is not
tolerated at all.

`|| rc=$?` puts the invocation in a compound command, which `set -e`
exempts, so `case` is reached. Both exit-2 paths verified: no env-file
(the new test's precondition) and no jq (PATH sandbox, same SKIP message
CI printed).

The test pins the tolerance itself rather than the checker: it points HOME
at an empty directory, which is exactly CI's condition, and asserts `task
check` exits 0 AND still says it could not verify -- silence would mean
skipped rather than tolerated.

Bump-Type: patch

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 20:14:19 +02:00
5 changed files with 103 additions and 4 deletions
+1 -1
View File
@@ -1,7 +1,7 @@
{ {
"name": "skills", "name": "skills",
"description": "Mathias's engineering skills library, plus the hooks and MCP servers that make it enforceable: TDD, clean code, debugging, specs, discovery, CI, and secret hygiene.", "description": "Mathias's engineering skills library, plus the hooks and MCP servers that make it enforceable: TDD, clean code, debugging, specs, discovery, CI, and secret hygiene.",
"version": "0.19.0", "version": "0.19.2",
"author": { "author": {
"name": "Mathias", "name": "Mathias",
"url": "https://git.d-ma.be/mathias" "url": "https://git.d-ma.be/mathias"
+7 -2
View File
@@ -145,8 +145,13 @@ tasks:
# not run (no jq, or no env-file — CI has neither); that is reported # not run (no jq, or no env-file — CI has neither); that is reported
# and not fatal, but it is never treated as a pass. # and not fatal, but it is never treated as a pass.
if [ -x "{{.TASKFILE_DIR}}/bin/check-mcp-env" ]; then if [ -x "{{.TASKFILE_DIR}}/bin/check-mcp-env" ]; then
"{{.TASKFILE_DIR}}/bin/check-mcp-env" "{{.TASKFILE_DIR}}/.mcp.json" "$HOME/.op-env" # `|| rc=$?` is load-bearing, not style. Task runs this block under
case $? in # `sh -e`, so a bare invocation exiting non-zero kills the shell
# before `case` is reached -- the tolerance below never ran, and the
# gate died with `exit status 2` on the commit that added it.
rc=0
"{{.TASKFILE_DIR}}/bin/check-mcp-env" "{{.TASKFILE_DIR}}/.mcp.json" "$HOME/.op-env" || rc=$?
case $rc in
0) ;; 0) ;;
2) echo "NOTE: MCP env wiring unverified on this host" >&2 ;; 2) echo "NOTE: MCP env wiring unverified on this host" >&2 ;;
*) fail=1 ;; *) fail=1 ;;
+10
View File
@@ -41,7 +41,17 @@ if [ ! -f "$CONFIG" ]; then
fi fi
if [ ! -f "$ENVFILE" ]; then if [ ! -f "$ENVFILE" ]; then
# EXPECTED IN CI, and permanently so. The env-file holds live credentials, so
# it is not on the runner and must not be -- which means this check can only
# ever verify on a host that has one. infra#368 added jq to the runner image
# believing that would let CI verify; it removed the jq branch below and
# changed nothing else, because this branch is the binding one.
#
# Said plainly so nobody concludes CI covers this. It does not, it cannot,
# and putting ~/.op-env on the runner to make it would be trading a real
# secret for a green tick.
echo "ERROR: no such env-file: $ENVFILE -- cannot verify MCP variables" >&2 echo "ERROR: no such env-file: $ENVFILE -- cannot verify MCP variables" >&2
echo " (expected in CI: this check only verifies on a host with ~/.op-env)" >&2
exit 2 exit 2
fi fi
@@ -0,0 +1,70 @@
#!/bin/sh
# `task check` must survive a check that reports "could not verify".
#
# The regression these exist for: bin/check-mcp-env exits 2 to mean "I could
# not run — no jq, or no env-file", which the gate deliberately treats as a
# reported NOTE rather than a failure. The Taskfile said exactly that:
#
# "$DIR/check-mcp-env" "$DIR/.mcp.json" "$HOME/.op-env"
# case $? in 0) ;; 2) echo "NOTE: ..." ;; *) fail=1 ;; esac
#
# and it did not work. Task runs each cmds block under `sh -e`, so a bare
# command exiting non-zero aborts the shell *before* `case $?` is ever
# reached. The whole gate died with `exit status 2`. Locally the check found
# jq and exited 0, so it went green; CI has neither jq nor ~/.op-env, so main
# went red on the very commit that added it (skills run #57, 2026-09-07).
#
# A tolerated exit code that is only tolerated on the host where it never
# fires is not tolerated at all. This pins the tolerance itself.
#
# No mocking: pointing HOME at an empty directory removes ~/.op-env, which is
# exactly the condition CI runs under, so check-mcp-env genuinely exits 2.
set -u
ROOT=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
pass=0
fail=0
ok() { pass=$((pass + 1)); printf ' ok %s\n' "$1"; }
no() { fail=$((fail + 1)); printf ' FAIL %s\n' "$1"; }
if ! command -v task >/dev/null 2>&1; then
printf ' SKIP: task not installed -- cannot exercise the gate\n'
exit 0
fi
TMP=$(mktemp -d)
trap 'rm -rf "$TMP"' EXIT
# Sanity: the precondition must actually hold, or the test proves nothing.
# An empty HOME means no ~/.op-env, so check-mcp-env cannot verify anything.
"$ROOT/bin/check-mcp-env" "$ROOT/.mcp.json" "$TMP/.op-env" >/dev/null 2>&1
rc=$?
if [ "$rc" -eq 2 ]; then
ok 'precondition: check-mcp-env exits 2 when the env-file is absent'
else
no "precondition: expected check-mcp-env to exit 2, got $rc"
fi
# The assertion. `task check` must still pass, and must say why it could not
# verify -- silence would mean the check was skipped rather than tolerated.
out=$(HOME="$TMP" task --dir "$ROOT" check 2>&1)
rc=$?
if [ "$rc" -eq 0 ]; then
ok 'task check passes when MCP env wiring cannot be verified'
else
no "task check exited $rc when check-mcp-env reported unverifiable"
printf '%s\n' "$out" | sed 's/^/ | /' | tail -8
fi
if printf '%s' "$out" | grep -q 'unverified'; then
ok 'the unverifiable check is reported, not silently skipped'
else
no 'no NOTE reported -- an unverifiable check must never look like a pass'
fi
printf ' %d passed, %d failed\n' "$pass" "$fail"
[ "$fail" -eq 0 ]
+15 -1
View File
@@ -24,7 +24,7 @@ decision_for() {
_tool=${2:-Bash} _tool=${2:-Bash}
jq -n --arg t "$_tool" --arg c "$_cmd" \ jq -n --arg t "$_tool" --arg c "$_cmd" \
'{tool_name: $t, tool_input: {command: $c}}' \ '{tool_name: $t, tool_input: {command: $c}}' \
| sh "$GUARD" \ | "$GUARD" \
| jq -r '.hookSpecificOutput.permissionDecision // "MISSING"' | jq -r '.hookSpecificOutput.permissionDecision // "MISSING"'
} }
@@ -42,6 +42,20 @@ expect() {
fi fi
} }
# Invoked as a PROGRAM above, never as `sh "$GUARD"`. Claude Code executes the
# hook directly, so its `#!/usr/bin/env bash` shebang decides the interpreter;
# piping it into `sh` overrides that and tests something production never runs.
#
# It is not academic. On Debian /bin/sh is dash, which has no `set -o
# pipefail`, so the guard died on line 23 before reading a byte of input and
# every deny case returned empty -- 20 of 20 assertions failed in CI while
# passing on a host where /bin/sh happens to be bash. A security gate that
# tests green and cannot start is the worst of both.
if [ ! -x "$GUARD" ]; then
printf 'FAIL: %s is not executable -- it must run via its own shebang\n' "$GUARD"
exit 1
fi
if [ ! -f "$GUARD" ]; then if [ ! -f "$GUARD" ]; then
printf 'FAIL: %s does not exist\n' "$GUARD" printf 'FAIL: %s does not exist\n' "$GUARD"
exit 1 exit 1