fix(check): tolerate an unverifiable check instead of dying on it
CI / Check / Test (push) Failing after 1s
CI / Tag release (push) Skipped

bin/check-mcp-env exits 2 for "could not run" -- no jq, or no env-file --
and the gate was written to report that as a NOTE rather than fail:

    "$DIR/check-mcp-env" "$DIR/.mcp.json" "$HOME/.op-env"
    case $? in 0) ;; 2) echo "NOTE: ..." ;; *) fail=1 ;; esac

That tolerance never executed. Task runs each cmds block under `sh -e`, so
a bare command exiting non-zero aborts the shell before `case $?` is
reached. main went red on the very commit that added the check (run #57),
with `task: Failed to run task "check": exit status 2`.

It passed locally because this host has both jq and ~/.op-env, so the
check exited 0 and the untested branch never ran. CI has neither. A
tolerated exit code that is only tolerated where it never fires is not
tolerated at all.

`|| rc=$?` puts the invocation in a compound command, which `set -e`
exempts, so `case` is reached. Both exit-2 paths verified: no env-file
(the new test's precondition) and no jq (PATH sandbox, same SKIP message
CI printed).

The test pins the tolerance itself rather than the checker: it points HOME
at an empty directory, which is exactly CI's condition, and asserts `task
check` exits 0 AND still says it could not verify -- silence would mean
skipped rather than tolerated.

Bump-Type: patch

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-07 20:14:19 +02:00
co-authored by Claude Opus 5
parent da76a6b3d2
commit 9fd5a054b3
3 changed files with 78 additions and 3 deletions
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "skills",
"description": "Mathias's engineering skills library, plus the hooks and MCP servers that make it enforceable: TDD, clean code, debugging, specs, discovery, CI, and secret hygiene.",
"version": "0.19.0",
"version": "0.19.1",
"author": {
"name": "Mathias",
"url": "https://git.d-ma.be/mathias"
+7 -2
View File
@@ -145,8 +145,13 @@ tasks:
# not run (no jq, or no env-file — CI has neither); that is reported
# and not fatal, but it is never treated as a pass.
if [ -x "{{.TASKFILE_DIR}}/bin/check-mcp-env" ]; then
"{{.TASKFILE_DIR}}/bin/check-mcp-env" "{{.TASKFILE_DIR}}/.mcp.json" "$HOME/.op-env"
case $? in
# `|| rc=$?` is load-bearing, not style. Task runs this block under
# `sh -e`, so a bare invocation exiting non-zero kills the shell
# before `case` is reached -- the tolerance below never ran, and the
# gate died with `exit status 2` on the commit that added it.
rc=0
"{{.TASKFILE_DIR}}/bin/check-mcp-env" "{{.TASKFILE_DIR}}/.mcp.json" "$HOME/.op-env" || rc=$?
case $rc in
0) ;;
2) echo "NOTE: MCP env wiring unverified on this host" >&2 ;;
*) fail=1 ;;
@@ -0,0 +1,70 @@
#!/bin/sh
# `task check` must survive a check that reports "could not verify".
#
# The regression these exist for: bin/check-mcp-env exits 2 to mean "I could
# not run — no jq, or no env-file", which the gate deliberately treats as a
# reported NOTE rather than a failure. The Taskfile said exactly that:
#
# "$DIR/check-mcp-env" "$DIR/.mcp.json" "$HOME/.op-env"
# case $? in 0) ;; 2) echo "NOTE: ..." ;; *) fail=1 ;; esac
#
# and it did not work. Task runs each cmds block under `sh -e`, so a bare
# command exiting non-zero aborts the shell *before* `case $?` is ever
# reached. The whole gate died with `exit status 2`. Locally the check found
# jq and exited 0, so it went green; CI has neither jq nor ~/.op-env, so main
# went red on the very commit that added it (skills run #57, 2026-09-07).
#
# A tolerated exit code that is only tolerated on the host where it never
# fires is not tolerated at all. This pins the tolerance itself.
#
# No mocking: pointing HOME at an empty directory removes ~/.op-env, which is
# exactly the condition CI runs under, so check-mcp-env genuinely exits 2.
set -u
ROOT=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
pass=0
fail=0
ok() { pass=$((pass + 1)); printf ' ok %s\n' "$1"; }
no() { fail=$((fail + 1)); printf ' FAIL %s\n' "$1"; }
if ! command -v task >/dev/null 2>&1; then
printf ' SKIP: task not installed -- cannot exercise the gate\n'
exit 0
fi
TMP=$(mktemp -d)
trap 'rm -rf "$TMP"' EXIT
# Sanity: the precondition must actually hold, or the test proves nothing.
# An empty HOME means no ~/.op-env, so check-mcp-env cannot verify anything.
"$ROOT/bin/check-mcp-env" "$ROOT/.mcp.json" "$TMP/.op-env" >/dev/null 2>&1
rc=$?
if [ "$rc" -eq 2 ]; then
ok 'precondition: check-mcp-env exits 2 when the env-file is absent'
else
no "precondition: expected check-mcp-env to exit 2, got $rc"
fi
# The assertion. `task check` must still pass, and must say why it could not
# verify -- silence would mean the check was skipped rather than tolerated.
out=$(HOME="$TMP" task --dir "$ROOT" check 2>&1)
rc=$?
if [ "$rc" -eq 0 ]; then
ok 'task check passes when MCP env wiring cannot be verified'
else
no "task check exited $rc when check-mcp-env reported unverifiable"
printf '%s\n' "$out" | sed 's/^/ | /' | tail -8
fi
if printf '%s' "$out" | grep -q 'unverified'; then
ok 'the unverifiable check is reported, not silently skipped'
else
no 'no NOTE reported -- an unverifiable check must never look like a pass'
fi
printf ' %d passed, %d failed\n' "$pass" "$fail"
[ "$fail" -eq 0 ]