mathiasandClaude Opus 5
9fd5a054b3
bin/check-mcp-env exits 2 for "could not run" -- no jq, or no env-file --
and the gate was written to report that as a NOTE rather than fail:
"$DIR/check-mcp-env" "$DIR/.mcp.json" "$HOME/.op-env"
case $? in 0) ;; 2) echo "NOTE: ..." ;; *) fail=1 ;; esac
That tolerance never executed. Task runs each cmds block under `sh -e`, so
a bare command exiting non-zero aborts the shell before `case $?` is
reached. main went red on the very commit that added the check (run #57),
with `task: Failed to run task "check": exit status 2`.
It passed locally because this host has both jq and ~/.op-env, so the
check exited 0 and the untested branch never ran. CI has neither. A
tolerated exit code that is only tolerated where it never fires is not
tolerated at all.
`|| rc=$?` puts the invocation in a compound command, which `set -e`
exempts, so `case` is reached. Both exit-2 paths verified: no env-file
(the new test's precondition) and no jq (PATH sandbox, same SKIP message
CI printed).
The test pins the tolerance itself rather than the checker: it points HOME
at an empty directory, which is exactly CI's condition, and asserts `task
check` exits 0 AND still says it could not verify -- silence would mean
skipped rather than tolerated.
Bump-Type: patch
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
71 lines
2.5 KiB
Bash
71 lines
2.5 KiB
Bash
#!/bin/sh
|
|
# `task check` must survive a check that reports "could not verify".
|
|
#
|
|
# The regression these exist for: bin/check-mcp-env exits 2 to mean "I could
|
|
# not run — no jq, or no env-file", which the gate deliberately treats as a
|
|
# reported NOTE rather than a failure. The Taskfile said exactly that:
|
|
#
|
|
# "$DIR/check-mcp-env" "$DIR/.mcp.json" "$HOME/.op-env"
|
|
# case $? in 0) ;; 2) echo "NOTE: ..." ;; *) fail=1 ;; esac
|
|
#
|
|
# and it did not work. Task runs each cmds block under `sh -e`, so a bare
|
|
# command exiting non-zero aborts the shell *before* `case $?` is ever
|
|
# reached. The whole gate died with `exit status 2`. Locally the check found
|
|
# jq and exited 0, so it went green; CI has neither jq nor ~/.op-env, so main
|
|
# went red on the very commit that added it (skills run #57, 2026-09-07).
|
|
#
|
|
# A tolerated exit code that is only tolerated on the host where it never
|
|
# fires is not tolerated at all. This pins the tolerance itself.
|
|
#
|
|
# No mocking: pointing HOME at an empty directory removes ~/.op-env, which is
|
|
# exactly the condition CI runs under, so check-mcp-env genuinely exits 2.
|
|
|
|
set -u
|
|
|
|
ROOT=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
|
|
|
|
pass=0
|
|
fail=0
|
|
|
|
ok() { pass=$((pass + 1)); printf ' ok %s\n' "$1"; }
|
|
no() { fail=$((fail + 1)); printf ' FAIL %s\n' "$1"; }
|
|
|
|
if ! command -v task >/dev/null 2>&1; then
|
|
printf ' SKIP: task not installed -- cannot exercise the gate\n'
|
|
exit 0
|
|
fi
|
|
|
|
TMP=$(mktemp -d)
|
|
trap 'rm -rf "$TMP"' EXIT
|
|
|
|
# Sanity: the precondition must actually hold, or the test proves nothing.
|
|
# An empty HOME means no ~/.op-env, so check-mcp-env cannot verify anything.
|
|
"$ROOT/bin/check-mcp-env" "$ROOT/.mcp.json" "$TMP/.op-env" >/dev/null 2>&1
|
|
rc=$?
|
|
if [ "$rc" -eq 2 ]; then
|
|
ok 'precondition: check-mcp-env exits 2 when the env-file is absent'
|
|
else
|
|
no "precondition: expected check-mcp-env to exit 2, got $rc"
|
|
fi
|
|
|
|
# The assertion. `task check` must still pass, and must say why it could not
|
|
# verify -- silence would mean the check was skipped rather than tolerated.
|
|
out=$(HOME="$TMP" task --dir "$ROOT" check 2>&1)
|
|
rc=$?
|
|
|
|
if [ "$rc" -eq 0 ]; then
|
|
ok 'task check passes when MCP env wiring cannot be verified'
|
|
else
|
|
no "task check exited $rc when check-mcp-env reported unverifiable"
|
|
printf '%s\n' "$out" | sed 's/^/ | /' | tail -8
|
|
fi
|
|
|
|
if printf '%s' "$out" | grep -q 'unverified'; then
|
|
ok 'the unverifiable check is reported, not silently skipped'
|
|
else
|
|
no 'no NOTE reported -- an unverifiable check must never look like a pass'
|
|
fi
|
|
|
|
printf ' %d passed, %d failed\n' "$pass" "$fail"
|
|
[ "$fail" -eq 0 ]
|