mathias
d6e892ac4c
Forward trace: requirement → decision → commit → gate → verification → CI on that sha → release → closure. The distinction the skill exists for is that each link is either checked or named as unchecked; a trace where every link is asserted is a narrative, and narratives are what audits exist to defeat. `trace.sh` does the mechanical part — commit, files, test files touched, release containing it, referenced issues as URLs — and then always prints the four things git cannot prove. The GREEN run's own accounting: "the tool did the mechanical 15%. Its real contribution is the 'Cannot verify from here' block, which is a checklist, not an output." Kept, because it is true. The RED run established the shape by tracing a commit by hand in sixteen steps. Its decisive moves were reading issue COMMENTS — where the decisions actually live — and running a probe that falsified a claim. Both are now steps. The GREEN run then used step 5 against a gate I shipped this morning, and broke it. It also found four things wrong with what I had written: - The slug sed used `[^/]+?` expecting a lazy quantifier. POSIX ERE has none, so `.git` was swallowed and every emitted evidence URL carried the suffix. It happened to work against Gitea and would 404 on GitHub — a bad failure for the one tool whose pitch is that an unchecked link must not look checked. - "Prints every issue it references" was overclaimed: it greps the commit body, so it structurally cannot find the issue that cites the COMMIT. That is the commoner break — the issue filed afterwards that falsifies the outcome — and it is now step 6 with the two searches that find it. - Step 3 told you to check which job passed, and the estate's own `workflow_run_status` returns no jobs. A step whose tooling dead-ends is where a trace becomes "the run was green"; the REST fallback is now written out. - The output format did not survive contact. It grew four slots, each forced by a real trace: two requirements differing in kind (a defect report and a standing bar), the gate as an artifact separate from the test, the RED/GREEN baseline against the parent, and the commit's factual claims checked. Without the last one, "commit → verified" means only "this sha exists". Also: `git describe --contains` reads local tags, so the tool now says so on every hit rather than only when it finds nothing. Closes #10 Bump-Type: minor
82 lines
2.6 KiB
Bash
82 lines
2.6 KiB
Bash
#!/bin/sh
|
|
# Tests for skills/traceability/trace.sh.
|
|
#
|
|
# The RED run for #10 established what a forward trace costs by hand: sixteen
|
|
# commands, of which the decisive ones were reading issue comments and running a
|
|
# probe against a claim. This tool does the mechanical half — what git can prove
|
|
# — and, more importantly, states what it CANNOT prove, so an unverifiable link
|
|
# never reads as a verified one.
|
|
|
|
set -u
|
|
|
|
ROOT=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
|
|
SUT="$ROOT/skills/traceability/trace.sh"
|
|
|
|
pass=0
|
|
fail=0
|
|
WORK=${TMPDIR:-/tmp}/traceability_test.$$
|
|
|
|
if [ ! -f "$SUT" ]; then
|
|
printf 'FAIL: %s does not exist\n' "$SUT"
|
|
exit 1
|
|
fi
|
|
|
|
cleanup() { rm -rf "$WORK"; }
|
|
trap cleanup EXIT INT TERM
|
|
|
|
mkdir -p "$WORK/repo/tests"
|
|
cd "$WORK/repo" || exit 1
|
|
git init -q .
|
|
git config user.email t@example.com
|
|
git config user.name test
|
|
# A remote is needed for the tool to turn "#37" into a checkable URL, which is
|
|
# the point of that section.
|
|
git remote add origin https://git.d-ma.be/mathias/demo.git
|
|
|
|
printf 'baseline\n' > README.md
|
|
git add -A && git commit -qm "chore: baseline"
|
|
|
|
printf 'gate\n' > gate.sh
|
|
printf 'case\n' > tests/gate_test.sh
|
|
git add -A && git commit -qm "feat(gate): add the thing
|
|
|
|
Implements the decision recorded in #5, filed as #37.
|
|
|
|
Closes #37"
|
|
SHA=$(git rev-parse --short HEAD)
|
|
git tag -a v1.2.0 -m "release v1.2.0"
|
|
|
|
printf 'more\n' >> gate.sh
|
|
git add -A && git commit -qm "chore: later work"
|
|
|
|
cd - >/dev/null || exit 1
|
|
out=$(sh "$SUT" "$SHA" "$WORK/repo" 2>&1)
|
|
|
|
check() {
|
|
if printf '%s' "$out" | grep -q "$1"; then
|
|
pass=$((pass + 1)); printf ' ok %s\n' "$2"
|
|
else
|
|
fail=$((fail + 1)); printf ' FAIL %s\n output had no %s\n%s\n' "$2" "$1" "$out"
|
|
fi
|
|
}
|
|
|
|
check '#5' 'lists every issue the commit references, not just the closing one'
|
|
check '#37' 'lists the closing reference'
|
|
check 'tests/gate_test.sh' 'names the test files the commit touched'
|
|
check 'v1.2.0' 'names the release that contains it'
|
|
check 'Cannot verify' 'states what it could not verify'
|
|
check 'issues/37' 'emits a checkable URL for the tracker link it cannot follow'
|
|
|
|
# A commit in no release must say so — "no tag" and "not checked" are different
|
|
# answers and only one of them is about the commit.
|
|
last=$(git -C "$WORK/repo" rev-parse --short HEAD)
|
|
out=$(sh "$SUT" "$last" "$WORK/repo" 2>&1)
|
|
if printf '%s' "$out" | grep -qi 'no release contains'; then
|
|
pass=$((pass + 1)); printf ' ok an unreleased commit is reported as unreleased\n'
|
|
else
|
|
fail=$((fail + 1)); printf ' FAIL an unreleased commit is reported as unreleased\n%s\n' "$out"
|
|
fi
|
|
|
|
printf '\ntraceability: %d passed, %d failed\n' "$pass" "$fail"
|
|
[ "$fail" -eq 0 ]
|