fix(check): read the session_log payload, not the file

A traceability probe falsified this gate four hours after I shipped it. The
check was `grep -q 'session_id' "$dir/SKILL.md"` — file-scoped. So the probe
stripped session_id from all three documented payloads in tdd/SKILL.md, added
one prose line saying "the tool has a required session_id field", and the gate
returned 0. The exact regression it exists to prevent was back in the file, and
the gate said nothing.

The duration_ms rule was no better: it matched one spelling, the historical one.
Both of these passed — `"duration_ms": <wall-clock>` with a JSON-quoted key, and
`duration_ms: elapsed wall-clock ms` with no angle brackets. It was a regression
test against the nine strings that existed that morning, described in the commit
message as a rule.

Now each documented payload is checked on its own line and reported with a line
number, and duration_ms is rejected in any spelling unless it carries an actual
number — the objection was always to a placeholder no agent can fill, not to the
field.

Five new cases, each one a probe from that run: prose mentioning the word while
the payload omits it, one bad payload among several, both duration_ms spellings,
and a real measurement being allowed.

Also fixed while writing them: my first regex had `[[:space:]]*` before a
negated class that did not exclude whitespace, so the star matched empty and the
space itself tripped the check — a real measurement failed. The tests caught it
immediately, which is the argument for writing them from the probes rather than
from the intent.

Refs #36, #39
Bump-Type: patch
This commit is contained in:
2026-08-29 15:34:11 +02:00
parent 88a4fa407b
commit 4d6ebb626f
2 changed files with 62 additions and 5 deletions
+29 -5
View File
@@ -25,15 +25,39 @@ for dir in "$ROOT"/skills/*/; do
grep -q 'session_log' "$dir/SKILL.md" || continue
if ! grep -q 'session_id' "$dir/SKILL.md"; then
echo "ERROR: skills/$name documents session_log without session_id — the tool's only required field" >&2
fail=1
# Read the PAYLOAD, not the file. The first version of this gate grepped the
# whole SKILL.md for "session_id", so a payload missing the field passed as
# long as the word appeared anywhere — including in prose explaining that the
# field is required. A traceability probe demonstrated it in one sed.
payloads=$(grep -n 'session_log' "$dir/SKILL.md" | grep '{' || true)
if [ -z "$payloads" ]; then
# Mentions the tool but documents no call. Nothing to check, and nothing to
# get wrong.
continue
fi
if grep -q 'duration_ms:[[:space:]]*<' "$dir/SKILL.md"; then
echo "ERROR: skills/$name documents duration_ms as a placeholder an agent cannot measure — omit it or pass a real measurement" >&2
printf '%s\n' "$payloads" | while IFS= read -r line; do
[ -n "$line" ] || continue
body=$(printf '%s' "$line" | sed 's/^[0-9]*://')
lineno=$(printf '%s' "$line" | cut -d: -f1)
printf '%s' "$body" | grep -q 'session_id' \
|| echo "ERROR: skills/$name:$lineno documents a session_log payload with no session_id — the tool's only required field" >&2
# Any spelling of the key, quoted or not, followed by anything that is not a
# bare number. A placeholder nobody can fill gets fabricated or dropped; a
# real measurement is fine.
if printf '%s' "$body" | grep -Eq '"?duration_ms"?[[:space:]]*:[[:space:]]*[^0-9,}[:space:]]'; then
echo "ERROR: skills/$name:$lineno documents duration_ms as something other than a real measurement — an agent does not observe a clock; omit it" >&2
fi
done > "$ROOT/.session_log_errors.$$" 2>&1
if [ -s "$ROOT/.session_log_errors.$$" ]; then
cat "$ROOT/.session_log_errors.$$" >&2
fail=1
fi
rm -f "$ROOT/.session_log_errors.$$"
done
exit $fail