mathias
4d6ebb626f
A traceability probe falsified this gate four hours after I shipped it. The check was `grep -q 'session_id' "$dir/SKILL.md"` — file-scoped. So the probe stripped session_id from all three documented payloads in tdd/SKILL.md, added one prose line saying "the tool has a required session_id field", and the gate returned 0. The exact regression it exists to prevent was back in the file, and the gate said nothing. The duration_ms rule was no better: it matched one spelling, the historical one. Both of these passed — `"duration_ms": <wall-clock>` with a JSON-quoted key, and `duration_ms: elapsed wall-clock ms` with no angle brackets. It was a regression test against the nine strings that existed that morning, described in the commit message as a rule. Now each documented payload is checked on its own line and reported with a line number, and duration_ms is rejected in any spelling unless it carries an actual number — the objection was always to a placeholder no agent can fill, not to the field. Five new cases, each one a probe from that run: prose mentioning the word while the payload omits it, one bad payload among several, both duration_ms spellings, and a real measurement being allowed. Also fixed while writing them: my first regex had `[[:space:]]*` before a negated class that did not exclude whitespace, so the star matched empty and the space itself tripped the check — a real measurement failed. The tests caught it immediately, which is the argument for writing them from the probes rather than from the intent. Refs #36, #39 Bump-Type: patch
64 lines
2.5 KiB
Bash
Executable File
64 lines
2.5 KiB
Bash
Executable File
#!/bin/sh
|
|
# check-session-log — a skill that documents a session_log call must document a
|
|
# call the tool would actually accept.
|
|
#
|
|
# Nine skills documented the payload and none named `session_id`, which is the
|
|
# tool's only REQUIRED field. An agent following any of them either failed the
|
|
# call or invented an id — and invented ids do not group, so the pass-rate the
|
|
# endpoint computes was being assembled from entries that could not be joined
|
|
# (#36, found by a pressure test against `debug`).
|
|
#
|
|
# Also rejected: `duration_ms: <wall-clock>`. An agent does not observe a clock.
|
|
# A documented field nobody can honestly populate gets fabricated or silently
|
|
# dropped, and fabricated numbers in a metrics stream are worse than no numbers.
|
|
#
|
|
# Usage: check-session-log [checkout-root]
|
|
|
|
set -u
|
|
|
|
ROOT=${1:-$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)}
|
|
fail=0
|
|
|
|
for dir in "$ROOT"/skills/*/; do
|
|
[ -f "$dir/SKILL.md" ] || continue
|
|
name=$(basename "$dir")
|
|
|
|
grep -q 'session_log' "$dir/SKILL.md" || continue
|
|
|
|
# Read the PAYLOAD, not the file. The first version of this gate grepped the
|
|
# whole SKILL.md for "session_id", so a payload missing the field passed as
|
|
# long as the word appeared anywhere — including in prose explaining that the
|
|
# field is required. A traceability probe demonstrated it in one sed.
|
|
payloads=$(grep -n 'session_log' "$dir/SKILL.md" | grep '{' || true)
|
|
|
|
if [ -z "$payloads" ]; then
|
|
# Mentions the tool but documents no call. Nothing to check, and nothing to
|
|
# get wrong.
|
|
continue
|
|
fi
|
|
|
|
printf '%s\n' "$payloads" | while IFS= read -r line; do
|
|
[ -n "$line" ] || continue
|
|
body=$(printf '%s' "$line" | sed 's/^[0-9]*://')
|
|
lineno=$(printf '%s' "$line" | cut -d: -f1)
|
|
|
|
printf '%s' "$body" | grep -q 'session_id' \
|
|
|| echo "ERROR: skills/$name:$lineno documents a session_log payload with no session_id — the tool's only required field" >&2
|
|
|
|
# Any spelling of the key, quoted or not, followed by anything that is not a
|
|
# bare number. A placeholder nobody can fill gets fabricated or dropped; a
|
|
# real measurement is fine.
|
|
if printf '%s' "$body" | grep -Eq '"?duration_ms"?[[:space:]]*:[[:space:]]*[^0-9,}[:space:]]'; then
|
|
echo "ERROR: skills/$name:$lineno documents duration_ms as something other than a real measurement — an agent does not observe a clock; omit it" >&2
|
|
fi
|
|
done > "$ROOT/.session_log_errors.$$" 2>&1
|
|
|
|
if [ -s "$ROOT/.session_log_errors.$$" ]; then
|
|
cat "$ROOT/.session_log_errors.$$" >&2
|
|
fail=1
|
|
fi
|
|
rm -f "$ROOT/.session_log_errors.$$"
|
|
done
|
|
|
|
exit $fail
|