mathias
b616c51a2c
Nine skills documented the payload. None named `session_id`, which is the tool's only required field. So every documented call was one the tool rejects — an agent following any of them either failed the call or invented an id, and invented ids do not group, which means the pass-rate the /pass-rate endpoint computes was being assembled from entries that cannot be joined. Found by the #19 pressure test against `debug`, which invented `debug-signed-artifact-missing-2026-08-29` and said so: "my three entries are only groupable by luck of naming". The grep afterwards showed it was fleet-wide, not a `debug` bug. Also removed: `duration_ms: <wall-clock>` in eight of them. An agent does not observe a clock. The same run omitted it "rather than fabricate", which is the good outcome; the bad outcome is the run that fabricates, and a made-up number in a metrics stream is worse than a missing one. It is an optional field and is now simply absent. bin/check-session-log gates both, with tests. This is exactly the case #19 has in mind for a small tool over prose: the rule is a grep, so it should be a gate, not a note asking people to remember. Closes part of #36 Bump-Type: patch
40 lines
1.4 KiB
Bash
Executable File
40 lines
1.4 KiB
Bash
Executable File
#!/bin/sh
|
|
# check-session-log — a skill that documents a session_log call must document a
|
|
# call the tool would actually accept.
|
|
#
|
|
# Nine skills documented the payload and none named `session_id`, which is the
|
|
# tool's only REQUIRED field. An agent following any of them either failed the
|
|
# call or invented an id — and invented ids do not group, so the pass-rate the
|
|
# endpoint computes was being assembled from entries that could not be joined
|
|
# (#36, found by a pressure test against `debug`).
|
|
#
|
|
# Also rejected: `duration_ms: <wall-clock>`. An agent does not observe a clock.
|
|
# A documented field nobody can honestly populate gets fabricated or silently
|
|
# dropped, and fabricated numbers in a metrics stream are worse than no numbers.
|
|
#
|
|
# Usage: check-session-log [checkout-root]
|
|
|
|
set -u
|
|
|
|
ROOT=${1:-$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)}
|
|
fail=0
|
|
|
|
for dir in "$ROOT"/skills/*/; do
|
|
[ -f "$dir/SKILL.md" ] || continue
|
|
name=$(basename "$dir")
|
|
|
|
grep -q 'session_log' "$dir/SKILL.md" || continue
|
|
|
|
if ! grep -q 'session_id' "$dir/SKILL.md"; then
|
|
echo "ERROR: skills/$name documents session_log without session_id — the tool's only required field" >&2
|
|
fail=1
|
|
fi
|
|
|
|
if grep -q 'duration_ms:[[:space:]]*<' "$dir/SKILL.md"; then
|
|
echo "ERROR: skills/$name documents duration_ms as a placeholder an agent cannot measure — omit it or pass a real measurement" >&2
|
|
fail=1
|
|
fi
|
|
done
|
|
|
|
exit $fail
|