session_log asks for a session_id no agent can obtain — two runs declined to log rather than invent one #39

Open
opened 2026-08-29 13:18:07 +00:00 by mathias · 0 comments
Owner

Introduced by me in b616c51 while fixing #36 item 3, and caught by two separate pressure-test runs within the hour.

The problem

Nine skills now document a session_log call with {session_id: "<this session's id>", ...}. session_id is the tool's only required field, so naming it was the fix. But nothing tells an agent how to obtain one, and an agent has no obvious way to read its own session identifier.

Both runs handled it the same way, and their reasoning is the argument for fixing it:

I did not call session_log for the red phase: I don't have this session's id, and final_status would be a fabrication since the red was never observed.

I logged nothing rather than fabricate a session id and poison that metric — flagging it as a deliberate omission, not an oversight.

Two well-behaved runs both declined to log. A less careful one invents an id, and invented ids do not group — which is the exact defect #36 item 3 set out to remove, now relocated rather than removed. The earlier debug run had already demonstrated the failure mode, inventing debug-signed-artifact-missing-2026-08-29 and noting its entries were "only groupable by luck of naming".

Where the id could come from

Not established — this is the work. Candidates:

  1. The harness already knows it. Claude Code hook payloads carry session_id; if the plugin can surface it to the model (an env var, a ${CLAUDE_SESSION_ID} expansion, a line injected by the SessionStart hook), the skills just say "use that".
  2. The MCP server derives it. session_log could accept an omitted session_id and fall back to a server-side notion of the caller. That moves the problem to a layer that can actually answer it, and makes every existing skill correct as written.
  3. A documented deterministic convention — <project>-<yyyy-mm-dd>-<task-slug>, held identical across every call in one task. Groups the task rather than the session, which is arguably what a pass-rate wants anyway. Weakest option: two agents on the same task the same day collide, and nothing enforces the shape.

I lean 2, then 1. Option 3 is what an agent will improvise if we say nothing, so leaving this open is choosing it by default.

Related, from the same run

debug's phases assume the skill ran. A correct intake stop — refusing to hypothesise because the report has no artifact — reached no phase, so it would have logged as a failed read-failure. Fixed in 0c7fac3 by adding an intake phase, but the general shape is worth checking across the other eight: a metric that records the skill's best behaviour as a failure trains the next model out of it.

Definition of done

An agent following any of the nine skills can produce a real session_id without inventing one, or the field is no longer required of it. bin/check-session-log stays as the gate either way.

Introduced by me in `b616c51` while fixing #36 item 3, and caught by two separate pressure-test runs within the hour. ## The problem Nine skills now document a `session_log` call with `{session_id: "<this session's id>", ...}`. `session_id` is the tool's only required field, so naming it was the fix. But **nothing tells an agent how to obtain one**, and an agent has no obvious way to read its own session identifier. Both runs handled it the same way, and their reasoning is the argument for fixing it: > I did not call `session_log` for the red phase: I don't have this session's id, and `final_status` would be a fabrication since the red was never observed. > I logged nothing rather than fabricate a session id and poison that metric — flagging it as a deliberate omission, not an oversight. Two well-behaved runs both declined to log. A less careful one invents an id, and invented ids do not group — which is the exact defect #36 item 3 set out to remove, now relocated rather than removed. The earlier `debug` run had already demonstrated the failure mode, inventing `debug-signed-artifact-missing-2026-08-29` and noting its entries were *"only groupable by luck of naming"*. ## Where the id could come from Not established — this is the work. Candidates: 1. **The harness already knows it.** Claude Code hook payloads carry `session_id`; if the plugin can surface it to the model (an env var, a `${CLAUDE_SESSION_ID}` expansion, a line injected by the SessionStart hook), the skills just say "use that". 2. **The MCP server derives it.** `session_log` could accept an omitted `session_id` and fall back to a server-side notion of the caller. That moves the problem to a layer that can actually answer it, and makes every existing skill correct as written. 3. **A documented deterministic convention** — `<project>-<yyyy-mm-dd>-<task-slug>`, held identical across every call in one task. Groups the *task* rather than the session, which is arguably what a pass-rate wants anyway. Weakest option: two agents on the same task the same day collide, and nothing enforces the shape. I lean 2, then 1. Option 3 is what an agent will improvise if we say nothing, so leaving this open is choosing it by default. ## Related, from the same run `debug`'s phases assume the skill ran. A correct **intake stop** — refusing to hypothesise because the report has no artifact — reached no phase, so it would have logged as a failed `read-failure`. Fixed in `0c7fac3` by adding an `intake` phase, but the general shape is worth checking across the other eight: a metric that records the skill's best behaviour as a failure trains the next model out of it. ## Definition of done An agent following any of the nine skills can produce a real `session_id` without inventing one, or the field is no longer required of it. `bin/check-session-log` stays as the gate either way.
mathias added the risk/mediumsize/mtype/fix labels 2026-09-16 22:22:05 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/skills#39