mathias
d6e892ac4c
Forward trace: requirement → decision → commit → gate → verification → CI on that sha → release → closure. The distinction the skill exists for is that each link is either checked or named as unchecked; a trace where every link is asserted is a narrative, and narratives are what audits exist to defeat. `trace.sh` does the mechanical part — commit, files, test files touched, release containing it, referenced issues as URLs — and then always prints the four things git cannot prove. The GREEN run's own accounting: "the tool did the mechanical 15%. Its real contribution is the 'Cannot verify from here' block, which is a checklist, not an output." Kept, because it is true. The RED run established the shape by tracing a commit by hand in sixteen steps. Its decisive moves were reading issue COMMENTS — where the decisions actually live — and running a probe that falsified a claim. Both are now steps. The GREEN run then used step 5 against a gate I shipped this morning, and broke it. It also found four things wrong with what I had written: - The slug sed used `[^/]+?` expecting a lazy quantifier. POSIX ERE has none, so `.git` was swallowed and every emitted evidence URL carried the suffix. It happened to work against Gitea and would 404 on GitHub — a bad failure for the one tool whose pitch is that an unchecked link must not look checked. - "Prints every issue it references" was overclaimed: it greps the commit body, so it structurally cannot find the issue that cites the COMMIT. That is the commoner break — the issue filed afterwards that falsifies the outcome — and it is now step 6 with the two searches that find it. - Step 3 told you to check which job passed, and the estate's own `workflow_run_status` returns no jobs. A step whose tooling dead-ends is where a trace becomes "the run was green"; the REST fallback is now written out. - The output format did not survive contact. It grew four slots, each forced by a real trace: two requirements differing in kind (a defect report and a standing bar), the gate as an artifact separate from the test, the RED/GREEN baseline against the parent, and the commit's factual claims checked. Without the last one, "commit → verified" means only "this sha exists". Also: `git describe --contains` reads local tags, so the tool now says so on every hit rather than only when it finds nothing. Closes #10 Bump-Type: minor
8.5 KiB
name, surfaces, summary, use_when, description
| name | surfaces | summary | use_when | description | ||
|---|---|---|---|---|---|---|
| traceability |
|
Forward trace — requirement to decision to commit to verification to release, and what cannot be checked | Proving why a change exists; auditing a claim; before an attestation or a compliance ask | Establish the forward trace for a change — requirement, decision, commit, verification, CI, release, closure — as a chain of links that are each checked rather than asserted. Names the links that cannot be verified from the repository instead of letting them pass as verified. Use when auditing why a change exists, preparing an attestation, or when a claim needs to be provable rather than plausible. |
Traceability
Overview
A trace is not a story about how a change came to be. It is a chain of links, each of which is either checked or named as unchecked. The value is entirely in that distinction: a trace where every link is asserted is a narrative, and narratives are what audits exist to defeat.
The chain in this estate:
issue (requirement) → comment (decision) → commit → test/gate → CI run on that sha
→ release tag → closing comment (evidence)
Note where the decisions live: in issue comments, not issue bodies. A tool that reads issue titles will trace the wrong thing.
When to Use
- Auditing why a change exists, especially one nobody remembers making
- Preparing an attestation, a compliance answer, or a client-facing "how do you know"
- Before claiming a requirement is satisfied
- When a claim is repeated in several places, which is the condition under which nobody checks it
Do not use for: a change you are about to make. That is adr (the decision) and
spec-driven-dev (the requirement). This skill reconstructs; it does not author.
The tool
skills/traceability/trace.sh <sha> [repo-path]
Prints what git can prove — the commit, its files, the test files it touched, the release that contains it, every issue it references as a URL — and then prints, every time, the three things git cannot prove. That last section is the point of the tool.
The sequence
1. Run the tool. It does the mechanical half in one command.
2. Open the issues it printed, and read the comments. The requirement is in the body; the decision is almost always in a comment, and so is the closure record. A trace built from commit messages alone will confidently cite an issue that says something else.
3. Match CI on the head sha, not on the branch. "The pipeline is green" is not
evidence about a commit. Find the run whose head_sha is this commit and read its
conclusion — and check which job and which step passed, because a run can be red for a
reason that has nothing to do with the gate you care about, and green overall while the
gate you care about was skipped.
The estate's workflow_run_status returns only the run-level conclusion — no jobs, no
steps. For that breakdown you have to drop to the API:
op run --env-file ~/.op-env -- sh -c 'curl -sS -H "Authorization: token $DMABE_GITEA_API_TOKEN" \
"https://git.d-ma.be/api/v1/repos/<owner>/<repo>/actions/runs/<id>/jobs"'
Said explicitly because a step whose tooling dead-ends is exactly where a trace becomes "the run was green".
4. Check that the tests actually cover the behaviour. A commit that touched a test file is not a commit that is verified. Read the assertions and ask what would have to break for them to fail. If nothing would, the link is decorative.
5. Probe the load-bearing claim — try to falsify it. This is the step that separates a trace from a summary, and it is the one that finds things.
Pick the claim the chain rests on, and attack it in a scratch copy:
git archive <sha> | (mkdir -p /tmp/probe && tar -x -C /tmp/probe)
# then break exactly the thing the claim protects, and run the gate
If the gate stays green, the claim is false — regardless of how many places assert it. This is not hypothetical: the probe that shaped this skill falsified a "the list only shrinks" claim that was stated in five places in the repository and enforced in none.
6. Look for what cites the commit, not only what it cites. A commit cannot reference
an issue filed after it, and that is precisely where an outcome gets falsified. Search the
tracker for the sha, across issues and comments, and git log --all --grep=<sha> for
later commits referring back. In the run that shaped this skill, the issue proving the
traced commit had relocated rather than fixed its defect was filed 90 minutes later and
was reachable only this way.
7. Report unverifiable links as unverifiable. Not as weak links, not as caveats at the end. Each gets its own line saying what would be needed to close it.
What a broken link looks like
| Symptom | What it means |
|---|---|
The commit cites #37, and #37 describes something else |
The reference is decoration; someone renumbered or misremembered |
| A claim appears in five places and a check in none | Convention wearing a mechanism's clothes. Probe it |
| The gate greps for a heading | Presence is not compliance. A heading with nothing under it passes |
git describe says unreleased |
Usually stale local tags, not an unreleased commit. git fetch --tags first |
The requirement is a bare #5 with no URL |
Recoverable only while the tracker lives. Not traceable from the repo alone |
| CI green, but on a different sha | The most common false link, and the easiest to accept by accident |
Output
State the chain, then the gaps, in that order:
CHAIN
requirement #36 item 3 (defect report) → verified, read: body + comments
requirement #19 (standing execution bar) → verified, read
decision comment 8509 → verified; names the commit
commit b616c51 → verified
claim "nine skills, none named session_id" → verified against b616c51^: 9/9
claim "session_id is the only required field" → verified against the live tool schema
gate bin/check-session-log → RED on parent (17 errors), GREEN on it
wiring Taskfile check step 5 → verified; CI runs task check
verification tests/check_session_log_test.sh → verified, 4 cases, both directions
CI run 4880, job "Check / Test" → verified, head_sha matches, success
release v0.13.3 → verified against the REMOTE
GAPS
- The gate greps the file, not the payload. Probe: stripped session_id from
every payload, left the word in prose — gate returned 0. FALSIFIED.
- Defect relocated, not removed: #39, filed 90 minutes later by the same
author, says so. The commit's outcome claim does not hold.
Four slots that are not optional, each learned by leaving them out:
- More than one requirement, and they differ in kind. A defect report and a standing policy bar are both requirements and collapsing them hides which one was not met.
- The gate, separately from the test. A commit that ships a check has three artifacts, not two, and the chain usually rests on the one in the middle.
- The RED/GREEN baseline. Run the gate against the parent commit. Without it you have shown a gate exists, not that it caught anything.
- The commit's factual claims, checked. "Nine skills, none named it" is falsifiable in one command. Without a slot for it, "commit → verified" means only "this sha exists".
Gaps ranked by how much they matter, not by where you found them. A trace whose gap section is empty should be suspicious of itself.
Escalation
Read-only. Reports the chain; changes nothing, and closes no issue on the strength of what it finds.
Stop and hand back when: a link's evidence is outside anything you can reach — a tracker you cannot read, a CI system you have no access to, a decision made in conversation. Say which link and what would close it. An inferred link presented as a traced one is worse than a gap, because the gap is honest and the inference is citable.
Also stop before acting on a falsified claim. Finding that a gate does not enforce what it says is a finding; deciding what to do about it is the owner's call.
Cross-References
lineage— the same discipline backwards: this asks "does the chain hold", that asks "was it tried before and why did it stop".adr— the decision layer of the chain. A trace that reaches an ADR has found the reasoning; a trace that reaches only a commit message has found a summary of it.