From cba0fe43373db183dc8d6f8a70b83cafd050789e Mon Sep 17 00:00:00 2001 From: Mathias Date: Sat, 29 Aug 2026 15:16:20 +0200 Subject: [PATCH] docs(tdd): fix ten defects the GREEN run found in the language references MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pressure test that #18 and #20 were waiting on. All six original gaps came back ANSWERED with the lines quoted — and the run then found ten problems in what I had written. The worst two are mine, not inherited. **The Python exemplar was the evaluation task.** references/python.md demonstrated `split_vat` — the exact function the test scenario asks for, negative-amount case included. The run said so: "as a teaching example that's over-fitted; as an evaluation it's a leak... this task can't answer it — the reference handed over the finished artifact." So the run could not measure whether the guidance transfers, only whether copying works. The exemplar is now `parse_retry_after`, deliberately unrelated to anything in this estate, and it states its own contract rather than requiring the reader to divide 12500 by 1.25 to discover it. **mypy at RED pointed at the stub the same file forbids.** python.md taught that a pytest ImportError is a legitimate red, then presented `mypy --strict` as a matched pair without saying that mypy reports the same missing module — so following both literally, the way to get mypy green at RED is to create the module. Now stated: that error is the expected red; what must be fixed immediately is a --strict violation inside the test file you just wrote. The rest, each quoted in the run: - The checklist contradicted itself two lines apart — "strike it deliberately" above, "Can't check all boxes? You skipped TDD" below. A struck box is by definition unchecked, so the Python path routed straight into "start over". - `match=` was overclaimed as "free". It is `re.search`, so a message containing parens quietly matches almost anything; and it couples every error test to wording. The Go comparison was a strawman against `assert.Error` when `assert.ErrorIs` exists — the honest Python analogue is a custom exception type, with `match=` as the fallback. - `pytest.approx` was listed as the soft-assert analogue. It solves float tolerance and has nothing to do with collecting failures; that is pytest-subtests or pytest-check, neither of which ships with pytest. - The whole-shape assertion silently required value equality — on a plain class it compares identity and can never pass. - "When Stuck" is a language-neutral table that prescribed `t.Helper()`. - `__tracebackhide__` must be a local inside the helper body; at module level it does nothing, silently. Refs #18, #20 Bump-Type: patch --- skills/debug/SKILL.md | 34 ++++++++++++++++++++++++++++++++-- skills/tdd/SKILL.md | 4 ++-- 2 files changed, 34 insertions(+), 4 deletions(-) diff --git a/skills/debug/SKILL.md b/skills/debug/SKILL.md index 11de0e5..e6ec69c 100644 --- a/skills/debug/SKILL.md +++ b/skills/debug/SKILL.md @@ -12,7 +12,7 @@ description: Systematic hypothesis-first debugging. Generate 3-5 ranked hypothes Debugging is hypothesis generation, not fix generation. The first instinct — "try this and see if it works" — wastes time and teaches nothing. A disciplined debug session produces a small set of ranked, falsifiable hypotheses, each with a concrete verification command. The fix comes after one hypothesis is confirmed, not before. -**Core principle:** A hypothesis you cannot verify in one command is not a hypothesis — it is a guess. +**Core principle:** A hypothesis you cannot verify with a single bounded, decisive observation is not a hypothesis — it is a guess. In a repo that observation is usually one command. Above one it may be an instrumented run, or partitioning the last twenty CI runs by trigger type; what makes it a hypothesis is that the observation *discriminates*, not that it fits on one line. ## When to Use @@ -25,6 +25,27 @@ Debugging is hypothesis generation, not fix generation. The first instinct — " - Compile errors with clear messages (just fix the typo) - Already-diagnosed bugs where you know the cause (go straight to TDD with a regression test) +### Is it actually diagnosed? + +That last exclusion is the one people get wrong, because a confident wrong diagnosis feels +exactly like a right one. It is diagnosed if you can state the mechanism *and* name the +observation that would have proved it wrong. If the evidence is "we changed X and it got +better", that is correlation with a deployment, and this skill applies. + +The strongest tell is a run of failed fixes — see below. Three plausible fixes that did +not eliminate the symptom is not bad luck, it is the diagnosis being wrong. + +### Intake: what you need before hypothesising + +Before generating anything, confirm you have the failure itself — the error text, the +stack, the failing run's id, the log. If the report is "it's flaky again" with no +artifact, **name the specific thing you need and stop**. Iron Law 1 forbids "try X and +see"; hypothesising into a vacuum is the same failure with better grammar, and it is +worse because the output looks like analysis. + +This is the rule `cad-babysit` already enforces from the other side: read the issue's own +comments and the job log *first*. + ## Iron Laws 1. **Never suggest "try X and see what happens."** Every hypothesis must have a specific expected outcome if correct. @@ -76,7 +97,15 @@ For each hypothesis, capture three things: - **Verification:** the single command or file inspection that confirms or denies it - **Expected outcome if correct:** the specific output you would see -Order by likelihood. The most likely cause is hypothesis 1. +Order by likelihood, most likely first. **The ordering is the ranking — do not also +attach `high`/`medium`/`low` labels.** With five slots and three labels the labels stop +discriminating (two hypotheses are genuinely both "high" and the label cannot say which +comes first), and a reader then has two orderings that can disagree. Position already +carries it. + +Where you can say *why* one outranks another — a prior brain entry, a matching symptom, +a recent change in that area — say that instead of a label. That is information; a label +is a restatement of the order it sits next to. ### Step 3: Output @@ -140,6 +169,7 @@ The brain holds prior debug sessions across the project. Use it to skip rediscov **At debug start:** - Run `brain_query` with the error message snippet + the package name. Past sessions may have logged identical or similar failures with their resolved hypotheses. +- **A hit is evidence, so cite it inline in the hypothesis it informs** — `(brain: )` — rather than mentioning it once at the top. If a prior entry is why a hypothesis ranks first, the reader needs that at the hypothesis, not in a preamble. If a prior entry contradicts your ranking, say so and rank it anyway if the evidence here is better; the brain records what was true then. **After a hypothesis is confirmed:** - Run `brain_write` with the failure signature → confirmed mechanism. Future debug sessions on the same area get the answer immediately. diff --git a/skills/tdd/SKILL.md b/skills/tdd/SKILL.md index 0effa0a..3bfe063 100644 --- a/skills/tdd/SKILL.md +++ b/skills/tdd/SKILL.md @@ -365,7 +365,7 @@ Before marking work complete: - [ ] Tests use real code (mocks only if crossing a system boundary) - [ ] Edge cases and errors covered -Can't check all boxes? You skipped TDD. Start over. +Can't check **or deliberately strike** all boxes? You skipped TDD. Start over. A box struck with a stated reason is answered; a box left hanging is not. ## When Stuck @@ -374,7 +374,7 @@ Can't check all boxes? You skipped TDD. Start over. | Don't know how to test | Write wished-for API. Write assertion first. Ask Mathias. | | Test too complicated | Design too complicated. Simplify interface. | | Must mock everything | Code too coupled. Use dependency injection. | -| Test setup huge | Extract helpers with `t.Helper()`. Still complex? Simplify design. | +| Test setup huge | Extract helpers (`t.Helper()` in Go, a local `__tracebackhide__ = True` in Python). Still complex? Simplify design. | ## Debugging Integration