The pressure test that #18 and #20 were waiting on. All six original gaps came back ANSWERED with the lines quoted — and the run then found ten problems in what I had written. The worst two are mine, not inherited. **The Python exemplar was the evaluation task.** references/python.md demonstrated `split_vat` — the exact function the test scenario asks for, negative-amount case included. The run said so: "as a teaching example that's over-fitted; as an evaluation it's a leak... this task can't answer it — the reference handed over the finished artifact." So the run could not measure whether the guidance transfers, only whether copying works. The exemplar is now `parse_retry_after`, deliberately unrelated to anything in this estate, and it states its own contract rather than requiring the reader to divide 12500 by 1.25 to discover it. **mypy at RED pointed at the stub the same file forbids.** python.md taught that a pytest ImportError is a legitimate red, then presented `mypy --strict` as a matched pair without saying that mypy reports the same missing module — so following both literally, the way to get mypy green at RED is to create the module. Now stated: that error is the expected red; what must be fixed immediately is a --strict violation inside the test file you just wrote. The rest, each quoted in the run: - The checklist contradicted itself two lines apart — "strike it deliberately" above, "Can't check all boxes? You skipped TDD" below. A struck box is by definition unchecked, so the Python path routed straight into "start over". - `match=` was overclaimed as "free". It is `re.search`, so a message containing parens quietly matches almost anything; and it couples every error test to wording. The Go comparison was a strawman against `assert.Error` when `assert.ErrorIs` exists — the honest Python analogue is a custom exception type, with `match=` as the fallback. - `pytest.approx` was listed as the soft-assert analogue. It solves float tolerance and has nothing to do with collecting failures; that is pytest-subtests or pytest-check, neither of which ships with pytest. - The whole-shape assertion silently required value equality — on a plain class it compares identity and can never pass. - "When Stuck" is a language-neutral table that prescribed `t.Helper()`. - `__tracebackhide__` must be a local inside the helper body; at module level it does nothing, silently. Refs #18, #20 Bump-Type: patch
13 KiB
name, surfaces, summary, use_when, description
| name | surfaces | summary | use_when | description | ||
|---|---|---|---|---|---|---|
| tdd |
|
Test-Driven Development enforcement | Starting any new feature or fixing a bug | Write failing tests first, then minimal code to pass. Non-negotiable for all new features and bug fixes. |
Test-Driven Development (TDD)
Overview
Write the test first. Watch it fail. Write minimal code to pass.
Core principle: If you didn't watch the test fail, you don't know if it tests the right thing.
Violating the letter of the rules is violating the spirit of the rules.
When to Use
Always:
- New features
- Bug fixes
- Refactoring
- Behavior changes
Exceptions (ask Mathias):
- Throwaway prototypes
- Generated code (sqlc output, templ output)
- Configuration files
Thinking "skip TDD just this once"? Stop. That's rationalization.
The Iron Law
NO PRODUCTION CODE WITHOUT A FAILING TEST FIRST
Write code before the test? Delete it. Start over.
No exceptions:
- Don't keep it as "reference"
- Don't "adapt" it while writing tests
- Don't look at it
- Delete means delete
Implement fresh from tests. Period.
Red-Green-Refactor
RED → verify fails correctly → GREEN → verify all pass → REFACTOR → stay green → RED (next)
RED - Write Failing Test
Write one minimal test showing what should happen.
Go example (good):
func TestRetryOperation_RetriesThreeTimes(t *testing.T) {
attempts := 0
op := func() error {
attempts++
if attempts < 3 {
return errors.New("fail")
}
return nil
}
err := RetryOperation(op, 3)
require.NoError(t, err)
assert.Equal(t, 3, attempts)
}
Clear name, tests real behavior, one thing.
Go example (bad):
func TestRetry(t *testing.T) {
// Vague name, tests nothing meaningful
err := RetryOperation(nil, 0)
assert.NoError(t, err)
}
Requirements:
- One behavior per test
- Clear name describing the behavior
- Real code (no mocks unless crossing a system boundary)
Verify RED - Watch It Fail
MANDATORY. Never skip.
go test -run TestRetryOperation_RetriesThreeTimes ./...
Confirm:
- Test fails, and fails because the feature is missing
- The failure message is the one you expected
- It is not failing on a typo in the test itself
Test passes? You're testing existing behavior. Fix test.
Failing because the thing does not exist yet? That is a legitimate red — a Go
compile error for an undefined function, a Python ImportError for a module you have
not written, a TypeScript "cannot find module". Read the message, confirm it names the
unit you are about to build, and proceed to GREEN. Do not write a stub to make the
runner happier. A stub is production code written before a failing test, which is the
Iron Law with extra steps.
Failing on a typo or a broken import elsewhere? Different thing. Fix it and re-run until the failure is the one above.
GREEN - Minimal Code
Write simplest code to pass the test.
Good:
func RetryOperation(op func() error, maxRetries int) error {
for i := 0; i < maxRetries; i++ {
if err := op(); err == nil {
return nil
}
}
return op()
}
Just enough to pass.
Bad:
func RetryOperation(op func() error, maxRetries int, opts ...RetryOption) error {
// YAGNI — don't add options, backoff, jitter before the test demands it
}
Over-engineered.
Don't add features, refactor other code, or "improve" beyond what the test demands.
Verify GREEN - Watch It Pass
MANDATORY.
go test ./...
Confirm:
- Test passes
- All other tests still pass
- No race conditions:
go test -race ./...
Test fails? Fix code, not test.
Other tests fail? Fix now.
REFACTOR - Clean Up
After green only:
- Remove duplication
- Improve names
- Extract helpers
- Apply clean code principles (load
clean-codeskill)
Keep tests green. Don't add behavior.
Repeat
Next failing test for next behavior.
Other languages
The discipline above is language-neutral; the examples below it are Go, because Go is this fleet's default stack and one good example beats five mediocre ones.
For anything else, the per-language conventions live in reference files — load the one
you need rather than translating Go idioms by eye. Translation is where the mistakes
happen: the Go table-driven exemplar below branches on a wantErr column inside the
loop, which is idiomatic Go and an anti-pattern in pytest, where it produces a
conditional assertion and folds two behaviours into one test.
references/python.md— pytest,@pytest.mark.parametrize,mypy --strictin the RED gate,pytest.raises(..., match=), fixtures vs helpersreferences/typescript.md— vitest/jest,it.each,expect(...).rejectsreferences/rust.md—#[cfg(test)]modules, table-driven via arrays,#[should_panic]references/java.md— JUnit 5,@ParameterizedTest,assertThrows
Go-Specific TDD Notes
Table-Driven Tests (Preferred Pattern)
func TestValidateEmail(t *testing.T) {
tests := []struct {
name string
email string
wantErr bool
}{
{"valid email", "user@example.com", false},
{"empty email", "", true},
{"no at sign", "userexample.com", true},
{"no domain", "user@", true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateEmail(tt.email)
if tt.wantErr {
assert.Error(t, err)
} else {
assert.NoError(t, err)
}
})
}
}
Table-driven tests are the Go idiom. Use them for behavior that varies across inputs.
Subtests with t.Run
Use t.Run to name subtests clearly. Failure messages include the subtest name.
t.Run("rejects empty input", func(t *testing.T) { ... })
t.Run("accepts valid UUID", func(t *testing.T) { ... })
Test File Conventions
- File:
<package>_test.goin the same directory - Package:
package foo_testfor black-box testing (preferred),package foofor white-box - Helpers: use
t.Helper()so stack traces point to the caller, not the helper
func assertNoError(t *testing.T, err error) {
t.Helper()
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
}
Running Tests
go test ./... # all packages
go test -run TestFoo ./pkg/... # specific test
go test -run TestFoo/subtest ./... # specific subtest
go test -race ./... # race detector (always run before commit)
go test -cover ./... # coverage
go test -v ./... # verbose
testify
Pre-approved. Use assert (continues on failure) and require (stops on failure):
require.NoError(t, err) // fatal if error
assert.Equal(t, expected, got) // non-fatal comparison
assert.ErrorIs(t, err, ErrFoo) // error chain check
Good Tests
| Quality | Good | Bad |
|---|---|---|
| Minimal | One thing. "and" in name? Split it. | TestValidatesEmailAndDomainAndWhitespace |
| Clear | Name describes behavior | TestFoo, Test1 |
| Shows intent | Demonstrates desired API | Obscures what code should do |
| Table-driven | Multiple cases, one test function | Copy-pasted test functions |
Common Rationalizations
| Excuse | Reality |
|---|---|
| "Too simple to test" | Simple code breaks. Test takes 30 seconds. |
| "I'll test after" | Tests passing immediately prove nothing. |
| "Tests after achieve same goals" | Tests-after = "what does this do?" Tests-first = "what should this do?" |
| "Already manually tested" | Ad-hoc ≠ systematic. No record, can't re-run. |
| "Deleting X hours is wasteful" | Sunk cost fallacy. Keeping unverified code is technical debt. |
| "Keep as reference, write tests first" | You'll adapt it. That's testing after. Delete means delete. |
| "Need to explore first" | Fine. Throw away exploration, start with TDD. |
| "Test hard = design unclear" | Listen to the test. Hard to test = hard to use. |
| "TDD will slow me down" | TDD faster than debugging. Pragmatic = test-first. |
| "Existing code has no tests" | You're improving it. Add tests for existing code. |
Red Flags - STOP and Start Over
- Code before test
- Test after implementation
- Test passes immediately without seeing it fail
- Can't explain why test failed
- Tests added "later"
- Rationalizing "just this once"
- "Already manually tested it"
- "Tests after achieve the same purpose"
- "Keep as reference" or "adapt existing code"
- "Already spent X hours, deleting is wasteful"
- "TDD is dogmatic, I'm being pragmatic"
- "This is different because..."
All of these mean: Delete code. Start over with TDD.
Example: Bug Fix
Bug: Empty email accepted
RED
func TestSubmitForm_RejectsEmptyEmail(t *testing.T) {
result := submitForm(FormData{Email: ""})
assert.Equal(t, "email required", result.Error)
}
Verify RED
$ go test -run TestSubmitForm_RejectsEmptyEmail ./...
FAIL: expected "email required", got ""
GREEN
func submitForm(data FormData) FormResult {
if strings.TrimSpace(data.Email) == "" {
return FormResult{Error: "email required"}
}
// ...
}
Verify GREEN
$ go test ./...
ok example.com/myapp 0.003s
REFACTOR Extract validation for multiple fields if needed.
Verification Checklist
Before marking work complete:
- Every new function/method has a test
- Watched each test fail before implementing
- Each test failed for expected reason (feature missing, not typo)
- Wrote minimal code to pass each test
- All tests pass
- The language's static gate is clean —
go vet,mypy --strict,tsc --noEmit,cargo clippy. A red type-check is a red build; it does not wait for GREEN. - The language's concurrency gate is clean where one exists and applies —
go test -race,cargo testunder Miri for unsafe code. There is no equivalent for a pure Python function, and a checklist box that can never be ticked teaches people to skip checklists. Tick it or strike it deliberately; do not leave it hanging. - Tests use real code (mocks only if crossing a system boundary)
- Edge cases and errors covered
Can't check or deliberately strike all boxes? You skipped TDD. Start over. A box struck with a stated reason is answered; a box left hanging is not.
When Stuck
| Problem | Solution |
|---|---|
| Don't know how to test | Write wished-for API. Write assertion first. Ask Mathias. |
| Test too complicated | Design too complicated. Simplify interface. |
| Must mock everything | Code too coupled. Use dependency injection. |
| Test setup huge | Extract helpers (t.Helper() in Go, a local __tracebackhide__ = True in Python). Still complex? Simplify design. |
Debugging Integration
Bug found? Write failing test reproducing it. Follow TDD cycle. Test proves fix and prevents regression.
Never fix bugs without a test.
Testing Anti-Patterns
When adding test utilities or mocks, load the references/testing-anti-patterns.md to avoid:
- Testing mock behavior instead of real behavior
- Adding test-only methods to production types
- Mocking without understanding what the dependency does
Brain MCP Integration
The brain MCP exposes session context across machines. Use it to make TDD cycles cumulative rather than one-shot.
At session start:
- Run
brain_querywith the feature name + "tdd" to surface prior cycles, anti-patterns, or testing decisions for this code area. Skip if the feature is brand new.
Never:
- Embed brain content in test code or assertions. The brain is context for you, not state for the system under test.
Logging
Call session_log once at the end of every phase to record the outcome.
Pass-rate is computed downstream by the /pass-rate HTTP endpoint, which
treats pass as success, fail as failure, skip as neither.
At end of red phase:
session_logwith{session_id: "<this session's id>", skill: "tdd", phase: "red", final_status: "pass" | "fail" | "skip", message: "<one-line summary>", project_root: "<absolute path>"}
At end of green phase:
session_logwith{session_id: "<this session's id>", skill: "tdd", phase: "green", final_status: "pass" | "fail" | "skip", message: "<one-line summary>", project_root: "<absolute path>"}
At end of refactor phase:
session_logwith{session_id: "<this session's id>", skill: "tdd", phase: "refactor", final_status: "pass" | "fail" | "skip", message: "<one-line summary>", project_root: "<absolute path>"}
Status semantics:
pass— the phase's intended outcome was reached (red: test fails as expected; green: test passes; refactor: tests still pass after refactor).fail— the phase's intended outcome was NOT reached (test compiled when it shouldn't, test still fails after green attempt, refactor broke tests).skip— phase was skipped intentionally (e.g. refactor not warranted).
Why this matters: pass-rate is what a future routing decision would be based on — whether a tdd call can be served by a local model instead of a cloud one. A skill that never logs contributes no evidence either way, and a decision made on no evidence is a coin flip wearing a metric.
Final Rule
Production code → test exists and failed first
Otherwise → not TDD
No exceptions without Mathias's permission.