The pressure test that #18 and #20 were waiting on. All six original gaps came back ANSWERED with the lines quoted — and the run then found ten problems in what I had written. The worst two are mine, not inherited. **The Python exemplar was the evaluation task.** references/python.md demonstrated `split_vat` — the exact function the test scenario asks for, negative-amount case included. The run said so: "as a teaching example that's over-fitted; as an evaluation it's a leak... this task can't answer it — the reference handed over the finished artifact." So the run could not measure whether the guidance transfers, only whether copying works. The exemplar is now `parse_retry_after`, deliberately unrelated to anything in this estate, and it states its own contract rather than requiring the reader to divide 12500 by 1.25 to discover it. **mypy at RED pointed at the stub the same file forbids.** python.md taught that a pytest ImportError is a legitimate red, then presented `mypy --strict` as a matched pair without saying that mypy reports the same missing module — so following both literally, the way to get mypy green at RED is to create the module. Now stated: that error is the expected red; what must be fixed immediately is a --strict violation inside the test file you just wrote. The rest, each quoted in the run: - The checklist contradicted itself two lines apart — "strike it deliberately" above, "Can't check all boxes? You skipped TDD" below. A struck box is by definition unchecked, so the Python path routed straight into "start over". - `match=` was overclaimed as "free". It is `re.search`, so a message containing parens quietly matches almost anything; and it couples every error test to wording. The Go comparison was a strawman against `assert.Error` when `assert.ErrorIs` exists — the honest Python analogue is a custom exception type, with `match=` as the fallback. - `pytest.approx` was listed as the soft-assert analogue. It solves float tolerance and has nothing to do with collecting failures; that is pytest-subtests or pytest-check, neither of which ships with pytest. - The whole-shape assertion silently required value equality — on a plain class it compares identity and can never pass. - "When Stuck" is a language-neutral table that prescribed `t.Helper()`. - `__tracebackhide__` must be a local inside the helper body; at module level it does nothing, silently. Refs #18, #20 Bump-Type: patch
432 lines
13 KiB
Markdown
432 lines
13 KiB
Markdown
---
|
|
name: tdd
|
|
surfaces: [claude-code, library]
|
|
summary: "Test-Driven Development enforcement"
|
|
use_when: "Starting any new feature or fixing a bug"
|
|
description: Write failing tests first, then minimal code to pass. Non-negotiable for all new features and bug fixes.
|
|
---
|
|
|
|
# Test-Driven Development (TDD)
|
|
|
|
## Overview
|
|
|
|
Write the test first. Watch it fail. Write minimal code to pass.
|
|
|
|
**Core principle:** If you didn't watch the test fail, you don't know if it tests the right thing.
|
|
|
|
**Violating the letter of the rules is violating the spirit of the rules.**
|
|
|
|
## When to Use
|
|
|
|
**Always:**
|
|
- New features
|
|
- Bug fixes
|
|
- Refactoring
|
|
- Behavior changes
|
|
|
|
**Exceptions (ask Mathias):**
|
|
- Throwaway prototypes
|
|
- Generated code (sqlc output, templ output)
|
|
- Configuration files
|
|
|
|
Thinking "skip TDD just this once"? Stop. That's rationalization.
|
|
|
|
## The Iron Law
|
|
|
|
```
|
|
NO PRODUCTION CODE WITHOUT A FAILING TEST FIRST
|
|
```
|
|
|
|
Write code before the test? Delete it. Start over.
|
|
|
|
**No exceptions:**
|
|
- Don't keep it as "reference"
|
|
- Don't "adapt" it while writing tests
|
|
- Don't look at it
|
|
- Delete means delete
|
|
|
|
Implement fresh from tests. Period.
|
|
|
|
## Red-Green-Refactor
|
|
|
|
```
|
|
RED → verify fails correctly → GREEN → verify all pass → REFACTOR → stay green → RED (next)
|
|
```
|
|
|
|
### RED - Write Failing Test
|
|
|
|
Write one minimal test showing what should happen.
|
|
|
|
**Go example (good):**
|
|
```go
|
|
func TestRetryOperation_RetriesThreeTimes(t *testing.T) {
|
|
attempts := 0
|
|
op := func() error {
|
|
attempts++
|
|
if attempts < 3 {
|
|
return errors.New("fail")
|
|
}
|
|
return nil
|
|
}
|
|
|
|
err := RetryOperation(op, 3)
|
|
|
|
require.NoError(t, err)
|
|
assert.Equal(t, 3, attempts)
|
|
}
|
|
```
|
|
Clear name, tests real behavior, one thing.
|
|
|
|
**Go example (bad):**
|
|
```go
|
|
func TestRetry(t *testing.T) {
|
|
// Vague name, tests nothing meaningful
|
|
err := RetryOperation(nil, 0)
|
|
assert.NoError(t, err)
|
|
}
|
|
```
|
|
|
|
**Requirements:**
|
|
- One behavior per test
|
|
- Clear name describing the behavior
|
|
- Real code (no mocks unless crossing a system boundary)
|
|
|
|
### Verify RED - Watch It Fail
|
|
|
|
**MANDATORY. Never skip.**
|
|
|
|
```bash
|
|
go test -run TestRetryOperation_RetriesThreeTimes ./...
|
|
```
|
|
|
|
Confirm:
|
|
- Test fails, and fails because the feature is missing
|
|
- The failure message is the one you expected
|
|
- It is not failing on a typo in the test itself
|
|
|
|
**Test passes?** You're testing existing behavior. Fix test.
|
|
|
|
**Failing because the thing does not exist yet?** That is a legitimate red — a Go
|
|
compile error for an undefined function, a Python `ImportError` for a module you have
|
|
not written, a TypeScript "cannot find module". Read the message, confirm it names the
|
|
unit you are about to build, and proceed to GREEN. **Do not write a stub to make the
|
|
runner happier.** A stub is production code written before a failing test, which is the
|
|
Iron Law with extra steps.
|
|
|
|
**Failing on a typo or a broken import elsewhere?** Different thing. Fix it and re-run
|
|
until the failure is the one above.
|
|
|
|
### GREEN - Minimal Code
|
|
|
|
Write simplest code to pass the test.
|
|
|
|
**Good:**
|
|
```go
|
|
func RetryOperation(op func() error, maxRetries int) error {
|
|
for i := 0; i < maxRetries; i++ {
|
|
if err := op(); err == nil {
|
|
return nil
|
|
}
|
|
}
|
|
return op()
|
|
}
|
|
```
|
|
Just enough to pass.
|
|
|
|
**Bad:**
|
|
```go
|
|
func RetryOperation(op func() error, maxRetries int, opts ...RetryOption) error {
|
|
// YAGNI — don't add options, backoff, jitter before the test demands it
|
|
}
|
|
```
|
|
Over-engineered.
|
|
|
|
Don't add features, refactor other code, or "improve" beyond what the test demands.
|
|
|
|
### Verify GREEN - Watch It Pass
|
|
|
|
**MANDATORY.**
|
|
|
|
```bash
|
|
go test ./...
|
|
```
|
|
|
|
Confirm:
|
|
- Test passes
|
|
- All other tests still pass
|
|
- No race conditions: `go test -race ./...`
|
|
|
|
**Test fails?** Fix code, not test.
|
|
|
|
**Other tests fail?** Fix now.
|
|
|
|
### REFACTOR - Clean Up
|
|
|
|
After green only:
|
|
- Remove duplication
|
|
- Improve names
|
|
- Extract helpers
|
|
- Apply clean code principles (load `clean-code` skill)
|
|
|
|
Keep tests green. Don't add behavior.
|
|
|
|
### Repeat
|
|
|
|
Next failing test for next behavior.
|
|
|
|
## Other languages
|
|
|
|
The discipline above is language-neutral; the examples below it are Go, because Go is
|
|
this fleet's default stack and one good example beats five mediocre ones.
|
|
|
|
For anything else, the per-language conventions live in reference files — load the one
|
|
you need rather than translating Go idioms by eye. Translation is where the mistakes
|
|
happen: the Go table-driven exemplar below branches on a `wantErr` column inside the
|
|
loop, which is idiomatic Go and an anti-pattern in pytest, where it produces a
|
|
conditional assertion and folds two behaviours into one test.
|
|
|
|
- `references/python.md` — pytest, `@pytest.mark.parametrize`, `mypy --strict` in the
|
|
RED gate, `pytest.raises(..., match=)`, fixtures vs helpers
|
|
- `references/typescript.md` — vitest/jest, `it.each`, `expect(...).rejects`
|
|
- `references/rust.md` — `#[cfg(test)]` modules, table-driven via arrays, `#[should_panic]`
|
|
- `references/java.md` — JUnit 5, `@ParameterizedTest`, `assertThrows`
|
|
|
|
## Go-Specific TDD Notes
|
|
|
|
### Table-Driven Tests (Preferred Pattern)
|
|
|
|
```go
|
|
func TestValidateEmail(t *testing.T) {
|
|
tests := []struct {
|
|
name string
|
|
email string
|
|
wantErr bool
|
|
}{
|
|
{"valid email", "user@example.com", false},
|
|
{"empty email", "", true},
|
|
{"no at sign", "userexample.com", true},
|
|
{"no domain", "user@", true},
|
|
}
|
|
|
|
for _, tt := range tests {
|
|
t.Run(tt.name, func(t *testing.T) {
|
|
err := ValidateEmail(tt.email)
|
|
if tt.wantErr {
|
|
assert.Error(t, err)
|
|
} else {
|
|
assert.NoError(t, err)
|
|
}
|
|
})
|
|
}
|
|
}
|
|
```
|
|
|
|
Table-driven tests are the Go idiom. Use them for behavior that varies across inputs.
|
|
|
|
### Subtests with t.Run
|
|
|
|
Use `t.Run` to name subtests clearly. Failure messages include the subtest name.
|
|
|
|
```go
|
|
t.Run("rejects empty input", func(t *testing.T) { ... })
|
|
t.Run("accepts valid UUID", func(t *testing.T) { ... })
|
|
```
|
|
|
|
### Test File Conventions
|
|
|
|
- File: `<package>_test.go` in the same directory
|
|
- Package: `package foo_test` for black-box testing (preferred), `package foo` for white-box
|
|
- Helpers: use `t.Helper()` so stack traces point to the caller, not the helper
|
|
|
|
```go
|
|
func assertNoError(t *testing.T, err error) {
|
|
t.Helper()
|
|
if err != nil {
|
|
t.Fatalf("unexpected error: %v", err)
|
|
}
|
|
}
|
|
```
|
|
|
|
### Running Tests
|
|
|
|
```bash
|
|
go test ./... # all packages
|
|
go test -run TestFoo ./pkg/... # specific test
|
|
go test -run TestFoo/subtest ./... # specific subtest
|
|
go test -race ./... # race detector (always run before commit)
|
|
go test -cover ./... # coverage
|
|
go test -v ./... # verbose
|
|
```
|
|
|
|
### testify
|
|
|
|
Pre-approved. Use `assert` (continues on failure) and `require` (stops on failure):
|
|
|
|
```go
|
|
require.NoError(t, err) // fatal if error
|
|
assert.Equal(t, expected, got) // non-fatal comparison
|
|
assert.ErrorIs(t, err, ErrFoo) // error chain check
|
|
```
|
|
|
|
## Good Tests
|
|
|
|
| Quality | Good | Bad |
|
|
|---------|------|-----|
|
|
| **Minimal** | One thing. "and" in name? Split it. | `TestValidatesEmailAndDomainAndWhitespace` |
|
|
| **Clear** | Name describes behavior | `TestFoo`, `Test1` |
|
|
| **Shows intent** | Demonstrates desired API | Obscures what code should do |
|
|
| **Table-driven** | Multiple cases, one test function | Copy-pasted test functions |
|
|
|
|
## Common Rationalizations
|
|
|
|
| Excuse | Reality |
|
|
|--------|---------|
|
|
| "Too simple to test" | Simple code breaks. Test takes 30 seconds. |
|
|
| "I'll test after" | Tests passing immediately prove nothing. |
|
|
| "Tests after achieve same goals" | Tests-after = "what does this do?" Tests-first = "what should this do?" |
|
|
| "Already manually tested" | Ad-hoc ≠ systematic. No record, can't re-run. |
|
|
| "Deleting X hours is wasteful" | Sunk cost fallacy. Keeping unverified code is technical debt. |
|
|
| "Keep as reference, write tests first" | You'll adapt it. That's testing after. Delete means delete. |
|
|
| "Need to explore first" | Fine. Throw away exploration, start with TDD. |
|
|
| "Test hard = design unclear" | Listen to the test. Hard to test = hard to use. |
|
|
| "TDD will slow me down" | TDD faster than debugging. Pragmatic = test-first. |
|
|
| "Existing code has no tests" | You're improving it. Add tests for existing code. |
|
|
|
|
## Red Flags - STOP and Start Over
|
|
|
|
- Code before test
|
|
- Test after implementation
|
|
- Test passes immediately without seeing it fail
|
|
- Can't explain why test failed
|
|
- Tests added "later"
|
|
- Rationalizing "just this once"
|
|
- "Already manually tested it"
|
|
- "Tests after achieve the same purpose"
|
|
- "Keep as reference" or "adapt existing code"
|
|
- "Already spent X hours, deleting is wasteful"
|
|
- "TDD is dogmatic, I'm being pragmatic"
|
|
- "This is different because..."
|
|
|
|
**All of these mean: Delete code. Start over with TDD.**
|
|
|
|
## Example: Bug Fix
|
|
|
|
**Bug:** Empty email accepted
|
|
|
|
**RED**
|
|
```go
|
|
func TestSubmitForm_RejectsEmptyEmail(t *testing.T) {
|
|
result := submitForm(FormData{Email: ""})
|
|
assert.Equal(t, "email required", result.Error)
|
|
}
|
|
```
|
|
|
|
**Verify RED**
|
|
```bash
|
|
$ go test -run TestSubmitForm_RejectsEmptyEmail ./...
|
|
FAIL: expected "email required", got ""
|
|
```
|
|
|
|
**GREEN**
|
|
```go
|
|
func submitForm(data FormData) FormResult {
|
|
if strings.TrimSpace(data.Email) == "" {
|
|
return FormResult{Error: "email required"}
|
|
}
|
|
// ...
|
|
}
|
|
```
|
|
|
|
**Verify GREEN**
|
|
```bash
|
|
$ go test ./...
|
|
ok example.com/myapp 0.003s
|
|
```
|
|
|
|
**REFACTOR**
|
|
Extract validation for multiple fields if needed.
|
|
|
|
## Verification Checklist
|
|
|
|
Before marking work complete:
|
|
|
|
- [ ] Every new function/method has a test
|
|
- [ ] Watched each test fail before implementing
|
|
- [ ] Each test failed for expected reason (feature missing, not typo)
|
|
- [ ] Wrote minimal code to pass each test
|
|
- [ ] All tests pass
|
|
- [ ] The language's static gate is clean — `go vet`, `mypy --strict`, `tsc --noEmit`,
|
|
`cargo clippy`. A red type-check is a red build; it does not wait for GREEN.
|
|
- [ ] The language's concurrency gate is clean **where one exists and applies** —
|
|
`go test -race`, `cargo test` under Miri for unsafe code. There is no equivalent
|
|
for a pure Python function, and a checklist box that can never be ticked teaches
|
|
people to skip checklists. Tick it or strike it deliberately; do not leave it
|
|
hanging.
|
|
- [ ] Tests use real code (mocks only if crossing a system boundary)
|
|
- [ ] Edge cases and errors covered
|
|
|
|
Can't check **or deliberately strike** all boxes? You skipped TDD. Start over. A box struck with a stated reason is answered; a box left hanging is not.
|
|
|
|
## When Stuck
|
|
|
|
| Problem | Solution |
|
|
|---------|----------|
|
|
| Don't know how to test | Write wished-for API. Write assertion first. Ask Mathias. |
|
|
| Test too complicated | Design too complicated. Simplify interface. |
|
|
| Must mock everything | Code too coupled. Use dependency injection. |
|
|
| Test setup huge | Extract helpers (`t.Helper()` in Go, a local `__tracebackhide__ = True` in Python). Still complex? Simplify design. |
|
|
|
|
## Debugging Integration
|
|
|
|
Bug found? Write failing test reproducing it. Follow TDD cycle. Test proves fix and prevents regression.
|
|
|
|
Never fix bugs without a test.
|
|
|
|
## Testing Anti-Patterns
|
|
|
|
When adding test utilities or mocks, load the `references/testing-anti-patterns.md` to avoid:
|
|
- Testing mock behavior instead of real behavior
|
|
- Adding test-only methods to production types
|
|
- Mocking without understanding what the dependency does
|
|
|
|
## Brain MCP Integration
|
|
|
|
The brain MCP exposes session context across machines. Use it to make TDD cycles cumulative rather than one-shot.
|
|
|
|
**At session start:**
|
|
- Run `brain_query` with the feature name + "tdd" to surface prior cycles, anti-patterns, or testing decisions for this code area. Skip if the feature is brand new.
|
|
|
|
**Never:**
|
|
- Embed brain content in test code or assertions. The brain is context for you, not state for the system under test.
|
|
|
|
### Logging
|
|
|
|
Call `session_log` once at the end of every phase to record the outcome.
|
|
Pass-rate is computed downstream by the `/pass-rate` HTTP endpoint, which
|
|
treats `pass` as success, `fail` as failure, `skip` as neither.
|
|
|
|
**At end of `red` phase:**
|
|
- `session_log` with `{session_id: "<this session's id>", skill: "tdd", phase: "red", final_status: "pass" | "fail" | "skip", message: "<one-line summary>", project_root: "<absolute path>"}`
|
|
|
|
**At end of `green` phase:**
|
|
- `session_log` with `{session_id: "<this session's id>", skill: "tdd", phase: "green", final_status: "pass" | "fail" | "skip", message: "<one-line summary>", project_root: "<absolute path>"}`
|
|
|
|
**At end of `refactor` phase:**
|
|
- `session_log` with `{session_id: "<this session's id>", skill: "tdd", phase: "refactor", final_status: "pass" | "fail" | "skip", message: "<one-line summary>", project_root: "<absolute path>"}`
|
|
|
|
**Status semantics:**
|
|
- `pass` — the phase's intended outcome was reached (red: test fails as expected; green: test passes; refactor: tests still pass after refactor).
|
|
- `fail` — the phase's intended outcome was NOT reached (test compiled when it shouldn't, test still fails after green attempt, refactor broke tests).
|
|
- `skip` — phase was skipped intentionally (e.g. refactor not warranted).
|
|
|
|
**Why this matters:** pass-rate is what a future routing decision would be based on — whether a `tdd` call can be served by a local model instead of a cloud one. A skill that never logs contributes no evidence either way, and a decision made on no evidence is a coin flip wearing a metric.
|
|
|
|
## Final Rule
|
|
|
|
```
|
|
Production code → test exists and failed first
|
|
Otherwise → not TDD
|
|
```
|
|
|
|
No exceptions without Mathias's permission.
|