The template cannot test itself — the guard skips check as well as build/deploy #12

Closed
opened 2026-09-17 14:31:03 +00:00 by mathias · 2 comments
Owner

The gap

The guard that stops the template building itself also stops it testing itself. On PR #11:

Detect unsubstituted template -> success   (is_template=true)
Lint / Test / Vet             -> skipped
Build & Import                -> skipped
Deploy via GitOps             -> skipped

All three real jobs are gated on needs.guard.outputs.is_template != 'true', and in this repo that is always true — by design. So no change to this template can ever be verified by its own CI.

That is not the fake-green bug from #9/#10, which was a wrong answer. This is the right answer with an unintended consequence: the repo that mints every other repo is the one repo with no working gate. Given #9 and #10 both shipped because an inline workflow block could only be reviewed by reading, that is the worst place in the estate to have no test signal.

It also means PR #11's own fix was merged on local evidence rather than a CI verdict:

guard in a simulated child  -> false   (CI runs)
guard in the template       -> true    (still skips its own build)
test suite, both states     -> 8 passed, 0 failed

Good evidence, and reproducible — but it is a person running a command, which this estate has repeatedly decided is not a control.

Why build and deploy should stay gated

They genuinely cannot work here. The module path is a placeholder, there is no image name to push, and there is no namespace to deploy into. Skipping them is correct.

Why check probably should not be

Linting, vetting and testing a template is meaningful work — arguably more meaningful than in a child, since a defect here is inherited by every repo ever created from it. template-go-agent#1 and #2 are exactly that: a scaffold that ships lint-dirty and fails task check from birth. Neither would survive a check job that actually ran in the template.

The open question is whether the Go half can run against a placeholder module path. module git.d-ma.be/mathias/<placeholder> is a syntactically valid module path and internal imports resolve locally, so go vet ./... and go test ./... may simply work. That needs establishing by running it, not by reasoning about it — the failure mode here is subtle and the estate's record on "it should be fine" is poor.

Proposed

  1. Establish empirically whether task check passes on the raw template. One run.
  2. If it does: ungate check from the guard, leaving build and deploy gated. The template then verifies every change to itself.
  3. If it does not: identify which specific step fails on a placeholder module path, and gate only that step — not the whole job.
  4. Either way scripts/tests/is-template_test.sh (added in #11) must run in this repo's CI. It is pure bash with no module dependency, so it has no reason to be gated at all.

Done when

  • A change that breaks scripts/is-template.sh fails this repo's CI, demonstrated by breaking it on purpose rather than by reading the YAML
  • build and deploy still skip on the template
  • A child repo is unaffected — verified by the substitution simulation in #11

Source

Observed on run 7480, PR #11. Related: #9, #10 (the bug that motivated the guard), template-go-agent#1/#2 (defects a working check job here would have caught).


Created via git-mcp on behalf of @mathias

## The gap The guard that stops the template building itself also stops it **testing** itself. On PR #11: ``` Detect unsubstituted template -> success (is_template=true) Lint / Test / Vet -> skipped Build & Import -> skipped Deploy via GitOps -> skipped ``` All three real jobs are gated on `needs.guard.outputs.is_template != 'true'`, and in this repo that is always `true` — by design. So **no change to this template can ever be verified by its own CI.** That is not the fake-green bug from #9/#10, which was a *wrong* answer. This is the right answer with an unintended consequence: the repo that mints every other repo is the one repo with no working gate. Given #9 and #10 both shipped *because* an inline workflow block could only be reviewed by reading, that is the worst place in the estate to have no test signal. It also means PR #11's own fix was merged on local evidence rather than a CI verdict: ``` guard in a simulated child -> false (CI runs) guard in the template -> true (still skips its own build) test suite, both states -> 8 passed, 0 failed ``` Good evidence, and reproducible — but it is a person running a command, which this estate has repeatedly decided is not a control. ## Why `build` and `deploy` should stay gated They genuinely cannot work here. The module path is a placeholder, there is no image name to push, and there is no namespace to deploy into. Skipping them is correct. ## Why `check` probably should not be Linting, vetting and testing a template is meaningful work — arguably *more* meaningful than in a child, since a defect here is inherited by every repo ever created from it. `template-go-agent#1` and `#2` are exactly that: a scaffold that ships lint-dirty and fails `task check` from birth. Neither would survive a `check` job that actually ran in the template. The open question is whether the Go half can run against a placeholder module path. `module git.d-ma.be/mathias/<placeholder>` is a syntactically valid module path and internal imports resolve locally, so `go vet ./...` and `go test ./...` may simply work. **That needs establishing by running it, not by reasoning about it** — the failure mode here is subtle and the estate's record on "it should be fine" is poor. ## Proposed 1. Establish empirically whether `task check` passes on the raw template. One run. 2. If it does: ungate `check` from the guard, leaving `build` and `deploy` gated. The template then verifies every change to itself. 3. If it does not: identify which specific step fails on a placeholder module path, and gate only that step — not the whole job. 4. Either way `scripts/tests/is-template_test.sh` (added in #11) must run in this repo's CI. It is pure bash with no module dependency, so it has no reason to be gated at all. ## Done when - A change that breaks `scripts/is-template.sh` fails this repo's CI, demonstrated by breaking it on purpose rather than by reading the YAML - `build` and `deploy` still skip on the template - A child repo is unaffected — verified by the substitution simulation in #11 ## Source Observed on run 7480, PR #11. Related: #9, #10 (the bug that motivated the guard), `template-go-agent#1`/`#2` (defects a working `check` job here would have caught). --- _Created via git-mcp on behalf of @mathias_
mathias added the risk/mediumsize/xstype/fix labels 2026-09-17 14:31:03 +00:00
Author
Owner

Gate review of PR #13 — not merged, one blocking finding. Full evidence: #13 (comment 11976).

All three of this issue's done-when clauses are satisfied, and I verified each by running it rather than reading the YAML:

  1. A change that breaks scripts/is-template.sh fails this repo's CI, demonstrated by breaking it on purpose — I narrowed the guard's regex to a literal placeholder token; scripts/tests/is-template_test.sh went 4 passed, 4 failed, exit 1. Because check is now ungated and runs that harness, this fails the repo's CI rather than passing silently.
  2. build and deploy still skip on the template — run 7511 (head 39b8e86): Detect unsubstituted template success, Lint / Test / Vet success — the first time this repo has ever gated itself, Build & Import skipped, Deploy via GitOps skipped.
  3. A child repo is unaffected — I substituted the tracked tree into a child (mysvc): guard answers false, go list ./... now shows cmd/mysvc, harness 5/5.

The open question in the body — whether the Go half can run against a placeholder module path — resolves to yes for go vet/go test/golangci-lint (CI: 0 issues.), and no for the entrypoint: ./... expansion skips cmd/__PROJECT_NAME__ because the directory name begins with an underscore. PR #13 handles that with a substitute-into-a-child harness rather than by gating a step, which is the right shape.

Why it is not merged. The new harness grades golangci-lint by grepping its stdout and discards its exit code (scripts/tests/generated-child_test.sh:93-98). I put a shim on PATH that exits 3 with a fatal error and the harness reported ok ... 5 passed, 0 failed, exit 0. Since that harness is the only gate covering the entrypoint, a crashed linter would leave the entrypoint unchecked with CI green — the same class of defect this issue was opened about. The fix is to assert on the exit status; three lines. Re-gating once it lands.

**Gate review of PR #13 — not merged, one blocking finding.** Full evidence: #13 (comment 11976). All three of this issue's done-when clauses are satisfied, and I verified each by running it rather than reading the YAML: 1. *A change that breaks `scripts/is-template.sh` fails this repo's CI, demonstrated by breaking it on purpose* — I narrowed the guard's regex to a literal placeholder token; `scripts/tests/is-template_test.sh` went `4 passed, 4 failed`, exit 1. Because `check` is now ungated and runs that harness, this fails the repo's CI rather than passing silently. 2. *`build` and `deploy` still skip on the template* — run **7511** (head `39b8e86`): `Detect unsubstituted template` success, `Lint / Test / Vet` **success — the first time this repo has ever gated itself**, `Build & Import` skipped, `Deploy via GitOps` skipped. 3. *A child repo is unaffected* — I substituted the tracked tree into a child (`mysvc`): guard answers `false`, `go list ./...` now shows `cmd/mysvc`, harness 5/5. The open question in the body — whether the Go half can run against a placeholder module path — resolves to **yes** for `go vet`/`go test`/`golangci-lint` (CI: `0 issues.`), and **no** for the entrypoint: `./...` expansion skips `cmd/__PROJECT_NAME__` because the directory name begins with an underscore. PR #13 handles that with a substitute-into-a-child harness rather than by gating a step, which is the right shape. **Why it is not merged.** The new harness grades `golangci-lint` by grepping its stdout and discards its exit code (`scripts/tests/generated-child_test.sh:93-98`). I put a shim on `PATH` that exits 3 with a fatal error and the harness reported `ok ... 5 passed, 0 failed`, exit 0. Since that harness is the only gate covering the entrypoint, a crashed linter would leave the entrypoint unchecked with CI green — the same class of defect this issue was opened about. The fix is to assert on the exit status; three lines. Re-gating once it lands.
Author
Owner

Closed by #13 (merged as rebase; main is now 2c4158a). Gated and verified independently — evidence below is mine, from running it, not a restatement of the PR.

Done-when clauses

1. "A change that breaks scripts/is-template.sh fails this repo's CI, demonstrated by breaking it on purpose rather than by reading the YAML."

I narrowed the guard's shape-matching regex to a literal placeholder token — the exact #9/#10 regression:

$ bash scripts/tests/is-template_test.sh
  FAIL the guard survives being substituted itself
     after substitution it answered 'true' for a generated repo — this is #9/#10
  FAIL the substituted guard still detects a raw template
  FAIL the guard contains no substitutable placeholder token
  FAIL any leftover placeholder counts
  4 passed, 4 failed        (exit 1)

That harness runs inside task check, and check is now ungated, so this fails the repo's CI rather than passing invisibly.

2. "build and deploy still skip on the template." Run 7514 (push to main, post-merge):

Detect unsubstituted template -> success
Lint / Test / Vet             -> success
Build & Import                -> skipped
Deploy via GitOps             -> skipped

3. "A child repo is unaffected." I substituted the tracked tree into a real child (mysvc): guard answers false, go list ./... now reports cmd/mysvc alongside internal/web, and the harness passes 5/5 there.

The open question in the body, resolved

Can the Go half run against a placeholder module path? Yes for go vet / go test / golangci-lint — CI reports 0 issues. on the raw template. No for the entrypoint: ./... expansion skips cmd/__PROJECT_NAME__ because the directory name begins with an underscore, so ungating alone would have produced a gate that still never saw the package every child is built around. #13 closes that with a substitute-into-a-child harness rather than by gating a step.

What I blocked on first, and why it is now satisfied

The first revision graded golangci-lint by grepping its stdout and discarded its exit code, so a crashed linter scored as a clean pass (5 passed, 0 failed, exit 0 against a shim exiting 3). That was the same defect class this issue is about, inside the fix for it. Corrected in 2c4158a, and I verified the correction with shims that decouple output from exit status, so the result cannot be an accident of text matching:

Shim stdout exit Result
looks clean, but crashed 0 issues. 3 FAIL — linter exited 3 — the child is UNCHECKED, not clean
looks dirty, but succeeded 7 issues: 0 pass
real errcheck in entrypoint findings 1 FAIL — cmd/childproj/main.go:17:10 ... (errcheck)
templ/golangci-lint off PATH — — FAIL, 0 passed, 1 failed (was exit 0)

The first two are the discriminating pair: grading is genuinely on the exit code, and no stdout grepping remains.

CI: run 7513 (PR head 2c4158a) success — Lint / Test / Vet success, build/deploy skipped; run 7514 (main) identical. No SKIP line anywhere in either log; both harnesses genuinely executed (is-template 8/8, generated-child 5/5). Local task check exit 0, zero FAIL lines.

Residual, filed separately

The harness now substitutes Dockerfile but does not execute it, so Dockerfile:8's ./cmd/__PROJECT_NAME__ path is still unvalidated — I broke it to ./cmd/does_not_exist and the harness passed 5/5. Not a regression and not in this issue's scope; filed as its own issue.

**Closed by #13** (merged as rebase; `main` is now `2c4158a`). Gated and verified independently — evidence below is mine, from running it, not a restatement of the PR. ### Done-when clauses **1. "A change that breaks `scripts/is-template.sh` fails this repo's CI, demonstrated by breaking it on purpose rather than by reading the YAML."** I narrowed the guard's shape-matching regex to a literal placeholder token — the exact #9/#10 regression: ``` $ bash scripts/tests/is-template_test.sh FAIL the guard survives being substituted itself after substitution it answered 'true' for a generated repo — this is #9/#10 FAIL the substituted guard still detects a raw template FAIL the guard contains no substitutable placeholder token FAIL any leftover placeholder counts 4 passed, 4 failed (exit 1) ``` That harness runs inside `task check`, and `check` is now ungated, so this fails the repo's CI rather than passing invisibly. **2. "`build` and `deploy` still skip on the template."** Run **7514** (push to `main`, post-merge): ``` Detect unsubstituted template -> success Lint / Test / Vet -> success Build & Import -> skipped Deploy via GitOps -> skipped ``` **3. "A child repo is unaffected."** I substituted the tracked tree into a real child (`mysvc`): guard answers `false`, `go list ./...` now reports `cmd/mysvc` alongside `internal/web`, and the harness passes 5/5 there. ### The open question in the body, resolved *Can the Go half run against a placeholder module path?* **Yes** for `go vet` / `go test` / `golangci-lint` — CI reports `0 issues.` on the raw template. **No** for the entrypoint: `./...` expansion skips `cmd/__PROJECT_NAME__` because the directory name begins with an underscore, so ungating alone would have produced a gate that still never saw the package every child is built around. #13 closes that with a substitute-into-a-child harness rather than by gating a step. ### What I blocked on first, and why it is now satisfied The first revision graded `golangci-lint` by grepping its stdout and discarded its exit code, so a crashed linter scored as a clean pass (`5 passed, 0 failed`, exit 0 against a shim exiting 3). That was the same defect class this issue is about, inside the fix for it. Corrected in `2c4158a`, and I verified the correction with shims that **decouple output from exit status**, so the result cannot be an accident of text matching: | Shim | stdout | exit | Result | |---|---|---|---| | looks clean, but crashed | `0 issues.` | 3 | **FAIL** — `linter exited 3 — the child is UNCHECKED, not clean` | | looks dirty, but succeeded | `7 issues:` | 0 | **pass** | | real `errcheck` in entrypoint | findings | 1 | **FAIL** — `cmd/childproj/main.go:17:10 ... (errcheck)` | | `templ`/`golangci-lint` off `PATH` | — | — | **FAIL**, `0 passed, 1 failed` (was `exit 0`) | The first two are the discriminating pair: grading is genuinely on the exit code, and no stdout grepping remains. **CI:** run **7513** (PR head `2c4158a`) success — `Lint / Test / Vet` success, build/deploy skipped; run **7514** (main) identical. No `SKIP` line anywhere in either log; both harnesses genuinely executed (`is-template` 8/8, `generated-child` 5/5). Local `task check` exit 0, zero FAIL lines. ### Residual, filed separately The harness now substitutes `Dockerfile` but does not *execute* it, so `Dockerfile:8`'s `./cmd/__PROJECT_NAME__` path is still unvalidated — I broke it to `./cmd/does_not_exist` and the harness passed 5/5. Not a regression and not in this issue's scope; filed as its own issue.
Sign in to join this conversation.