Template
mathiasandClaude Opus 5
4038339727
Found by an adversarial review of the same harness in template-go-web (template-go-web#13). This copy had the identical defect. The harness matched golangci-lint's human-readable summary to decide pass or fail. Any failure that is not "found issues" -- bad config, unwritable TMPDIR, OOM, or simply a future change to the output format -- produces non-matching text and was scored as a pass. That is the exact defect class this harness exists to end. It is the only gate covering cmd/__PROJECT_NAME__, so a linter that dies leaves the entrypoint unchecked with CI green. golangci-lint exits 0 clean / 1 issues found / >1 the linter itself failed. Now graded on that, with the >1 case reported as UNCHECKED rather than clean. Also: a missing tool now fails instead of `exit 0` with zero assertions, which was indistinguishable from a real pass to anything reading the exit code. And Dockerfile joins the substituted file set -- it references the entrypoint path, which is the one thing a rename has historically broken here. Verified with a shimmed linter exiting 3: 3 passed, 1 failed. Normal run 4/4. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
122 lines
5.5 KiB
Bash
Executable File
122 lines
5.5 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Does a repo generated from this template pass its own gate?
|
|
#
|
|
# WHY THIS EXISTS
|
|
#
|
|
# `task check` on the template is green and has been for months, while children
|
|
# generated from it were red from their first commit (#1, #2). The template's
|
|
# own gate cannot see the defect, for a reason that is invisible unless you go
|
|
# looking:
|
|
#
|
|
# Go ignores directories whose names begin with "_" or "." when expanding
|
|
# package patterns like ./...
|
|
#
|
|
# The entrypoint lives in cmd/__PROJECT_NAME__, which begins with an underscore.
|
|
# So `go build ./...`, `go vet ./...`, `go test ./...` AND
|
|
# `golangci-lint run ./...` all silently skip the one package every generated
|
|
# repo is built around. Confirmed:
|
|
#
|
|
# $ go list ./...
|
|
# __MODULE_PATH__/internal/agent
|
|
# __MODULE_PATH__/pkg/litellm <- no cmd package
|
|
#
|
|
# After substitution the directory has a real name, becomes visible, and the
|
|
# defects appear in the child. Template green, children red, by construction.
|
|
#
|
|
# WHY NOT JUST LINT THE DIRECTORY DIRECTLY
|
|
#
|
|
# Tried: `golangci-lint run ./cmd/__PROJECT_NAME__` does work (naming a path
|
|
# directly bypasses the pattern exclusion). But it reports a gofumpt violation
|
|
# on the import block that does NOT exist in a generated repo -- the formatter's
|
|
# import grouping depends on the module path, and the template's is a
|
|
# placeholder rather than a real path. Wiring that into `check` would turn the
|
|
# template's gate red over an artifact of being a template.
|
|
#
|
|
# So this simulates the substitution and gates on the RESULT, which is the only
|
|
# artifact anyone actually runs. Same discipline as template-go-web's
|
|
# scripts/tests/is-template_test.sh: a test for transformation-sensitive code
|
|
# has to undergo the transformation.
|
|
#
|
|
# Usage: ./scripts/tests/generated-child_test.sh
|
|
|
|
set -uo pipefail
|
|
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
|
|
|
pass=0; fail=0
|
|
ok() { printf ' ok %s\n' "$1"; pass=$((pass+1)); }
|
|
bad() { printf ' FAIL %s\n %s\n' "$1" "${2:-}"; fail=$((fail+1)); }
|
|
|
|
echo "generated-child tests"
|
|
|
|
# A missing tool is a FAILURE, not a skip. An `exit 0` with zero assertions is
|
|
# indistinguishable from a real pass to anything reading the exit code -- which
|
|
# is the whole failure mode this harness exists to close.
|
|
if ! command -v golangci-lint >/dev/null 2>&1; then
|
|
bad "golangci-lint is available" "not on PATH — the generated child cannot be gated"
|
|
echo; echo "$pass passed, $fail failed"; exit 1
|
|
fi
|
|
|
|
tmp="$(mktemp -d)"; trap 'rm -rf "$tmp"' EXIT
|
|
|
|
# Build the child exactly as a scaffolder does: the tracked tree, the entrypoint
|
|
# directory renamed, every placeholder substituted.
|
|
#
|
|
# Tracked files from the WORKING TREE, not `git archive HEAD`. This runs inside
|
|
# `task check`, so it has to gate what is about to be committed -- archiving
|
|
# HEAD tests the previous commit and reports stale results against a fix sitting
|
|
# unstaged, which it did on first use here. `git ls-files` keeps the file set
|
|
# honest (no build output, no local scratch) while taking current content.
|
|
( cd "$ROOT" && git ls-files -z ) | while IFS= read -r -d '' f; do
|
|
mkdir -p "$tmp/$(dirname "$f")"
|
|
cp "$ROOT/$f" "$tmp/$f"
|
|
done
|
|
if [ ! -d "$tmp/cmd/__PROJECT_NAME__" ]; then
|
|
bad "the template still has a placeholder entrypoint directory" \
|
|
"cmd/__PROJECT_NAME__ not found — has the layout changed?"
|
|
echo; echo "$pass passed, $fail failed"; exit 1
|
|
fi
|
|
mv "$tmp/cmd/__PROJECT_NAME__" "$tmp/cmd/childproj"
|
|
find "$tmp" -type f \( -name '*.go' -o -name '*.mod' -o -name '*.yml' -o -name '*.md' -o -name 'Dockerfile' \) \
|
|
-exec sed -i \
|
|
-e 's|__MODULE_PATH__|git.d-ma.be/mathias/childproj|g' \
|
|
-e 's|__PROJECT_NAME__|childproj|g' {} +
|
|
|
|
# The whole point: the entrypoint must be visible to Go tooling in the child.
|
|
# If this regresses, everything below it passes for the wrong reason.
|
|
if ( cd "$tmp" && go list ./... 2>/dev/null ) | grep -q '/cmd/childproj$'; then
|
|
ok "the generated child's entrypoint is visible to go list"
|
|
else
|
|
bad "the generated child's entrypoint is visible to go list" \
|
|
"cmd package missing from ./... — the child would be unchecked too"
|
|
fi
|
|
|
|
out="$(cd "$tmp" && go build ./... 2>&1)"
|
|
[ -z "$out" ] && ok "the generated child builds" \
|
|
|| bad "the generated child builds" "$(printf '%s' "$out" | head -3)"
|
|
|
|
out="$(cd "$tmp" && go vet ./... 2>&1)"
|
|
[ -z "$out" ] && ok "the generated child passes go vet" \
|
|
|| bad "the generated child passes go vet" "$(printf '%s' "$out" | head -3)"
|
|
|
|
# The one that caught #1/#2: errcheck on the deferred shutdown, in a package the
|
|
# template's own `golangci-lint run ./...` never analysed.
|
|
# Graded on the EXIT CODE, never on grepping the output. golangci-lint exits
|
|
# 0 clean / 1 issues found / >1 the linter itself failed. Matching its
|
|
# human-readable summary instead scores a CRASHED linter as a pass -- bad
|
|
# config, unwritable TMPDIR, OOM, or a future change to its output format all
|
|
# produce non-matching text. Since this harness is the only gate covering the
|
|
# entrypoint, that would leave it unchecked with CI green, which is precisely
|
|
# the defect class this file exists to end.
|
|
out="$(cd "$tmp" && golangci-lint run ./... 2>&1)"; rc=$?
|
|
case "$rc" in
|
|
0) ok "the generated child passes golangci-lint" ;;
|
|
1) bad "the generated child passes golangci-lint" \
|
|
"$(printf '%s' "$out" | grep -E '\.go:[0-9]+' | head -3)" ;;
|
|
*) bad "golangci-lint ran successfully" \
|
|
"linter exited $rc — the child is UNCHECKED, not clean: $(printf '%s' "$out" | tail -2)" ;;
|
|
esac
|
|
|
|
echo
|
|
echo "$pass passed, $fail failed"
|
|
[ "$fail" -eq 0 ]
|