fix(health): /healthz answers now, and /readyz takes a degraded pod out
CD / Deploy via GitOps (push) Successful in 5s
CD / Lint / Test / Vet (push) Successful in 30s
CD / Build & Import (push) Successful in 15s

Closes #57. Consumes mcp-chassis v0.6.0 (mcp-chassis#4).

Two halves of one six-day outage. The chassis fix alone would not have fixed
this one.

/HEALTHZ WAS A CONSTANT

newHealthzHandler marshalled its body ONCE at startup and served those bytes for
the process lifetime. On 2026-09-14 it reported jwt.status=degraded for six days
after Authentik had recovered, because the answer was computed before Authentik
was up and could never change. A health endpoint that cannot change its answer
is not a health endpoint.

It now evaluates readiness per request. That is also what drives recovery: the
chassis resolves a pending issuer on demand, so every probe is a retry.

/READYZ IS NEW, AND IT IS THE ACTUAL CONTROL

Through all three occurrences the pod was 1/1 Running and readiness-green while
401ing every OAuth caller. k8s had no signal and kept routing to it. /readyz
answers 503 while JWT auth is degraded, so the pod leaves the Service instead of
serving guaranteed failures, and the kubelet's probe cadence becomes the retry
loop — no background goroutine, no context to own.

/healthz deliberately stays 200. The process is alive; a liveness probe must not
restart it, because restarting does not make the IdP come back sooner.

A server with no issuer configured is always ready. Making readiness depend on an
issuer it was never given would pull every static-token-only MCP out of its
Service — asserted, not assumed.

THE RETRY-WITHOUT-THE-K8S-ISSUER FALLBACK IS GONE

It existed because one unreachable issuer used to fail the whole constructor and
take Authentik auth down with it. Since v0.6.0 each issuer resolves lazily and
independently, so a down issuer degrades only itself. The warning it logged —
"multi-issuer init failed; retrying without the k8s issuer" — was the first line
of today's incident; it no longer has a reason to exist.

A constructor error now means a config error, which exits non-zero rather than
warning and continuing.

TESTS

healthz_live_test.go, written first and red (newReadyzHandler undefined, and
newHealthzHandler's signature could not express a state that changes). The
sharpest assertion is that readiness is evaluated TWICE for two requests — a
handler that caches cannot pass it.

The existing table test is ported, not weakened: same three cases, same expected
statuses, only the mechanism by which state reaches the handler changed.

Verified: task check EXIT=0 (lint, vet, race). The lint failure it surfaced was
pre-existing and is fixed in the preceding commit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-14 20:06:15 +02:00
co-authored by Claude Opus 5
parent aaaa4b4d64
commit 09bde6df8a
7 changed files with 201 additions and 30 deletions
+2
View File
@@ -1,5 +1,7 @@
git.d-ma.be/mathias/mcp-chassis v0.5.0 h1:0w3dt4t4r8OtZBHIOoZkR6p6L3L6YDiqhMh9bTI/w/8=
git.d-ma.be/mathias/mcp-chassis v0.5.0/go.mod h1:Ks7EK2UnGAN0H3rJjKUxUagX8/ZBdtLrOlcUbv0RwH8=
git.d-ma.be/mathias/mcp-chassis v0.6.0 h1:/MuxT9IyzZbN26+COlI2FK0moTyXMLG/qXc/cm0i0Vg=
git.d-ma.be/mathias/mcp-chassis v0.6.0/go.mod h1:Ks7EK2UnGAN0H3rJjKUxUagX8/ZBdtLrOlcUbv0RwH8=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=