Closes #57. Consumes mcp-chassis v0.6.0 (mcp-chassis#4). Two halves of one six-day outage. The chassis fix alone would not have fixed this one. /HEALTHZ WAS A CONSTANT newHealthzHandler marshalled its body ONCE at startup and served those bytes for the process lifetime. On 2026-09-14 it reported jwt.status=degraded for six days after Authentik had recovered, because the answer was computed before Authentik was up and could never change. A health endpoint that cannot change its answer is not a health endpoint. It now evaluates readiness per request. That is also what drives recovery: the chassis resolves a pending issuer on demand, so every probe is a retry. /READYZ IS NEW, AND IT IS THE ACTUAL CONTROL Through all three occurrences the pod was 1/1 Running and readiness-green while 401ing every OAuth caller. k8s had no signal and kept routing to it. /readyz answers 503 while JWT auth is degraded, so the pod leaves the Service instead of serving guaranteed failures, and the kubelet's probe cadence becomes the retry loop — no background goroutine, no context to own. /healthz deliberately stays 200. The process is alive; a liveness probe must not restart it, because restarting does not make the IdP come back sooner. A server with no issuer configured is always ready. Making readiness depend on an issuer it was never given would pull every static-token-only MCP out of its Service — asserted, not assumed. THE RETRY-WITHOUT-THE-K8S-ISSUER FALLBACK IS GONE It existed because one unreachable issuer used to fail the whole constructor and take Authentik auth down with it. Since v0.6.0 each issuer resolves lazily and independently, so a down issuer degrades only itself. The warning it logged — "multi-issuer init failed; retrying without the k8s issuer" — was the first line of today's incident; it no longer has a reason to exist. A constructor error now means a config error, which exits non-zero rather than warning and continuing. TESTS healthz_live_test.go, written first and red (newReadyzHandler undefined, and newHealthzHandler's signature could not express a state that changes). The sharpest assertion is that readiness is evaluated TWICE for two requests — a handler that caches cannot pass it. The existing table test is ported, not weakened: same three cases, same expected statuses, only the mechanism by which state reaches the handler changed. Verified: task check EXIT=0 (lint, vet, race). The lint failure it surfaced was pre-existing and is fixed in the preceding commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
100 lines
3.3 KiB
Go
100 lines
3.3 KiB
Go
package main
|
|
|
|
import (
|
|
"context"
|
|
"encoding/json"
|
|
"net/http"
|
|
)
|
|
|
|
type healthStatus struct {
|
|
OK bool `json:"ok"`
|
|
JWT jwtStatus `json:"jwt"`
|
|
}
|
|
|
|
// jwtStatus surfaces the runtime state of the Dex JWT validator so ops
|
|
// can distinguish "Dex unreachable at startup" from "JWT auth not
|
|
// configured" — both previously degraded silently to static-token-only
|
|
// (refs hyperguild/gitea-mcp#6).
|
|
type jwtStatus struct {
|
|
// Status is one of: "disabled" (DEX_ISSUER_URL not set),
|
|
// "enabled" (validator initialized), "degraded" (configured but
|
|
// init failed; only static-token auth currently accepted).
|
|
Status string `json:"status"`
|
|
LastError string `json:"last_error,omitempty"`
|
|
}
|
|
|
|
// readinessFn reports whether JWT validation is currently possible, and why
|
|
// not. chassisauth.JWTValidator.Ready satisfies it.
|
|
type readinessFn func(context.Context) (bool, error)
|
|
|
|
// newHealthzHandler reports the CURRENT state, evaluated per request.
|
|
//
|
|
// It used to marshal one body at startup and serve those bytes forever. That is
|
|
// how gitea-mcp reported jwt.status=degraded for six days after Authentik had
|
|
// recovered (mcp-chassis#4): the answer was computed before the IdP was up and
|
|
// could never change. Fixing the validator alone would not have fixed this —
|
|
// the validator would recover and /healthz would keep saying degraded.
|
|
//
|
|
// Calling ready() per request is also what drives recovery: the chassis resolves
|
|
// a pending issuer on demand, so every probe is a retry.
|
|
func newHealthzHandler(dexConfigured bool, ready readinessFn) http.HandlerFunc {
|
|
return func(w http.ResponseWriter, r *http.Request) {
|
|
status := healthStatus{OK: true, JWT: jwtStatus{Status: "disabled"}}
|
|
if dexConfigured {
|
|
ok, err := ready(r.Context())
|
|
status.JWT.Status = jwtStatusFor(true, ok)
|
|
if err != nil {
|
|
status.JWT.LastError = err.Error()
|
|
}
|
|
}
|
|
w.Header().Set("Content-Type", "application/json")
|
|
_ = json.NewEncoder(w).Encode(status)
|
|
}
|
|
}
|
|
|
|
// newReadyzHandler is the kubelet's probe: 503 while JWT auth is degraded.
|
|
//
|
|
// /healthz stays 200 deliberately — the process is alive and a liveness probe
|
|
// must not restart it, because restarting does not make the IdP come back any
|
|
// sooner. Readiness is the right lever: k8s stops routing OAuth callers to a pod
|
|
// that would 401 them, and the probe's own cadence becomes the retry loop with
|
|
// no background goroutine to own.
|
|
//
|
|
// A server with no issuer configured is always ready. Making readiness depend on
|
|
// an issuer it was never given would pull every static-token-only MCP out of its
|
|
// Service.
|
|
func newReadyzHandler(dexConfigured bool, ready readinessFn) http.HandlerFunc {
|
|
return func(w http.ResponseWriter, r *http.Request) {
|
|
if !dexConfigured {
|
|
w.WriteHeader(http.StatusOK)
|
|
return
|
|
}
|
|
ok, err := ready(r.Context())
|
|
if !ok {
|
|
w.Header().Set("Content-Type", "application/json")
|
|
w.WriteHeader(http.StatusServiceUnavailable)
|
|
msg := ""
|
|
if err != nil {
|
|
msg = err.Error()
|
|
}
|
|
_ = json.NewEncoder(w).Encode(healthStatus{
|
|
OK: false,
|
|
JWT: jwtStatus{Status: "degraded", LastError: msg},
|
|
})
|
|
return
|
|
}
|
|
w.WriteHeader(http.StatusOK)
|
|
}
|
|
}
|
|
|
|
func jwtStatusFor(dexConfigured, validatorReady bool) string {
|
|
switch {
|
|
case !dexConfigured:
|
|
return "disabled"
|
|
case validatorReady:
|
|
return "enabled"
|
|
default:
|
|
return "degraded"
|
|
}
|
|
}
|