Files
gitea-mcp/cmd/gitea-mcp/healthz_live_test.go
T
mathiasandClaude Opus 5 09bde6df8a
CD / Deploy via GitOps (push) Successful in 5s
CD / Lint / Test / Vet (push) Successful in 30s
CD / Build & Import (push) Successful in 15s
fix(health): /healthz answers now, and /readyz takes a degraded pod out
Closes #57. Consumes mcp-chassis v0.6.0 (mcp-chassis#4).

Two halves of one six-day outage. The chassis fix alone would not have fixed
this one.

/HEALTHZ WAS A CONSTANT

newHealthzHandler marshalled its body ONCE at startup and served those bytes for
the process lifetime. On 2026-09-14 it reported jwt.status=degraded for six days
after Authentik had recovered, because the answer was computed before Authentik
was up and could never change. A health endpoint that cannot change its answer
is not a health endpoint.

It now evaluates readiness per request. That is also what drives recovery: the
chassis resolves a pending issuer on demand, so every probe is a retry.

/READYZ IS NEW, AND IT IS THE ACTUAL CONTROL

Through all three occurrences the pod was 1/1 Running and readiness-green while
401ing every OAuth caller. k8s had no signal and kept routing to it. /readyz
answers 503 while JWT auth is degraded, so the pod leaves the Service instead of
serving guaranteed failures, and the kubelet's probe cadence becomes the retry
loop — no background goroutine, no context to own.

/healthz deliberately stays 200. The process is alive; a liveness probe must not
restart it, because restarting does not make the IdP come back sooner.

A server with no issuer configured is always ready. Making readiness depend on an
issuer it was never given would pull every static-token-only MCP out of its
Service — asserted, not assumed.

THE RETRY-WITHOUT-THE-K8S-ISSUER FALLBACK IS GONE

It existed because one unreachable issuer used to fail the whole constructor and
take Authentik auth down with it. Since v0.6.0 each issuer resolves lazily and
independently, so a down issuer degrades only itself. The warning it logged —
"multi-issuer init failed; retrying without the k8s issuer" — was the first line
of today's incident; it no longer has a reason to exist.

A constructor error now means a config error, which exits non-zero rather than
warning and continuing.

TESTS

healthz_live_test.go, written first and red (newReadyzHandler undefined, and
newHealthzHandler's signature could not express a state that changes). The
sharpest assertion is that readiness is evaluated TWICE for two requests — a
handler that caches cannot pass it.

The existing table test is ported, not weakened: same three cases, same expected
statuses, only the mechanism by which state reaches the handler changed.

Verified: task check EXIT=0 (lint, vet, race). The lint failure it surfaced was
pre-existing and is fixed in the preceding commit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-14 20:06:15 +02:00

107 lines
3.6 KiB
Go

package main
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"github.com/stretchr/testify/require"
)
// readinessFunc is what the handler asks. Named here so the test can drive it.
type fakeReadiness struct{ calls int }
// ready reports not-ready on the first call and ready afterwards — the shape of
// an IdP that was down when the process booted and came back a minute later.
func (f *fakeReadiness) ready(context.Context) (bool, error) {
f.calls++
if f.calls == 1 {
return false, errBoot
}
return true, nil
}
var errBoot = &bootErr{}
type bootErr struct{}
func (*bootErr) Error() string { return "oidc discovery: status 503" }
// A health endpoint that answers from a value captured at startup is not a
// health endpoint.
//
// newHealthzHandler marshalled its body ONCE and served the same bytes for the
// process lifetime. On 2026-09-14 gitea-mcp reported jwt.status=degraded for six
// days after Authentik recovered, because the answer had been computed before
// Authentik was up and could never change. The pod was 1/1 Running and
// readiness-green throughout.
//
// Fixing mcp-chassis#4 alone does not fix this: the validator would recover and
// /healthz would keep saying degraded.
func TestHealthz_ReflectsCurrentState_NotBootState(t *testing.T) {
f := &fakeReadiness{}
h := newHealthzHandler(true, f.ready)
first := decodeHealth(t, h)
require.Equal(t, "degraded", first.JWT.Status, "first call: issuer still down")
require.NotEmpty(t, first.JWT.LastError, "degraded must say why")
second := decodeHealth(t, h)
require.Equal(t, "enabled", second.JWT.Status,
"the issuer recovered; /healthz must notice without a restart")
require.Empty(t, second.JWT.LastError)
require.Equal(t, 2, f.calls, "readiness must be evaluated per request, not cached at startup")
}
// With no issuer configured, JWT auth is disabled by choice and readiness is
// never consulted — a static-token-only server must not be reported degraded.
func TestHealthz_NoIssuer_IsDisabledNotDegraded(t *testing.T) {
f := &fakeReadiness{}
h := newHealthzHandler(false, f.ready)
got := decodeHealth(t, h)
require.Equal(t, "disabled", got.JWT.Status)
require.Zero(t, f.calls, "no issuer configured means nothing to probe")
}
// readyz is what the kubelet probes. It must FAIL while JWT auth is degraded, so
// k8s stops routing to a pod that will 401 every OAuth caller — and so the
// probe's own cadence becomes the retry loop.
func TestReadyz_FailsWhileDegraded_ThenPasses(t *testing.T) {
f := &fakeReadiness{}
h := newReadyzHandler(true, f.ready)
rec := httptest.NewRecorder()
h(rec, httptest.NewRequest(http.MethodGet, "/readyz", nil))
require.Equal(t, http.StatusServiceUnavailable, rec.Code,
"a pod that cannot validate any JWT must not be in the Service")
rec = httptest.NewRecorder()
h(rec, httptest.NewRequest(http.MethodGet, "/readyz", nil))
require.Equal(t, http.StatusOK, rec.Code, "recovered without a restart")
}
// A static-token-only server is always ready. Making readyz depend on an issuer
// it was never given would take every static-only MCP out of its Service.
func TestReadyz_NoIssuer_AlwaysReady(t *testing.T) {
f := &fakeReadiness{}
h := newReadyzHandler(false, f.ready)
rec := httptest.NewRecorder()
h(rec, httptest.NewRequest(http.MethodGet, "/readyz", nil))
require.Equal(t, http.StatusOK, rec.Code)
require.Zero(t, f.calls)
}
func decodeHealth(t *testing.T, h http.HandlerFunc) healthStatus {
t.Helper()
rec := httptest.NewRecorder()
h(rec, httptest.NewRequest(http.MethodGet, "/healthz", nil))
var got healthStatus
require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &got))
return got
}