fix(auth): a validator whose issuer is down at boot is built anyway

Closes #4. Closes gitea-mcp#57.

Filed 2026-07-17, filed again 2026-09-07, recurred 2026-09-14. Each occurrence
ended with a human running `kubectl rollout restart`; the last one ran for six
days and was found by chance while gathering context for unrelated work.

THE BUG

NewMultiJWTValidator resolved OIDC discovery eagerly and returned (nil, err) if
the issuer was not answering. BearerMiddleware reads a nil validator as "this
server does not do JWT auth", so it fell through to the static-token comparison
and rejected every Authentik JWT:

    WARN mcp auth rejected reason=static_token_mismatch token_type=jwt status=401

Two states collapsed into one nil:

    static-token-only by design      -> nil   correct
    JWT required, IdP down at boot   -> nil   wrong, and permanent

Permanent because discovery was attempted exactly once, at construction.

WHY THE FIX IS SMALL

Everything downstream was already right. Validate distinguishes ErrUnavailable
(issuer unreachable -> 503, client retries) from a definitive rejection (-> 401),
and BearerMiddleware already honours that — gitea-mcp#6 got it right. That path
was simply unreachable when the validator was never built.

So: build it anyway. jwks_uri now resolves lazily on first use and re-resolves on
failure, no sooner than WithRetryInterval apart (default 30s) so a down issuer
costs one discovery request per interval, not one per request. Until it resolves,
Validate returns ErrUnavailable and the existing machinery does the rest. The
constructor still warms discovery eagerly on a healthy start; it just no longer
fails on it.

Ready(ctx) reports whether every issuer resolved, and attempts resolution for any
that have not. That makes a readiness probe the retry loop — no background
goroutine, no context to leak, and the platform's own cadence drives recovery.
A nil validator is Ready: "JWT auth disabled" is a configuration, not a fault.

WHAT DID NOT CHANGE

No issuer configured still returns (nil, nil). That is the one case where nil is
not a lie, and collapsing it would turn every static-only server into a JWT
server. Pinned by TestNewMultiJWTValidator_NoIssuer_StillReturnsNilNil.

ONE EXISTING TEST CHANGED ITS MECHANISM, NOT ITS CLAIM

TestNewMultiJWTValidator_UsesPerIssuerHTTPClientForAuthedFetch asserted that the
constructor ERRORS when discovery 401s without the fetch credential. It cannot,
now, by design. Rewritten to assert the property that actually mattered, and
more directly than before: without the credential the validator is not Ready AND
a correctly-minted token from that very issuer is refused as ErrUnavailable. The
old test only checked that a constructor returned an error.

Written test-first: jwt_bootrace_test.go red (API absent), then green. Also red
in the behavioural sense — the constructor-errors assertion above had to be
rewritten precisely because the semantics genuinely changed.

Verified: golangci-lint 0 issues; go vet clean; go test -race -count=1 ./... ok.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

.gitea/workflows/ci.yml

@@ -0,0 +1,30 @@
+name: CI
+
+# This repo had NO CI at all until 2026-09-05 -- no .gitea/workflows step ran
+# anything, so nothing was verified on push. Found by infra's task-properties
+# gate (scripts/check-task-properties.py), which asserts what CI RUNS rather
+# than what a repo contains.
+#
+# Verified before this was added, so turning verification on could not turn the
+# repo red: go build, go test and golangci-lint against the canonical
+# .golangci.yml all pass at 0 issues.
+
+on: [push]
+
+jobs:
+  check:
+    name: Lint / Test / Vet
+    runs-on: self-hosted
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with: { go-version: "1.26", cache: false } # self-hosted runner: Go cache persists on disk between runs (infra#360)
+      - name: Install golangci-lint
+        run: |
+          curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/HEAD/install.sh \
+            | sh -s -- -b "$(go env GOPATH)/bin" v2.11.4
+          golangci-lint --version
+
+      - run: golangci-lint run ./...
+      - run: go vet ./...
+      - run: CGO_ENABLED=1 go test -race -count=1 ./...

.golangci.yml

@@ -0,0 +1,52 @@
+# Canonical golangci-lint configuration for the estate's Go repos (infra#309
+# pattern: one canonical file here, a byte-identical copy in each adopter,
+# and the drift gate in THIS repo comparing sha256).
+#
+# Why this exists: 26 Go repos, 3 had a lint config, and all three were
+# different. The 23 without one ran golangci-lint's v2 default set, which
+# does NOT include a formatter -- so gofmt/gofumpt drift went uncaught until
+# it was found by hand (dma-cli#12: two files drifted, nothing flagged them).
+#
+# The two configs this replaces (cobalt-dingo, coo-agent) were near-identical
+# copies of each other AND both invalid under v2: they carried v1-style
+# `linters-settings` and `issues.exclude-rules` keys, which `golangci-lint
+# config verify` rejects outright. Their revive settings and test exclusions
+# were therefore not in effect. Consolidating fixes that too.
+#
+# Per-repo variation does NOT belong here -- this file must stay byte-identical
+# across adopters so the gate is a checksum, not a diff a human has to judge.
+# A repo with a genuine exception (agentsquad excludes its seeded-bug fixture
+# tree, which exists precisely to contain broken code) keeps its own config and
+# is not listed as an adopter.
+
+version: "2"
+
+linters:
+  default: none
+  enable:
+    - errcheck # unchecked errors
+    - govet # suspicious constructs
+    - staticcheck # broad static analysis
+    - unused # unused code
+    - misspell # spelling
+    - revive # idiomatic Go
+    - exhaustive # exhaustive switch on enums
+
+  settings:
+    revive:
+      rules:
+        - name: var-naming
+          severity: warning
+
+  exclusions:
+    rules:
+      # Test files: table-driven cases legitimately carry unused struct fields
+      # and long functions.
+      - path: _test\.go
+        linters:
+          - revive
+          - unused
+
+formatters:
+  enable:
+    - gofumpt

auth/bearer.go

@@ -1,7 +1,12 @@
 package auth
 
 import (
+	"crypto/sha256"
 	"crypto/subtle"
+	"encoding/hex"
+	"errors"
+	"log/slog"
+	"net"
 	"net/http"
 	"strings"
 )
@@ -44,7 +49,7 @@ func BearerMiddleware(
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		rawToken, ok := strings.CutPrefix(r.Header.Get("Authorization"), "Bearer ")
 		if !ok || rawToken == "" {
-			unauthorized(w, realm, resourceMetadataURL)
+			deny(w, r, realm, resourceMetadataURL, http.StatusUnauthorized, "no_token", rawToken)
 			return
 		}
 
@@ -57,21 +62,89 @@ func BearerMiddleware(
 
 		// 2. Then Dex JWT, if configured.
 		if validator != nil {
-			if _, err := validator.Validate(r.Context(), rawToken); err == nil {
+			switch _, err := validator.Validate(r.Context(), rawToken); {
+			case err == nil:
 				next.ServeHTTP(w, r)
 				return
+			case errors.Is(err, ErrUnavailable):
+				// Dex/JWKS is down — the token might be fine, we just can't
+				// check. Answer 503 so the caller retries rather than treating
+				// it as a hard auth failure (gitea-mcp#6).
+				deny(w, r, realm, resourceMetadataURL, http.StatusServiceUnavailable, "jwt_dex_unavailable", rawToken)
+				return
+			default:
+				deny(w, r, realm, resourceMetadataURL, http.StatusUnauthorized, "jwt_invalid", rawToken)
+				return
 			}
 		}
 
-		// 3. Reject with an OAuth resource-metadata challenge if configured.
-		unauthorized(w, realm, resourceMetadataURL)
+		// 3. Static token was wrong and no JWT validator is configured.
+		deny(w, r, realm, resourceMetadataURL, http.StatusUnauthorized, "static_token_mismatch", rawToken)
 	})
 }
 
-func unauthorized(w http.ResponseWriter, realm, resourceMetadataURL string) {
+// deny writes the auth rejection and emits a structured audit log line
+// (gitea-mcp#9). The log records the reason, client IP, presented token type,
+// and a non-reversible token fingerprint — never the raw token, so nothing
+// secret lands in stdout/log aggregation. slog adds the timestamp.
+func deny(w http.ResponseWriter, r *http.Request, realm, resourceMetadataURL string, status int, reason, rawToken string) {
+	slog.Warn("mcp auth rejected",
+		"reason", reason,
+		"status", status,
+		"remote", clientIP(r),
+		"token_type", tokenType(rawToken),
+		"token_fp", fingerprint(rawToken),
+		"path", r.URL.Path,
+	)
+
+	if status == http.StatusServiceUnavailable {
+		// RFC 6750 §3.1: temporarily_unavailable signals a transient failure.
+		w.Header().Set("WWW-Authenticate", `Bearer realm="`+realm+`", error="temporarily_unavailable"`)
+		http.Error(w, "temporarily unavailable", http.StatusServiceUnavailable)
+		return
+	}
+
 	if resourceMetadataURL != "" {
 		w.Header().Set("WWW-Authenticate",
 			`Bearer realm="`+realm+`", resource_metadata="`+resourceMetadataURL+`"`)
 	}
 	http.Error(w, "unauthorized", http.StatusUnauthorized)
 }
+
+// clientIP prefers the left-most X-Forwarded-For entry (the original client
+// behind the reverse proxy) and falls back to the transport peer.
+func clientIP(r *http.Request) string {
+	if xff := r.Header.Get("X-Forwarded-For"); xff != "" {
+		if i := strings.IndexByte(xff, ','); i > 0 {
+			return strings.TrimSpace(xff[:i])
+		}
+		return strings.TrimSpace(xff)
+	}
+	if host, _, err := net.SplitHostPort(r.RemoteAddr); err == nil {
+		return host
+	}
+	return r.RemoteAddr
+}
+
+// tokenType classifies the presented credential without revealing it: a
+// three-segment dotted value is a JWT, anything else an opaque/static token.
+func tokenType(raw string) string {
+	switch {
+	case raw == "":
+		return "none"
+	case strings.Count(raw, ".") == 2:
+		return "jwt"
+	default:
+		return "opaque"
+	}
+}
+
+// fingerprint is a short, non-reversible correlator (truncated SHA-256) so ops
+// can group repeated attempts from one client without ever logging token bytes.
+func fingerprint(raw string) string {
+	if raw == "" {
+		return ""
+	}
+	sum := sha256.Sum256([]byte(raw))
+	return hex.EncodeToString(sum[:])[:12]
+}

auth/bearer_test.go

@@ -1,13 +1,79 @@
 package auth
 
 import (
+	"bytes"
+	"context"
+	"log/slog"
 	"net/http"
 	"net/http/httptest"
 	"testing"
 
+	"github.com/lestrrat-go/jwx/v2/jwk"
+	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 )
 
+// unavailableValidator returns a validator whose JWKS cache cannot be fetched,
+// so Validate fails with ErrUnavailable — simulating a Dex/JWKS outage.
+func unavailableValidator(t *testing.T) *JWTValidator {
+	t.Helper()
+	return &JWTValidator{
+		entries: []*issuerEntry{{
+			issuer:  "https://dex.example",
+			jwksURI: "http://127.0.0.1:0/jwks", // already "resolved", unregistered in cache → Get fails
+		}},
+		cache: jwk.NewCache(context.Background()),
+	}
+}
+
+// #6: a JWKS/Dex outage at validation time is ErrUnavailable, distinct from a
+// present-but-invalid token.
+func TestValidate_JWKSUnreachable_IsUnavailable(t *testing.T) {
+	t.Parallel()
+	_, err := unavailableValidator(t).Validate(context.Background(), "some.jwt.token")
+	require.Error(t, err)
+	require.ErrorIs(t, err, ErrUnavailable)
+}
+
+// #6: when JWT validation can't complete because Dex is down, answer 503
+// temporarily_unavailable rather than a generic 401.
+func TestBearerMiddleware_DexDown_503(t *testing.T) {
+	t.Parallel()
+	h := BearerMiddleware("static-tok", unavailableValidator(t), "gitea", "",
+		http.HandlerFunc(func(http.ResponseWriter, *http.Request) {
+			t.Fatal("next must NOT be called when Dex is down")
+		}))
+	rec := httptest.NewRecorder()
+	req := httptest.NewRequest(http.MethodGet, "/", nil)
+	req.Header.Set("Authorization", "Bearer aaa.bbb.ccc") // not the static token → JWT path
+	h.ServeHTTP(rec, req)
+
+	require.Equal(t, http.StatusServiceUnavailable, rec.Code)
+	require.Contains(t, rec.Header().Get("WWW-Authenticate"), `error="temporarily_unavailable"`)
+}
+
+// #9: every rejection is audit-logged with reason, remote, token type, and a
+// non-reversible token fingerprint — never the raw token.
+func TestBearerMiddleware_AuditLogsRejection(t *testing.T) {
+	var buf bytes.Buffer
+	prev := slog.Default()
+	slog.SetDefault(slog.New(slog.NewJSONHandler(&buf, &slog.HandlerOptions{Level: slog.LevelInfo})))
+	defer slog.SetDefault(prev)
+
+	h := BearerMiddleware("expected-secret", nil, "gitea", "",
+		http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))
+	rec := httptest.NewRecorder()
+	req := httptest.NewRequest(http.MethodGet, "/", nil)
+	req.Header.Set("Authorization", "Bearer wrong-secret-value")
+	h.ServeHTTP(rec, req)
+
+	require.Equal(t, http.StatusUnauthorized, rec.Code)
+	logged := buf.String()
+	assert.Contains(t, logged, "static_token_mismatch")
+	assert.Contains(t, logged, "token_fp")
+	assert.NotContains(t, logged, "wrong-secret-value", "raw token must never be logged")
+}
+
 func TestBearerMiddleware_StaticTokenWins(t *testing.T) {
 	t.Parallel()
 

auth/jwt.go

@@ -1,116 +1,314 @@
-// Package auth provides the Dex-JWT + static-Bearer authentication primitives
-// shared by every Mathias-owned MCP server (gitea-mcp, brain-mcp / ingestion,
-// future MCPs spawned from template-go-agent).
+// Package auth provides the Dex/Authentik-JWT + static-Bearer authentication
+// primitives shared by every Mathias-owned MCP server (gitea-mcp, brain-mcp /
+// ingestion, future MCPs spawned from template-go-agent).
 //
 // Replaces ~80 LOC of near-identical jwt.go in each consumer, ~50 LOC of
 // Bearer middleware, and ~25 LOC of RFC 9728 protected-resource metadata
 // handler. See `gitea.d-ma.be/mathias/infra` docs/superpowers/handoffs/
 // 2026-05-22-mcp-chassis-spike.md for the design rationale and the
 // abort-criterion check.
+//
+// A validator trusts a LIST of OIDC issuers (infra ADR-0011): one MCP server can
+// accept Authentik JWTs (interactive/web) AND k3s cluster ServiceAccount tokens
+// (in-cluster, audience-bound, kubelet-rotated) at once.
 package auth
 
 import (
 	"context"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"net/http"
+	"sync"
 	"time"
 
 	"github.com/lestrrat-go/jwx/v2/jwk"
 	"github.com/lestrrat-go/jwx/v2/jwt"
 )
 
-// JWTValidator validates Bearer JWTs issued by a Dex (OIDC) authorization server.
-// Audience is optional; leave empty to skip audience validation.
-//
-// A nil *JWTValidator behaves as "JWT auth disabled" — Validate returns an
-// error without panicking. Callers can construct one validator at startup
-// keyed on whether DEX_ISSUER_URL is set, and pass nil through the rest of
-// the codebase without further branching.
-type JWTValidator struct {
-	issuer   string
-	audience string
-	jwksURI  string
-	cache    *jwk.Cache
+// ErrUnavailable indicates JWT validation could not be COMPLETED because the
+// JWKS / issuer endpoint was unreachable — a transient condition — as opposed to
+// the token being present and invalid. Callers (e.g. BearerMiddleware) map it
+// to HTTP 503 temporarily_unavailable rather than a generic 401, so an issuer
+// outage is distinguishable from a bad token (gitea-mcp#6).
+var ErrUnavailable = errors.New("jwt validation temporarily unavailable")
+
+// IssuerConfig names one trusted OIDC issuer and its (optional) required
+// audience. Trusting a list of these is what lets a single MCP server accept
+// tokens from Authentik (D3/D4) and the k3s cluster OIDC issuer (D1 SA tokens)
+// simultaneously — infra ADR-0011 Decision 5.
+type IssuerConfig struct {
+	IssuerURL string
+	Audience  string // "" = skip audience validation for this issuer
+	// HTTPClient, if non-nil, is used to fetch THIS issuer's OIDC discovery
+	// document and JWKS. Supply one to reach an issuer that needs a custom CA
+	// and/or credential — notably the in-cluster k8s OIDC endpoint, which serves
+	// its discovery/JWKS over the cluster CA AND requires an authenticated
+	// (ServiceAccount-bearer) request (anonymous access is 401). nil uses
+	// http.DefaultClient — correct for public issuers like Authentik.
+	HTTPClient *http.Client
 }
 
-// NewJWTValidator fetches the OIDC discovery document from issuerURL,
-// extracts jwks_uri, warms the JWKS cache, and returns a ready validator.
-// Empty issuerURL returns (nil, nil) so callers can use a single
-// constructor regardless of whether Dex is configured.
-func NewJWTValidator(ctx context.Context, issuerURL, audience string) (*JWTValidator, error) {
+// issuerEntry is one trusted issuer. jwksURI is resolved LAZILY: OIDC discovery
+// may not be answerable when the process starts, and an issuer that is merely
+// not-yet-serving must not become a permanent auth outage (mcp-chassis#4,
+// gitea-mcp#57 — three recurrences, each ended by a human running
+// `kubectl rollout restart`).
+//
+// Everything downstream of resolution was already correct. Validate reports
+// ErrUnavailable when it cannot reach an issuer, and BearerMiddleware already
+// answers 503-and-retry for that. Resolving eagerly in the constructor was the
+// one step that made the correct path unreachable.
+type issuerEntry struct {
+	issuer     string
+	audience   string
+	httpClient *http.Client
+
+	mu          sync.Mutex
+	jwksURI     string    // "" until discovery has succeeded
+	lastAttempt time.Time // zero until the first attempt
+	lastErr     error
+}
+
+// resolve returns this issuer's jwks_uri, running OIDC discovery if it has not
+// succeeded yet. Failures are retried, no sooner than retryInterval apart, so a
+// down issuer costs one discovery request per interval rather than one per
+// inbound request.
+func (e *issuerEntry) resolve(ctx context.Context, cache *jwk.Cache, retryInterval, refreshInterval time.Duration) (string, error) {
+	e.mu.Lock()
+	defer e.mu.Unlock()
+
+	if e.jwksURI != "" {
+		return e.jwksURI, nil
+	}
+	if !e.lastAttempt.IsZero() && time.Since(e.lastAttempt) < retryInterval {
+		// Back off, but report the real reason rather than a generic one — the
+		// operator needs the issuer's actual error, not "throttled".
+		return "", e.lastErr
+	}
+	e.lastAttempt = time.Now()
+
+	client := e.httpClient
+	if client == nil {
+		client = http.DefaultClient
+	}
+	jwksURI, err := discoverJWKSURI(ctx, client, e.issuer)
+	if err != nil {
+		e.lastErr = fmt.Errorf("%w: issuer %s: %v", ErrUnavailable, e.issuer, err)
+		return "", e.lastErr
+	}
+
+	regOpts := []jwk.RegisterOption{jwk.WithMinRefreshInterval(refreshInterval)}
+	if e.httpClient != nil {
+		regOpts = append(regOpts, jwk.WithHTTPClient(e.httpClient))
+	}
+	if err := cache.Register(jwksURI, regOpts...); err != nil {
+		e.lastErr = fmt.Errorf("%w: register jwks cache (%s): %v", ErrUnavailable, e.issuer, err)
+		return "", e.lastErr
+	}
+	if _, err := cache.Refresh(ctx, jwksURI); err != nil {
+		// Registered but unfetchable. Unregister so the next attempt re-runs
+		// discovery cleanly rather than inheriting a half-built entry.
+		_ = cache.Unregister(jwksURI)
+		e.lastErr = fmt.Errorf("%w: initial jwks fetch (%s): %v", ErrUnavailable, e.issuer, err)
+		return "", e.lastErr
+	}
+
+	e.jwksURI, e.lastErr = jwksURI, nil
+	return jwksURI, nil
+}
+
+// Option configures a validator. Variadic so every existing caller compiles
+// unchanged — a shared library cannot afford a breaking signature for this.
+type Option func(*options)
+
+type options struct {
+	retryInterval   time.Duration
+	refreshInterval time.Duration
+}
+
+// defaultRetryInterval bounds how often a down issuer is re-probed. A readiness
+// probe typically runs every 10s, so recovery lands within one interval of the
+// issuer coming back — without turning an IdP outage into a discovery flood.
+const defaultRetryInterval = 30 * time.Second
+
+// defaultRefreshInterval is the JWKS cache's minimum refresh interval. 1h
+// preserves the value that was hardcoded here before options existed; making it
+// configurable is mcp-chassis#1.
+const defaultRefreshInterval = time.Hour
+
+// WithRetryInterval sets how long to wait before re-attempting OIDC discovery
+// for an issuer that is not yet resolved. Zero disables the wait, which is for
+// tests; production wants the default.
+func WithRetryInterval(d time.Duration) Option {
+	return func(o *options) { o.retryInterval = d }
+}
+
+// JWTValidator validates Bearer JWTs against one or more trusted OIDC issuers.
+//
+// A nil *JWTValidator behaves as "JWT auth disabled" — Validate returns an
+// error without panicking. Callers can construct one validator at startup keyed
+// on whether any issuer is configured, and pass nil through the rest of the
+// codebase without further branching.
+type JWTValidator struct {
+	entries []*issuerEntry
+	cache   *jwk.Cache
+	opts    options
+}
+
+// NewJWTValidator builds a single-issuer validator — the common case, and
+// backward compatible with every existing caller. Empty issuerURL returns
+// (nil, nil) so callers can use one constructor regardless of whether an issuer
+// is configured.
+func NewJWTValidator(ctx context.Context, issuerURL, audience string, opts ...Option) (*JWTValidator, error) {
 	if issuerURL == "" {
 		return nil, nil
 	}
+	return NewMultiJWTValidator(ctx, []IssuerConfig{{IssuerURL: issuerURL, Audience: audience}}, opts...)
+}
 
+// NewMultiJWTValidator builds a validator that trusts every issuer in the list.
+// For each, it fetches the OIDC discovery document, registers jwks_uri in a
+// shared cache, and warms it. Entries with an empty IssuerURL are skipped; an
+// empty/nil resulting set returns (nil, nil) — "JWT auth disabled".
+func NewMultiJWTValidator(ctx context.Context, issuers []IssuerConfig, opts ...Option) (*JWTValidator, error) {
+	o := options{retryInterval: defaultRetryInterval, refreshInterval: defaultRefreshInterval}
+	for _, fn := range opts {
+		fn(&o)
+	}
+
+	var entries []*issuerEntry
+	for _, ic := range issuers {
+		if ic.IssuerURL == "" {
+			continue
+		}
+		entries = append(entries, &issuerEntry{
+			issuer:     ic.IssuerURL,
+			audience:   ic.Audience,
+			httpClient: ic.HTTPClient,
+		})
+	}
+	if len(entries) == 0 {
+		// No issuer configured: JWT auth is genuinely disabled, and nil is the
+		// right answer. This is the ONE case where nil is not a lie, which is
+		// why the boot-race fix must not collapse it — see
+		// TestNewMultiJWTValidator_NoIssuer_StillReturnsNilNil.
+		return nil, nil
+	}
+
+	v := &JWTValidator{entries: entries, cache: jwk.NewCache(ctx), opts: o}
+
+	// Try once, eagerly, so a healthy start still warms the cache before the
+	// first request — but do NOT fail on it. A validator that exists and is
+	// temporarily unresolved is strictly better than no validator: the first
+	// answers 503-and-retry, the second answers 401 forever.
+	for _, e := range v.entries {
+		_, _ = e.resolve(ctx, v.cache, 0, o.refreshInterval)
+	}
+	return v, nil
+}
+
+// Ready reports whether every trusted issuer has been resolved, attempting
+// resolution for any that have not. It is safe and intended to be called from a
+// readiness probe: the probe's own cadence then drives retry, so recovery needs
+// no background goroutine and no process restart.
+//
+// A nil validator is ready — "JWT auth disabled" is a configuration, not a
+// fault.
+func (v *JWTValidator) Ready(ctx context.Context) (bool, error) {
+	if v == nil {
+		return true, nil
+	}
+	for _, e := range v.entries {
+		if _, err := e.resolve(ctx, v.cache, v.opts.retryInterval, v.opts.refreshInterval); err != nil {
+			return false, err
+		}
+	}
+	return true, nil
+}
+
+// discoverJWKSURI fetches the OIDC discovery document from issuerURL and returns
+// its jwks_uri.
+func discoverJWKSURI(ctx context.Context, client *http.Client, issuerURL string) (string, error) {
 	req, err := http.NewRequestWithContext(ctx, http.MethodGet,
 		issuerURL+"/.well-known/openid-configuration", nil)
 	if err != nil {
-		return nil, fmt.Errorf("build oidc discovery request: %w", err)
+		return "", fmt.Errorf("build oidc discovery request: %w", err)
 	}
-	resp, err := http.DefaultClient.Do(req)
+	resp, err := client.Do(req)
 	if err != nil {
-		return nil, fmt.Errorf("fetch oidc discovery: %w", err)
+		return "", fmt.Errorf("fetch oidc discovery: %w", err)
 	}
 	defer func() { _ = resp.Body.Close() }()
 	if resp.StatusCode != http.StatusOK {
-		return nil, fmt.Errorf("oidc discovery: status %d", resp.StatusCode)
+		return "", fmt.Errorf("oidc discovery: status %d", resp.StatusCode)
 	}
 
 	var doc struct {
 		JWKSURI string `json:"jwks_uri"`
 	}
 	if err := json.NewDecoder(resp.Body).Decode(&doc); err != nil {
-		return nil, fmt.Errorf("decode oidc discovery: %w", err)
+		return "", fmt.Errorf("decode oidc discovery: %w", err)
 	}
 	if doc.JWKSURI == "" {
-		return nil, fmt.Errorf("oidc discovery: empty jwks_uri")
+		return "", fmt.Errorf("oidc discovery: empty jwks_uri")
 	}
-
-	cache := jwk.NewCache(ctx)
-	if err := cache.Register(doc.JWKSURI, jwk.WithMinRefreshInterval(time.Hour)); err != nil {
-		return nil, fmt.Errorf("register jwks cache: %w", err)
-	}
-	if _, err := cache.Refresh(ctx, doc.JWKSURI); err != nil {
-		return nil, fmt.Errorf("initial jwks fetch: %w", err)
-	}
-
-	return &JWTValidator{
-		issuer:   issuerURL,
-		audience: audience,
-		jwksURI:  doc.JWKSURI,
-		cache:    cache,
-	}, nil
+	return doc.JWKSURI, nil
 }
 
-// Validate parses and validates rawToken against the OIDC issuer. Returns
-// the subject claim on success. A nil receiver returns
-// (`""`, errDisabled) so callers can dispatch on `err != nil` without
-// nil-checks at every call site.
+// Validate parses rawToken against each trusted issuer and returns the subject
+// claim on the first success. If no issuer accepts it: when EVERY issuer's JWKS
+// was unreachable the error wraps ErrUnavailable (caller answers 503); otherwise
+// at least one issuer reached a signature/claims verdict and rejected it, so the
+// error is a definitive rejection (401). A nil receiver returns errDisabled.
 func (v *JWTValidator) Validate(ctx context.Context, rawToken string) (string, error) {
 	if v == nil {
 		return "", errDisabled
 	}
 
-	keySet, err := v.cache.Get(ctx, v.jwksURI)
-	if err != nil {
-		return "", fmt.Errorf("get jwks: %w", err)
+	var lastErr error
+	sawDecision := false // at least one issuer reached a real verify verdict
+	for _, e := range v.entries {
+		jwksURI, err := e.resolve(ctx, v.cache, v.opts.retryInterval, v.opts.refreshInterval)
+		if err != nil {
+			// Not resolved yet — transient by construction (resolve wraps
+			// ErrUnavailable). Try the next issuer.
+			lastErr = err
+			continue
+		}
+		keySet, err := v.cache.Get(ctx, jwksURI)
+		if err != nil {
+			// This issuer's JWKS is unreachable — transient. Try the next; only
+			// if ALL issuers are unreachable do we surface ErrUnavailable.
+			lastErr = fmt.Errorf("%w: get jwks (%s): %v", ErrUnavailable, e.issuer, err)
+			continue
+		}
+
+		opts := []jwt.ParseOption{
+			jwt.WithKeySet(keySet),
+			jwt.WithValidate(true),
+			jwt.WithIssuer(e.issuer),
+		}
+		if e.audience != "" {
+			opts = append(opts, jwt.WithAudience(e.audience))
+		}
+
+		tok, err := jwt.ParseString(rawToken, opts...)
+		if err == nil {
+			return tok.Subject(), nil
+		}
+		sawDecision = true
+		lastErr = fmt.Errorf("validate jwt: %w", err)
 	}
 
-	opts := []jwt.ParseOption{
-		jwt.WithKeySet(keySet),
-		jwt.WithValidate(true),
-		jwt.WithIssuer(v.issuer),
+	if lastErr == nil {
+		lastErr = errDisabled
 	}
-	if v.audience != "" {
-		opts = append(opts, jwt.WithAudience(v.audience))
+	if !sawDecision {
+		// Every issuer's JWKS was unreachable — lastErr already wraps ErrUnavailable.
+		return "", lastErr
 	}
-
-	tok, err := jwt.ParseString(rawToken, opts...)
-	if err != nil {
-		return "", fmt.Errorf("validate jwt: %w", err)
-	}
-	return tok.Subject(), nil
+	return "", lastErr
 }
 
 // errDisabled is the sentinel returned by Validate on a nil receiver.

auth/jwt_bootrace_test.go

@@ -0,0 +1,115 @@
+package auth
+
+import (
+	"context"
+	"errors"
+	"testing"
+
+	"github.com/stretchr/testify/require"
+)
+
+// The boot race, pinned.
+//
+// WHAT HAPPENED, THREE TIMES
+// gitea-mcp starts before Authentik is serving. OIDC discovery returns 502/503.
+// The constructor errors, the caller is left holding nil, and `nil` means "this
+// server does not do JWT auth" — so every subsequent Authentik JWT is
+// string-compared against the static token and rejected:
+//
+//	WARN mcp auth rejected reason=static_token_mismatch token_type=jwt status=401
+//
+// Permanently, because discovery was attempted exactly once. The pod stays 1/1
+// Running and readiness-green throughout.
+//
+//	2026-07-17  gitea-mcp#57  ~1 debugging session
+//	2026-09-07  mcp-chassis#4 ~16 hours
+//	2026-09-14  (this)        ~6 days, found by chance
+//
+// Each recovery was a human running `kubectl rollout restart`.
+//
+// WHY THE FIX IS SMALL
+// Everything after construction is already correct. `Validate` distinguishes
+// ErrUnavailable (issuer unreachable → 503, retry) from a definitive rejection
+// (→ 401), and BearerMiddleware already honours that. The correct behaviour was
+// unreachable for this case only because the validator never got built.
+//
+// So: build it anyway, resolve discovery lazily, and let the existing machinery
+// do what it already does.
+
+// A validator must be constructible while its issuer is down, must report
+// ErrUnavailable rather than a rejection while it stays down, and must start
+// working when the issuer returns — without a restart.
+func TestNewMultiJWTValidator_SurvivesIssuerDownAtBoot(t *testing.T) {
+	ctx := context.Background()
+	iss := newTestIssuer(t)
+	// Three, because three resolution attempts happen before recovery is
+	// asserted: the constructor's eager warm-up, the Validate below, and the
+	// Ready below. Each must see the issuer still down.
+	iss.discoveryFailures = 3
+
+	// 1. Construction succeeds despite discovery being down. This is the whole
+	//    bug: today this returns (nil, err) and the caller degrades silently.
+	v, err := NewMultiJWTValidator(ctx,
+		[]IssuerConfig{{IssuerURL: iss.url, Audience: "test-aud"}},
+		WithRetryInterval(0), // no suppression, so the test drives the retries
+	)
+	require.NoError(t, err, "a validator whose issuer is down at boot must still be constructed")
+	require.NotNil(t, v, "nil would be indistinguishable from 'JWT auth disabled'")
+
+	// 2. While down, a well-formed token is UNAVAILABLE, never a rejection.
+	//    BearerMiddleware turns this into 503 + retry; a rejection would become
+	//    401 static_token_mismatch, which is the lie that cost three sessions.
+	tok := iss.mint(t, "test-aud", "alice")
+	_, err = v.Validate(ctx, tok)
+	require.Error(t, err)
+	require.ErrorIs(t, err, ErrUnavailable,
+		"issuer down must be transient, not a verdict on the token")
+
+	// 3. The server knows it is degraded, so it can stop calling itself healthy.
+	ready, why := v.Ready(ctx)
+	require.False(t, ready)
+	require.Error(t, why, "Ready must say WHY, so /healthz can report it")
+
+	// 4. The issuer recovers. The SAME validator starts working — no restart,
+	//    no rollout, no human.
+	require.GreaterOrEqual(t, iss.discoveryAttempts(), 1, "discovery must be retried, not cached as failed")
+	sub, err := v.Validate(ctx, tok)
+	require.NoError(t, err, "a recovered issuer must work without reconstructing the validator")
+	require.Equal(t, "alice", sub)
+
+	ready, why = v.Ready(ctx)
+	require.True(t, ready)
+	require.NoError(t, why)
+}
+
+// Regression guard. A server that genuinely wants static-token-only auth passes
+// no issuer, and must keep getting nil — that is what makes BearerMiddleware
+// report static_token_mismatch correctly for it. Fixing the boot race must not
+// turn every static-only server into a JWT server.
+func TestNewMultiJWTValidator_NoIssuer_StillReturnsNilNil(t *testing.T) {
+	v, err := NewMultiJWTValidator(context.Background(), []IssuerConfig{{IssuerURL: ""}})
+	require.NoError(t, err)
+	require.Nil(t, v, "no issuer configured still means JWT auth disabled")
+}
+
+// A permanently bad issuer must not look like a transient one forever in a way
+// that hides a config error — but it must still be ErrUnavailable rather than a
+// token verdict, because the token was never the problem. The distinction that
+// matters is the one Ready() reports.
+func TestJWTValidator_UnresolvableIssuer_IsUnavailableAndNotReady(t *testing.T) {
+	ctx := context.Background()
+	v, err := NewMultiJWTValidator(ctx,
+		[]IssuerConfig{{IssuerURL: "http://127.0.0.1:1/nonexistent"}},
+		WithRetryInterval(0),
+	)
+	require.NoError(t, err)
+	require.NotNil(t, v)
+
+	_, err = v.Validate(ctx, "not.a.real.token")
+	require.True(t, errors.Is(err, ErrUnavailable),
+		"an unreachable issuer is unavailable, whatever the token looks like")
+
+	ready, why := v.Ready(ctx)
+	require.False(t, ready)
+	require.Error(t, why)
+}

auth/jwt_test.go

@@ -0,0 +1,238 @@
+package auth
+
+import (
+	"context"
+	"crypto/rand"
+	"crypto/rsa"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/lestrrat-go/jwx/v2/jwa"
+	"github.com/lestrrat-go/jwx/v2/jwk"
+	"github.com/lestrrat-go/jwx/v2/jwt"
+	"github.com/stretchr/testify/require"
+)
+
+// testIssuer is an in-process OIDC issuer: it serves an openid-configuration
+// discovery doc + a JWKS, and mints signed JWTs — enough to exercise the real
+// signature/issuer/audience validation path (which the chassis previously had
+// no happy-path test for). Used to prove multi-issuer validation (ADR-0011).
+type testIssuer struct {
+	url           string
+	priv          jwk.Key
+	requireBearer string // if set, discovery+JWKS return 401 without this bearer
+
+	// mu guards the boot-race fields below, which a test mutates while the
+	// httptest server is serving.
+	mu sync.Mutex
+	// discoveryFailures is how many initial discovery requests answer 503 —
+	// the IdP-not-yet-serving case that disabled OAuth for six days
+	// (mcp-chassis#4, gitea-mcp#57).
+	discoveryFailures int
+	discoveryHits     int
+}
+
+func (ti *testIssuer) authed(r *http.Request) bool {
+	return ti.requireBearer == "" || r.Header.Get("Authorization") == "Bearer "+ti.requireBearer
+}
+
+// discoveryDown reports whether this discovery request should answer 503, and
+// counts the attempt. Attempts are counted even while down, so a test can assert
+// that resolution was RETRIED rather than cached as a permanent failure.
+func (ti *testIssuer) discoveryDown() bool {
+	ti.mu.Lock()
+	defer ti.mu.Unlock()
+	ti.discoveryHits++
+	return ti.discoveryHits <= ti.discoveryFailures
+}
+
+func (ti *testIssuer) discoveryAttempts() int {
+	ti.mu.Lock()
+	defer ti.mu.Unlock()
+	return ti.discoveryHits
+}
+
+// bearerRT adds a static Authorization header to every request — a minimal stand-in
+// for the in-cluster case where the JWKS fetch carries the pod's SA token.
+type bearerRT struct{ token string }
+
+func (b bearerRT) RoundTrip(r *http.Request) (*http.Response, error) {
+	r.Header.Set("Authorization", "Bearer "+b.token)
+	return http.DefaultTransport.RoundTrip(r)
+}
+
+func newTestIssuer(t *testing.T) *testIssuer {
+	t.Helper()
+	raw, err := rsa.GenerateKey(rand.Reader, 2048)
+	require.NoError(t, err)
+	priv, err := jwk.FromRaw(raw)
+	require.NoError(t, err)
+	require.NoError(t, priv.Set(jwk.KeyIDKey, "test-kid"))
+	require.NoError(t, priv.Set(jwk.AlgorithmKey, jwa.RS256))
+	pub, err := priv.PublicKey()
+	require.NoError(t, err)
+	set := jwk.NewSet()
+	require.NoError(t, set.AddKey(pub))
+
+	ti := &testIssuer{priv: priv}
+	mux := http.NewServeMux()
+	mux.HandleFunc("/jwks", func(w http.ResponseWriter, r *http.Request) {
+		if !ti.authed(r) {
+			w.WriteHeader(http.StatusUnauthorized)
+			return
+		}
+		_ = json.NewEncoder(w).Encode(set)
+	})
+	mux.HandleFunc("/.well-known/openid-configuration", func(w http.ResponseWriter, r *http.Request) {
+		if ti.discoveryDown() {
+			w.WriteHeader(http.StatusServiceUnavailable)
+			return
+		}
+		if !ti.authed(r) {
+			w.WriteHeader(http.StatusUnauthorized)
+			return
+		}
+		_ = json.NewEncoder(w).Encode(map[string]string{
+			"issuer":   ti.url,
+			"jwks_uri": ti.url + "/jwks",
+		})
+	})
+	srv := httptest.NewServer(mux)
+	t.Cleanup(srv.Close)
+	ti.url = srv.URL
+	return ti
+}
+
+func (ti *testIssuer) mint(t *testing.T, aud, sub string) string {
+	t.Helper()
+	tok, err := jwt.NewBuilder().
+		Issuer(ti.url).
+		Subject(sub).
+		Audience([]string{aud}).
+		IssuedAt(time.Now()).
+		Expiration(time.Now().Add(time.Hour)).
+		Build()
+	require.NoError(t, err)
+	signed, err := jwt.Sign(tok, jwt.WithKey(jwa.RS256, ti.priv))
+	require.NoError(t, err)
+	return string(signed)
+}
+
+// The core ADR-0011 claim: one validator, several trusted issuers (e.g.
+// Authentik + the k3s cluster OIDC), a token from any trusted issuer validates.
+func TestNewMultiJWTValidator_AcceptsEitherTrustedIssuer(t *testing.T) {
+	ctx := context.Background()
+	a := newTestIssuer(t)
+	b := newTestIssuer(t)
+
+	v, err := NewMultiJWTValidator(ctx, []IssuerConfig{
+		{IssuerURL: a.url, Audience: "brain-mcp"},
+		{IssuerURL: b.url, Audience: "brain-mcp"},
+	})
+	require.NoError(t, err)
+	require.NotNil(t, v)
+
+	subA, err := v.Validate(ctx, a.mint(t, "brain-mcp", "alice"))
+	require.NoError(t, err)
+	require.Equal(t, "alice", subA)
+
+	subB, err := v.Validate(ctx, b.mint(t, "brain-mcp", "bob"))
+	require.NoError(t, err)
+	require.Equal(t, "bob", subB)
+}
+
+// A token from an issuer NOT in the trusted list is rejected — definitively
+// (401), not as a transient JWKS outage (503).
+func TestNewMultiJWTValidator_RejectsUntrustedIssuer(t *testing.T) {
+	ctx := context.Background()
+	trusted := newTestIssuer(t)
+	untrusted := newTestIssuer(t)
+
+	v, err := NewMultiJWTValidator(ctx, []IssuerConfig{{IssuerURL: trusted.url, Audience: "brain-mcp"}})
+	require.NoError(t, err)
+
+	_, err = v.Validate(ctx, untrusted.mint(t, "brain-mcp", "eve"))
+	require.Error(t, err)
+	require.NotErrorIs(t, err, ErrUnavailable)
+}
+
+// Audience is enforced per issuer — the k8s-SA-token replay guard from the ADR
+// spike. A token minted for a different audience is rejected.
+func TestNewMultiJWTValidator_EnforcesPerIssuerAudience(t *testing.T) {
+	ctx := context.Background()
+	a := newTestIssuer(t)
+
+	v, err := NewMultiJWTValidator(ctx, []IssuerConfig{{IssuerURL: a.url, Audience: "brain-mcp"}})
+	require.NoError(t, err)
+
+	_, err = v.Validate(ctx, a.mint(t, "some-other-service", "alice"))
+	require.Error(t, err)
+}
+
+// Backward compatibility: the existing single-issuer constructor still works and
+// validates a real signed token end-to-end.
+func TestNewJWTValidator_SingleIssuer_BackwardCompatible(t *testing.T) {
+	ctx := context.Background()
+	a := newTestIssuer(t)
+
+	v, err := NewJWTValidator(ctx, a.url, "brain-mcp")
+	require.NoError(t, err)
+	require.NotNil(t, v)
+
+	sub, err := v.Validate(ctx, a.mint(t, "brain-mcp", "carol"))
+	require.NoError(t, err)
+	require.Equal(t, "carol", sub)
+}
+
+func TestNewMultiJWTValidator_EmptyList_ReturnsNilNil(t *testing.T) {
+	v, err := NewMultiJWTValidator(context.Background(), nil)
+	require.NoError(t, err)
+	require.Nil(t, v)
+}
+
+// The in-cluster k8s issuer requires an AUTHENTICATED discovery/JWKS fetch
+// (anonymous is 401, confirmed against k3s). IssuerConfig.HTTPClient must carry
+// that credential (the server pod's SA token) + trust the cluster CA.
+func TestNewMultiJWTValidator_UsesPerIssuerHTTPClientForAuthedFetch(t *testing.T) {
+	ctx := context.Background()
+	iss := newTestIssuer(t)
+	iss.requireBearer = "fetch-secret" // discovery + JWKS now demand this bearer
+
+	// nil client (http.DefaultClient, no credential) → discovery 401 → the issuer
+	// never resolves.
+	//
+	// This used to assert that the CONSTRUCTOR errored. It no longer does: since
+	// mcp-chassis#4, construction cannot fail on an unreachable issuer, because
+	// "could not resolve" and "JWT auth not wanted" must stop collapsing into the
+	// same nil. The property that actually matters is unchanged and asserted more
+	// directly here — without the credential, a correctly-minted token from this
+	// very issuer still gets nowhere, and the validator says it is not ready.
+	unauth, err := NewMultiJWTValidator(ctx,
+		[]IssuerConfig{{IssuerURL: iss.url, Audience: "sa-poc"}},
+		WithRetryInterval(0),
+	)
+	require.NoError(t, err)
+	require.NotNil(t, unauth)
+
+	ready, why := unauth.Ready(ctx)
+	require.False(t, ready, "an issuer whose discovery 401s is not ready")
+	require.Error(t, why)
+
+	_, err = unauth.Validate(ctx, iss.mint(t, "sa-poc", "poc-pod"))
+	require.Error(t, err, "unauthenticated fetch must authenticate nobody, even with a genuine token")
+	require.ErrorIs(t, err, ErrUnavailable)
+
+	// A client that presents the fetch credential → build succeeds, token validates.
+	client := &http.Client{Transport: bearerRT{token: "fetch-secret"}}
+	v, err := NewMultiJWTValidator(ctx, []IssuerConfig{
+		{IssuerURL: iss.url, Audience: "sa-poc", HTTPClient: client},
+	})
+	require.NoError(t, err)
+	sub, err := v.Validate(ctx, iss.mint(t, "sa-poc", "poc-pod"))
+	require.NoError(t, err)
+	require.Equal(t, "poc-pod", sub)
+}