Found 2026-09-07 debugging why the claude.ai gitea connector could not authenticate. gitea-mcp had been rejecting every OAuth JWT for ~16 hours. The chassis behaved exactly as written; the gap is that "the validator could not be built" is indistinguishable from "no JWT auth wanted".
What happened
koala rebooted. gitea-mcp started before Authentik was serving:
05:07:46 WARN multi-issuer init failed; retrying without the k8s issuer
err="issuer https://auth.d-ma.be/application/o/claude-ai/: oidc discovery: status 502"
05:07:46 WARN jwt validator init failed; JWT auth degraded
05:07:46 INFO gitea-mcp starting addr=:8080 version=2bb1fef
Then, for every subsequent request, ~370 of these:
Authentik was healthy within minutes. gitea-mcp never noticed. A kubectl rollout restart fixed it instantly — /healthz went from jwt.status=degraded to jwt=enabled.
Why the chassis produced that message
auth/bearer.go is correct and its own comment says so:
ifvalidator!=nil{switch_,err:=validator.Validate(r.Context(),rawToken);{caseerr==nil:// allowcaseerrors.Is(err,ErrUnavailable):// Dex/JWKS is down — the token might be fine, we just can't// check. Answer 503 so the caller retries rather than treating// it as a hard auth failure (gitea-mcp#6).deny(...,http.StatusServiceUnavailable,"jwt_dex_unavailable",...)default:deny(...,http.StatusUnauthorized,"jwt_invalid",...)}}// 3. Static token was wrong and no JWT validator is configured.deny(...,http.StatusUnauthorized,"static_token_mismatch",...)
ErrUnavailable already encodes precisely the right idea — we cannot check, so 503 and retry — and gitea-mcp#6 got that right. But it only applies to a validator that exists.
NewJWTValidator / NewMultiJWTValidator (auth/jwt.go:74, :85) resolve discovery eagerly at construction: discoverJWKSURI returns oidc discovery: status 502, the constructor errors, and the caller is left holding nil. From the middleware's perspective nil means "this server does not do JWT auth", so it falls through to the static comparison and reports static_token_mismatch.
Two states collapse into one:
intent
validator
middleware sees
static-token-only server, by design
nil
correct
JWT required, IdP was down at boot
nil
wrong — reports as static-only, forever
The failure is permanent because discovery is attempted exactly once, at construction.
Proposed fix — in the chassis, because every MCP server here shares it
1. Make the validator constructible without successful discovery. Resolve jwks_uri lazily on first use, and re-resolve on failure with backoff. Until resolution succeeds, Validate returns ErrUnavailable — which the middleware already turns into a 503 the client retries. A boot-time 502 then self-heals within seconds instead of requiring a human and a rollout restart.
This needs no change to bearer.go. The correct behaviour is already there; it is currently unreachable for this case.
2. Do not let a nil validator silently mean "static only" when an issuer was configured. Either a constructor variant that cannot fail (per 1), or an explicit RequireJWT flag so the middleware can answer 503 rather than 401 when JWT auth is required but unavailable. Silently downgrading an auth mode is the wrong default.
3. Expose the validator's readiness so servers can report it./healthz returned {"ok":true,"jwt":{"status":"degraded",...}} — the process knew it was broken and still called itself healthy. That is gitea-mcp's handler, but the chassis owns the state it needs, so a Validator.Ready() (bool, error) (or similar) would let every server get this right the same way. A component that knows it is degraded should not report itself healthy.
Blast radius
Every MCP server in the estate uses chassisauth.BearerMiddleware — gitea-mcp, brain-mcp, infra-mcp. All share this failure mode. On this reboot brain-mcp happened to win the race and gitea-mcp lost; nothing about the design made that outcome deterministic, and next reboot it could be either.
Any cold start where the IdP is slower to become ready than the MCP servers silently disables OAuth on whichever ones lost.
Suggested test
A fake issuer that 502s for the first N discovery attempts and then succeeds. Assertions:
construction succeeds despite the initial 502
Validate returns ErrUnavailable while discovery is failing → middleware answers 503, not 401, and never static_token_mismatch
once the issuer recovers, a valid token is accepted with no restart
a genuinely static-token-only server (no issuer configured) still reports static_token_mismatch — the existing behaviour must not regress
Related
gitea-mcp#6 — added the ErrUnavailable → 503 path this extends
infra#349 — the reboot that exposed it
The estate hit three variants of "component reports healthy while unusable" this week: llama.cpp silently falling back to CPU, LiteLLMTierDown firing unread, and this. Worth treating as a pattern rather than three incidents.
Found 2026-09-07 debugging why the claude.ai gitea connector could not authenticate. gitea-mcp had been rejecting every OAuth JWT for ~16 hours. The chassis behaved exactly as written; the gap is that "the validator could not be built" is indistinguishable from "no JWT auth wanted".
## What happened
koala rebooted. gitea-mcp started before Authentik was serving:
```
05:07:46 WARN multi-issuer init failed; retrying without the k8s issuer
err="issuer https://auth.d-ma.be/application/o/claude-ai/: oidc discovery: status 502"
05:07:46 WARN jwt validator init failed; JWT auth degraded
05:07:46 INFO gitea-mcp starting addr=:8080 version=2bb1fef
```
Then, for every subsequent request, ~370 of these:
```
WARN mcp auth rejected reason=static_token_mismatch token_type=jwt status=401
```
Authentik was healthy within minutes. gitea-mcp never noticed. A `kubectl rollout restart` fixed it instantly — `/healthz` went from `jwt.status=degraded` to `jwt=enabled`.
## Why the chassis produced that message
`auth/bearer.go` is correct and its own comment says so:
```go
if validator != nil {
switch _, err := validator.Validate(r.Context(), rawToken); {
case err == nil: // allow
case errors.Is(err, ErrUnavailable):
// Dex/JWKS is down — the token might be fine, we just can't
// check. Answer 503 so the caller retries rather than treating
// it as a hard auth failure (gitea-mcp#6).
deny(..., http.StatusServiceUnavailable, "jwt_dex_unavailable", ...)
default:
deny(..., http.StatusUnauthorized, "jwt_invalid", ...)
}
}
// 3. Static token was wrong and no JWT validator is configured.
deny(..., http.StatusUnauthorized, "static_token_mismatch", ...)
```
`ErrUnavailable` already encodes precisely the right idea — *we cannot check, so 503 and retry* — and gitea-mcp#6 got that right. But it only applies to a validator that **exists**.
`NewJWTValidator` / `NewMultiJWTValidator` (`auth/jwt.go:74`, `:85`) resolve discovery **eagerly at construction**: `discoverJWKSURI` returns `oidc discovery: status 502`, the constructor errors, and the caller is left holding `nil`. From the middleware's perspective `nil` means "this server does not do JWT auth", so it falls through to the static comparison and reports `static_token_mismatch`.
Two states collapse into one:
| intent | validator | middleware sees |
|---|---|---|
| static-token-only server, by design | `nil` | correct |
| JWT required, IdP was down at boot | `nil` | **wrong** — reports as static-only, forever |
The failure is permanent because discovery is attempted exactly once, at construction.
## Proposed fix — in the chassis, because every MCP server here shares it
**1. Make the validator constructible without successful discovery.** Resolve `jwks_uri` lazily on first use, and re-resolve on failure with backoff. Until resolution succeeds, `Validate` returns `ErrUnavailable` — which the middleware *already* turns into a 503 the client retries. A boot-time 502 then self-heals within seconds instead of requiring a human and a `rollout restart`.
This needs no change to `bearer.go`. The correct behaviour is already there; it is currently unreachable for this case.
**2. Do not let a nil validator silently mean "static only" when an issuer was configured.** Either a constructor variant that cannot fail (per 1), or an explicit `RequireJWT` flag so the middleware can answer 503 rather than 401 when JWT auth is required but unavailable. Silently downgrading an auth mode is the wrong default.
**3. Expose the validator's readiness so servers can report it.** `/healthz` returned `{"ok":true,"jwt":{"status":"degraded",...}}` — the process knew it was broken and still called itself healthy. That is gitea-mcp's handler, but the chassis owns the state it needs, so a `Validator.Ready() (bool, error)` (or similar) would let every server get this right the same way. A component that knows it is degraded should not report itself healthy.
## Blast radius
Every MCP server in the estate uses `chassisauth.BearerMiddleware` — gitea-mcp, brain-mcp, infra-mcp. All share this failure mode. On this reboot **brain-mcp happened to win the race and gitea-mcp lost**; nothing about the design made that outcome deterministic, and next reboot it could be either.
Any cold start where the IdP is slower to become ready than the MCP servers silently disables OAuth on whichever ones lost.
## Suggested test
A fake issuer that 502s for the first N discovery attempts and then succeeds. Assertions:
- construction succeeds despite the initial 502
- `Validate` returns `ErrUnavailable` while discovery is failing → middleware answers **503**, not 401, and never `static_token_mismatch`
- once the issuer recovers, a valid token is accepted with **no restart**
- a genuinely static-token-only server (no issuer configured) still reports `static_token_mismatch` — the existing behaviour must not regress
## Related
- gitea-mcp#6 — added the `ErrUnavailable` → 503 path this extends
- infra#349 — the reboot that exposed it
- The estate hit three variants of "component reports healthy while unusable" this week: llama.cpp silently falling back to CPU, `LiteLLMTierDown` firing unread, and this. Worth treating as a pattern rather than three incidents.
---
_Created via git-mcp on behalf of @mathias_
Fixed and deployed. mcp-chassis v0.6.0, commit 3257364.
It recurred on 2026-09-14 — a third time, running six days before anyone noticed. That one was found by chance while gathering context for unrelated work, which is the strongest argument this issue could have had.
The fix is the one this issue specified
Resolve jwks_uri lazily on first use, and re-resolve on failure with backoff. Until resolution succeeds, Validate returns ErrUnavailable — which the middleware already turns into a 503 the client retries.
Exactly that. issuerEntry now holds jwksURI behind a mutex, empty until discovery succeeds, and resolve() retries no sooner than WithRetryInterval apart (default 30s) so a down issuer costs one discovery request per interval rather than one per request. The constructor still warms discovery eagerly on a healthy start — it just no longer fails on it.
bearer.go is unchanged, as this issue predicted. The correct behaviour was already there and merely unreachable.
Ready(ctx) — and the probe is the retry
Point 3 asked for a readiness accessor. Ready(ctx) reports whether every issuer resolved and attempts resolution for any that have not. That makes a Kubernetes readiness probe the retry loop: no background goroutine, no context to own, no lifecycle to get wrong, and the platform's own cadence drives recovery. gitea-mcp now probes /readyz every 5s.
A nil validator is Ready — "JWT auth disabled" is a configuration, not a fault.
One correction to this issue's blast radius
Every MCP server in the estate uses chassisauth.BearerMiddleware — gitea-mcp, brain-mcp, infra-mcp. All share this failure mode. On this reboot brain-mcp happened to win the race and gitea-mcp lost.
Not so, and the difference is instructive. Measured 2026-09-14:
go.mod consumers of this module are cad, gitea-mcp, sa-token-poc. hyperguild (which serves brain-mcp.d-ma.be) does not import it.
At the same second, same Authentik 503, the two behaved differently:
17:36:10 ingestion (brain-mcp): WARN "oidc discovery not ready, retrying" → recovered
17:36:10 gitea-mcp (this module): WARN "jwt validator init failed; JWT auth degraded" → dead 6 days
brain-mcp did not win a race. It has its own auth implementation that already retried, which is the behaviour this change brings to the chassis. infra-mcp is upstream third-party (kubernetes_mcp_server:v0.0.60) and uses none of this.
So the real blast radius was cad + gitea-mcp + sa-token-poc — and the reference implementation for the correct shape was sitting in the one server that had never adopted the shared library.
Tests
auth/jwt_bootrace_test.go, written first, red on the absent API. The four assertions this issue asked for, all present. The sharpest is the last:
construction succeeds despite the initial 503
Validate returns ErrUnavailable while down — never a token verdict
Ready is false and says why
the issuer recovers and the same validator works, with no reconstruction
a genuinely static-token-only server still returns (nil, nil) — no regression
One existing test changed its mechanism, not its claim. TestNewMultiJWTValidator_UsesPerIssuerHTTPClientForAuthedFetch asserted the constructor errors when discovery 401s; it cannot now, by design. Rewritten to assert the property that actually mattered, more directly than before: without the credential the validator is not ready and a correctly-minted token from that very issuer is refused as ErrUnavailable. The old version only checked that a function returned an error.
golangci-lint 0 issues, go vet clean, go test -race -count=1 ok.
Downstream
gitea-mcp v0.14.0 consumes it, infra v1.36.0 points readiness at /readyz and adds GiteaMcpUnavailable (critical, so infra#355's edge files a ticket unattended). Live and verified: pod Ready, which now means/readyz answered 200, which means the validator resolved. Zero auth rejections since.
cad and sa-token-poc still pin v0.5.0 and inherit the bug until bumped — filing that separately rather than leaving it here.
## Fixed and deployed. `mcp-chassis v0.6.0`, commit `3257364`.
It recurred on **2026-09-14** — a third time, running six days before anyone noticed. That one was found by chance while gathering context for unrelated work, which is the strongest argument this issue could have had.
### The fix is the one this issue specified
> Resolve `jwks_uri` lazily on first use, and re-resolve on failure with backoff. Until resolution succeeds, `Validate` returns `ErrUnavailable` — which the middleware *already* turns into a 503 the client retries.
Exactly that. `issuerEntry` now holds `jwksURI` behind a mutex, empty until discovery succeeds, and `resolve()` retries no sooner than `WithRetryInterval` apart (default 30s) so a down issuer costs one discovery request per interval rather than one per request. The constructor still warms discovery eagerly on a healthy start — it just no longer *fails* on it.
`bearer.go` is unchanged, as this issue predicted. The correct behaviour was already there and merely unreachable.
### `Ready(ctx)` — and the probe is the retry
Point 3 asked for a readiness accessor. `Ready(ctx)` reports whether every issuer resolved **and attempts resolution for any that have not**. That makes a Kubernetes readiness probe the retry loop: no background goroutine, no context to own, no lifecycle to get wrong, and the platform's own cadence drives recovery. `gitea-mcp` now probes `/readyz` every 5s.
A nil validator is `Ready` — "JWT auth disabled" is a configuration, not a fault.
### One correction to this issue's blast radius
> Every MCP server in the estate uses `chassisauth.BearerMiddleware` — gitea-mcp, brain-mcp, infra-mcp. All share this failure mode. On this reboot **brain-mcp happened to win the race and gitea-mcp lost**.
Not so, and the difference is instructive. Measured 2026-09-14:
- `go.mod` consumers of this module are **`cad`, `gitea-mcp`, `sa-token-poc`**. `hyperguild` (which serves `brain-mcp.d-ma.be`) does not import it.
- At the same second, same Authentik 503, the two behaved differently:
```
17:36:10 ingestion (brain-mcp): WARN "oidc discovery not ready, retrying" → recovered
17:36:10 gitea-mcp (this module): WARN "jwt validator init failed; JWT auth degraded" → dead 6 days
```
brain-mcp did not win a race. It has its own auth implementation that **already retried**, which is the behaviour this change brings to the chassis. `infra-mcp` is upstream third-party (`kubernetes_mcp_server:v0.0.60`) and uses none of this.
So the real blast radius was `cad` + `gitea-mcp` + `sa-token-poc` — and the reference implementation for the correct shape was sitting in the one server that had never adopted the shared library.
### Tests
`auth/jwt_bootrace_test.go`, written first, red on the absent API. The four assertions this issue asked for, all present. The sharpest is the last:
- construction succeeds despite the initial 503
- `Validate` returns `ErrUnavailable` while down — never a token verdict
- `Ready` is false and says why
- **the issuer recovers and the same validator works, with no reconstruction**
- a genuinely static-token-only server still returns `(nil, nil)` — no regression
One existing test changed its mechanism, not its claim. `TestNewMultiJWTValidator_UsesPerIssuerHTTPClientForAuthedFetch` asserted the constructor *errors* when discovery 401s; it cannot now, by design. Rewritten to assert the property that actually mattered, more directly than before: without the credential the validator is not ready **and** a correctly-minted token from that very issuer is refused as `ErrUnavailable`. The old version only checked that a function returned an error.
`golangci-lint` 0 issues, `go vet` clean, `go test -race -count=1` ok.
### Downstream
`gitea-mcp v0.14.0` consumes it, `infra v1.36.0` points readiness at `/readyz` and adds `GiteaMcpUnavailable` (critical, so infra#355's edge files a ticket unattended). Live and verified: pod `Ready`, which now *means* `/readyz` answered 200, which means the validator resolved. Zero auth rejections since.
`cad` and `sa-token-poc` still pin `v0.5.0` and inherit the bug until bumped — filing that separately rather than leaving it here.
Closing.
---
_Created via git-mcp on behalf of @mathias_
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Found 2026-09-07 debugging why the claude.ai gitea connector could not authenticate. gitea-mcp had been rejecting every OAuth JWT for ~16 hours. The chassis behaved exactly as written; the gap is that "the validator could not be built" is indistinguishable from "no JWT auth wanted".
What happened
koala rebooted. gitea-mcp started before Authentik was serving:
Then, for every subsequent request, ~370 of these:
Authentik was healthy within minutes. gitea-mcp never noticed. A
kubectl rollout restartfixed it instantly —/healthzwent fromjwt.status=degradedtojwt=enabled.Why the chassis produced that message
auth/bearer.gois correct and its own comment says so:ErrUnavailablealready encodes precisely the right idea — we cannot check, so 503 and retry — and gitea-mcp#6 got that right. But it only applies to a validator that exists.NewJWTValidator/NewMultiJWTValidator(auth/jwt.go:74,:85) resolve discovery eagerly at construction:discoverJWKSURIreturnsoidc discovery: status 502, the constructor errors, and the caller is left holdingnil. From the middleware's perspectivenilmeans "this server does not do JWT auth", so it falls through to the static comparison and reportsstatic_token_mismatch.Two states collapse into one:
nilnilThe failure is permanent because discovery is attempted exactly once, at construction.
Proposed fix — in the chassis, because every MCP server here shares it
1. Make the validator constructible without successful discovery. Resolve
jwks_urilazily on first use, and re-resolve on failure with backoff. Until resolution succeeds,ValidatereturnsErrUnavailable— which the middleware already turns into a 503 the client retries. A boot-time 502 then self-heals within seconds instead of requiring a human and arollout restart.This needs no change to
bearer.go. The correct behaviour is already there; it is currently unreachable for this case.2. Do not let a nil validator silently mean "static only" when an issuer was configured. Either a constructor variant that cannot fail (per 1), or an explicit
RequireJWTflag so the middleware can answer 503 rather than 401 when JWT auth is required but unavailable. Silently downgrading an auth mode is the wrong default.3. Expose the validator's readiness so servers can report it.
/healthzreturned{"ok":true,"jwt":{"status":"degraded",...}}— the process knew it was broken and still called itself healthy. That is gitea-mcp's handler, but the chassis owns the state it needs, so aValidator.Ready() (bool, error)(or similar) would let every server get this right the same way. A component that knows it is degraded should not report itself healthy.Blast radius
Every MCP server in the estate uses
chassisauth.BearerMiddleware— gitea-mcp, brain-mcp, infra-mcp. All share this failure mode. On this reboot brain-mcp happened to win the race and gitea-mcp lost; nothing about the design made that outcome deterministic, and next reboot it could be either.Any cold start where the IdP is slower to become ready than the MCP servers silently disables OAuth on whichever ones lost.
Suggested test
A fake issuer that 502s for the first N discovery attempts and then succeeds. Assertions:
ValidatereturnsErrUnavailablewhile discovery is failing → middleware answers 503, not 401, and neverstatic_token_mismatchstatic_token_mismatch— the existing behaviour must not regressRelated
ErrUnavailable→ 503 path this extendsLiteLLMTierDownfiring unread, and this. Worth treating as a pattern rather than three incidents.Created via git-mcp on behalf of @mathias
Fixed and deployed.
mcp-chassis v0.6.0, commit3257364.It recurred on 2026-09-14 — a third time, running six days before anyone noticed. That one was found by chance while gathering context for unrelated work, which is the strongest argument this issue could have had.
The fix is the one this issue specified
Exactly that.
issuerEntrynow holdsjwksURIbehind a mutex, empty until discovery succeeds, andresolve()retries no sooner thanWithRetryIntervalapart (default 30s) so a down issuer costs one discovery request per interval rather than one per request. The constructor still warms discovery eagerly on a healthy start — it just no longer fails on it.bearer.gois unchanged, as this issue predicted. The correct behaviour was already there and merely unreachable.Ready(ctx)— and the probe is the retryPoint 3 asked for a readiness accessor.
Ready(ctx)reports whether every issuer resolved and attempts resolution for any that have not. That makes a Kubernetes readiness probe the retry loop: no background goroutine, no context to own, no lifecycle to get wrong, and the platform's own cadence drives recovery.gitea-mcpnow probes/readyzevery 5s.A nil validator is
Ready— "JWT auth disabled" is a configuration, not a fault.One correction to this issue's blast radius
Not so, and the difference is instructive. Measured 2026-09-14:
go.modconsumers of this module arecad,gitea-mcp,sa-token-poc.hyperguild(which servesbrain-mcp.d-ma.be) does not import it.brain-mcp did not win a race. It has its own auth implementation that already retried, which is the behaviour this change brings to the chassis.
infra-mcpis upstream third-party (kubernetes_mcp_server:v0.0.60) and uses none of this.So the real blast radius was
cad+gitea-mcp+sa-token-poc— and the reference implementation for the correct shape was sitting in the one server that had never adopted the shared library.Tests
auth/jwt_bootrace_test.go, written first, red on the absent API. The four assertions this issue asked for, all present. The sharpest is the last:ValidatereturnsErrUnavailablewhile down — never a token verdictReadyis false and says why(nil, nil)— no regressionOne existing test changed its mechanism, not its claim.
TestNewMultiJWTValidator_UsesPerIssuerHTTPClientForAuthedFetchasserted the constructor errors when discovery 401s; it cannot now, by design. Rewritten to assert the property that actually mattered, more directly than before: without the credential the validator is not ready and a correctly-minted token from that very issuer is refused asErrUnavailable. The old version only checked that a function returned an error.golangci-lint0 issues,go vetclean,go test -race -count=1ok.Downstream
gitea-mcp v0.14.0consumes it,infra v1.36.0points readiness at/readyzand addsGiteaMcpUnavailable(critical, so infra#355's edge files a ticket unattended). Live and verified: podReady, which now means/readyzanswered 200, which means the validator resolved. Zero auth rejections since.cadandsa-token-pocstill pinv0.5.0and inherit the bug until bumped — filing that separately rather than leaving it here.Closing.
Created via git-mcp on behalf of @mathias