JWT validator init is one-shot at startup — IdP unavailable during co-boot degrades auth permanently and silently #57

Open
opened 2026-07-17 13:02:45 +00:00 by mathias · 0 comments
Owner

Symptom

After a koala reboot, every claude.ai connector request to git-mcp.d-ma.be returned 401. The claude.ai UI reported Authorization with gitea failed. You can check your credentials and permissions. — misdirecting toward credentials, which were valid throughout.

Root cause

gitea-mcp v0.11.0 initializes the JWT validator exactly once at startup. On 2026-07-17 the pod started at 11:58:46Z and reached OIDC discovery at 11:59:00Z, roughly 12s before Authentik was serving traffic:

11:59:00 WARN multi-issuer init failed; retrying without the k8s issuer
         err="issuer https://auth.d-ma.be/application/o/claude-ai/: oidc discovery: status 502"
11:59:01 WARN jwt validator init failed; JWT auth degraded
11:59:01 INFO gitea-mcp starting addr=:8080 version=v0.11.0

There is no retry. The validator stays dead for the process lifetime. BearerMiddleware then falls through to static-token comparison for every inbound token, including JWTs, so a valid Authentik JWT is string-compared against the static token and rejected:

WARN mcp auth rejected reason="static_token_mismatch" status=401 token_type="jwt" token_fp="872d2da957ac" path="/mcp"

Why this is worse than a transient failure

The pod stays Running, 1/1, readiness green. Kubernetes has no signal that JWT auth is dead, so it keeps routing traffic to a pod that 401s every JWT-authenticated caller. The only recovery is a manual kubectl rollout restart. Every co-boot of koala (where Authentik and gitea-mcp race) is a coin flip on whether JWT auth comes up at all.

Confirmed fix on 2026-07-17 was kubectl rollout restart deploy/gitea-mcp -n gitea-mcp once discovery returned 200 — clean boot, no degradation warnings, connector reconnected with no user re-auth.

Proposed fix

Pick one (1 or 2 preferred, 3 as defense-in-depth):

  1. Retry discovery with backoff in the background until it succeeds; swap the validator in once ready.
  2. Lazy-init on first JWT seen — avoids the boot race entirely, cost is paid once on first JWT request.
  3. Fail the readiness probe while the validator is down so k8s does not route to a degraded pod. On its own this turns a silent 401 into a visible CrashLoop/NotReady, which is a better failure but still an outage; pair with 1 or 2.

Option 3 alone would also have surfaced this within seconds instead of costing a debugging session.

Acceptance criterion

A test that starts gitea-mcp with the OIDC issuer returning 502, then brings the issuer up, and asserts a valid JWT authenticates successfully without a process restart.

Refs

  • Related: [[mcp-chassis-shared-auth-library]] (BearerMiddleware(staticToken, jwtValidator, …))
  • Misleading log reason filed separately.
  • Observed on v0.11.0, koala k3s, 2026-07-17.
## Symptom After a koala reboot, every claude.ai connector request to `git-mcp.d-ma.be` returned 401. The claude.ai UI reported `Authorization with gitea failed. You can check your credentials and permissions.` — misdirecting toward credentials, which were valid throughout. ## Root cause gitea-mcp v0.11.0 initializes the JWT validator exactly once at startup. On 2026-07-17 the pod started at `11:58:46Z` and reached OIDC discovery at `11:59:00Z`, roughly 12s before Authentik was serving traffic: ``` 11:59:00 WARN multi-issuer init failed; retrying without the k8s issuer err="issuer https://auth.d-ma.be/application/o/claude-ai/: oidc discovery: status 502" 11:59:01 WARN jwt validator init failed; JWT auth degraded 11:59:01 INFO gitea-mcp starting addr=:8080 version=v0.11.0 ``` There is no retry. The validator stays dead for the process lifetime. `BearerMiddleware` then falls through to static-token comparison for *every* inbound token, including JWTs, so a valid Authentik JWT is string-compared against the static token and rejected: ``` WARN mcp auth rejected reason="static_token_mismatch" status=401 token_type="jwt" token_fp="872d2da957ac" path="/mcp" ``` ## Why this is worse than a transient failure The pod stays `Running`, `1/1`, readiness green. Kubernetes has no signal that JWT auth is dead, so it keeps routing traffic to a pod that 401s every JWT-authenticated caller. The only recovery is a manual `kubectl rollout restart`. Every co-boot of koala (where Authentik and gitea-mcp race) is a coin flip on whether JWT auth comes up at all. Confirmed fix on 2026-07-17 was `kubectl rollout restart deploy/gitea-mcp -n gitea-mcp` once discovery returned 200 — clean boot, no degradation warnings, connector reconnected with no user re-auth. ## Proposed fix Pick one (1 or 2 preferred, 3 as defense-in-depth): 1. **Retry discovery with backoff** in the background until it succeeds; swap the validator in once ready. 2. **Lazy-init on first JWT seen** — avoids the boot race entirely, cost is paid once on first JWT request. 3. **Fail the readiness probe while the validator is down** so k8s does not route to a degraded pod. On its own this turns a silent 401 into a visible CrashLoop/NotReady, which is a better failure but still an outage; pair with 1 or 2. Option 3 alone would also have surfaced this within seconds instead of costing a debugging session. ## Acceptance criterion A test that starts gitea-mcp with the OIDC issuer returning 502, then brings the issuer up, and asserts a valid JWT authenticates successfully without a process restart. ## Refs - Related: `[[mcp-chassis-shared-auth-library]]` (`BearerMiddleware(staticToken, jwtValidator, …)`) - Misleading log reason filed separately. - Observed on v0.11.0, koala k3s, 2026-07-17.
Sign in to join this conversation.
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/gitea-mcp#57