After a koala reboot, every claude.ai connector request to git-mcp.d-ma.be returned 401. The claude.ai UI reported Authorization with gitea failed. You can check your credentials and permissions. — misdirecting toward credentials, which were valid throughout.
Root cause
gitea-mcp v0.11.0 initializes the JWT validator exactly once at startup. On 2026-07-17 the pod started at 11:58:46Z and reached OIDC discovery at 11:59:00Z, roughly 12s before Authentik was serving traffic:
11:59:00 WARN multi-issuer init failed; retrying without the k8s issuer
err="issuer https://auth.d-ma.be/application/o/claude-ai/: oidc discovery: status 502"
11:59:01 WARN jwt validator init failed; JWT auth degraded
11:59:01 INFO gitea-mcp starting addr=:8080 version=v0.11.0
There is no retry. The validator stays dead for the process lifetime. BearerMiddleware then falls through to static-token comparison for every inbound token, including JWTs, so a valid Authentik JWT is string-compared against the static token and rejected:
The pod stays Running, 1/1, readiness green. Kubernetes has no signal that JWT auth is dead, so it keeps routing traffic to a pod that 401s every JWT-authenticated caller. The only recovery is a manual kubectl rollout restart. Every co-boot of koala (where Authentik and gitea-mcp race) is a coin flip on whether JWT auth comes up at all.
Confirmed fix on 2026-07-17 was kubectl rollout restart deploy/gitea-mcp -n gitea-mcp once discovery returned 200 — clean boot, no degradation warnings, connector reconnected with no user re-auth.
Proposed fix
Pick one (1 or 2 preferred, 3 as defense-in-depth):
Retry discovery with backoff in the background until it succeeds; swap the validator in once ready.
Lazy-init on first JWT seen — avoids the boot race entirely, cost is paid once on first JWT request.
Fail the readiness probe while the validator is down so k8s does not route to a degraded pod. On its own this turns a silent 401 into a visible CrashLoop/NotReady, which is a better failure but still an outage; pair with 1 or 2.
Option 3 alone would also have surfaced this within seconds instead of costing a debugging session.
Acceptance criterion
A test that starts gitea-mcp with the OIDC issuer returning 502, then brings the issuer up, and asserts a valid JWT authenticates successfully without a process restart.
## Symptom
After a koala reboot, every claude.ai connector request to `git-mcp.d-ma.be` returned 401. The claude.ai UI reported `Authorization with gitea failed. You can check your credentials and permissions.` — misdirecting toward credentials, which were valid throughout.
## Root cause
gitea-mcp v0.11.0 initializes the JWT validator exactly once at startup. On 2026-07-17 the pod started at `11:58:46Z` and reached OIDC discovery at `11:59:00Z`, roughly 12s before Authentik was serving traffic:
```
11:59:00 WARN multi-issuer init failed; retrying without the k8s issuer
err="issuer https://auth.d-ma.be/application/o/claude-ai/: oidc discovery: status 502"
11:59:01 WARN jwt validator init failed; JWT auth degraded
11:59:01 INFO gitea-mcp starting addr=:8080 version=v0.11.0
```
There is no retry. The validator stays dead for the process lifetime. `BearerMiddleware` then falls through to static-token comparison for *every* inbound token, including JWTs, so a valid Authentik JWT is string-compared against the static token and rejected:
```
WARN mcp auth rejected reason="static_token_mismatch" status=401 token_type="jwt" token_fp="872d2da957ac" path="/mcp"
```
## Why this is worse than a transient failure
The pod stays `Running`, `1/1`, readiness green. Kubernetes has no signal that JWT auth is dead, so it keeps routing traffic to a pod that 401s every JWT-authenticated caller. The only recovery is a manual `kubectl rollout restart`. Every co-boot of koala (where Authentik and gitea-mcp race) is a coin flip on whether JWT auth comes up at all.
Confirmed fix on 2026-07-17 was `kubectl rollout restart deploy/gitea-mcp -n gitea-mcp` once discovery returned 200 — clean boot, no degradation warnings, connector reconnected with no user re-auth.
## Proposed fix
Pick one (1 or 2 preferred, 3 as defense-in-depth):
1. **Retry discovery with backoff** in the background until it succeeds; swap the validator in once ready.
2. **Lazy-init on first JWT seen** — avoids the boot race entirely, cost is paid once on first JWT request.
3. **Fail the readiness probe while the validator is down** so k8s does not route to a degraded pod. On its own this turns a silent 401 into a visible CrashLoop/NotReady, which is a better failure but still an outage; pair with 1 or 2.
Option 3 alone would also have surfaced this within seconds instead of costing a debugging session.
## Acceptance criterion
A test that starts gitea-mcp with the OIDC issuer returning 502, then brings the issuer up, and asserts a valid JWT authenticates successfully without a process restart.
## Refs
- Related: `[[mcp-chassis-shared-auth-library]]` (`BearerMiddleware(staticToken, jwtValidator, …)`)
- Misleading log reason filed separately.
- Observed on v0.11.0, koala k3s, 2026-07-17.
Fixed and live. v0.14.0, commits aaaa4b4 + 09bde6d.
Filed 2026-07-17. It recurred 2026-09-07 (mcp-chassis#4) and again 2026-09-14, that time running six days before anyone noticed — found by chance, not by anything watching.
This issue proposed three fixes and said "1 or 2 preferred, 3 as defense-in-depth". All three landed, because they turn out to be one mechanism.
1 + 2 — retry, in the chassis (mcp-chassis#4, v0.6.0)
jwks_uri now resolves lazily and re-resolves on failure, no sooner than a retry interval apart. Until it resolves, Validate returns ErrUnavailable, which BearerMiddleware already turns into 503-and-retry. The constructor no longer fails on an unreachable issuer, so nil stops meaning two different things.
Fixed there rather than here, because cad and sa-token-poc share the same library and the same bug.
3 — readiness, and it is not just defence in depth
The pod stays Running, 1/1, readiness green. Kubernetes has no signal that JWT auth is dead.
/readyz (new) answers 503 while the validator cannot reach its issuer; infra v1.36.0 points the readinessProbe at it. The degraded pod now leaves the Service instead of serving guaranteed 401s.
But it does more than surface the fault: evaluating readiness is what retries the issuer. The kubelet's 5s cadence is the retry loop — no background goroutine, no context to own, no lifecycle to get wrong. Option 3 turned out to be the delivery mechanism for option 1, not an alternative to it.
/healthz stays 200 deliberately. The process is alive, and restarting it does not make Authentik come back sooner — pointing liveness at /readyz would turn an upstream outage into a CrashLoopBackOff.
The half that lived in this repo, and would have survived the chassis fix
newHealthzHandler marshalled its body once at startup and served those bytes for the process lifetime. On 2026-09-14 it reported jwt.status=degraded for six days after Authentik had recovered, because the answer predated the recovery and could never change.
Fixing the validator alone would not have fixed that: the validator would recover and /healthz would keep saying degraded. A health endpoint that cannot change its answer is not a health endpoint. It now evaluates per request.
Also removed
The multi-issuer init failed; retrying without the k8s issuer fallback. It existed because one unreachable issuer used to fail the whole constructor and take Authentik auth down with it; each issuer now resolves independently, so a down issuer degrades only itself. That warning was the first line of the 2026-09-14 incident and no longer has a reason to exist. A constructor error now means a config error and exits non-zero rather than warning and continuing.
Acceptance criterion
A test that starts gitea-mcp with the OIDC issuer returning 502, then brings the issuer up, and asserts a valid JWT authenticates successfully without a process restart.
mcp-chassis auth/jwt_bootrace_test.go — exactly that, at the layer where the bug lived. Here, cmd/gitea-mcp/healthz_live_test.go pins the other half; its sharpest assertion is that readiness is evaluated twice for two requests, which a handler that caches cannot pass.
Detection, so a fourth time pages
GiteaMcpUnavailable (infra alerts.yaml), critical — so infra#355's edge turns it into a tracked issue unattended. KubeDeploymentReplicasMismatch already covered the condition, but as a warning it would reach ntfy and stop there, which is how the previous three ended.
Verified live
gitea-mcp:09bde6d deployed via CD, Flux at main@sha1:f0eaf5d, readinessProbe /readyz applied, pod Ready — which now means/readyz answered 200, which means the validator resolved. Zero auth rejections since. Rule loaded in Prometheus, inactive.
## Fixed and live. `v0.14.0`, commits `aaaa4b4` + `09bde6d`.
Filed 2026-07-17. It recurred 2026-09-07 (mcp-chassis#4) and again **2026-09-14**, that time running six days before anyone noticed — found by chance, not by anything watching.
This issue proposed three fixes and said *"1 or 2 preferred, 3 as defense-in-depth"*. All three landed, because they turn out to be one mechanism.
### 1 + 2 — retry, in the chassis (mcp-chassis#4, v0.6.0)
`jwks_uri` now resolves lazily and re-resolves on failure, no sooner than a retry interval apart. Until it resolves, `Validate` returns `ErrUnavailable`, which `BearerMiddleware` already turns into 503-and-retry. The constructor no longer fails on an unreachable issuer, so `nil` stops meaning two different things.
Fixed there rather than here, because `cad` and `sa-token-poc` share the same library and the same bug.
### 3 — readiness, and it is not just defence in depth
> The pod stays `Running`, `1/1`, readiness green. Kubernetes has no signal that JWT auth is dead.
`/readyz` (new) answers 503 while the validator cannot reach its issuer; `infra v1.36.0` points the readinessProbe at it. The degraded pod now leaves the Service instead of serving guaranteed 401s.
But it does more than surface the fault: **evaluating readiness is what retries the issuer**. The kubelet's 5s cadence is the retry loop — no background goroutine, no context to own, no lifecycle to get wrong. Option 3 turned out to be the delivery mechanism for option 1, not an alternative to it.
`/healthz` stays 200 deliberately. The process is alive, and restarting it does not make Authentik come back sooner — pointing liveness at `/readyz` would turn an upstream outage into a CrashLoopBackOff.
### The half that lived in this repo, and would have survived the chassis fix
`newHealthzHandler` marshalled its body **once at startup** and served those bytes for the process lifetime. On 2026-09-14 it reported `jwt.status=degraded` for six days after Authentik had recovered, because the answer predated the recovery and could never change.
Fixing the validator alone would not have fixed that: the validator would recover and `/healthz` would keep saying degraded. A health endpoint that cannot change its answer is not a health endpoint. It now evaluates per request.
### Also removed
The `multi-issuer init failed; retrying without the k8s issuer` fallback. It existed because one unreachable issuer used to fail the whole constructor and take Authentik auth down with it; each issuer now resolves independently, so a down issuer degrades only itself. That warning was the first line of the 2026-09-14 incident and no longer has a reason to exist. A constructor error now means a config error and exits non-zero rather than warning and continuing.
### Acceptance criterion
> A test that starts gitea-mcp with the OIDC issuer returning 502, then brings the issuer up, and asserts a valid JWT authenticates successfully without a process restart.
`mcp-chassis auth/jwt_bootrace_test.go` — exactly that, at the layer where the bug lived. Here, `cmd/gitea-mcp/healthz_live_test.go` pins the other half; its sharpest assertion is that readiness is evaluated **twice for two requests**, which a handler that caches cannot pass.
### Detection, so a fourth time pages
`GiteaMcpUnavailable` (infra `alerts.yaml`), **critical** — so infra#355's edge turns it into a tracked issue unattended. `KubeDeploymentReplicasMismatch` already covered the condition, but as a `warning` it would reach ntfy and stop there, which is how the previous three ended.
### Verified live
`gitea-mcp:09bde6d` deployed via CD, Flux at `main@sha1:f0eaf5d`, readinessProbe `/readyz` applied, pod `Ready` — which now *means* `/readyz` answered 200, which means the validator resolved. Zero auth rejections since. Rule loaded in Prometheus, `inactive`.
Closing.
---
_Created via git-mcp on behalf of @mathias_
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Symptom
After a koala reboot, every claude.ai connector request to
git-mcp.d-ma.bereturned 401. The claude.ai UI reportedAuthorization with gitea failed. You can check your credentials and permissions.— misdirecting toward credentials, which were valid throughout.Root cause
gitea-mcp v0.11.0 initializes the JWT validator exactly once at startup. On 2026-07-17 the pod started at
11:58:46Zand reached OIDC discovery at11:59:00Z, roughly 12s before Authentik was serving traffic:There is no retry. The validator stays dead for the process lifetime.
BearerMiddlewarethen falls through to static-token comparison for every inbound token, including JWTs, so a valid Authentik JWT is string-compared against the static token and rejected:Why this is worse than a transient failure
The pod stays
Running,1/1, readiness green. Kubernetes has no signal that JWT auth is dead, so it keeps routing traffic to a pod that 401s every JWT-authenticated caller. The only recovery is a manualkubectl rollout restart. Every co-boot of koala (where Authentik and gitea-mcp race) is a coin flip on whether JWT auth comes up at all.Confirmed fix on 2026-07-17 was
kubectl rollout restart deploy/gitea-mcp -n gitea-mcponce discovery returned 200 — clean boot, no degradation warnings, connector reconnected with no user re-auth.Proposed fix
Pick one (1 or 2 preferred, 3 as defense-in-depth):
Option 3 alone would also have surfaced this within seconds instead of costing a debugging session.
Acceptance criterion
A test that starts gitea-mcp with the OIDC issuer returning 502, then brings the issuer up, and asserts a valid JWT authenticates successfully without a process restart.
Refs
[[mcp-chassis-shared-auth-library]](BearerMiddleware(staticToken, jwtValidator, …))Fixed and live.
v0.14.0, commitsaaaa4b4+09bde6d.Filed 2026-07-17. It recurred 2026-09-07 (mcp-chassis#4) and again 2026-09-14, that time running six days before anyone noticed — found by chance, not by anything watching.
This issue proposed three fixes and said "1 or 2 preferred, 3 as defense-in-depth". All three landed, because they turn out to be one mechanism.
1 + 2 — retry, in the chassis (mcp-chassis#4, v0.6.0)
jwks_urinow resolves lazily and re-resolves on failure, no sooner than a retry interval apart. Until it resolves,ValidatereturnsErrUnavailable, whichBearerMiddlewarealready turns into 503-and-retry. The constructor no longer fails on an unreachable issuer, sonilstops meaning two different things.Fixed there rather than here, because
cadandsa-token-pocshare the same library and the same bug.3 — readiness, and it is not just defence in depth
/readyz(new) answers 503 while the validator cannot reach its issuer;infra v1.36.0points the readinessProbe at it. The degraded pod now leaves the Service instead of serving guaranteed 401s.But it does more than surface the fault: evaluating readiness is what retries the issuer. The kubelet's 5s cadence is the retry loop — no background goroutine, no context to own, no lifecycle to get wrong. Option 3 turned out to be the delivery mechanism for option 1, not an alternative to it.
/healthzstays 200 deliberately. The process is alive, and restarting it does not make Authentik come back sooner — pointing liveness at/readyzwould turn an upstream outage into a CrashLoopBackOff.The half that lived in this repo, and would have survived the chassis fix
newHealthzHandlermarshalled its body once at startup and served those bytes for the process lifetime. On 2026-09-14 it reportedjwt.status=degradedfor six days after Authentik had recovered, because the answer predated the recovery and could never change.Fixing the validator alone would not have fixed that: the validator would recover and
/healthzwould keep saying degraded. A health endpoint that cannot change its answer is not a health endpoint. It now evaluates per request.Also removed
The
multi-issuer init failed; retrying without the k8s issuerfallback. It existed because one unreachable issuer used to fail the whole constructor and take Authentik auth down with it; each issuer now resolves independently, so a down issuer degrades only itself. That warning was the first line of the 2026-09-14 incident and no longer has a reason to exist. A constructor error now means a config error and exits non-zero rather than warning and continuing.Acceptance criterion
mcp-chassis auth/jwt_bootrace_test.go— exactly that, at the layer where the bug lived. Here,cmd/gitea-mcp/healthz_live_test.gopins the other half; its sharpest assertion is that readiness is evaluated twice for two requests, which a handler that caches cannot pass.Detection, so a fourth time pages
GiteaMcpUnavailable(infraalerts.yaml), critical — so infra#355's edge turns it into a tracked issue unattended.KubeDeploymentReplicasMismatchalready covered the condition, but as awarningit would reach ntfy and stop there, which is how the previous three ended.Verified live
gitea-mcp:09bde6ddeployed via CD, Flux atmain@sha1:f0eaf5d, readinessProbe/readyzapplied, podReady— which now means/readyzanswered 200, which means the validator resolved. Zero auth rejections since. Rule loaded in Prometheus,inactive.Closing.
Created via git-mcp on behalf of @mathias