mathias/hyperguild
1
0
Fork 0
Code Issues 30 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

mcp-chassis: validate a LIST of trusted issuers (accept k8s SA tokens) — ADR-0011 #77

New Issue
Closed
opened 2026-07-06 19:31:42 +00:00 by mathias · 5 comments
mathias commented 2026-07-06 19:31:42 +00:00
Owner
Copy Link
Copy Source

Implements the one hard dependency of infra ADR-0011 (service-to-service auth: native identity per trust domain).

Problem

mcp-chassis BearerMiddleware/JWTValidator matches iss exactly against a single DEX_ISSUER_URL. That's why every in-cluster caller falls back to a long-lived static bearer. ADR-0011's whole plan (in-cluster pods authenticate with kubelet-rotated, audience-bound k8s ServiceAccount tokens) hinges on the chassis trusting more than one issuer.

Change

Make the validator accept a configurable list of trusted issuers, each with its own JWKS source and (optional) required audience:

  • Authentik (D3/D4, existing).
  • The k3s cluster OIDC issuer https://kubernetes.default.svc.cluster.local, JWKS from https://kubernetes.default.svc/openid/v1/jwks (in-cluster, cached) — for D1 SA tokens.
  • (later) an optional trusted Tailscale-proxy identity header for D2.

Additive and backward-compatible: keep accepting the current Authentik/static bearers through the migration.

Proven viable (live-cluster spike, 2026-07-06)

k3s exposes OIDC discovery + audience-scoped kubectl create token out of the box — no cluster-config change. A jwx/v2 verifier (the library the chassis already uses) validated a real SA token: accepts for its audience, rejects wrong-audience (replay block), rejects wrong-issuer. Verifier used:

func Verify(token []byte, jwks jwk.Set, issuer, audience string) error {
    _, err := jwt.Parse(bytes.TrimSpace(token),
        jwt.WithKeySet(jwks), jwt.WithValidate(true),
        jwt.WithIssuer(issuer), jwt.WithAudience(audience))
    return err
}

For a trusted-issuer LIST: pick the entry whose issuer matches the token's iss (peek unverified), then jwt.Parse with that entry's JWKS + required audience. In-cluster, fetch+cache JWKS from kubernetes.default.svc.

Acceptance

  • JWTValidator accepts a []TrustedIssuer{ Issuer, JWKSURL, Audience } (config/env).
  • A SA token with iss=https://kubernetes.default.svc.cluster.local + matching audience validates; wrong-audience and wrong-issuer are rejected (unit tests).
  • Existing Authentik/static-bearer paths still work (no regression).
  • gitea-mcp + brain-mcp wired to trust the cluster issuer.

Then infra rolls SA-token projected volumes onto the in-cluster callers (ADR-0011 migration step 2+). Full rationale + spike detail: brain wiki/homelab/decisions/service-to-service-auth-native-identity-per-domain-2026-07-06; infra ADR-0011.

Implements the one hard dependency of infra **ADR-0011** (service-to-service auth: native identity per trust domain). ## Problem mcp-chassis `BearerMiddleware`/`JWTValidator` matches `iss` **exactly** against a single `DEX_ISSUER_URL`. That's why every in-cluster caller falls back to a long-lived static bearer. ADR-0011's whole plan (in-cluster pods authenticate with kubelet-rotated, audience-bound k8s ServiceAccount tokens) hinges on the chassis trusting **more than one** issuer. ## Change Make the validator accept a configurable **list** of trusted issuers, each with its own JWKS source and (optional) required audience: - Authentik (D3/D4, existing). - The **k3s cluster OIDC issuer** `https://kubernetes.default.svc.cluster.local`, JWKS from `https://kubernetes.default.svc/openid/v1/jwks` (in-cluster, cached) — for D1 SA tokens. - (later) an optional trusted Tailscale-proxy identity header for D2. Additive and backward-compatible: keep accepting the current Authentik/static bearers through the migration. ## Proven viable (live-cluster spike, 2026-07-06) k3s exposes OIDC discovery + audience-scoped `kubectl create token` out of the box — **no cluster-config change**. A `jwx/v2` verifier (the library the chassis already uses) validated a real SA token: accepts for its audience, **rejects wrong-audience** (replay block), rejects wrong-issuer. Verifier used: ```go func Verify(token []byte, jwks jwk.Set, issuer, audience string) error { _, err := jwt.Parse(bytes.TrimSpace(token), jwt.WithKeySet(jwks), jwt.WithValidate(true), jwt.WithIssuer(issuer), jwt.WithAudience(audience)) return err } ``` For a trusted-issuer LIST: pick the entry whose `issuer` matches the token's `iss` (peek unverified), then `jwt.Parse` with that entry's JWKS + required audience. In-cluster, fetch+cache JWKS from `kubernetes.default.svc`. ## Acceptance - [ ] `JWTValidator` accepts a `[]TrustedIssuer{ Issuer, JWKSURL, Audience }` (config/env). - [ ] A SA token with `iss=https://kubernetes.default.svc.cluster.local` + matching audience validates; wrong-audience and wrong-issuer are rejected (unit tests). - [ ] Existing Authentik/static-bearer paths still work (no regression). - [ ] gitea-mcp + brain-mcp wired to trust the cluster issuer. Then infra rolls SA-token projected volumes onto the in-cluster callers (ADR-0011 migration step 2+). Full rationale + spike detail: brain `wiki/homelab/decisions/service-to-service-auth-native-identity-per-domain-2026-07-06`; infra ADR-0011.
mathias commented 2026-07-06 21:15:24 +00:00
Author
Owner
Copy Link
Copy Source

Done — shipped in mcp-chassis v0.4.0 (75eb8b8). Recommend close.

Heads-up: this issue was misfiled — the chassis is its own repo git.d-ma.be/mathias/mcp-chassis (module .../auth), not hyperguild. The change landed there.

What shipped (TDD, backward-compatible)

  • IssuerConfig{IssuerURL, Audience} + NewMultiJWTValidator([]IssuerConfig).
  • JWTValidator now holds a list of issuer entries + a shared jwk.Cache; Validate tries each trusted issuer, returns the subject on first success. All-issuers-JWKS-unreachable → ErrUnavailable (503); any definitive reject → 401.
  • NewJWTValidator (single issuer) and BearerMiddleware signatures UNCHANGED → gitea-mcp / ingestion compile and behave identically; opting into multi-issuer is a one-line constructor swap.
  • Added auth/jwt_test.go: an in-process OIDC issuer harness (discovery + JWKS + token minting) — the chassis had no happy-path JWT test before. New tests: accept-either-trusted-issuer, reject-untrusted (401 not 503), per-issuer audience enforcement, single-issuer backward compat. Full suite green (incl. all prior tests), -race/vet/gofmt clean.

Acceptance

  • validator accepts []TrustedIssuer{Issuer, Audience} (IssuerConfig)
  • SA-token issuer + matching audience validates; wrong-audience + wrong-issuer rejected
  • existing Authentik/static-bearer paths unregressed
  • gitea-mcp + brain-mcp wired to trust the cluster issuer — remaining follow-up

Remaining (ADR-0011 migration step 2+, not this issue)

Bump the mcp-chassis dep in gitea-mcp + ingestion; swap their NewJWTValidator(...) → NewMultiJWTValidator(..., {IssuerURL: "https://kubernetes.default.svc.cluster.local", Audience: "<svc>"}); mount an audience-scoped projected SA token on one pod (issue-reconcile is the clean guinea pig) → prove D1 end-to-end through a live MCP server.

Note there's no CI in mcp-chassis (.gitea/workflows absent) — a shared auth library arguably wants a test gate; worth a separate issue.

Recommend closing (chassis change complete); track the consumer wiring under ADR-0011.

## Done — shipped in mcp-chassis `v0.4.0` (`75eb8b8`). Recommend close. Heads-up: **this issue was misfiled** — the chassis is its own repo `git.d-ma.be/mathias/mcp-chassis` (module `.../auth`), not hyperguild. The change landed there. ### What shipped (TDD, backward-compatible) - `IssuerConfig{IssuerURL, Audience}` + **`NewMultiJWTValidator([]IssuerConfig)`**. - `JWTValidator` now holds a list of issuer entries + a shared `jwk.Cache`; `Validate` tries each trusted issuer, returns the subject on first success. All-issuers-JWKS-unreachable → `ErrUnavailable` (503); any definitive reject → 401. - **`NewJWTValidator` (single issuer) and `BearerMiddleware` signatures UNCHANGED** → gitea-mcp / ingestion compile and behave identically; opting into multi-issuer is a one-line constructor swap. - Added `auth/jwt_test.go`: an in-process OIDC issuer harness (discovery + JWKS + token minting) — **the chassis had no happy-path JWT test before**. New tests: accept-either-trusted-issuer, reject-untrusted (401 not 503), per-issuer audience enforcement, single-issuer backward compat. Full suite green (incl. all prior tests), `-race`/`vet`/`gofmt` clean. ### Acceptance - [x] validator accepts `[]TrustedIssuer{Issuer, Audience}` (`IssuerConfig`) - [x] SA-token issuer + matching audience validates; wrong-audience + wrong-issuer rejected - [x] existing Authentik/static-bearer paths unregressed - [ ] gitea-mcp + brain-mcp wired to trust the cluster issuer — **remaining follow-up** ### Remaining (ADR-0011 migration step 2+, not this issue) Bump the `mcp-chassis` dep in gitea-mcp + ingestion; swap their `NewJWTValidator(...)` → `NewMultiJWTValidator(..., {IssuerURL: "https://kubernetes.default.svc.cluster.local", Audience: "<svc>"})`; mount an audience-scoped projected SA token on one pod (issue-reconcile is the clean guinea pig) → prove D1 end-to-end through a live MCP server. Note there's **no CI in mcp-chassis** (`.gitea/workflows` absent) — a shared auth library arguably wants a test gate; worth a separate issue. **Recommend closing** (chassis change complete); track the consumer wiring under ADR-0011.
mathias commented 2026-07-06 22:23:52 +00:00
Author
Owner
Copy Link
Copy Source

Consumer wiring DONE + proven live on gitea-mcp

The remaining acceptance item ("gitea-mcp wired to trust the cluster issuer") is complete and verified end-to-end on the live server.

  • gitea-mcp (240c3ec): bumped chassis v0.3.0 → v0.5.0; switched NewJWTValidator → NewMultiJWTValidator (Authentik + k8s issuer); added cmd/gitea-mcp/k8soidc.go (cluster-CA + SA-bearer client for the authenticated JWKS fetch). Additive + fallback: if the in-cluster OIDC is unreachable at startup the k8s issuer is dropped so Authentik/static auth is never affected. Smoke-tested off-pod (graceful skip).

  • infra (v0.27.0): K8S_ISSUER_URL + K8S_MCP_AUDIENCE=gitea-mcp on the deployment. Rolled via Flux; the live pod loaded both issuers with no fallback (authed JWKS fetch works from the real pod).

  • Live proof: a real aud=gitea-mcp SA token → the live gitea-mcp via the public ingress returned HTTP 200 (full MCP initialize); no-token = 401; the claude-ai/static path stayed up throughout.

  • gitea-mcp wired to trust the cluster issuer

Remaining is per-caller migration (give an in-cluster caller a projected aud=gitea-mcp token and drop its static bearer — issue-reconcile is the clean first) + optionally wiring ingestion/brain-mcp the same way. Tracked under ADR-0011, not this issue. This one is fully done — close it.

## Consumer wiring DONE + proven live on gitea-mcp The remaining acceptance item ("gitea-mcp wired to trust the cluster issuer") is complete and verified end-to-end on the **live** server. - **gitea-mcp** (`240c3ec`): bumped chassis v0.3.0 → **v0.5.0**; switched `NewJWTValidator` → `NewMultiJWTValidator` (Authentik + k8s issuer); added `cmd/gitea-mcp/k8soidc.go` (cluster-CA + SA-bearer client for the authenticated JWKS fetch). Additive + fallback: if the in-cluster OIDC is unreachable at startup the k8s issuer is dropped so Authentik/static auth is never affected. Smoke-tested off-pod (graceful skip). - **infra** (`v0.27.0`): `K8S_ISSUER_URL` + `K8S_MCP_AUDIENCE=gitea-mcp` on the deployment. Rolled via Flux; the live pod loaded **both** issuers with **no fallback** (authed JWKS fetch works from the real pod). - **Live proof:** a real `aud=gitea-mcp` SA token → the live gitea-mcp via the public ingress returned **HTTP 200** (full MCP `initialize`); no-token = 401; the claude-ai/static path stayed up throughout. - [x] gitea-mcp wired to trust the cluster issuer Remaining is per-caller migration (give an in-cluster caller a projected `aud=gitea-mcp` token and drop its static bearer — issue-reconcile is the clean first) + optionally wiring ingestion/brain-mcp the same way. Tracked under ADR-0011, not this issue. **This one is fully done — close it.**
mathias added the status:dispatch label 2026-08-24 21:50:02 +00:00
mathias added the status:dispatched label 2026-08-24 21:51:06 +00:00
mathias commented 2026-08-24 21:51:06 +00:00
Author
Owner
Copy Link
Copy Source

🤖 dispatch: routing: executor=claude (default — no executor: label) — starting headless run

🤖 dispatch: routing: executor=claude (default — no executor:<name> label) — starting headless run
mathias commented 2026-08-24 21:51:08 +00:00
Author
Owner
Copy Link
Copy Source

🤖 dispatch: executor claude failed: runner error: agent run ended in error (see stderr)

🤖 dispatch: executor claude failed: runner error: agent run ended in error (see stderr)
mathias commented 2026-08-26 20:29:37 +00:00
Author
Owner
Copy Link
Copy Source

Closing — this was done and marked "recommend close" back on 2026-07-06 (both comments above), fully shipped in mcp-chassis v0.4.0 + live-proven on gitea-mcp. My dispatch label just now was a mistake — I should have checked live state before labeling (same lesson as the missing_edge cluster earlier this sprint); it wasted a run for no reason since the actual work already existed. Closing for real this time. Session close, 2026-08-26.

Closing — this was done and marked "recommend close" back on 2026-07-06 (both comments above), fully shipped in `mcp-chassis` v0.4.0 + live-proven on gitea-mcp. My dispatch label just now was a mistake — I should have checked live state before labeling (same lesson as the missing_edge cluster earlier this sprint); it wasted a run for no reason since the actual work already existed. Closing for real this time. Session close, 2026-08-26.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agent/ready
Meets the agent-ready contract; authorizes unattended execution
 blocked/client
Waiting on the client
 blocked/supplier
Waiting on a supplier
 blocked/upstream
Waiting on an upstream project or vendor
 risk/high
Irreversible, security-relevant, or control plane
 risk/low
Reversible, local, test-covered
 risk/medium
Touches shared state, deploys or data
 size/l
Too big - split it
 size/m
Under two days
 size/s
Under half a day
 size/xs
Under an hour
 status:dispatch
status:dispatched
status:review
status:reviewing
type/chore
Maintenance, no behaviour change
 type/decision
Human decision required; never agent/ready
 type/docs
Documentation only
 type/feat
New capability
 type/fix
Corrects broken behaviour
 type/ops
Infrastructure, deploy, runtime
 type/research
Investigate; output is a finding, not code
 type/test
Test coverage only
No labels status:dispatch status:dispatched
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
renovate-bot mathias
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/hyperguild#77