Note the ordering: it completes all its real work — postgres hybrid retrieval connected, graph backfill of 794 nodes, three session batches ingested — and only then dies on the JWT validator. Every restart redoes that work.
The 502 is ingress-nginx failing to reach dex, which wasn't serving yet. Nothing was misconfigured; ingestion just started first.
Why it matters
It self-heals via CrashLoopBackOff (it recovered once dex came up), so this is low severity in effect. But:
Backoff is exponential — a slower dex start means a longer brain outage.
Every attempt re-runs graph backfill and re-ingests batches. Wasteful, and worth confirming it's idempotent (offsets suggest yes, but unverified).
The failure reads as a hard error in logs when it's a benign ordering race, which costs diagnosis time.
Prior art
The brain already has this pattern: knowledge/flux-kustomization-depends-on-bootstrap-ordering.md — "prerequisite resources must be wrapped in their own Kustomization with dependsOn entries on any Kustomization that requires them". This is another instance of the same cold-bootstrap race, at app level rather than Flux level.
Proposed fix
Retry OIDC discovery with bounded exponential backoff at startup rather than exiting on first failure. A transient 502 from a not-yet-ready auth upstream is expected during cold boot, not fatal.
Options:
App-level retry (preferred) — retry discovery for a bounded window before giving up. Fixes it regardless of orchestration.
Flux dependsOn — order the ingestion Kustomization after dex/ingress. Consistent with the recorded prior art, but doesn't help outside Flux and doesn't cover a dex restart at runtime.
Defer validator construction — build the JWT validator lazily on first authenticated request instead of at boot.
(1) or (3); they compose.
Secondary
Consider constructing the JWT validator before the expensive graph backfill and ingestion work, so a fatal config error fails fast instead of after several seconds of redone work each cycle.
Acceptance criterion
A cold cluster boot brings ingestion to 1/1 Running with 0 restarts. Regression test: OIDC discovery returning 502 on first N attempts then succeeding should not crash the process.
Source
ingestion/internal/claudewatcher + cmd/server/main.go in this repo. (mathias/ingestion-svc is a tombstone — "superseded by hyperguild/ingestion", 2026-05-14.)
## Problem
`supervisor/ingestion` exits on startup if OIDC discovery fails, instead of retrying. On a cold cluster boot it races ahead of dex and dies.
Observed 2026-07-17 after a koala reboot — CrashLoopBackOff, 4 restarts before it caught:
```
{"level":"INFO","msg":"graph backfill complete","indexed":794}
{"level":"INFO","msg":"claudewatcher started","sessions_dir":"/host-home-claude/projects","host":"koala"}
{"level":"INFO","msg":"claudewatcher: ingested batch","turns_kept":20,...}
{"level":"ERROR","msg":"build jwt validator","err":"oidc discovery: status 502"}
```
Note the ordering: it completes all its real work — postgres hybrid retrieval connected, graph backfill of 794 nodes, three session batches ingested — and only *then* dies on the JWT validator. Every restart redoes that work.
The `502` is ingress-nginx failing to reach dex, which wasn't serving yet. Nothing was misconfigured; ingestion just started first.
## Why it matters
It self-heals via CrashLoopBackOff (it recovered once dex came up), so this is low severity in effect. But:
- Backoff is exponential — a slower dex start means a longer brain outage.
- Every attempt re-runs graph backfill and re-ingests batches. Wasteful, and worth confirming it's idempotent (offsets suggest yes, but unverified).
- The failure reads as a hard error in logs when it's a benign ordering race, which costs diagnosis time.
## Prior art
The brain already has this pattern: `knowledge/flux-kustomization-depends-on-bootstrap-ordering.md` — *"prerequisite resources must be wrapped in their own Kustomization with `dependsOn` entries on any Kustomization that requires them"*. This is another instance of the same cold-bootstrap race, at app level rather than Flux level.
## Proposed fix
Retry OIDC discovery with bounded exponential backoff at startup rather than exiting on first failure. A transient 502 from a not-yet-ready auth upstream is expected during cold boot, not fatal.
Options:
1. **App-level retry** (preferred) — retry discovery for a bounded window before giving up. Fixes it regardless of orchestration.
2. **Flux `dependsOn`** — order the ingestion Kustomization after dex/ingress. Consistent with the recorded prior art, but doesn't help outside Flux and doesn't cover a dex restart at runtime.
3. **Defer validator construction** — build the JWT validator lazily on first authenticated request instead of at boot.
(1) or (3); they compose.
## Secondary
Consider constructing the JWT validator *before* the expensive graph backfill and ingestion work, so a fatal config error fails fast instead of after several seconds of redone work each cycle.
## Acceptance criterion
A cold cluster boot brings ingestion to `1/1 Running` with 0 restarts. Regression test: OIDC discovery returning 502 on first N attempts then succeeding should not crash the process.
## Source
`ingestion/internal/claudewatcher` + `cmd/server/main.go` in this repo. (`mathias/ingestion-svc` is a tombstone — "superseded by hyperguild/ingestion", 2026-05-14.)
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Problem
supervisor/ingestionexits on startup if OIDC discovery fails, instead of retrying. On a cold cluster boot it races ahead of dex and dies.Observed 2026-07-17 after a koala reboot — CrashLoopBackOff, 4 restarts before it caught:
Note the ordering: it completes all its real work — postgres hybrid retrieval connected, graph backfill of 794 nodes, three session batches ingested — and only then dies on the JWT validator. Every restart redoes that work.
The
502is ingress-nginx failing to reach dex, which wasn't serving yet. Nothing was misconfigured; ingestion just started first.Why it matters
It self-heals via CrashLoopBackOff (it recovered once dex came up), so this is low severity in effect. But:
Prior art
The brain already has this pattern:
knowledge/flux-kustomization-depends-on-bootstrap-ordering.md— "prerequisite resources must be wrapped in their own Kustomization withdependsOnentries on any Kustomization that requires them". This is another instance of the same cold-bootstrap race, at app level rather than Flux level.Proposed fix
Retry OIDC discovery with bounded exponential backoff at startup rather than exiting on first failure. A transient 502 from a not-yet-ready auth upstream is expected during cold boot, not fatal.
Options:
dependsOn— order the ingestion Kustomization after dex/ingress. Consistent with the recorded prior art, but doesn't help outside Flux and doesn't cover a dex restart at runtime.(1) or (3); they compose.
Secondary
Consider constructing the JWT validator before the expensive graph backfill and ingestion work, so a fatal config error fails fast instead of after several seconds of redone work each cycle.
Acceptance criterion
A cold cluster boot brings ingestion to
1/1 Runningwith 0 restarts. Regression test: OIDC discovery returning 502 on first N attempts then succeeding should not crash the process.Source
ingestion/internal/claudewatcher+cmd/server/main.goin this repo. (mathias/ingestion-svcis a tombstone — "superseded by hyperguild/ingestion", 2026-05-14.)