ingestion: retry OIDC discovery instead of exiting on cold-boot 502 #83

Open
opened 2026-07-17 12:15:08 +00:00 by mathias · 0 comments
Owner

Problem

supervisor/ingestion exits on startup if OIDC discovery fails, instead of retrying. On a cold cluster boot it races ahead of dex and dies.

Observed 2026-07-17 after a koala reboot — CrashLoopBackOff, 4 restarts before it caught:

{"level":"INFO","msg":"graph backfill complete","indexed":794}
{"level":"INFO","msg":"claudewatcher started","sessions_dir":"/host-home-claude/projects","host":"koala"}
{"level":"INFO","msg":"claudewatcher: ingested batch","turns_kept":20,...}
{"level":"ERROR","msg":"build jwt validator","err":"oidc discovery: status 502"}

Note the ordering: it completes all its real work — postgres hybrid retrieval connected, graph backfill of 794 nodes, three session batches ingested — and only then dies on the JWT validator. Every restart redoes that work.

The 502 is ingress-nginx failing to reach dex, which wasn't serving yet. Nothing was misconfigured; ingestion just started first.

Why it matters

It self-heals via CrashLoopBackOff (it recovered once dex came up), so this is low severity in effect. But:

  • Backoff is exponential — a slower dex start means a longer brain outage.
  • Every attempt re-runs graph backfill and re-ingests batches. Wasteful, and worth confirming it's idempotent (offsets suggest yes, but unverified).
  • The failure reads as a hard error in logs when it's a benign ordering race, which costs diagnosis time.

Prior art

The brain already has this pattern: knowledge/flux-kustomization-depends-on-bootstrap-ordering.md"prerequisite resources must be wrapped in their own Kustomization with dependsOn entries on any Kustomization that requires them". This is another instance of the same cold-bootstrap race, at app level rather than Flux level.

Proposed fix

Retry OIDC discovery with bounded exponential backoff at startup rather than exiting on first failure. A transient 502 from a not-yet-ready auth upstream is expected during cold boot, not fatal.

Options:

  1. App-level retry (preferred) — retry discovery for a bounded window before giving up. Fixes it regardless of orchestration.
  2. Flux dependsOn — order the ingestion Kustomization after dex/ingress. Consistent with the recorded prior art, but doesn't help outside Flux and doesn't cover a dex restart at runtime.
  3. Defer validator construction — build the JWT validator lazily on first authenticated request instead of at boot.

(1) or (3); they compose.

Secondary

Consider constructing the JWT validator before the expensive graph backfill and ingestion work, so a fatal config error fails fast instead of after several seconds of redone work each cycle.

Acceptance criterion

A cold cluster boot brings ingestion to 1/1 Running with 0 restarts. Regression test: OIDC discovery returning 502 on first N attempts then succeeding should not crash the process.

Source

ingestion/internal/claudewatcher + cmd/server/main.go in this repo. (mathias/ingestion-svc is a tombstone — "superseded by hyperguild/ingestion", 2026-05-14.)

## Problem `supervisor/ingestion` exits on startup if OIDC discovery fails, instead of retrying. On a cold cluster boot it races ahead of dex and dies. Observed 2026-07-17 after a koala reboot — CrashLoopBackOff, 4 restarts before it caught: ``` {"level":"INFO","msg":"graph backfill complete","indexed":794} {"level":"INFO","msg":"claudewatcher started","sessions_dir":"/host-home-claude/projects","host":"koala"} {"level":"INFO","msg":"claudewatcher: ingested batch","turns_kept":20,...} {"level":"ERROR","msg":"build jwt validator","err":"oidc discovery: status 502"} ``` Note the ordering: it completes all its real work — postgres hybrid retrieval connected, graph backfill of 794 nodes, three session batches ingested — and only *then* dies on the JWT validator. Every restart redoes that work. The `502` is ingress-nginx failing to reach dex, which wasn't serving yet. Nothing was misconfigured; ingestion just started first. ## Why it matters It self-heals via CrashLoopBackOff (it recovered once dex came up), so this is low severity in effect. But: - Backoff is exponential — a slower dex start means a longer brain outage. - Every attempt re-runs graph backfill and re-ingests batches. Wasteful, and worth confirming it's idempotent (offsets suggest yes, but unverified). - The failure reads as a hard error in logs when it's a benign ordering race, which costs diagnosis time. ## Prior art The brain already has this pattern: `knowledge/flux-kustomization-depends-on-bootstrap-ordering.md` — *"prerequisite resources must be wrapped in their own Kustomization with `dependsOn` entries on any Kustomization that requires them"*. This is another instance of the same cold-bootstrap race, at app level rather than Flux level. ## Proposed fix Retry OIDC discovery with bounded exponential backoff at startup rather than exiting on first failure. A transient 502 from a not-yet-ready auth upstream is expected during cold boot, not fatal. Options: 1. **App-level retry** (preferred) — retry discovery for a bounded window before giving up. Fixes it regardless of orchestration. 2. **Flux `dependsOn`** — order the ingestion Kustomization after dex/ingress. Consistent with the recorded prior art, but doesn't help outside Flux and doesn't cover a dex restart at runtime. 3. **Defer validator construction** — build the JWT validator lazily on first authenticated request instead of at boot. (1) or (3); they compose. ## Secondary Consider constructing the JWT validator *before* the expensive graph backfill and ingestion work, so a fatal config error fails fast instead of after several seconds of redone work each cycle. ## Acceptance criterion A cold cluster boot brings ingestion to `1/1 Running` with 0 restarts. Regression test: OIDC discovery returning 502 on first N attempts then succeeding should not crash the process. ## Source `ingestion/internal/claudewatcher` + `cmd/server/main.go` in this repo. (`mathias/ingestion-svc` is a tombstone — "superseded by hyperguild/ingestion", 2026-05-14.)
Sign in to join this conversation.
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/hyperguild#83