mathias/hyperguild
1
0
Fork 0
Code Issues 30 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

ingestion: retry OIDC discovery instead of exiting on cold-boot 502 #83

New Issue
Closed
opened 2026-07-17 12:15:08 +00:00 by mathias · 4 comments
mathias commented 2026-07-17 12:15:08 +00:00
Owner
Copy Link
Copy Source

Problem

supervisor/ingestion exits on startup if OIDC discovery fails, instead of retrying. On a cold cluster boot it races ahead of dex and dies.

Observed 2026-07-17 after a koala reboot — CrashLoopBackOff, 4 restarts before it caught:

{"level":"INFO","msg":"graph backfill complete","indexed":794}
{"level":"INFO","msg":"claudewatcher started","sessions_dir":"/host-home-claude/projects","host":"koala"}
{"level":"INFO","msg":"claudewatcher: ingested batch","turns_kept":20,...}
{"level":"ERROR","msg":"build jwt validator","err":"oidc discovery: status 502"}

Note the ordering: it completes all its real work — postgres hybrid retrieval connected, graph backfill of 794 nodes, three session batches ingested — and only then dies on the JWT validator. Every restart redoes that work.

The 502 is ingress-nginx failing to reach dex, which wasn't serving yet. Nothing was misconfigured; ingestion just started first.

Why it matters

It self-heals via CrashLoopBackOff (it recovered once dex came up), so this is low severity in effect. But:

  • Backoff is exponential — a slower dex start means a longer brain outage.
  • Every attempt re-runs graph backfill and re-ingests batches. Wasteful, and worth confirming it's idempotent (offsets suggest yes, but unverified).
  • The failure reads as a hard error in logs when it's a benign ordering race, which costs diagnosis time.

Prior art

The brain already has this pattern: knowledge/flux-kustomization-depends-on-bootstrap-ordering.md — "prerequisite resources must be wrapped in their own Kustomization with dependsOn entries on any Kustomization that requires them". This is another instance of the same cold-bootstrap race, at app level rather than Flux level.

Proposed fix

Retry OIDC discovery with bounded exponential backoff at startup rather than exiting on first failure. A transient 502 from a not-yet-ready auth upstream is expected during cold boot, not fatal.

Options:

  1. App-level retry (preferred) — retry discovery for a bounded window before giving up. Fixes it regardless of orchestration.
  2. Flux dependsOn — order the ingestion Kustomization after dex/ingress. Consistent with the recorded prior art, but doesn't help outside Flux and doesn't cover a dex restart at runtime.
  3. Defer validator construction — build the JWT validator lazily on first authenticated request instead of at boot.

(1) or (3); they compose.

Secondary

Consider constructing the JWT validator before the expensive graph backfill and ingestion work, so a fatal config error fails fast instead of after several seconds of redone work each cycle.

Acceptance criterion

A cold cluster boot brings ingestion to 1/1 Running with 0 restarts. Regression test: OIDC discovery returning 502 on first N attempts then succeeding should not crash the process.

Source

ingestion/internal/claudewatcher + cmd/server/main.go in this repo. (mathias/ingestion-svc is a tombstone — "superseded by hyperguild/ingestion", 2026-05-14.)

## Problem `supervisor/ingestion` exits on startup if OIDC discovery fails, instead of retrying. On a cold cluster boot it races ahead of dex and dies. Observed 2026-07-17 after a koala reboot — CrashLoopBackOff, 4 restarts before it caught: ``` {"level":"INFO","msg":"graph backfill complete","indexed":794} {"level":"INFO","msg":"claudewatcher started","sessions_dir":"/host-home-claude/projects","host":"koala"} {"level":"INFO","msg":"claudewatcher: ingested batch","turns_kept":20,...} {"level":"ERROR","msg":"build jwt validator","err":"oidc discovery: status 502"} ``` Note the ordering: it completes all its real work — postgres hybrid retrieval connected, graph backfill of 794 nodes, three session batches ingested — and only *then* dies on the JWT validator. Every restart redoes that work. The `502` is ingress-nginx failing to reach dex, which wasn't serving yet. Nothing was misconfigured; ingestion just started first. ## Why it matters It self-heals via CrashLoopBackOff (it recovered once dex came up), so this is low severity in effect. But: - Backoff is exponential — a slower dex start means a longer brain outage. - Every attempt re-runs graph backfill and re-ingests batches. Wasteful, and worth confirming it's idempotent (offsets suggest yes, but unverified). - The failure reads as a hard error in logs when it's a benign ordering race, which costs diagnosis time. ## Prior art The brain already has this pattern: `knowledge/flux-kustomization-depends-on-bootstrap-ordering.md` — *"prerequisite resources must be wrapped in their own Kustomization with `dependsOn` entries on any Kustomization that requires them"*. This is another instance of the same cold-bootstrap race, at app level rather than Flux level. ## Proposed fix Retry OIDC discovery with bounded exponential backoff at startup rather than exiting on first failure. A transient 502 from a not-yet-ready auth upstream is expected during cold boot, not fatal. Options: 1. **App-level retry** (preferred) — retry discovery for a bounded window before giving up. Fixes it regardless of orchestration. 2. **Flux `dependsOn`** — order the ingestion Kustomization after dex/ingress. Consistent with the recorded prior art, but doesn't help outside Flux and doesn't cover a dex restart at runtime. 3. **Defer validator construction** — build the JWT validator lazily on first authenticated request instead of at boot. (1) or (3); they compose. ## Secondary Consider constructing the JWT validator *before* the expensive graph backfill and ingestion work, so a fatal config error fails fast instead of after several seconds of redone work each cycle. ## Acceptance criterion A cold cluster boot brings ingestion to `1/1 Running` with 0 restarts. Regression test: OIDC discovery returning 502 on first N attempts then succeeding should not crash the process. ## Source `ingestion/internal/claudewatcher` + `cmd/server/main.go` in this repo. (`mathias/ingestion-svc` is a tombstone — "superseded by hyperguild/ingestion", 2026-05-14.)
mathias commented 2026-08-17 22:22:16 +00:00
Author
Owner
Copy Link
Copy Source

Fixed with (1) app-level retry + (3) fail-fast ordering — 9a232ad. They compose, as the issue predicted, and each fixes a different half.

Retry: 8 attempts, 1s doubling to a 30s cap, wrapping chassisauth.NewJWTValidator. A 502 from a not-yet-serving dex is now a WARN line per attempt instead of os.Exit(1).

Ordering (the secondary ask): the validator is built immediately after ctx, ahead of the watcher, the graph backfill and claudewatcher. So a real config error — wrong DEX_ISSUER_URL, wrong audience — now fails in seconds instead of after 794 nodes and three session batches.

Why the budget stays bounded

Retrying forever is the tempting version and it is worse. A wrong issuer URL would become a process that is Running, answers no authenticated request, and never says why — strictly harder to diagnose than the crash loop this issue is about. The bound keeps the loud failure for the failure that deserves it, and removes it for the benign race.

startupretry.Do is generic and tested directly

main() wiring is not testable, so the logic is a package. Five properties pinned, red first:

  • transient failures then success → no crash, exact call count
  • budget exhausted → error wraps the last failure, so errors.Is still finds the original diagnosis
  • context cancelled mid-wait → returns promptly with context.Canceled, not after the full backoff
  • delays grow and stay under the cap
  • zero-value policy still makes one attempt — a caller who forgets to configure it gets the failure, not a silent no-op returning (nil, nil)

That last one matters here specifically because NewJWTValidator returns (nil, nil) when DEX_ISSUER_URL is empty. A retry helper that quietly did nothing would be indistinguishable from auth-disabled.

Not verified

The acceptance criterion — cold cluster boot reaching 1/1 Running with 0 restarts — needs a koala reboot, which is not something to do unattended. task check is green and the unit test reproduces the exact 502-then-success sequence; the real confirmation is the next cold boot.

Also unverified, and still worth checking when someone is at a terminal: whether the redone graph backfill and batch ingestion were genuinely idempotent. The offsets suggested yes, and after this change a restart is far less likely to happen at all — but "less likely to happen" is not the same as "was safe when it did", and this issue is the only place that question is written down.

Leaving open until a cold boot confirms it, since that is the stated acceptance.

Fixed with **(1) app-level retry + (3) fail-fast ordering** — `9a232ad`. They compose, as the issue predicted, and each fixes a different half. **Retry**: 8 attempts, 1s doubling to a 30s cap, wrapping `chassisauth.NewJWTValidator`. A 502 from a not-yet-serving dex is now a `WARN` line per attempt instead of `os.Exit(1)`. **Ordering (the secondary ask)**: the validator is built immediately after `ctx`, ahead of the watcher, the graph backfill and claudewatcher. So a real config error — wrong `DEX_ISSUER_URL`, wrong audience — now fails in seconds instead of after 794 nodes and three session batches. ### Why the budget stays bounded Retrying forever is the tempting version and it is worse. A wrong issuer URL would become a process that is `Running`, answers no authenticated request, and never says why — strictly harder to diagnose than the crash loop this issue is about. The bound keeps the loud failure for the failure that deserves it, and removes it for the benign race. ### `startupretry.Do` is generic and tested directly `main()` wiring is not testable, so the logic is a package. Five properties pinned, red first: - transient failures then success → no crash, exact call count - budget exhausted → error wraps the last failure, so `errors.Is` still finds the original diagnosis - context cancelled mid-wait → returns promptly with `context.Canceled`, not after the full backoff - delays grow and stay under the cap - **zero-value policy still makes one attempt** — a caller who forgets to configure it gets the failure, not a silent no-op returning `(nil, nil)` That last one matters here specifically because `NewJWTValidator` returns `(nil, nil)` when `DEX_ISSUER_URL` is empty. A retry helper that quietly did nothing would be indistinguishable from auth-disabled. ### Not verified The acceptance criterion — cold cluster boot reaching `1/1 Running` with 0 restarts — needs a koala reboot, which is not something to do unattended. `task check` is green and the unit test reproduces the exact 502-then-success sequence; the real confirmation is the next cold boot. Also unverified, and still worth checking when someone is at a terminal: whether the redone graph backfill and batch ingestion were genuinely idempotent. The offsets suggested yes, and after this change a restart is far less likely to happen at all — but "less likely to happen" is not the same as "was safe when it did", and this issue is the only place that question is written down. Leaving open until a cold boot confirms it, since that is the stated acceptance.
mathias added the status:dispatch label 2026-08-24 21:49:56 +00:00
mathias added the status:dispatched label 2026-08-24 21:51:01 +00:00
mathias commented 2026-08-24 21:51:01 +00:00
Author
Owner
Copy Link
Copy Source

🤖 dispatch: routing: executor=claude (default — no executor: label) — starting headless run

🤖 dispatch: routing: executor=claude (default — no executor:<name> label) — starting headless run
mathias commented 2026-08-24 21:51:03 +00:00
Author
Owner
Copy Link
Copy Source

🤖 dispatch: executor claude failed: runner error: agent run ended in error (see stderr)

🤖 dispatch: executor claude failed: runner error: agent run ended in error (see stderr)
mathias added the risk/mediumsize/stype/fix labels 2026-08-27 05:25:42 +00:00
mathias removed the status:dispatch label 2026-08-27 05:31:46 +00:00
mathias removed the status:dispatched label 2026-10-05 05:03:04 +00:00
mathias commented 2026-10-05 05:03:41 +00:00
Author
Owner
Copy Link
Copy Source

Closed by 432e09d. The retry itself shipped in 9a232ad; this commit adds the stated regression test against a real issuer. I extracted main()'s closure into buildJWTValidator without changing behaviour, and tested it with an httptest OIDC issuer:

  • 3× 502 then 200: succeeds after exactly 4 discoveries
  • a 502 on every attempt: stops at the 8-attempt budget and returns the last error (status 502)
  • ctx cancelled during backoff: returns context.Canceled promptly

Flip test: setting Attempts=1 turns all three red. CI=1 task check exits 0.

Still not verified: the live acceptance, a cold koala boot reaching 1/1 Running with 0 restarts. That needs a reboot, which I won't do unattended. I also removed the stale status:dispatched label.

Created via git-mcp on behalf of @mathias

**Closed by `432e09d`.** The retry itself shipped in `9a232ad`; this commit adds the stated regression test against a real issuer. I extracted `main()`'s closure into `buildJWTValidator` without changing behaviour, and tested it with an `httptest` OIDC issuer: - 3× 502 then 200: succeeds after exactly 4 discoveries - a 502 on every attempt: stops at the 8-attempt budget and returns the last error (`status 502`) - ctx cancelled during backoff: returns `context.Canceled` promptly Flip test: setting `Attempts=1` turns all three red. `CI=1 task check` exits 0. **Still not verified:** the live acceptance, a cold koala boot reaching 1/1 Running with 0 restarts. That needs a reboot, which I won't do unattended. I also removed the stale `status:dispatched` label. --- _Created via git-mcp on behalf of @mathias_
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agent/ready
Meets the agent-ready contract; authorizes unattended execution
 blocked/client
Waiting on the client
 blocked/supplier
Waiting on a supplier
 blocked/upstream
Waiting on an upstream project or vendor
 risk/high
Irreversible, security-relevant, or control plane
 risk/low
Reversible, local, test-covered
 risk/medium
Touches shared state, deploys or data
 size/l
Too big - split it
 size/m
Under two days
 size/s
Under half a day
 size/xs
Under an hour
 status:dispatch
status:dispatched
status:review
status:reviewing
type/chore
Maintenance, no behaviour change
 type/decision
Human decision required; never agent/ready
 type/docs
Documentation only
 type/feat
New capability
 type/fix
Corrects broken behaviour
 type/ops
Infrastructure, deploy, runtime
 type/research
Investigate; output is a finding, not code
 type/test
Test coverage only
No labels risk/medium size/s type/fix
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
renovate-bot mathias
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/hyperguild#83