mathiasandClaude Opus 5
9a232ad444
On a cluster restart this pod can beat dex to serving, gets a 502 from ingress, and exits — after connecting postgres, backfilling 794 graph nodes and ingesting three session batches, all of which the restart redoes. Nothing was misconfigured; it started first. The validator now retries with bounded exponential backoff (8 attempts, 1s doubling to 30s) and is built immediately after ctx, before the watcher and the backfill. That second part is the other half of the issue: a genuine config error now fails in seconds rather than after all that work. Bounded on purpose. Retrying forever would convert a wrong issuer URL into a process that looks alive and never serves, which is harder to diagnose than a crash loop. startupretry.Do is generic and tested directly: transient-then-success, budget exhaustion surfacing the last error via errors.Is, prompt return on context cancellation, growing-and-capped delays, and a zero-value policy still making one attempt rather than silently succeeding. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BeAp5LTscnz2W7Ubt4eJ6m
76 lines
2.1 KiB
Go
76 lines
2.1 KiB
Go
// Package startupretry retries a startup dependency that may not be serving
|
|
// yet, instead of exiting on its first failure.
|
|
//
|
|
// It exists for the cold-boot ordering race (#83): on a cluster restart the
|
|
// ingestion pod can reach OIDC discovery before dex is answering, gets a 502
|
|
// from ingress, and dies — after having already connected postgres, backfilled
|
|
// the graph and ingested session batches, all of which the restart redoes.
|
|
// Nothing is misconfigured in that scenario; the pod simply started first.
|
|
//
|
|
// The budget is bounded on purpose. Retrying forever would turn a genuine
|
|
// misconfiguration (wrong issuer URL, bad audience) into a process that looks
|
|
// healthy and never serves, which is harder to diagnose than a crash loop.
|
|
package startupretry
|
|
|
|
import (
|
|
"context"
|
|
"fmt"
|
|
"time"
|
|
)
|
|
|
|
// Policy bounds the retry: at most Attempts calls, with the delay doubling
|
|
// from Base and capped at Max.
|
|
type Policy struct {
|
|
Attempts int // total calls, including the first; < 1 means 1
|
|
Base time.Duration // delay before the second attempt
|
|
Max time.Duration // ceiling for the doubling delay
|
|
}
|
|
|
|
// Do calls fn until it succeeds, the attempt budget is spent, or ctx ends.
|
|
//
|
|
// The returned error wraps the last failure, so callers keep the diagnosis
|
|
// they would have had without the retry.
|
|
func Do[T any](ctx context.Context, p Policy, fn func(context.Context) (T, error)) (T, error) {
|
|
var zero T
|
|
|
|
attempts := p.Attempts
|
|
if attempts < 1 {
|
|
attempts = 1
|
|
}
|
|
|
|
delay := p.Base
|
|
var lastErr error
|
|
|
|
for i := range attempts {
|
|
if err := ctx.Err(); err != nil {
|
|
return zero, fmt.Errorf("startupretry: %w", err)
|
|
}
|
|
|
|
v, err := fn(ctx)
|
|
if err == nil {
|
|
return v, nil
|
|
}
|
|
lastErr = err
|
|
|
|
if i == attempts-1 {
|
|
break
|
|
}
|
|
|
|
if delay <= 0 {
|
|
delay = time.Second
|
|
}
|
|
if p.Max > 0 && delay > p.Max {
|
|
delay = p.Max
|
|
}
|
|
|
|
select {
|
|
case <-ctx.Done():
|
|
return zero, fmt.Errorf("startupretry: %w", ctx.Err())
|
|
case <-time.After(delay):
|
|
}
|
|
delay *= 2
|
|
}
|
|
|
|
return zero, fmt.Errorf("startupretry: gave up after %d attempts: %w", attempts, lastErr)
|
|
}
|