fix(ingestion): tripwire alerts survive outages; drop only permanent rejections
PR #160 round 2: dropping a message after 5 failures could not tell a poison message from an ntfy outage. A 15-minute outage dropped the only incident with nothing reported, and drops were not counted (74 of 300 removals were neither delivered nor counted). - audit.NtfyNotifier returns *RejectedError for non-2xx; Permanent() is true for 4xx other than 429. - The tripwire queue drops a message only on a permanent rejection and counts it; every other failure is an outage and is retried, however long. Each flush tries every queued message, so an undeliverable one cannot block the rest. - Overflow and drops go into one counter, sent as a summary retried every tick and never dropped. Every removal is delivered or counted. Tests (red first): an outage of ~100 ticks still delivers the only incident; delivered + counted equals the number of removals; a permanently rejected message is dropped and counted. The short-outage test is renamed to say it is short. The Watch doc no longer claims 'loses none' unconditionally. Refs #157 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -49,7 +49,19 @@ func (n *NtfyNotifier) Notify(ctx context.Context, msg string) error {
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
return fmt.Errorf("ntfy notify: status %d", resp.StatusCode)
|
||||
return &RejectedError{Status: resp.StatusCode}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// RejectedError is a non-2xx ntfy response.
|
||||
type RejectedError struct{ Status int }
|
||||
|
||||
func (e *RejectedError) Error() string { return fmt.Sprintf("ntfy notify: status %d", e.Status) }
|
||||
|
||||
// Permanent reports whether resending the same message cannot succeed: a 4xx
|
||||
// other than 429 (rate limited). Callers may drop such a message; anything
|
||||
// else (5xx, 429, transport errors) is an outage and must be retried.
|
||||
func (e *RejectedError) Permanent() bool {
|
||||
return e.Status >= 400 && e.Status < 500 && e.Status != http.StatusTooManyRequests
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user