mathias/hyperguild
1
0
Fork 0
Code Issues 30 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

The audit ntfy notifier is dark in production — BRAIN_NTFY_URL is unset on ingestion #100

New Issue
Closed
opened 2026-08-22 06:22:50 +00:00 by mathias · 2 comments
mathias commented 2026-08-22 06:22:50 +00:00
Owner
Copy Link
Copy Source

Found by live-verifying #97 against production at v0.15.0, which is exactly what that verification was for.

What is broken

buildNotifier() returns a nil interface when BRAIN_NTFY_URL is empty, and every consumer skips a nil notifier silently — by design, so an unconfigured homelab does not error. The ingestion deployment sets BRAIN_LOKI_URL but not BRAIN_NTFY_URL:

$ kubectl get deploy -n supervisor ingestion -o jsonpath='{...env[*].name}'
BRAIN_LOKI_URL      ← set
BRAIN_NTFY_URL      ← absent
BRAIN_NTFY_TOKEN    ← absent

So two alerts that look shipped are not:

  1. #97's governance-rewrite alert. Verified live: two real brain_edit_root calls landed both audit records in loki with correct before/after hashes, and produced no notification. The audit half works; the alert half is dark.
  2. The capture-audit degradation alert — silent since it shipped. DegradingSink.Record fires the "audit BUFFERED LOCALLY (loki unreachable)" notification through the same nil notifier. That is the alert whose entire job is to tell you the central audit trail is down, and it has never been able to fire. Pre-existing; #97 only made it visible.

#97's premise — "brain-sync already posts to ntfy on failure, so the wiring and topic exist" — was true of brain-sync, a different component with its own config. The ingestion deployment never had it.

DECISION NEEDED — which topic, and how does it authenticate?

ntfy-acl-bootstrap.yaml already provisions candidates, and the estate has precedent in both directions:

  • A) Anonymous write-only to a dedicated topic, mirroring brain-sync exactly. That file's own comment records the reasoning: "Anonymous write-only to this one topic avoids cross-namespace cred plumbing; the ntfy Service is ClusterIP-only so only in-cluster pods can reach it." Would need ntfy access everyone brain-audit wo (or a new brain-governance topic) plus one env var. Cheapest, and consistent with the last two decisions of this shape.
  • B) Authenticate as the existing brain-audit user. Already has wo on brain-audit, but authenticates via a standalone access token living in brain-gardener's gitea secret — so this means cross-namespace credential plumbing into supervisor. The same file flags that user as created ad-hoc and never codified ("exactly the evaporation anti-pattern infra#40 warned about").

Recommendation: A, with a dedicated brain-governance topic rather than reusing brain-audit. A governance rewrite is rare and must be read; audit-degradation alerts are operational noise by comparison. Mixing them on one topic is how the rare one gets muted. That does mean two env vars eventually, or accepting that both share BRAIN_NTFY_URL until the notifier takes a per-event topic.

Not acting on this unilaterally — it widens write access on a shared notification service and picks a topic taxonomy.

Acceptance

  • caveman: alarm actually make noise when someone change rules
  • BRAIN_NTFY_URL set on the ingestion deployment via the infra repo (GitOps, not kubectl edit)
  • ACL grant added to ntfy-acl-bootstrap.yaml — codified, not applied ad-hoc
  • Live proof: one real brain_edit_root produces a notification on a phone, containing path + both hashes and no file content
  • Live proof for the degradation path too: with loki unreachable, a capture buffers and alerts

Risk: LOW to fix. The current state is the risk — a security control whose alert cannot fire, and an outage alert that cannot report the outage.

Refs #97, #96, ingestion/cmd/server/main.go (buildNotifier), internal/audit/degrading.go, infra k3s/apps/monitoring/ntfy-acl-bootstrap.yaml.

Found by live-verifying #97 against production at `v0.15.0`, which is exactly what that verification was for. ## What is broken `buildNotifier()` returns a **nil interface** when `BRAIN_NTFY_URL` is empty, and every consumer skips a nil notifier silently — by design, so an unconfigured homelab does not error. The `ingestion` deployment sets `BRAIN_LOKI_URL` but **not** `BRAIN_NTFY_URL`: ``` $ kubectl get deploy -n supervisor ingestion -o jsonpath='{...env[*].name}' BRAIN_LOKI_URL ← set BRAIN_NTFY_URL ← absent BRAIN_NTFY_TOKEN ← absent ``` So two alerts that look shipped are not: 1. **#97's governance-rewrite alert.** Verified live: two real `brain_edit_root` calls landed both audit records in loki with correct before/after hashes, and produced **no notification**. The audit half works; the alert half is dark. 2. **The capture-audit degradation alert — silent since it shipped.** `DegradingSink.Record` fires the "audit BUFFERED LOCALLY (loki unreachable)" notification through the same nil notifier. That is the alert whose entire job is to tell you the central audit trail is down, and it has never been able to fire. Pre-existing; #97 only made it visible. #97's premise — *"brain-sync already posts to ntfy on failure, so the wiring and topic exist"* — was true of **brain-sync**, a different component with its own config. The `ingestion` deployment never had it. ## DECISION NEEDED — which topic, and how does it authenticate? `ntfy-acl-bootstrap.yaml` already provisions candidates, and the estate has precedent in both directions: - **A) Anonymous write-only to a dedicated topic**, mirroring brain-sync exactly. That file's own comment records the reasoning: *"Anonymous write-only to this one topic avoids cross-namespace cred plumbing; the ntfy Service is ClusterIP-only so only in-cluster pods can reach it."* Would need `ntfy access everyone brain-audit wo` (or a new `brain-governance` topic) plus one env var. Cheapest, and consistent with the last two decisions of this shape. - **B) Authenticate as the existing `brain-audit` user.** Already has `wo` on `brain-audit`, but authenticates via a standalone access token living in brain-gardener's gitea secret — so this means cross-namespace credential plumbing into `supervisor`. The same file flags that user as created ad-hoc and never codified ("exactly the evaporation anti-pattern infra#40 warned about"). **Recommendation: A**, with a **dedicated `brain-governance` topic** rather than reusing `brain-audit`. A governance rewrite is rare and must be read; audit-degradation alerts are operational noise by comparison. Mixing them on one topic is how the rare one gets muted. That does mean two env vars eventually, or accepting that both share `BRAIN_NTFY_URL` until the notifier takes a per-event topic. Not acting on this unilaterally — it widens write access on a shared notification service and picks a topic taxonomy. ## Acceptance - [ ] `caveman: alarm actually make noise when someone change rules` - [ ] `BRAIN_NTFY_URL` set on the `ingestion` deployment via the infra repo (GitOps, not `kubectl edit`) - [ ] ACL grant added to `ntfy-acl-bootstrap.yaml` — codified, not applied ad-hoc - [ ] Live proof: one real `brain_edit_root` produces a notification on a phone, containing path + both hashes and no file content - [ ] Live proof for the degradation path too: with loki unreachable, a capture buffers **and** alerts **Risk:** LOW to fix. The current state is the risk — a security control whose alert cannot fire, and an outage alert that cannot report the outage. Refs #97, #96, `ingestion/cmd/server/main.go` (`buildNotifier`), `internal/audit/degrading.go`, infra `k3s/apps/monitoring/ntfy-acl-bootstrap.yaml`.
mathias commented 2026-08-22 08:50:33 +00:00
Author
Owner
Copy Link
Copy Source

Decided and built: option A, one topic — infra PR #297

Taking the DECISION NEEDED on the "keep it simple, keep it robust" principle.

Anonymous write-only to a dedicated brain-alerts topic. Not the brain-audit credential plumbed cross-namespace.

  • Simple: one env var, no credential to mint, rotate, or sync into another namespace.
  • Robust: ntfy is ClusterIP-only, so only in-cluster pods can reach it — the same reasoning already recorded in ntfy-acl-bootstrap.yaml for brain-sync. Option B additionally depends on a user that file itself flags as created ad-hoc and never codified.

One topic, not two, which reverses the taxonomy I proposed earlier. Both alerts are rare, and a second env var is a second thing to forget — which is precisely the failure being fixed here. Still deliberately off brain-audit: a rare "someone rewrote the rules" alert must not share a channel with routine gardener digest traffic, or it gets muted along with it.

The ACL grant is codified in the bootstrap script rather than applied by hand.

This issue does not close on merge

Config is not delivery — that is the entire lesson here. After Flux reconciles:

  1. Pod restarted with BRAIN_NTFY_URL present
  2. One real brain_edit_root → a notification actually arrives on your phone (path + both hashes, no content)
  3. Ideally the degradation path too: loki unreachable → capture buffers and alerts

Only step 2 closes this. I can drive 1 and 3; step 2 needs you to confirm the message landed, since the failure mode is silence and I cannot observe your phone.

Worth its own issue

buildNotifier() returning nil silently is right for an unconfigured homelab and wrong for a security-relevant alert. A loud-on-unconfigured startup line — governance alerting DISABLED: BRAIN_NTFY_URL unset — would have made this a five-second discovery instead of a live-fire test. That is a code change in this repo; say the word and I'll file it.

## Decided and built: option A, one topic — infra PR #297 Taking the DECISION NEEDED on the "keep it simple, keep it robust" principle. **Anonymous write-only to a dedicated `brain-alerts` topic.** Not the `brain-audit` credential plumbed cross-namespace. - *Simple:* one env var, no credential to mint, rotate, or sync into another namespace. - *Robust:* `ntfy` is ClusterIP-only, so only in-cluster pods can reach it — the same reasoning already recorded in `ntfy-acl-bootstrap.yaml` for brain-sync. Option B additionally depends on a user that file itself flags as created ad-hoc and never codified. **One topic, not two,** which reverses the taxonomy I proposed earlier. Both alerts are rare, and a second env var is a second thing to forget — which is precisely the failure being fixed here. Still deliberately **off** `brain-audit`: a rare "someone rewrote the rules" alert must not share a channel with routine gardener digest traffic, or it gets muted along with it. The ACL grant is codified in the bootstrap script rather than applied by hand. ## This issue does not close on merge Config is not delivery — that is the entire lesson here. After Flux reconciles: 1. Pod restarted with `BRAIN_NTFY_URL` present 2. **One real `brain_edit_root` → a notification actually arrives on your phone** (path + both hashes, no content) 3. Ideally the degradation path too: loki unreachable → capture buffers *and* alerts Only step 2 closes this. I can drive 1 and 3; step 2 needs you to confirm the message landed, since the failure mode is silence and I cannot observe your phone. ## Worth its own issue `buildNotifier()` returning nil silently is right for an unconfigured homelab and wrong for a security-relevant alert. A **loud-on-unconfigured startup line** — `governance alerting DISABLED: BRAIN_NTFY_URL unset` — would have made this a five-second discovery instead of a live-fire test. That is a code change in this repo; say the word and I'll file it.
mathias commented 2026-08-23 05:29:07 +00:00
Author
Owner
Copy Link
Copy Source

Confirmed received. Closing.

A real brain_edit_root on CLAUDE.md at 21:28 local (a601ab7f → 9d0056f9) produced a notification that arrived on the phone. That was the one step nobody in this chain could verify but Mathias, and it is the step that decides whether the issue is fixed.

Full chain now verified end to end, each hop with its own evidence rather than inference:

Hop Evidence
gate fires WARN brain_edit_root applied principal=local-cli
audit record queried loki: actor: brain_edit_root, items: [CLAUDE.md], before/after hashes chained
env var live BRAIN_NTFY_URL=…/brain-alerts on the running pod
ACL permits publish POST /brain-alerts 403 → 200 after the acl-revision bump
ntfy accepts no notify failed in the pod log, i.e. Notify returned nil
delivered message on the phone

Two corrections worth keeping on the record

The first "confirmation" was the wrong topic. A control message seen at 10:59 was mine, posted to brain-sync-alert — the known-good control. It proved the mechanism worked and said nothing about brain-alerts. Nearly closed this on it.

The ACL grant needed more than the ConfigMap edit. The bootstrap runs as an initContainer, and the deployment's own annotation must be bumped to re-run it. Editing the ConfigMap changed the script, not the state it had produced — so the grant sat inert on a 10-day-old pod and anonymous publish returned 403. Caught by testing the publish path rather than reading the manifest; fixed in infra dfc5940.

Both are the same shape as the bug this issue is about: configuration present, effect absent.

Left open deliberately, filed separately

ntfy's topic history returns empty even as admin with since=all, despite a PVC and a cache file being configured — so the phone is currently the only observer of this channel, and an alert fired while it is offline cannot be recovered or audited after the fact. Unexplained, and not something to close this issue over; filed on its own.

The startup warning that would have made this a five-second discovery instead of a live-fire test is #107, shipped in PR #108.

## Confirmed received. Closing. A real `brain_edit_root` on `CLAUDE.md` at 21:28 local (`a601ab7f → 9d0056f9`) produced a notification that **arrived on the phone**. That was the one step nobody in this chain could verify but Mathias, and it is the step that decides whether the issue is fixed. Full chain now verified end to end, each hop with its own evidence rather than inference: | Hop | Evidence | |---|---| | gate fires | `WARN brain_edit_root applied principal=local-cli` | | audit record | queried loki: `actor: brain_edit_root`, `items: [CLAUDE.md]`, before/after hashes chained | | env var live | `BRAIN_NTFY_URL=…/brain-alerts` on the running pod | | ACL permits publish | `POST /brain-alerts` 403 → 200 after the `acl-revision` bump | | ntfy accepts | no `notify failed` in the pod log, i.e. `Notify` returned nil | | **delivered** | **message on the phone** | ## Two corrections worth keeping on the record **The first "confirmation" was the wrong topic.** A `control` message seen at 10:59 was mine, posted to `brain-sync-alert` — the known-good control. It proved the mechanism worked and said nothing about `brain-alerts`. Nearly closed this on it. **The ACL grant needed more than the ConfigMap edit.** The bootstrap runs as an initContainer, and the deployment's own annotation must be bumped to re-run it. Editing the ConfigMap changed the script, not the state it had produced — so the grant sat inert on a 10-day-old pod and anonymous publish returned 403. Caught by testing the publish path rather than reading the manifest; fixed in infra `dfc5940`. Both are the same shape as the bug this issue is about: configuration present, effect absent. ## Left open deliberately, filed separately ntfy's topic history returns empty even as admin with `since=all`, despite a PVC and a cache file being configured — so the phone is currently the only observer of this channel, and an alert fired while it is offline cannot be recovered or audited after the fact. Unexplained, and not something to close this issue over; filed on its own. The startup warning that would have made this a five-second discovery instead of a live-fire test is #107, shipped in PR #108.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agent/ready
Meets the agent-ready contract; authorizes unattended execution
 blocked/client
Waiting on the client
 blocked/supplier
Waiting on a supplier
 blocked/upstream
Waiting on an upstream project or vendor
 risk/high
Irreversible, security-relevant, or control plane
 risk/low
Reversible, local, test-covered
 risk/medium
Touches shared state, deploys or data
 size/l
Too big - split it
 size/m
Under two days
 size/s
Under half a day
 size/xs
Under an hour
 status:dispatch
status:dispatched
status:review
status:reviewing
type/chore
Maintenance, no behaviour change
 type/decision
Human decision required; never agent/ready
 type/docs
Documentation only
 type/feat
New capability
 type/fix
Corrects broken behaviour
 type/ops
Infrastructure, deploy, runtime
 type/research
Investigate; output is a finding, not code
 type/test
Test coverage only
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
renovate-bot mathias
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/hyperguild#100