mathias/hyperguild
1
0
Fork 0
Code Issues 30 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

chore(security): rotate brain_app postgres password — leaked in pre-fix pod logs #20

New Issue
Closed
opened 2026-05-19 11:10:55 +00:00 by mathias · 2 comments
mathias commented 2026-05-19 11:10:55 +00:00
Owner
Copy Link
Copy Source

Context

The original cmd/server/main.go "redaction" for the BRAIN_PG_DSN log line was wrong — it sliced up to @, which is after the password in a Postgres URL, so the password landed in the structured log line of the ingestion pod under image 7a13c756... on 2026-05-18 → 2026-05-19.

Fixed in 4af10364 via url.URL.Redacted() (now prints xxxxx). But the old log lines remain in kubectl logs --previous history and any sink the cluster logs are shipped to until that retention rolls off (typically 7–14 days for container runtime; longer if logs are shipped to a SIEM).

Action

Rotate brain_app postgres password end-to-end:

  1. Generate new password: umask 077 && openssl rand -hex 24 > /tmp/brain_app_pg.pwd
  2. Apply via init SQL (the script is idempotent — \if :role_exists ALTER ROLE ... WITH PASSWORD): 
    kubectl exec -i -n databases postgres18-0 -- \
  psql -U postgres -v password="$(cat /tmp/brain_app_pg.pwd)" \
  < scripts/brain-embeddings-init.sql
    (Note: bare value, NO surrounding single quotes — :'password' does its own quoting.)
  3. Rebuild DSN and SOPS-set: 
    DSN="postgres://brain_app:$(cat /tmp/brain_app_pg.pwd)@postgres18.databases.svc.cluster.local:5432/brain?sslmode=disable"
cd ~/dev/AI/infra
sops set k3s/apps/supervisor/secrets.enc.yaml '["stringData"]["BRAIN_PG_DSN"]' "\"$DSN\""
  4. Commit the SOPS change and bump secrets-revision on ingestion-deployment.yaml to roll the pod.
  5. rm /tmp/brain_app_pg.pwd

Acceptance criteria

  • New brain_app password generated and stored only in SOPS
  • Pod restart picks up the new DSN cleanly (brain hybrid retrieval enabled line + no auth errors)
  • /tmp/brain_app_pg.pwd deleted
  • Optional: old pod-log entries containing the leaked password purged sooner via kubectl delete pod once more after retention concern eases

Severity

Low — the leak was internal-only (kubectl logs require cluster admin), and the affected DB sits behind Tailscale. Worth doing as hygiene, not urgent.

## Context The original `cmd/server/main.go` "redaction" for the `BRAIN_PG_DSN` log line was wrong — it sliced up to `@`, which is *after* the password in a Postgres URL, so the password landed in the structured log line of the ingestion pod under image `7a13c756...` on 2026-05-18 → 2026-05-19. Fixed in `4af10364` via `url.URL.Redacted()` (now prints `xxxxx`). But the old log lines remain in `kubectl logs --previous` history and any sink the cluster logs are shipped to until that retention rolls off (typically 7–14 days for container runtime; longer if logs are shipped to a SIEM). ## Action Rotate `brain_app` postgres password end-to-end: 1. Generate new password: `umask 077 && openssl rand -hex 24 > /tmp/brain_app_pg.pwd` 2. Apply via init SQL (the script is idempotent — `\if :role_exists ALTER ROLE ... WITH PASSWORD`): ```bash kubectl exec -i -n databases postgres18-0 -- \ psql -U postgres -v password="$(cat /tmp/brain_app_pg.pwd)" \ < scripts/brain-embeddings-init.sql ``` (Note: bare value, NO surrounding single quotes — `:'password'` does its own quoting.) 3. Rebuild DSN and SOPS-set: ```bash DSN="postgres://brain_app:$(cat /tmp/brain_app_pg.pwd)@postgres18.databases.svc.cluster.local:5432/brain?sslmode=disable" cd ~/dev/AI/infra sops set k3s/apps/supervisor/secrets.enc.yaml '["stringData"]["BRAIN_PG_DSN"]' "\"$DSN\"" ``` 4. Commit the SOPS change and bump `secrets-revision` on `ingestion-deployment.yaml` to roll the pod. 5. `rm /tmp/brain_app_pg.pwd` ## Acceptance criteria - [ ] New `brain_app` password generated and stored only in SOPS - [ ] Pod restart picks up the new DSN cleanly (`brain hybrid retrieval enabled` line + no auth errors) - [ ] `/tmp/brain_app_pg.pwd` deleted - [ ] Optional: old pod-log entries containing the leaked password purged sooner via `kubectl delete pod` once more after retention concern eases ## Severity Low — the leak was internal-only (kubectl logs require cluster admin), and the affected DB sits behind Tailscale. Worth doing as hygiene, not urgent.
mathias commented 2026-05-27 10:53:59 +00:00
Author
Owner
Copy Link
Copy Source

Closing — procedure was pre-ESO. Rewritten as #30 with ExternalSecret + 1Password Connect migration baked into the rotation so future rotations become a 1P field edit instead of a SOPS commit.

Closing — procedure was pre-ESO. Rewritten as #30 with ExternalSecret + 1Password Connect migration baked into the rotation so future rotations become a 1P field edit instead of a SOPS commit.
mathias commented 2026-05-27 11:24:29 +00:00
Author
Owner
Copy Link
Copy Source

✅ Superseded and completed — 2026-05-27

This issue (brain_app postgres rotation) was superseded by the broader ESO migration work in hyperguild #30.

Migration complete per commit 7481687:

  • claude-ingest-client-block ExternalSecret synced via 1Password/ESO
  • supervisor-secrets Secret pruned by Flux — gone
  • secrets.enc.yaml deleted from repo entirely
  • Pod rolled clean: claudewatcher client-block guard registered ✓
  • Memory updated: reference_homelab_client_list.md + MEMORY.md reflect 1P/ESO location

Supervisor namespace is now 100% ESO/1Password-managed, zero SOPS.
Future client list edits: op item edit only.

Closing as superseded.

## ✅ Superseded and completed — 2026-05-27 This issue (brain_app postgres rotation) was superseded by the broader ESO migration work in hyperguild #30. Migration complete per commit `7481687`: - `claude-ingest-client-block` ExternalSecret synced via 1Password/ESO - `supervisor-secrets` Secret pruned by Flux — gone - `secrets.enc.yaml` deleted from repo entirely - Pod rolled clean: `claudewatcher client-block guard registered` ✓ - Memory updated: reference_homelab_client_list.md + MEMORY.md reflect 1P/ESO location **Supervisor namespace is now 100% ESO/1Password-managed, zero SOPS.** Future client list edits: `op item edit` only. Closing as superseded.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agent/ready
Meets the agent-ready contract; authorizes unattended execution
 blocked/client
Waiting on the client
 blocked/supplier
Waiting on a supplier
 blocked/upstream
Waiting on an upstream project or vendor
 risk/high
Irreversible, security-relevant, or control plane
 risk/low
Reversible, local, test-covered
 risk/medium
Touches shared state, deploys or data
 size/l
Too big - split it
 size/m
Under two days
 size/s
Under half a day
 size/xs
Under an hour
 status:dispatch
status:dispatched
status:review
status:reviewing
type/chore
Maintenance, no behaviour change
 type/decision
Human decision required; never agent/ready
 type/docs
Documentation only
 type/feat
New capability
 type/fix
Corrects broken behaviour
 type/ops
Infrastructure, deploy, runtime
 type/research
Investigate; output is a finding, not code
 type/test
Test coverage only
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
renovate-bot mathias
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/hyperguild#20