The original cmd/server/main.go "redaction" for the BRAIN_PG_DSN log line was wrong — it sliced up to @, which is after the password in a Postgres URL, so the password landed in the structured log line of the ingestion pod under image 7a13c756... on 2026-05-18 → 2026-05-19.
Fixed in 4af10364 via url.URL.Redacted() (now prints xxxxx). But the old log lines remain in kubectl logs --previous history and any sink the cluster logs are shipped to until that retention rolls off (typically 7–14 days for container runtime; longer if logs are shipped to a SIEM).
(Note: bare value, NO surrounding single quotes — :'password' does its own quoting.)
Rebuild DSN and SOPS-set:
DSN="postgres://brain_app:$(cat /tmp/brain_app_pg.pwd)@postgres18.databases.svc.cluster.local:5432/brain?sslmode=disable"cd ~/dev/AI/infra
sops set k3s/apps/supervisor/secrets.enc.yaml '["stringData"]["BRAIN_PG_DSN"]'"\"$DSN\""
Commit the SOPS change and bump secrets-revision on ingestion-deployment.yaml to roll the pod.
rm /tmp/brain_app_pg.pwd
Acceptance criteria
New brain_app password generated and stored only in SOPS
Pod restart picks up the new DSN cleanly (brain hybrid retrieval enabled line + no auth errors)
/tmp/brain_app_pg.pwd deleted
Optional: old pod-log entries containing the leaked password purged sooner via kubectl delete pod once more after retention concern eases
Severity
Low — the leak was internal-only (kubectl logs require cluster admin), and the affected DB sits behind Tailscale. Worth doing as hygiene, not urgent.
## Context
The original `cmd/server/main.go` "redaction" for the `BRAIN_PG_DSN` log line was wrong — it sliced up to `@`, which is *after* the password in a Postgres URL, so the password landed in the structured log line of the ingestion pod under image `7a13c756...` on 2026-05-18 → 2026-05-19.
Fixed in `4af10364` via `url.URL.Redacted()` (now prints `xxxxx`). But the old log lines remain in `kubectl logs --previous` history and any sink the cluster logs are shipped to until that retention rolls off (typically 7–14 days for container runtime; longer if logs are shipped to a SIEM).
## Action
Rotate `brain_app` postgres password end-to-end:
1. Generate new password: `umask 077 && openssl rand -hex 24 > /tmp/brain_app_pg.pwd`
2. Apply via init SQL (the script is idempotent — `\if :role_exists ALTER ROLE ... WITH PASSWORD`):
```bash
kubectl exec -i -n databases postgres18-0 -- \
psql -U postgres -v password="$(cat /tmp/brain_app_pg.pwd)" \
< scripts/brain-embeddings-init.sql
```
(Note: bare value, NO surrounding single quotes — `:'password'` does its own quoting.)
3. Rebuild DSN and SOPS-set:
```bash
DSN="postgres://brain_app:$(cat /tmp/brain_app_pg.pwd)@postgres18.databases.svc.cluster.local:5432/brain?sslmode=disable"
cd ~/dev/AI/infra
sops set k3s/apps/supervisor/secrets.enc.yaml '["stringData"]["BRAIN_PG_DSN"]' "\"$DSN\""
```
4. Commit the SOPS change and bump `secrets-revision` on `ingestion-deployment.yaml` to roll the pod.
5. `rm /tmp/brain_app_pg.pwd`
## Acceptance criteria
- [ ] New `brain_app` password generated and stored only in SOPS
- [ ] Pod restart picks up the new DSN cleanly (`brain hybrid retrieval enabled` line + no auth errors)
- [ ] `/tmp/brain_app_pg.pwd` deleted
- [ ] Optional: old pod-log entries containing the leaked password purged sooner via `kubectl delete pod` once more after retention concern eases
## Severity
Low — the leak was internal-only (kubectl logs require cluster admin), and the affected DB sits behind Tailscale. Worth doing as hygiene, not urgent.
Closing — procedure was pre-ESO. Rewritten as #30 with ExternalSecret + 1Password Connect migration baked into the rotation so future rotations become a 1P field edit instead of a SOPS commit.
Closing — procedure was pre-ESO. Rewritten as #30 with ExternalSecret + 1Password Connect migration baked into the rotation so future rotations become a 1P field edit instead of a SOPS commit.
Supervisor namespace is now 100% ESO/1Password-managed, zero SOPS.
Future client list edits: op item edit only.
Closing as superseded.
## ✅ Superseded and completed — 2026-05-27
This issue (brain_app postgres rotation) was superseded by the broader ESO migration work in hyperguild #30.
Migration complete per commit `7481687`:
- `claude-ingest-client-block` ExternalSecret synced via 1Password/ESO
- `supervisor-secrets` Secret pruned by Flux — gone
- `secrets.enc.yaml` deleted from repo entirely
- Pod rolled clean: `claudewatcher client-block guard registered` ✓
- Memory updated: reference_homelab_client_list.md + MEMORY.md reflect 1P/ESO location
**Supervisor namespace is now 100% ESO/1Password-managed, zero SOPS.**
Future client list edits: `op item edit` only.
Closing as superseded.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Context
The original
cmd/server/main.go"redaction" for theBRAIN_PG_DSNlog line was wrong — it sliced up to@, which is after the password in a Postgres URL, so the password landed in the structured log line of the ingestion pod under image7a13c756...on 2026-05-18 → 2026-05-19.Fixed in
4af10364viaurl.URL.Redacted()(now printsxxxxx). But the old log lines remain inkubectl logs --previoushistory and any sink the cluster logs are shipped to until that retention rolls off (typically 7–14 days for container runtime; longer if logs are shipped to a SIEM).Action
Rotate
brain_apppostgres password end-to-end:umask 077 && openssl rand -hex 24 > /tmp/brain_app_pg.pwd\if :role_exists ALTER ROLE ... WITH PASSWORD)::'password'does its own quoting.)secrets-revisiononingestion-deployment.yamlto roll the pod.rm /tmp/brain_app_pg.pwdAcceptance criteria
brain_apppassword generated and stored only in SOPSbrain hybrid retrieval enabledline + no auth errors)/tmp/brain_app_pg.pwddeletedkubectl delete podonce more after retention concern easesSeverity
Low — the leak was internal-only (kubectl logs require cluster admin), and the affected DB sits behind Tailscale. Worth doing as hygiene, not urgent.
Closing — procedure was pre-ESO. Rewritten as #30 with ExternalSecret + 1Password Connect migration baked into the rotation so future rotations become a 1P field edit instead of a SOPS commit.
✅ Superseded and completed — 2026-05-27
This issue (brain_app postgres rotation) was superseded by the broader ESO migration work in hyperguild #30.
Migration complete per commit
7481687:claude-ingest-client-blockExternalSecret synced via 1Password/ESOsupervisor-secretsSecret pruned by Flux — gonesecrets.enc.yamldeleted from repo entirelyclaudewatcher client-block guard registered✓Supervisor namespace is now 100% ESO/1Password-managed, zero SOPS.
Future client list edits:
op item editonly.Closing as superseded.