mathias/hyperguild
1
0
Fork 0
Code Issues 30 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

chore(security): rotate brain_app postgres password + migrate BRAIN_PG_DSN from SOPS to 1Password via ESO #30

New Issue
Closed
opened 2026-05-27 10:53:54 +00:00 by mathias · 1 comment
mathias commented 2026-05-27 10:53:54 +00:00
Owner
Copy Link
Copy Source

Context

Supersedes #20. The original cmd/server/main.go redaction for BRAIN_PG_DSN was wrong — it sliced up to @, which is after the password in a Postgres URL, so the password landed in the structured log line of the ingestion pod under image 7a13c756... on 2026-05-18 → 2026-05-19. Fixed in 4af10364 via url.URL.Redacted(). Old log lines remain in kubectl logs --previous history and any sink container logs ship to until retention rolls off (typically 7–14 days; longer if shipped to a SIEM).

Since #20 was filed, External Secrets Operator + 1Password Connect went live (ClusterSecretStore/onepassword, HomeLab vault). BRAIN_PG_DSN is still on the old SOPS path (k3s/apps/supervisor/secrets.enc.yaml) and referenced as secretKeyRef: supervisor-secrets in ingestion-deployment.yaml. Rotate + migrate in the same change so future rotations are 1P-only with no git commit needed.

Action

  1. Generate new password: 
    umask 077 && openssl rand -hex 24 > /tmp/brain_app_pg.pwd
  2. Apply via init SQL (idempotent ALTER ROLE): 
    kubectl exec -i -n databases postgres18-0 -- \
  psql -U postgres -v password="$(cat /tmp/brain_app_pg.pwd)" \
  < scripts/brain-embeddings-init.sql
    (Bare value, no surrounding single quotes — :'password' does its own quoting.)
  3. Store the full DSN in 1Password — vault HomeLab, item BRAIN_PG_DSN, field password: 
    postgres://brain_app:<new-pwd>@postgres18.databases.svc.cluster.local:5432/brain?sslmode=disable
  4. Create k3s/apps/supervisor/brain-pg-dsn-externalsecret.yaml mirroring brain-mcp-token-externalsecret.yaml: 
    apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: brain-pg-dsn
  namespace: supervisor
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: onepassword
    kind: ClusterSecretStore
  target:
    name: brain-pg-dsn
    creationPolicy: Owner
  data:
    - secretKey: BRAIN_PG_DSN
      remoteRef:
        key: BRAIN_PG_DSN
        property: password
  5. Update ingestion-deployment.yaml BRAIN_PG_DSN env block: secretKeyRef.name: supervisor-secrets → brain-pg-dsn.
  6. Remove BRAIN_PG_DSN from k3s/apps/supervisor/secrets.enc.yaml (re-sops without the key; CLAUDE_INGEST_CLIENT_BLOCK stays — separate migration).
  7. Add the new ExternalSecret to k3s/apps/supervisor/kustomization.yaml.
  8. Bump secrets-revision annotation on ingestion-deployment.yaml to roll the pod.
  9. Commit (one logical change), watch Flux reconcile (flux get kustomization apps --watch).
  10. rm /tmp/brain_app_pg.pwd.

Acceptance criteria

  • New brain_app password generated, full DSN stored only in 1Password HomeLab vault
  • BRAIN_PG_DSN removed from secrets.enc.yaml
  • kubectl get externalsecret -n supervisor brain-pg-dsn shows SecretSynced=True
  • Pod restart picks up the new DSN cleanly (brain hybrid retrieval enabled log line, no auth errors)
  • /tmp/brain_app_pg.pwd deleted
  • Optional: extra kubectl delete pod after retention concern eases to flush leaked log lines sooner

Follow-up (out of scope)

  • Migrate CLAUDE_INGEST_CLIENT_BLOCK to 1Password, retire secrets.enc.yaml entirely.

Severity

Low — the leak was internal-only (kubectl logs require cluster admin), and the affected DB sits behind Tailscale. Worth doing as hygiene, not urgent. Migration win is the real value: future rotations become a 1P field edit, no commit.

## Context Supersedes #20. The original `cmd/server/main.go` redaction for `BRAIN_PG_DSN` was wrong — it sliced up to `@`, which is *after* the password in a Postgres URL, so the password landed in the structured log line of the ingestion pod under image `7a13c756...` on 2026-05-18 → 2026-05-19. Fixed in `4af10364` via `url.URL.Redacted()`. Old log lines remain in `kubectl logs --previous` history and any sink container logs ship to until retention rolls off (typically 7–14 days; longer if shipped to a SIEM). Since #20 was filed, External Secrets Operator + 1Password Connect went live (`ClusterSecretStore/onepassword`, HomeLab vault). `BRAIN_PG_DSN` is still on the old SOPS path (`k3s/apps/supervisor/secrets.enc.yaml`) and referenced as `secretKeyRef: supervisor-secrets` in `ingestion-deployment.yaml`. Rotate + migrate in the same change so future rotations are 1P-only with no git commit needed. ## Action 1. Generate new password: ```bash umask 077 && openssl rand -hex 24 > /tmp/brain_app_pg.pwd ``` 2. Apply via init SQL (idempotent `ALTER ROLE`): ```bash kubectl exec -i -n databases postgres18-0 -- \ psql -U postgres -v password="$(cat /tmp/brain_app_pg.pwd)" \ < scripts/brain-embeddings-init.sql ``` (Bare value, no surrounding single quotes — `:'password'` does its own quoting.) 3. Store the full DSN in 1Password — vault `HomeLab`, item `BRAIN_PG_DSN`, field `password`: ``` postgres://brain_app:<new-pwd>@postgres18.databases.svc.cluster.local:5432/brain?sslmode=disable ``` 4. Create `k3s/apps/supervisor/brain-pg-dsn-externalsecret.yaml` mirroring `brain-mcp-token-externalsecret.yaml`: ```yaml apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: name: brain-pg-dsn namespace: supervisor spec: refreshInterval: 1h secretStoreRef: name: onepassword kind: ClusterSecretStore target: name: brain-pg-dsn creationPolicy: Owner data: - secretKey: BRAIN_PG_DSN remoteRef: key: BRAIN_PG_DSN property: password ``` 5. Update `ingestion-deployment.yaml` BRAIN_PG_DSN env block: `secretKeyRef.name: supervisor-secrets` → `brain-pg-dsn`. 6. Remove `BRAIN_PG_DSN` from `k3s/apps/supervisor/secrets.enc.yaml` (re-sops without the key; `CLAUDE_INGEST_CLIENT_BLOCK` stays — separate migration). 7. Add the new ExternalSecret to `k3s/apps/supervisor/kustomization.yaml`. 8. Bump `secrets-revision` annotation on `ingestion-deployment.yaml` to roll the pod. 9. Commit (one logical change), watch Flux reconcile (`flux get kustomization apps --watch`). 10. `rm /tmp/brain_app_pg.pwd`. ## Acceptance criteria - [ ] New `brain_app` password generated, full DSN stored only in 1Password HomeLab vault - [ ] `BRAIN_PG_DSN` removed from `secrets.enc.yaml` - [ ] `kubectl get externalsecret -n supervisor brain-pg-dsn` shows `SecretSynced=True` - [ ] Pod restart picks up the new DSN cleanly (`brain hybrid retrieval enabled` log line, no auth errors) - [ ] `/tmp/brain_app_pg.pwd` deleted - [ ] Optional: extra `kubectl delete pod` after retention concern eases to flush leaked log lines sooner ## Follow-up (out of scope) - Migrate `CLAUDE_INGEST_CLIENT_BLOCK` to 1Password, retire `secrets.enc.yaml` entirely. ## Severity Low — the leak was internal-only (kubectl logs require cluster admin), and the affected DB sits behind Tailscale. Worth doing as hygiene, not urgent. Migration win is the real value: future rotations become a 1P field edit, no commit.
mathias commented 2026-05-27 11:18:30 +00:00
Author
Owner
Copy Link
Copy Source

Done — commit b048594 on infra/main.

  • DB password rotated via ALTER ROLE brain_app in postgres18-0, verified with psql -U brain_app -d brain (1 row)
  • DSN stored in 1P HomeLab vault, item BRAIN_PG_DSN, field password
  • brain-pg-dsn ExternalSecret reconciled SecretSynced=True
  • BRAIN_PG_DSN removed from secrets.enc.yaml, CLAUDE_INGEST_CLIENT_BLOCK retained
  • Ingestion pod rolled, log line: brain hybrid retrieval enabled with redacted DSN (xxxxx)
  • embed sync added=14 deleted=0 errors=0 — DB writes working
  • Flux apps Kustomization Ready=True at revision b0485949
  • /tmp/brain_app_pg.pwd shredded

Acceptance criteria met. Closing.

Follow-up: migrate CLAUDE_INGEST_CLIENT_BLOCK to 1P and retire secrets.enc.yaml (separate issue).

Done — commit `b048594` on infra/main. - DB password rotated via `ALTER ROLE brain_app` in `postgres18-0`, verified with `psql -U brain_app -d brain` (1 row) - DSN stored in 1P HomeLab vault, item `BRAIN_PG_DSN`, field `password` - `brain-pg-dsn` ExternalSecret reconciled `SecretSynced=True` - `BRAIN_PG_DSN` removed from `secrets.enc.yaml`, `CLAUDE_INGEST_CLIENT_BLOCK` retained - Ingestion pod rolled, log line: `brain hybrid retrieval enabled` with redacted DSN (`xxxxx`) - `embed sync added=14 deleted=0 errors=0` — DB writes working - Flux `apps` Kustomization `Ready=True` at revision `b0485949` - `/tmp/brain_app_pg.pwd` shredded Acceptance criteria met. Closing. Follow-up: migrate `CLAUDE_INGEST_CLIENT_BLOCK` to 1P and retire `secrets.enc.yaml` (separate issue).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agent/ready
Meets the agent-ready contract; authorizes unattended execution
 blocked/client
Waiting on the client
 blocked/supplier
Waiting on a supplier
 blocked/upstream
Waiting on an upstream project or vendor
 risk/high
Irreversible, security-relevant, or control plane
 risk/low
Reversible, local, test-covered
 risk/medium
Touches shared state, deploys or data
 size/l
Too big - split it
 size/m
Under two days
 size/s
Under half a day
 size/xs
Under an hour
 status:dispatch
status:dispatched
status:review
status:reviewing
type/chore
Maintenance, no behaviour change
 type/decision
Human decision required; never agent/ready
 type/docs
Documentation only
 type/feat
New capability
 type/fix
Corrects broken behaviour
 type/ops
Infrastructure, deploy, runtime
 type/research
Investigate; output is a finding, not code
 type/test
Test coverage only
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
renovate-bot mathias
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/hyperguild#30