Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ac84ddf5a9 | ||
|
|
db18785bfc | ||
|
|
138d6e2545 | ||
|
|
41b7105b31 |
@@ -84,15 +84,3 @@ When acting as a coding agent on this project:
|
|||||||
4. Never modify files outside the project root without explicit permission
|
4. Never modify files outside the project root without explicit permission
|
||||||
5. When adding a dependency, explain why in the commit message
|
5. When adding a dependency, explain why in the commit message
|
||||||
6. For client projects: never send code or context to cloud APIs — use local models via LiteLLM
|
6. For client projects: never send code or context to cloud APIs — use local models via LiteLLM
|
||||||
|
|
||||||
## `main` is protected: ship by PR (local-dev#42, decided 2026-10-08)
|
|
||||||
|
|
||||||
Gitea refuses direct pushes to `main` in this repo, and a PR merges only after its CI status (`CI / Lint / Test / Vet (pull_request)`) passes.
|
|
||||||
**Why:** local-dev rule 5's reviewed lane (risk/medium+ is reviewed by a human) was enforced on the merge tools
|
|
||||||
(`tbd_ship`, `pr_merge`: gitea-mcp v0.17.0/v0.18.0, merge only for an issue labelled exactly `risk/low` with green CI)
|
|
||||||
but not on a direct commit to `main`, which bypassed it. This overrides rule 5's "risk/low commits straight to main"
|
|
||||||
for this repo; control code is where the lane matters.
|
|
||||||
**How:** `tbd_ship` for a one-file change, or branch -> PR -> `pr_merge` (name the `risk/low` issue); above `risk/low`
|
|
||||||
open the PR and a human merges it in the Gitea UI after review. Tags are not protected. CD jobs push image tags to
|
|
||||||
the **infra** repo with deploy keys, never to this repo's `main`. Undo (Mathias only): Settings -> Branches, or
|
|
||||||
`PATCH /repos/mathias/hyperguild/branch_protections/main {"enable_push": true}`.
|
|
||||||
|
|||||||
+17
-81
@@ -164,63 +164,6 @@ These rules apply to every task across every project, regardless of harness.
|
|||||||
the thing it was named for was exploitable. If you cannot name the mechanism,
|
the thing it was named for was exploitable. If you cannot name the mechanism,
|
||||||
you do not yet know what the test observes.
|
you do not yet know what the test observes.
|
||||||
|
|
||||||
10. **Short sessions, one rc per repo, edit only in a locked worktree of the repo
|
|
||||||
you are changing** (ADR-0002, accepted 2026-10-06).
|
|
||||||
- One session per issue or task, ending with rule 6's close ritual. Related
|
|
||||||
issues may share a session only when they share a worktree and an owner.
|
|
||||||
State that must outlive the session goes in an issue, not the chat.
|
|
||||||
- Start `claude rc` in the repo you will edit (`infra`, `trucker-ecommerce`,
|
|
||||||
`hyperguild`, ...), not in `~/dev`: the rc worktree is a worktree of the
|
|
||||||
launch directory's repo, so launched from `~/dev` it protects nothing you
|
|
||||||
edit (measured: 73% of edits landed in shared checkouts). Start it only
|
|
||||||
while working there.
|
|
||||||
- Keep at most one thin `~/dev` coordinator session. It files issues and hands
|
|
||||||
out tasks; it reads and edits no sub-repo code.
|
|
||||||
- Split cross-repo work per repo. The contract between the halves lives in an
|
|
||||||
issue on each repo.
|
|
||||||
- Edit in a worktree of that repo, locked at creation:
|
|
||||||
`scripts/wt.sh add <repo> <task>` does it (fetch, `wt/<task>` from origin/main, lock);
|
|
||||||
`scripts/wt.sh done <repo>-<task>` removes it after the push and refuses dirty or
|
|
||||||
unpushed work. By hand: `git worktree add ~/worktrees/<name> -b <branch> origin/main`
|
|
||||||
then `git worktree lock --reason "manual: <purpose>" ~/worktrees/<name>`.
|
|
||||||
- Before working a repo another session may touch, announce your issue list to
|
|
||||||
that session (`SendMessage`) and ask for theirs.
|
|
||||||
- Auto-memory is keyed by cwd, so a repo rc starts without it. Cross-repo
|
|
||||||
working rules live in brain (`wiki/homelab/decisions/2026-10-06-062416-auto.md`, wing homelab, hall
|
|
||||||
decisions) and the section below, not in per-directory memory.
|
|
||||||
|
|
||||||
## Standing footguns
|
|
||||||
|
|
||||||
Each cost real time once. Detail and evidence: brain `wiki/homelab/decisions/2026-10-06-062416-auto.md` (title standing-working-rules).
|
|
||||||
|
|
||||||
- **Gate then commit with `&&`, never `;`.** "Green" means exit code 0 and
|
|
||||||
`grep -c FAIL` is 0, not a filtered grep.
|
|
||||||
- **Stage by path, never `git add -A` or `git add .`** (they stage from the repo
|
|
||||||
root and swept in other sessions' work twice). Check `git status --porcelain` first.
|
|
||||||
- **Before committing in a shared checkout:** `git branch --show-current`.
|
|
||||||
- **Before tagging:** `git fetch origin --tags`, then `git describe --tags --abbrev=0 <sha>`
|
|
||||||
and `git tag --sort=-v:refname | head`; go above both. Never take the last tag
|
|
||||||
from a report.
|
|
||||||
- **Run only what the project's gate runs.** No repo-wide formatter unless the gate
|
|
||||||
runs it.
|
|
||||||
- **Reproduce a CI red at the cited commit before applying an issue's prescribed fix.**
|
|
||||||
- **Read an issue's comments (`issue_list_comments`), not just its body, before working it.**
|
|
||||||
- **A comment naming a control ("behind branch protection") is not evidence; query
|
|
||||||
the control.** A working detector can hide a missing preventer.
|
|
||||||
- **On koala the MCP servers are `mcp__plugin_skills_{brain,gitea,infra}__*`.** The
|
|
||||||
per-turn auth reminder is not evidence; call the tool or run `claude mcp list`.
|
|
||||||
Brain writes always pass `wing` + `hall`.
|
|
||||||
- **`cat /etc/hostname` before assuming which host you are on.**
|
|
||||||
- **Never run bare `git remote -v` or `git config --list`** (url.insteadOf expands
|
|
||||||
into a live token); use `git config --get remote.origin.url`.
|
|
||||||
- **On koala `sudo` is denied outright; k8s exec, env dumps and flux reconcile need
|
|
||||||
per-action approval; decrypting a SOPS file and printing in one Bash call is
|
|
||||||
hook-blocked.** Do the repo-side work and hand the host-side step over in an issue.
|
|
||||||
- **`set -o pipefail` plus an optional `grep | head | awk` dies on no match:** append `|| true`.
|
|
||||||
- **Gitea `file_write_branch` takes plain text, not base64.**
|
|
||||||
- **`OP_SERVICE_ACCOUNT_TOKEN` in the shell is read-only for `op`.** Operational
|
|
||||||
secrets go in the AgentSecrets vault via `set -a; . ~/.op-agent-write.env; set +a`.
|
|
||||||
|
|
||||||
## CAD sprint patterns
|
## CAD sprint patterns
|
||||||
|
|
||||||
Two CAD sprint archetypes (defs: brain `wiki/homelab/decisions/continuous-agentic-development-cad-concept-2026-06-16`):
|
Two CAD sprint archetypes (defs: brain `wiki/homelab/decisions/continuous-agentic-development-cad-concept-2026-06-16`):
|
||||||
@@ -301,20 +244,25 @@ Three machines on Tailscale:
|
|||||||
|
|
||||||
All development repos live at `~/dev/` (softlink from `~/Documents/local-dev/`).
|
All development repos live at `~/dev/` (softlink from `~/Documents/local-dev/`).
|
||||||
|
|
||||||
Organized by **role**, because the role decides which rules apply (ADR-0003):
|
Organized in thematic folders:
|
||||||
|
|
||||||
| Folder | Role | Rules that key off it |
|
| Folder | Focus | Count |
|
||||||
|--------|------|-----------------------|
|
|--------|-------|-------|
|
||||||
| `platform/` | infra, brain, hyperguild, gitea-mcp, dispatch, shared tooling | control code (rule 8), GitOps review lane |
|
| `GO/` | Go web frameworks, API integrations, learning projects | ~10 |
|
||||||
| `products/` | tapir, cobalt-dingo, openbanking-api, ... | ordinary trunk-based work |
|
| `AI/` | ML research, AI frameworks (FinRL, DSPy, crawl4ai) | ~6 |
|
||||||
| `clients/` | client work | local models only, never cloud APIs (Client work rules) |
|
| `AGENTS/` | Autonomous agents, coding agents, MCP servers, infra | ~15 |
|
||||||
| `spikes/` | throwaway | no ceremony |
|
| `QKX/` | Invoice processing, financial automation, payment systems | ~13 |
|
||||||
|
| `XT/` | Climate data, sustainability (Klimatkollen, Garbo) | ~2 |
|
||||||
|
|
||||||
New projects are born into a role folder (`task new-project -- <name> <client|personal>
|
See `~/dev/PROJECT_SUMMARY.md` for detailed descriptions of each project.
|
||||||
<role-folder>`); a client tag requires `clients/` and `clients/` requires a client tag.
|
|
||||||
Existing repos still sit in the old `AGENTS/` and `AI/` folders until they are moved one by
|
### Key active projects
|
||||||
one (ADR-0003 step 3). `~/dev/PROJECT_SUMMARY.md` (generated) lists every repo and where it
|
|
||||||
is. How a session runs on koala (worktrees, rc per repo, the reaper, memory): `docs/working-on-koala.md`.
|
- **super-koala** (`AGENTS/`) — multi-component agent stack with LangGraph, DSPy, MCP
|
||||||
|
- **azure-tiger** (`QKX/`) — invoice extraction → ISO 20022 payment instructions
|
||||||
|
- **gocrwl** (`AGENTS/`) — Go web crawler with containerized deployment
|
||||||
|
- **koala-ai-stack** (`AGENTS/`) — local AI server infrastructure management
|
||||||
|
- **klimatkollen** (`XT/`) — Swedish municipal climate data platform
|
||||||
|
|
||||||
## Knowledge base — actively use it
|
## Knowledge base — actively use it
|
||||||
|
|
||||||
@@ -566,16 +514,4 @@ When acting as a coding agent on this project:
|
|||||||
5. When adding a dependency, explain why in the commit message
|
5. When adding a dependency, explain why in the commit message
|
||||||
6. For client projects: never send code or context to cloud APIs — use local models via LiteLLM
|
6. For client projects: never send code or context to cloud APIs — use local models via LiteLLM
|
||||||
|
|
||||||
## `main` is protected: ship by PR (local-dev#42, decided 2026-10-08)
|
|
||||||
|
|
||||||
Gitea refuses direct pushes to `main` in this repo, and a PR merges only after its CI status (`CI / Lint / Test / Vet (pull_request)`) passes.
|
|
||||||
**Why:** local-dev rule 5's reviewed lane (risk/medium+ is reviewed by a human) was enforced on the merge tools
|
|
||||||
(`tbd_ship`, `pr_merge`: gitea-mcp v0.17.0/v0.18.0, merge only for an issue labelled exactly `risk/low` with green CI)
|
|
||||||
but not on a direct commit to `main`, which bypassed it. This overrides rule 5's "risk/low commits straight to main"
|
|
||||||
for this repo; control code is where the lane matters.
|
|
||||||
**How:** `tbd_ship` for a one-file change, or branch -> PR -> `pr_merge` (name the `risk/low` issue); above `risk/low`
|
|
||||||
open the PR and a human merges it in the Gitea UI after review. Tags are not protected. CD jobs push image tags to
|
|
||||||
the **infra** repo with deploy keys, never to this repo's `main`. Undo (Mathias only): Settings -> Branches, or
|
|
||||||
`PATCH /repos/mathias/hyperguild/branch_protections/main {"enable_push": true}`.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|||||||
@@ -159,63 +159,6 @@ These rules apply to every task across every project, regardless of harness.
|
|||||||
the thing it was named for was exploitable. If you cannot name the mechanism,
|
the thing it was named for was exploitable. If you cannot name the mechanism,
|
||||||
you do not yet know what the test observes.
|
you do not yet know what the test observes.
|
||||||
|
|
||||||
10. **Short sessions, one rc per repo, edit only in a locked worktree of the repo
|
|
||||||
you are changing** (ADR-0002, accepted 2026-10-06).
|
|
||||||
- One session per issue or task, ending with rule 6's close ritual. Related
|
|
||||||
issues may share a session only when they share a worktree and an owner.
|
|
||||||
State that must outlive the session goes in an issue, not the chat.
|
|
||||||
- Start `claude rc` in the repo you will edit (`infra`, `trucker-ecommerce`,
|
|
||||||
`hyperguild`, ...), not in `~/dev`: the rc worktree is a worktree of the
|
|
||||||
launch directory's repo, so launched from `~/dev` it protects nothing you
|
|
||||||
edit (measured: 73% of edits landed in shared checkouts). Start it only
|
|
||||||
while working there.
|
|
||||||
- Keep at most one thin `~/dev` coordinator session. It files issues and hands
|
|
||||||
out tasks; it reads and edits no sub-repo code.
|
|
||||||
- Split cross-repo work per repo. The contract between the halves lives in an
|
|
||||||
issue on each repo.
|
|
||||||
- Edit in a worktree of that repo, locked at creation:
|
|
||||||
`scripts/wt.sh add <repo> <task>` does it (fetch, `wt/<task>` from origin/main, lock);
|
|
||||||
`scripts/wt.sh done <repo>-<task>` removes it after the push and refuses dirty or
|
|
||||||
unpushed work. By hand: `git worktree add ~/worktrees/<name> -b <branch> origin/main`
|
|
||||||
then `git worktree lock --reason "manual: <purpose>" ~/worktrees/<name>`.
|
|
||||||
- Before working a repo another session may touch, announce your issue list to
|
|
||||||
that session (`SendMessage`) and ask for theirs.
|
|
||||||
- Auto-memory is keyed by cwd, so a repo rc starts without it. Cross-repo
|
|
||||||
working rules live in brain (`wiki/homelab/decisions/2026-10-06-062416-auto.md`, wing homelab, hall
|
|
||||||
decisions) and the section below, not in per-directory memory.
|
|
||||||
|
|
||||||
## Standing footguns
|
|
||||||
|
|
||||||
Each cost real time once. Detail and evidence: brain `wiki/homelab/decisions/2026-10-06-062416-auto.md` (title standing-working-rules).
|
|
||||||
|
|
||||||
- **Gate then commit with `&&`, never `;`.** "Green" means exit code 0 and
|
|
||||||
`grep -c FAIL` is 0, not a filtered grep.
|
|
||||||
- **Stage by path, never `git add -A` or `git add .`** (they stage from the repo
|
|
||||||
root and swept in other sessions' work twice). Check `git status --porcelain` first.
|
|
||||||
- **Before committing in a shared checkout:** `git branch --show-current`.
|
|
||||||
- **Before tagging:** `git fetch origin --tags`, then `git describe --tags --abbrev=0 <sha>`
|
|
||||||
and `git tag --sort=-v:refname | head`; go above both. Never take the last tag
|
|
||||||
from a report.
|
|
||||||
- **Run only what the project's gate runs.** No repo-wide formatter unless the gate
|
|
||||||
runs it.
|
|
||||||
- **Reproduce a CI red at the cited commit before applying an issue's prescribed fix.**
|
|
||||||
- **Read an issue's comments (`issue_list_comments`), not just its body, before working it.**
|
|
||||||
- **A comment naming a control ("behind branch protection") is not evidence; query
|
|
||||||
the control.** A working detector can hide a missing preventer.
|
|
||||||
- **On koala the MCP servers are `mcp__plugin_skills_{brain,gitea,infra}__*`.** The
|
|
||||||
per-turn auth reminder is not evidence; call the tool or run `claude mcp list`.
|
|
||||||
Brain writes always pass `wing` + `hall`.
|
|
||||||
- **`cat /etc/hostname` before assuming which host you are on.**
|
|
||||||
- **Never run bare `git remote -v` or `git config --list`** (url.insteadOf expands
|
|
||||||
into a live token); use `git config --get remote.origin.url`.
|
|
||||||
- **On koala `sudo` is denied outright; k8s exec, env dumps and flux reconcile need
|
|
||||||
per-action approval; decrypting a SOPS file and printing in one Bash call is
|
|
||||||
hook-blocked.** Do the repo-side work and hand the host-side step over in an issue.
|
|
||||||
- **`set -o pipefail` plus an optional `grep | head | awk` dies on no match:** append `|| true`.
|
|
||||||
- **Gitea `file_write_branch` takes plain text, not base64.**
|
|
||||||
- **`OP_SERVICE_ACCOUNT_TOKEN` in the shell is read-only for `op`.** Operational
|
|
||||||
secrets go in the AgentSecrets vault via `set -a; . ~/.op-agent-write.env; set +a`.
|
|
||||||
|
|
||||||
## CAD sprint patterns
|
## CAD sprint patterns
|
||||||
|
|
||||||
Two CAD sprint archetypes (defs: brain `wiki/homelab/decisions/continuous-agentic-development-cad-concept-2026-06-16`):
|
Two CAD sprint archetypes (defs: brain `wiki/homelab/decisions/continuous-agentic-development-cad-concept-2026-06-16`):
|
||||||
@@ -296,20 +239,25 @@ Three machines on Tailscale:
|
|||||||
|
|
||||||
All development repos live at `~/dev/` (softlink from `~/Documents/local-dev/`).
|
All development repos live at `~/dev/` (softlink from `~/Documents/local-dev/`).
|
||||||
|
|
||||||
Organized by **role**, because the role decides which rules apply (ADR-0003):
|
Organized in thematic folders:
|
||||||
|
|
||||||
| Folder | Role | Rules that key off it |
|
| Folder | Focus | Count |
|
||||||
|--------|------|-----------------------|
|
|--------|-------|-------|
|
||||||
| `platform/` | infra, brain, hyperguild, gitea-mcp, dispatch, shared tooling | control code (rule 8), GitOps review lane |
|
| `GO/` | Go web frameworks, API integrations, learning projects | ~10 |
|
||||||
| `products/` | tapir, cobalt-dingo, openbanking-api, ... | ordinary trunk-based work |
|
| `AI/` | ML research, AI frameworks (FinRL, DSPy, crawl4ai) | ~6 |
|
||||||
| `clients/` | client work | local models only, never cloud APIs (Client work rules) |
|
| `AGENTS/` | Autonomous agents, coding agents, MCP servers, infra | ~15 |
|
||||||
| `spikes/` | throwaway | no ceremony |
|
| `QKX/` | Invoice processing, financial automation, payment systems | ~13 |
|
||||||
|
| `XT/` | Climate data, sustainability (Klimatkollen, Garbo) | ~2 |
|
||||||
|
|
||||||
New projects are born into a role folder (`task new-project -- <name> <client|personal>
|
See `~/dev/PROJECT_SUMMARY.md` for detailed descriptions of each project.
|
||||||
<role-folder>`); a client tag requires `clients/` and `clients/` requires a client tag.
|
|
||||||
Existing repos still sit in the old `AGENTS/` and `AI/` folders until they are moved one by
|
### Key active projects
|
||||||
one (ADR-0003 step 3). `~/dev/PROJECT_SUMMARY.md` (generated) lists every repo and where it
|
|
||||||
is. How a session runs on koala (worktrees, rc per repo, the reaper, memory): `docs/working-on-koala.md`.
|
- **super-koala** (`AGENTS/`) — multi-component agent stack with LangGraph, DSPy, MCP
|
||||||
|
- **azure-tiger** (`QKX/`) — invoice extraction → ISO 20022 payment instructions
|
||||||
|
- **gocrwl** (`AGENTS/`) — Go web crawler with containerized deployment
|
||||||
|
- **koala-ai-stack** (`AGENTS/`) — local AI server infrastructure management
|
||||||
|
- **klimatkollen** (`XT/`) — Swedish municipal climate data platform
|
||||||
|
|
||||||
## Knowledge base — actively use it
|
## Knowledge base — actively use it
|
||||||
|
|
||||||
@@ -560,15 +508,3 @@ When acting as a coding agent on this project:
|
|||||||
4. Never modify files outside the project root without explicit permission
|
4. Never modify files outside the project root without explicit permission
|
||||||
5. When adding a dependency, explain why in the commit message
|
5. When adding a dependency, explain why in the commit message
|
||||||
6. For client projects: never send code or context to cloud APIs — use local models via LiteLLM
|
6. For client projects: never send code or context to cloud APIs — use local models via LiteLLM
|
||||||
|
|
||||||
## `main` is protected: ship by PR (local-dev#42, decided 2026-10-08)
|
|
||||||
|
|
||||||
Gitea refuses direct pushes to `main` in this repo, and a PR merges only after its CI status (`CI / Lint / Test / Vet (pull_request)`) passes.
|
|
||||||
**Why:** local-dev rule 5's reviewed lane (risk/medium+ is reviewed by a human) was enforced on the merge tools
|
|
||||||
(`tbd_ship`, `pr_merge`: gitea-mcp v0.17.0/v0.18.0, merge only for an issue labelled exactly `risk/low` with green CI)
|
|
||||||
but not on a direct commit to `main`, which bypassed it. This overrides rule 5's "risk/low commits straight to main"
|
|
||||||
for this repo; control code is where the lane matters.
|
|
||||||
**How:** `tbd_ship` for a one-file change, or branch -> PR -> `pr_merge` (name the `risk/low` issue); above `risk/low`
|
|
||||||
open the PR and a human merges it in the Gitea UI after review. Tags are not protected. CD jobs push image tags to
|
|
||||||
the **infra** repo with deploy keys, never to this repo's `main`. Undo (Mathias only): Settings -> Branches, or
|
|
||||||
`PATCH /repos/mathias/hyperguild/branch_protections/main {"enable_push": true}`.
|
|
||||||
|
|||||||
@@ -84,15 +84,3 @@ When acting as a coding agent on this project:
|
|||||||
4. Never modify files outside the project root without explicit permission
|
4. Never modify files outside the project root without explicit permission
|
||||||
5. When adding a dependency, explain why in the commit message
|
5. When adding a dependency, explain why in the commit message
|
||||||
6. For client projects: never send code or context to cloud APIs — use local models via LiteLLM
|
6. For client projects: never send code or context to cloud APIs — use local models via LiteLLM
|
||||||
|
|
||||||
## `main` is protected: ship by PR (local-dev#42, decided 2026-10-08)
|
|
||||||
|
|
||||||
Gitea refuses direct pushes to `main` in this repo, and a PR merges only after its CI status (`CI / Lint / Test / Vet (pull_request)`) passes.
|
|
||||||
**Why:** local-dev rule 5's reviewed lane (risk/medium+ is reviewed by a human) was enforced on the merge tools
|
|
||||||
(`tbd_ship`, `pr_merge`: gitea-mcp v0.17.0/v0.18.0, merge only for an issue labelled exactly `risk/low` with green CI)
|
|
||||||
but not on a direct commit to `main`, which bypassed it. This overrides rule 5's "risk/low commits straight to main"
|
|
||||||
for this repo; control code is where the lane matters.
|
|
||||||
**How:** `tbd_ship` for a one-file change, or branch -> PR -> `pr_merge` (name the `risk/low` issue); above `risk/low`
|
|
||||||
open the PR and a human merges it in the Gitea UI after review. Tags are not protected. CD jobs push image tags to
|
|
||||||
the **infra** repo with deploy keys, never to this repo's `main`. Undo (Mathias only): Settings -> Branches, or
|
|
||||||
`PATCH /repos/mathias/hyperguild/branch_protections/main {"enable_push": true}`.
|
|
||||||
|
|||||||
@@ -179,12 +179,9 @@ func alertingWarnings(getenv func(string) string) []string {
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
// buildNotifier returns the ntfy notifier for one alert source, titled so
|
func buildNotifier() audit.Notifier {
|
||||||
// the source is recognisable (hyperguild#163), or nil when ntfy is not
|
if n := audit.NewNtfyNotifier(os.Getenv("BRAIN_NTFY_URL"), os.Getenv("BRAIN_NTFY_TOKEN")); n != nil {
|
||||||
// configured.
|
return n
|
||||||
func buildNotifier(getenv func(string) string, title string) audit.Notifier {
|
|
||||||
if n := audit.NewNtfyNotifier(getenv("BRAIN_NTFY_URL"), getenv("BRAIN_NTFY_TOKEN")); n != nil {
|
|
||||||
return n.WithTitle(title)
|
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -206,7 +203,7 @@ func buildAuditSink(ctx context.Context, brainDir string, logger *slog.Logger) c
|
|||||||
logger.Error("capture audit buffer init", "err", err)
|
logger.Error("capture audit buffer init", "err", err)
|
||||||
os.Exit(1)
|
os.Exit(1)
|
||||||
}
|
}
|
||||||
notifier := buildNotifier(os.Getenv, "brain-capture audit degraded")
|
notifier := buildNotifier()
|
||||||
reconcileInterval := time.Duration(envInt("BRAIN_AUDIT_RECONCILE_INTERVAL", 60)) * time.Second
|
reconcileInterval := time.Duration(envInt("BRAIN_AUDIT_RECONCILE_INTERVAL", 60)) * time.Second
|
||||||
audit.StartReconcile(ctx, central, buffer, notifier, reconcileInterval)
|
audit.StartReconcile(ctx, central, buffer, notifier, reconcileInterval)
|
||||||
logger.Info("capture audit: loki+buffer sink", "loki", lokiURL, "reconcile_s", int(reconcileInterval.Seconds()))
|
logger.Info("capture audit: loki+buffer sink", "loki", lokiURL, "reconcile_s", int(reconcileInterval.Seconds()))
|
||||||
@@ -369,7 +366,7 @@ func main() {
|
|||||||
|
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
// hyperguild#157: report a symlink that appears in the brain while running.
|
// hyperguild#157: report a symlink that appears in the brain while running.
|
||||||
go symlinkguard.Watch(ctx, brainDir, symlinkScanInterval(os.Getenv), buildNotifier(os.Getenv, "brain symlink tripwire"))
|
go symlinkguard.Watch(ctx, brainDir, symlinkScanInterval(os.Getenv), buildNotifier())
|
||||||
|
|
||||||
// Never fatal on an unreachable issuer (#83): on a cold cluster boot this
|
// Never fatal on an unreachable issuer (#83): on a cold cluster boot this
|
||||||
// pod can beat dex to serving and get a 502 from ingress, and exiting meant
|
// pod can beat dex to serving and get a 502 from ingress, and exiting meant
|
||||||
@@ -549,7 +546,7 @@ func main() {
|
|||||||
// capture already gets, plus an out-of-band alert. Wired here because
|
// capture already gets, plus an out-of-band alert. Wired here because
|
||||||
// brain_edit_root needs the capture plumbing anyway — that is what
|
// brain_edit_root needs the capture plumbing anyway — that is what
|
||||||
// re-derives the static principal its gate checks.
|
// re-derives the static principal its gate checks.
|
||||||
mcpSrv.WithGovernanceObserver(auditSink, buildNotifier(os.Getenv, "brain governance edit"))
|
mcpSrv.WithGovernanceObserver(auditSink, buildNotifier())
|
||||||
logger.Info("capture enabled (REST + MCP tool)", "sovereign_principals", len(sovereign),
|
logger.Info("capture enabled (REST + MCP tool)", "sovereign_principals", len(sovereign),
|
||||||
"governance_observer", true)
|
"governance_observer", true)
|
||||||
} else {
|
} else {
|
||||||
|
|||||||
@@ -159,16 +159,3 @@ func TestBuildMuxObservesRequestsAndServesMetrics(t *testing.T) {
|
|||||||
require.Equal(t, http.StatusOK, code)
|
require.Equal(t, http.StatusOK, code)
|
||||||
assert.Contains(t, body, `path="GET /pending"`)
|
assert.Contains(t, body, `path="GET /pending"`)
|
||||||
}
|
}
|
||||||
|
|
||||||
// hyperguild#185: an authenticated request to an unregistered path reached
|
|
||||||
// the inner REST mux, which left Pattern empty, and the metrics label fell
|
|
||||||
// back to the raw URL path: one Prometheus series per unique path any token
|
|
||||||
// holder sends. It is now labelled with a fixed catch-all.
|
|
||||||
func TestBuildMuxLabelsAnUnregisteredPathWithAFixedLabel(t *testing.T) {
|
|
||||||
r := testMux(t)
|
|
||||||
code, _ := do(t, r, "GET", "/zqrandom-185-segment", "Bearer "+routesTok)
|
|
||||||
require.Equal(t, http.StatusNotFound, code)
|
|
||||||
_, body := do(t, r, "GET", "/metrics", "")
|
|
||||||
assert.NotContains(t, body, "zqrandom-185-segment", "the raw path became a metrics label")
|
|
||||||
assert.Contains(t, body, `path="unmatched"`)
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -1,13 +1,9 @@
|
|||||||
package main
|
package main
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
"syscall"
|
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
@@ -67,26 +63,3 @@ func TestStartupSymlinkCheckRefusesAnUnlistableBrainRoot(t *testing.T) {
|
|||||||
t.Cleanup(func() { _ = os.Chmod(brain, 0o755) })
|
t.Cleanup(func() { _ = os.Chmod(brain, 0o755) })
|
||||||
assert.Error(t, startupSymlinkCheck(brain))
|
assert.Error(t, startupSymlinkCheck(brain))
|
||||||
}
|
}
|
||||||
|
|
||||||
// hyperguild#179: a FIFO in the brain is refused at startup with a clear
|
|
||||||
// message, instead of hanging or exiting later in a reader.
|
|
||||||
func TestStartupSymlinkCheckRefusesABrainWithAFIFO(t *testing.T) {
|
|
||||||
brain := t.TempDir()
|
|
||||||
require.NoError(t, syscall.Mkfifo(filepath.Join(brain, "classification.yaml"), 0o644))
|
|
||||||
err := startupSymlinkCheck(brain)
|
|
||||||
require.Error(t, err)
|
|
||||||
assert.Contains(t, err.Error(), "classification.yaml")
|
|
||||||
}
|
|
||||||
|
|
||||||
// hyperguild#163: each alert source gets its own ntfy title.
|
|
||||||
func TestBuildNotifierUsesTheGivenTitle(t *testing.T) {
|
|
||||||
var got string
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { got = r.Header.Get("Title") }))
|
|
||||||
defer srv.Close()
|
|
||||||
env := map[string]string{"BRAIN_NTFY_URL": srv.URL}
|
|
||||||
n := buildNotifier(func(k string) string { return env[k] }, "brain symlink tripwire")
|
|
||||||
require.NotNil(t, n)
|
|
||||||
require.NoError(t, n.Notify(context.Background(), "m"))
|
|
||||||
assert.Equal(t, "brain symlink tripwire", got)
|
|
||||||
assert.Nil(t, buildNotifier(func(string) string { return "" }, "x"), "no ntfy URL, no notifier")
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -1,34 +0,0 @@
|
|||||||
package api_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
|
|
||||||
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
|
|
||||||
)
|
|
||||||
|
|
||||||
// hyperguild#178: an unreadable directory under wiki/ made the link check
|
|
||||||
// fail and brain_write refuse the note, losing the learning. The note is now
|
|
||||||
// written with its links as given.
|
|
||||||
func TestWriteNoteKeepsTheLearningWhenTheLinkCheckCannotIndex(t *testing.T) {
|
|
||||||
if os.Geteuid() == 0 {
|
|
||||||
t.Skip("root ignores directory permissions")
|
|
||||||
}
|
|
||||||
brain := t.TempDir()
|
|
||||||
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki", "w", "facts"), 0o755))
|
|
||||||
require.NoError(t, os.WriteFile(filepath.Join(brain, "wiki", "w", "facts", "real.md"), []byte("x"), 0o644))
|
|
||||||
locked := filepath.Join(brain, "wiki", "other", "locked")
|
|
||||||
require.NoError(t, os.MkdirAll(locked, 0o755))
|
|
||||||
require.NoError(t, os.Chmod(locked, 0o000))
|
|
||||||
t.Cleanup(func() { _ = os.Chmod(locked, 0o755) })
|
|
||||||
|
|
||||||
rel, err := api.WriteNote(brain, api.WriteNoteOptions{Wing: "w", Hall: "facts", Filename: "n", Content: "learning [[real]] and [[gone]]"})
|
|
||||||
require.NoError(t, err)
|
|
||||||
got, err := os.ReadFile(filepath.Join(brain, filepath.FromSlash(rel)))
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Contains(t, string(got), "[[gone]]", "links kept as given when the index is partial")
|
|
||||||
}
|
|
||||||
@@ -13,14 +13,9 @@ import (
|
|||||||
type NtfyNotifier struct {
|
type NtfyNotifier struct {
|
||||||
topicURL string
|
topicURL string
|
||||||
token string
|
token string
|
||||||
title string
|
|
||||||
http *http.Client
|
http *http.Client
|
||||||
}
|
}
|
||||||
|
|
||||||
// defaultNtfyTitle is the title of a notifier built without WithTitle: the
|
|
||||||
// capture audit's degraded-state alert, its first user.
|
|
||||||
const defaultNtfyTitle = "brain-capture audit degraded"
|
|
||||||
|
|
||||||
// NewNtfyNotifier constructs a notifier for the given ntfy topic URL
|
// NewNtfyNotifier constructs a notifier for the given ntfy topic URL
|
||||||
// (e.g. https://ntfy.sh/my-topic). token is an optional bearer for
|
// (e.g. https://ntfy.sh/my-topic). token is an optional bearer for
|
||||||
// protected ntfy instances; it is held here and only sent in the
|
// protected ntfy instances; it is held here and only sent in the
|
||||||
@@ -32,31 +27,17 @@ func NewNtfyNotifier(topicURL, token string) *NtfyNotifier {
|
|||||||
return &NtfyNotifier{
|
return &NtfyNotifier{
|
||||||
topicURL: strings.TrimRight(topicURL, "/"),
|
topicURL: strings.TrimRight(topicURL, "/"),
|
||||||
token: token,
|
token: token,
|
||||||
title: defaultNtfyTitle,
|
|
||||||
http: &http.Client{Timeout: 10 * time.Second},
|
http: &http.Client{Timeout: 10 * time.Second},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// WithTitle returns a copy that sends title as the ntfy Title header, so each
|
|
||||||
// alert source (the capture audit, the symlink tripwire, governance edits)
|
|
||||||
// is recognisable on the phone (hyperguild#163). A nil notifier (ntfy not
|
|
||||||
// configured) stays nil.
|
|
||||||
func (n *NtfyNotifier) WithTitle(title string) *NtfyNotifier {
|
|
||||||
if n == nil {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
c := *n
|
|
||||||
c.title = title
|
|
||||||
return &c
|
|
||||||
}
|
|
||||||
|
|
||||||
// Notify posts a message to the ntfy topic.
|
// Notify posts a message to the ntfy topic.
|
||||||
func (n *NtfyNotifier) Notify(ctx context.Context, msg string) error {
|
func (n *NtfyNotifier) Notify(ctx context.Context, msg string) error {
|
||||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, n.topicURL, strings.NewReader(msg))
|
req, err := http.NewRequestWithContext(ctx, http.MethodPost, n.topicURL, strings.NewReader(msg))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
req.Header.Set("Title", n.title)
|
req.Header.Set("Title", "brain-capture audit degraded")
|
||||||
req.Header.Set("Priority", "high")
|
req.Header.Set("Priority", "high")
|
||||||
req.Header.Set("Tags", "warning,brain")
|
req.Header.Set("Tags", "warning,brain")
|
||||||
if n.token != "" {
|
if n.token != "" {
|
||||||
|
|||||||
@@ -1,29 +0,0 @@
|
|||||||
package audit
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
)
|
|
||||||
|
|
||||||
// hyperguild#163: every alert went out titled "brain-capture audit
|
|
||||||
// degraded", including the symlink tripwire's and governance alerts.
|
|
||||||
func TestNtfyNotifierSendsItsOwnTitle(t *testing.T) {
|
|
||||||
var got []string
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
||||||
got = append(got, r.Header.Get("Title"))
|
|
||||||
}))
|
|
||||||
defer srv.Close()
|
|
||||||
base := NewNtfyNotifier(srv.URL, "")
|
|
||||||
require.NoError(t, base.Notify(context.Background(), "m"))
|
|
||||||
require.NoError(t, base.WithTitle("brain symlink tripwire").Notify(context.Background(), "m"))
|
|
||||||
require.NoError(t, base.Notify(context.Background(), "m"))
|
|
||||||
assert.Equal(t, []string{"brain-capture audit degraded", "brain symlink tripwire", "brain-capture audit degraded"}, got,
|
|
||||||
"WithTitle returns a copy; the base keeps its default title")
|
|
||||||
var nilN *NtfyNotifier
|
|
||||||
assert.Nil(t, nilN.WithTitle("x"), "nil (ntfy not configured) stays nil")
|
|
||||||
}
|
|
||||||
@@ -47,12 +47,7 @@ var (
|
|||||||
func StripUnresolvedLinks(brainDir, content string) (string, []string, error) {
|
func StripUnresolvedLinks(brainDir, content string) (string, []string, error) {
|
||||||
index, err := entryIndex(brainDir)
|
index, err := entryIndex(brainDir)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
// A partial index cannot tell an invented link from one into the
|
return "", nil, err
|
||||||
// part it could not read, so unwrap nothing, and do not refuse the
|
|
||||||
// write either: this runs before every brain_write, and a refused
|
|
||||||
// write is a lost learning (hyperguild#178).
|
|
||||||
slog.Warn("link check skipped: cannot index the brain", "err", err)
|
|
||||||
return content, nil, nil
|
|
||||||
}
|
}
|
||||||
if len(index) == 0 {
|
if len(index) == 0 {
|
||||||
return content, nil, nil
|
return content, nil, nil
|
||||||
|
|||||||
@@ -1,41 +0,0 @@
|
|||||||
package brain_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
|
|
||||||
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
|
|
||||||
)
|
|
||||||
|
|
||||||
// hyperguild#178: StripUnresolvedLinks runs before every brain_write and
|
|
||||||
// brain_update. An unreadable directory under knowledge/ or wiki/ made it
|
|
||||||
// error, the write was refused and the learning lost. When the link index
|
|
||||||
// cannot be built, the content is now returned unchanged (no link is
|
|
||||||
// unwrapped, none is invented) and no error stops the write.
|
|
||||||
func TestStripUnresolvedLinksKeepsContentWhenTheIndexCannotBeBuilt(t *testing.T) {
|
|
||||||
if os.Geteuid() == 0 {
|
|
||||||
t.Skip("root ignores directory permissions")
|
|
||||||
}
|
|
||||||
b := t.TempDir()
|
|
||||||
require.NoError(t, os.MkdirAll(filepath.Join(b, "wiki", "w", "facts"), 0o755))
|
|
||||||
require.NoError(t, os.WriteFile(filepath.Join(b, "wiki", "w", "facts", "real.md"), []byte("x"), 0o644))
|
|
||||||
locked := filepath.Join(b, "wiki", "other", "locked")
|
|
||||||
require.NoError(t, os.MkdirAll(locked, 0o755))
|
|
||||||
require.NoError(t, os.Chmod(locked, 0o000))
|
|
||||||
t.Cleanup(func() { _ = os.Chmod(locked, 0o755) })
|
|
||||||
|
|
||||||
in := "learning [[real]] and [[gone]]"
|
|
||||||
out, unlinked, err := brain.StripUnresolvedLinks(b, in)
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Equal(t, in, out, "content unchanged: no link unwrapped on a partial index")
|
|
||||||
assert.Empty(t, unlinked)
|
|
||||||
|
|
||||||
require.NoError(t, os.Chmod(locked, 0o755))
|
|
||||||
out, _, err = brain.StripUnresolvedLinks(b, in)
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.NotContains(t, out, "[[gone]]", "positive control: with a readable tree the strip runs")
|
|
||||||
}
|
|
||||||
@@ -83,9 +83,8 @@ func (r *Registry) Observe(path, status string, d time.Duration) {
|
|||||||
|
|
||||||
// Middleware wraps next, observing every request's duration + status.
|
// Middleware wraps next, observing every request's duration + status.
|
||||||
// The metric label `path` uses the request's Pattern (Go 1.22+ ServeMux),
|
// The metric label `path` uses the request's Pattern (Go 1.22+ ServeMux),
|
||||||
// or the fixed label "unmatched" if no Pattern is set: falling back to the
|
// falling back to the URL path if no Pattern is set. Pattern keeps
|
||||||
// raw URL path gave any caller one series per unique path (hyperguild#185).
|
// cardinality bounded (one series per route, not one per unique URL).
|
||||||
// Pattern keeps cardinality bounded (one series per route).
|
|
||||||
func (r *Registry) Middleware(next http.Handler) http.Handler {
|
func (r *Registry) Middleware(next http.Handler) http.Handler {
|
||||||
return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
|
return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
|
||||||
rec := &statusRecorder{ResponseWriter: w, code: http.StatusOK}
|
rec := &statusRecorder{ResponseWriter: w, code: http.StatusOK}
|
||||||
@@ -93,7 +92,7 @@ func (r *Registry) Middleware(next http.Handler) http.Handler {
|
|||||||
next.ServeHTTP(rec, req)
|
next.ServeHTTP(rec, req)
|
||||||
path := req.Pattern
|
path := req.Pattern
|
||||||
if path == "" {
|
if path == "" {
|
||||||
path = "unmatched"
|
path = req.URL.Path
|
||||||
}
|
}
|
||||||
r.Observe(path, statusClass(rec.code), time.Since(start))
|
r.Observe(path, statusClass(rec.code), time.Since(start))
|
||||||
})
|
})
|
||||||
|
|||||||
@@ -71,16 +71,11 @@ func TestMiddleware_RecordsTiming(t *testing.T) {
|
|||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
r := New()
|
r := New()
|
||||||
// Routed through a ServeMux, as in production: the mux sets the
|
handler := r.Middleware(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||||
// request's Pattern, which is the label (an unrouted request is
|
|
||||||
// "unmatched", hyperguild#185).
|
|
||||||
mux := http.NewServeMux()
|
|
||||||
mux.HandleFunc("/query", func(w http.ResponseWriter, _ *http.Request) {
|
|
||||||
time.Sleep(2 * time.Millisecond)
|
time.Sleep(2 * time.Millisecond)
|
||||||
w.WriteHeader(http.StatusOK)
|
w.WriteHeader(http.StatusOK)
|
||||||
_, _ = io.WriteString(w, "ok")
|
_, _ = io.WriteString(w, "ok")
|
||||||
})
|
}))
|
||||||
handler := r.Middleware(mux)
|
|
||||||
|
|
||||||
srv := httptest.NewServer(handler)
|
srv := httptest.NewServer(handler)
|
||||||
defer srv.Close()
|
defer srv.Close()
|
||||||
@@ -122,19 +117,3 @@ func TestStatusRecorder_DefaultsTo200(t *testing.T) {
|
|||||||
t.Errorf("code %d, want 200", rec.Code)
|
t.Errorf("code %d, want 200", rec.Code)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// A request with no matched Pattern is labelled "unmatched", never with its
|
|
||||||
// raw URL path (unbounded cardinality, hyperguild#185).
|
|
||||||
func TestMiddleware_LabelsAnUnmatchedRequestWithAFixedLabel(t *testing.T) {
|
|
||||||
r := New()
|
|
||||||
h := r.Middleware(http.NotFoundHandler())
|
|
||||||
h.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("GET", "/zqrandom-185-unit", nil))
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
r.Handler().ServeHTTP(rec, httptest.NewRequest("GET", "/metrics", nil))
|
|
||||||
if strings.Contains(rec.Body.String(), "zqrandom-185-unit") {
|
|
||||||
t.Errorf("the raw path became a metrics label:\n%s", rec.Body.String())
|
|
||||||
}
|
|
||||||
if !strings.Contains(rec.Body.String(), `path="unmatched"`) {
|
|
||||||
t.Errorf("no unmatched label:\n%s", rec.Body.String())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -1,104 +0,0 @@
|
|||||||
package symlinkguard
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"net"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"syscall"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
)
|
|
||||||
|
|
||||||
// hyperguild#179: the tripwire looked only for symlinks, so a FIFO (or
|
|
||||||
// socket) in the brain passed it. A FIFO at classification.yaml then made
|
|
||||||
// startup exit; at a note path it could hang readers that skip the handle
|
|
||||||
// check.
|
|
||||||
func TestScanReportsFIFOsAndSocketsAsSpecialFiles(t *testing.T) {
|
|
||||||
brain := t.TempDir()
|
|
||||||
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki", "w"), 0o755))
|
|
||||||
require.NoError(t, syscall.Mkfifo(filepath.Join(brain, "classification.yaml"), 0o644))
|
|
||||||
require.NoError(t, syscall.Mkfifo(filepath.Join(brain, "wiki", "w", "f.md"), 0o644))
|
|
||||||
l, err := net.Listen("unix", filepath.Join(brain, "s.sock"))
|
|
||||||
require.NoError(t, err)
|
|
||||||
defer func() { _ = l.Close() }()
|
|
||||||
require.NoError(t, os.WriteFile(filepath.Join(brain, "wiki", "w", "n.md"), []byte("n"), 0o644))
|
|
||||||
|
|
||||||
r, err := Scan(brain)
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Equal(t, []string{"classification.yaml", "s.sock", "wiki/w/f.md"}, r.Special)
|
|
||||||
assert.False(t, r.Clean())
|
|
||||||
assert.Contains(t, r.String(), "special file")
|
|
||||||
}
|
|
||||||
|
|
||||||
// Watch removes a FIFO as it removes symlinks, and leaves a regular file
|
|
||||||
// (which Scan never reports) alone.
|
|
||||||
func TestWatchRemovesAFIFOAndLeavesAnUnreportedRegularFile(t *testing.T) {
|
|
||||||
brain := t.TempDir()
|
|
||||||
fifo := filepath.Join(brain, "f.md")
|
|
||||||
require.NoError(t, syscall.Mkfifo(fifo, 0o644))
|
|
||||||
regular := filepath.Join(brain, "n.md")
|
|
||||||
require.NoError(t, os.WriteFile(regular, []byte("n"), 0o644))
|
|
||||||
n := &recNotifier{}
|
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
|
||||||
defer cancel()
|
|
||||||
go Watch(ctx, brain, 10*time.Millisecond, n)
|
|
||||||
require.Eventually(t, func() bool {
|
|
||||||
_, err := os.Lstat(fifo)
|
|
||||||
return os.IsNotExist(err)
|
|
||||||
}, 2*time.Second, 10*time.Millisecond, "the FIFO is removed")
|
|
||||||
require.Eventually(t, func() bool { return n.count() >= 1 }, 2*time.Second, 10*time.Millisecond)
|
|
||||||
b, err := os.ReadFile(regular)
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Equal(t, "n", string(b))
|
|
||||||
n.mu.Lock()
|
|
||||||
defer n.mu.Unlock()
|
|
||||||
assert.Contains(t, n.msgs[0], "f.md")
|
|
||||||
}
|
|
||||||
|
|
||||||
// removeSpecial re-checks the type: a path that is a regular file or a
|
|
||||||
// directory at the re-check is not removed. (A swap between the re-check and
|
|
||||||
// the unlink can still remove one; Linux has no conditional unlink. The same
|
|
||||||
// window exists in removeSymlinks.)
|
|
||||||
func TestRemoveSpecialRefusesRegularFilesAndDirectories(t *testing.T) {
|
|
||||||
brain := t.TempDir()
|
|
||||||
require.NoError(t, os.WriteFile(filepath.Join(brain, "n.md"), []byte("n"), 0o644))
|
|
||||||
require.NoError(t, os.MkdirAll(filepath.Join(brain, "d"), 0o755))
|
|
||||||
require.NoError(t, syscall.Mkfifo(filepath.Join(brain, "f.md"), 0o644))
|
|
||||||
removed, failed := removeSpecial(brain, []string{"n.md", "d", "f.md"})
|
|
||||||
assert.Equal(t, []string{"f.md"}, removed)
|
|
||||||
assert.Equal(t, []string{"n.md", "d"}, failed)
|
|
||||||
_, err := os.Stat(filepath.Join(brain, "n.md"))
|
|
||||||
assert.NoError(t, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// A special file Watch cannot remove (its directory is read-only) stays
|
|
||||||
// reported: the persistent-finding alert names it.
|
|
||||||
func TestWatchReportsASpecialFileItCannotRemove(t *testing.T) {
|
|
||||||
if os.Geteuid() == 0 {
|
|
||||||
t.Skip("root ignores directory permissions")
|
|
||||||
}
|
|
||||||
brain := t.TempDir()
|
|
||||||
ro := filepath.Join(brain, "ro")
|
|
||||||
require.NoError(t, os.MkdirAll(ro, 0o755))
|
|
||||||
require.NoError(t, syscall.Mkfifo(filepath.Join(ro, "f.md"), 0o644))
|
|
||||||
require.NoError(t, os.Chmod(ro, 0o555))
|
|
||||||
t.Cleanup(func() { _ = os.Chmod(ro, 0o755) })
|
|
||||||
n := &recNotifier{}
|
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
|
||||||
defer cancel()
|
|
||||||
go Watch(ctx, brain, 10*time.Millisecond, n)
|
|
||||||
require.Eventually(t, func() bool { return n.count() >= 1 }, 2*time.Second, 10*time.Millisecond)
|
|
||||||
_, err := os.Lstat(filepath.Join(ro, "f.md"))
|
|
||||||
require.NoError(t, err, "the FIFO could not be removed")
|
|
||||||
n.mu.Lock()
|
|
||||||
defer n.mu.Unlock()
|
|
||||||
assert.Contains(t, n.msgs[0], "ro/f.md")
|
|
||||||
assert.Contains(t, n.msgs[0], "special file")
|
|
||||||
for _, m := range n.msgs {
|
|
||||||
assert.NotContains(t, m, "removed", "a file it could not remove was reported as removed")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -46,16 +46,10 @@ type Notifier interface {
|
|||||||
type Result struct {
|
type Result struct {
|
||||||
Symlinks []string
|
Symlinks []string
|
||||||
Unreadable []string
|
Unreadable []string
|
||||||
// Special holds FIFOs, sockets and devices (hyperguild#179): a reader
|
|
||||||
// that opens one by path can hang, and a FIFO at classification.yaml
|
|
||||||
// stops startup.
|
|
||||||
Special []string
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Clean reports whether the scan found nothing to act on.
|
// Clean reports whether the scan found nothing to act on.
|
||||||
func (r Result) Clean() bool {
|
func (r Result) Clean() bool { return len(r.Symlinks) == 0 && len(r.Unreadable) == 0 }
|
||||||
return len(r.Symlinks) == 0 && len(r.Unreadable) == 0 && len(r.Special) == 0
|
|
||||||
}
|
|
||||||
|
|
||||||
// String describes the findings for a log line or alert. Paths are quoted, so
|
// String describes the findings for a log line or alert. Paths are quoted, so
|
||||||
// a name containing a newline cannot blur two findings together.
|
// a name containing a newline cannot blur two findings together.
|
||||||
@@ -65,9 +59,6 @@ func (r Result) String() string {
|
|||||||
if len(r.Symlinks) > 0 {
|
if len(r.Symlinks) > 0 {
|
||||||
parts = append(parts, fmt.Sprintf("%d symlink(s): %s", len(r.Symlinks), q(r.Symlinks)))
|
parts = append(parts, fmt.Sprintf("%d symlink(s): %s", len(r.Symlinks), q(r.Symlinks)))
|
||||||
}
|
}
|
||||||
if len(r.Special) > 0 {
|
|
||||||
parts = append(parts, fmt.Sprintf("%d special file(s) (FIFO, socket or device; delete them): %s", len(r.Special), q(r.Special)))
|
|
||||||
}
|
|
||||||
if len(r.Unreadable) > 0 {
|
if len(r.Unreadable) > 0 {
|
||||||
parts = append(parts, fmt.Sprintf("%d unreadable dir(s): %s", len(r.Unreadable), q(r.Unreadable)))
|
parts = append(parts, fmt.Sprintf("%d unreadable dir(s): %s", len(r.Unreadable), q(r.Unreadable)))
|
||||||
}
|
}
|
||||||
@@ -122,16 +113,12 @@ func Scan(brainDir string) (Result, error) {
|
|||||||
if d.IsDir() && p == rootGit {
|
if d.IsDir() && p == rootGit {
|
||||||
return filepath.SkipDir
|
return filepath.SkipDir
|
||||||
}
|
}
|
||||||
switch {
|
if d.Type()&os.ModeSymlink != 0 {
|
||||||
case d.Type()&os.ModeSymlink != 0:
|
|
||||||
r.Symlinks = append(r.Symlinks, rel)
|
r.Symlinks = append(r.Symlinks, rel)
|
||||||
case !d.IsDir() && !d.Type().IsRegular():
|
|
||||||
r.Special = append(r.Special, rel)
|
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
})
|
})
|
||||||
sort.Strings(r.Symlinks)
|
sort.Strings(r.Symlinks)
|
||||||
sort.Strings(r.Special)
|
|
||||||
sort.Strings(r.Unreadable)
|
sort.Strings(r.Unreadable)
|
||||||
if walkErr != nil {
|
if walkErr != nil {
|
||||||
// Only the root's own error reaches here (deeper errors are recorded
|
// Only the root's own error reaches here (deeper errors are recorded
|
||||||
@@ -195,20 +182,14 @@ func Watch(ctx context.Context, brainDir string, interval time.Duration, n Notif
|
|||||||
q.push(fmt.Sprintf("brain symlink tripwire (hyperguild#157): removed %d symlink(s) from the brain dir: %s",
|
q.push(fmt.Sprintf("brain symlink tripwire (hyperguild#157): removed %d symlink(s) from the brain dir: %s",
|
||||||
len(removed), Result{}.quoted(removed)))
|
len(removed), Result{}.quoted(removed)))
|
||||||
}
|
}
|
||||||
sRemoved, sFailed := removeSpecial(brainDir, r.Special)
|
key = Result{Symlinks: failed, Unreadable: r.Unreadable}.String()
|
||||||
if len(sRemoved) > 0 {
|
|
||||||
slog.Error("brain symlink tripwire removed special files", "count", len(sRemoved))
|
|
||||||
q.push(fmt.Sprintf("brain symlink tripwire (hyperguild#179): removed %d special file(s) (FIFO, socket or device) from the brain dir: %s",
|
|
||||||
len(sRemoved), Result{}.quoted(sRemoved)))
|
|
||||||
}
|
|
||||||
key = Result{Symlinks: failed, Unreadable: r.Unreadable, Special: sFailed}.String()
|
|
||||||
}
|
}
|
||||||
if key != reported {
|
if key != reported {
|
||||||
reported = key
|
reported = key
|
||||||
if key != "" {
|
if key != "" {
|
||||||
slog.Error("brain symlink tripwire", "finding", key)
|
slog.Error("brain symlink tripwire", "finding", key)
|
||||||
q.push("brain symlink tripwire (hyperguild#157, #179): " + key +
|
q.push("brain symlink tripwire (hyperguild#157): " + key +
|
||||||
". Symlinks can lead readers out of the brain and special files can hang them; remove them.")
|
". Readers may follow symlinks out of the brain; fix them.")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -341,40 +322,6 @@ func permanent(err error) bool {
|
|||||||
return errors.As(err, &p) && p.Permanent()
|
return errors.As(err, &p) && p.Permanent()
|
||||||
}
|
}
|
||||||
|
|
||||||
// removeSpecial unlinks each brain-relative FIFO, socket or device through
|
|
||||||
// an os.Root on the brain dir, after re-checking that it is still one: a
|
|
||||||
// path that is a regular file, a directory or a symlink at the re-check is
|
|
||||||
// left alone and reported as failed. The re-check narrows, but cannot close,
|
|
||||||
// the window before the unlink (Linux has no conditional unlink); the same
|
|
||||||
// window exists in removeSymlinks, and only a local writer can use it.
|
|
||||||
func removeSpecial(brainDir string, rels []string) (removed, failed []string) {
|
|
||||||
if len(rels) == 0 {
|
|
||||||
return nil, nil
|
|
||||||
}
|
|
||||||
resolved, err := filepath.EvalSymlinks(brainDir)
|
|
||||||
if err != nil {
|
|
||||||
return nil, rels
|
|
||||||
}
|
|
||||||
root, err := os.OpenRoot(resolved)
|
|
||||||
if err != nil {
|
|
||||||
return nil, rels
|
|
||||||
}
|
|
||||||
defer func() { _ = root.Close() }()
|
|
||||||
for _, rel := range rels {
|
|
||||||
fi, err := root.Lstat(rel)
|
|
||||||
if err != nil || fi.IsDir() || fi.Mode().IsRegular() || fi.Mode()&os.ModeSymlink != 0 {
|
|
||||||
failed = append(failed, rel)
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
if err := root.Remove(rel); err != nil {
|
|
||||||
failed = append(failed, rel)
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
removed = append(removed, rel)
|
|
||||||
}
|
|
||||||
return removed, failed
|
|
||||||
}
|
|
||||||
|
|
||||||
// removeSymlinks unlinks each brain-relative symlink through an os.Root on
|
// removeSymlinks unlinks each brain-relative symlink through an os.Root on
|
||||||
// the brain dir, after re-checking it is still a symlink. Removing a symlink
|
// the brain dir, after re-checking it is still a symlink. Removing a symlink
|
||||||
// never touches its target. It returns what was removed and what was not.
|
// never touches its target. It returns what was removed and what was not.
|
||||||
|
|||||||
@@ -381,19 +381,18 @@ func TestWatchQueueIsBoundedDuringAnOutage(t *testing.T) {
|
|||||||
require.Eventually(t, func() bool { _, err := os.Lstat(link); return os.IsNotExist(err) }, 2*time.Second, 2*time.Millisecond)
|
require.Eventually(t, func() bool { _, err := os.Lstat(link); return os.IsNotExist(err) }, 2*time.Second, 2*time.Millisecond)
|
||||||
}
|
}
|
||||||
n.down.Store(false)
|
n.down.Store(false)
|
||||||
// Synchronise on the summary being delivered (hyperguild#193: a fixed
|
require.Eventually(t, func() bool { return n.count() >= 1 }, 2*time.Second, 5*time.Millisecond)
|
||||||
// sleep flaked under load); once it is, the queue has been flushed.
|
time.Sleep(100 * time.Millisecond)
|
||||||
require.Eventually(t, func() bool {
|
assert.LessOrEqual(t, n.count(), maxPending+1, "queued alerts plus at most one suppression summary")
|
||||||
n.mu.Lock()
|
n.mu.Lock()
|
||||||
defer n.mu.Unlock()
|
defer n.mu.Unlock()
|
||||||
|
found := false
|
||||||
for _, m := range n.msgs {
|
for _, m := range n.msgs {
|
||||||
if strings.Contains(m, "suppressed") {
|
if strings.Contains(m, "suppressed") {
|
||||||
return true
|
found = true
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return false
|
assert.True(t, found, "suppressed alerts are counted in a summary")
|
||||||
}, 10*time.Second, 5*time.Millisecond, "suppressed alerts are counted in a summary")
|
|
||||||
assert.LessOrEqual(t, n.count(), maxPending+1, "queued alerts plus at most one suppression summary")
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Within the bound, every alert queued during an outage is delivered after
|
// Within the bound, every alert queued during an outage is delivered after
|
||||||
@@ -412,21 +411,11 @@ func TestWatchDeliversEveryQueuedAlertAfterAShortOutage(t *testing.T) {
|
|||||||
require.Eventually(t, func() bool { _, err := os.Lstat(link); return os.IsNotExist(err) }, 2*time.Second, 2*time.Millisecond)
|
require.Eventually(t, func() bool { _, err := os.Lstat(link); return os.IsNotExist(err) }, 2*time.Second, 2*time.Millisecond)
|
||||||
}
|
}
|
||||||
n.down.Store(false)
|
n.down.Store(false)
|
||||||
require.Eventually(t, func() bool { return n.count() >= 3 }, 10*time.Second, 5*time.Millisecond)
|
require.Eventually(t, func() bool { return n.count() == 3 }, 2*time.Second, 5*time.Millisecond)
|
||||||
n.mu.Lock()
|
n.mu.Lock()
|
||||||
defer n.mu.Unlock()
|
defer n.mu.Unlock()
|
||||||
// Each removal delivered exactly once, in any order: the outage ends
|
for i, name := range []string{"qa", "qb", "qc"} {
|
||||||
// asynchronously, possibly mid-flush, so a later item can send before an
|
assert.Contains(t, n.msgs[i], strconv.Quote(name))
|
||||||
// earlier one (hyperguild#193). Order is not what this test is about.
|
|
||||||
assert.Len(t, n.msgs, 3)
|
|
||||||
for _, name := range []string{"qa", "qb", "qc"} {
|
|
||||||
hits := 0
|
|
||||||
for _, m := range n.msgs {
|
|
||||||
if strings.Contains(m, strconv.Quote(name)) {
|
|
||||||
hits++
|
|
||||||
}
|
|
||||||
}
|
|
||||||
assert.Equal(t, 1, hits, "%s delivered exactly once", name)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -468,30 +457,29 @@ func TestWatchAccountsForEveryRemovalDuringAnOutage(t *testing.T) {
|
|||||||
require.Eventually(t, func() bool { _, err := os.Lstat(link); return os.IsNotExist(err) }, 2*time.Second, 1*time.Millisecond)
|
require.Eventually(t, func() bool { _, err := os.Lstat(link); return os.IsNotExist(err) }, 2*time.Second, 1*time.Millisecond)
|
||||||
}
|
}
|
||||||
n.down.Store(false)
|
n.down.Store(false)
|
||||||
// Synchronise on the record being complete, not on a fixed sleep after
|
require.Eventually(t, func() bool {
|
||||||
// the first summary (hyperguild#193): sum every "removed N" alert and
|
|
||||||
// every suppression count until they account for all plants.
|
|
||||||
account := func() (delivered, counted int) {
|
|
||||||
n.mu.Lock()
|
n.mu.Lock()
|
||||||
defer n.mu.Unlock()
|
defer n.mu.Unlock()
|
||||||
for _, m := range n.msgs {
|
for _, m := range n.msgs {
|
||||||
var k int
|
if strings.Contains(m, "suppressed") {
|
||||||
if i := strings.Index(m, "removed "); i >= 0 && strings.Contains(m, "symlink(s)") {
|
return true
|
||||||
if _, err := fmt.Sscanf(m[i+len("removed "):], "%d", &k); err == nil {
|
|
||||||
delivered += k
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
return false
|
||||||
|
}, 2*time.Second, 5*time.Millisecond)
|
||||||
|
time.Sleep(50 * time.Millisecond)
|
||||||
|
n.mu.Lock()
|
||||||
|
defer n.mu.Unlock()
|
||||||
|
delivered, counted := 0, 0
|
||||||
|
for _, m := range n.msgs {
|
||||||
|
if strings.Contains(m, "removed 1 symlink(s)") {
|
||||||
|
delivered++
|
||||||
|
}
|
||||||
if i := strings.Index(m, "): "); i >= 0 && strings.Contains(m, "suppressed") {
|
if i := strings.Index(m, "): "); i >= 0 && strings.Contains(m, "suppressed") {
|
||||||
if _, err := fmt.Sscanf(m[i+3:], "%d", &k); err == nil {
|
_, _ = fmt.Sscanf(m[i+3:], "%d", &counted)
|
||||||
counted += k
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
assert.Equal(t, plants, delivered+counted, "delivered %d + counted %d must equal %d removals", delivered, counted, plants)
|
||||||
return delivered, counted
|
|
||||||
}
|
|
||||||
require.Eventually(t, func() bool { d, c := account(); return d+c >= plants }, 10*time.Second, 5*time.Millisecond)
|
|
||||||
d, c := account()
|
|
||||||
assert.Equal(t, plants, d+c, "delivered %d + counted %d must equal %d removals", d, c, plants)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// A poison message is still dropped (others send while it fails), and the
|
// A poison message is still dropped (others send while it fails), and the
|
||||||
|
|||||||
Reference in New Issue
Block a user