Compare commits
56
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
80f3fd3c5e | ||
|
|
0582dfdf6b | ||
|
|
4c2276b46b | ||
|
|
90752f2ac9 | ||
|
|
27b1579b80 | ||
|
|
49286ba8b8 | ||
|
|
6f5fb481ad | ||
|
|
74d6a08c7b | ||
|
|
87c40fe364 | ||
|
|
6b6d430383 | ||
|
|
ae02d4764c | ||
|
|
8e488b6e4b | ||
|
|
3ccf6b46ac | ||
|
|
44207f8ccf | ||
|
|
603d2f9dfb | ||
|
|
8988c291ca | ||
|
|
6717221989 | ||
|
|
409b85096b | ||
|
|
1025d77795 | ||
|
|
750dc93bdb | ||
|
|
ce5a5268e7 | ||
|
|
8342de5c49 | ||
|
|
d8c7defc78 | ||
|
|
2cc4f1b30d | ||
|
|
858ade6504 | ||
|
|
bd33a0a4e8 | ||
|
|
1e14c56a49 | ||
|
|
006a891423 | ||
|
|
dd4a825f0a | ||
|
|
b4b36c9bfd | ||
|
|
96c63906c2 | ||
|
|
b9657973ff | ||
|
|
20d874a23d | ||
|
|
2c6220782d | ||
|
|
ad793f320d | ||
|
|
62de8ac2e6 | ||
|
|
365130459f | ||
|
|
c06509afe4 | ||
|
|
85d8ecb5c7 | ||
|
|
5bbf47064d | ||
|
|
2212ceedda | ||
|
|
e6e14102ee | ||
|
|
97ec9e7ce8 | ||
|
|
87ae782527 | ||
|
|
6bf61d75c0 | ||
|
|
18d01b64d7 | ||
|
|
5dfd2c0f7e | ||
|
|
22d0653b61 | ||
|
|
d6ddac4a03 | ||
|
|
1a2a93d2f4 | ||
|
|
4422dd1f69 | ||
|
|
f65f0b7826 | ||
|
|
77c44f298c | ||
|
|
f4636c7ef0 | ||
|
|
bd989f887f | ||
|
|
4770f97ffd |
@@ -84,3 +84,15 @@ When acting as a coding agent on this project:
|
||||
4. Never modify files outside the project root without explicit permission
|
||||
5. When adding a dependency, explain why in the commit message
|
||||
6. For client projects: never send code or context to cloud APIs — use local models via LiteLLM
|
||||
|
||||
## `main` is protected: ship by PR (local-dev#42, decided 2026-10-08)
|
||||
|
||||
Gitea refuses direct pushes to `main` in this repo, and a PR merges only after its CI status (`CI / Lint / Test / Vet (pull_request)`) passes.
|
||||
**Why:** local-dev rule 5's reviewed lane (risk/medium+ is reviewed by a human) was enforced on the merge tools
|
||||
(`tbd_ship`, `pr_merge`: gitea-mcp v0.17.0/v0.18.0, merge only for an issue labelled exactly `risk/low` with green CI)
|
||||
but not on a direct commit to `main`, which bypassed it. This overrides rule 5's "risk/low commits straight to main"
|
||||
for this repo; control code is where the lane matters.
|
||||
**How:** `tbd_ship` for a one-file change, or branch -> PR -> `pr_merge` (name the `risk/low` issue); above `risk/low`
|
||||
open the PR and a human merges it in the Gitea UI after review. Tags are not protected. CD jobs push image tags to
|
||||
the **infra** repo with deploy keys, never to this repo's `main`. Undo (Mathias only): Settings -> Branches, or
|
||||
`PATCH /repos/mathias/hyperguild/branch_protections/main {"enable_push": true}`.
|
||||
|
||||
+81
-17
@@ -164,6 +164,63 @@ These rules apply to every task across every project, regardless of harness.
|
||||
the thing it was named for was exploitable. If you cannot name the mechanism,
|
||||
you do not yet know what the test observes.
|
||||
|
||||
10. **Short sessions, one rc per repo, edit only in a locked worktree of the repo
|
||||
you are changing** (ADR-0002, accepted 2026-10-06).
|
||||
- One session per issue or task, ending with rule 6's close ritual. Related
|
||||
issues may share a session only when they share a worktree and an owner.
|
||||
State that must outlive the session goes in an issue, not the chat.
|
||||
- Start `claude rc` in the repo you will edit (`infra`, `trucker-ecommerce`,
|
||||
`hyperguild`, ...), not in `~/dev`: the rc worktree is a worktree of the
|
||||
launch directory's repo, so launched from `~/dev` it protects nothing you
|
||||
edit (measured: 73% of edits landed in shared checkouts). Start it only
|
||||
while working there.
|
||||
- Keep at most one thin `~/dev` coordinator session. It files issues and hands
|
||||
out tasks; it reads and edits no sub-repo code.
|
||||
- Split cross-repo work per repo. The contract between the halves lives in an
|
||||
issue on each repo.
|
||||
- Edit in a worktree of that repo, locked at creation:
|
||||
`scripts/wt.sh add <repo> <task>` does it (fetch, `wt/<task>` from origin/main, lock);
|
||||
`scripts/wt.sh done <repo>-<task>` removes it after the push and refuses dirty or
|
||||
unpushed work. By hand: `git worktree add ~/worktrees/<name> -b <branch> origin/main`
|
||||
then `git worktree lock --reason "manual: <purpose>" ~/worktrees/<name>`.
|
||||
- Before working a repo another session may touch, announce your issue list to
|
||||
that session (`SendMessage`) and ask for theirs.
|
||||
- Auto-memory is keyed by cwd, so a repo rc starts without it. Cross-repo
|
||||
working rules live in brain (`wiki/homelab/decisions/2026-10-06-062416-auto.md`, wing homelab, hall
|
||||
decisions) and the section below, not in per-directory memory.
|
||||
|
||||
## Standing footguns
|
||||
|
||||
Each cost real time once. Detail and evidence: brain `wiki/homelab/decisions/2026-10-06-062416-auto.md` (title standing-working-rules).
|
||||
|
||||
- **Gate then commit with `&&`, never `;`.** "Green" means exit code 0 and
|
||||
`grep -c FAIL` is 0, not a filtered grep.
|
||||
- **Stage by path, never `git add -A` or `git add .`** (they stage from the repo
|
||||
root and swept in other sessions' work twice). Check `git status --porcelain` first.
|
||||
- **Before committing in a shared checkout:** `git branch --show-current`.
|
||||
- **Before tagging:** `git fetch origin --tags`, then `git describe --tags --abbrev=0 <sha>`
|
||||
and `git tag --sort=-v:refname | head`; go above both. Never take the last tag
|
||||
from a report.
|
||||
- **Run only what the project's gate runs.** No repo-wide formatter unless the gate
|
||||
runs it.
|
||||
- **Reproduce a CI red at the cited commit before applying an issue's prescribed fix.**
|
||||
- **Read an issue's comments (`issue_list_comments`), not just its body, before working it.**
|
||||
- **A comment naming a control ("behind branch protection") is not evidence; query
|
||||
the control.** A working detector can hide a missing preventer.
|
||||
- **On koala the MCP servers are `mcp__plugin_skills_{brain,gitea,infra}__*`.** The
|
||||
per-turn auth reminder is not evidence; call the tool or run `claude mcp list`.
|
||||
Brain writes always pass `wing` + `hall`.
|
||||
- **`cat /etc/hostname` before assuming which host you are on.**
|
||||
- **Never run bare `git remote -v` or `git config --list`** (url.insteadOf expands
|
||||
into a live token); use `git config --get remote.origin.url`.
|
||||
- **On koala `sudo` is denied outright; k8s exec, env dumps and flux reconcile need
|
||||
per-action approval; decrypting a SOPS file and printing in one Bash call is
|
||||
hook-blocked.** Do the repo-side work and hand the host-side step over in an issue.
|
||||
- **`set -o pipefail` plus an optional `grep | head | awk` dies on no match:** append `|| true`.
|
||||
- **Gitea `file_write_branch` takes plain text, not base64.**
|
||||
- **`OP_SERVICE_ACCOUNT_TOKEN` in the shell is read-only for `op`.** Operational
|
||||
secrets go in the AgentSecrets vault via `set -a; . ~/.op-agent-write.env; set +a`.
|
||||
|
||||
## CAD sprint patterns
|
||||
|
||||
Two CAD sprint archetypes (defs: brain `wiki/homelab/decisions/continuous-agentic-development-cad-concept-2026-06-16`):
|
||||
@@ -244,25 +301,20 @@ Three machines on Tailscale:
|
||||
|
||||
All development repos live at `~/dev/` (softlink from `~/Documents/local-dev/`).
|
||||
|
||||
Organized in thematic folders:
|
||||
Organized by **role**, because the role decides which rules apply (ADR-0003):
|
||||
|
||||
| Folder | Focus | Count |
|
||||
|--------|-------|-------|
|
||||
| `GO/` | Go web frameworks, API integrations, learning projects | ~10 |
|
||||
| `AI/` | ML research, AI frameworks (FinRL, DSPy, crawl4ai) | ~6 |
|
||||
| `AGENTS/` | Autonomous agents, coding agents, MCP servers, infra | ~15 |
|
||||
| `QKX/` | Invoice processing, financial automation, payment systems | ~13 |
|
||||
| `XT/` | Climate data, sustainability (Klimatkollen, Garbo) | ~2 |
|
||||
| Folder | Role | Rules that key off it |
|
||||
|--------|------|-----------------------|
|
||||
| `platform/` | infra, brain, hyperguild, gitea-mcp, dispatch, shared tooling | control code (rule 8), GitOps review lane |
|
||||
| `products/` | tapir, cobalt-dingo, openbanking-api, ... | ordinary trunk-based work |
|
||||
| `clients/` | client work | local models only, never cloud APIs (Client work rules) |
|
||||
| `spikes/` | throwaway | no ceremony |
|
||||
|
||||
See `~/dev/PROJECT_SUMMARY.md` for detailed descriptions of each project.
|
||||
|
||||
### Key active projects
|
||||
|
||||
- **super-koala** (`AGENTS/`) — multi-component agent stack with LangGraph, DSPy, MCP
|
||||
- **azure-tiger** (`QKX/`) — invoice extraction → ISO 20022 payment instructions
|
||||
- **gocrwl** (`AGENTS/`) — Go web crawler with containerized deployment
|
||||
- **koala-ai-stack** (`AGENTS/`) — local AI server infrastructure management
|
||||
- **klimatkollen** (`XT/`) — Swedish municipal climate data platform
|
||||
New projects are born into a role folder (`task new-project -- <name> <client|personal>
|
||||
<role-folder>`); a client tag requires `clients/` and `clients/` requires a client tag.
|
||||
Existing repos still sit in the old `AGENTS/` and `AI/` folders until they are moved one by
|
||||
one (ADR-0003 step 3). `~/dev/PROJECT_SUMMARY.md` (generated) lists every repo and where it
|
||||
is. How a session runs on koala (worktrees, rc per repo, the reaper, memory): `docs/working-on-koala.md`.
|
||||
|
||||
## Knowledge base — actively use it
|
||||
|
||||
@@ -514,4 +566,16 @@ When acting as a coding agent on this project:
|
||||
5. When adding a dependency, explain why in the commit message
|
||||
6. For client projects: never send code or context to cloud APIs — use local models via LiteLLM
|
||||
|
||||
## `main` is protected: ship by PR (local-dev#42, decided 2026-10-08)
|
||||
|
||||
Gitea refuses direct pushes to `main` in this repo, and a PR merges only after its CI status (`CI / Lint / Test / Vet (pull_request)`) passes.
|
||||
**Why:** local-dev rule 5's reviewed lane (risk/medium+ is reviewed by a human) was enforced on the merge tools
|
||||
(`tbd_ship`, `pr_merge`: gitea-mcp v0.17.0/v0.18.0, merge only for an issue labelled exactly `risk/low` with green CI)
|
||||
but not on a direct commit to `main`, which bypassed it. This overrides rule 5's "risk/low commits straight to main"
|
||||
for this repo; control code is where the lane matters.
|
||||
**How:** `tbd_ship` for a one-file change, or branch -> PR -> `pr_merge` (name the `risk/low` issue); above `risk/low`
|
||||
open the PR and a human merges it in the Gitea UI after review. Tags are not protected. CD jobs push image tags to
|
||||
the **infra** repo with deploy keys, never to this repo's `main`. Undo (Mathias only): Settings -> Branches, or
|
||||
`PATCH /repos/mathias/hyperguild/branch_protections/main {"enable_push": true}`.
|
||||
|
||||
---
|
||||
|
||||
@@ -159,6 +159,63 @@ These rules apply to every task across every project, regardless of harness.
|
||||
the thing it was named for was exploitable. If you cannot name the mechanism,
|
||||
you do not yet know what the test observes.
|
||||
|
||||
10. **Short sessions, one rc per repo, edit only in a locked worktree of the repo
|
||||
you are changing** (ADR-0002, accepted 2026-10-06).
|
||||
- One session per issue or task, ending with rule 6's close ritual. Related
|
||||
issues may share a session only when they share a worktree and an owner.
|
||||
State that must outlive the session goes in an issue, not the chat.
|
||||
- Start `claude rc` in the repo you will edit (`infra`, `trucker-ecommerce`,
|
||||
`hyperguild`, ...), not in `~/dev`: the rc worktree is a worktree of the
|
||||
launch directory's repo, so launched from `~/dev` it protects nothing you
|
||||
edit (measured: 73% of edits landed in shared checkouts). Start it only
|
||||
while working there.
|
||||
- Keep at most one thin `~/dev` coordinator session. It files issues and hands
|
||||
out tasks; it reads and edits no sub-repo code.
|
||||
- Split cross-repo work per repo. The contract between the halves lives in an
|
||||
issue on each repo.
|
||||
- Edit in a worktree of that repo, locked at creation:
|
||||
`scripts/wt.sh add <repo> <task>` does it (fetch, `wt/<task>` from origin/main, lock);
|
||||
`scripts/wt.sh done <repo>-<task>` removes it after the push and refuses dirty or
|
||||
unpushed work. By hand: `git worktree add ~/worktrees/<name> -b <branch> origin/main`
|
||||
then `git worktree lock --reason "manual: <purpose>" ~/worktrees/<name>`.
|
||||
- Before working a repo another session may touch, announce your issue list to
|
||||
that session (`SendMessage`) and ask for theirs.
|
||||
- Auto-memory is keyed by cwd, so a repo rc starts without it. Cross-repo
|
||||
working rules live in brain (`wiki/homelab/decisions/2026-10-06-062416-auto.md`, wing homelab, hall
|
||||
decisions) and the section below, not in per-directory memory.
|
||||
|
||||
## Standing footguns
|
||||
|
||||
Each cost real time once. Detail and evidence: brain `wiki/homelab/decisions/2026-10-06-062416-auto.md` (title standing-working-rules).
|
||||
|
||||
- **Gate then commit with `&&`, never `;`.** "Green" means exit code 0 and
|
||||
`grep -c FAIL` is 0, not a filtered grep.
|
||||
- **Stage by path, never `git add -A` or `git add .`** (they stage from the repo
|
||||
root and swept in other sessions' work twice). Check `git status --porcelain` first.
|
||||
- **Before committing in a shared checkout:** `git branch --show-current`.
|
||||
- **Before tagging:** `git fetch origin --tags`, then `git describe --tags --abbrev=0 <sha>`
|
||||
and `git tag --sort=-v:refname | head`; go above both. Never take the last tag
|
||||
from a report.
|
||||
- **Run only what the project's gate runs.** No repo-wide formatter unless the gate
|
||||
runs it.
|
||||
- **Reproduce a CI red at the cited commit before applying an issue's prescribed fix.**
|
||||
- **Read an issue's comments (`issue_list_comments`), not just its body, before working it.**
|
||||
- **A comment naming a control ("behind branch protection") is not evidence; query
|
||||
the control.** A working detector can hide a missing preventer.
|
||||
- **On koala the MCP servers are `mcp__plugin_skills_{brain,gitea,infra}__*`.** The
|
||||
per-turn auth reminder is not evidence; call the tool or run `claude mcp list`.
|
||||
Brain writes always pass `wing` + `hall`.
|
||||
- **`cat /etc/hostname` before assuming which host you are on.**
|
||||
- **Never run bare `git remote -v` or `git config --list`** (url.insteadOf expands
|
||||
into a live token); use `git config --get remote.origin.url`.
|
||||
- **On koala `sudo` is denied outright; k8s exec, env dumps and flux reconcile need
|
||||
per-action approval; decrypting a SOPS file and printing in one Bash call is
|
||||
hook-blocked.** Do the repo-side work and hand the host-side step over in an issue.
|
||||
- **`set -o pipefail` plus an optional `grep | head | awk` dies on no match:** append `|| true`.
|
||||
- **Gitea `file_write_branch` takes plain text, not base64.**
|
||||
- **`OP_SERVICE_ACCOUNT_TOKEN` in the shell is read-only for `op`.** Operational
|
||||
secrets go in the AgentSecrets vault via `set -a; . ~/.op-agent-write.env; set +a`.
|
||||
|
||||
## CAD sprint patterns
|
||||
|
||||
Two CAD sprint archetypes (defs: brain `wiki/homelab/decisions/continuous-agentic-development-cad-concept-2026-06-16`):
|
||||
@@ -239,25 +296,20 @@ Three machines on Tailscale:
|
||||
|
||||
All development repos live at `~/dev/` (softlink from `~/Documents/local-dev/`).
|
||||
|
||||
Organized in thematic folders:
|
||||
Organized by **role**, because the role decides which rules apply (ADR-0003):
|
||||
|
||||
| Folder | Focus | Count |
|
||||
|--------|-------|-------|
|
||||
| `GO/` | Go web frameworks, API integrations, learning projects | ~10 |
|
||||
| `AI/` | ML research, AI frameworks (FinRL, DSPy, crawl4ai) | ~6 |
|
||||
| `AGENTS/` | Autonomous agents, coding agents, MCP servers, infra | ~15 |
|
||||
| `QKX/` | Invoice processing, financial automation, payment systems | ~13 |
|
||||
| `XT/` | Climate data, sustainability (Klimatkollen, Garbo) | ~2 |
|
||||
| Folder | Role | Rules that key off it |
|
||||
|--------|------|-----------------------|
|
||||
| `platform/` | infra, brain, hyperguild, gitea-mcp, dispatch, shared tooling | control code (rule 8), GitOps review lane |
|
||||
| `products/` | tapir, cobalt-dingo, openbanking-api, ... | ordinary trunk-based work |
|
||||
| `clients/` | client work | local models only, never cloud APIs (Client work rules) |
|
||||
| `spikes/` | throwaway | no ceremony |
|
||||
|
||||
See `~/dev/PROJECT_SUMMARY.md` for detailed descriptions of each project.
|
||||
|
||||
### Key active projects
|
||||
|
||||
- **super-koala** (`AGENTS/`) — multi-component agent stack with LangGraph, DSPy, MCP
|
||||
- **azure-tiger** (`QKX/`) — invoice extraction → ISO 20022 payment instructions
|
||||
- **gocrwl** (`AGENTS/`) — Go web crawler with containerized deployment
|
||||
- **koala-ai-stack** (`AGENTS/`) — local AI server infrastructure management
|
||||
- **klimatkollen** (`XT/`) — Swedish municipal climate data platform
|
||||
New projects are born into a role folder (`task new-project -- <name> <client|personal>
|
||||
<role-folder>`); a client tag requires `clients/` and `clients/` requires a client tag.
|
||||
Existing repos still sit in the old `AGENTS/` and `AI/` folders until they are moved one by
|
||||
one (ADR-0003 step 3). `~/dev/PROJECT_SUMMARY.md` (generated) lists every repo and where it
|
||||
is. How a session runs on koala (worktrees, rc per repo, the reaper, memory): `docs/working-on-koala.md`.
|
||||
|
||||
## Knowledge base — actively use it
|
||||
|
||||
@@ -508,3 +560,15 @@ When acting as a coding agent on this project:
|
||||
4. Never modify files outside the project root without explicit permission
|
||||
5. When adding a dependency, explain why in the commit message
|
||||
6. For client projects: never send code or context to cloud APIs — use local models via LiteLLM
|
||||
|
||||
## `main` is protected: ship by PR (local-dev#42, decided 2026-10-08)
|
||||
|
||||
Gitea refuses direct pushes to `main` in this repo, and a PR merges only after its CI status (`CI / Lint / Test / Vet (pull_request)`) passes.
|
||||
**Why:** local-dev rule 5's reviewed lane (risk/medium+ is reviewed by a human) was enforced on the merge tools
|
||||
(`tbd_ship`, `pr_merge`: gitea-mcp v0.17.0/v0.18.0, merge only for an issue labelled exactly `risk/low` with green CI)
|
||||
but not on a direct commit to `main`, which bypassed it. This overrides rule 5's "risk/low commits straight to main"
|
||||
for this repo; control code is where the lane matters.
|
||||
**How:** `tbd_ship` for a one-file change, or branch -> PR -> `pr_merge` (name the `risk/low` issue); above `risk/low`
|
||||
open the PR and a human merges it in the Gitea UI after review. Tags are not protected. CD jobs push image tags to
|
||||
the **infra** repo with deploy keys, never to this repo's `main`. Undo (Mathias only): Settings -> Branches, or
|
||||
`PATCH /repos/mathias/hyperguild/branch_protections/main {"enable_push": true}`.
|
||||
|
||||
@@ -84,3 +84,15 @@ When acting as a coding agent on this project:
|
||||
4. Never modify files outside the project root without explicit permission
|
||||
5. When adding a dependency, explain why in the commit message
|
||||
6. For client projects: never send code or context to cloud APIs — use local models via LiteLLM
|
||||
|
||||
## `main` is protected: ship by PR (local-dev#42, decided 2026-10-08)
|
||||
|
||||
Gitea refuses direct pushes to `main` in this repo, and a PR merges only after its CI status (`CI / Lint / Test / Vet (pull_request)`) passes.
|
||||
**Why:** local-dev rule 5's reviewed lane (risk/medium+ is reviewed by a human) was enforced on the merge tools
|
||||
(`tbd_ship`, `pr_merge`: gitea-mcp v0.17.0/v0.18.0, merge only for an issue labelled exactly `risk/low` with green CI)
|
||||
but not on a direct commit to `main`, which bypassed it. This overrides rule 5's "risk/low commits straight to main"
|
||||
for this repo; control code is where the lane matters.
|
||||
**How:** `tbd_ship` for a one-file change, or branch -> PR -> `pr_merge` (name the `risk/low` issue); above `risk/low`
|
||||
open the PR and a human merges it in the Gitea UI after review. Tags are not protected. CD jobs push image tags to
|
||||
the **infra** repo with deploy keys, never to this repo's `main`. Undo (Mathias only): Settings -> Branches, or
|
||||
`PATCH /repos/mathias/hyperguild/branch_protections/main {"enable_push": true}`.
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
# The raw/ inbox
|
||||
|
||||
`raw/` is an inbox: only unprocessed material sits there. Syncthing shares it
|
||||
with the machines you drop files from. (hyperguild#205, #208, #209)
|
||||
|
||||
## What happens to a file you drop
|
||||
|
||||
| You drop | Result |
|
||||
|---|---|
|
||||
| `raw/clip.md` (no folder, no `wing:`) | extracted and filed as a source in `wiki/web-clip/sources/`; its concepts and entities update the shared `wiki/concepts/` and `wiki/entities/` pages; the original moves to `raw/processed/<date>/`, so `raw/` is empty again |
|
||||
| `raw/<wing>/clip.md` or `wing: <wing>` in the frontmatter | the same, with the source in `wiki/<wing>/sources/`. The wing must already exist; a typo is parked, never created. This is your deliberate choice and is not second-guessed |
|
||||
| a note whose text matches the client-content pattern, with no folder | parked in `raw/needs-review/`, not extracted |
|
||||
| a folder or `wing:` naming an unknown, reserved or non-ASCII wing, or two disagreeing signals | parked in `raw/needs-review/` |
|
||||
| a note over the size cap, or one that cannot be archived | stays in `raw/`, reported once, not retried until it changes |
|
||||
| a note the model returned nothing usable for (no source page could be written) | fails: the note is kept in `raw/failed/` and the reason is in `log.md`; it is never archived as processed |
|
||||
|
||||
Concepts and entities are shared across wings because their slug is the graph's
|
||||
identity. A source whose slug is taken gets a wing-qualified one (`notes-homelab`).
|
||||
|
||||
## Parked notes
|
||||
|
||||
`brain_pending` lists them with `queue: needs-review`, a proposed wing and the
|
||||
reason. To act on one: `brain_promote` / `brain_reject` (pass `queue` when the
|
||||
same name also sits in the inbox), or move the file into `raw/<wing>/` to have it
|
||||
extracted. Parked pdf/txt files are listed but can only be moved or rejected.
|
||||
|
||||
## Switching it on
|
||||
|
||||
Routing is off by default (the retired layout is used). Set on the ingestion
|
||||
deployment:
|
||||
|
||||
| Variable | Meaning |
|
||||
|---|---|
|
||||
| `INGEST_WING_ROUTING` | `1`/`true` enables the rules above; unset or `0`/`false` is off; any other value stops the server at startup |
|
||||
| `INGEST_DEFAULT_WING` | wing for unsignalled notes (default `web-clip`; must be a plain name, not `sources`/`concepts`/`entities`/`wiki`) |
|
||||
| `INGEST_CLIENT_BLOCK` | case-insensitive regexp over note text; a match parks an unsignalled note. Falls back to `CLAUDE_INGEST_CLIENT_BLOCK`, which the deployment injects as a bare list of client names (`acme|globex`): it is then matched as whole names, with word edges decided by Unicode letters and digits (an underscore or punctuation is a boundary, so `trucker_ecommerce` matches `trucker`; `acmecorp`, `acme2` and plurals do not), and a list that is not a valid regexp, has an empty alternative, or could break out of its group stops the server at startup. A full pattern that matches the empty string is refused at startup too. Matched after folding fullwidth letters, removing zero-width characters, normalising dashes and removing whitespace. Patterns with spaces, `\\s` or `\\b` work (whitespace is collapsed in one view and removed in another). It catches what it names and nothing else: leetspeak, punctuation splits (`a.c.m.e`), encodings and homoglyphs outside a small Cyrillic/Greek table are not caught. Large notes are checked in 1 MiB chunks |
|
||||
|
||||
A bad default or pattern stops the server at startup. A wing created while the
|
||||
server runs is usable without a restart.
|
||||
|
||||
## Known limits
|
||||
|
||||
- The client filter works on 1 MiB chunks that overlap by 4 KiB. A match that can
|
||||
span more than 4 KiB (crafted padding between two words of a multi-word
|
||||
pattern, exactly at a chunk edge) is not guaranteed to be found; this affects
|
||||
notes over 1 MiB only. On a 64 MiB note the check takes about ten seconds.
|
||||
- A pattern that matches the empty string (`.*`, `x*`) parks every unsignalled
|
||||
note. That fails closed and only the operator's own configuration can do it.
|
||||
|
||||
- A deliberate folder into a sensitive wing places the note there, and the shared
|
||||
concept/entity pages then cite that source by title. Client material does not
|
||||
belong in the brain at all; the pattern only guards the unsignalled lane.
|
||||
- A merge into a shared page refreshes `last_updated` and `origin` to the latest
|
||||
ingest; the first wing's attribution is not kept.
|
||||
- A marker file (`<note>.processed`) is only written when a processed original
|
||||
cannot be removed. One at least as new as its note holds the note (and says
|
||||
so once); an older or future-dated one is ignored and removed.
|
||||
@@ -17,6 +17,7 @@ import (
|
||||
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/audit"
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/capturehttp"
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/classification"
|
||||
@@ -36,6 +37,7 @@ import (
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/watcher"
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/webhook"
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/wing"
|
||||
"k8s.io/client-go/kubernetes"
|
||||
"k8s.io/client-go/rest"
|
||||
"k8s.io/client-go/tools/clientcmd"
|
||||
@@ -89,11 +91,15 @@ func (s *claudeSink) Ingest(ctx context.Context, b claudewatcher.Batch) error {
|
||||
sb.WriteString("\n\n")
|
||||
}
|
||||
slug := "session-" + b.Host + "-" + b.SessionID
|
||||
dest := filepath.Join(s.brainDir, "archive", "claude-sessions", b.Host, slug+".md")
|
||||
if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
|
||||
return fmt.Errorf("create claude-sessions archive dir: %w", err)
|
||||
dest := "archive/claude-sessions/" + b.Host + "/" + slug + ".md"
|
||||
// Written through the brain root, never through a symlinked directory
|
||||
// or file (hyperguild#158).
|
||||
bfs, err := brainfs.Open(s.brainDir)
|
||||
if err != nil {
|
||||
return fmt.Errorf("open brain: %w", err)
|
||||
}
|
||||
if err := os.WriteFile(dest, []byte(sb.String()), 0o644); err != nil {
|
||||
defer func() { _ = bfs.Close() }()
|
||||
if err := bfs.PutRegular(dest, []byte(sb.String())); err != nil {
|
||||
return fmt.Errorf("write claude session archive: %w", err)
|
||||
}
|
||||
s.logger.Debug("claude session archived (non-indexed)", "path", dest)
|
||||
@@ -144,6 +150,37 @@ func envInt(key string, fallback int) int {
|
||||
return fallback
|
||||
}
|
||||
|
||||
// ingestClientPattern picks the client-content pattern for raw/ routing.
|
||||
// INGEST_CLIENT_BLOCK is a complete regexp and is used as is. Otherwise the
|
||||
// shared CLAUDE_INGEST_CLIENT_BLOCK is used: the deployment injects it as a bare
|
||||
// list of client names (regexp alternation), which wing.WrapNames bounds so a
|
||||
// short name does not match inside ordinary words, and which it refuses if it
|
||||
// would break out of its group or match the empty string.
|
||||
func ingestClientPattern(getenv func(string) string) (string, error) {
|
||||
if p := getenv("INGEST_CLIENT_BLOCK"); p != "" {
|
||||
return p, nil
|
||||
}
|
||||
if names := getenv("CLAUDE_INGEST_CLIENT_BLOCK"); names != "" {
|
||||
return wing.WrapNames(names)
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
|
||||
// envFlag reads a boolean switch. Unset is off; a value strconv.ParseBool does
|
||||
// not understand is an error, so a typo ("ture") cannot silently leave a
|
||||
// safety-relevant switch off.
|
||||
func envFlag(key string) (bool, error) {
|
||||
v := os.Getenv(key)
|
||||
if v == "" {
|
||||
return false, nil
|
||||
}
|
||||
b, err := strconv.ParseBool(v)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("%s=%q: want 1, 0, true or false", key, v)
|
||||
}
|
||||
return b, nil
|
||||
}
|
||||
|
||||
// buildNotifier returns the ntfy notifier, or a nil interface when
|
||||
// BRAIN_NTFY_URL is unset. A nil *interface* (not a typed-nil) is required so
|
||||
// DegradingSink/Reconcile and the governance observer skip it cleanly.
|
||||
@@ -179,9 +216,12 @@ func alertingWarnings(getenv func(string) string) []string {
|
||||
return out
|
||||
}
|
||||
|
||||
func buildNotifier() audit.Notifier {
|
||||
if n := audit.NewNtfyNotifier(os.Getenv("BRAIN_NTFY_URL"), os.Getenv("BRAIN_NTFY_TOKEN")); n != nil {
|
||||
return n
|
||||
// buildNotifier returns the ntfy notifier for one alert source, titled so
|
||||
// the source is recognisable (hyperguild#163), or nil when ntfy is not
|
||||
// configured.
|
||||
func buildNotifier(getenv func(string) string, title string) audit.Notifier {
|
||||
if n := audit.NewNtfyNotifier(getenv("BRAIN_NTFY_URL"), getenv("BRAIN_NTFY_TOKEN")); n != nil {
|
||||
return n.WithTitle(title)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -203,7 +243,7 @@ func buildAuditSink(ctx context.Context, brainDir string, logger *slog.Logger) c
|
||||
logger.Error("capture audit buffer init", "err", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
notifier := buildNotifier()
|
||||
notifier := buildNotifier(os.Getenv, "brain-capture audit degraded")
|
||||
reconcileInterval := time.Duration(envInt("BRAIN_AUDIT_RECONCILE_INTERVAL", 60)) * time.Second
|
||||
audit.StartReconcile(ctx, central, buffer, notifier, reconcileInterval)
|
||||
logger.Info("capture audit: loki+buffer sink", "loki", lokiURL, "reconcile_s", int(reconcileInterval.Seconds()))
|
||||
@@ -366,7 +406,7 @@ func main() {
|
||||
|
||||
ctx := context.Background()
|
||||
// hyperguild#157: report a symlink that appears in the brain while running.
|
||||
go symlinkguard.Watch(ctx, brainDir, symlinkScanInterval(os.Getenv), buildNotifier())
|
||||
go symlinkguard.Watch(ctx, brainDir, symlinkScanInterval(os.Getenv), buildNotifier(os.Getenv, "brain symlink tripwire"))
|
||||
|
||||
// Never fatal on an unreachable issuer (#83): on a cold cluster boot this
|
||||
// pod can beat dex to serving and get a 502 from ingress, and exiting meant
|
||||
@@ -386,10 +426,40 @@ func main() {
|
||||
}
|
||||
|
||||
if watchInterval > 0 {
|
||||
// INGEST_WING_ROUTING=1 files raw/ notes by rule (hyperguild#208): a
|
||||
// deliberate raw/<wing>/ folder or `wing:` frontmatter, else the default
|
||||
// wing (INGEST_DEFAULT_WING, "web-clip"); a note whose text matches the
|
||||
// client-content pattern (INGEST_CLIENT_BLOCK, else
|
||||
// CLAUDE_INGEST_CLIENT_BLOCK) or that names an unknown wing is parked in
|
||||
// raw/needs-review/. Off = the retired layout. Set explicitly, a bad
|
||||
// default or pattern is fatal: silently falling back would write into
|
||||
// the layout this flag exists to leave. The interface stays nil when off
|
||||
// (a nil *Router inside it would not be nil).
|
||||
var placer wing.Placer
|
||||
routing, ferr := envFlag("INGEST_WING_ROUTING")
|
||||
if ferr != nil {
|
||||
logger.Error("INGEST_WING_ROUTING", "err", ferr)
|
||||
os.Exit(1)
|
||||
}
|
||||
if routing {
|
||||
pattern, perr := ingestClientPattern(os.Getenv)
|
||||
if perr != nil {
|
||||
logger.Error("client-content pattern", "err", perr)
|
||||
os.Exit(1)
|
||||
}
|
||||
router, rerr := wing.NewRouter(brainDir, envOr("INGEST_DEFAULT_WING", "web-clip"), pattern)
|
||||
if rerr != nil {
|
||||
logger.Error("build wing router", "err", rerr)
|
||||
os.Exit(1)
|
||||
}
|
||||
placer = router
|
||||
logger.Info("wing routing enabled", "default_wing", router.Default, "client_filter", router.ClientBlock != nil)
|
||||
}
|
||||
watcher.Start(ctx, watcher.Config{
|
||||
BrainDir: brainDir,
|
||||
Interval: time.Duration(watchInterval) * time.Second,
|
||||
Pipeline: pipelineCfg,
|
||||
Resolver: placer,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -546,7 +616,7 @@ func main() {
|
||||
// capture already gets, plus an out-of-band alert. Wired here because
|
||||
// brain_edit_root needs the capture plumbing anyway — that is what
|
||||
// re-derives the static principal its gate checks.
|
||||
mcpSrv.WithGovernanceObserver(auditSink, buildNotifier())
|
||||
mcpSrv.WithGovernanceObserver(auditSink, buildNotifier(os.Getenv, "brain governance edit"))
|
||||
logger.Info("capture enabled (REST + MCP tool)", "sovereign_principals", len(sovereign),
|
||||
"governance_observer", true)
|
||||
} else {
|
||||
|
||||
@@ -76,3 +76,83 @@ func TestAlertingWarnings(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// hyperguild#158: the claude-session archive was written with os.MkdirAll +
|
||||
// os.WriteFile by path, so a symlinked archive directory redirected it.
|
||||
func TestClaudeSinkDoesNotWriteThroughASymlinkedArchiveDir(t *testing.T) {
|
||||
for name, mk := range map[string]func(t *testing.T, dir string) (link, target string){
|
||||
"outside": func(t *testing.T, _ string) (string, string) { o := t.TempDir(); return o, o },
|
||||
"in-brain": func(t *testing.T, dir string) (string, string) {
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(dir, "raw"), 0o755))
|
||||
return filepath.Join("..", "raw"), filepath.Join(dir, "raw")
|
||||
},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
link, target := mk(t, dir)
|
||||
require.NoError(t, os.Symlink(link, filepath.Join(dir, "archive")))
|
||||
sink := &claudeSink{brainDir: dir, logger: slog.New(slog.NewTextHandler(os.Stderr, nil))}
|
||||
err := sink.Ingest(context.Background(), claudewatcher.Batch{Host: "koala", SessionID: "abc",
|
||||
Turns: []claudewatcher.Turn{{Type: "assistant", Content: "x"}}})
|
||||
assert.Error(t, err)
|
||||
ents, _ := os.ReadDir(target)
|
||||
assert.Empty(t, ents, "the archive was written at the link's target")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnvFlag(t *testing.T) {
|
||||
tests := []struct {
|
||||
name, set string
|
||||
want bool
|
||||
wantErr bool
|
||||
}{
|
||||
{"unset is off", "", false, false},
|
||||
{"1", "1", true, false},
|
||||
{"true", "true", true, false},
|
||||
{"TRUE", "TRUE", true, false},
|
||||
{"0", "0", false, false},
|
||||
{"false", "false", false, false},
|
||||
{"a typo is an error, not silently off", "ture", false, true},
|
||||
{"yes is not a bool here", "yes", false, true},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Setenv("TEST_ENV_FLAG", tt.set)
|
||||
got, err := envFlag("TEST_ENV_FLAG")
|
||||
if tt.wantErr {
|
||||
assert.Error(t, err)
|
||||
return
|
||||
}
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, tt.want, got)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The deployment injects CLAUDE_INGEST_CLIENT_BLOCK as a bare list of client
|
||||
// names (regexp alternation). Used raw, a short name would match inside ordinary
|
||||
// words; wrapped with \b, a name with a non-ASCII edge letter would never match.
|
||||
func TestIngestClientPattern(t *testing.T) {
|
||||
get := func(m map[string]string) func(string) string { return func(k string) string { return m[k] } }
|
||||
t.Run("nothing set", func(t *testing.T) {
|
||||
p, err := ingestClientPattern(get(map[string]string{}))
|
||||
assert.NoError(t, err)
|
||||
assert.Empty(t, p)
|
||||
})
|
||||
t.Run("explicit pattern is a full regexp, used as is", func(t *testing.T) {
|
||||
p, err := ingestClientPattern(get(map[string]string{"INGEST_CLIENT_BLOCK": `acme\s+corp`, "CLAUDE_INGEST_CLIENT_BLOCK": "other"}))
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, `acme\s+corp`, p)
|
||||
})
|
||||
t.Run("the shared names are wrapped", func(t *testing.T) {
|
||||
p, err := ingestClientPattern(get(map[string]string{"CLAUDE_INGEST_CLIENT_BLOCK": "dif|trucker"}))
|
||||
assert.NoError(t, err)
|
||||
assert.Contains(t, p, "dif|trucker")
|
||||
assert.NotEqual(t, "dif|trucker", p)
|
||||
})
|
||||
t.Run("a shared value that would break out of the group is a startup error", func(t *testing.T) {
|
||||
_, err := ingestClientPattern(get(map[string]string{"CLAUDE_INGEST_CLIENT_BLOCK": "x)|.*("}))
|
||||
assert.Error(t, err)
|
||||
})
|
||||
}
|
||||
|
||||
@@ -159,3 +159,16 @@ func TestBuildMuxObservesRequestsAndServesMetrics(t *testing.T) {
|
||||
require.Equal(t, http.StatusOK, code)
|
||||
assert.Contains(t, body, `path="GET /pending"`)
|
||||
}
|
||||
|
||||
// hyperguild#185: an authenticated request to an unregistered path reached
|
||||
// the inner REST mux, which left Pattern empty, and the metrics label fell
|
||||
// back to the raw URL path: one Prometheus series per unique path any token
|
||||
// holder sends. It is now labelled with a fixed catch-all.
|
||||
func TestBuildMuxLabelsAnUnregisteredPathWithAFixedLabel(t *testing.T) {
|
||||
r := testMux(t)
|
||||
code, _ := do(t, r, "GET", "/zqrandom-185-segment", "Bearer "+routesTok)
|
||||
require.Equal(t, http.StatusNotFound, code)
|
||||
_, body := do(t, r, "GET", "/metrics", "")
|
||||
assert.NotContains(t, body, "zqrandom-185-segment", "the raw path became a metrics label")
|
||||
assert.Contains(t, body, `path="unmatched"`)
|
||||
}
|
||||
|
||||
@@ -1,9 +1,13 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"syscall"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
@@ -63,3 +67,26 @@ func TestStartupSymlinkCheckRefusesAnUnlistableBrainRoot(t *testing.T) {
|
||||
t.Cleanup(func() { _ = os.Chmod(brain, 0o755) })
|
||||
assert.Error(t, startupSymlinkCheck(brain))
|
||||
}
|
||||
|
||||
// hyperguild#179: a FIFO in the brain is refused at startup with a clear
|
||||
// message, instead of hanging or exiting later in a reader.
|
||||
func TestStartupSymlinkCheckRefusesABrainWithAFIFO(t *testing.T) {
|
||||
brain := t.TempDir()
|
||||
require.NoError(t, syscall.Mkfifo(filepath.Join(brain, "classification.yaml"), 0o644))
|
||||
err := startupSymlinkCheck(brain)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "classification.yaml")
|
||||
}
|
||||
|
||||
// hyperguild#163: each alert source gets its own ntfy title.
|
||||
func TestBuildNotifierUsesTheGivenTitle(t *testing.T) {
|
||||
var got string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { got = r.Header.Get("Title") }))
|
||||
defer srv.Close()
|
||||
env := map[string]string{"BRAIN_NTFY_URL": srv.URL}
|
||||
n := buildNotifier(func(k string) string { return env[k] }, "brain symlink tripwire")
|
||||
require.NotNil(t, n)
|
||||
require.NoError(t, n.Notify(context.Background(), "m"))
|
||||
assert.Equal(t, "brain symlink tripwire", got)
|
||||
assert.Nil(t, buildNotifier(func(string) string { return "" }, "x"), "no ntfy URL, no notifier")
|
||||
}
|
||||
|
||||
+1
-1
@@ -69,6 +69,6 @@ require (
|
||||
golang.org/x/crypto v0.51.0 // indirect
|
||||
golang.org/x/sync v0.21.0 // indirect
|
||||
golang.org/x/sys v0.45.0
|
||||
golang.org/x/text v0.39.0 // indirect
|
||||
golang.org/x/text v0.39.0
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
)
|
||||
|
||||
@@ -83,6 +83,18 @@ func (f *frontmatter) get(key string) string {
|
||||
return ""
|
||||
}
|
||||
|
||||
// drop removes every `key: value` line for key (nothing else), so a
|
||||
// machine-written key does not travel with a note into the wiki.
|
||||
func (f *frontmatter) drop(key string) {
|
||||
kept := f.lines[:0]
|
||||
for _, l := range f.lines {
|
||||
if l.key != key {
|
||||
kept = append(kept, l)
|
||||
}
|
||||
}
|
||||
f.lines = kept
|
||||
}
|
||||
|
||||
// set overrides the value for an existing key in place, or appends a new
|
||||
// `key: value` line when the key is absent.
|
||||
func (f *frontmatter) set(key, value string) {
|
||||
|
||||
@@ -617,6 +617,7 @@ type promoteRequest struct {
|
||||
Wing string `json:"wing"`
|
||||
Hall string `json:"hall"`
|
||||
Slug string `json:"slug,omitempty"`
|
||||
Queue string `json:"queue,omitempty"`
|
||||
}
|
||||
|
||||
// Promote handles POST /promote — move a raw/ note into the wiki. A bad
|
||||
@@ -630,7 +631,7 @@ func (h *Handler) Promote(w http.ResponseWriter, r *http.Request) {
|
||||
// Field by field: this door carries no principal, so the promoted note is
|
||||
// stamped origin: unknown and the body cannot set provenance (#145).
|
||||
rel, err := PromoteNote(h.brainDir, PromoteOptions{
|
||||
Filename: req.Filename, Wing: req.Wing, Hall: req.Hall, Slug: req.Slug,
|
||||
Filename: req.Filename, Wing: req.Wing, Hall: req.Hall, Slug: req.Slug, Queue: req.Queue,
|
||||
})
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func writeParked(t *testing.T, brainDir, name, content string) {
|
||||
t.Helper()
|
||||
dir := filepath.Join(brainDir, "raw", "needs-review")
|
||||
require.NoError(t, os.MkdirAll(dir, 0o755))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(dir, name), []byte(content), 0o644))
|
||||
}
|
||||
|
||||
func TestListPending_ShowsParkedNotesWithTheirProposal(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
writeRaw(t, dir, "fresh.md", "---\ncreated_at: 2026-10-20T08:00:00Z\n---\nA note not yet processed.\n")
|
||||
writeParked(t, dir, "unsure.md", "---\nproposed_wing: radar\nproposal_confidence: 0.55\nproposal_reason: \"below threshold\"\ncreated_at: 2026-10-19T08:00:00Z\n---\nBody of the unsure note.\n")
|
||||
|
||||
got, err := ListPending(dir)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, got, 2)
|
||||
|
||||
assert.Equal(t, "unsure.md", got[0].Filename, "oldest first across both queues")
|
||||
assert.Equal(t, "needs-review", got[0].Queue)
|
||||
assert.Equal(t, "radar", got[0].ProposedWing)
|
||||
assert.Equal(t, "below threshold", got[0].ProposalReason)
|
||||
assert.InDelta(t, 0.55, got[0].ProposalConfidence, 1e-9)
|
||||
assert.Contains(t, got[0].Excerpt, "Body of the unsure note")
|
||||
assert.NotContains(t, got[0].Excerpt, "proposed_wing", "excerpt is body, not frontmatter")
|
||||
|
||||
assert.Equal(t, "fresh.md", got[1].Filename)
|
||||
assert.Equal(t, QueueInbox, got[1].Queue)
|
||||
assert.Empty(t, got[1].ProposedWing)
|
||||
}
|
||||
|
||||
func TestListPending_ParkedSymlinkIsNotListed(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
outside := filepath.Join(t.TempDir(), "secret.md")
|
||||
require.NoError(t, os.WriteFile(outside, []byte("outside"), 0o644))
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(dir, "raw", "needs-review"), 0o755))
|
||||
require.NoError(t, os.Symlink(outside, filepath.Join(dir, "raw", "needs-review", "link.md")))
|
||||
|
||||
got, err := ListPending(dir)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, got, "same symlink rule as the top level (#153)")
|
||||
}
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -20,6 +21,15 @@ type PendingNote struct {
|
||||
CreatedAt string `json:"created_at"`
|
||||
SizeBytes int64 `json:"size_bytes"`
|
||||
Excerpt string `json:"excerpt"`
|
||||
|
||||
// Set only for a note parked in raw/needs-review/ by the wing resolver
|
||||
// (hyperguild#205): it was not placed automatically, and the proposal says
|
||||
// what the resolver would have done and why it did not. Approve by moving
|
||||
// the note into raw/<wing>/ and the watcher ingests it there.
|
||||
Queue string `json:"queue"`
|
||||
ProposedWing string `json:"proposed_wing,omitempty"`
|
||||
ProposalReason string `json:"proposal_reason,omitempty"`
|
||||
ProposalConfidence float64 `json:"proposal_confidence,omitempty"`
|
||||
}
|
||||
|
||||
// datePrefix matches a leading YYYY-MM-DD- on a raw filename, stripped when
|
||||
@@ -38,18 +48,65 @@ func ListPending(brainDir string) ([]PendingNote, error) {
|
||||
return nil, fmt.Errorf("read raw dir: %w", err)
|
||||
}
|
||||
defer func() { _ = root.Close() }()
|
||||
|
||||
out, err := listNotes(root, QueueInbox)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// raw/needs-review/ is read through its own root, so a symlinked directory
|
||||
// or note there is refused exactly like at the top level (#153).
|
||||
if sub, serr := root.OpenRoot("needs-review"); serr == nil {
|
||||
defer func() { _ = sub.Close() }()
|
||||
parked, perr := listNotes(sub, QueueNeedsReview)
|
||||
if perr != nil {
|
||||
return nil, perr
|
||||
}
|
||||
out = append(out, parked...)
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool { return out[i].CreatedAt < out[j].CreatedAt })
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// listNotes lists the regular .md notes directly under root. queue labels the
|
||||
// notes ("" for the top of raw/) and enables reading a wing proposal.
|
||||
func listNotes(root *os.Root, queue string) ([]PendingNote, error) {
|
||||
entries, err := fs.ReadDir(root.FS(), ".")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read raw dir: %w", err)
|
||||
}
|
||||
|
||||
out := make([]PendingNote, 0, len(entries))
|
||||
for _, e := range entries {
|
||||
if !e.Type().IsRegular() || !strings.HasSuffix(e.Name(), ".md") || strings.HasPrefix(e.Name(), "tunnel-candidates-") {
|
||||
isMD := strings.HasSuffix(e.Name(), ".md")
|
||||
// A parked pdf or txt is listed too (the watcher parks any supported
|
||||
// file); at the top level only markdown ever was.
|
||||
parkedOther := queue == QueueNeedsReview && (strings.HasSuffix(e.Name(), ".txt") || strings.HasSuffix(e.Name(), ".pdf"))
|
||||
if !e.Type().IsRegular() || (!isMD && !parkedOther) || strings.HasPrefix(e.Name(), "tunnel-candidates-") {
|
||||
continue // directories, and symlinks whose target may be anywhere (#153)
|
||||
}
|
||||
if !isMD {
|
||||
info, ierr := root.Lstat(e.Name())
|
||||
if ierr != nil {
|
||||
continue
|
||||
}
|
||||
out = append(out, PendingNote{
|
||||
Filename: e.Name(), CreatedAt: info.ModTime().UTC().Format(time.RFC3339), SizeBytes: info.Size(),
|
||||
Excerpt: "(not markdown; the wing proposal is in log.md. Move the file into raw/<wing>/ to ingest it.)",
|
||||
Queue: queue,
|
||||
})
|
||||
continue
|
||||
}
|
||||
raw, info, readErr := readRawNote(root, e.Name())
|
||||
if readErr != nil {
|
||||
// Not skipped: a note that cannot be read (over the cap, say) must
|
||||
// not silently vanish from the queue it is waiting in.
|
||||
fi, ierr := root.Lstat(e.Name())
|
||||
if ierr != nil {
|
||||
continue
|
||||
}
|
||||
out = append(out, PendingNote{
|
||||
Filename: e.Name(), CreatedAt: fi.ModTime().UTC().Format(time.RFC3339), SizeBytes: fi.Size(),
|
||||
Excerpt: "(cannot be read: " + readErr.Error() + "; promote and reject cannot act on it: delete or move it by hand)", Queue: queue,
|
||||
})
|
||||
continue
|
||||
}
|
||||
fm, body := parseFrontmatter(string(raw))
|
||||
@@ -57,14 +114,20 @@ func ListPending(brainDir string) ([]PendingNote, error) {
|
||||
if created == "" {
|
||||
created = info.ModTime().UTC().Format(time.RFC3339)
|
||||
}
|
||||
out = append(out, PendingNote{
|
||||
n := PendingNote{
|
||||
Filename: e.Name(),
|
||||
CreatedAt: created,
|
||||
SizeBytes: info.Size(),
|
||||
Excerpt: excerpt(body, 200),
|
||||
})
|
||||
Queue: queue,
|
||||
}
|
||||
if queue == QueueNeedsReview {
|
||||
n.ProposedWing = fm.get("proposed_wing")
|
||||
n.ProposalReason = unquoteScalar(fm.get("proposal_reason"))
|
||||
n.ProposalConfidence, _ = strconv.ParseFloat(fm.get("proposal_confidence"), 64)
|
||||
}
|
||||
out = append(out, n)
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool { return out[i].CreatedAt < out[j].CreatedAt })
|
||||
return out, nil
|
||||
}
|
||||
|
||||
@@ -74,6 +137,9 @@ type PromoteOptions struct {
|
||||
Wing string
|
||||
Hall string
|
||||
Slug string // optional; defaults to Filename minus date prefix + .md
|
||||
// Queue names which listing the note is in: QueueInbox or QueueNeedsReview.
|
||||
// Empty is accepted only when the name exists in exactly one of them.
|
||||
Queue string
|
||||
// Origin/WrittenBy re-attribute the promoted note to the promoter: the raw
|
||||
// note's own frontmatter claims are not provenance (hyperguild#145).
|
||||
Origin string
|
||||
@@ -96,21 +162,30 @@ func PromoteNote(brainDir string, opts PromoteOptions) (string, error) {
|
||||
return "", fmt.Errorf("invalid filename %q", opts.Filename)
|
||||
}
|
||||
|
||||
if !strings.HasSuffix(base, ".md") {
|
||||
return "", fmt.Errorf("only a markdown note can be promoted; move %q into raw/<wing>/ to ingest it", base)
|
||||
}
|
||||
slug := opts.Slug
|
||||
if slug == "" {
|
||||
slug = datePrefix.ReplaceAllString(strings.TrimSuffix(base, ".md"), "")
|
||||
}
|
||||
// The retired layout's directory names are not wings: a note promoted into
|
||||
// "sources" would land in wiki/sources/<hall>/, inside the retired dir.
|
||||
switch brain.Sanitise(opts.Wing) {
|
||||
case "sources", "concepts", "entities", "wiki":
|
||||
return "", fmt.Errorf("%q is not a wing", opts.Wing)
|
||||
}
|
||||
// NotePath validates hall + wing + slug; do this before reading anything.
|
||||
dest, err := brain.NotePath(brainDir, opts.Wing, opts.Hall, slug)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
src := filepath.Join(brainDir, "raw", base)
|
||||
raw, err := readRawSource(brainDir, base)
|
||||
raw, srcRel, err := readRawSource(brainDir, base, opts.Queue)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
src := filepath.Join(brainDir, filepath.FromSlash(srcRel))
|
||||
|
||||
// Collision: never silently overwrite an existing promoted note.
|
||||
if _, statErr := os.Stat(dest); statErr == nil {
|
||||
@@ -119,6 +194,11 @@ func PromoteNote(brainDir string, opts PromoteOptions) (string, error) {
|
||||
}
|
||||
|
||||
fm, body := parseFrontmatter(string(raw))
|
||||
// The resolver's proposal described where the note was going to be placed;
|
||||
// once a human has placed it, it is noise (and `wing` is set below).
|
||||
for _, k := range []string{"proposed_wing", "proposal_confidence", "proposal_reason"} {
|
||||
fm.drop(k)
|
||||
}
|
||||
now := time.Now().UTC().Format(time.RFC3339)
|
||||
fm.set("wing", brain.Sanitise(opts.Wing))
|
||||
fm.set("hall", opts.Hall)
|
||||
@@ -156,6 +236,7 @@ func PromoteNote(brainDir string, opts PromoteOptions) (string, error) {
|
||||
type RejectOptions struct {
|
||||
Filename string // basename in brain/raw/
|
||||
Reason string // optional
|
||||
Queue string // as PromoteOptions.Queue
|
||||
}
|
||||
|
||||
// RejectNote records a human decision not to promote a brain/raw/ note,
|
||||
@@ -178,11 +259,11 @@ func RejectNote(brainDir string, opts RejectOptions) (string, error) {
|
||||
return "", fmt.Errorf("invalid filename %q", opts.Filename)
|
||||
}
|
||||
|
||||
src := filepath.Join(brainDir, "raw", base)
|
||||
raw, err := readRawSource(brainDir, base)
|
||||
raw, srcRel, err := readRawSource(brainDir, base, opts.Queue)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
src := filepath.Join(brainDir, filepath.FromSlash(srcRel))
|
||||
|
||||
dest := filepath.Join(brainDir, "raw", "rejected", base)
|
||||
if _, statErr := os.Stat(dest); statErr == nil {
|
||||
@@ -190,15 +271,19 @@ func RejectNote(brainDir string, opts RejectOptions) (string, error) {
|
||||
return "", fmt.Errorf("a rejected note named %s already exists", filepath.ToSlash(rel))
|
||||
}
|
||||
|
||||
fm, body := parseFrontmatter(string(raw))
|
||||
fm.set("rejected_at", time.Now().UTC().Format(time.RFC3339))
|
||||
if opts.Reason != "" {
|
||||
fm.set("reject_reason", opts.Reason)
|
||||
}
|
||||
content := string(raw)
|
||||
if strings.HasSuffix(base, ".md") {
|
||||
fm, body := parseFrontmatter(string(raw))
|
||||
fm.set("rejected_at", time.Now().UTC().Format(time.RFC3339))
|
||||
if opts.Reason != "" {
|
||||
fm.set("reject_reason", opts.Reason)
|
||||
}
|
||||
content = fm.render() + body
|
||||
} // a pdf or txt is moved byte for byte: prepending frontmatter would alter it
|
||||
|
||||
// Write-then-delete: the source survives any write failure. Both go
|
||||
// through the brain root (hyperguild#188).
|
||||
if err := createBrainFile(brainDir, dest, []byte(fm.render()+body)); err != nil {
|
||||
if err := createBrainFile(brainDir, dest, []byte(content)); err != nil {
|
||||
return "", fmt.Errorf("write rejected note: %w", err)
|
||||
}
|
||||
if err := removeBrainFile(brainDir, src); err != nil {
|
||||
@@ -221,21 +306,67 @@ func excerpt(s string, n int) string {
|
||||
|
||||
// readRawSource reads a pending note for promote/reject through the raw root,
|
||||
// refusing symlinks and a symlinked raw/ (hyperguild#153).
|
||||
func readRawSource(brainDir, base string) ([]byte, error) {
|
||||
func readRawSource(brainDir, base, queue string) ([]byte, string, error) {
|
||||
if queue != "" && queue != QueueInbox && queue != QueueNeedsReview {
|
||||
return nil, "", fmt.Errorf("unknown queue %q (want %q or %q)", queue, QueueInbox, QueueNeedsReview)
|
||||
}
|
||||
root, _, err := OpenRawRoot(brainDir)
|
||||
if err != nil {
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return nil, fmt.Errorf("pending note %q does not exist in raw/", base)
|
||||
return nil, "", fmt.Errorf("pending note %q does not exist in raw/", base)
|
||||
}
|
||||
return nil, fmt.Errorf("read source: %w", err)
|
||||
return nil, "", fmt.Errorf("read source: %w", err)
|
||||
}
|
||||
defer func() { _ = root.Close() }()
|
||||
raw, _, err := readRawNote(root, base)
|
||||
if err != nil {
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return nil, fmt.Errorf("pending note %q does not exist in raw/", base)
|
||||
}
|
||||
return nil, fmt.Errorf("read source: %w", err)
|
||||
|
||||
var topRaw, parkedRaw []byte
|
||||
var topErr, parkedErr error
|
||||
if queue != QueueNeedsReview {
|
||||
topRaw, _, topErr = readRawNote(root, base)
|
||||
}
|
||||
return raw, nil
|
||||
if queue != QueueInbox {
|
||||
// The sub-root refuses a symlinked directory like OpenRawRoot does.
|
||||
if sub, serr := root.OpenRoot(parkedDir); serr != nil {
|
||||
parkedErr = serr
|
||||
} else {
|
||||
parkedRaw, _, parkedErr = readRawNote(sub, base)
|
||||
_ = sub.Close()
|
||||
}
|
||||
}
|
||||
haveTop := queue != QueueNeedsReview && topErr == nil
|
||||
haveParked := queue != QueueInbox && parkedErr == nil
|
||||
switch {
|
||||
case haveTop && haveParked:
|
||||
return nil, "", fmt.Errorf("%q is in both raw/ and raw/%s/: name the queue (%q or %q)", base, parkedDir, QueueInbox, QueueNeedsReview)
|
||||
case haveTop:
|
||||
return topRaw, "raw/" + base, nil
|
||||
case haveParked:
|
||||
return parkedRaw, "raw/" + parkedDir + "/" + base, nil
|
||||
}
|
||||
for _, e := range []error{topErr, parkedErr} {
|
||||
if e != nil && !errors.Is(e, os.ErrNotExist) {
|
||||
return nil, "", fmt.Errorf("read source: %w", e)
|
||||
}
|
||||
}
|
||||
return nil, "", fmt.Errorf("pending note %q does not exist in raw/", base)
|
||||
}
|
||||
|
||||
// parkedDir is where the watcher parks notes it could not place.
|
||||
const parkedDir = "needs-review"
|
||||
|
||||
// The two review queues brain_pending lists. promote and reject take one of
|
||||
// them (or neither, when the name is unambiguous).
|
||||
const (
|
||||
QueueInbox = "inbox" // raw/<name>
|
||||
QueueNeedsReview = "needs-review" // raw/needs-review/<name>
|
||||
)
|
||||
|
||||
// unquoteScalar undoes the Go-style double quoting the watcher writes around a
|
||||
// free-text value (strconv.Quote keeps it on one line); a bare value is
|
||||
// returned as is.
|
||||
func unquoteScalar(v string) string {
|
||||
if u, err := strconv.Unquote(v); err == nil {
|
||||
return u
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
@@ -0,0 +1,210 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// brain_pending lists a top-level note and a parked note of the same name as two
|
||||
// entries with the same filename. Acting on "a.md" must not silently pick one.
|
||||
func TestPromoteNote_ANameInBothQueuesNeedsTheQueueToBeNamed(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
writeRaw(t, dir, "a.md", "---\ncreated_at: 2026-10-20T08:00:00Z\n---\nTop-level body.\n")
|
||||
writeParked(t, dir, "a.md", parkedNote)
|
||||
|
||||
_, err := PromoteNote(dir, PromoteOptions{Filename: "a.md", Wing: "homelab", Hall: "facts"})
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "both")
|
||||
assert.Contains(t, err.Error(), "queue")
|
||||
_, statErr := os.Stat(filepath.Join(dir, "wiki/homelab/facts/a.md"))
|
||||
assert.ErrorIs(t, statErr, os.ErrNotExist, "nothing was promoted")
|
||||
|
||||
rel, err := PromoteNote(dir, PromoteOptions{Filename: "a.md", Wing: "homelab", Hall: "facts", Queue: QueueNeedsReview})
|
||||
require.NoError(t, err)
|
||||
got, _ := os.ReadFile(filepath.Join(dir, rel))
|
||||
assert.Contains(t, string(got), "Body of the parked note.")
|
||||
_, statErr = os.Stat(filepath.Join(dir, "raw", "a.md"))
|
||||
assert.NoError(t, statErr, "the top-level note of the same name is untouched")
|
||||
}
|
||||
|
||||
func TestPromoteNote_QueueInboxSelectsTheTopLevelNote(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
writeRaw(t, dir, "a.md", "---\ncreated_at: 2026-10-20T08:00:00Z\n---\nTop-level body.\n")
|
||||
writeParked(t, dir, "a.md", parkedNote)
|
||||
|
||||
rel, err := PromoteNote(dir, PromoteOptions{Filename: "a.md", Wing: "homelab", Hall: "facts", Queue: QueueInbox})
|
||||
require.NoError(t, err)
|
||||
got, _ := os.ReadFile(filepath.Join(dir, rel))
|
||||
assert.Contains(t, string(got), "Top-level body.")
|
||||
_, statErr := os.Stat(filepath.Join(dir, "raw", "needs-review", "a.md"))
|
||||
assert.NoError(t, statErr, "the parked note is untouched")
|
||||
}
|
||||
|
||||
func TestPromoteNote_ANamedQueueThatDoesNotHoldTheNoteIsAnError(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
writeRaw(t, dir, "a.md", "top")
|
||||
_, err := PromoteNote(dir, PromoteOptions{Filename: "a.md", Wing: "homelab", Hall: "facts", Queue: QueueNeedsReview})
|
||||
assert.Error(t, err)
|
||||
_, err = PromoteNote(dir, PromoteOptions{Filename: "a.md", Wing: "homelab", Hall: "facts", Queue: "elsewhere"})
|
||||
assert.Error(t, err, "an unknown queue name is refused")
|
||||
}
|
||||
|
||||
func TestRejectNote_ANameInBothQueuesNeedsTheQueueToBeNamed(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
writeRaw(t, dir, "a.md", "top")
|
||||
writeParked(t, dir, "a.md", parkedNote)
|
||||
|
||||
_, err := RejectNote(dir, RejectOptions{Filename: "a.md"})
|
||||
require.Error(t, err)
|
||||
_, err = RejectNote(dir, RejectOptions{Filename: "a.md", Queue: QueueNeedsReview})
|
||||
require.NoError(t, err)
|
||||
_, statErr := os.Stat(filepath.Join(dir, "raw", "a.md"))
|
||||
assert.NoError(t, statErr, "the top-level note is untouched")
|
||||
}
|
||||
|
||||
func TestPromoteNote_NeverIntoARetiredLayoutName(t *testing.T) {
|
||||
for _, w := range []string{"sources", "concepts", "entities", "wiki"} {
|
||||
t.Run(w, func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
writeRaw(t, dir, "a.md", "top")
|
||||
_, err := PromoteNote(dir, PromoteOptions{Filename: "a.md", Wing: w, Hall: "facts"})
|
||||
assert.Error(t, err)
|
||||
_, statErr := os.Stat(filepath.Join(dir, "wiki", w))
|
||||
assert.ErrorIs(t, statErr, os.ErrNotExist)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestListPending_LabelsEveryEntryWithItsQueue(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
writeRaw(t, dir, "fresh.md", "---\ncreated_at: 2026-10-20T08:00:00Z\n---\nx\n")
|
||||
writeParked(t, dir, "p.md", parkedNote)
|
||||
got, err := ListPending(dir)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, got, 2)
|
||||
queues := map[string]string{}
|
||||
for _, n := range got {
|
||||
queues[n.Filename] = n.Queue
|
||||
}
|
||||
assert.Equal(t, QueueInbox, queues["fresh.md"])
|
||||
assert.Equal(t, QueueNeedsReview, queues["p.md"])
|
||||
}
|
||||
|
||||
// Parked notes that are not markdown (a pdf, a txt) were invisible: the only
|
||||
// trace was a log.md line.
|
||||
func TestListPending_ShowsParkedNonMarkdownFiles(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(dir, "raw", "needs-review"), 0o755))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(dir, "raw", "needs-review", "paper.pdf"), []byte("%PDF-1.4 binary"), 0o644))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(dir, "raw", "needs-review", "plain.txt"), []byte("some text"), 0o644))
|
||||
|
||||
got, err := ListPending(dir)
|
||||
require.NoError(t, err)
|
||||
names := map[string]PendingNote{}
|
||||
for _, n := range got {
|
||||
names[n.Filename] = n
|
||||
}
|
||||
require.Contains(t, names, "paper.pdf")
|
||||
require.Contains(t, names, "plain.txt")
|
||||
assert.Equal(t, QueueNeedsReview, names["paper.pdf"].Queue)
|
||||
assert.Contains(t, names["paper.pdf"].Excerpt, "proposal", "points the reader at where the proposal is")
|
||||
}
|
||||
|
||||
func TestPromoteNote_OnlyMarkdownCanBePromoted(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(dir, "raw", "needs-review"), 0o755))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(dir, "raw", "needs-review", "paper.pdf"), []byte("%PDF binary"), 0o644))
|
||||
_, err := PromoteNote(dir, PromoteOptions{Filename: "paper.pdf", Wing: "homelab", Hall: "facts"})
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "markdown")
|
||||
_, statErr := os.Stat(filepath.Join(dir, "raw", "needs-review", "paper.pdf"))
|
||||
assert.NoError(t, statErr, "refused, untouched")
|
||||
}
|
||||
|
||||
// brain.Sanitise normalises case, trailing dots and separators; the reserved-name
|
||||
// check must look at the normalised name or "Sources" and "sources." walk past it.
|
||||
func TestPromoteNote_ReservedWingNamesAreRefusedInEveryDisguise(t *testing.T) {
|
||||
for _, w := range []string{"Sources", "sources.", "SOURCES ", "Entities/", "concepts\\", " wiki "} {
|
||||
t.Run(w, func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
writeRaw(t, dir, "a.md", "top")
|
||||
_, err := PromoteNote(dir, PromoteOptions{Filename: "a.md", Wing: w, Hall: "facts"})
|
||||
assert.Error(t, err)
|
||||
_, statErr := os.Stat(filepath.Join(dir, "raw", "a.md"))
|
||||
assert.NoError(t, statErr, "refused, untouched")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Reject moves the bytes; it must not prepend frontmatter to a pdf or txt.
|
||||
func TestRejectNote_AParkedNonMarkdownFileIsMovedUntouched(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
pdf := "%PDF-1.4\r\n\x00\xff binary \r\n"
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(dir, "raw", "needs-review"), 0o755))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(dir, "raw", "needs-review", "paper.pdf"), []byte(pdf), 0o644))
|
||||
|
||||
rel, err := RejectNote(dir, RejectOptions{Filename: "paper.pdf", Reason: "not relevant"})
|
||||
require.NoError(t, err)
|
||||
got, err := os.ReadFile(filepath.Join(dir, rel))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, pdf, string(got), "byte for byte")
|
||||
_, statErr := os.Stat(filepath.Join(dir, "raw", "needs-review", "paper.pdf"))
|
||||
assert.ErrorIs(t, statErr, os.ErrNotExist)
|
||||
}
|
||||
|
||||
func TestReadRawNote_IsBoundedByTheCap(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
require.NoError(t, os.WriteFile(filepath.Join(dir, "big.md"), []byte(strings.Repeat("x", 200)), 0o644))
|
||||
root, err := os.OpenRoot(dir)
|
||||
require.NoError(t, err)
|
||||
defer func() { _ = root.Close() }()
|
||||
prev := maxRawNoteBytes
|
||||
maxRawNoteBytes = 100
|
||||
t.Cleanup(func() { maxRawNoteBytes = prev })
|
||||
|
||||
_, _, err = readRawNote(root, "big.md")
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "exceeds")
|
||||
}
|
||||
|
||||
type endless struct{ n int64 }
|
||||
|
||||
func (e *endless) Read(p []byte) (int, error) {
|
||||
if e.n > 8<<20 { // a regression fails the test instead of exhausting memory
|
||||
return 0, errors.New("endless reader drained")
|
||||
}
|
||||
e.n += int64(len(p))
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
func TestReadCapped_StopsAtTheCap(t *testing.T) {
|
||||
e := &endless{}
|
||||
_, err := readCapped(e, 1000)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "exceeds")
|
||||
assert.LessOrEqual(t, e.n, int64(1000+1+32*1024), "pulled far more than the cap: %d", e.n)
|
||||
}
|
||||
|
||||
// An .md over the read cap used to vanish from the listing (readRawNote failed
|
||||
// and the entry was skipped). It must stay visible, with the reason.
|
||||
func TestListPending_AnUnreadableNoteIsListedWithTheReason(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(dir, "raw"), 0o755))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(dir, "raw", "huge.md"), []byte(strings.Repeat("x", 200)), 0o644))
|
||||
prev := maxRawNoteBytes
|
||||
maxRawNoteBytes = 100
|
||||
t.Cleanup(func() { maxRawNoteBytes = prev })
|
||||
|
||||
got, err := ListPending(dir)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, got, 1)
|
||||
assert.Equal(t, "huge.md", got[0].Filename)
|
||||
assert.Contains(t, got[0].Excerpt, "cannot be read")
|
||||
assert.Contains(t, got[0].Excerpt, "exceeds")
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
const parkedNote = "---\nproposed_wing: radar\nproposal_confidence: 0.40\nproposal_reason: \"text matches the client-content marker\"\ncreated_at: 2026-10-19T08:00:00Z\n---\nBody of the parked note.\n"
|
||||
|
||||
// brain_pending lists a parked note under its bare filename, so promote and
|
||||
// reject must be able to act on exactly that name (#208: they could not).
|
||||
func TestPromoteNote_ActsOnANoteParkedInNeedsReview(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
writeParked(t, dir, "unsure.md", parkedNote)
|
||||
|
||||
rel, err := PromoteNote(dir, PromoteOptions{Filename: "unsure.md", Wing: "homelab", Hall: "facts"})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "wiki/homelab/facts/unsure.md", rel)
|
||||
|
||||
got, err := os.ReadFile(filepath.Join(dir, "wiki/homelab/facts/unsure.md"))
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, string(got), "Body of the parked note.")
|
||||
assert.Contains(t, string(got), "wing: homelab")
|
||||
assert.NotContains(t, string(got), "proposed_wing", "the resolver's proposal is not part of the promoted note")
|
||||
assert.NotContains(t, string(got), "proposal_reason")
|
||||
|
||||
_, statErr := os.Stat(filepath.Join(dir, "raw", "needs-review", "unsure.md"))
|
||||
assert.ErrorIs(t, statErr, os.ErrNotExist, "the parked note is consumed")
|
||||
pending, err := ListPending(dir)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, pending)
|
||||
}
|
||||
|
||||
func TestRejectNote_ActsOnANoteParkedInNeedsReview(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
writeParked(t, dir, "unsure.md", parkedNote)
|
||||
|
||||
rel, err := RejectNote(dir, RejectOptions{Filename: "unsure.md", Reason: "not worth keeping"})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "raw/rejected/unsure.md", rel)
|
||||
_, statErr := os.Stat(filepath.Join(dir, "raw", "needs-review", "unsure.md"))
|
||||
assert.ErrorIs(t, statErr, os.ErrNotExist)
|
||||
pending, _ := ListPending(dir)
|
||||
assert.Empty(t, pending, "a rejected note is not re-listed")
|
||||
}
|
||||
|
||||
func TestPromoteNote_ASymlinkedNeedsReviewIsNotRead(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
outside := t.TempDir()
|
||||
require.NoError(t, os.WriteFile(filepath.Join(outside, "secret.md"), []byte("outside"), 0o644))
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(dir, "raw"), 0o755))
|
||||
require.NoError(t, os.Symlink(outside, filepath.Join(dir, "raw", "needs-review")))
|
||||
|
||||
_, err := PromoteNote(dir, PromoteOptions{Filename: "secret.md", Wing: "homelab", Hall: "facts"})
|
||||
assert.Error(t, err)
|
||||
_, statErr := os.Stat(filepath.Join(dir, "wiki/homelab/facts/secret.md"))
|
||||
assert.ErrorIs(t, statErr, os.ErrNotExist)
|
||||
}
|
||||
@@ -38,6 +38,10 @@ func OpenRawRoot(brainDir string) (*os.Root, string, error) {
|
||||
return r, p, nil
|
||||
}
|
||||
|
||||
// maxRawNoteBytes bounds what promote, reject and the pending listing read from
|
||||
// one raw/ entry. A variable so a test can lower it.
|
||||
var maxRawNoteBytes int64 = 64 << 20
|
||||
|
||||
// readRawNote reads one raw/ entry by basename through the root, refusing
|
||||
// anything that is not a regular file.
|
||||
//
|
||||
@@ -59,9 +63,22 @@ func readRawNote(r *os.Root, base string) ([]byte, os.FileInfo, error) {
|
||||
return nil, nil, err
|
||||
}
|
||||
defer func() { _ = f.Close() }()
|
||||
b, err := io.ReadAll(f)
|
||||
b, err := readCapped(f, maxRawNoteBytes)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
return nil, nil, fmt.Errorf("%s: %w", base, err)
|
||||
}
|
||||
return b, fi, nil
|
||||
}
|
||||
|
||||
// readCapped reads r but never more than max+1 bytes, so an oversize or growing
|
||||
// file costs at most that much memory.
|
||||
func readCapped(r io.Reader, max int64) ([]byte, error) {
|
||||
b, err := io.ReadAll(io.LimitReader(r, max+1))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if int64(len(b)) > max {
|
||||
return nil, fmt.Errorf("note exceeds %d bytes", max)
|
||||
}
|
||||
return b, nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
package api_test
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
|
||||
)
|
||||
|
||||
// hyperguild#178: an unreadable directory under wiki/ made the link check
|
||||
// fail and brain_write refuse the note, losing the learning. The note is now
|
||||
// written with its links as given.
|
||||
func TestWriteNoteKeepsTheLearningWhenTheLinkCheckCannotIndex(t *testing.T) {
|
||||
if os.Geteuid() == 0 {
|
||||
t.Skip("root ignores directory permissions")
|
||||
}
|
||||
brain := t.TempDir()
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki", "w", "facts"), 0o755))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(brain, "wiki", "w", "facts", "real.md"), []byte("x"), 0o644))
|
||||
locked := filepath.Join(brain, "wiki", "other", "locked")
|
||||
require.NoError(t, os.MkdirAll(locked, 0o755))
|
||||
require.NoError(t, os.Chmod(locked, 0o000))
|
||||
t.Cleanup(func() { _ = os.Chmod(locked, 0o755) })
|
||||
|
||||
rel, err := api.WriteNote(brain, api.WriteNoteOptions{Wing: "w", Hall: "facts", Filename: "n", Content: "learning [[real]] and [[gone]]"})
|
||||
require.NoError(t, err)
|
||||
got, err := os.ReadFile(filepath.Join(brain, filepath.FromSlash(rel)))
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, string(got), "[[gone]]", "links kept as given when the index is partial")
|
||||
}
|
||||
@@ -13,9 +13,14 @@ import (
|
||||
type NtfyNotifier struct {
|
||||
topicURL string
|
||||
token string
|
||||
title string
|
||||
http *http.Client
|
||||
}
|
||||
|
||||
// defaultNtfyTitle is the title of a notifier built without WithTitle: the
|
||||
// capture audit's degraded-state alert, its first user.
|
||||
const defaultNtfyTitle = "brain-capture audit degraded"
|
||||
|
||||
// NewNtfyNotifier constructs a notifier for the given ntfy topic URL
|
||||
// (e.g. https://ntfy.sh/my-topic). token is an optional bearer for
|
||||
// protected ntfy instances; it is held here and only sent in the
|
||||
@@ -27,17 +32,31 @@ func NewNtfyNotifier(topicURL, token string) *NtfyNotifier {
|
||||
return &NtfyNotifier{
|
||||
topicURL: strings.TrimRight(topicURL, "/"),
|
||||
token: token,
|
||||
title: defaultNtfyTitle,
|
||||
http: &http.Client{Timeout: 10 * time.Second},
|
||||
}
|
||||
}
|
||||
|
||||
// WithTitle returns a copy that sends title as the ntfy Title header, so each
|
||||
// alert source (the capture audit, the symlink tripwire, governance edits)
|
||||
// is recognisable on the phone (hyperguild#163). A nil notifier (ntfy not
|
||||
// configured) stays nil.
|
||||
func (n *NtfyNotifier) WithTitle(title string) *NtfyNotifier {
|
||||
if n == nil {
|
||||
return nil
|
||||
}
|
||||
c := *n
|
||||
c.title = title
|
||||
return &c
|
||||
}
|
||||
|
||||
// Notify posts a message to the ntfy topic.
|
||||
func (n *NtfyNotifier) Notify(ctx context.Context, msg string) error {
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, n.topicURL, strings.NewReader(msg))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req.Header.Set("Title", "brain-capture audit degraded")
|
||||
req.Header.Set("Title", n.title)
|
||||
req.Header.Set("Priority", "high")
|
||||
req.Header.Set("Tags", "warning,brain")
|
||||
if n.token != "" {
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
package audit
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// hyperguild#163: every alert went out titled "brain-capture audit
|
||||
// degraded", including the symlink tripwire's and governance alerts.
|
||||
func TestNtfyNotifierSendsItsOwnTitle(t *testing.T) {
|
||||
var got []string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
got = append(got, r.Header.Get("Title"))
|
||||
}))
|
||||
defer srv.Close()
|
||||
base := NewNtfyNotifier(srv.URL, "")
|
||||
require.NoError(t, base.Notify(context.Background(), "m"))
|
||||
require.NoError(t, base.WithTitle("brain symlink tripwire").Notify(context.Background(), "m"))
|
||||
require.NoError(t, base.Notify(context.Background(), "m"))
|
||||
assert.Equal(t, []string{"brain-capture audit degraded", "brain symlink tripwire", "brain-capture audit degraded"}, got,
|
||||
"WithTitle returns a copy; the base keeps its default title")
|
||||
var nilN *NtfyNotifier
|
||||
assert.Nil(t, nilN.WithTitle("x"), "nil (ntfy not configured) stays nil")
|
||||
}
|
||||
@@ -47,7 +47,12 @@ var (
|
||||
func StripUnresolvedLinks(brainDir, content string) (string, []string, error) {
|
||||
index, err := entryIndex(brainDir)
|
||||
if err != nil {
|
||||
return "", nil, err
|
||||
// A partial index cannot tell an invented link from one into the
|
||||
// part it could not read, so unwrap nothing, and do not refuse the
|
||||
// write either: this runs before every brain_write, and a refused
|
||||
// write is a lost learning (hyperguild#178).
|
||||
slog.Warn("link check skipped: cannot index the brain", "err", err)
|
||||
return content, nil, nil
|
||||
}
|
||||
if len(index) == 0 {
|
||||
return content, nil, nil
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
package brain_test
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
|
||||
)
|
||||
|
||||
// hyperguild#178: StripUnresolvedLinks runs before every brain_write and
|
||||
// brain_update. An unreadable directory under knowledge/ or wiki/ made it
|
||||
// error, the write was refused and the learning lost. When the link index
|
||||
// cannot be built, the content is now returned unchanged (no link is
|
||||
// unwrapped, none is invented) and no error stops the write.
|
||||
func TestStripUnresolvedLinksKeepsContentWhenTheIndexCannotBeBuilt(t *testing.T) {
|
||||
if os.Geteuid() == 0 {
|
||||
t.Skip("root ignores directory permissions")
|
||||
}
|
||||
b := t.TempDir()
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(b, "wiki", "w", "facts"), 0o755))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(b, "wiki", "w", "facts", "real.md"), []byte("x"), 0o644))
|
||||
locked := filepath.Join(b, "wiki", "other", "locked")
|
||||
require.NoError(t, os.MkdirAll(locked, 0o755))
|
||||
require.NoError(t, os.Chmod(locked, 0o000))
|
||||
t.Cleanup(func() { _ = os.Chmod(locked, 0o755) })
|
||||
|
||||
in := "learning [[real]] and [[gone]]"
|
||||
out, unlinked, err := brain.StripUnresolvedLinks(b, in)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, in, out, "content unchanged: no link unwrapped on a partial index")
|
||||
assert.Empty(t, unlinked)
|
||||
|
||||
require.NoError(t, os.Chmod(locked, 0o755))
|
||||
out, _, err = brain.StripUnresolvedLinks(b, in)
|
||||
require.NoError(t, err)
|
||||
assert.NotContains(t, out, "[[gone]]", "positive control: with a readable tree the strip runs")
|
||||
}
|
||||
@@ -109,7 +109,7 @@ func (b *Brain) ReadDirRegular(relDir string) ([]string, error) {
|
||||
// after following a symlink, and a ".." after a symlinked directory would
|
||||
// otherwise pass the check and redirect the read or write.
|
||||
func cleanRel(rel string) error {
|
||||
if rel == "" || path.Clean(rel) != rel || !filepath.IsLocal(rel) || strings.Contains(rel, "\\") {
|
||||
if rel == "" || path.Clean(rel) != rel || !filepath.IsLocal(rel) {
|
||||
return fmt.Errorf("%q: not a clean brain-relative path: %w", rel, ErrNotRegular)
|
||||
}
|
||||
return nil
|
||||
@@ -243,6 +243,27 @@ func (b *Brain) RealParent(rel string) error {
|
||||
return b.realDir(path.Dir(rel))
|
||||
}
|
||||
|
||||
// PutFile opens the brain dir as a root and calls PutRegular, for one-off
|
||||
// writers that hold no Brain.
|
||||
func PutFile(brainDir, rel string, data []byte) error {
|
||||
b, err := Open(brainDir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() { _ = b.Close() }()
|
||||
return b.PutRegular(rel, data)
|
||||
}
|
||||
|
||||
// AppendFile opens the brain dir as a root and calls AppendRegular.
|
||||
func AppendFile(brainDir, rel string, data []byte) error {
|
||||
b, err := Open(brainDir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() { _ = b.Close() }()
|
||||
return b.AppendRegular(rel, data)
|
||||
}
|
||||
|
||||
// mkdirReal makes sure every component of rel exists as a real directory,
|
||||
// creating missing ones through the root; an existing component that is
|
||||
// not a real directory is ErrNotRegular.
|
||||
|
||||
@@ -529,3 +529,20 @@ func TestCreateRegularCreatesOnlyNewFilesThroughRealDirectories(t *testing.T) {
|
||||
got, _ = os.ReadFile(filepath.Join(dir, "wiki", "new", "facts", "n.md"))
|
||||
assert.Equal(t, "one", string(got))
|
||||
}
|
||||
|
||||
// On Linux a backslash is an ordinary filename byte, not a separator, so a
|
||||
// name containing one stays a single in-root component (hyperguild#158: the
|
||||
// watcher retried such a note forever while cleanRel refused it).
|
||||
func TestReadAndPutRegularTreatABackslashAsAFilenameByte(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
b, err := Open(dir)
|
||||
require.NoError(t, err)
|
||||
defer func() { _ = b.Close() }()
|
||||
require.NoError(t, b.PutRegular(`a\..\b.md`, []byte("x")))
|
||||
got, err := os.ReadFile(filepath.Join(dir, `a\..\b.md`))
|
||||
require.NoError(t, err, "the name was not kept as one component")
|
||||
assert.Equal(t, "x", string(got))
|
||||
got, err = b.ReadRegular(`a\..\b.md`)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "x", string(got))
|
||||
}
|
||||
|
||||
@@ -88,7 +88,6 @@ func TestNoPackageReferencesExtractTextOutsideExtract(t *testing.T) {
|
||||
// osOpenAllowed names the functions in guarded files that may call os.Open*
|
||||
// on a path that is not a raw/ read.
|
||||
var osOpenAllowed = map[string]bool{
|
||||
"internal/watcher/watcher.go:appendWatcherLog": true, // ingest log append, not a raw/ read
|
||||
// Text reads by path; outside this package only TextIn's temp copy reaches
|
||||
// it (TestNoPackageReferencesExtractTextOutsideExtract enforces that).
|
||||
"internal/extract/extract.go:Text": true,
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
package extract
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
@@ -50,6 +51,18 @@ func fileExt(path string) string {
|
||||
// external tool (pdf, docmark) are copied to a temp file from the handle
|
||||
// opened through the root, then extracted from the copy.
|
||||
func TextIn(r *os.Root, rel string) (string, error) {
|
||||
return TextInMax(r, rel, DefaultMaxBytes)
|
||||
}
|
||||
|
||||
// DefaultMaxBytes bounds what TextIn reads or copies from one file.
|
||||
const DefaultMaxBytes = 64 << 20
|
||||
|
||||
// ErrTooLarge reports a file that exceeded the read cap while being read: the
|
||||
// size seen at an earlier Lstat is not a bound, the file may have grown.
|
||||
var ErrTooLarge = errors.New("file exceeds the read cap")
|
||||
|
||||
// TextInMax is TextIn with an explicit cap on the bytes read (hyperguild#198).
|
||||
func TextInMax(r *os.Root, rel string, max int64) (string, error) {
|
||||
fi, err := r.Lstat(rel)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("stat %s: %w", rel, err)
|
||||
@@ -65,10 +78,13 @@ func TextIn(r *os.Root, rel string) (string, error) {
|
||||
ext := strings.ToLower(fileExt(rel))
|
||||
switch ext {
|
||||
case ".md", ".txt":
|
||||
b, err := io.ReadAll(f)
|
||||
b, err := io.ReadAll(io.LimitReader(f, max+1))
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("read %s: %w", rel, err)
|
||||
}
|
||||
if int64(len(b)) > max {
|
||||
return "", fmt.Errorf("%s: %w (%d bytes)", rel, ErrTooLarge, max)
|
||||
}
|
||||
return string(b), nil
|
||||
case ".pdf", ".docx", ".xlsx", ".pptx", ".png", ".jpg", ".jpeg":
|
||||
tmp, err := os.CreateTemp("", "ingest-*"+ext)
|
||||
@@ -76,10 +92,15 @@ func TextIn(r *os.Root, rel string) (string, error) {
|
||||
return "", fmt.Errorf("temp copy: %w", err)
|
||||
}
|
||||
defer func() { _ = os.Remove(tmp.Name()) }()
|
||||
if _, err := io.Copy(tmp, f); err != nil {
|
||||
n, err := io.Copy(tmp, io.LimitReader(f, max+1))
|
||||
if err != nil {
|
||||
_ = tmp.Close()
|
||||
return "", fmt.Errorf("copy %s: %w", rel, err)
|
||||
}
|
||||
if n > max {
|
||||
_ = tmp.Close()
|
||||
return "", fmt.Errorf("%s: %w (%d bytes)", rel, ErrTooLarge, max)
|
||||
}
|
||||
if err := tmp.Close(); err != nil {
|
||||
return "", fmt.Errorf("copy %s: %w", rel, err)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
package extract
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func rootWith(t *testing.T, name string, size int) *os.Root {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
require.NoError(t, os.WriteFile(filepath.Join(dir, name), []byte(strings.Repeat("x", size)), 0o644))
|
||||
r, err := os.OpenRoot(dir)
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = r.Close() })
|
||||
return r
|
||||
}
|
||||
|
||||
// hyperguild#198: the size check in the watcher is a pre-check on Lstat; a note
|
||||
// that grows before it is read must still be capped at the read.
|
||||
func TestTextInMax(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
size int
|
||||
max int64
|
||||
wantErr bool
|
||||
}{
|
||||
{"under the cap", 100, 1024, false},
|
||||
{"exactly at the cap", 1024, 1024, false},
|
||||
{"one byte over", 1025, 1024, true},
|
||||
{"far over", 1 << 20, 1024, true},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
r := rootWith(t, "n.md", tt.size)
|
||||
got, err := TextInMax(r, "n.md", tt.max)
|
||||
if tt.wantErr {
|
||||
require.Error(t, err)
|
||||
assert.True(t, errors.Is(err, ErrTooLarge))
|
||||
return
|
||||
}
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, got, tt.size)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestTextInMax_TheBinaryTempCopyIsCappedToo(t *testing.T) {
|
||||
r := rootWith(t, "n.pdf", 4096)
|
||||
_, err := TextInMax(r, "n.pdf", 1024)
|
||||
require.Error(t, err)
|
||||
assert.True(t, errors.Is(err, ErrTooLarge), "the temp copy of a pdf must not exceed the cap either: %v", err)
|
||||
}
|
||||
|
||||
func TestTextIn_HasADefaultCap(t *testing.T) {
|
||||
assert.Positive(t, DefaultMaxBytes)
|
||||
}
|
||||
@@ -94,6 +94,7 @@ func (s *Server) tools() []map[string]any {
|
||||
"wing": str("target wing, e.g. jepa-fx"),
|
||||
"hall": enum("target hall", halls...),
|
||||
"slug": str("optional target slug; defaults to filename minus date prefix"),
|
||||
"queue": enum("which brain_pending listing the note is in; required only when the name exists in both", "inbox", "needs-review"),
|
||||
}),
|
||||
},
|
||||
{
|
||||
@@ -102,6 +103,7 @@ func (s *Server) tools() []map[string]any {
|
||||
"inputSchema": schema([]string{"filename"}, map[string]any{
|
||||
"filename": str("basename in brain/raw/, e.g. 2026-04-23-fed-sentiment-scrape.md"),
|
||||
"reason": str("optional; why this isn't worth promoting"),
|
||||
"queue": enum("which brain_pending listing the note is in; required only when the name exists in both", "inbox", "needs-review"),
|
||||
}),
|
||||
},
|
||||
noteToolDescriptors()[0],
|
||||
@@ -383,6 +385,7 @@ type brainPromoteArgs struct {
|
||||
Wing string `json:"wing"`
|
||||
Hall string `json:"hall"`
|
||||
Slug string `json:"slug,omitempty"`
|
||||
Queue string `json:"queue,omitempty"`
|
||||
}
|
||||
|
||||
// brainPromote moves a raw/ note into the structured wiki (frontmatter
|
||||
@@ -395,7 +398,7 @@ func (s *Server) brainPromote(ctx context.Context, args json.RawMessage) (json.R
|
||||
}
|
||||
origin, writtenBy := provenanceFrom(ctx)
|
||||
relPath, err := api.PromoteNote(s.brainDir, api.PromoteOptions{
|
||||
Filename: a.Filename, Wing: a.Wing, Hall: a.Hall, Slug: a.Slug,
|
||||
Filename: a.Filename, Wing: a.Wing, Hall: a.Hall, Slug: a.Slug, Queue: a.Queue,
|
||||
Origin: origin, WrittenBy: writtenBy,
|
||||
})
|
||||
if err != nil {
|
||||
@@ -408,6 +411,7 @@ func (s *Server) brainPromote(ctx context.Context, args json.RawMessage) (json.R
|
||||
type brainRejectArgs struct {
|
||||
Filename string `json:"filename"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
Queue string `json:"queue,omitempty"`
|
||||
}
|
||||
|
||||
// brainReject records a decision not to promote a raw/ note (api.RejectNote:
|
||||
@@ -418,7 +422,7 @@ func (s *Server) brainReject(_ context.Context, args json.RawMessage) (json.RawM
|
||||
if err := json.Unmarshal(args, &a); err != nil {
|
||||
return nil, fmt.Errorf("parse args: %w", err)
|
||||
}
|
||||
relPath, err := api.RejectNote(s.brainDir, api.RejectOptions{Filename: a.Filename, Reason: a.Reason})
|
||||
relPath, err := api.RejectNote(s.brainDir, api.RejectOptions{Filename: a.Filename, Reason: a.Reason, Queue: a.Queue})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -431,3 +431,28 @@ func TestBrainRejectMissingSourceErrors(t *testing.T) {
|
||||
resp := toolCall(t, srv, "brain_reject", map[string]any{"filename": "ghost.md"})
|
||||
require.NotNil(t, resp["error"])
|
||||
}
|
||||
|
||||
// brain_pending lists two entries for one name when a top-level note and a
|
||||
// parked one share it; the tool must let the caller say which it means.
|
||||
func TestBrainPromoteAndRejectTakeTheQueueOfAnAmbiguousName(t *testing.T) {
|
||||
brainDir := t.TempDir()
|
||||
raw := filepath.Join(brainDir, "raw")
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(raw, "needs-review"), 0o755))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(raw, "a.md"), []byte("---\ncreated_at: 2026-10-20T00:00:00Z\n---\nTOP\n"), 0o644))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(raw, "needs-review", "a.md"), []byte("---\nproposed_wing: homelab\n---\nPARKED\n"), 0o644))
|
||||
srv := mcp.NewServer(brainDir, nil, nil, nil)
|
||||
|
||||
resp := toolCall(t, srv, "brain_promote", map[string]any{"filename": "a.md", "wing": "homelab", "hall": "facts"})
|
||||
require.NotNil(t, resp["error"], "an ambiguous name is refused, not guessed")
|
||||
|
||||
resp = toolCall(t, srv, "brain_promote", map[string]any{"filename": "a.md", "wing": "homelab", "hall": "facts", "queue": "needs-review"})
|
||||
require.Nil(t, resp["error"], "got: %v", resp["error"])
|
||||
got, err := os.ReadFile(filepath.Join(brainDir, "wiki/homelab/facts/a.md"))
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, string(got), "PARKED")
|
||||
|
||||
resp = toolCall(t, srv, "brain_reject", map[string]any{"filename": "a.md", "queue": "inbox", "reason": "no"})
|
||||
require.Nil(t, resp["error"], "got: %v", resp["error"])
|
||||
_, err = os.Stat(filepath.Join(brainDir, "raw/rejected/a.md"))
|
||||
assert.NoError(t, err)
|
||||
}
|
||||
|
||||
@@ -83,8 +83,9 @@ func (r *Registry) Observe(path, status string, d time.Duration) {
|
||||
|
||||
// Middleware wraps next, observing every request's duration + status.
|
||||
// The metric label `path` uses the request's Pattern (Go 1.22+ ServeMux),
|
||||
// falling back to the URL path if no Pattern is set. Pattern keeps
|
||||
// cardinality bounded (one series per route, not one per unique URL).
|
||||
// or the fixed label "unmatched" if no Pattern is set: falling back to the
|
||||
// raw URL path gave any caller one series per unique path (hyperguild#185).
|
||||
// Pattern keeps cardinality bounded (one series per route).
|
||||
func (r *Registry) Middleware(next http.Handler) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
|
||||
rec := &statusRecorder{ResponseWriter: w, code: http.StatusOK}
|
||||
@@ -92,7 +93,7 @@ func (r *Registry) Middleware(next http.Handler) http.Handler {
|
||||
next.ServeHTTP(rec, req)
|
||||
path := req.Pattern
|
||||
if path == "" {
|
||||
path = req.URL.Path
|
||||
path = "unmatched"
|
||||
}
|
||||
r.Observe(path, statusClass(rec.code), time.Since(start))
|
||||
})
|
||||
|
||||
@@ -71,11 +71,16 @@ func TestMiddleware_RecordsTiming(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
r := New()
|
||||
handler := r.Middleware(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
// Routed through a ServeMux, as in production: the mux sets the
|
||||
// request's Pattern, which is the label (an unrouted request is
|
||||
// "unmatched", hyperguild#185).
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("/query", func(w http.ResponseWriter, _ *http.Request) {
|
||||
time.Sleep(2 * time.Millisecond)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
_, _ = io.WriteString(w, "ok")
|
||||
}))
|
||||
})
|
||||
handler := r.Middleware(mux)
|
||||
|
||||
srv := httptest.NewServer(handler)
|
||||
defer srv.Close()
|
||||
@@ -117,3 +122,19 @@ func TestStatusRecorder_DefaultsTo200(t *testing.T) {
|
||||
t.Errorf("code %d, want 200", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
// A request with no matched Pattern is labelled "unmatched", never with its
|
||||
// raw URL path (unbounded cardinality, hyperguild#185).
|
||||
func TestMiddleware_LabelsAnUnmatchedRequestWithAFixedLabel(t *testing.T) {
|
||||
r := New()
|
||||
h := r.Middleware(http.NotFoundHandler())
|
||||
h.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("GET", "/zqrandom-185-unit", nil))
|
||||
rec := httptest.NewRecorder()
|
||||
r.Handler().ServeHTTP(rec, httptest.NewRequest("GET", "/metrics", nil))
|
||||
if strings.Contains(rec.Body.String(), "zqrandom-185-unit") {
|
||||
t.Errorf("the raw path became a metrics label:\n%s", rec.Body.String())
|
||||
}
|
||||
if !strings.Contains(rec.Body.String(), `path="unmatched"`) {
|
||||
t.Errorf("no unmatched label:\n%s", rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,8 +7,6 @@ import (
|
||||
"fmt"
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/wiki"
|
||||
@@ -90,8 +88,7 @@ func BackfillRefs(ctx context.Context, brainDir string) (int, error) {
|
||||
}
|
||||
|
||||
for relPath, page := range pending {
|
||||
dest := filepath.Join(brainDir, filepath.FromSlash(relPath))
|
||||
if err := os.WriteFile(dest, []byte(page.Content), 0o644); err != nil {
|
||||
if err := bfs.PutRegular(relPath, []byte(page.Content)); err != nil {
|
||||
return 0, fmt.Errorf("write %s: %w", relPath, err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -14,12 +14,21 @@ import (
|
||||
// (derived from the filename, not the LLM title). Pages whose title resolves to an
|
||||
// empty slug are skipped and returned as warnings instead.
|
||||
func BuildPages(rawPages []RawPage, sourceSlug, date string) ([]wiki.Page, []string) {
|
||||
return BuildPagesIn("", rawPages, sourceSlug, date)
|
||||
}
|
||||
|
||||
// BuildPagesIn is BuildPages for a wing: a source goes to wiki/<wing>/sources/
|
||||
// with wing/hall frontmatter. Concepts and entities stay in the shared
|
||||
// wiki/concepts/ and wiki/entities/, since their slug is the graph's global
|
||||
// identity and they are cited from many wings. An empty wing is the retired
|
||||
// layout for sources too.
|
||||
func BuildPagesIn(wing string, rawPages []RawPage, sourceSlug, date string) ([]wiki.Page, []string) {
|
||||
out := make([]wiki.Page, 0, len(rawPages))
|
||||
var warnings []string
|
||||
for _, rp := range rawPages {
|
||||
// The type becomes a directory under wiki/; "../raw" wrote outside it
|
||||
// (PR #147 review). Only a plain lower-case name is a page type.
|
||||
if !validPageType.MatchString(rp.Type) {
|
||||
if !validPageType.MatchString(rp.Type) || (wing != "" && !layoutType(rp.Type)) {
|
||||
warnings = append(warnings, fmt.Sprintf("skipped page with invalid type %q", rp.Type))
|
||||
continue
|
||||
}
|
||||
@@ -28,11 +37,16 @@ func BuildPages(rawPages []RawPage, sourceSlug, date string) ([]wiki.Page, []str
|
||||
warnings = append(warnings, fmt.Sprintf("skipped page with empty title (type: %s)", rp.Type))
|
||||
continue
|
||||
}
|
||||
out = append(out, buildPage(rp, sourceSlug, date))
|
||||
out = append(out, buildPage(rp, sourceSlug, date, wing))
|
||||
}
|
||||
return out, warnings
|
||||
}
|
||||
|
||||
// layoutType reports whether a wing-mode page type has a place in the layout.
|
||||
// The type comes from the model, so from the clip being ingested: anything else
|
||||
// would name a directory (wiki/<type>/) of the model's choosing.
|
||||
func layoutType(t string) bool { return t == "source" || t == "concept" || t == "entity" }
|
||||
|
||||
var validPageType = regexp.MustCompile(`^[a-z][a-z0-9-]*$`)
|
||||
|
||||
func computeSlug(rp RawPage, sourceSlug string) string {
|
||||
@@ -42,7 +56,7 @@ func computeSlug(rp RawPage, sourceSlug string) string {
|
||||
return wiki.Slug(rp.Title)
|
||||
}
|
||||
|
||||
func buildPage(rp RawPage, sourceSlug, date string) wiki.Page {
|
||||
func buildPage(rp RawPage, sourceSlug, date, wing string) wiki.Page {
|
||||
var slug, dir string
|
||||
switch rp.Type {
|
||||
case "source":
|
||||
@@ -59,8 +73,14 @@ func buildPage(rp RawPage, sourceSlug, date string) wiki.Page {
|
||||
dir = "wiki/" + rp.Type
|
||||
}
|
||||
|
||||
if wing != "" && rp.Type == "source" {
|
||||
dir = "wiki/" + wing + "/sources"
|
||||
}
|
||||
path := dir + "/" + slug + ".md"
|
||||
fm := buildFrontmatter(rp, date)
|
||||
if wing != "" && rp.Type == "source" {
|
||||
fm = addWingFrontmatter(fm, wing, "sources")
|
||||
}
|
||||
|
||||
return wiki.Page{
|
||||
Path: path,
|
||||
@@ -122,6 +142,12 @@ func buildFrontmatter(rp RawPage, date string) string {
|
||||
return sb.String()
|
||||
}
|
||||
|
||||
// addWingFrontmatter inserts wing/hall right after the opening fence of an
|
||||
// already-built frontmatter block.
|
||||
func addWingFrontmatter(fm, wing, hall string) string {
|
||||
return strings.Replace(fm, "---\n", "---\nwing: "+wing+"\nhall: "+hall+"\n", 1)
|
||||
}
|
||||
|
||||
func yamlScalar(s string) string {
|
||||
// A raw newline or CR would end the scalar for every line-oriented
|
||||
// frontmatter reader and start a new key (PR #147 review: a title of
|
||||
|
||||
@@ -7,10 +7,11 @@ import (
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -26,6 +27,11 @@ type Config struct {
|
||||
Complete CompleteFunc
|
||||
ChunkSize int // 0 = no chunking
|
||||
Schema string // overrides brain/schema.md when set (useful in tests)
|
||||
|
||||
// Wing, when set, writes into wiki/<wing>/{sources,facts}/ and scopes the
|
||||
// inventory (title resolution, back-references) to that wing alone. Empty
|
||||
// keeps the retired wiki/{sources,concepts,entities}/ layout.
|
||||
Wing string
|
||||
}
|
||||
|
||||
// Result is the outcome of a pipeline run.
|
||||
@@ -42,7 +48,16 @@ type Result struct {
|
||||
// Run ingests content and writes structured wiki pages to brainDir/wiki/.
|
||||
// In dry-run mode, pages are returned but not written to disk.
|
||||
func Run(ctx context.Context, cfg Config, brainDir, content, source string, dryRun bool) (Result, error) {
|
||||
inventory, err := wiki.LoadInventory(brainDir)
|
||||
if cfg.Wing != "" && brain.Sanitise(cfg.Wing) != cfg.Wing {
|
||||
return Result{}, fmt.Errorf("invalid wing %q: must be a plain lower-case wing name", cfg.Wing)
|
||||
}
|
||||
var inventory map[wiki.PageType][]wiki.Entry
|
||||
var err error
|
||||
if cfg.Wing != "" {
|
||||
inventory, err = wiki.LoadWingInventory(brainDir, cfg.Wing)
|
||||
} else {
|
||||
inventory, err = wiki.LoadInventory(brainDir)
|
||||
}
|
||||
if err != nil {
|
||||
return Result{}, fmt.Errorf("load inventory: %w", err)
|
||||
}
|
||||
@@ -53,6 +68,12 @@ func Run(ctx context.Context, cfg Config, brainDir, content, source string, dryR
|
||||
}
|
||||
|
||||
sourceSlug := wiki.Slug(source)
|
||||
if cfg.Wing != "" {
|
||||
var rerr error
|
||||
if sourceSlug, rerr = resolveSourceSlug(brainDir, cfg.Wing, sourceSlug); rerr != nil {
|
||||
return Result{}, rerr
|
||||
}
|
||||
}
|
||||
date := time.Now().UTC().Format("2006-01-02")
|
||||
|
||||
// Drift check BEFORE any extraction: an unchanged source costs a full LLM
|
||||
@@ -60,7 +81,7 @@ func Run(ctx context.Context, cfg Config, brainDir, content, source string, dryR
|
||||
// The merge is already idempotent (#103), so this is cost and signal, not
|
||||
// correctness — which is why it is safe for it to be an early return.
|
||||
bodyHash := hashBody(content)
|
||||
if !dryRun && bodyHash != "" && bodyHash == storedSourceHash(brainDir, sourceSlug) {
|
||||
if !dryRun && bodyHash != "" && bodyHash == storedSourceHash(brainDir, cfg.Wing, sourceSlug) {
|
||||
return Result{Skipped: true, Warnings: []string{
|
||||
fmt.Sprintf("source %q unchanged since last ingest (hash %s) — extraction skipped", source, bodyHash[:12]),
|
||||
}}, nil
|
||||
@@ -71,7 +92,7 @@ func Run(ctx context.Context, cfg Config, brainDir, content, source string, dryR
|
||||
// only exists once the model has produced a title. So page-aware
|
||||
// extraction (#106) is scoped to the source page: it's the one page every
|
||||
// chunk's prompt can be given in advance.
|
||||
existingPage := readSourcePage(brainDir, sourceSlug)
|
||||
existingPage := readSourcePage(brainDir, cfg.Wing, sourceSlug)
|
||||
|
||||
chunks := Chunk(content, cfg.ChunkSize)
|
||||
|
||||
@@ -93,7 +114,19 @@ func Run(ctx context.Context, cfg Config, brainDir, content, source string, dryR
|
||||
meta.BodyHash = bodyHash
|
||||
applySourceMeta(allRaw, meta)
|
||||
|
||||
return buildAndWrite(ctx, allRaw, sourceSlug, date, brainDir, source, inventory, allWarnings, dryRun)
|
||||
res, err := buildAndWrite(ctx, cfg.Wing, allRaw, sourceSlug, date, brainDir, source, inventory, allWarnings, dryRun)
|
||||
if err != nil || cfg.Wing == "" {
|
||||
return res, err
|
||||
}
|
||||
// In a wing every ingest must leave its source page: it is what ties the
|
||||
// note to the wiki (and carries the hash the drift check reads). Without one
|
||||
// nothing of the note exists, and the caller would archive it as processed.
|
||||
for _, p := range res.Pages {
|
||||
if isSourcePath(p) {
|
||||
return res, nil
|
||||
}
|
||||
}
|
||||
return res, fmt.Errorf("no source page was written for %q; nothing of the note was ingested (%s)", source, strings.Join(res.Warnings, "; "))
|
||||
}
|
||||
|
||||
// RunRaw runs the pipeline on pre-parsed RawPages, skipping the LLM extraction
|
||||
@@ -112,26 +145,37 @@ func RunRaw(ctx context.Context, brainDir, source string, rawPages []RawPage, dr
|
||||
sourceSlug := wiki.Slug(source)
|
||||
date := time.Now().UTC().Format("2006-01-02")
|
||||
|
||||
return buildAndWrite(ctx, rawPages, sourceSlug, date, brainDir, source, inventory, nil, dryRun)
|
||||
return buildAndWrite(ctx, "", rawPages, sourceSlug, date, brainDir, source, inventory, nil, dryRun)
|
||||
}
|
||||
|
||||
// buildAndWrite runs BuildPages through write for both Run and RunRaw.
|
||||
func buildAndWrite(ctx context.Context, rawPages []RawPage, sourceSlug, date, brainDir, source string, inventory map[wiki.PageType][]wiki.Entry, warnings []string, dryRun bool) (Result, error) {
|
||||
pages, buildWarnings := BuildPages(rawPages, sourceSlug, date)
|
||||
func buildAndWrite(ctx context.Context, wing string, rawPages []RawPage, sourceSlug, date, brainDir, source string, inventory map[wiki.PageType][]wiki.Entry, warnings []string, dryRun bool) (Result, error) {
|
||||
pages, buildWarnings := BuildPagesIn(wing, rawPages, sourceSlug, date)
|
||||
warnings = append(warnings, buildWarnings...)
|
||||
resolved := Resolve(pages, inventory)
|
||||
canonicalized, linkWarnings := CanonicalizeLinks(resolved, inventory)
|
||||
warnings = append(warnings, linkWarnings...)
|
||||
withRefs := injectSourceRefs(canonicalized, inventory, brainDir)
|
||||
merged := mergeAll(withRefs)
|
||||
if wing != "" {
|
||||
var clashWarnings []string
|
||||
var cerr error
|
||||
merged, clashWarnings, cerr = dropSlugClashes(brainDir, merged)
|
||||
if cerr != nil {
|
||||
return Result{}, cerr
|
||||
}
|
||||
warnings = append(warnings, clashWarnings...)
|
||||
}
|
||||
|
||||
if wing != "" {
|
||||
// The source page carries source_hash, which the drift check trusts: it
|
||||
// goes last, so a batch that fails midway leaves nothing to hide behind.
|
||||
sort.SliceStable(merged, func(i, j int) bool { return !isSourcePath(merged[i].Path) && isSourcePath(merged[j].Path) })
|
||||
}
|
||||
|
||||
var written []string
|
||||
for _, page := range merged {
|
||||
if !dryRun {
|
||||
dest := filepath.Join(brainDir, filepath.FromSlash(page.Path))
|
||||
if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
|
||||
return Result{}, fmt.Errorf("mkdir for %s: %w", page.Path, err)
|
||||
}
|
||||
content, conflicts, err := reviseOrCreate(brainDir, page)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
@@ -150,7 +194,9 @@ func buildAndWrite(ctx context.Context, rawPages []RawPage, sourceSlug, date, br
|
||||
if err != nil {
|
||||
return Result{}, fmt.Errorf("stamp %s: %w", page.Path, err)
|
||||
}
|
||||
if err := os.WriteFile(dest, []byte(stamped), 0o644); err != nil {
|
||||
// Written through the brain root, never through a symlinked
|
||||
// directory or page (hyperguild#158).
|
||||
if err := writeBrainFile(brainDir, page.Path, []byte(stamped)); err != nil {
|
||||
return Result{}, fmt.Errorf("write %s: %w", page.Path, err)
|
||||
}
|
||||
}
|
||||
@@ -158,7 +204,11 @@ func buildAndWrite(ctx context.Context, rawPages []RawPage, sourceSlug, date, br
|
||||
}
|
||||
|
||||
if !dryRun {
|
||||
if err := wiki.RebuildIndex(brainDir, date); err != nil {
|
||||
if wing != "" {
|
||||
if err := brain.BuildWingIndex(brainDir, wing); err != nil {
|
||||
warnings = append(warnings, fmt.Sprintf("rebuild wing index: %v", err))
|
||||
}
|
||||
} else if err := wiki.RebuildIndex(brainDir, date); err != nil {
|
||||
warnings = append(warnings, fmt.Sprintf("rebuild index: %v", err))
|
||||
}
|
||||
if err := wiki.AppendLog(brainDir, source, written, warnings, date); err != nil {
|
||||
@@ -193,14 +243,29 @@ func hashBody(content string) string {
|
||||
// wiki/sources/<sourceSlug>.md, or "" when it doesn't exist yet.
|
||||
// It reads through the brain root, regular files only (hyperguild#157): the
|
||||
// content goes into the LLM prompt.
|
||||
func readSourcePage(brainDir, sourceSlug string) string {
|
||||
raw, err := readBrainFile(brainDir, "wiki/sources/"+sourceSlug+".md")
|
||||
func readSourcePage(brainDir, wing, sourceSlug string) string {
|
||||
rel := "wiki/sources/" + sourceSlug + ".md"
|
||||
if wing != "" {
|
||||
rel = "wiki/" + wing + "/sources/" + sourceSlug + ".md"
|
||||
}
|
||||
raw, err := readBrainFile(brainDir, rel)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return string(raw)
|
||||
}
|
||||
|
||||
// writeBrainFile creates or replaces one brain-relative file through the
|
||||
// brain root (hyperguild#158).
|
||||
func writeBrainFile(brainDir, rel string, data []byte) error {
|
||||
bfs, err := brainfs.Open(brainDir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() { _ = bfs.Close() }()
|
||||
return bfs.PutRegular(rel, data)
|
||||
}
|
||||
|
||||
// readBrainFile reads one brain-relative file through the brain root,
|
||||
// regular files only (hyperguild#157).
|
||||
func readBrainFile(brainDir, rel string) ([]byte, error) {
|
||||
@@ -218,8 +283,8 @@ func readBrainFile(brainDir, rel string) ([]byte, error) {
|
||||
// Absence always means "extract". That is the fail-safe direction: a missed
|
||||
// skip costs one extraction, while a wrong skip loses an update silently — and
|
||||
// silence is the failure mode this repo keeps paying for.
|
||||
func storedSourceHash(brainDir, sourceSlug string) string {
|
||||
for _, line := range strings.Split(readSourcePage(brainDir, sourceSlug), "\n") {
|
||||
func storedSourceHash(brainDir, wing, sourceSlug string) string {
|
||||
for _, line := range strings.Split(readSourcePage(brainDir, wing, sourceSlug), "\n") {
|
||||
if key, value, found := strings.Cut(line, ":"); found && strings.TrimSpace(key) == "source_hash" {
|
||||
return strings.TrimSpace(value)
|
||||
}
|
||||
@@ -289,3 +354,150 @@ func loadSchema(brainDir string) string {
|
||||
}
|
||||
return strings.TrimSpace(string(b))
|
||||
}
|
||||
|
||||
// resolveSourceSlug returns the slug this wing's source page uses. A slug is a
|
||||
// page's graph identity and must be unique brain-wide, but two wings dropping a
|
||||
// file of the same name (notes.md, readme.md) is ordinary, so when the plain
|
||||
// slug is taken by anything but this wing's own source page the wing name is
|
||||
// appended (talk-homelab, then talk-homelab-2, ...). The choice is stable: a
|
||||
// re-ingest finds the page it made, because that page is "its own".
|
||||
func resolveSourceSlug(brainDir, wing, base string) (string, error) {
|
||||
for n := 1; n <= 100; n++ {
|
||||
cand := base
|
||||
switch n {
|
||||
case 1:
|
||||
case 2:
|
||||
cand = base + "-" + wing
|
||||
default:
|
||||
cand = fmt.Sprintf("%s-%s-%d", base, wing, n-1)
|
||||
}
|
||||
existing, err := wiki.FindSlugs(brainDir, cand)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("check source slug: %w", err)
|
||||
}
|
||||
own := "wiki/" + wing + "/sources/" + cand + ".md"
|
||||
free, hasOwn := true, false
|
||||
for _, p := range existing {
|
||||
if p == own {
|
||||
hasOwn = true
|
||||
} else {
|
||||
free = false
|
||||
}
|
||||
}
|
||||
// The wing's own page is its identity: keep using it even if some other
|
||||
// page shares the slug (that other page is the one dropSlugClashes skips).
|
||||
if hasOwn {
|
||||
return cand, nil
|
||||
}
|
||||
if free {
|
||||
return cand, nil
|
||||
}
|
||||
}
|
||||
return "", fmt.Errorf("no free source slug for %q in wing %q", base, wing)
|
||||
}
|
||||
|
||||
// dropSlugClashes removes, from the pages about to be written in a wing, any
|
||||
// page whose slug belongs to a different page. A slug is the graph identity, so
|
||||
// two pages must not share one; and a page that shares one cannot be updated
|
||||
// again, because every later ingest sees the other as its clash.
|
||||
//
|
||||
// Two kinds of clash, both by slug, neither failing the ingest (one extracted
|
||||
// page must not be able to lock a source out of re-ingest):
|
||||
// - inside the batch (a clip named notes.md whose model output also has an
|
||||
// entity "Notes"): the source wins over an entity, an entity over a concept;
|
||||
// - against the brain: an entity or concept whose slug is another page's. The
|
||||
// SOURCE page is exempt: it is the note's own output, and its slug was
|
||||
// already chosen free (or its own) by resolveSourceSlug.
|
||||
//
|
||||
// A page's own path is never a clash (a shared entity merging into itself).
|
||||
func dropSlugClashes(brainDir string, pages []wiki.Page) ([]wiki.Page, []string, error) {
|
||||
rank := func(p wiki.Page) int {
|
||||
switch {
|
||||
case isSourcePath(p.Path):
|
||||
return 0
|
||||
case strings.HasPrefix(p.Path, "wiki/entities/"):
|
||||
return 1
|
||||
case strings.HasPrefix(p.Path, "wiki/concepts/"):
|
||||
return 2
|
||||
}
|
||||
return 3
|
||||
}
|
||||
slugOf := func(p wiki.Page) string { return strings.TrimSuffix(filepath.Base(p.Path), ".md") }
|
||||
|
||||
// Group by slug; where several pages with different paths want one slug,
|
||||
// the source wins, then a page that already exists on disk (so its update
|
||||
// is not lost), then entity over concept.
|
||||
bySlug := map[string][]wiki.Page{}
|
||||
for _, p := range pages {
|
||||
bySlug[slugOf(p)] = append(bySlug[slugOf(p)], p)
|
||||
}
|
||||
winner := map[string]wiki.Page{}
|
||||
for slug, group := range bySlug {
|
||||
best := group[0]
|
||||
if distinctPaths(group) > 1 {
|
||||
existing, err := wiki.FindSlugs(brainDir, slug)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("check slug %q: %w", slug, err)
|
||||
}
|
||||
onDisk := func(p wiki.Page) int {
|
||||
for _, e := range existing {
|
||||
if e == p.Path {
|
||||
return 0
|
||||
}
|
||||
}
|
||||
return 1
|
||||
}
|
||||
for _, p := range group[1:] {
|
||||
if rank(p) != rank(best) && (rank(p) == 0 || rank(best) == 0) {
|
||||
if rank(p) < rank(best) {
|
||||
best = p
|
||||
}
|
||||
continue
|
||||
}
|
||||
if onDisk(p) < onDisk(best) || (onDisk(p) == onDisk(best) && rank(p) < rank(best)) {
|
||||
best = p
|
||||
}
|
||||
}
|
||||
}
|
||||
winner[slug] = best
|
||||
}
|
||||
|
||||
var kept []wiki.Page
|
||||
var warnings []string
|
||||
for _, p := range pages {
|
||||
slug := slugOf(p)
|
||||
if w := winner[slug]; w.Path != p.Path {
|
||||
warnings = append(warnings, fmt.Sprintf("skipped %s: slug %q is also used by %s in this batch", p.Path, slug, w.Path))
|
||||
continue
|
||||
}
|
||||
if rank(p) == 0 {
|
||||
kept = append(kept, p)
|
||||
continue
|
||||
}
|
||||
existing, err := wiki.FindSlugs(brainDir, slug)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("check slug %q: %w", slug, err)
|
||||
}
|
||||
clash := ""
|
||||
for _, e := range existing {
|
||||
if e != p.Path {
|
||||
clash = e
|
||||
break
|
||||
}
|
||||
}
|
||||
if clash != "" {
|
||||
warnings = append(warnings, fmt.Sprintf("skipped %s: slug %q is already used by %s", p.Path, slug, clash))
|
||||
continue
|
||||
}
|
||||
kept = append(kept, p)
|
||||
}
|
||||
return kept, warnings, nil
|
||||
}
|
||||
|
||||
func distinctPaths(pages []wiki.Page) int {
|
||||
seen := map[string]bool{}
|
||||
for _, p := range pages {
|
||||
seen[p.Path] = true
|
||||
}
|
||||
return len(seen)
|
||||
}
|
||||
|
||||
@@ -23,7 +23,7 @@ func injectSourceRefs(pages []wiki.Page, inventory map[wiki.PageType][]wiki.Entr
|
||||
|
||||
var sourceContent string
|
||||
for _, p := range pages {
|
||||
if strings.HasPrefix(p.Path, "wiki/sources/") &&
|
||||
if isSourcePath(p.Path) &&
|
||||
strings.TrimSuffix(filepath.Base(p.Path), ".md") == sourceSlug {
|
||||
sourceContent = p.Content
|
||||
break
|
||||
@@ -35,7 +35,7 @@ func injectSourceRefs(pages []wiki.Page, inventory map[wiki.PageType][]wiki.Entr
|
||||
|
||||
bySlug := make(map[string]int, len(pages))
|
||||
for i, p := range pages {
|
||||
if !strings.HasPrefix(p.Path, "wiki/sources/") {
|
||||
if !isSourcePath(p.Path) {
|
||||
bySlug[strings.TrimSuffix(filepath.Base(p.Path), ".md")] = i
|
||||
}
|
||||
}
|
||||
@@ -52,12 +52,13 @@ func injectSourceRefs(pages []wiki.Page, inventory map[wiki.PageType][]wiki.Entr
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
b, err := readBrainFile(brainDir, "wiki/"+string(pt)+"/"+slug+".md")
|
||||
path := entryPath(slug, pt, inventory)
|
||||
b, err := readBrainFile(brainDir, path)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
page := wiki.Page{
|
||||
Path: "wiki/" + string(pt) + "/" + slug + ".md",
|
||||
Path: path,
|
||||
Content: string(b),
|
||||
}
|
||||
pages = append(pages, addSourceRef(page, sourceRef))
|
||||
@@ -88,7 +89,7 @@ func extractWikilinks(content string) map[string]bool {
|
||||
// findSourcePage returns the slug and title of the first wiki/sources/ page in pages.
|
||||
func findSourcePage(pages []wiki.Page) (slug, title string, found bool) {
|
||||
for _, p := range pages {
|
||||
if strings.HasPrefix(p.Path, "wiki/sources/") {
|
||||
if isSourcePath(p.Path) {
|
||||
slug = strings.TrimSuffix(filepath.Base(p.Path), ".md")
|
||||
title = extractTitle(p.Content)
|
||||
if title == "" {
|
||||
@@ -111,3 +112,27 @@ func findInInventory(slug string, inventory map[wiki.PageType][]wiki.Entry) (wik
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
// isSourcePath reports whether path is a source page in either layout:
|
||||
// wiki/sources/<slug>.md or wiki/<wing>/sources/<slug>.md.
|
||||
func isSourcePath(path string) bool {
|
||||
parts := strings.Split(filepath.ToSlash(path), "/")
|
||||
switch {
|
||||
case len(parts) == 3 && parts[0] == "wiki" && parts[1] == "sources":
|
||||
return true
|
||||
case len(parts) == 4 && parts[0] == "wiki" && parts[2] == "sources":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// entryPath is the brain-relative path of an inventory page: the recorded Path
|
||||
// for a wing-layout entry, else the retired wiki/<type>/<slug>.md.
|
||||
func entryPath(slug string, pt wiki.PageType, inventory map[wiki.PageType][]wiki.Entry) string {
|
||||
for _, e := range inventory[pt] {
|
||||
if e.Slug == slug && e.Path != "" {
|
||||
return e.Path
|
||||
}
|
||||
}
|
||||
return "wiki/" + string(pt) + "/" + slug + ".md"
|
||||
}
|
||||
|
||||
@@ -32,7 +32,14 @@ func Resolve(proposed []wiki.Page, inventory map[wiki.PageType][]wiki.Entry) []w
|
||||
|
||||
out := make([]wiki.Page, 0, len(proposed))
|
||||
for _, page := range proposed {
|
||||
pt := pageTypeFromPath(page.Path)
|
||||
if isWingSourcePath(page.Path) {
|
||||
// In a wing the source's path is its filename-derived slug. Matching
|
||||
// by TITLE would merge two different notes the model happened to
|
||||
// title alike ("Home", "Untitled") into one page.
|
||||
out = append(out, page)
|
||||
continue
|
||||
}
|
||||
pt := pageTypeOf(page)
|
||||
title := extractTitle(page.Content)
|
||||
k := key{pt: pt, normalized: normalizeTitle(title)}
|
||||
if canonicalSlug, ok := lookup[k]; ok {
|
||||
@@ -55,6 +62,17 @@ func normalizeTitle(s string) string {
|
||||
return strings.Join(strings.Fields(s), " ")
|
||||
}
|
||||
|
||||
// pageTypeOf is the page's type: from its directory in the retired layout, or
|
||||
// "sources" for wiki/<wing>/sources/<slug>.md. Concepts and entities are always
|
||||
// in their shared directories.
|
||||
func pageTypeOf(p wiki.Page) wiki.PageType {
|
||||
parts := strings.Split(filepath.ToSlash(p.Path), "/")
|
||||
if len(parts) == 4 && parts[0] == "wiki" && parts[2] == "sources" {
|
||||
return wiki.PageTypeSource
|
||||
}
|
||||
return pageTypeFromPath(p.Path)
|
||||
}
|
||||
|
||||
// pageTypeFromPath extracts the wiki.PageType from a path like "wiki/entities/foo.md".
|
||||
func pageTypeFromPath(path string) wiki.PageType {
|
||||
parts := strings.Split(filepath.ToSlash(path), "/")
|
||||
@@ -86,3 +104,9 @@ func extractTitle(content string) string {
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// isWingSourcePath reports whether path is wiki/<wing>/sources/<slug>.md.
|
||||
func isWingSourcePath(path string) bool {
|
||||
parts := strings.Split(filepath.ToSlash(path), "/")
|
||||
return len(parts) == 4 && parts[0] == "wiki" && parts[2] == "sources"
|
||||
}
|
||||
|
||||
@@ -35,17 +35,17 @@ func TestReadSourcePageAndSchemaRefuseSymlinksAndSurviveSwapRaces(t *testing.T)
|
||||
inBrain := brainfstest.WriteSentinelAt(t, filepath.Join(brain, "raw", "s.md"))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(brain, "wiki", "sources", "ok.md"), []byte("zqinside source"), 0o644))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(brain, "schema.md"), []byte("zqinside schema"), 0o644))
|
||||
assert.Equal(t, "zqinside source", readSourcePage(brain, "ok"), "positive control")
|
||||
assert.Equal(t, "zqinside source", readSourcePage(brain, "", "ok"), "positive control")
|
||||
assert.Equal(t, "zqinside schema", loadSchema(brain), "positive control")
|
||||
|
||||
require.NoError(t, os.Symlink(brainfstest.Outside(t), filepath.Join(brain, "wiki", "sources", "out.md")))
|
||||
require.NoError(t, os.Symlink(filepath.Join("..", "..", "raw", "s.md"), filepath.Join(brain, "wiki", "sources", "in.md")))
|
||||
assert.NotContains(t, readSourcePage(brain, "out"), brainfstest.Sentinel)
|
||||
assert.NotContains(t, readSourcePage(brain, "in"), brainfstest.Sentinel)
|
||||
assert.NotContains(t, storedSourceHash(brain, "in"), brainfstest.Sentinel)
|
||||
assert.NotContains(t, readSourcePage(brain, "", "out"), brainfstest.Sentinel)
|
||||
assert.NotContains(t, readSourcePage(brain, "", "in"), brainfstest.Sentinel)
|
||||
assert.NotContains(t, storedSourceHash(brain, "", "in"), brainfstest.Sentinel)
|
||||
|
||||
src := filepath.Join(brain, "wiki", "sources", "r.md")
|
||||
read := func() string { return readSourcePage(brain, "r") }
|
||||
read := func() string { return readSourcePage(brain, "", "r") }
|
||||
brainfstest.AssertNoLeak(t, src, "zqinside", 1500*time.Millisecond, read)
|
||||
brainfstest.AssertNoLeakFrom(t, src, "zqinside", inBrain, 1500*time.Millisecond, read)
|
||||
brainfstest.AssertNoHang(t, src, "zqinside", 1500*time.Millisecond, 6*time.Second, func() { _ = read() })
|
||||
@@ -165,3 +165,53 @@ func TestBackfillRefsDoesNotCopyAPageSwappedInBehindASymlink(t *testing.T) {
|
||||
require.Contains(t, read(), "[[s|S]]", "positive control: the regular page gets the source ref")
|
||||
brainfstest.AssertNoLeakFrom(t, page, inside, target, 3*time.Second, read)
|
||||
}
|
||||
|
||||
// hyperguild#158: BackfillRefs read a regular concept page, a symlink could
|
||||
// be swapped in, and os.WriteFile then followed it and overwrote the link's
|
||||
// target. The target is not restored between rounds here, so any
|
||||
// write-through shows.
|
||||
func TestBackfillRefsNeverWritesThroughAPageSwappedForASymlink(t *testing.T) {
|
||||
brain := t.TempDir()
|
||||
mkdirs(t, brain, "wiki/sources", "wiki/concepts", "raw")
|
||||
require.NoError(t, os.WriteFile(filepath.Join(brain, "wiki", "sources", "s.md"), []byte("---\ntitle: S\n---\n[[c|C]]\n"), 0o644))
|
||||
page := filepath.Join(brain, "wiki", "concepts", "c.md")
|
||||
target := filepath.Join(brain, "raw", "secret.md")
|
||||
require.NoError(t, os.WriteFile(target, []byte("untouched"), 0o644))
|
||||
inside := "---\ntitle: C\n---\nbody\n"
|
||||
require.NoError(t, os.WriteFile(page, []byte(inside), 0o644))
|
||||
brainfstest.StartSwap(t, page, inside, target, false)
|
||||
ok := 0
|
||||
for deadline := time.Now().Add(2 * time.Second); time.Now().Before(deadline); {
|
||||
if _, err := BackfillRefs(context.Background(), brain); err == nil {
|
||||
ok++
|
||||
}
|
||||
}
|
||||
got, _ := os.ReadFile(target)
|
||||
assert.Equal(t, "untouched", string(got), "BackfillRefs wrote through a swapped-in symlink")
|
||||
assert.Positive(t, ok, "positive control: some backfills ran")
|
||||
}
|
||||
|
||||
// hyperguild#158: buildAndWrite wrote each page with os.WriteFile by path, so
|
||||
// a page read as regular and swapped for a symlink before the write was
|
||||
// written through. The target is not restored between rounds, so any
|
||||
// write-through shows.
|
||||
func TestBuildAndWriteNeverWritesThroughAPageSwappedForASymlink(t *testing.T) {
|
||||
brain := t.TempDir()
|
||||
mkdirs(t, brain, "wiki/sources", "wiki/concepts", "wiki/entities", "raw")
|
||||
page := filepath.Join(brain, "wiki", "concepts", "zq-concept.md")
|
||||
target := filepath.Join(brain, "raw", "secret.md")
|
||||
require.NoError(t, os.WriteFile(target, []byte("untouched"), 0o644))
|
||||
inside := "---\ntitle: Zq Concept\n---\nbody\n"
|
||||
require.NoError(t, os.WriteFile(page, []byte(inside), 0o644))
|
||||
pages := []RawPage{{Title: "Zq Concept", Type: "concept", Content: "## About\n\nmore\n"}}
|
||||
brainfstest.StartSwap(t, page, inside, target, false)
|
||||
ok := 0
|
||||
for deadline := time.Now().Add(2 * time.Second); time.Now().Before(deadline); {
|
||||
if _, err := RunRaw(context.Background(), brain, "zq source", pages, false); err == nil {
|
||||
ok++
|
||||
}
|
||||
}
|
||||
got, _ := os.ReadFile(target)
|
||||
assert.Equal(t, "untouched", string(got), "buildAndWrite wrote through a swapped-in symlink")
|
||||
assert.Positive(t, ok, "positive control: some runs returned without error")
|
||||
}
|
||||
|
||||
@@ -0,0 +1,396 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func wingCfg(wing string, pages ...RawPage) Config {
|
||||
return Config{
|
||||
Wing: wing,
|
||||
Schema: "# Schema",
|
||||
Complete: func(context.Context, string, string) (string, error) {
|
||||
return mustJSON(pages), nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func sourceEntityConcept() []RawPage {
|
||||
return []RawPage{
|
||||
{Title: "Dark Factory Talk", Type: "source", Subtype: "video", Domain: "ai",
|
||||
Content: "## Summary\n\nA talk.\n\n## Entities Mentioned\n\n[[Jev]]\n\n## Concepts Introduced or Reinforced\n\n[[Dark Factory]]\n"},
|
||||
{Title: "Jev", Type: "entity", Subtype: "tool", Domain: "ai",
|
||||
Content: "## Definition\n\nA decision model.\n\n## Sources\n"},
|
||||
{Title: "Dark Factory", Type: "concept", Domain: "ai",
|
||||
Content: "## Definition\n\nA fully automated build line.\n\n## Sources\n"},
|
||||
}
|
||||
}
|
||||
|
||||
func exists(dir, rel string) bool {
|
||||
_, err := os.Stat(filepath.Join(dir, rel))
|
||||
return err == nil
|
||||
}
|
||||
|
||||
func read(t *testing.T, dir, rel string) string {
|
||||
t.Helper()
|
||||
b, err := os.ReadFile(filepath.Join(dir, rel))
|
||||
require.NoError(t, err)
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// With a wing, the SOURCE belongs to the wing; concepts and entities are the
|
||||
// shared cross-wing layer (wiki/concepts, wiki/entities) because their slug is
|
||||
// the graph's global identity and they are cited from many wings.
|
||||
func TestRun_WithAWingTheSourceIsPerWingAndConceptsEntitiesAreShared(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
_, err := Run(context.Background(), wingCfg("web-clip", sourceEntityConcept()...), dir, "body of the talk", "dark-factory-talk", false)
|
||||
require.NoError(t, err)
|
||||
|
||||
src := read(t, dir, "wiki/web-clip/sources/dark-factory-talk.md")
|
||||
assert.Contains(t, src, "wing: web-clip")
|
||||
assert.Contains(t, src, "hall: sources")
|
||||
assert.True(t, exists(dir, "wiki/entities/jev.md"))
|
||||
assert.True(t, exists(dir, "wiki/concepts/dark-factory.md"))
|
||||
assert.NotContains(t, read(t, dir, "wiki/entities/jev.md"), "wing:", "a shared page belongs to no single wing")
|
||||
assert.False(t, exists(dir, "wiki/web-clip/facts"), "no per-wing copy of a shared page")
|
||||
assert.False(t, exists(dir, "wiki/sources"), "the retired source layout is not used")
|
||||
}
|
||||
|
||||
func TestRun_WithAWingTheSharedPageLinksBackToTheSource(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
_, err := Run(context.Background(), wingCfg("web-clip", sourceEntityConcept()...), dir, "body", "dark-factory-talk", false)
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, read(t, dir, "wiki/entities/jev.md"), "[[dark-factory-talk|Dark Factory Talk]]")
|
||||
}
|
||||
|
||||
func TestRun_TheSameEntityCitedFromTwoWingsIsOnePageWithBothSources(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
_, err := Run(context.Background(), wingCfg("web-clip", sourceEntityConcept()...), dir, "first body", "talk-one", false)
|
||||
require.NoError(t, err)
|
||||
// A wing must already exist to be named; create it the way a note would.
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki/homelab/facts"), 0o755))
|
||||
|
||||
again := []RawPage{
|
||||
{Title: "Second Talk", Type: "source", Subtype: "video", Content: "## Summary\n\nMore.\n\n## Entities Mentioned\n\n[[The Jev]]\n"},
|
||||
{Title: "The Jev", Type: "entity", Subtype: "tool", Content: "## Definition\n\nStill a decision model.\n\n## Sources\n"},
|
||||
}
|
||||
_, err = Run(context.Background(), wingCfg("homelab", again...), dir, "second body", "talk-two", false)
|
||||
require.NoError(t, err)
|
||||
|
||||
files, err := filepath.Glob(filepath.Join(dir, "wiki/entities/*.md"))
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, files, 1, "one entity page, matched by title with the leading article ignored")
|
||||
jev := read(t, dir, "wiki/entities/jev.md")
|
||||
assert.Contains(t, jev, "talk-one")
|
||||
assert.Contains(t, jev, "talk-two")
|
||||
assert.False(t, exists(dir, "wiki/homelab/facts/jev.md"), "no second copy in the second wing")
|
||||
}
|
||||
|
||||
// A source slug is the graph's identity too: a second wing must not take over
|
||||
// the slug of a page that lives elsewhere. Two wings dropping a file of the same
|
||||
// name (notes.md, readme.md) is ordinary, so the second source gets a distinct,
|
||||
// stable slug instead of failing.
|
||||
func TestRun_ASourceWhoseSlugIsTakenElsewhereGetsADistinctStableSlug(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
_, err := Run(context.Background(), wingCfg("web-clip", sourceEntityConcept()...), dir, "body", "talk", false)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki/homelab/facts"), 0o755))
|
||||
|
||||
_, err = Run(context.Background(), wingCfg("homelab", sourceEntityConcept()...), dir, "different body", "talk", false)
|
||||
require.NoError(t, err)
|
||||
assert.True(t, exists(dir, "wiki/web-clip/sources/talk.md"), "the first wing's page is untouched")
|
||||
assert.True(t, exists(dir, "wiki/homelab/sources/talk-homelab.md"), "the second gets a wing-qualified slug")
|
||||
assert.False(t, exists(dir, "wiki/homelab/sources/talk.md"))
|
||||
|
||||
_, err = Run(context.Background(), wingCfg("homelab", sourceEntityConcept()...), dir, "different body, edited", "talk", false)
|
||||
require.NoError(t, err)
|
||||
files, _ := filepath.Glob(filepath.Join(dir, "wiki/homelab/sources/*.md"))
|
||||
assert.Len(t, files, 1, "re-ingesting finds the same page, it does not make talk-homelab-2")
|
||||
}
|
||||
|
||||
func TestRun_ASourceWhoseSlugIsTakenByARetiredPageOrASharedPageGetsADistinctSlug(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki/sources"), 0o755))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(dir, "wiki/sources/talk.md"), []byte("---\ntitle: Talk\n---\nold"), 0o644))
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki/entities"), 0o755))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(dir, "wiki/entities/other.md"), []byte("---\ntitle: Other\n---\nx"), 0o644))
|
||||
|
||||
_, err := Run(context.Background(), wingCfg("web-clip", sourceEntityConcept()...), dir, "body", "talk", false)
|
||||
require.NoError(t, err)
|
||||
assert.True(t, exists(dir, "wiki/web-clip/sources/talk-web-clip.md"))
|
||||
assert.Equal(t, "---\ntitle: Talk\n---\nold", read(t, dir, "wiki/sources/talk.md"), "the retired page is untouched")
|
||||
}
|
||||
|
||||
// The other direction: a model-extracted entity or concept whose slug is a
|
||||
// source's slug in some wing would lock that source out of re-ingest. The page
|
||||
// is skipped with a warning; the ingest itself succeeds.
|
||||
func TestRun_AnEntityWhoseSlugIsASourceElsewhereIsSkippedNotWritten(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
_, err := Run(context.Background(), wingCfg("web-clip", []RawPage{
|
||||
{Title: "Jev", Type: "source", Subtype: "video", Content: "## Summary\n\nA clip titled Jev.\n"},
|
||||
}...), dir, "body", "jev", false)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki/homelab/facts"), 0o755))
|
||||
|
||||
res, err := Run(context.Background(), wingCfg("homelab", sourceEntityConcept()...), dir, "another body", "other-talk", false)
|
||||
require.NoError(t, err)
|
||||
assert.False(t, exists(dir, "wiki/entities/jev.md"), "the entity would take the source's slug")
|
||||
assert.Contains(t, strings.Join(res.Warnings, "\n"), "jev")
|
||||
|
||||
_, err = Run(context.Background(), wingCfg("web-clip", []RawPage{
|
||||
{Title: "Jev", Type: "source", Subtype: "video", Content: "## Summary\n\nEdited.\n"},
|
||||
}...), dir, "body, edited", "jev", false)
|
||||
require.NoError(t, err, "the first wing can still re-ingest its own source")
|
||||
}
|
||||
|
||||
// The type comes from the model, so from the clip: in a wing it may only be one
|
||||
// of the three the layout has a place for.
|
||||
func TestRun_WithAWingAModelChosenTypeCannotWriteOutsideTheLayout(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki/alpha/facts"), 0o755))
|
||||
pages := append(sourceEntityConcept(),
|
||||
RawPage{Title: "Planted", Type: "sources", Content: "## x\n\nplanted\n"},
|
||||
RawPage{Title: "Loose", Type: "alpha", Content: "## x\n\nloose\n"},
|
||||
RawPage{Title: "Sneaky", Type: "homelab", Content: "## x\n\nsneaky\n"})
|
||||
res, err := Run(context.Background(), wingCfg("web-clip", pages...), dir, "body", "talk", false)
|
||||
require.NoError(t, err)
|
||||
assert.False(t, exists(dir, "wiki/sources/planted.md"))
|
||||
assert.False(t, exists(dir, "wiki/alpha/loose.md"))
|
||||
assert.False(t, exists(dir, "wiki/homelab"))
|
||||
assert.Contains(t, strings.Join(res.Warnings, "\n"), "invalid type")
|
||||
}
|
||||
|
||||
func TestRun_ReIngestingTheSameSourceIntoItsOwnWingIsNotACollision(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
_, err := Run(context.Background(), wingCfg("web-clip", sourceEntityConcept()...), dir, "body one", "talk", false)
|
||||
require.NoError(t, err)
|
||||
_, err = Run(context.Background(), wingCfg("web-clip", sourceEntityConcept()...), dir, "body two, edited", "talk", false)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
func TestRun_WithAWingRejectsANameThatIsNotAPlainWing(t *testing.T) {
|
||||
for _, bad := range []string{"../escape", "a/b", "Radar", "has space", "."} {
|
||||
t.Run(bad, func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
_, err := Run(context.Background(), wingCfg(bad, sourceEntityConcept()...), dir, "body", "talk", false)
|
||||
require.Error(t, err)
|
||||
assert.Len(t, listAll(t, dir), 1, "nothing is written for a rejected wing (only the temp dir itself exists)")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func listAll(t *testing.T, dir string) []string {
|
||||
t.Helper()
|
||||
var out []string
|
||||
_ = filepath.WalkDir(dir, func(p string, _ os.DirEntry, _ error) error { out = append(out, p); return nil })
|
||||
return out
|
||||
}
|
||||
|
||||
// Title resolution can mask a wrong slug choice at the Run level, so the
|
||||
// stability of the choice is pinned directly: the page this wing made earlier
|
||||
// under the qualified slug is "its own" and is found again.
|
||||
func TestResolveSourceSlug(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
for _, rel := range []string{
|
||||
"wiki/web-clip/sources/talk.md",
|
||||
"wiki/homelab/sources/talk-homelab.md",
|
||||
"wiki/entities/kube.md",
|
||||
} {
|
||||
require.NoError(t, os.MkdirAll(filepath.Dir(filepath.Join(dir, rel)), 0o755))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(dir, rel), []byte("---\ntitle: x\n---\n"), 0o644))
|
||||
}
|
||||
tests := []struct {
|
||||
name, wing, base, want string
|
||||
}{
|
||||
{"free slug is used as is", "homelab", "fresh", "fresh"},
|
||||
{"own page under the plain slug", "web-clip", "talk", "talk"},
|
||||
{"taken elsewhere: wing-qualified", "tapir", "talk", "talk-tapir"},
|
||||
{"own earlier qualified page is found again", "homelab", "talk", "talk-homelab"},
|
||||
{"taken by a shared page", "tapir", "kube", "kube-tapir"},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got, err := resolveSourceSlug(dir, tt.wing, tt.base)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, tt.want, got)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSourceSlug_AFurtherClashGetsANumber(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
for _, rel := range []string{"wiki/a/sources/talk.md", "wiki/entities/talk-b.md"} {
|
||||
require.NoError(t, os.MkdirAll(filepath.Dir(filepath.Join(dir, rel)), 0o755))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(dir, rel), []byte("x"), 0o644))
|
||||
}
|
||||
got, err := resolveSourceSlug(dir, "b", "talk")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "talk-b-2", got)
|
||||
}
|
||||
|
||||
// From the second review of #211: slug clashes INSIDE one batch. dropSlugClashes
|
||||
// only looked at what was already on disk, so a source and an entity extracted
|
||||
// together with one slug were both written, and neither could ever be updated.
|
||||
func TestRun_ASourceAndAnEntityWithOneSlugInTheSameBatchWriteOnlyTheSource(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
pages := []RawPage{
|
||||
{Title: "Notes", Type: "source", Subtype: "note", Content: "## Summary\n\nfirst\n"},
|
||||
{Title: "Notes", Type: "entity", Subtype: "tool", Content: "## Definition\n\nan entity named like its source\n"},
|
||||
}
|
||||
res, err := Run(context.Background(), wingCfg("homelab", pages...), dir, "body one", "notes", false)
|
||||
require.NoError(t, err)
|
||||
assert.True(t, exists(dir, "wiki/homelab/sources/notes.md"))
|
||||
assert.False(t, exists(dir, "wiki/entities/notes.md"), "two pages must not share a slug")
|
||||
assert.Contains(t, strings.Join(res.Warnings, "\n"), "notes")
|
||||
|
||||
pages[0].Content = "## Summary\n\nfirst\n\n## Key Claims\n\n- second\n"
|
||||
_, err = Run(context.Background(), wingCfg("homelab", pages...), dir, "body two, changed", "notes", false)
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, read(t, dir, "wiki/homelab/sources/notes.md"), "second", "the source can still be updated")
|
||||
}
|
||||
|
||||
func TestRun_AConceptAndAnEntityWithOneSlugInTheSameBatchWriteOne(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
pages := []RawPage{
|
||||
{Title: "Talk", Type: "source", Subtype: "video", Content: "## Summary\n\nx\n"},
|
||||
{Title: "Foo", Type: "concept", Content: "## Definition\n\na concept\n"},
|
||||
{Title: "Foo", Type: "entity", Subtype: "tool", Content: "## Definition\n\nan entity\n"},
|
||||
}
|
||||
res, err := Run(context.Background(), wingCfg("homelab", pages...), dir, "body", "talk", false)
|
||||
require.NoError(t, err)
|
||||
written := 0
|
||||
for _, rel := range []string{"wiki/concepts/foo.md", "wiki/entities/foo.md"} {
|
||||
if exists(dir, rel) {
|
||||
written++
|
||||
}
|
||||
}
|
||||
assert.Equal(t, 1, written, "one slug, one page")
|
||||
assert.Contains(t, strings.Join(res.Warnings, "\n"), "foo")
|
||||
}
|
||||
|
||||
// A brain that already holds the bad pair (written before this fix) must not
|
||||
// lock the source out: the source page is the note's own output and is never
|
||||
// the one skipped.
|
||||
func TestRun_AnExistingSourceEntityPairDoesNotLockTheSourceOut(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki/homelab/sources"), 0o755))
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki/entities"), 0o755))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(dir, "wiki/homelab/sources/notes.md"),
|
||||
[]byte("---\ntitle: 'Notes'\ntype: 'note'\nwing: homelab\nhall: sources\naliases:\n - 'Notes'\n---\n\n## Summary\n\nold\n"), 0o644))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(dir, "wiki/entities/notes.md"),
|
||||
[]byte("---\ntitle: 'Notes'\naliases:\n - 'Notes'\n---\n\n## Definition\n\nstale entity\n"), 0o644))
|
||||
|
||||
pages := []RawPage{{Title: "Notes", Type: "source", Subtype: "note", Content: "## Summary\n\nnew text\n\n## Key Claims\n\n- fresh\n"}}
|
||||
_, err := Run(context.Background(), wingCfg("homelab", pages...), dir, "changed body", "notes", false)
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, read(t, dir, "wiki/homelab/sources/notes.md"), "fresh", "the source took the update")
|
||||
assert.Contains(t, read(t, dir, "wiki/entities/notes.md"), "stale entity", "the entity is untouched")
|
||||
}
|
||||
|
||||
// Pass 3 of #211: a note must never leave the inbox as "processed" when nothing
|
||||
// of it was written.
|
||||
func TestRun_WithAWingAnIngestThatWritesNoSourcePageIsAnError(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
pages []RawPage
|
||||
prep func(t *testing.T, dir string)
|
||||
}{
|
||||
{"the model returned nothing", nil, nil},
|
||||
{"only pages of a type the layout has no place for", []RawPage{{Title: "Claim", Type: "claim", Content: "## x\n\ny\n"}}, nil},
|
||||
{"only an entity, and its slug is taken by a source", []RawPage{{Title: "S1", Type: "entity", Subtype: "tool", Content: "## Definition\n\nx\n"}},
|
||||
func(t *testing.T, dir string) {
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki/web-clip/sources"), 0o755))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(dir, "wiki/web-clip/sources/s1.md"), []byte("---\ntitle: 'S1'\n---\nx"), 0o644))
|
||||
}},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
if tt.prep != nil {
|
||||
tt.prep(t, dir)
|
||||
}
|
||||
res, err := Run(context.Background(), wingCfg("web-clip", tt.pages...), dir, "a note body", "other-note", false)
|
||||
require.Error(t, err, "nothing was ingested; the caller must not archive it as done")
|
||||
assert.Empty(t, res.Pages)
|
||||
assert.Contains(t, err.Error(), "no source page")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRun_WithAWingAnUnchangedSourceIsStillASkipNotAnError(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
_, err := Run(context.Background(), wingCfg("web-clip", sourceEntityConcept()...), dir, "same body", "talk", false)
|
||||
require.NoError(t, err)
|
||||
res, err := Run(context.Background(), wingCfg("web-clip", sourceEntityConcept()...), dir, "same body", "talk", false)
|
||||
require.NoError(t, err)
|
||||
assert.True(t, res.Skipped)
|
||||
}
|
||||
|
||||
// Title resolution must not merge two different notes into one source page just
|
||||
// because the model titled both "Home": in a wing the source path is already
|
||||
// deterministic (the filename), so the title must not redirect it.
|
||||
func TestRun_WithAWingTwoNotesTheModelTitlesAlikeKeepTheirOwnSourcePages(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
home := func(body string) []RawPage {
|
||||
return []RawPage{{Title: "Home", Type: "source", Subtype: "article", Content: "## Summary\n\n" + body + "\n"}}
|
||||
}
|
||||
_, err := Run(context.Background(), wingCfg("web-clip", home("note A text")...), dir, "body A", "note-a", false)
|
||||
require.NoError(t, err)
|
||||
_, err = Run(context.Background(), wingCfg("web-clip", home("note B text")...), dir, "body B", "note-b", false)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.True(t, exists(dir, "wiki/web-clip/sources/note-a.md"))
|
||||
assert.True(t, exists(dir, "wiki/web-clip/sources/note-b.md"), "note B has its own page")
|
||||
assert.Contains(t, read(t, dir, "wiki/web-clip/sources/note-a.md"), "note A text")
|
||||
assert.Contains(t, read(t, dir, "wiki/web-clip/sources/note-b.md"), "note B text")
|
||||
|
||||
res, err := Run(context.Background(), wingCfg("web-clip", home("note A text")...), dir, "body A", "note-a", false)
|
||||
require.NoError(t, err)
|
||||
assert.True(t, res.Skipped, "note A's hash was not overwritten by note B")
|
||||
}
|
||||
|
||||
// Both a concept and an entity named "Foo" arrive and the CONCEPT already exists
|
||||
// on disk: the page that exists wins the tie, so its source reference is added.
|
||||
func TestRun_ABatchTieBetweenAConceptAndAnEntityPrefersThePageAlreadyOnDisk(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki/concepts"), 0o755))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(dir, "wiki/concepts/foo.md"),
|
||||
[]byte("---\ntitle: 'Foo'\naliases:\n - 'Foo'\n---\n\n## Definition\n\nexisting concept\n\n## Sources\n"), 0o644))
|
||||
pages := []RawPage{
|
||||
{Title: "Talk", Type: "source", Subtype: "video", Content: "## Summary\n\nx\n\n## Concepts Introduced or Reinforced\n\n[[Foo]]\n"},
|
||||
{Title: "Foo", Type: "entity", Subtype: "tool", Content: "## Definition\n\nan entity\n"},
|
||||
{Title: "Foo", Type: "concept", Content: "## Definition\n\na concept\n\n## Sources\n"},
|
||||
}
|
||||
_, err := Run(context.Background(), wingCfg("web-clip", pages...), dir, "body", "talk", false)
|
||||
require.NoError(t, err)
|
||||
assert.False(t, exists(dir, "wiki/entities/foo.md"))
|
||||
assert.Contains(t, read(t, dir, "wiki/concepts/foo.md"), "talk", "the existing concept got the new source reference")
|
||||
}
|
||||
|
||||
// Pass 4 of #211: the source page carries the hash the drift check reads. If it
|
||||
// is written FIRST and a later page fails to write, the next ingest of the same
|
||||
// content is skipped and the rest of the batch is lost for good. Written LAST,
|
||||
// a failed batch leaves no hash and is simply redone.
|
||||
func TestRun_WithAWingAFailedBatchLeavesNoSourcePageToHideBehind(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
// An entity path that cannot be written: a directory where the page goes.
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki/entities/jev.md"), 0o755))
|
||||
|
||||
_, err := Run(context.Background(), wingCfg("web-clip", sourceEntityConcept()...), dir, "same body", "talk", false)
|
||||
require.Error(t, err, "the entity write fails")
|
||||
assert.False(t, exists(dir, "wiki/web-clip/sources/talk.md"),
|
||||
"the source page (and its hash) must not exist when the batch did not complete")
|
||||
|
||||
require.NoError(t, os.RemoveAll(filepath.Join(dir, "wiki/entities/jev.md")))
|
||||
res, err := Run(context.Background(), wingCfg("web-clip", sourceEntityConcept()...), dir, "same body", "talk", false)
|
||||
require.NoError(t, err)
|
||||
assert.False(t, res.Skipped, "the retry is a real ingest, not a hash skip")
|
||||
assert.True(t, exists(dir, "wiki/entities/jev.md"))
|
||||
assert.True(t, exists(dir, "wiki/web-clip/sources/talk.md"))
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
package symlinkguard
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"syscall"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// hyperguild#179: the tripwire looked only for symlinks, so a FIFO (or
|
||||
// socket) in the brain passed it. A FIFO at classification.yaml then made
|
||||
// startup exit; at a note path it could hang readers that skip the handle
|
||||
// check.
|
||||
func TestScanReportsFIFOsAndSocketsAsSpecialFiles(t *testing.T) {
|
||||
brain := t.TempDir()
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki", "w"), 0o755))
|
||||
require.NoError(t, syscall.Mkfifo(filepath.Join(brain, "classification.yaml"), 0o644))
|
||||
require.NoError(t, syscall.Mkfifo(filepath.Join(brain, "wiki", "w", "f.md"), 0o644))
|
||||
l, err := net.Listen("unix", filepath.Join(brain, "s.sock"))
|
||||
require.NoError(t, err)
|
||||
defer func() { _ = l.Close() }()
|
||||
require.NoError(t, os.WriteFile(filepath.Join(brain, "wiki", "w", "n.md"), []byte("n"), 0o644))
|
||||
|
||||
r, err := Scan(brain)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{"classification.yaml", "s.sock", "wiki/w/f.md"}, r.Special)
|
||||
assert.False(t, r.Clean())
|
||||
assert.Contains(t, r.String(), "special file")
|
||||
}
|
||||
|
||||
// Watch removes a FIFO as it removes symlinks, and leaves a regular file
|
||||
// (which Scan never reports) alone.
|
||||
func TestWatchRemovesAFIFOAndLeavesAnUnreportedRegularFile(t *testing.T) {
|
||||
brain := t.TempDir()
|
||||
fifo := filepath.Join(brain, "f.md")
|
||||
require.NoError(t, syscall.Mkfifo(fifo, 0o644))
|
||||
regular := filepath.Join(brain, "n.md")
|
||||
require.NoError(t, os.WriteFile(regular, []byte("n"), 0o644))
|
||||
n := &recNotifier{}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
go Watch(ctx, brain, 10*time.Millisecond, n)
|
||||
require.Eventually(t, func() bool {
|
||||
_, err := os.Lstat(fifo)
|
||||
return os.IsNotExist(err)
|
||||
}, 2*time.Second, 10*time.Millisecond, "the FIFO is removed")
|
||||
require.Eventually(t, func() bool { return n.count() >= 1 }, 2*time.Second, 10*time.Millisecond)
|
||||
b, err := os.ReadFile(regular)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "n", string(b))
|
||||
n.mu.Lock()
|
||||
defer n.mu.Unlock()
|
||||
assert.Contains(t, n.msgs[0], "f.md")
|
||||
}
|
||||
|
||||
// removeSpecial re-checks the type: a path that is a regular file or a
|
||||
// directory at the re-check is not removed. (A swap between the re-check and
|
||||
// the unlink can still remove one; Linux has no conditional unlink. The same
|
||||
// window exists in removeSymlinks.)
|
||||
func TestRemoveSpecialRefusesRegularFilesAndDirectories(t *testing.T) {
|
||||
brain := t.TempDir()
|
||||
require.NoError(t, os.WriteFile(filepath.Join(brain, "n.md"), []byte("n"), 0o644))
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(brain, "d"), 0o755))
|
||||
require.NoError(t, syscall.Mkfifo(filepath.Join(brain, "f.md"), 0o644))
|
||||
removed, failed := removeSpecial(brain, []string{"n.md", "d", "f.md"})
|
||||
assert.Equal(t, []string{"f.md"}, removed)
|
||||
assert.Equal(t, []string{"n.md", "d"}, failed)
|
||||
_, err := os.Stat(filepath.Join(brain, "n.md"))
|
||||
assert.NoError(t, err)
|
||||
}
|
||||
|
||||
// A special file Watch cannot remove (its directory is read-only) stays
|
||||
// reported: the persistent-finding alert names it.
|
||||
func TestWatchReportsASpecialFileItCannotRemove(t *testing.T) {
|
||||
if os.Geteuid() == 0 {
|
||||
t.Skip("root ignores directory permissions")
|
||||
}
|
||||
brain := t.TempDir()
|
||||
ro := filepath.Join(brain, "ro")
|
||||
require.NoError(t, os.MkdirAll(ro, 0o755))
|
||||
require.NoError(t, syscall.Mkfifo(filepath.Join(ro, "f.md"), 0o644))
|
||||
require.NoError(t, os.Chmod(ro, 0o555))
|
||||
t.Cleanup(func() { _ = os.Chmod(ro, 0o755) })
|
||||
n := &recNotifier{}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
go Watch(ctx, brain, 10*time.Millisecond, n)
|
||||
require.Eventually(t, func() bool { return n.count() >= 1 }, 2*time.Second, 10*time.Millisecond)
|
||||
_, err := os.Lstat(filepath.Join(ro, "f.md"))
|
||||
require.NoError(t, err, "the FIFO could not be removed")
|
||||
n.mu.Lock()
|
||||
defer n.mu.Unlock()
|
||||
assert.Contains(t, n.msgs[0], "ro/f.md")
|
||||
assert.Contains(t, n.msgs[0], "special file")
|
||||
for _, m := range n.msgs {
|
||||
assert.NotContains(t, m, "removed", "a file it could not remove was reported as removed")
|
||||
}
|
||||
}
|
||||
@@ -46,10 +46,16 @@ type Notifier interface {
|
||||
type Result struct {
|
||||
Symlinks []string
|
||||
Unreadable []string
|
||||
// Special holds FIFOs, sockets and devices (hyperguild#179): a reader
|
||||
// that opens one by path can hang, and a FIFO at classification.yaml
|
||||
// stops startup.
|
||||
Special []string
|
||||
}
|
||||
|
||||
// Clean reports whether the scan found nothing to act on.
|
||||
func (r Result) Clean() bool { return len(r.Symlinks) == 0 && len(r.Unreadable) == 0 }
|
||||
func (r Result) Clean() bool {
|
||||
return len(r.Symlinks) == 0 && len(r.Unreadable) == 0 && len(r.Special) == 0
|
||||
}
|
||||
|
||||
// String describes the findings for a log line or alert. Paths are quoted, so
|
||||
// a name containing a newline cannot blur two findings together.
|
||||
@@ -59,6 +65,9 @@ func (r Result) String() string {
|
||||
if len(r.Symlinks) > 0 {
|
||||
parts = append(parts, fmt.Sprintf("%d symlink(s): %s", len(r.Symlinks), q(r.Symlinks)))
|
||||
}
|
||||
if len(r.Special) > 0 {
|
||||
parts = append(parts, fmt.Sprintf("%d special file(s) (FIFO, socket or device; delete them): %s", len(r.Special), q(r.Special)))
|
||||
}
|
||||
if len(r.Unreadable) > 0 {
|
||||
parts = append(parts, fmt.Sprintf("%d unreadable dir(s): %s", len(r.Unreadable), q(r.Unreadable)))
|
||||
}
|
||||
@@ -113,12 +122,16 @@ func Scan(brainDir string) (Result, error) {
|
||||
if d.IsDir() && p == rootGit {
|
||||
return filepath.SkipDir
|
||||
}
|
||||
if d.Type()&os.ModeSymlink != 0 {
|
||||
switch {
|
||||
case d.Type()&os.ModeSymlink != 0:
|
||||
r.Symlinks = append(r.Symlinks, rel)
|
||||
case !d.IsDir() && !d.Type().IsRegular():
|
||||
r.Special = append(r.Special, rel)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
sort.Strings(r.Symlinks)
|
||||
sort.Strings(r.Special)
|
||||
sort.Strings(r.Unreadable)
|
||||
if walkErr != nil {
|
||||
// Only the root's own error reaches here (deeper errors are recorded
|
||||
@@ -182,14 +195,20 @@ func Watch(ctx context.Context, brainDir string, interval time.Duration, n Notif
|
||||
q.push(fmt.Sprintf("brain symlink tripwire (hyperguild#157): removed %d symlink(s) from the brain dir: %s",
|
||||
len(removed), Result{}.quoted(removed)))
|
||||
}
|
||||
key = Result{Symlinks: failed, Unreadable: r.Unreadable}.String()
|
||||
sRemoved, sFailed := removeSpecial(brainDir, r.Special)
|
||||
if len(sRemoved) > 0 {
|
||||
slog.Error("brain symlink tripwire removed special files", "count", len(sRemoved))
|
||||
q.push(fmt.Sprintf("brain symlink tripwire (hyperguild#179): removed %d special file(s) (FIFO, socket or device) from the brain dir: %s",
|
||||
len(sRemoved), Result{}.quoted(sRemoved)))
|
||||
}
|
||||
key = Result{Symlinks: failed, Unreadable: r.Unreadable, Special: sFailed}.String()
|
||||
}
|
||||
if key != reported {
|
||||
reported = key
|
||||
if key != "" {
|
||||
slog.Error("brain symlink tripwire", "finding", key)
|
||||
q.push("brain symlink tripwire (hyperguild#157): " + key +
|
||||
". Readers may follow symlinks out of the brain; fix them.")
|
||||
q.push("brain symlink tripwire (hyperguild#157, #179): " + key +
|
||||
". Symlinks can lead readers out of the brain and special files can hang them; remove them.")
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -322,6 +341,40 @@ func permanent(err error) bool {
|
||||
return errors.As(err, &p) && p.Permanent()
|
||||
}
|
||||
|
||||
// removeSpecial unlinks each brain-relative FIFO, socket or device through
|
||||
// an os.Root on the brain dir, after re-checking that it is still one: a
|
||||
// path that is a regular file, a directory or a symlink at the re-check is
|
||||
// left alone and reported as failed. The re-check narrows, but cannot close,
|
||||
// the window before the unlink (Linux has no conditional unlink); the same
|
||||
// window exists in removeSymlinks, and only a local writer can use it.
|
||||
func removeSpecial(brainDir string, rels []string) (removed, failed []string) {
|
||||
if len(rels) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
resolved, err := filepath.EvalSymlinks(brainDir)
|
||||
if err != nil {
|
||||
return nil, rels
|
||||
}
|
||||
root, err := os.OpenRoot(resolved)
|
||||
if err != nil {
|
||||
return nil, rels
|
||||
}
|
||||
defer func() { _ = root.Close() }()
|
||||
for _, rel := range rels {
|
||||
fi, err := root.Lstat(rel)
|
||||
if err != nil || fi.IsDir() || fi.Mode().IsRegular() || fi.Mode()&os.ModeSymlink != 0 {
|
||||
failed = append(failed, rel)
|
||||
continue
|
||||
}
|
||||
if err := root.Remove(rel); err != nil {
|
||||
failed = append(failed, rel)
|
||||
continue
|
||||
}
|
||||
removed = append(removed, rel)
|
||||
}
|
||||
return removed, failed
|
||||
}
|
||||
|
||||
// removeSymlinks unlinks each brain-relative symlink through an os.Root on
|
||||
// the brain dir, after re-checking it is still a symlink. Removing a symlink
|
||||
// never touches its target. It returns what was removed and what was not.
|
||||
|
||||
@@ -381,18 +381,19 @@ func TestWatchQueueIsBoundedDuringAnOutage(t *testing.T) {
|
||||
require.Eventually(t, func() bool { _, err := os.Lstat(link); return os.IsNotExist(err) }, 2*time.Second, 2*time.Millisecond)
|
||||
}
|
||||
n.down.Store(false)
|
||||
require.Eventually(t, func() bool { return n.count() >= 1 }, 2*time.Second, 5*time.Millisecond)
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
assert.LessOrEqual(t, n.count(), maxPending+1, "queued alerts plus at most one suppression summary")
|
||||
n.mu.Lock()
|
||||
defer n.mu.Unlock()
|
||||
found := false
|
||||
for _, m := range n.msgs {
|
||||
if strings.Contains(m, "suppressed") {
|
||||
found = true
|
||||
// Synchronise on the summary being delivered (hyperguild#193: a fixed
|
||||
// sleep flaked under load); once it is, the queue has been flushed.
|
||||
require.Eventually(t, func() bool {
|
||||
n.mu.Lock()
|
||||
defer n.mu.Unlock()
|
||||
for _, m := range n.msgs {
|
||||
if strings.Contains(m, "suppressed") {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
assert.True(t, found, "suppressed alerts are counted in a summary")
|
||||
return false
|
||||
}, 10*time.Second, 5*time.Millisecond, "suppressed alerts are counted in a summary")
|
||||
assert.LessOrEqual(t, n.count(), maxPending+1, "queued alerts plus at most one suppression summary")
|
||||
}
|
||||
|
||||
// Within the bound, every alert queued during an outage is delivered after
|
||||
@@ -411,11 +412,21 @@ func TestWatchDeliversEveryQueuedAlertAfterAShortOutage(t *testing.T) {
|
||||
require.Eventually(t, func() bool { _, err := os.Lstat(link); return os.IsNotExist(err) }, 2*time.Second, 2*time.Millisecond)
|
||||
}
|
||||
n.down.Store(false)
|
||||
require.Eventually(t, func() bool { return n.count() == 3 }, 2*time.Second, 5*time.Millisecond)
|
||||
require.Eventually(t, func() bool { return n.count() >= 3 }, 10*time.Second, 5*time.Millisecond)
|
||||
n.mu.Lock()
|
||||
defer n.mu.Unlock()
|
||||
for i, name := range []string{"qa", "qb", "qc"} {
|
||||
assert.Contains(t, n.msgs[i], strconv.Quote(name))
|
||||
// Each removal delivered exactly once, in any order: the outage ends
|
||||
// asynchronously, possibly mid-flush, so a later item can send before an
|
||||
// earlier one (hyperguild#193). Order is not what this test is about.
|
||||
assert.Len(t, n.msgs, 3)
|
||||
for _, name := range []string{"qa", "qb", "qc"} {
|
||||
hits := 0
|
||||
for _, m := range n.msgs {
|
||||
if strings.Contains(m, strconv.Quote(name)) {
|
||||
hits++
|
||||
}
|
||||
}
|
||||
assert.Equal(t, 1, hits, "%s delivered exactly once", name)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -457,29 +468,30 @@ func TestWatchAccountsForEveryRemovalDuringAnOutage(t *testing.T) {
|
||||
require.Eventually(t, func() bool { _, err := os.Lstat(link); return os.IsNotExist(err) }, 2*time.Second, 1*time.Millisecond)
|
||||
}
|
||||
n.down.Store(false)
|
||||
require.Eventually(t, func() bool {
|
||||
// Synchronise on the record being complete, not on a fixed sleep after
|
||||
// the first summary (hyperguild#193): sum every "removed N" alert and
|
||||
// every suppression count until they account for all plants.
|
||||
account := func() (delivered, counted int) {
|
||||
n.mu.Lock()
|
||||
defer n.mu.Unlock()
|
||||
for _, m := range n.msgs {
|
||||
if strings.Contains(m, "suppressed") {
|
||||
return true
|
||||
var k int
|
||||
if i := strings.Index(m, "removed "); i >= 0 && strings.Contains(m, "symlink(s)") {
|
||||
if _, err := fmt.Sscanf(m[i+len("removed "):], "%d", &k); err == nil {
|
||||
delivered += k
|
||||
}
|
||||
}
|
||||
if i := strings.Index(m, "): "); i >= 0 && strings.Contains(m, "suppressed") {
|
||||
if _, err := fmt.Sscanf(m[i+3:], "%d", &k); err == nil {
|
||||
counted += k
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}, 2*time.Second, 5*time.Millisecond)
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
n.mu.Lock()
|
||||
defer n.mu.Unlock()
|
||||
delivered, counted := 0, 0
|
||||
for _, m := range n.msgs {
|
||||
if strings.Contains(m, "removed 1 symlink(s)") {
|
||||
delivered++
|
||||
}
|
||||
if i := strings.Index(m, "): "); i >= 0 && strings.Contains(m, "suppressed") {
|
||||
_, _ = fmt.Sscanf(m[i+3:], "%d", &counted)
|
||||
}
|
||||
return delivered, counted
|
||||
}
|
||||
assert.Equal(t, plants, delivered+counted, "delivered %d + counted %d must equal %d removals", delivered, counted, plants)
|
||||
require.Eventually(t, func() bool { d, c := account(); return d+c >= plants }, 10*time.Second, 5*time.Millisecond)
|
||||
d, c := account()
|
||||
assert.Equal(t, plants, d+c, "delivered %d + counted %d must equal %d removals", d, c, plants)
|
||||
}
|
||||
|
||||
// A poison message is still dropped (others send while it fails), and the
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
package watcher
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/wing"
|
||||
)
|
||||
|
||||
// These tests are the sprint's goal (hyperguild#209) stated as behaviour, with
|
||||
// the real pipeline and real file layout and only the model stubbed: a person
|
||||
// drops a clip into raw/ and does nothing else.
|
||||
|
||||
func acceptanceCfg(t *testing.T) (Config, string, *atomic.Int32) {
|
||||
t.Helper()
|
||||
brainDir := t.TempDir()
|
||||
var calls atomic.Int32
|
||||
// Each call extracts a source with its own title, as real clips have: two
|
||||
// sources with one title are deliberately merged into one page by the
|
||||
// pipeline's title resolution.
|
||||
pagesFor := func(n int32) []pipeline.RawPage {
|
||||
return []pipeline.RawPage{
|
||||
{Title: fmt.Sprintf("Dark Factory Talk %d", n), Type: "source", Subtype: "video", Domain: "ai",
|
||||
Content: "## Summary\n\nA talk.\n\n## Entities Mentioned\n\n[[Jev]]\n\n## Concepts Introduced or Reinforced\n\n[[Dark Factory]]\n"},
|
||||
{Title: "Jev", Type: "entity", Subtype: "tool", Content: "## Definition\n\nA decision model.\n\n## Sources\n"},
|
||||
{Title: "Dark Factory", Type: "concept", Content: "## Definition\n\nA fully automated build line.\n\n## Sources\n"},
|
||||
}
|
||||
}
|
||||
return Config{
|
||||
BrainDir: brainDir,
|
||||
Resolver: wing.Router{BrainDir: brainDir, Default: "web-clip", ClientBlock: regexp.MustCompile(`(?i)acme-client`)},
|
||||
Pipeline: pipeline.Config{
|
||||
Schema: "# Schema",
|
||||
Complete: func(context.Context, string, string) (string, error) {
|
||||
return mustJSONPages(pagesFor(calls.Add(1))), nil
|
||||
},
|
||||
},
|
||||
}, brainDir, &calls
|
||||
}
|
||||
|
||||
// topLevel lists the regular files directly in raw/: the inbox.
|
||||
func topLevel(t *testing.T, brainDir string) []string {
|
||||
t.Helper()
|
||||
entries, err := os.ReadDir(filepath.Join(brainDir, "raw"))
|
||||
require.NoError(t, err)
|
||||
var out []string
|
||||
for _, e := range entries {
|
||||
if e.Type().IsRegular() {
|
||||
out = append(out, e.Name())
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func TestAcceptance_ADroppedClipLandsInWebClipAndTheInboxIsEmpty(t *testing.T) {
|
||||
cfg, brainDir, calls := acceptanceCfg(t)
|
||||
put(t, brainDir, "dark-factory-talk.md", "Transcript of a talk about dark factories.")
|
||||
|
||||
require.Empty(t, processDir(context.Background(), cfg, "2026-10-20"))
|
||||
|
||||
assert.True(t, has(brainDir, "wiki/web-clip/sources/dark-factory-talk.md"), "the clip is a page in web-clip")
|
||||
assert.True(t, has(brainDir, "wiki/entities/jev.md"), "its entity is in the shared layer")
|
||||
assert.True(t, has(brainDir, "wiki/concepts/dark-factory.md"), "its concept is in the shared layer")
|
||||
assert.Empty(t, topLevel(t, brainDir), "raw/ holds nothing unprocessed")
|
||||
assert.False(t, has(brainDir, "raw/needs-review"), "and nothing waits for a human")
|
||||
assert.Equal(t, int32(1), calls.Load(), "one extraction, no classification step")
|
||||
}
|
||||
|
||||
func TestAcceptance_TheSameEntityFromTwoClipsIsOnePage(t *testing.T) {
|
||||
cfg, brainDir, _ := acceptanceCfg(t)
|
||||
put(t, brainDir, "talk-one.md", "First talk.")
|
||||
require.Empty(t, processDir(context.Background(), cfg, "2026-10-20"))
|
||||
put(t, brainDir, "talk-two.md", "Second talk, different words.")
|
||||
require.Empty(t, processDir(context.Background(), cfg, "2026-10-20"))
|
||||
|
||||
files, err := filepath.Glob(filepath.Join(brainDir, "wiki", "entities", "*.md"))
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, files, 1)
|
||||
b, _ := os.ReadFile(filepath.Join(brainDir, "wiki/entities/jev.md"))
|
||||
assert.True(t, strings.Contains(string(b), "talk-one") && strings.Contains(string(b), "talk-two"), "both sources are cited")
|
||||
}
|
||||
|
||||
func TestAcceptance_AClipThatLooksLikeClientMaterialIsParkedNotFiled(t *testing.T) {
|
||||
cfg, brainDir, calls := acceptanceCfg(t)
|
||||
put(t, brainDir, "kickoff.md", "Notes from the ACME-CLIENT kickoff call.")
|
||||
|
||||
require.Empty(t, processDir(context.Background(), cfg, "2026-10-20"))
|
||||
|
||||
assert.Zero(t, calls.Load(), "not even extracted")
|
||||
assert.False(t, has(brainDir, "wiki/web-clip"), "never filed into a shared wing")
|
||||
assert.True(t, has(brainDir, "raw/needs-review/kickoff.md"), "waits for the owner")
|
||||
assert.Empty(t, topLevel(t, brainDir))
|
||||
}
|
||||
|
||||
func TestAcceptance_ASubfolderNamingAnExistingWingPlacesThereAndATypoParks(t *testing.T) {
|
||||
cfg, brainDir, _ := acceptanceCfg(t)
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(brainDir, "wiki", "homelab", "facts"), 0o755))
|
||||
put(t, brainDir, "homelab/ops-note.md", "A note about the cluster.")
|
||||
put(t, brainDir, "hoemlab/typo-note.md", "A misfiled note.")
|
||||
|
||||
require.Empty(t, processDir(context.Background(), cfg, "2026-10-20"))
|
||||
|
||||
assert.True(t, has(brainDir, "wiki/homelab/sources/ops-note.md"))
|
||||
assert.False(t, has(brainDir, "wiki/hoemlab"), "a typo never creates a wing")
|
||||
assert.True(t, has(brainDir, "raw/needs-review/typo-note.md"))
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
package watcher
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
|
||||
)
|
||||
|
||||
// Kept out of watcher.go on purpose: that file is covered by the raw-reader
|
||||
// guard (internal/extract/callsites_test.go), which forbids any .Open selector
|
||||
// there. This function reads nothing from raw/; it creates a file under the
|
||||
// brain root, which refuses symlinked directories and never follows a final
|
||||
// symlink.
|
||||
|
||||
// createParked writes data to raw/needs-review/<filename> through the brain
|
||||
// root, never replacing a parked note: on a name clash it tries -2, -3, ...
|
||||
// A needs-review that is a symlink, or any non-directory, is refused.
|
||||
func createParked(brainDir, filename string, data []byte) error {
|
||||
return createUnique(brainDir, "raw/needs-review", filename, data)
|
||||
}
|
||||
|
||||
// createUnique writes data to dirRel/filename through the brain root, never
|
||||
// replacing an existing file: on a name clash it tries -2, -3, ... A dirRel that
|
||||
// is a symlink, or any non-directory, is refused.
|
||||
func createUnique(brainDir, dirRel, filename string, data []byte) error {
|
||||
bfs, err := brainfs.Open(brainDir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() { _ = bfs.Close() }()
|
||||
ext := filepath.Ext(filename)
|
||||
stem := strings.TrimSuffix(filename, ext)
|
||||
for n := 1; n < 10000; n++ {
|
||||
name := filename
|
||||
if n > 1 {
|
||||
name = fmt.Sprintf("%s-%d%s", stem, n, ext)
|
||||
}
|
||||
err := bfs.CreateRegular(dirRel+"/"+name, data)
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
if !errors.Is(err, fs.ErrExist) {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("no free name for %q in %s", filename, dirRel)
|
||||
}
|
||||
@@ -3,17 +3,22 @@ package watcher
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
"unicode"
|
||||
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/extract"
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/wing"
|
||||
)
|
||||
|
||||
// Config holds watcher configuration.
|
||||
@@ -21,8 +26,27 @@ type Config struct {
|
||||
BrainDir string
|
||||
Interval time.Duration
|
||||
Pipeline pipeline.Config
|
||||
|
||||
// MaxNoteBytes refuses a note larger than this before anything is read into
|
||||
// memory (hyperguild#198). 0 means defaultMaxNoteBytes.
|
||||
MaxNoteBytes int64
|
||||
|
||||
// Resolver, when set, decides each note's wing (hyperguild#205, #208): a
|
||||
// deliberate subfolder/frontmatter wing, else the default wing. A note it
|
||||
// cannot place is parked in raw/needs-review/ for a human. Nil keeps the
|
||||
// retired layout. Set it only to a non-nil value: a nil pointer inside the
|
||||
// interface is not nil.
|
||||
Resolver wing.Placer
|
||||
}
|
||||
|
||||
// defaultMaxNoteBytes bounds a single note: extraction and the archive copy both
|
||||
// hold it in memory.
|
||||
const defaultMaxNoteBytes = 64 << 20
|
||||
|
||||
// errTooLarge marks a note refused for its size; like a refused write it is
|
||||
// reported once and not retried until the note changes.
|
||||
var errTooLarge = extract.ErrTooLarge
|
||||
|
||||
// Start launches the watcher in a background goroutine.
|
||||
// It returns immediately. The watcher stops when ctx is cancelled.
|
||||
func Start(ctx context.Context, cfg Config) {
|
||||
@@ -85,7 +109,7 @@ func processDir(ctx context.Context, cfg Config, date string) []error {
|
||||
// path that treats raw/ as "awaiting processing" — this is that path.
|
||||
if d.IsDir() {
|
||||
name := d.Name()
|
||||
if name == "processed" || name == "failed" || name == "rejected" {
|
||||
if name == "processed" || name == "failed" || name == "rejected" || name == "needs-review" {
|
||||
return filepath.SkipDir
|
||||
}
|
||||
return nil
|
||||
@@ -109,14 +133,33 @@ func processDir(ctx context.Context, cfg Config, date string) []error {
|
||||
}
|
||||
|
||||
// Skip files that have already been processed or permanently failed.
|
||||
if _, err := os.Stat(path + ".processed"); err == nil {
|
||||
// Any entry at the marker path counts (Lstat, not Stat): a dangling
|
||||
// marker symlink made os.Stat report nothing and the note was
|
||||
// reprocessed every interval (hyperguild#158).
|
||||
if skip, stale := markerBlocks(path); skip {
|
||||
warnMarkerOnce(path)
|
||||
return nil
|
||||
} else {
|
||||
for _, m := range stale {
|
||||
slog.Warn("watcher: ignoring a stale marker older than its note", "marker", m)
|
||||
if rmErr := root.Remove(relOf(rawDir, m)); rmErr != nil {
|
||||
slog.Error("watcher: failed to remove a stale marker", "marker", m, "error", rmErr)
|
||||
}
|
||||
}
|
||||
}
|
||||
if _, err := os.Stat(path + ".failed"); err == nil {
|
||||
// A note whose archive copy or marker was refused is not retried
|
||||
// until it changes: a refusal has no marker to stop the next round,
|
||||
// and each round can cost an LLM call (hyperguild#158).
|
||||
info, ierr := d.Info()
|
||||
if ierr == nil && refusedUnchanged(path, info) {
|
||||
return nil
|
||||
}
|
||||
|
||||
if err := processFile(ctx, cfg, root, rawDir, path, date); err != nil {
|
||||
if ierr == nil && (errors.Is(err, brainfs.ErrNotRegular) || errors.Is(err, errTooLarge)) {
|
||||
rememberRefused(path, info)
|
||||
slog.Error("watcher: note refused; not retried until it changes", "file", filepath.Base(path), "error", err)
|
||||
}
|
||||
errs = append(errs, fmt.Errorf("process %s: %w", filepath.Base(path), err))
|
||||
}
|
||||
return nil
|
||||
@@ -127,9 +170,12 @@ func processDir(ctx context.Context, cfg Config, date string) []error {
|
||||
return errs
|
||||
}
|
||||
|
||||
// processFile reads a file, calls pipeline.Run, copies it to processed/ or failed/,
|
||||
// and writes a marker file next to the original so the watcher skips it next poll.
|
||||
// The original file is never deleted, keeping Syncthing-connected vaults (e.g. Obsidian) intact.
|
||||
// processFile reads a file, calls pipeline.Run, and archives it into processed/
|
||||
// or failed/. The original is then removed from raw/ so that raw/ only ever
|
||||
// holds unprocessed input (inbox zero). The removal syncs to Syncthing peers;
|
||||
// the archive copy under raw/processed/ or raw/failed/ is what they keep. A
|
||||
// marker is written next to the original only if the removal fails, so a
|
||||
// stuck file is not re-ingested every poll.
|
||||
func processFile(ctx context.Context, cfg Config, root *os.Root, rawDir, path, date string) error {
|
||||
filename := filepath.Base(path)
|
||||
source := deriveSource(filename)
|
||||
@@ -138,25 +184,43 @@ func processFile(ctx context.Context, cfg Config, root *os.Root, rawDir, path, d
|
||||
return fmt.Errorf("relative path: %w", err)
|
||||
}
|
||||
|
||||
content, err := extract.TextIn(root, rel)
|
||||
before, err := root.Lstat(rel)
|
||||
if err != nil {
|
||||
return fmt.Errorf("stat before ingest: %w", err)
|
||||
}
|
||||
limit := cfg.MaxNoteBytes
|
||||
if limit <= 0 {
|
||||
limit = defaultMaxNoteBytes
|
||||
}
|
||||
if before.Size() > limit {
|
||||
return fmt.Errorf("%s is %d bytes, over the %d byte cap: %w", filename, before.Size(), limit, errTooLarge)
|
||||
}
|
||||
content, err := extract.TextInMax(root, rel, limit)
|
||||
if err != nil {
|
||||
return fmt.Errorf("extract text: %w", err)
|
||||
}
|
||||
|
||||
_, runErr := pipeline.Run(ctx, cfg.Pipeline, cfg.BrainDir, content, source, false)
|
||||
if runErr != nil {
|
||||
// Copy to failed/ and leave a .failed marker so we don't retry.
|
||||
failedDir := filepath.Join(cfg.BrainDir, "raw", "failed")
|
||||
if mkErr := os.MkdirAll(failedDir, 0o755); mkErr != nil {
|
||||
return fmt.Errorf("mkdir failed dir: %w", mkErr)
|
||||
pcfg := cfg.Pipeline
|
||||
if cfg.Resolver != nil {
|
||||
d := cfg.Resolver.Resolve(ctx, wing.Note{Dir: topDir(rel), FrontmatterWing: frontmatterWing(content), Text: content})
|
||||
if d.Source == wing.Exception {
|
||||
return parkForReview(cfg, root, rel, path, filename, content, d, date, before)
|
||||
}
|
||||
dest := filepath.Join(failedDir, filename)
|
||||
if cpErr := copyFromRoot(root, rel, dest); cpErr != nil {
|
||||
pcfg.Wing = d.Wing
|
||||
slog.Info("watcher: wing resolved", "file", filename, "wing", d.Wing, "how", d.Source, "reason", d.Reason)
|
||||
}
|
||||
_, runErr := pipeline.Run(ctx, pcfg, cfg.BrainDir, content, source, false)
|
||||
if runErr != nil {
|
||||
// Copy to failed/ and leave a .failed marker so we don't retry. Both
|
||||
// written through the brain root (hyperguild#158).
|
||||
if cpErr := copyFromRoot(root, rel, cfg.BrainDir, "raw/failed", filename, limit); cpErr != nil {
|
||||
return fmt.Errorf("copy to failed: %w", cpErr)
|
||||
}
|
||||
if mkErr := os.WriteFile(path+".failed", []byte(runErr.Error()), 0o644); mkErr != nil {
|
||||
slog.Error("watcher: failed to write .failed marker", "error", mkErr)
|
||||
if changedSince(root, rel, before) {
|
||||
slog.Warn("watcher: note changed during ingest; leaving it in raw/ for the next poll", "file", filename)
|
||||
return nil
|
||||
}
|
||||
retire(cfg, root, rel, path, ".failed", runErr.Error(), before)
|
||||
|
||||
slog.Warn("watcher: file failed", "file", filename, "error", runErr)
|
||||
|
||||
@@ -168,28 +232,75 @@ func processFile(ctx context.Context, cfg Config, root *os.Root, rawDir, path, d
|
||||
}
|
||||
|
||||
// Copy to processed/YYYY-MM-DD/ and leave a .processed marker so we don't re-ingest.
|
||||
processedDir := filepath.Join(cfg.BrainDir, "raw", "processed", date)
|
||||
if err := os.MkdirAll(processedDir, 0o755); err != nil {
|
||||
return fmt.Errorf("mkdir processed dir: %w", err)
|
||||
}
|
||||
dest := filepath.Join(processedDir, filename)
|
||||
if _, err := os.Stat(dest); err == nil {
|
||||
// Archive copy already exists; append timestamp to avoid overwriting.
|
||||
ext := filepath.Ext(filename)
|
||||
base := strings.TrimSuffix(filename, ext)
|
||||
dest = filepath.Join(processedDir, base+"-"+time.Now().UTC().Format("150405")+ext)
|
||||
}
|
||||
if err := copyFromRoot(root, rel, dest); err != nil {
|
||||
// Written through the brain root (hyperguild#158).
|
||||
if err := copyFromRoot(root, rel, cfg.BrainDir, "raw/processed/"+date, filename, limit); err != nil {
|
||||
return fmt.Errorf("copy to processed: %w", err)
|
||||
}
|
||||
if err := os.WriteFile(path+".processed", []byte(date), 0o644); err != nil {
|
||||
slog.Error("watcher: failed to write .processed marker", "error", err)
|
||||
if changedSince(root, rel, before) {
|
||||
slog.Warn("watcher: note changed during ingest; leaving it in raw/ for the next poll", "file", filename)
|
||||
return nil
|
||||
}
|
||||
retire(cfg, root, rel, path, ".processed", date, before)
|
||||
|
||||
slog.Info("watcher: file processed", "file", filename, "source", source)
|
||||
return nil
|
||||
}
|
||||
|
||||
// changedSince reports whether the note at rel differs (size or mtime) from
|
||||
// before. A note replaced mid-ingest (Syncthing) was extracted from older
|
||||
// content than it now holds, so it must not be removed. Limits: a same-size
|
||||
// rewrite with a restored mtime is not detected (the new bytes then survive
|
||||
// only in the archive copy), nor a change between this check and Remove.
|
||||
func changedSince(root *os.Root, rel string, before os.FileInfo) bool {
|
||||
now, err := root.Lstat(rel)
|
||||
if err != nil {
|
||||
return true
|
||||
}
|
||||
return now.Size() != before.Size() || !now.ModTime().Equal(before.ModTime())
|
||||
}
|
||||
|
||||
// retire removes an archived original from raw/ (inbox zero). If the removal
|
||||
// fails it falls back to a marker written through the brain root (#158), and if
|
||||
// even that fails the note is remembered, with the size and mtime it had, so it
|
||||
// is not ingested again until it changes.
|
||||
func retire(cfg Config, root *os.Root, rel, path, markerSuffix, markerBody string, before os.FileInfo) {
|
||||
rmErr := root.Remove(rel)
|
||||
if rmErr == nil {
|
||||
return
|
||||
}
|
||||
slog.Error("watcher: failed to remove archived original; writing marker", "file", rel, "error", rmErr)
|
||||
if err := putBrainFile(cfg.BrainDir, "raw/"+filepath.ToSlash(rel)+markerSuffix, []byte(markerBody)); err != nil {
|
||||
slog.Error("watcher: failed to write marker; skipping this note until it changes", "file", rel, "error", err)
|
||||
rememberRefused(path, before)
|
||||
}
|
||||
}
|
||||
|
||||
// refused remembers notes whose archive copy or marker write was refused,
|
||||
// by path, with the size and mtime they had, for the life of the process.
|
||||
var refused sync.Map
|
||||
|
||||
type refusedStamp struct {
|
||||
size int64
|
||||
mtime time.Time
|
||||
}
|
||||
|
||||
func rememberRefused(path string, info os.FileInfo) {
|
||||
refused.Store(path, refusedStamp{size: info.Size(), mtime: info.ModTime()})
|
||||
}
|
||||
|
||||
func refusedUnchanged(path string, info os.FileInfo) bool {
|
||||
v, ok := refused.Load(path)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
st := v.(refusedStamp)
|
||||
if st.size == info.Size() && st.mtime.Equal(info.ModTime()) {
|
||||
return true
|
||||
}
|
||||
refused.Delete(path)
|
||||
return false
|
||||
}
|
||||
|
||||
// deriveSource turns a filename into a human-readable source name.
|
||||
// "shape-up-book.md" → "Shape Up Book"
|
||||
func deriveSource(filename string) string {
|
||||
@@ -214,42 +325,231 @@ func appendWatcherLog(brainDir, filename string, runErr error, date string) erro
|
||||
entry := fmt.Sprintf("## %s — watcher error\n\n- **File:** %s\n- **Error:** %s\n\n",
|
||||
date, filename, runErr.Error())
|
||||
|
||||
logPath := filepath.Join(brainDir, "log.md")
|
||||
f, err := os.OpenFile(logPath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
|
||||
if err != nil {
|
||||
return fmt.Errorf("open log: %w", err)
|
||||
}
|
||||
|
||||
if _, err = f.WriteString(entry); err != nil {
|
||||
f.Close() //nolint:errcheck
|
||||
if err := brainfs.AppendFile(brainDir, "log.md", []byte(entry)); err != nil {
|
||||
return fmt.Errorf("write log: %w", err)
|
||||
}
|
||||
return f.Close()
|
||||
return nil
|
||||
}
|
||||
|
||||
// copyFromRoot archives a raw/ file read through the root, refusing anything
|
||||
// that is not a regular file, so a swapped-in symlink cannot copy an outside
|
||||
// file into raw/processed/ or raw/failed/ (hyperguild#153).
|
||||
func copyFromRoot(root *os.Root, rel, dst string) error {
|
||||
fi, err := root.Lstat(rel)
|
||||
// putBrainFile creates or replaces a brain-relative file through the brain
|
||||
// root, never through a symlinked directory or file (hyperguild#158).
|
||||
func putBrainFile(brainDir, rel string, data []byte) error {
|
||||
return brainfs.PutFile(brainDir, rel, data)
|
||||
}
|
||||
|
||||
// copyFromRoot archives a raw/ file read through the root into dstDir under a
|
||||
// name no existing file has (so a note never replaces another's copy: the copy
|
||||
// is the only one left once the original is retired), refusing anything that is
|
||||
// not a regular file and anything over max bytes at the read, not just at the
|
||||
// earlier Lstat (hyperguild#158, #198).
|
||||
func copyFromRoot(root *os.Root, rel, brainDir, dstDir, filename string, max int64) error {
|
||||
data, err := readNoteMax(root, rel, max)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return createUnique(brainDir, dstDir, filename, data)
|
||||
}
|
||||
|
||||
// readNoteMax reads a raw/ note through the root, at most max bytes.
|
||||
func readNoteMax(root *os.Root, rel string, max int64) ([]byte, error) {
|
||||
fi, err := root.Lstat(rel)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !fi.Mode().IsRegular() {
|
||||
return fmt.Errorf("%s: not a regular file", rel)
|
||||
return nil, fmt.Errorf("%s: not a regular file", rel)
|
||||
}
|
||||
in, _, err := extract.OpenRegular(root, rel)
|
||||
if err != nil {
|
||||
return err
|
||||
return nil, err
|
||||
}
|
||||
defer func() { _ = in.Close() }()
|
||||
out, err := os.Create(dst)
|
||||
if err != nil {
|
||||
return err
|
||||
data, err := readAllMax(in, max)
|
||||
if errors.Is(err, errTooLarge) {
|
||||
return nil, fmt.Errorf("%s: %w (%d bytes)", rel, errTooLarge, max)
|
||||
}
|
||||
if _, err := io.Copy(out, in); err != nil {
|
||||
_ = out.Close()
|
||||
return err
|
||||
return data, err
|
||||
}
|
||||
|
||||
// readAllMax reads r but never more than max+1 bytes, so an oversize or growing
|
||||
// file costs at most that much memory, and reports errTooLarge past max.
|
||||
func readAllMax(r io.Reader, max int64) ([]byte, error) {
|
||||
data, err := io.ReadAll(io.LimitReader(r, max+1))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out.Close()
|
||||
if int64(len(data)) > max {
|
||||
return nil, errTooLarge
|
||||
}
|
||||
return data, nil
|
||||
}
|
||||
|
||||
// topDir is the first directory of a raw/-relative path, "" for a top-level file.
|
||||
func topDir(rel string) string {
|
||||
if i := strings.IndexByte(filepath.ToSlash(rel), '/'); i > 0 {
|
||||
return filepath.ToSlash(rel)[:i]
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// frontmatterWing reads a top-level `wing:` from a note's leading frontmatter,
|
||||
// "" if there is none. The block counts only if it has a real closing fence (a
|
||||
// line that is exactly "---"), and an indented `wing:` belongs to some other
|
||||
// mapping, not to the note.
|
||||
func frontmatterWing(content string) string {
|
||||
content = strings.TrimPrefix(content, "\ufeff") // a byte-order mark must not hide the block
|
||||
content = strings.ReplaceAll(content, "\r\n", "\n")
|
||||
if !strings.HasPrefix(content, "---\n") {
|
||||
return ""
|
||||
}
|
||||
block, _, found := cutFence(content[4:])
|
||||
if !found {
|
||||
return ""
|
||||
}
|
||||
for _, line := range strings.Split(block, "\n") {
|
||||
if line == "" || line[0] == ' ' || line[0] == '\t' {
|
||||
continue
|
||||
}
|
||||
if key, val, ok := strings.Cut(line, ":"); ok && strings.TrimSpace(key) == "wing" {
|
||||
return strings.Trim(strings.TrimSpace(val), `"'`)
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// parkForReview moves a note the resolver could not place into
|
||||
// raw/needs-review/, recording the proposal in its frontmatter so a human can
|
||||
// approve it by moving it into raw/<wing>/. Nothing is extracted or written to
|
||||
// the wiki.
|
||||
//
|
||||
// The parked copy is created through the brain root (real directories only,
|
||||
// O_EXCL, never overwriting) and is the only copy once the original is retired,
|
||||
// so the original is removed only after that write succeeds and only if it has
|
||||
// not changed since it was read: a newer version delivered while the resolver
|
||||
// ran stays in raw/ for the next poll.
|
||||
func parkForReview(cfg Config, root *os.Root, rel, path, filename, content string, d wing.Decision, date string, before os.FileInfo) error {
|
||||
var data []byte
|
||||
if strings.ToLower(filepath.Ext(filename)) == ".pdf" {
|
||||
// Binary: park the bytes untouched; the proposal is in log.md only.
|
||||
var err error
|
||||
data, err = readNoteMax(root, rel, defaultMaxNoteBytes)
|
||||
if err != nil {
|
||||
return fmt.Errorf("park pdf: %w", err)
|
||||
}
|
||||
} else {
|
||||
data = []byte(withProposal(content, d))
|
||||
}
|
||||
if err := createParked(cfg.BrainDir, filename, data); err != nil {
|
||||
return fmt.Errorf("park note: %w", err)
|
||||
}
|
||||
if err := appendWatcherLog(cfg.BrainDir, filename, fmt.Errorf("needs review (proposed wing %q): %s", d.Wing, d.Reason), date); err != nil {
|
||||
slog.Error("watcher: failed to write log entry", "error", err)
|
||||
}
|
||||
slog.Info("watcher: parked for review", "file", filename, "proposed_wing", d.Wing, "reason", d.Reason)
|
||||
if changedSince(root, rel, before) {
|
||||
slog.Warn("watcher: note changed while it was being resolved; leaving the newer version in raw/", "file", filename)
|
||||
return nil
|
||||
}
|
||||
retire(cfg, root, rel, path, ".processed", date, before)
|
||||
return nil
|
||||
}
|
||||
|
||||
// proposalKeys are the frontmatter keys the resolver owns in a parked note.
|
||||
var proposalKeys = map[string]bool{"proposed_wing": true, "proposal_confidence": true, "proposal_reason": true}
|
||||
|
||||
// withProposal adds the resolver's proposal to a note's frontmatter, creating
|
||||
// the block when the note has none. Every value is on one line: the wing only
|
||||
// if it is a plain wing name (else it appears in the reason alone), strings
|
||||
// Go-quoted. Any proposal keys the note already carried are dropped first, so a
|
||||
// note cannot pre-empt the resolver's (frontmatter reads are first-wins).
|
||||
// Line endings are normalised to LF so a CRLF note keeps one valid block.
|
||||
func withProposal(content string, d wing.Decision) string {
|
||||
content = strings.ReplaceAll(content, "\r\n", "\n")
|
||||
reason := d.Reason
|
||||
var kv strings.Builder
|
||||
if d.Wing != "" && brain.Sanitise(d.Wing) == d.Wing {
|
||||
fmt.Fprintf(&kv, "proposed_wing: %s\n", d.Wing)
|
||||
} else if d.Wing != "" {
|
||||
reason = fmt.Sprintf("proposed wing %q is not a plain wing name; %s", d.Wing, reason)
|
||||
}
|
||||
fmt.Fprintf(&kv, "proposal_confidence: %.2f\nproposal_reason: %q\n", d.Confidence, reason)
|
||||
|
||||
if !strings.HasPrefix(content, "---\n") {
|
||||
return "---\n" + kv.String() + "---\n" + content
|
||||
}
|
||||
block, rest, found := cutFence(content[4:])
|
||||
if !found {
|
||||
return "---\n" + kv.String() + "---\n" + content
|
||||
}
|
||||
var kept []string
|
||||
for _, line := range strings.Split(block, "\n") {
|
||||
if key, _, ok := strings.Cut(line, ":"); ok && proposalKeys[strings.TrimSpace(key)] && !strings.HasPrefix(line, " ") {
|
||||
continue
|
||||
}
|
||||
kept = append(kept, line)
|
||||
}
|
||||
return "---\n" + kv.String() + strings.Join(kept, "\n") + "\n---" + rest
|
||||
}
|
||||
|
||||
// cutFence splits s at the first closing fence: a line that is exactly "---".
|
||||
// "----" or "---x" is not a fence, matching how the frontmatter readers parse.
|
||||
func cutFence(s string) (block, rest string, found bool) {
|
||||
if i := strings.Index(s, "\n---\n"); i >= 0 {
|
||||
return s[:i], s[i+4:], true
|
||||
}
|
||||
if strings.HasSuffix(s, "\n---") {
|
||||
return s[:len(s)-4], "", true
|
||||
}
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
// markerBlocks reports whether a marker next to path means "already handled".
|
||||
// Any non-regular entry at the marker path counts (a dangling symlink must not
|
||||
// be written through, hyperguild#158). A regular marker describes an EARLIER
|
||||
// note of the same name, so it does not block this one, and is returned in
|
||||
// stale, when it is older than the note or dated implausibly far in the future
|
||||
// (clock skew, a peer): those are removed by the caller. Heuristic by nature:
|
||||
// a marker at least as new as the note still means "done".
|
||||
func markerBlocks(path string) (skip bool, stale []string) {
|
||||
note, err := os.Lstat(path)
|
||||
if err != nil {
|
||||
return false, nil
|
||||
}
|
||||
for _, suffix := range []string{".processed", ".failed"} {
|
||||
m, err := os.Lstat(path + suffix)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if !m.Mode().IsRegular() {
|
||||
return true, nil
|
||||
}
|
||||
if m.ModTime().Before(note.ModTime()) || m.ModTime().After(time.Now().Add(markerSkew)) {
|
||||
stale = append(stale, path+suffix)
|
||||
continue
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
return false, stale
|
||||
}
|
||||
|
||||
// markerSkew is how far ahead of now a marker may be dated before it is not
|
||||
// believed.
|
||||
const markerSkew = 10 * time.Minute
|
||||
|
||||
// markerWarned remembers notes already reported as held by a marker, so the
|
||||
// 30-second poll says it once instead of forever.
|
||||
var markerWarned sync.Map
|
||||
|
||||
func warnMarkerOnce(path string) {
|
||||
if _, seen := markerWarned.LoadOrStore(path, struct{}{}); !seen {
|
||||
slog.Warn("watcher: note is held by a marker newer than itself and will not be processed; delete the marker or edit the note", "file", path)
|
||||
}
|
||||
}
|
||||
|
||||
func relOf(rawDir, path string) string {
|
||||
rel, err := filepath.Rel(rawDir, path)
|
||||
if err != nil {
|
||||
return filepath.Base(path)
|
||||
}
|
||||
return rel
|
||||
}
|
||||
|
||||
@@ -4,7 +4,9 @@ package watcher
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
@@ -89,13 +91,12 @@ func TestStart_ProcessesFile(t *testing.T) {
|
||||
}
|
||||
require.True(t, found, "file should be copied to processed/")
|
||||
|
||||
// Original file should still exist (copy, not move — keeps Obsidian vault intact).
|
||||
// Inbox zero: once archived, the original leaves raw/ and no marker is
|
||||
// left behind, so what remains in raw/ is exactly the unprocessed set.
|
||||
_, err := os.Stat(rawFile)
|
||||
assert.NoError(t, err, "original file should remain in raw/")
|
||||
|
||||
// A .processed marker should exist next to the original.
|
||||
assert.ErrorIs(t, err, os.ErrNotExist, "original should be removed from raw/ once archived")
|
||||
_, err = os.Stat(rawFile + ".processed")
|
||||
assert.NoError(t, err, ".processed marker should be written")
|
||||
assert.ErrorIs(t, err, os.ErrNotExist, "no .processed marker: the archive copy is the record")
|
||||
|
||||
// Wiki page should exist.
|
||||
wikiPath := filepath.Join(brainDir, "wiki", "sources", "shape-up-book.md")
|
||||
@@ -142,13 +143,11 @@ func TestStart_MovesToFailedOnError(t *testing.T) {
|
||||
}
|
||||
require.True(t, found, "file should be copied to failed/")
|
||||
|
||||
// Original file should still exist (copy, not move — keeps Obsidian vault intact).
|
||||
// Inbox zero: a quarantined original leaves raw/ too, no marker beside it.
|
||||
_, err := os.Stat(rawFile)
|
||||
assert.NoError(t, err, "original file should remain in raw/")
|
||||
|
||||
// A .failed marker should exist next to the original.
|
||||
assert.ErrorIs(t, err, os.ErrNotExist, "original should be removed from raw/ once quarantined")
|
||||
_, err = os.Stat(rawFile + ".failed")
|
||||
assert.NoError(t, err, ".failed marker should be written")
|
||||
assert.ErrorIs(t, err, os.ErrNotExist, "no .failed marker: failed/ is the record")
|
||||
|
||||
// log.md should contain a watcher error entry.
|
||||
logContent, err := os.ReadFile(filepath.Join(brainDir, "log.md"))
|
||||
@@ -390,3 +389,324 @@ func TestProcessDir_DoesNotHangUnderAFIFOSwapRace(t *testing.T) {
|
||||
t.Fatalf("the watcher hung on a swapped-in FIFO after %d iterations", iters.Load())
|
||||
}
|
||||
}
|
||||
|
||||
// A note whose removal AND marker write both fail (read-only raw/) must not be
|
||||
// re-ingested on every poll: each pass would write another archive copy.
|
||||
func TestProcessDir_DoesNotReingestANoteItCannotRetire(t *testing.T) {
|
||||
brainDir := setupBrainDir(t)
|
||||
rawDir := filepath.Join(brainDir, "raw")
|
||||
require.NoError(t, os.WriteFile(filepath.Join(rawDir, "stuck.md"), []byte("Content."), 0o644))
|
||||
// processed/ must exist and stay writable; only raw/ itself is locked.
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(rawDir, "processed"), 0o755))
|
||||
require.NoError(t, os.Chmod(rawDir, 0o555))
|
||||
t.Cleanup(func() { _ = os.Chmod(rawDir, 0o755) })
|
||||
|
||||
var calls atomic.Int32
|
||||
cfg := Config{
|
||||
BrainDir: brainDir,
|
||||
Pipeline: pipeline.Config{
|
||||
Complete: func(ctx context.Context, s, u string) (string, error) {
|
||||
calls.Add(1)
|
||||
return successComplete(pipeline.RawPage{Title: "Stuck", Type: "source", Subtype: "article", Content: "x\n"})(ctx, s, u)
|
||||
},
|
||||
Schema: "# Schema",
|
||||
},
|
||||
}
|
||||
for i := 0; i < 3; i++ {
|
||||
processDir(context.Background(), cfg, "2026-10-19")
|
||||
}
|
||||
// The LLM count alone proves nothing: pipeline.Run's source-hash check skips
|
||||
// extraction on an unchanged note, so a re-run is invisible there. What a
|
||||
// re-run does leave behind is another archive copy, every poll.
|
||||
archived, err := filepath.Glob(filepath.Join(rawDir, "processed", "2026-10-19", "*"))
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, archived, 1, "a note that cannot be retired is archived once, not once per poll")
|
||||
assert.Equal(t, int32(1), calls.Load())
|
||||
}
|
||||
|
||||
// If the archive copy cannot be written, the original is the only copy left:
|
||||
// it must stay in raw/.
|
||||
func TestProcessDir_KeepsTheOriginalWhenTheArchiveCopyFails(t *testing.T) {
|
||||
brainDir := setupBrainDir(t)
|
||||
rawDir := filepath.Join(brainDir, "raw")
|
||||
note := filepath.Join(rawDir, "keep-me.md")
|
||||
require.NoError(t, os.WriteFile(note, []byte("Content."), 0o644))
|
||||
// The dated archive dir exists but is read-only: MkdirAll succeeds, the
|
||||
// copy itself (os.Create) fails.
|
||||
dated := filepath.Join(rawDir, "processed", "2026-10-19")
|
||||
require.NoError(t, os.MkdirAll(dated, 0o755))
|
||||
require.NoError(t, os.Chmod(dated, 0o555))
|
||||
t.Cleanup(func() { _ = os.Chmod(dated, 0o755) })
|
||||
|
||||
cfg := Config{
|
||||
BrainDir: brainDir,
|
||||
Pipeline: pipeline.Config{
|
||||
Complete: successComplete(pipeline.RawPage{Title: "Keep Me", Type: "source", Subtype: "article", Content: "x\n"}),
|
||||
Schema: "# Schema",
|
||||
},
|
||||
}
|
||||
errs := processDir(context.Background(), cfg, "2026-10-19")
|
||||
|
||||
assert.NotEmpty(t, errs, "the failed archive is reported")
|
||||
_, err := os.Stat(note)
|
||||
assert.NoError(t, err, "original stays in raw/ when it could not be archived")
|
||||
}
|
||||
|
||||
// If the note is replaced (e.g. by Syncthing) while it is being ingested, the
|
||||
// wiki page was extracted from the old content; removing the original would
|
||||
// leave the new content only in the archive and never re-ingest it. Observes
|
||||
// size/mtime change only: a same-size rewrite with a restored mtime is not caught.
|
||||
func TestProcessDir_KeepsANoteThatChangedWhileItWasBeingIngested(t *testing.T) {
|
||||
brainDir := setupBrainDir(t)
|
||||
note := filepath.Join(brainDir, "raw", "edited.md")
|
||||
require.NoError(t, os.WriteFile(note, []byte("old"), 0o644))
|
||||
|
||||
cfg := Config{
|
||||
BrainDir: brainDir,
|
||||
Pipeline: pipeline.Config{
|
||||
Complete: func(ctx context.Context, s, u string) (string, error) {
|
||||
require.NoError(t, os.WriteFile(note, []byte("newer and longer content"), 0o644))
|
||||
return successComplete(pipeline.RawPage{Title: "Edited", Type: "source", Subtype: "article", Content: "x\n"})(ctx, s, u)
|
||||
},
|
||||
Schema: "# Schema",
|
||||
},
|
||||
}
|
||||
processDir(context.Background(), cfg, "2026-10-19")
|
||||
|
||||
got, err := os.ReadFile(note)
|
||||
require.NoError(t, err, "a note that changed during ingest must stay in raw/")
|
||||
assert.Equal(t, "newer and longer content", string(got))
|
||||
}
|
||||
|
||||
// hyperguild#198: an oversized note must not be read into memory. It is left in
|
||||
// raw/ (visible, not lost), reported once, and not retried until it changes.
|
||||
func TestProcessDir_ANoteOverTheSizeCapIsRefusedOnceAndKept(t *testing.T) {
|
||||
brainDir := setupBrainDir(t)
|
||||
big := filepath.Join(brainDir, "raw", "big.md")
|
||||
require.NoError(t, os.WriteFile(big, []byte(strings.Repeat("x", 2048)), 0o644))
|
||||
var calls atomic.Int32
|
||||
cfg := Config{
|
||||
BrainDir: brainDir,
|
||||
MaxNoteBytes: 1024,
|
||||
Pipeline: pipeline.Config{
|
||||
Complete: func(ctx context.Context, s, u string) (string, error) {
|
||||
calls.Add(1)
|
||||
return successComplete(pipeline.RawPage{Title: "Big", Type: "source", Content: "x\n"})(ctx, s, u)
|
||||
},
|
||||
Schema: "# Schema",
|
||||
},
|
||||
}
|
||||
|
||||
first := processDir(context.Background(), cfg, "2026-10-20")
|
||||
require.NotEmpty(t, first, "the refusal is reported")
|
||||
assert.Contains(t, first[0].Error(), "over the 1024 byte cap")
|
||||
second := processDir(context.Background(), cfg, "2026-10-20")
|
||||
assert.Empty(t, second, "and reported once, not every poll")
|
||||
|
||||
assert.Zero(t, calls.Load(), "no extraction")
|
||||
_, err := os.Stat(big)
|
||||
assert.NoError(t, err, "the note stays in raw/")
|
||||
entries, _ := os.ReadDir(filepath.Join(brainDir, "raw", "processed"))
|
||||
assert.Empty(t, entries, "nothing was archived")
|
||||
}
|
||||
|
||||
// hyperguild#199: a name near NAME_MAX cannot take a ".processed" suffix. Under
|
||||
// inbox zero the marker is only the fallback when removal fails, and any failure
|
||||
// there is remembered, so the note is still not handled every poll.
|
||||
func TestProcessDir_ALongNameWhoseMarkerCannotBeWrittenIsNotHandledEveryPoll(t *testing.T) {
|
||||
brainDir := setupBrainDir(t)
|
||||
rawDir := filepath.Join(brainDir, "raw")
|
||||
name := strings.Repeat("a", 250) + ".md" // 253 bytes; +".processed" exceeds 255
|
||||
require.NoError(t, os.WriteFile(filepath.Join(rawDir, name), []byte("Content."), 0o644))
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(rawDir, "processed"), 0o755))
|
||||
require.NoError(t, os.Chmod(rawDir, 0o555)) // removal fails, so the marker fallback runs
|
||||
t.Cleanup(func() { _ = os.Chmod(rawDir, 0o755) })
|
||||
cfg := Config{BrainDir: brainDir, Pipeline: pipeline.Config{
|
||||
Complete: successComplete(pipeline.RawPage{Title: "Long", Type: "source", Content: "x\n"}), Schema: "# Schema"}}
|
||||
|
||||
for i := 0; i < 3; i++ {
|
||||
errs := processDir(context.Background(), cfg, "2026-10-20")
|
||||
// Without the remember step the second poll tries to archive a second
|
||||
// copy under a name that is too long and errors, every poll.
|
||||
assert.Empty(t, errs, "poll %d handled the note again", i)
|
||||
}
|
||||
archived, err := filepath.Glob(filepath.Join(rawDir, "processed", "2026-10-20", "*"))
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, archived, 1)
|
||||
}
|
||||
|
||||
// Three notes named n.md in three folders, one poll. The archive and failed/
|
||||
// copies are the only copy once the originals are retired, so none may replace
|
||||
// another.
|
||||
func TestProcessDir_SameNamedNotesAreAllArchivedNoneOverwritten(t *testing.T) {
|
||||
for _, outcome := range []string{"processed", "failed"} {
|
||||
t.Run(outcome, func(t *testing.T) {
|
||||
brainDir := setupBrainDir(t)
|
||||
for _, d := range []string{"a", "b", "c"} {
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(brainDir, "raw", d), 0o755))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(brainDir, "raw", d, "n.md"), []byte("CONTENT-"+d), 0o644))
|
||||
}
|
||||
cfg := Config{BrainDir: brainDir, Pipeline: pipeline.Config{Schema: "# Schema",
|
||||
Complete: successComplete(pipeline.RawPage{Title: "N", Type: "source", Content: "x\n"})}}
|
||||
if outcome == "failed" {
|
||||
cfg.Pipeline.Complete = errorComplete
|
||||
}
|
||||
processDir(context.Background(), cfg, "2026-10-20")
|
||||
|
||||
var all string
|
||||
files, _ := filepath.Glob(filepath.Join(brainDir, "raw", outcome, "*", "*"))
|
||||
more, _ := filepath.Glob(filepath.Join(brainDir, "raw", outcome, "*"))
|
||||
for _, f := range append(files, more...) {
|
||||
if b, err := os.ReadFile(f); err == nil {
|
||||
all += string(b)
|
||||
}
|
||||
}
|
||||
for _, d := range []string{"a", "b", "c"} {
|
||||
assert.Contains(t, all, "CONTENT-"+d, "the copy of note %s was overwritten", d)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A marker older than the note describes an earlier note of the same name; it
|
||||
// must not hide the new one, silently, forever (the legacy .processed markers
|
||||
// from before inbox zero are exactly this).
|
||||
func TestProcessDir_AStaleMarkerDoesNotHideANewerNoteOfTheSameName(t *testing.T) {
|
||||
brainDir := setupBrainDir(t)
|
||||
rawDir := filepath.Join(brainDir, "raw")
|
||||
marker := filepath.Join(rawDir, "idea.md.processed")
|
||||
require.NoError(t, os.WriteFile(marker, []byte("2026-06-16"), 0o644))
|
||||
old := time.Now().Add(-48 * time.Hour)
|
||||
require.NoError(t, os.Chtimes(marker, old, old))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(rawDir, "idea.md"), []byte("A new idea, same file name."), 0o644))
|
||||
|
||||
var calls atomic.Int32
|
||||
cfg := Config{BrainDir: brainDir, Pipeline: pipeline.Config{Schema: "# Schema",
|
||||
Complete: func(ctx context.Context, s, u string) (string, error) {
|
||||
calls.Add(1)
|
||||
return successComplete(pipeline.RawPage{Title: "Idea", Type: "source", Content: "x\n"})(ctx, s, u)
|
||||
}}}
|
||||
require.Empty(t, processDir(context.Background(), cfg, "2026-10-20"))
|
||||
|
||||
assert.Equal(t, int32(1), calls.Load(), "the new note is processed")
|
||||
_, err := os.Stat(filepath.Join(rawDir, "idea.md"))
|
||||
assert.ErrorIs(t, err, os.ErrNotExist, "and retired")
|
||||
_, err = os.Lstat(marker)
|
||||
assert.ErrorIs(t, err, os.ErrNotExist, "the stale marker is cleaned up")
|
||||
}
|
||||
|
||||
func TestProcessDir_AMarkerNewerThanTheNoteStillMeansDone(t *testing.T) {
|
||||
brainDir := setupBrainDir(t)
|
||||
rawDir := filepath.Join(brainDir, "raw")
|
||||
note := filepath.Join(rawDir, "stuck.md")
|
||||
require.NoError(t, os.WriteFile(note, []byte("Content."), 0o644))
|
||||
past := time.Now().Add(-time.Hour)
|
||||
require.NoError(t, os.Chtimes(note, past, past))
|
||||
require.NoError(t, os.WriteFile(note+".processed", []byte("d"), 0o644)) // newer than the note
|
||||
|
||||
var calls atomic.Int32
|
||||
cfg := Config{BrainDir: brainDir, Pipeline: pipeline.Config{Schema: "# Schema",
|
||||
Complete: func(context.Context, string, string) (string, error) { calls.Add(1); return "[]", nil }}}
|
||||
processDir(context.Background(), cfg, "2026-10-20")
|
||||
assert.Zero(t, calls.Load(), "a fallback marker written after the note still stops reprocessing")
|
||||
}
|
||||
|
||||
// The archive copy is a second read of the note; the cap must hold there too,
|
||||
// not only at the earlier Lstat (a note can grow in between).
|
||||
func TestReadNoteMax(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
require.NoError(t, os.WriteFile(filepath.Join(dir, "n.md"), []byte(strings.Repeat("x", 100)), 0o644))
|
||||
root, err := os.OpenRoot(dir)
|
||||
require.NoError(t, err)
|
||||
defer func() { _ = root.Close() }()
|
||||
|
||||
got, err := readNoteMax(root, "n.md", 100)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, got, 100, "exactly at the cap is allowed")
|
||||
|
||||
_, err = readNoteMax(root, "n.md", 99)
|
||||
require.Error(t, err)
|
||||
assert.ErrorIs(t, err, errTooLarge)
|
||||
}
|
||||
|
||||
// countingReader reports how much was pulled from it: an endless file must be
|
||||
// abandoned at max+1 bytes, not slurped.
|
||||
type countingReader struct{ n int64 }
|
||||
|
||||
func (c *countingReader) Read(p []byte) (int, error) {
|
||||
if c.n > 8<<20 { // a regression must fail the test, not exhaust memory
|
||||
return 0, errors.New("endless reader drained")
|
||||
}
|
||||
for i := range p {
|
||||
p[i] = 'x'
|
||||
}
|
||||
c.n += int64(len(p))
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
func TestReadAllMax_StopsReadingAtTheCap(t *testing.T) {
|
||||
c := &countingReader{}
|
||||
_, err := readAllMax(c, 1000)
|
||||
require.ErrorIs(t, err, errTooLarge)
|
||||
assert.LessOrEqual(t, c.n, int64(1000+1+32*1024), "pulled far more than the cap from an endless reader: %d", c.n)
|
||||
}
|
||||
|
||||
// From the second review of #210: a marker dated in the future (clock skew,
|
||||
// Syncthing) must not hide the note for good, and when a marker does block it
|
||||
// has to say so, once.
|
||||
func TestProcessDir_AFutureDatedMarkerDoesNotHideTheNote(t *testing.T) {
|
||||
brainDir := setupBrainDir(t)
|
||||
rawDir := filepath.Join(brainDir, "raw")
|
||||
marker := filepath.Join(rawDir, "idea.md.processed")
|
||||
require.NoError(t, os.WriteFile(marker, []byte("x"), 0o644))
|
||||
future := time.Now().Add(72 * time.Hour)
|
||||
require.NoError(t, os.Chtimes(marker, future, future))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(rawDir, "idea.md"), []byte("A new idea."), 0o644))
|
||||
|
||||
var calls atomic.Int32
|
||||
cfg := Config{BrainDir: brainDir, Pipeline: pipeline.Config{Schema: "# Schema",
|
||||
Complete: func(ctx context.Context, s, u string) (string, error) {
|
||||
calls.Add(1)
|
||||
return successComplete(pipeline.RawPage{Title: "Idea", Type: "source", Content: "x\n"})(ctx, s, u)
|
||||
}}}
|
||||
processDir(context.Background(), cfg, "2026-10-20")
|
||||
assert.Equal(t, int32(1), calls.Load(), "a marker from the future is not evidence the note was handled")
|
||||
}
|
||||
|
||||
func TestProcessDir_BothStaleMarkersAreRemoved(t *testing.T) {
|
||||
brainDir := setupBrainDir(t)
|
||||
rawDir := filepath.Join(brainDir, "raw")
|
||||
old := time.Now().Add(-48 * time.Hour)
|
||||
for _, suffix := range []string{".processed", ".failed"} {
|
||||
require.NoError(t, os.WriteFile(filepath.Join(rawDir, "idea.md"+suffix), []byte("x"), 0o644))
|
||||
require.NoError(t, os.Chtimes(filepath.Join(rawDir, "idea.md"+suffix), old, old))
|
||||
}
|
||||
require.NoError(t, os.WriteFile(filepath.Join(rawDir, "idea.md"), []byte("A new idea."), 0o644))
|
||||
cfg := Config{BrainDir: brainDir, Pipeline: pipeline.Config{Schema: "# Schema",
|
||||
Complete: successComplete(pipeline.RawPage{Title: "Idea", Type: "source", Content: "x\n"})}}
|
||||
processDir(context.Background(), cfg, "2026-10-20")
|
||||
for _, suffix := range []string{".processed", ".failed"} {
|
||||
_, err := os.Lstat(filepath.Join(rawDir, "idea.md"+suffix))
|
||||
assert.ErrorIs(t, err, os.ErrNotExist, "orphan %s marker left behind", suffix)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProcessDir_AMarkerThatBlocksANoteIsReportedOnce(t *testing.T) {
|
||||
var logs strings.Builder
|
||||
prev := slog.Default()
|
||||
slog.SetDefault(slog.New(slog.NewTextHandler(&logs, nil)))
|
||||
t.Cleanup(func() { slog.SetDefault(prev) })
|
||||
|
||||
brainDir := setupBrainDir(t)
|
||||
rawDir := filepath.Join(brainDir, "raw")
|
||||
note := filepath.Join(rawDir, "held.md")
|
||||
require.NoError(t, os.WriteFile(note, []byte("Content."), 0o644))
|
||||
past := time.Now().Add(-time.Hour)
|
||||
require.NoError(t, os.Chtimes(note, past, past))
|
||||
require.NoError(t, os.WriteFile(note+".processed", []byte("d"), 0o644)) // newer: blocks
|
||||
cfg := Config{BrainDir: brainDir, Pipeline: pipeline.Config{Schema: "# Schema", Complete: errorComplete}}
|
||||
for i := 0; i < 3; i++ {
|
||||
processDir(context.Background(), cfg, "2026-10-20")
|
||||
}
|
||||
assert.Equal(t, 1, strings.Count(logs.String(), "held by a marker"), "said once, not every poll: %s", logs.String())
|
||||
}
|
||||
|
||||
@@ -0,0 +1,358 @@
|
||||
package watcher
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/wing"
|
||||
)
|
||||
|
||||
func wingTestCfg(t *testing.T, mk func(brainDir string) wing.Placer, llmCalls *atomic.Int32) (Config, string) {
|
||||
t.Helper()
|
||||
brainDir := setupBrainDir(t)
|
||||
cfg := Config{
|
||||
BrainDir: brainDir,
|
||||
Pipeline: pipeline.Config{
|
||||
Schema: "# Schema",
|
||||
Complete: func(ctx context.Context, s, u string) (string, error) {
|
||||
llmCalls.Add(1)
|
||||
return successComplete(pipeline.RawPage{Title: "Talk", Type: "source", Subtype: "video", Content: "## Summary\n\nx\n"})(ctx, s, u)
|
||||
},
|
||||
},
|
||||
}
|
||||
if mk != nil {
|
||||
cfg.Resolver = mk(brainDir)
|
||||
}
|
||||
return cfg, brainDir
|
||||
}
|
||||
|
||||
// router is the production Router with a client-content marker.
|
||||
func router(brainDir string) wing.Placer {
|
||||
return wing.Router{BrainDir: brainDir, Default: "web-clip", ClientBlock: regexp.MustCompile(`(?i)acme-client`)}
|
||||
}
|
||||
|
||||
func makeWing(t *testing.T, brainDir, name string) {
|
||||
t.Helper()
|
||||
require.NoError(t, os.MkdirAll(filepath.Join(brainDir, "wiki", name, "facts"), 0o755))
|
||||
}
|
||||
|
||||
func put(t *testing.T, brainDir, rel, body string) {
|
||||
t.Helper()
|
||||
full := filepath.Join(brainDir, "raw", rel)
|
||||
require.NoError(t, os.MkdirAll(filepath.Dir(full), 0o755))
|
||||
require.NoError(t, os.WriteFile(full, []byte(body), 0o644))
|
||||
}
|
||||
|
||||
func has(brainDir, rel string) bool {
|
||||
_, err := os.Stat(filepath.Join(brainDir, rel))
|
||||
return err == nil
|
||||
}
|
||||
|
||||
func mustJSONPages(pages []pipeline.RawPage) string {
|
||||
b, err := json.Marshal(pages)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
func parkedBody(t *testing.T, brainDir, name string) string {
|
||||
t.Helper()
|
||||
b, err := os.ReadFile(filepath.Join(brainDir, "raw", "needs-review", name))
|
||||
require.NoError(t, err)
|
||||
return string(b)
|
||||
}
|
||||
|
||||
func linesStartingWith(s, prefix string) int {
|
||||
n := 0
|
||||
for _, l := range strings.Split(s, "\n") {
|
||||
if strings.HasPrefix(l, prefix) {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func TestProcessDir_ASubfolderNamesTheWing(t *testing.T) {
|
||||
var llmCalls atomic.Int32
|
||||
cfg, brainDir := wingTestCfg(t, router, &llmCalls)
|
||||
makeWing(t, brainDir, "radar")
|
||||
put(t, brainDir, "radar/dark-factory.md", "A talk about factories.")
|
||||
|
||||
require.Empty(t, processDir(context.Background(), cfg, "2026-10-20"))
|
||||
|
||||
assert.True(t, has(brainDir, "wiki/radar/sources/dark-factory.md"), "lands in the named wing")
|
||||
assert.False(t, has(brainDir, "raw/radar/dark-factory.md"), "inbox zero holds for subfolders too")
|
||||
}
|
||||
|
||||
func TestProcessDir_ADeliberateSubfolderIntoAnyExistingWingPlaces(t *testing.T) {
|
||||
var llmCalls atomic.Int32
|
||||
cfg, brainDir := wingTestCfg(t, router, &llmCalls)
|
||||
makeWing(t, brainDir, "rikatillsammans")
|
||||
put(t, brainDir, "rikatillsammans/meeting.md", "Notes the owner put there on purpose.")
|
||||
|
||||
require.Empty(t, processDir(context.Background(), cfg, "2026-10-20"))
|
||||
|
||||
assert.True(t, has(brainDir, "wiki/rikatillsammans/sources/meeting.md"),
|
||||
"a folder is the owner's deliberate choice; extraction runs on the local model")
|
||||
}
|
||||
|
||||
func TestProcessDir_ANoteTheRouterCannotPlaceIsParkedNotIngested(t *testing.T) {
|
||||
var llmCalls atomic.Int32
|
||||
cfg, brainDir := wingTestCfg(t, router, &llmCalls)
|
||||
put(t, brainDir, "kickoff.md", "Some note about the ACME-CLIENT kickoff.")
|
||||
|
||||
require.Empty(t, processDir(context.Background(), cfg, "2026-10-20"))
|
||||
|
||||
assert.Zero(t, llmCalls.Load(), "no extraction runs for an unplaced note")
|
||||
assert.False(t, has(brainDir, "raw/kickoff.md"), "leaves the top of raw/")
|
||||
parked := parkedBody(t, brainDir, "kickoff.md")
|
||||
assert.Contains(t, parked, "proposal_reason:")
|
||||
assert.Contains(t, parked, "client-content marker")
|
||||
assert.Contains(t, parked, "ACME-CLIENT kickoff", "the note body is preserved")
|
||||
}
|
||||
|
||||
func TestProcessDir_ATypoedWingFolderIsParkedAndNeverCreated(t *testing.T) {
|
||||
var llmCalls atomic.Int32
|
||||
cfg, brainDir := wingTestCfg(t, router, &llmCalls)
|
||||
makeWing(t, brainDir, "homelab")
|
||||
put(t, brainDir, "hoemlab/n.md", "Note.")
|
||||
|
||||
require.Empty(t, processDir(context.Background(), cfg, "2026-10-20"))
|
||||
|
||||
assert.False(t, has(brainDir, "wiki/hoemlab"))
|
||||
assert.True(t, has(brainDir, "raw/needs-review/n.md"))
|
||||
assert.Zero(t, llmCalls.Load())
|
||||
}
|
||||
|
||||
func TestProcessDir_ParkedNotesAreNotReprocessed(t *testing.T) {
|
||||
var llmCalls atomic.Int32
|
||||
cfg, brainDir := wingTestCfg(t, router, &llmCalls)
|
||||
put(t, brainDir, "needs-review/waiting.md", "---\nproposed_wing: radar\n---\nA parked note.")
|
||||
|
||||
require.Empty(t, processDir(context.Background(), cfg, "2026-10-20"))
|
||||
|
||||
assert.Zero(t, llmCalls.Load(), "needs-review/ is a human queue, like rejected/")
|
||||
assert.True(t, has(brainDir, "raw/needs-review/waiting.md"))
|
||||
}
|
||||
|
||||
func TestProcessDir_WithoutAResolverTheRetiredLayoutIsUnchanged(t *testing.T) {
|
||||
var llmCalls atomic.Int32
|
||||
cfg, brainDir := wingTestCfg(t, nil, &llmCalls)
|
||||
put(t, brainDir, "plain.md", "Plain note.")
|
||||
|
||||
require.Empty(t, processDir(context.Background(), cfg, "2026-10-20"))
|
||||
|
||||
assert.True(t, has(brainDir, "wiki/sources/plain.md"))
|
||||
}
|
||||
|
||||
// The reason comes from file and folder names and ends up in frontmatter: a
|
||||
// newline in it must not be able to start a new key (PR #147: a title once
|
||||
// forged `origin:`).
|
||||
func TestProcessDir_AFolderNameCannotForgeFrontmatterKeys(t *testing.T) {
|
||||
var llmCalls atomic.Int32
|
||||
cfg, brainDir := wingTestCfg(t, router, &llmCalls)
|
||||
put(t, brainDir, "radar\nproposal_confidence: 0.99\nwing: homelab/forge.md", "Note.")
|
||||
|
||||
require.Empty(t, processDir(context.Background(), cfg, "2026-10-20"))
|
||||
|
||||
entries, err := os.ReadDir(filepath.Join(brainDir, "raw", "needs-review"))
|
||||
require.NoError(t, err)
|
||||
require.Len(t, entries, 1)
|
||||
b := parkedBody(t, brainDir, entries[0].Name())
|
||||
assert.Equal(t, 1, linesStartingWith(b, "proposal_confidence:"), "exactly one confidence key at line start")
|
||||
assert.Equal(t, 0, linesStartingWith(b, "wing:"), "no injected wing key")
|
||||
assert.Equal(t, 0, linesStartingWith(b, "proposed_wing:"), "a non-plain wing name is never a proposal")
|
||||
}
|
||||
|
||||
func TestProcessDir_ANotesOwnProposalKeysDoNotOutrankTheRouters(t *testing.T) {
|
||||
var llmCalls atomic.Int32
|
||||
cfg, brainDir := wingTestCfg(t, router, &llmCalls)
|
||||
put(t, brainDir, "n.md", "---\nproposed_wing: homelab\nproposal_confidence: 1.00\nproposal_reason: \"self-approved\"\n---\nACME-CLIENT body.")
|
||||
|
||||
require.Empty(t, processDir(context.Background(), cfg, "2026-10-20"))
|
||||
|
||||
b := parkedBody(t, brainDir, "n.md")
|
||||
assert.NotContains(t, b, "self-approved")
|
||||
assert.Equal(t, 1, linesStartingWith(b, "proposal_reason:"))
|
||||
assert.Equal(t, 0, linesStartingWith(b, "proposed_wing:"))
|
||||
}
|
||||
|
||||
func TestProcessDir_ACRLFNoteWithAWingKeyIsStillRead(t *testing.T) {
|
||||
var llmCalls atomic.Int32
|
||||
cfg, brainDir := wingTestCfg(t, router, &llmCalls)
|
||||
makeWing(t, brainDir, "radar")
|
||||
put(t, brainDir, "n.md", "---\r\nwing: radar\r\n---\r\nBody.\r\n")
|
||||
|
||||
require.Empty(t, processDir(context.Background(), cfg, "2026-10-20"))
|
||||
|
||||
assert.True(t, has(brainDir, "wiki/radar/sources/n.md"), "the declared wing is honoured")
|
||||
}
|
||||
|
||||
// editingPlacer rewrites the note while the placement decision is being made,
|
||||
// then parks it: the realistic window is a slow decision with Syncthing
|
||||
// delivering a newer version meanwhile.
|
||||
type editingPlacer struct{ edit func() }
|
||||
|
||||
func (e editingPlacer) Resolve(context.Context, wing.Note) wing.Decision {
|
||||
e.edit()
|
||||
return wing.Decision{Source: wing.Exception, Reason: "test: parked after an edit"}
|
||||
}
|
||||
|
||||
func TestProcessDir_ANoteEditedWhileItIsBeingPlacedIsNotDeleted(t *testing.T) {
|
||||
var llmCalls atomic.Int32
|
||||
var notePath string
|
||||
cfg, brainDir := wingTestCfg(t, func(string) wing.Placer {
|
||||
return editingPlacer{edit: func() {
|
||||
require.NoError(t, os.WriteFile(notePath, []byte("NEWER and longer version"), 0o644))
|
||||
}}
|
||||
}, &llmCalls)
|
||||
notePath = filepath.Join(brainDir, "raw", "n.md")
|
||||
put(t, brainDir, "n.md", "ORIGINAL")
|
||||
|
||||
require.Empty(t, processDir(context.Background(), cfg, "2026-10-20"))
|
||||
|
||||
got, err := os.ReadFile(notePath)
|
||||
require.NoError(t, err, "the edited note must still be in raw/")
|
||||
assert.Equal(t, "NEWER and longer version", string(got))
|
||||
}
|
||||
|
||||
// A planted symlink at raw/needs-review must not redirect a parked note out of
|
||||
// the brain, nor lose it from the review queue.
|
||||
func TestProcessDir_AParkedNoteIsNeverWrittenThroughASymlinkedNeedsReview(t *testing.T) {
|
||||
var llmCalls atomic.Int32
|
||||
cfg, brainDir := wingTestCfg(t, router, &llmCalls)
|
||||
outside := t.TempDir()
|
||||
require.NoError(t, os.Symlink(outside, filepath.Join(brainDir, "raw", "needs-review")))
|
||||
put(t, brainDir, "n.md", "ACME-CLIENT note.")
|
||||
|
||||
errs := processDir(context.Background(), cfg, "2026-10-20")
|
||||
|
||||
assert.NotEmpty(t, errs, "parking is refused, loudly")
|
||||
entries, _ := os.ReadDir(outside)
|
||||
assert.Empty(t, entries, "nothing was written through the symlink")
|
||||
assert.True(t, has(brainDir, "raw/n.md"), "the note stays where it was")
|
||||
}
|
||||
|
||||
func TestWithProposal_ANonFenceDashLineDoesNotCloseTheBlock(t *testing.T) {
|
||||
out := withProposal("---\nfoo: bar\n----\nBody", wing.Decision{Wing: "radar", Confidence: 0.4, Reason: "r"})
|
||||
lines := strings.Split(out, "\n")
|
||||
require.Equal(t, "---", lines[0])
|
||||
closing := -1
|
||||
for i := 1; i < len(lines); i++ {
|
||||
if lines[i] == "---" {
|
||||
closing = i
|
||||
break
|
||||
}
|
||||
}
|
||||
require.Positive(t, closing, "the proposal block is closed by a real fence")
|
||||
assert.Contains(t, strings.Join(lines[:closing], "\n"), "proposal_reason:")
|
||||
assert.NotContains(t, strings.Join(lines[:closing], "\n"), "foo: bar", "the unfenced note is body, not part of the block")
|
||||
}
|
||||
|
||||
// The parked copy is the only copy once the original is retired, so two notes
|
||||
// that share a name must both survive.
|
||||
func TestProcessDir_SameNamedNotesAreAllParkedNoneOverwritten(t *testing.T) {
|
||||
var llmCalls atomic.Int32
|
||||
cfg, brainDir := wingTestCfg(t, router, &llmCalls)
|
||||
for i, body := range []string{"ONE", "TWO", "THREE", "FOUR"} {
|
||||
put(t, brainDir, filepath.Join("typo"+string(rune('1'+i)), "x.md"), body)
|
||||
}
|
||||
require.Empty(t, processDir(context.Background(), cfg, "2026-10-20"))
|
||||
|
||||
files, err := filepath.Glob(filepath.Join(brainDir, "raw", "needs-review", "*.md"))
|
||||
require.NoError(t, err)
|
||||
var all string
|
||||
for _, f := range files {
|
||||
b, _ := os.ReadFile(f)
|
||||
all += string(b)
|
||||
}
|
||||
for _, body := range []string{"ONE", "TWO", "THREE", "FOUR"} {
|
||||
assert.Contains(t, all, body, "note %s must not have been overwritten", body)
|
||||
}
|
||||
assert.Len(t, files, 4)
|
||||
}
|
||||
|
||||
// Syncthing can bring a removed note back (an offline peer, a restored backup).
|
||||
// The source-hash drift check must keep that from costing a second extraction.
|
||||
func TestProcessDir_ANoteThatReappearsUnchangedIsNotExtractedAgain(t *testing.T) {
|
||||
var llmCalls atomic.Int32
|
||||
cfg, brainDir := wingTestCfg(t, router, &llmCalls)
|
||||
makeWing(t, brainDir, "radar")
|
||||
|
||||
put(t, brainDir, "radar/talk.md", "The same talk, byte for byte.")
|
||||
require.Empty(t, processDir(context.Background(), cfg, "2026-10-20"))
|
||||
require.Equal(t, int32(1), llmCalls.Load(), "positive control: the first drop is extracted")
|
||||
require.False(t, has(brainDir, "raw/radar/talk.md"))
|
||||
|
||||
put(t, brainDir, "radar/talk.md", "The same talk, byte for byte.") // the peer sends it back
|
||||
require.Empty(t, processDir(context.Background(), cfg, "2026-10-21"))
|
||||
|
||||
assert.Equal(t, int32(1), llmCalls.Load(), "an unchanged source is not extracted twice")
|
||||
assert.False(t, has(brainDir, "raw/radar/talk.md"), "and inbox zero still holds")
|
||||
}
|
||||
|
||||
func TestProcessDir_ANoteThatReappearsChangedIsExtractedAgain(t *testing.T) {
|
||||
var llmCalls atomic.Int32
|
||||
cfg, brainDir := wingTestCfg(t, router, &llmCalls)
|
||||
makeWing(t, brainDir, "radar")
|
||||
|
||||
put(t, brainDir, "radar/talk.md", "Version one.")
|
||||
require.Empty(t, processDir(context.Background(), cfg, "2026-10-20"))
|
||||
put(t, brainDir, "radar/talk.md", "Version two, edited.")
|
||||
require.Empty(t, processDir(context.Background(), cfg, "2026-10-21"))
|
||||
|
||||
assert.Equal(t, int32(2), llmCalls.Load(), "a real edit must still be picked up")
|
||||
}
|
||||
|
||||
func TestFrontmatterWing(t *testing.T) {
|
||||
tests := []struct {
|
||||
name, in, want string
|
||||
}{
|
||||
{"plain key", "---\nwing: radar\n---\nbody", "radar"},
|
||||
{"quoted", "---\nwing: \"radar\"\n---\nbody", "radar"},
|
||||
{"CRLF", "---\r\nwing: radar\r\n---\r\nbody", "radar"},
|
||||
{"no frontmatter", "wing: radar\nbody", ""},
|
||||
{"an indented key belongs to another mapping", "---\nmeta:\n wing: radar\n---\nbody", ""},
|
||||
{"a key before a non-fence dash line is not frontmatter", "---\nwing: radar\n----\nbody", ""},
|
||||
{"no closing fence", "---\nwing: radar\nbody", ""},
|
||||
{"closing fence at end of input", "---\nwing: radar\n---", "radar"},
|
||||
{"another key first", "---\ntitle: x\nwing: radar\n---\nbody", "radar"},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
assert.Equal(t, tt.want, frontmatterWing(tt.in))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestFrontmatterWing_ABOMDoesNotHideTheKey(t *testing.T) {
|
||||
assert.Equal(t, "homelab", frontmatterWing("\ufeff---\nwing: homelab\n---\nbody"))
|
||||
}
|
||||
|
||||
// A note that produced nothing must not leave the inbox as "processed".
|
||||
func TestProcessDir_AWingIngestThatWritesNothingFailsTheNoteVisibly(t *testing.T) {
|
||||
var llmCalls atomic.Int32
|
||||
cfg, brainDir := wingTestCfg(t, router, &llmCalls)
|
||||
cfg.Pipeline.Complete = func(context.Context, string, string) (string, error) { return "[]", nil }
|
||||
put(t, brainDir, "empty-result.md", "A note the model returned nothing for.")
|
||||
|
||||
require.Empty(t, processDir(context.Background(), cfg, "2026-10-20"))
|
||||
|
||||
assert.True(t, has(brainDir, "raw/failed/empty-result.md"), "kept where a human looks")
|
||||
processed, _ := filepath.Glob(filepath.Join(brainDir, "raw", "processed", "*", "*"))
|
||||
assert.Empty(t, processed, "never archived as processed")
|
||||
logb, err := os.ReadFile(filepath.Join(brainDir, "log.md"))
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, string(logb), "no source page")
|
||||
}
|
||||
@@ -0,0 +1,190 @@
|
||||
package watcher
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
|
||||
)
|
||||
|
||||
func okPipeline() pipeline.Config {
|
||||
return pipeline.Config{Complete: func(context.Context, string, string) (string, error) {
|
||||
b, _ := json.Marshal([]pipeline.RawPage{{Title: "X", Type: "source", Subtype: "article", Content: "## S\n\nx\n"}})
|
||||
return string(b), nil
|
||||
}, Schema: "# Schema"}
|
||||
}
|
||||
|
||||
func failingPipeline() pipeline.Config {
|
||||
return pipeline.Config{Complete: func(context.Context, string, string) (string, error) {
|
||||
return "", errors.New("model down")
|
||||
}, Schema: "# Schema"}
|
||||
}
|
||||
|
||||
// hyperguild#158: the watcher wrote the archive copy under raw/processed or
|
||||
// raw/failed, and its log.md entry, by path, so a symlinked archive
|
||||
// directory or log redirected the write elsewhere in the brain or outside
|
||||
// it. (A pre-planted marker symlink makes the watcher skip the note, so the
|
||||
// marker write itself is reachable only by a race.)
|
||||
func TestProcessDirDoesNotWriteThroughASymlinkedArchiveDir(t *testing.T) {
|
||||
for name, pcfg := range map[string]pipeline.Config{"processed": okPipeline(), "failed": failingPipeline()} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
brainDir := setupBrainDir(t)
|
||||
outside := t.TempDir()
|
||||
victim := filepath.Join(brainDir, "wiki", "concepts", "v.md")
|
||||
require.NoError(t, os.WriteFile(victim, []byte("untouched"), 0o644))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(brainDir, "raw", "n.md"), []byte("note"), 0o644))
|
||||
require.NoError(t, os.Symlink(outside, filepath.Join(brainDir, "raw", name)))
|
||||
_ = processDir(context.Background(), Config{BrainDir: brainDir, Interval: time.Hour, Pipeline: pcfg}, "2026-10-05")
|
||||
got, _ := os.ReadFile(victim)
|
||||
assert.Equal(t, "untouched", string(got))
|
||||
ents, _ := os.ReadDir(outside)
|
||||
assert.Empty(t, ents, "the archive copy was written outside the brain")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestProcessDirWritesTheArchiveInRealDirectoriesAndRetiresTheOriginal(t *testing.T) {
|
||||
brainDir := setupBrainDir(t)
|
||||
require.NoError(t, os.WriteFile(filepath.Join(brainDir, "raw", "n.md"), []byte("note"), 0o644))
|
||||
errs := processDir(context.Background(), Config{BrainDir: brainDir, Interval: time.Hour, Pipeline: okPipeline()}, "2026-10-05")
|
||||
assert.Empty(t, errs)
|
||||
// Inbox zero (v0.25.0): no marker is written beside a note that was removed.
|
||||
_, err := os.Stat(filepath.Join(brainDir, "raw", "n.md.processed"))
|
||||
assert.ErrorIs(t, err, os.ErrNotExist, "a marker is only the fallback when the removal fails")
|
||||
_, err = os.Stat(filepath.Join(brainDir, "raw", "n.md"))
|
||||
assert.ErrorIs(t, err, os.ErrNotExist, "positive control: the original is retired")
|
||||
_, err = os.Stat(filepath.Join(brainDir, "raw", "processed", "2026-10-05", "n.md"))
|
||||
assert.NoError(t, err, "positive control: archive copy written")
|
||||
}
|
||||
|
||||
// The log.md entry for a failed note is appended through the brain root: with
|
||||
// a real raw/failed (so the archive copy succeeds and the log is reached), a
|
||||
// symlinked log.md is not written through.
|
||||
func TestProcessDirDoesNotAppendTheLogThroughASymlink(t *testing.T) {
|
||||
brainDir := setupBrainDir(t)
|
||||
victim := filepath.Join(brainDir, "wiki", "concepts", "v.md")
|
||||
require.NoError(t, os.WriteFile(victim, []byte("untouched"), 0o644))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(brainDir, "raw", "n.md"), []byte("note"), 0o644))
|
||||
require.NoError(t, os.Symlink(filepath.Join("wiki", "concepts", "v.md"), filepath.Join(brainDir, "log.md")))
|
||||
_ = processDir(context.Background(), Config{BrainDir: brainDir, Interval: time.Hour, Pipeline: failingPipeline()}, "2026-10-05")
|
||||
_, err := os.Stat(filepath.Join(brainDir, "raw", "failed", "n.md"))
|
||||
require.NoError(t, err, "positive control: the archive copy was written, so the log append was reached")
|
||||
got, _ := os.ReadFile(victim)
|
||||
assert.Equal(t, "untouched", string(got), "the log was appended through a symlink")
|
||||
}
|
||||
|
||||
// A marker path holding anything (here a dangling symlink) counts as a
|
||||
// marker: the note is skipped and nothing is created at the link's target.
|
||||
// (os.Stat followed the link, saw nothing, and processed the note again.)
|
||||
func TestProcessDirTreatsAnyEntryAtTheMarkerPathAsAMarker(t *testing.T) {
|
||||
for _, kind := range []string{"processed", "failed"} {
|
||||
t.Run(kind, func(t *testing.T) {
|
||||
brainDir := setupBrainDir(t)
|
||||
outside := filepath.Join(t.TempDir(), "created")
|
||||
require.NoError(t, os.WriteFile(filepath.Join(brainDir, "raw", "n.md"), []byte("note"), 0o644))
|
||||
require.NoError(t, os.Symlink(outside, filepath.Join(brainDir, "raw", "n.md."+kind)))
|
||||
calls := 0
|
||||
cfg := Config{BrainDir: brainDir, Interval: time.Hour, Pipeline: pipeline.Config{
|
||||
Complete: func(context.Context, string, string) (string, error) { calls++; return "[]", nil }, Schema: "# Schema"}}
|
||||
_ = processDir(context.Background(), cfg, "2026-10-05")
|
||||
assert.Zero(t, calls, "the note was processed despite an entry at its marker path")
|
||||
_, err := os.Lstat(outside)
|
||||
assert.True(t, os.IsNotExist(err), "a file was created at the marker link's target")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A link planted at the marker path while pipeline.Run is in flight (in
|
||||
// production, an LLM call) ends up with its target untouched, and the note is
|
||||
// processed once, even after the link is removed (as symlinkguard does). Under
|
||||
// inbox zero the original is removed, so the marker write that this scenario
|
||||
// was written to redirect is never reached: this test does NOT pin that the
|
||||
// marker-fallback write goes through the brain root (see the PR description).
|
||||
func TestProcessDirLeavesAMarkerLinkPlantedDuringTheRunUntouchedAndProcessesTheNoteOnce(t *testing.T) {
|
||||
for _, kind := range []string{"processed", "failed"} {
|
||||
for _, target := range []string{"outside", "in-brain"} {
|
||||
t.Run(kind+"/"+target, func(t *testing.T) {
|
||||
brainDir := setupBrainDir(t)
|
||||
raw := filepath.Join(brainDir, "raw")
|
||||
marker := filepath.Join(raw, "n.md."+kind)
|
||||
require.NoError(t, os.WriteFile(filepath.Join(raw, "n.md"), []byte("note"), 0o644))
|
||||
// The in-brain target is relative, so os.Root itself would
|
||||
// follow it; only the regular-file check refuses it.
|
||||
victim := filepath.Join(raw, "victim")
|
||||
link := filepath.Join(t.TempDir(), "created")
|
||||
if target == "in-brain" {
|
||||
require.NoError(t, os.WriteFile(victim, []byte("untouched"), 0o644))
|
||||
link = "victim"
|
||||
}
|
||||
ok := okPipeline().Complete
|
||||
calls := 0
|
||||
cfg := Config{BrainDir: brainDir, Interval: time.Hour, Pipeline: pipeline.Config{Schema: "# Schema",
|
||||
Complete: func(ctx context.Context, sys, user string) (string, error) {
|
||||
calls++
|
||||
require.NoError(t, os.Symlink(link, marker))
|
||||
if kind == "failed" {
|
||||
return "", errors.New("model down")
|
||||
}
|
||||
return ok(ctx, sys, user)
|
||||
}}}
|
||||
for i := 0; i < 3; i++ {
|
||||
_ = processDir(context.Background(), cfg, "2026-10-05")
|
||||
_ = os.Remove(marker)
|
||||
}
|
||||
assert.Equal(t, 1, calls, "the note was retried after the link was removed")
|
||||
if target == "in-brain" {
|
||||
got, err := os.ReadFile(victim)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "untouched", string(got), "the marker was written through the link")
|
||||
} else {
|
||||
_, err := os.Lstat(link)
|
||||
assert.True(t, os.IsNotExist(err), "the marker was written through the link")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A refused archive copy must not retry every interval: the note is
|
||||
// remembered and skipped until it changes (a symlinked raw/failed cost one
|
||||
// LLM call per round before).
|
||||
func TestProcessDirDoesNotRetryANoteWhoseArchiveCopyIsRefused(t *testing.T) {
|
||||
brainDir := setupBrainDir(t)
|
||||
require.NoError(t, os.WriteFile(filepath.Join(brainDir, "raw", "n.md"), []byte("note"), 0o644))
|
||||
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brainDir, "raw", "failed")))
|
||||
calls := 0
|
||||
cfg := Config{BrainDir: brainDir, Interval: time.Hour, Pipeline: pipeline.Config{
|
||||
Complete: func(context.Context, string, string) (string, error) { calls++; return "", errors.New("model down") }, Schema: "# Schema"}}
|
||||
for i := 0; i < 3; i++ {
|
||||
_ = processDir(context.Background(), cfg, "2026-10-05")
|
||||
}
|
||||
assert.Equal(t, 1, calls, "the refused note was retried every round")
|
||||
|
||||
// Once the note changes it is tried again.
|
||||
require.NoError(t, os.WriteFile(filepath.Join(brainDir, "raw", "n.md"), []byte("note, edited"), 0o644))
|
||||
future := time.Now().Add(2 * time.Second)
|
||||
require.NoError(t, os.Chtimes(filepath.Join(brainDir, "raw", "n.md"), future, future))
|
||||
_ = processDir(context.Background(), cfg, "2026-10-05")
|
||||
assert.Equal(t, 2, calls, "positive control: a changed note is tried again")
|
||||
}
|
||||
|
||||
// A note whose name has a backslash is legal on Linux and must be processed
|
||||
// (cleanRel refused it, so it was never archived or marked).
|
||||
func TestProcessDirProcessesANoteWithABackslashInItsName(t *testing.T) {
|
||||
brainDir := setupBrainDir(t)
|
||||
require.NoError(t, os.WriteFile(filepath.Join(brainDir, "raw", "a\\b.md"), []byte("note"), 0o644))
|
||||
errs := processDir(context.Background(), Config{BrainDir: brainDir, Interval: time.Hour, Pipeline: okPipeline()}, "2026-10-05")
|
||||
assert.Empty(t, errs)
|
||||
_, err := os.Stat(filepath.Join(brainDir, "raw", "processed", "2026-10-05", "a\\b.md"))
|
||||
assert.NoError(t, err, "archived")
|
||||
_, err = os.Stat(filepath.Join(brainDir, "raw", "a\\b.md"))
|
||||
assert.ErrorIs(t, err, os.ErrNotExist, "and retired")
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
package wiki
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func put(t *testing.T, dir, rel string) {
|
||||
t.Helper()
|
||||
full := filepath.Join(dir, rel)
|
||||
require.NoError(t, os.MkdirAll(filepath.Dir(full), 0o755))
|
||||
require.NoError(t, os.WriteFile(full, []byte("---\ntitle: x\n---\n"), 0o644))
|
||||
}
|
||||
|
||||
func TestFindSlugs(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
put(t, dir, "wiki/sources/legacy-src.md")
|
||||
put(t, dir, "wiki/concepts/a-concept.md")
|
||||
put(t, dir, "wiki/entities/an-entity.md")
|
||||
put(t, dir, "wiki/homelab/failures/in-a-hall.md")
|
||||
put(t, dir, "wiki/web-clip/sources/a-clip.md")
|
||||
// One slug in two places: the first hit must not hide the second.
|
||||
put(t, dir, "wiki/sources/shared-slug.md")
|
||||
put(t, dir, "wiki/entities/shared-slug.md")
|
||||
put(t, dir, "wiki/aaa/facts/shared-slug.md")
|
||||
|
||||
tests := []struct {
|
||||
slug string
|
||||
want []string
|
||||
}{
|
||||
{"legacy-src", []string{"wiki/sources/legacy-src.md"}},
|
||||
{"a-concept", []string{"wiki/concepts/a-concept.md"}},
|
||||
{"an-entity", []string{"wiki/entities/an-entity.md"}},
|
||||
{"in-a-hall", []string{"wiki/homelab/failures/in-a-hall.md"}},
|
||||
{"a-clip", []string{"wiki/web-clip/sources/a-clip.md"}},
|
||||
{"shared-slug", []string{"wiki/aaa/facts/shared-slug.md", "wiki/entities/shared-slug.md", "wiki/sources/shared-slug.md"}},
|
||||
{"nothing-here", nil},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.slug, func(t *testing.T) {
|
||||
got, err := FindSlugs(dir, tt.slug)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, tt.want, got)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestFindSlugs_NoWikiDirFindsNothing(t *testing.T) {
|
||||
got, err := FindSlugs(t.TempDir(), "x")
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, got)
|
||||
}
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
|
||||
@@ -96,3 +97,98 @@ func readFrontmatter(bfs *brainfs.Brain, rel, fallbackSlug string) (title string
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// LoadWingInventory returns what an ingest into one wing can match against:
|
||||
// that wing's sources (wiki/<wing>/sources/) and the SHARED concepts and
|
||||
// entities (wiki/concepts/, wiki/entities/), which belong to no single wing
|
||||
// because their slug is the graph's global identity. Other wings' sources are
|
||||
// not read.
|
||||
func LoadWingInventory(brainDir, wing string) (map[PageType][]Entry, error) {
|
||||
result, err := LoadInventory(brainDir)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
result[PageTypeSource] = []Entry{} // the retired source layout is not this wing's
|
||||
bfs, err := brainfs.Open(brainDir)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("open brain: %w", err)
|
||||
}
|
||||
defer func() { _ = bfs.Close() }()
|
||||
dir := "wiki/" + wing + "/sources"
|
||||
names, err := bfs.ReadDirRegular(dir)
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
return result, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read dir %s: %w", dir, err)
|
||||
}
|
||||
for _, name := range names {
|
||||
if !strings.HasSuffix(name, ".md") {
|
||||
continue
|
||||
}
|
||||
rel := dir + "/" + name
|
||||
slug := strings.TrimSuffix(name, ".md")
|
||||
title, aliases := readFrontmatter(bfs, rel, slug)
|
||||
result[PageTypeSource] = append(result[PageTypeSource], Entry{Slug: slug, Title: title, Aliases: aliases, Type: PageTypeSource, Path: rel})
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// FindSlugs returns the brain-relative path of EVERY existing page whose
|
||||
// basename is slug, searching the retired dirs (wiki/{sources,concepts,
|
||||
// entities}/) and every wing hall (wiki/<wing>/<hall>/), sorted. The slug is a
|
||||
// page's graph identity, so two pages must not share one; returning all of
|
||||
// them (not the first) is what lets a caller tell "only my own page" from "my
|
||||
// own page and somebody else's".
|
||||
func FindSlugs(brainDir, slug string) ([]string, error) {
|
||||
bfs, err := brainfs.Open(brainDir)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("open brain: %w", err)
|
||||
}
|
||||
defer func() { _ = bfs.Close() }()
|
||||
file := slug + ".md"
|
||||
var found []string
|
||||
check := func(dir string) error {
|
||||
names, err := bfs.ReadDirRegular(dir)
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, n := range names {
|
||||
if n == file {
|
||||
found = append(found, dir+"/"+file)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
for _, d := range []string{"wiki/sources", "wiki/concepts", "wiki/entities"} {
|
||||
if err := check(d); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
wings, err := bfs.ReadDirDirs("wiki")
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
return found, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, w := range wings {
|
||||
if w == "sources" || w == "concepts" || w == "entities" {
|
||||
continue
|
||||
}
|
||||
halls, err := bfs.ReadDirDirs("wiki/" + w)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
for _, h := range halls {
|
||||
if err := check("wiki/" + w + "/" + h); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
}
|
||||
sort.Strings(found)
|
||||
return found, nil
|
||||
}
|
||||
|
||||
@@ -22,4 +22,5 @@ type Entry struct {
|
||||
Title string
|
||||
Aliases []string
|
||||
Type PageType
|
||||
Path string // brain-relative path; set for wing-layout entries
|
||||
}
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
package wing_test
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func writePage(t *testing.T, dir, rel, body string) {
|
||||
t.Helper()
|
||||
full := filepath.Join(dir, rel)
|
||||
require.NoError(t, os.MkdirAll(filepath.Dir(full), 0o755))
|
||||
require.NoError(t, os.WriteFile(full, []byte(body), 0o644))
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
package wing
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
|
||||
)
|
||||
|
||||
// legacyDirs are the retired wiki/{concepts,entities,sources}/ layout, not wings.
|
||||
var legacyDirs = map[string]bool{"concepts": true, "entities": true, "sources": true}
|
||||
|
||||
// KnownWings lists wiki/<wing>/ directories that contain at least one hall
|
||||
// subdirectory, sorted. The retired layout is not a wing. A missing wiki/ is
|
||||
// an error: an unreadable brain must not look like "no wings".
|
||||
func KnownWings(brainDir string) ([]string, error) {
|
||||
entries, err := os.ReadDir(filepath.Join(brainDir, "wiki"))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("list wings: %w", err)
|
||||
}
|
||||
var out []string
|
||||
for _, e := range entries {
|
||||
if !e.IsDir() || legacyDirs[e.Name()] || !plainWingName(e.Name()) {
|
||||
continue
|
||||
}
|
||||
subs, err := os.ReadDir(filepath.Join(brainDir, "wiki", e.Name()))
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
for _, s := range subs {
|
||||
if s.IsDir() {
|
||||
out = append(out, e.Name())
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// plainWingName reports whether name is already in the canonical lower-case form
|
||||
// the pipeline writes. Anything else would make the pipeline create a different
|
||||
// wing than the one matched.
|
||||
func plainWingName(name string) bool { return brain.Sanitise(name) == name }
|
||||
@@ -0,0 +1,42 @@
|
||||
package wing_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/wing"
|
||||
)
|
||||
|
||||
func TestKnownWings_ListsWingsWithHallsAndSkipsTheRetiredLayout(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
writePage(t, dir, "wiki/homelab/facts/a.md", "x")
|
||||
writePage(t, dir, "wiki/tapir/decisions/b.md", "x")
|
||||
writePage(t, dir, "wiki/concepts/c.md", "x") // retired layout, a file directly under a legacy dir
|
||||
writePage(t, dir, "wiki/sources/d.md", "x") // retired layout
|
||||
writePage(t, dir, "wiki/entities/e.md", "x") // retired layout
|
||||
writePage(t, dir, "wiki/loose-note.md", "x") // a file, not a wing
|
||||
writePage(t, dir, "wiki/emptywing/_index.md", "x") // a dir with no hall subdirectory
|
||||
got, err := wing.KnownWings(dir)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{"homelab", "tapir"}, got)
|
||||
}
|
||||
|
||||
func TestKnownWings_NoWikiDirIsAnError(t *testing.T) {
|
||||
_, err := wing.KnownWings(t.TempDir())
|
||||
assert.Error(t, err, "a brain without wiki/ must not look like a brain with no wings")
|
||||
}
|
||||
|
||||
// Known matches case-insensitively and the pipeline writes the lower-case name,
|
||||
// so a wiki directory that is not already plain lower-case would make the
|
||||
// pipeline create a different, new wing. It is not a candidate.
|
||||
func TestKnownWings_SkipsADirectoryThatIsNotAPlainWingName(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
writePage(t, dir, "wiki/Radar/facts/a.md", "x")
|
||||
writePage(t, dir, "wiki/has space/facts/a.md", "x")
|
||||
writePage(t, dir, "wiki/ok/facts/a.md", "x")
|
||||
got, err := wing.KnownWings(dir)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{"ok"}, got)
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
// Package wing decides which brain wing an incoming raw note belongs to
|
||||
// without the human typing it (hyperguild#205, #208).
|
||||
//
|
||||
// An earlier design classified each note with a local model and cross-checked
|
||||
// it with a text score. Measured against pages whose wing a human had chosen,
|
||||
// the model was right 60% of the time with a saturated self-reported
|
||||
// confidence, and the cross-check was right 11% of the time, so the gate it
|
||||
// formed placed 2 of 55 pages (hyperguild#208). The Router that replaced it
|
||||
// decides with rules only: a deliberate folder or frontmatter wing, else one
|
||||
// default wing; nothing it cannot decide safely is guessed.
|
||||
package wing
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Source says how a Decision was reached.
|
||||
type Source string
|
||||
|
||||
const (
|
||||
Explicit Source = "explicit" // subfolder or frontmatter named the wing
|
||||
Default Source = "default" // no signal; the configured default wing
|
||||
Exception Source = "exception" // needs a human
|
||||
)
|
||||
|
||||
// Note is what the Placer sees of an incoming raw note.
|
||||
type Note struct {
|
||||
Dir string // first subfolder under raw/, "" when dropped at the top
|
||||
FrontmatterWing string // `wing:` from the note's frontmatter, "" if absent
|
||||
Text string
|
||||
}
|
||||
|
||||
// Decision is the outcome. For an Exception, Wing holds the proposal when
|
||||
// there was one (so a reviewer can approve it) and is empty otherwise.
|
||||
type Decision struct {
|
||||
Wing string
|
||||
Source Source
|
||||
Confidence float64
|
||||
Reason string
|
||||
}
|
||||
|
||||
// Placer decides where a raw note goes. Router is the production
|
||||
// implementation; the interface lets the watcher stay ignorant of how.
|
||||
type Placer interface {
|
||||
Resolve(ctx context.Context, n Note) Decision
|
||||
}
|
||||
|
||||
func exception(w string, conf float64, reason string) Decision {
|
||||
return Decision{Wing: w, Source: Exception, Confidence: conf, Reason: reason}
|
||||
}
|
||||
|
||||
func norm(s string) string { return strings.ToLower(strings.TrimSpace(s)) }
|
||||
@@ -0,0 +1,248 @@
|
||||
package wing
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strings"
|
||||
"unicode"
|
||||
|
||||
xnorm "golang.org/x/text/unicode/norm"
|
||||
)
|
||||
|
||||
// Router places a note with rules only, no model (hyperguild#208):
|
||||
//
|
||||
// 1. A subfolder or frontmatter `wing:` is the owner's deliberate choice. It
|
||||
// must name a plain, EXISTING wing (a typo never creates one) and then it
|
||||
// places, whatever the wing is: extraction runs on the local model, and it
|
||||
// is the owner who dropped the note there. Two signals that disagree park.
|
||||
// 2. With no signal, a note whose text matches ClientBlock is parked; any
|
||||
// other note goes to Default, the one wing that is created on first use.
|
||||
//
|
||||
// Wings are read from disk on every decision, so a wing created while the
|
||||
// server runs is usable at once.
|
||||
type Router struct {
|
||||
BrainDir string
|
||||
Default string
|
||||
ClientBlock *regexp.Regexp // nil = no content check
|
||||
|
||||
// FilterChunk is how many bytes of a note the content filter folds and
|
||||
// matches at a time (0 = 1 MiB). Folding allocates several times its input,
|
||||
// so a 64 MiB note is checked in pieces (with an overlap, so a marker on a
|
||||
// boundary is still found) instead of all at once.
|
||||
FilterChunk int
|
||||
}
|
||||
|
||||
// Resolve implements Placer. It never returns an error: anything it cannot
|
||||
// decide safely is an Exception with the reason spelled out.
|
||||
func (r Router) Resolve(_ context.Context, n Note) Decision {
|
||||
for _, raw := range []string{n.Dir, n.FrontmatterWing} {
|
||||
if !isASCII(raw) {
|
||||
// strings.ToLower folds some non-ASCII runes to ASCII (the Kelvin
|
||||
// sign becomes "k"), which would let a lookalike name a real wing.
|
||||
return exception("", 0, fmt.Sprintf("wing name %q contains non-ASCII characters", strings.TrimSpace(raw)))
|
||||
}
|
||||
}
|
||||
dir, fm := norm(n.Dir), norm(n.FrontmatterWing)
|
||||
switch {
|
||||
case dir != "" && fm != "" && dir != fm:
|
||||
return exception("", 0, fmt.Sprintf("subfolder %q and frontmatter wing %q disagree", dir, fm))
|
||||
case dir != "":
|
||||
return r.explicit(dir, "subfolder")
|
||||
case fm != "":
|
||||
return r.explicit(fm, "frontmatter")
|
||||
}
|
||||
if r.ClientBlock != nil && matchesClient(r.ClientBlock, n.Text, r.FilterChunk) {
|
||||
return exception("", 0, "text matches the client-content marker; not defaulted into a shared wing")
|
||||
}
|
||||
if r.Default == "" || !usableWingName(r.Default) {
|
||||
return exception("", 0, "no valid default wing configured")
|
||||
}
|
||||
return Decision{Wing: r.Default, Source: Default, Confidence: 1, Reason: "no wing named; default wing"}
|
||||
}
|
||||
|
||||
func (r Router) explicit(w, via string) Decision {
|
||||
if !usableWingName(w) {
|
||||
return exception("", 0, fmt.Sprintf("%s names %q, which is not a usable wing name", via, w))
|
||||
}
|
||||
if w == r.Default {
|
||||
return Decision{Wing: w, Source: Explicit, Confidence: 1, Reason: "named by " + via}
|
||||
}
|
||||
known, err := KnownWings(r.BrainDir)
|
||||
if err != nil {
|
||||
return exception(w, 0, fmt.Sprintf("cannot list wings: %v", err))
|
||||
}
|
||||
if !slices.Contains(known, w) {
|
||||
return exception(w, 0, fmt.Sprintf("%s names %q: not an existing wing (new wings are never created from a folder name)", via, strings.TrimSpace(w)))
|
||||
}
|
||||
return Decision{Wing: w, Source: Explicit, Confidence: 1, Reason: "named by " + via}
|
||||
}
|
||||
|
||||
// NewRouter builds the production Router. defaultWing must be a plain wing
|
||||
// name; clientPattern, when set, is a case-insensitive regexp over note text
|
||||
// whose match parks an unsignalled note instead of filing it. Both fail at
|
||||
// startup rather than later: a client filter that silently does nothing is
|
||||
// worse than none.
|
||||
func NewRouter(brainDir, defaultWing, clientPattern string) (Router, error) {
|
||||
if defaultWing == "" || !usableWingName(defaultWing) {
|
||||
return Router{}, fmt.Errorf("default wing %q is not a usable wing name", defaultWing)
|
||||
}
|
||||
r := Router{BrainDir: brainDir, Default: defaultWing}
|
||||
if clientPattern != "" {
|
||||
re, err := regexp.Compile("(?i)" + clientPattern)
|
||||
if err != nil {
|
||||
return Router{}, fmt.Errorf("client-content pattern: %w", err)
|
||||
}
|
||||
if re.MatchString("x") || re.MatchString("") {
|
||||
return Router{}, fmt.Errorf("client-content pattern %q matches text that names no client; it would park every note", clientPattern)
|
||||
}
|
||||
r.ClientBlock = re
|
||||
}
|
||||
return r, nil
|
||||
}
|
||||
|
||||
// reservedWingNames are directory names under wiki/ that are not wings: the
|
||||
// retired layout and the root. A note must never be proposed into one.
|
||||
var reservedWingNames = map[string]bool{"sources": true, "concepts": true, "entities": true, "wiki": true}
|
||||
|
||||
// usableWingName: a plain name that is not reserved.
|
||||
func usableWingName(w string) bool { return plainWingName(w) && !reservedWingNames[w] }
|
||||
|
||||
func isASCII(s string) bool {
|
||||
for _, r := range s {
|
||||
if r > unicode.MaxASCII {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
const (
|
||||
defaultFilterChunk = 1 << 20
|
||||
filterOverlap = 4096
|
||||
)
|
||||
|
||||
// matchesClient applies the client-content pattern to the text in bounded
|
||||
// chunks, each overlapping the next, so memory stays proportional to the chunk
|
||||
// and a marker on a chunk edge is still found.
|
||||
func matchesClient(re *regexp.Regexp, text string, chunk int) bool {
|
||||
if chunk <= 0 {
|
||||
chunk = defaultFilterChunk
|
||||
}
|
||||
overlap := filterOverlap
|
||||
if overlap >= chunk {
|
||||
overlap = chunk / 2
|
||||
}
|
||||
for start := 0; start < len(text); {
|
||||
end := start + chunk
|
||||
if end >= len(text) {
|
||||
end = len(text)
|
||||
}
|
||||
if matchesFolded(re, text[start:end]) {
|
||||
return true
|
||||
}
|
||||
if end == len(text) {
|
||||
return false
|
||||
}
|
||||
// A cut inside a rune is harmless: the overlap carries every match that
|
||||
// spans at most filterOverlap bytes whole into the next chunk. A match
|
||||
// longer than that (a pattern like `acme.{5000}corp`, or padding
|
||||
// between two words) can be missed on a chunk edge; documented limit.
|
||||
start = end - overlap
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// matchesFolded matches the pattern against the chunk as it is and against
|
||||
// folded forms, so the cheap evasions do not defeat it. Folding: NFKD (fullwidth
|
||||
// letters, accents become base letter + mark), then marks, format characters and
|
||||
// invisible fillers dropped, Cyrillic/Greek lookalikes mapped to Latin, every
|
||||
// dash made "-". Views: whitespace collapsed to one space (so "acme corp" and
|
||||
// \s work), whitespace removed (so "acme-\nclient" works), and the same with a
|
||||
// soft hyphen read as a dash. It catches what it names and nothing else:
|
||||
// leetspeak, punctuation splits (a.c.m.e), encodings and the many homoglyphs
|
||||
// not in the small table below are not caught. It is a filter, not a classifier.
|
||||
func matchesFolded(re *regexp.Regexp, chunk string) bool {
|
||||
if re.MatchString(chunk) {
|
||||
return true
|
||||
}
|
||||
decomposed := xnorm.NFKD.String(chunk)
|
||||
return re.MatchString(fold(decomposed, false, true)) ||
|
||||
re.MatchString(fold(decomposed, false, false)) ||
|
||||
re.MatchString(fold(decomposed, true, false))
|
||||
}
|
||||
|
||||
// fold builds one view. softAsDash reads U+00AD as "-" instead of dropping it;
|
||||
// keepSpace collapses whitespace to a single space instead of removing it.
|
||||
func fold(s string, softAsDash, keepSpace bool) string {
|
||||
var b strings.Builder
|
||||
b.Grow(len(s))
|
||||
space := false
|
||||
for _, r := range s {
|
||||
switch {
|
||||
case r == '\u00ad' && softAsDash:
|
||||
b.WriteRune('-')
|
||||
space = false
|
||||
continue
|
||||
case unicode.Is(unicode.Mn, r) || unicode.Is(unicode.Me, r) || unicode.Is(unicode.Mc, r) ||
|
||||
unicode.Is(unicode.Cf, r) || invisibleFiller[r]:
|
||||
continue
|
||||
case unicode.IsSpace(r) || unicode.Is(unicode.Zs, r):
|
||||
if keepSpace && !space {
|
||||
b.WriteByte(' ')
|
||||
space = true
|
||||
}
|
||||
continue
|
||||
}
|
||||
space = false
|
||||
switch {
|
||||
case unicode.Is(unicode.Pd, r) || r == '\u2212':
|
||||
b.WriteRune('-')
|
||||
default:
|
||||
if l, ok := latinLookalike[r]; ok {
|
||||
r = l
|
||||
}
|
||||
b.WriteRune(r)
|
||||
}
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// invisibleFiller are characters that render as nothing but are not format
|
||||
// characters: Hangul fillers and the blank Braille pattern.
|
||||
var invisibleFiller = map[rune]bool{0x115F: true, 0x1160: true, 0x3164: true, 0xFFA0: true, 0x2800: true}
|
||||
|
||||
// latinLookalike maps the commonest Cyrillic and Greek letters that read as
|
||||
// Latin ones. Deliberately small; see matchesFolded.
|
||||
var latinLookalike = map[rune]rune{
|
||||
'\u0430': 'a', '\u0435': 'e', '\u043e': 'o', '\u0440': 'p', '\u0441': 'c', '\u0445': 'x', '\u0443': 'y',
|
||||
'\u0456': 'i', '\u0458': 'j', '\u0455': 's', '\u04bb': 'h', '\u0501': 'd', '\u043a': 'k', '\u043d': 'h', '\u0442': 't',
|
||||
'\u0410': 'A', '\u0412': 'B', '\u0415': 'E', '\u041a': 'K', '\u041c': 'M', '\u041d': 'H', '\u041e': 'O', '\u0420': 'P', '\u0421': 'C', '\u0422': 'T', '\u0425': 'X',
|
||||
'\u03bf': 'o', '\u03bd': 'v', '\u03c1': 'p', '\u0391': 'A', '\u0392': 'B', '\u0395': 'E', '\u039f': 'O', '\u03a1': 'P',
|
||||
}
|
||||
|
||||
// WrapNames turns a bare list of client names (a regexp alternation such as
|
||||
// "acme|globex") into a pattern that matches a name only as a whole word, with
|
||||
// word edges decided by Unicode letters and digits: Go's \b is ASCII-only, so
|
||||
// with it a name that begins or ends in a letter like Å, é or Ø would never
|
||||
// match. An underscore or punctuation is a boundary (trucker_ecommerce matches
|
||||
// "trucker"), a letter or digit glued on is not (acmecorp, acme2).
|
||||
//
|
||||
// The list is validated first, on its own: it must compile as a regexp, must not
|
||||
// match the empty string (an empty alternative such as "a||b" would match
|
||||
// everywhere), and must be self-contained: a value like "x)|.*(" compiles alone
|
||||
// only if it is balanced, so it cannot break out of the group it is put in.
|
||||
func WrapNames(names string) (string, error) {
|
||||
if strings.TrimSpace(names) == "" {
|
||||
return "", fmt.Errorf("client names list is empty")
|
||||
}
|
||||
re, err := regexp.Compile("(?i)" + names)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("client names are not a valid regexp alternation: %w", err)
|
||||
}
|
||||
if re.MatchString("") {
|
||||
return "", fmt.Errorf("client names contain an empty alternative; it would match everywhere")
|
||||
}
|
||||
return `(?:^|[^\p{L}\p{N}])(?:` + names + `)(?:[^\p{L}\p{N}]|$)`, nil
|
||||
}
|
||||
@@ -0,0 +1,292 @@
|
||||
package wing_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
|
||||
"github.com/mathiasbq/hyperguild/ingestion/internal/wing"
|
||||
)
|
||||
|
||||
func routerBrain(t *testing.T) string {
|
||||
dir := t.TempDir()
|
||||
writePage(t, dir, "wiki/homelab/facts/a.md", "x")
|
||||
writePage(t, dir, "wiki/rikatillsammans/facts/b.md", "x")
|
||||
return dir
|
||||
}
|
||||
|
||||
func TestRouter_Decisions(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
note wing.Note
|
||||
want wing.Source
|
||||
wantWing string
|
||||
}{
|
||||
{"no signal goes to the default wing", wing.Note{Text: "a talk about agents"}, wing.Default, "web-clip"},
|
||||
{"a subfolder naming an existing wing places there", wing.Note{Dir: "homelab", Text: "x"}, wing.Explicit, "homelab"},
|
||||
{"a deliberate subfolder into a client wing is the owner's call and is placed", wing.Note{Dir: "rikatillsammans", Text: "x"}, wing.Explicit, "rikatillsammans"},
|
||||
{"frontmatter wing places like a subfolder", wing.Note{FrontmatterWing: "homelab", Text: "x"}, wing.Explicit, "homelab"},
|
||||
{"subfolder is case-insensitive", wing.Note{Dir: "HomeLab", Text: "x"}, wing.Explicit, "homelab"},
|
||||
{"a typo never creates a wing", wing.Note{Dir: "hoemlab", Text: "x"}, wing.Exception, "hoemlab"},
|
||||
{"conflicting signals are parked", wing.Note{Dir: "homelab", FrontmatterWing: "rikatillsammans", Text: "x"}, wing.Exception, ""},
|
||||
{"the default wing may be named explicitly even before it exists", wing.Note{Dir: "web-clip", Text: "x"}, wing.Explicit, "web-clip"},
|
||||
{"client marker with no signal is parked, not defaulted", wing.Note{Text: "notes from the ACME-CLIENT kickoff"}, wing.Exception, ""},
|
||||
{"client marker does not block a deliberate subfolder", wing.Note{Dir: "homelab", Text: "ACME-CLIENT"}, wing.Explicit, "homelab"},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
r := wing.Router{BrainDir: routerBrain(t), Default: "web-clip", ClientBlock: regexp.MustCompile(`(?i)acme-client`)}
|
||||
d := r.Resolve(context.Background(), tt.note)
|
||||
assert.Equal(t, tt.want, d.Source)
|
||||
assert.Equal(t, tt.wantWing, d.Wing)
|
||||
assert.NotEmpty(t, d.Reason)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRouter_ReadsTheWingsFreshOnEveryDecision(t *testing.T) {
|
||||
dir := routerBrain(t)
|
||||
r := wing.Router{BrainDir: dir, Default: "web-clip"}
|
||||
assert.Equal(t, wing.Exception, r.Resolve(context.Background(), wing.Note{Dir: "newwing", Text: "x"}).Source)
|
||||
|
||||
writePage(t, dir, "wiki/newwing/facts/c.md", "x") // a wing created after startup
|
||||
assert.Equal(t, wing.Explicit, r.Resolve(context.Background(), wing.Note{Dir: "newwing", Text: "x"}).Source,
|
||||
"a wing made while the server runs is usable without a restart")
|
||||
}
|
||||
|
||||
func TestRouter_AnUnreadableBrainParksInsteadOfGuessing(t *testing.T) {
|
||||
r := wing.Router{BrainDir: t.TempDir() + "/missing", Default: "web-clip"}
|
||||
d := r.Resolve(context.Background(), wing.Note{Dir: "homelab", Text: "x"})
|
||||
assert.Equal(t, wing.Exception, d.Source)
|
||||
}
|
||||
|
||||
func TestRouter_WithoutADefaultAnUnsignalledNoteIsParked(t *testing.T) {
|
||||
r := wing.Router{BrainDir: routerBrain(t)}
|
||||
assert.Equal(t, wing.Exception, r.Resolve(context.Background(), wing.Note{Text: "x"}).Source)
|
||||
}
|
||||
|
||||
func TestRouter_ADefaultThatIsNotAPlainNameIsRefused(t *testing.T) {
|
||||
r := wing.Router{BrainDir: routerBrain(t), Default: "../escape"}
|
||||
assert.Equal(t, wing.Exception, r.Resolve(context.Background(), wing.Note{Text: "x"}).Source)
|
||||
}
|
||||
|
||||
func TestRouter_ANameThatIsNotPlainIsRefusedAsSuch(t *testing.T) {
|
||||
for _, bad := range []string{"../x", "a/b", "has space", "dot.ted"} {
|
||||
t.Run(bad, func(t *testing.T) {
|
||||
// Default equals the bad name: the "named explicitly" shortcut for the
|
||||
// default wing must not wave a malformed name through.
|
||||
r := wing.Router{BrainDir: routerBrain(t), Default: bad}
|
||||
d := r.Resolve(context.Background(), wing.Note{Dir: bad, Text: "x"})
|
||||
assert.Equal(t, wing.Exception, d.Source)
|
||||
assert.Empty(t, d.Wing, "a malformed name is never echoed as a proposal")
|
||||
assert.Contains(t, d.Reason, "not a usable wing name")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewRouter(t *testing.T) {
|
||||
dir := routerBrain(t)
|
||||
t.Run("valid", func(t *testing.T) {
|
||||
r, err := wing.NewRouter(dir, "web-clip", "acme-client|globex")
|
||||
assert.NoError(t, err)
|
||||
d := r.Resolve(context.Background(), wing.Note{Text: "kickoff with GLOBEX"})
|
||||
assert.Equal(t, wing.Exception, d.Source, "the marker is matched case-insensitively")
|
||||
})
|
||||
t.Run("no marker pattern means no content check", func(t *testing.T) {
|
||||
r, err := wing.NewRouter(dir, "web-clip", "")
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, wing.Default, r.Resolve(context.Background(), wing.Note{Text: "anything"}).Source)
|
||||
})
|
||||
t.Run("a default that is not a plain wing name is a startup error", func(t *testing.T) {
|
||||
_, err := wing.NewRouter(dir, "Web Clip", "")
|
||||
assert.Error(t, err)
|
||||
})
|
||||
t.Run("an empty default is a startup error", func(t *testing.T) {
|
||||
_, err := wing.NewRouter(dir, "", "")
|
||||
assert.Error(t, err)
|
||||
})
|
||||
t.Run("a pattern that does not compile is a startup error", func(t *testing.T) {
|
||||
_, err := wing.NewRouter(dir, "web-clip", "([unclosed")
|
||||
assert.Error(t, err, "a client filter that silently does nothing is worse than none")
|
||||
})
|
||||
}
|
||||
|
||||
// From the review of #211: the content filter is the only guard on the default
|
||||
// lane, so it must survive the cheap evasions.
|
||||
func TestRouter_ClientMarkerSurvivesCheapEvasions(t *testing.T) {
|
||||
r := wing.Router{BrainDir: routerBrain(t), Default: "web-clip", ClientBlock: regexp.MustCompile(`(?i)acme-client`)}
|
||||
tests := []struct{ name, text string }{
|
||||
{"non-breaking hyphen", "notes on acme‑client"},
|
||||
{"zero-width joiner inside", "notes on acme\u200d-client"},
|
||||
{"zero-width space inside", "notes on ac\u200bme-client"},
|
||||
{"split across lines", "notes on acme-\nclient"},
|
||||
{"spaced hyphen", "notes on acme - client"},
|
||||
{"fullwidth letters", "notes on acme-client"},
|
||||
{"soft hyphen", "notes on acme\u00ad-client"},
|
||||
{"en dash", "notes on acme–client"},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
d := r.Resolve(context.Background(), wing.Note{Text: tt.text})
|
||||
assert.Equal(t, wing.Exception, d.Source, tt.text)
|
||||
})
|
||||
}
|
||||
t.Run("an unrelated text still files", func(t *testing.T) {
|
||||
d := r.Resolve(context.Background(), wing.Note{Text: "a talk about agents and a client-server model"})
|
||||
assert.Equal(t, wing.Default, d.Source)
|
||||
})
|
||||
}
|
||||
|
||||
func TestRouter_FolderNamesAreAsciiAndNotReserved(t *testing.T) {
|
||||
dir := routerBrain(t)
|
||||
writePage(t, dir, "wiki/kubernetes/facts/a.md", "x")
|
||||
r := wing.Router{BrainDir: dir, Default: "web-clip"}
|
||||
tests := []struct {
|
||||
name string
|
||||
dir string
|
||||
}{
|
||||
{"Kelvin sign lowercases to an ASCII k", "Kubernetes"},
|
||||
{"cyrillic lookalike", "h\u043emelab"},
|
||||
{"retired layout name: sources", "sources"},
|
||||
{"retired layout name: concepts", "concepts"},
|
||||
{"retired layout name: entities", "entities"},
|
||||
{"wiki", "wiki"},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
d := r.Resolve(context.Background(), wing.Note{Dir: tt.dir, Text: "x"})
|
||||
assert.Equal(t, wing.Exception, d.Source)
|
||||
assert.Empty(t, d.Wing, "never proposed as a wing")
|
||||
})
|
||||
}
|
||||
t.Run("plain ASCII case folding still works", func(t *testing.T) {
|
||||
d := r.Resolve(context.Background(), wing.Note{Dir: "KubernetEs", Text: "x"})
|
||||
assert.Equal(t, wing.Explicit, d.Source)
|
||||
assert.Equal(t, "kubernetes", d.Wing)
|
||||
})
|
||||
}
|
||||
|
||||
func TestNewRouter_AReservedDefaultIsRefused(t *testing.T) {
|
||||
_, err := wing.NewRouter(routerBrain(t), "sources", "")
|
||||
assert.Error(t, err)
|
||||
}
|
||||
|
||||
// From the second review of #211: the multi-word, \s and \b patterns an operator
|
||||
// would actually write, and the next tier of evasions.
|
||||
func TestRouter_ClientMarkerHandlesRealisticPatternsAndHarderEvasions(t *testing.T) {
|
||||
tests := []struct {
|
||||
name, pattern, text string
|
||||
}{
|
||||
{"space in pattern vs NBSP", `acme corp`, "about acme\u00a0corp today"},
|
||||
{"space in pattern vs two spaces", `acme corp`, "about acme corp today"},
|
||||
{"space in pattern vs newline", `acme corp`, "about acme\ncorp today"},
|
||||
{"backslash-s pattern vs NBSP", `acme\s+corp`, "about acme\u00a0corp today"},
|
||||
{"word boundary vs a zero-width char inside", `\bacme\b`, "the ac\u200bme corp"},
|
||||
{"dash pattern vs a soft hyphen as the dash", `acme-client`, "acme\u00adclient"},
|
||||
{"cyrillic a", `acme-client`, "\u0430cme-client"},
|
||||
{"cyrillic e", `acme-client`, "acm\u0435-client"},
|
||||
{"cyrillic o and c", `coop`, "c\u043e\u043ep"},
|
||||
{"combining acute", `acme-client`, "a\u0301cme-client"},
|
||||
{"precomposed accent", `acme-client`, "\u00e1cme-client"},
|
||||
{"hangul filler", `acme-client`, "ac\u3164me-client"},
|
||||
{"halfwidth hangul filler", `acme-client`, "ac\uffa0me-client"},
|
||||
{"braille blank", `acme-client`, "ac\u2800me-client"},
|
||||
{"grapheme joiner", `acme-client`, "ac\u034fme-client"},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
r, err := wing.NewRouter(routerBrain(t), "web-clip", tt.pattern)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, wing.Exception, r.Resolve(context.Background(), wing.Note{Text: tt.text}).Source, "%q should be caught by %q", tt.text, tt.pattern)
|
||||
})
|
||||
}
|
||||
t.Run("ordinary text is not caught by a multi-word pattern", func(t *testing.T) {
|
||||
r, _ := wing.NewRouter(routerBrain(t), "web-clip", `acme corp`)
|
||||
assert.Equal(t, wing.Default, r.Resolve(context.Background(), wing.Note{Text: "acme is not corp, and corp is not acme"}).Source)
|
||||
})
|
||||
}
|
||||
|
||||
// The folding allocates several times the text, so a large note is checked in
|
||||
// bounded chunks. A marker that straddles a chunk edge must still be found.
|
||||
func TestRouter_ClientMarkerIsFoundAcrossAChunkBoundary(t *testing.T) {
|
||||
r, err := wing.NewRouter(routerBrain(t), "web-clip", `acme-client`)
|
||||
assert.NoError(t, err)
|
||||
r.FilterChunk = 64
|
||||
for _, at := range []int{0, 30, 55, 58, 62, 64, 70, 120, 190} {
|
||||
text := strings.Repeat("x", at) + "acme-client" + strings.Repeat("y", 200)
|
||||
assert.Equal(t, wing.Exception, r.Resolve(context.Background(), wing.Note{Text: text}).Source, "marker at offset %d was missed", at)
|
||||
}
|
||||
clean := strings.Repeat("lorem ipsum ", 100)
|
||||
assert.Equal(t, wing.Default, r.Resolve(context.Background(), wing.Note{Text: clean}).Source)
|
||||
}
|
||||
|
||||
func TestRouter_ClientMarkerChunkingDoesNotSplitAMultiByteRune(t *testing.T) {
|
||||
r, err := wing.NewRouter(routerBrain(t), "web-clip", `acme-client`)
|
||||
assert.NoError(t, err)
|
||||
r.FilterChunk = 65 // odd, over two-byte runes: every boundary falls inside a rune unless it is moved
|
||||
for pad := 0; pad < 8; pad++ {
|
||||
text := strings.Repeat("\u00e9", 40+pad) + "acme-client" + strings.Repeat("\u00e9", 40)
|
||||
assert.Equal(t, wing.Exception, r.Resolve(context.Background(), wing.Note{Text: text}).Source, "pad %d", pad)
|
||||
}
|
||||
clean := strings.Repeat("\u00e9", 300)
|
||||
assert.Equal(t, wing.Default, r.Resolve(context.Background(), wing.Note{Text: clean}).Source, "no panic or false match on a text of 2-byte runes")
|
||||
}
|
||||
|
||||
// From the review of #214: the shared secret is a bare list of client names.
|
||||
func TestWrapNames(t *testing.T) {
|
||||
park := func(t *testing.T, names, text string) bool {
|
||||
t.Helper()
|
||||
pat, err := wing.WrapNames(names)
|
||||
assert.NoError(t, err)
|
||||
r, err := wing.NewRouter(routerBrain(t), "web-clip", pat)
|
||||
assert.NoError(t, err)
|
||||
return r.Resolve(context.Background(), wing.Note{Text: text}).Source == wing.Exception
|
||||
}
|
||||
t.Run("a short name does not match inside ordinary words", func(t *testing.T) {
|
||||
assert.False(t, park(t, "dif|trucker", "a different approach to modifying prompts"))
|
||||
assert.True(t, park(t, "dif|trucker", "kickoff for the DIF project"))
|
||||
})
|
||||
t.Run("names with a non-ASCII letter at the edge still match (Go's \\b would never)", func(t *testing.T) {
|
||||
assert.True(t, park(t, "Åke|Société|Ørsted", "Åke called"))
|
||||
assert.True(t, park(t, "Åke|Société|Ørsted", "la Société wrote"))
|
||||
assert.True(t, park(t, "Åke|Société|Ørsted", "Ørsted wind"))
|
||||
})
|
||||
t.Run("identifiers with underscores and punctuation count as boundaries", func(t *testing.T) {
|
||||
assert.True(t, park(t, "trucker", "repo trucker_ecommerce was cloned"))
|
||||
assert.True(t, park(t, "trucker", "(Trucker)"))
|
||||
})
|
||||
t.Run("a name inside a longer word or with digits glued on is not a boundary", func(t *testing.T) {
|
||||
assert.False(t, park(t, "acme", "acmecorp"))
|
||||
assert.False(t, park(t, "acme", "acme2"))
|
||||
})
|
||||
}
|
||||
|
||||
func TestWrapNames_RefusesWhatWouldBreakOrBlockEverything(t *testing.T) {
|
||||
for _, bad := range []string{
|
||||
"x)|.*(", // breaks out of the group
|
||||
"acme)|(b", // changes the structure
|
||||
"a||b", // empty alternative matches everywhere
|
||||
"acme|", // trailing empty alternative
|
||||
"|acme", // leading empty alternative
|
||||
"", // nothing
|
||||
"c++ corp", // not a valid regexp
|
||||
"([unclosed", // not a valid regexp
|
||||
} {
|
||||
t.Run(bad, func(t *testing.T) {
|
||||
_, err := wing.WrapNames(bad)
|
||||
assert.Error(t, err)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewRouter_RefusesAPatternThatMatchesTheEmptyString(t *testing.T) {
|
||||
for _, p := range []string{`.*`, `x*`, `^`, `\b`, `(a|)`} {
|
||||
t.Run(p, func(t *testing.T) {
|
||||
_, err := wing.NewRouter(routerBrain(t), "web-clip", p)
|
||||
assert.Error(t, err, "it would park every note")
|
||||
})
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user