mathiasandClaude Sonnet 5.5
27b1579b80
- in a wing, an ingest that writes no source page is an error: the note goes to
raw/failed with the warnings in log.md instead of leaving the inbox as
'processed' (model returned [], only invalid types, only clashing entities)
- the wing's source path is deterministic, so Resolve no longer redirects a
source onto another note's page because the model gave both the same title
('Home'); that overwrote the hash and the text
- a batch tie between concept and entity prefers the page already on disk, so
its update is not lost; the wing's own source page keeps its slug even when a
foreign page shares it
- brain_pending lists an unreadable (over-cap) note with the reason instead of
dropping it; the client filter's 4 KiB match-span limit is documented
Each has a test that fails without it (mutation-checked).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
58 lines
3.9 KiB
Markdown
58 lines
3.9 KiB
Markdown
# The raw/ inbox
|
|
|
|
`raw/` is an inbox: only unprocessed material sits there. Syncthing shares it
|
|
with the machines you drop files from. (hyperguild#205, #208, #209)
|
|
|
|
## What happens to a file you drop
|
|
|
|
| You drop | Result |
|
|
|---|---|
|
|
| `raw/clip.md` (no folder, no `wing:`) | extracted and filed as a source in `wiki/web-clip/sources/`; its concepts and entities update the shared `wiki/concepts/` and `wiki/entities/` pages; the original moves to `raw/processed/<date>/`, so `raw/` is empty again |
|
|
| `raw/<wing>/clip.md` or `wing: <wing>` in the frontmatter | the same, with the source in `wiki/<wing>/sources/`. The wing must already exist; a typo is parked, never created. This is your deliberate choice and is not second-guessed |
|
|
| a note whose text matches the client-content pattern, with no folder | parked in `raw/needs-review/`, not extracted |
|
|
| a folder or `wing:` naming an unknown, reserved or non-ASCII wing, or two disagreeing signals | parked in `raw/needs-review/` |
|
|
| a note over the size cap, or one that cannot be archived | stays in `raw/`, reported once, not retried until it changes |
|
|
| a note the model returned nothing usable for (no source page could be written) | fails: the note is kept in `raw/failed/` and the reason is in `log.md`; it is never archived as processed |
|
|
|
|
Concepts and entities are shared across wings because their slug is the graph's
|
|
identity. A source whose slug is taken gets a wing-qualified one (`notes-homelab`).
|
|
|
|
## Parked notes
|
|
|
|
`brain_pending` lists them with `queue: needs-review`, a proposed wing and the
|
|
reason. To act on one: `brain_promote` / `brain_reject` (pass `queue` when the
|
|
same name also sits in the inbox), or move the file into `raw/<wing>/` to have it
|
|
extracted. Parked pdf/txt files are listed but can only be moved or rejected.
|
|
|
|
## Switching it on
|
|
|
|
Routing is off by default (the retired layout is used). Set on the ingestion
|
|
deployment:
|
|
|
|
| Variable | Meaning |
|
|
|---|---|
|
|
| `INGEST_WING_ROUTING` | `1`/`true` enables the rules above; unset or `0`/`false` is off; any other value stops the server at startup |
|
|
| `INGEST_DEFAULT_WING` | wing for unsignalled notes (default `web-clip`; must be a plain name, not `sources`/`concepts`/`entities`/`wiki`) |
|
|
| `INGEST_CLIENT_BLOCK` | case-insensitive regexp over note text; a match parks an unsignalled note. Falls back to `CLAUDE_INGEST_CLIENT_BLOCK`. Matched after folding fullwidth letters, removing zero-width characters, normalising dashes and removing whitespace. Patterns with spaces, `\\s` or `\\b` work (whitespace is collapsed in one view and removed in another). It catches what it names and nothing else: leetspeak, punctuation splits (`a.c.m.e`), encodings and homoglyphs outside a small Cyrillic/Greek table are not caught. Large notes are checked in 1 MiB chunks |
|
|
|
|
A bad default or pattern stops the server at startup. A wing created while the
|
|
server runs is usable without a restart.
|
|
|
|
## Known limits
|
|
|
|
- The client filter works on 1 MiB chunks that overlap by 4 KiB. A match that can
|
|
span more than 4 KiB (crafted padding between two words of a multi-word
|
|
pattern, exactly at a chunk edge) is not guaranteed to be found; this affects
|
|
notes over 1 MiB only. On a 64 MiB note the check takes about ten seconds.
|
|
- A pattern that matches the empty string (`.*`, `x*`) parks every unsignalled
|
|
note. That fails closed and only the operator's own configuration can do it.
|
|
|
|
- A deliberate folder into a sensitive wing places the note there, and the shared
|
|
concept/entity pages then cite that source by title. Client material does not
|
|
belong in the brain at all; the pattern only guards the unsignalled lane.
|
|
- A merge into a shared page refreshes `last_updated` and `origin` to the latest
|
|
ingest; the first wing's attribution is not kept.
|
|
- A marker file (`<note>.processed`) is only written when a processed original
|
|
cannot be removed. One at least as new as its note holds the note (and says
|
|
so once); an older or future-dated one is ignored and removed.
|