mathiasandClaude Sonnet 5.5
49286ba8b8
- brain_reject on a parked pdf/txt moves the bytes; it no longer prepends YAML - raw/ notes are read through a cap (readCapped); an endless reader test fails instead of exhausting memory - a byte-order mark no longer hides a note's wing: key - INGEST_WING_ROUTING is parsed strictly; a typo stops the server rather than leaving routing silently off - tests for the reserved-wing check in every disguise (Sources, sources., ...) Mutation-checked. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
3.3 KiB
The raw/ inbox
raw/ is an inbox: only unprocessed material sits there. Syncthing shares it
with the machines you drop files from. (hyperguild#205, #208, #209)
What happens to a file you drop
| You drop | Result |
|---|---|
raw/clip.md (no folder, no wing:) |
extracted and filed as a source in wiki/web-clip/sources/; its concepts and entities update the shared wiki/concepts/ and wiki/entities/ pages; the original moves to raw/processed/<date>/, so raw/ is empty again |
raw/<wing>/clip.md or wing: <wing> in the frontmatter |
the same, with the source in wiki/<wing>/sources/. The wing must already exist; a typo is parked, never created. This is your deliberate choice and is not second-guessed |
| a note whose text matches the client-content pattern, with no folder | parked in raw/needs-review/, not extracted |
a folder or wing: naming an unknown, reserved or non-ASCII wing, or two disagreeing signals |
parked in raw/needs-review/ |
| a note over the size cap, or one that cannot be archived | stays in raw/, reported once, not retried until it changes |
Concepts and entities are shared across wings because their slug is the graph's
identity. A source whose slug is taken gets a wing-qualified one (notes-homelab).
Parked notes
brain_pending lists them with queue: needs-review, a proposed wing and the
reason. To act on one: brain_promote / brain_reject (pass queue when the
same name also sits in the inbox), or move the file into raw/<wing>/ to have it
extracted. Parked pdf/txt files are listed but can only be moved or rejected.
Switching it on
Routing is off by default (the retired layout is used). Set on the ingestion deployment:
| Variable | Meaning |
|---|---|
INGEST_WING_ROUTING |
1/true enables the rules above; unset or 0/false is off; any other value stops the server at startup |
INGEST_DEFAULT_WING |
wing for unsignalled notes (default web-clip; must be a plain name, not sources/concepts/entities/wiki) |
INGEST_CLIENT_BLOCK |
case-insensitive regexp over note text; a match parks an unsignalled note. Falls back to CLAUDE_INGEST_CLIENT_BLOCK. Matched after folding fullwidth letters, removing zero-width characters, normalising dashes and removing whitespace. Patterns with spaces, \\s or \\b work (whitespace is collapsed in one view and removed in another). It catches what it names and nothing else: leetspeak, punctuation splits (a.c.m.e), encodings and homoglyphs outside a small Cyrillic/Greek table are not caught. Large notes are checked in 1 MiB chunks |
A bad default or pattern stops the server at startup. A wing created while the server runs is usable without a restart.
Known limits
- A deliberate folder into a sensitive wing places the note there, and the shared concept/entity pages then cite that source by title. Client material does not belong in the brain at all; the pattern only guards the unsignalled lane.
- A merge into a shared page refreshes
last_updatedandoriginto the latest ingest; the first wing's attribution is not kept. - A marker file (
<note>.processed) is only written when a processed original cannot be removed. One at least as new as its note holds the note (and says so once); an older or future-dated one is ignored and removed.