mathias/hyperguild
1
0
Fork 0
Code Issues 30 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

watcher: archive copy reads the whole note into memory (io.ReadAll), with no size cap #198

New Issue
Open
opened 2026-10-05 21:47:40 +00:00 by mathias · 0 comments
mathias commented 2026-10-05 21:47:40 +00:00
Owner
Copy Link
Copy Source

Source: PR #195 review round 2 (non-blocking), 2026-10-05.

copyFromRoot in ingestion/internal/watcher/watcher.go now reads the source note with io.ReadAll and writes it with brainfs.PutFile, so the write goes through real directories (#158).

  • For .md and .txt this doubles peak memory, since TextIn already reads the whole file.
  • For .pdf it is a new full in-memory copy, and there is no upstream size cap.
  • A big drop into raw/ can therefore push the ingestion pod towards its memory limit.

Fix options:

  • Stream through a CreateRegular-style writer (an open handle checked like PutRegular, then io.Copy).
  • Or cap the size with ReadRegularMax and refuse oversize notes into raw/failed with a clear reason.

Created via git-mcp on behalf of @mathias

**Source:** PR #195 review round 2 (non-blocking), 2026-10-05. `copyFromRoot` in `ingestion/internal/watcher/watcher.go` now reads the source note with `io.ReadAll` and writes it with `brainfs.PutFile`, so the write goes through real directories (#158). - For `.md` and `.txt` this doubles peak memory, since TextIn already reads the whole file. - For `.pdf` it is a new full in-memory copy, and there is no upstream size cap. - A big drop into `raw/` can therefore push the ingestion pod towards its memory limit. **Fix options:** - Stream through a `CreateRegular`-style writer (an open handle checked like PutRegular, then `io.Copy`). - Or cap the size with `ReadRegularMax` and refuse oversize notes into `raw/failed` with a clear reason. --- _Created via git-mcp on behalf of @mathias_
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agent/ready
Meets the agent-ready contract; authorizes unattended execution
 blocked/client
Waiting on the client
 blocked/supplier
Waiting on a supplier
 blocked/upstream
Waiting on an upstream project or vendor
 risk/high
Irreversible, security-relevant, or control plane
 risk/low
Reversible, local, test-covered
 risk/medium
Touches shared state, deploys or data
 size/l
Too big - split it
 size/m
Under two days
 size/s
Under half a day
 size/xs
Under an hour
 status:dispatch
status:dispatched
status:review
status:reviewing
type/chore
Maintenance, no behaviour change
 type/decision
Human decision required; never agent/ready
 type/docs
Documentation only
 type/feat
New capability
 type/fix
Corrects broken behaviour
 type/ops
Infrastructure, deploy, runtime
 type/research
Investigate; output is a finding, not code
 type/test
Test coverage only
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
renovate-bot mathias
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/hyperguild#198