A doc with no headings and no blank-line paragraphs (JSON-lines, e.g.
wiki/telos/decisions/human-intent-column.md) survived both chunk passes whole
and was sent to nomic-embed over its context window → 'input length exceeds
the context length' (400, the steady embed errors=1). Add a final hard-split
pass (line then UTF-8 rune boundaries) so no chunk exceeds maxBytes. TDD.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Long markdown files (>~8KB) silently failed to embed because nomic-embed-text
on iguana has a 2048-token context. embed sync logged errors=1 every cycle
with no useful body until #37 added per-item logging — three files exceed
the ceiling: finbert source (8 KB), koala-machine-state (7.1 KB),
litellm-absorption (8.8 KB). Curated knowledge entries should never be
vector-blind.
Approach: chunk-before-embed, no schema change.
vectorstore/chunk.go (new)
- ChunkMarkdown splits at H1/H2 boundaries; sections over maxBytes are
further split at paragraph boundaries, packing greedily under budget.
- NumberChunks assigns "<parent>#NNNN" storage paths (1-based, zero-padded
to 4 digits — handles files with up to ~10k sections in stable sort order).
- ParentPath strips the chunk suffix for retrieval-side dedup.
vectorstore/sync.go
- After ChunkMarkdown produces N pieces, each is embedded + upserted as a
separate brain_embeddings row at "<parent>#NNNN". maxChunkBytes = 4000
(≈1000 nomic tokens, well under the 2048 ceiling with headroom for
unicode/code blocks).
- "Already embedded?" check now reduces known paths to parent set via
ParentPath, so the first chunk hit short-circuits the file.
- Delete walk also reduces via ParentPath; when a parent file disappears,
every chunk row (and any pre-existing bare-path row, for backward
compatibility with rows written before this change) gets dropped.
search/search.go
- hybridMerge collapses chunk-path vector hits to parent via ParentPath
before scope check, RRF accumulation, and hydration. A file with three
chunk hits returns one result row, not three.
Backward compatibility: pre-existing bare-path rows in brain_embeddings
keep working — ParentPath returns them unchanged, knownParents handles
them as if they were "wiki/foo.md#NNNN" hits, sync skips re-embed, and
search dedup is a no-op for them. No migration required to ship.
Tests:
- chunk_test.go covers short / heading split / oversized section /
content preservation / chunk numbering / parent-path stripping.
- sync_test.go adds long-file chunking, single-chunk-row short file,
skip-if-any-chunk-known, delete-all-chunks-of-disappeared-file.
Existing tests updated for #NNNN paths.
- search_test.go adds chunk-paths-dedupe-to-parent.
Closes gitea/mathias/infra#38.
mathiasandClaude Opus 4.8