mathias/hyperguild
1
0
Fork 0
Code Issues 30 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

wiki/index.md contains a truncated UTF-8 sequence — the index writer cuts mid-character #93

New Issue
Closed
opened 2026-08-17 22:11:01 +00:00 by mathias · 1 comment
mathias commented 2026-08-17 22:11:01 +00:00
Owner
Copy Link
Copy Source

Found by the frontmatter lint shipped in #90. It is the one file in mathias/brain that the lint cannot classify, because it cannot be decoded at all.

Symptom

wiki/index.md: not valid UTF-8
first bad byte at offset 590:
  b'...|Local Model Agentic Coding Capability \xe2\x80\xe2\x80\xa6\n- [[brain-wiki'

\xe2\x80 is the first two bytes of a three-byte character (em-dash \xe2\x80\x94, ellipsis \xe2\x80\xa6, …) with its third byte missing — immediately followed by a complete \xe2\x80\xa6. So a title was cut mid-character and then the truncation marker was appended.

That is the signature of truncating by bytes where the intent was characters: something like s[:n] + "…" over a byte slice, or a Go string sliced at a fixed index without a rune boundary check.

Why it matters beyond one file

  • The file is generated, so patching the bytes fixes nothing — the next regeneration reproduces it.
  • Any consumer doing a strict decode (yaml.safe_load, utf8.Valid, most JSON encoders) rejects the whole index, not just the bad line. The frontmatter lint had to be written to report-and-continue specifically so this one file could not hide the other 17 findings.
  • Whatever writer does this is presumably used for other truncated fields too. The index is where it happened to become visible.

Where to look

The index writer that emits wiki/index.md — the - [[wing-slug|Title]] list lines. Find where a title gets shortened before the … is appended.

Fix shape

Truncate on a rune boundary, not a byte offset:

func truncate(s string, max int) string {
    if utf8.RuneCountInString(s) <= max {
        return s
    }
    return string([]rune(s)[:max]) + "…"
}

Acceptance

  • caveman: me cut long title between letters, not inside letter
  • Red-first test: a title whose cut point lands inside a multi-byte character produces valid UTF-8
  • utf8.Valid asserted over the generated index in a test — the property, not one example
  • wiki/index.md regenerates clean; task lint:frontmatter reports zero findings across mathias/brain
  • Check whether the same truncation helper is used elsewhere; fix at the helper, not at the call site

Refs

  • #90 (frontmatter lint — how this surfaced), ingestion/internal/frontmatter
Found by the frontmatter lint shipped in #90. It is the one file in `mathias/brain` that the lint cannot classify, because it cannot be decoded at all. ## Symptom ``` wiki/index.md: not valid UTF-8 first bad byte at offset 590: b'...|Local Model Agentic Coding Capability \xe2\x80\xe2\x80\xa6\n- [[brain-wiki' ``` `\xe2\x80` is the first two bytes of a three-byte character (em-dash `\xe2\x80\x94`, ellipsis `\xe2\x80\xa6`, …) with its third byte missing — immediately followed by a complete `\xe2\x80\xa6`. So a title was cut **mid-character** and then the truncation marker was appended. That is the signature of truncating by **bytes** where the intent was characters: something like `s[:n] + "…"` over a byte slice, or a Go `string` sliced at a fixed index without a rune boundary check. ## Why it matters beyond one file - The file is generated, so patching the bytes fixes nothing — the next regeneration reproduces it. - Any consumer doing a strict decode (`yaml.safe_load`, `utf8.Valid`, most JSON encoders) rejects the whole index, not just the bad line. The frontmatter lint had to be written to report-and-continue specifically so this one file could not hide the other 17 findings. - Whatever writer does this is presumably used for other truncated fields too. The index is where it happened to become visible. ## Where to look The index writer that emits `wiki/index.md` — the `- [[wing-slug|Title]]` list lines. Find where a title gets shortened before the `…` is appended. ## Fix shape Truncate on a rune boundary, not a byte offset: ```go func truncate(s string, max int) string { if utf8.RuneCountInString(s) <= max { return s } return string([]rune(s)[:max]) + "…" } ``` ## Acceptance - [ ] `caveman: me cut long title between letters, not inside letter` - [ ] Red-first test: a title whose cut point lands inside a multi-byte character produces valid UTF-8 - [ ] `utf8.Valid` asserted over the generated index in a test — the property, not one example - [ ] `wiki/index.md` regenerates clean; `task lint:frontmatter` reports zero findings across `mathias/brain` - [ ] Check whether the same truncation helper is used elsewhere; fix at the helper, not at the call site ## Refs - #90 (frontmatter lint — how this surfaced), `ingestion/internal/frontmatter`
mathias commented 2026-08-18 05:20:08 +00:00
Author
Owner
Copy Link
Copy Source

Fixed — 125c99b, plus mathias/brain@8ef0564 regenerating the file. task check green.

  • Truncate on a rune boundary, not a byte offset
  • Red-first property test
  • utf8.Valid asserted as a property, not on one example
  • wiki/index.md regenerates clean; frontmatterlint reports 0 broken across the whole brain
  • Same helper used everywhere — fixed at the helper, not the call site

It was four call sites, not one

if len(line) > 100 { return line[:100] + "…" }

len() counts bytes, the slice cuts bytes, so on å ä ö (2 bytes) or an em dash (3) the cut lands mid-character:

Site Feeds
internal/wiki/index.go:66 wiki/index.md on disk
internal/mcp/tools_context.go:176 brain_context excerpts
internal/mcp/tools_context.go:197 brain_context excerpts
internal/search/search.go:431 brain_query excerpts

Only the first one was visible, and that is the part worth keeping. wiki/index.md is written raw to disk, so a strict decoder rejects the whole file — which is how the frontmatter lint found it. The other three feed MCP responses, where encoding/json silently substitutes U+FFFD. Those have been quietly eating the last character of every truncated Swedish excerpt, with no error, no log line, and nothing to notice — for however long they have existed.

So the loud bug was the lucky one. Fixing only index.go would have closed this issue and left three live instances feeding models slightly-corrupted text.

The test is the property, not the example

s := strings.Repeat("räv — ål — ökäd ", 20)
for limit := 0; limit <= len(s)+5; limit++ {
    if !utf8.ValidString(textutil.Truncate(s, limit)) { t.Fatalf(...) }
}

Walking every offset matters here specifically: a single example passes or fails on where that one cut happens to land, so an off-by-one could ship by picking a lucky boundary. Three more cases pin behaviour a naive rune fix gets wrong — the marker is appended only when something was actually dropped (an ellipsis on complete text is a lie about the content), the limit counts characters rather than bytes, and a non-positive limit returns empty instead of panicking.

The repair

8 entries in the index had an orphaned lead byte where an em dash was sliced in half:

-  …Local Model Agentic Coding Capability �…
+  …Local Model Agentic Coding Capability — …

Regenerated with the fixed writer rather than patched by hand — the file is generated, so hand-patching the bytes would have lasted until the next rebuild.

Closing.

Fixed — `125c99b`, plus `mathias/brain@8ef0564` regenerating the file. `task check` green. - [x] Truncate on a rune boundary, not a byte offset - [x] Red-first property test - [x] `utf8.Valid` asserted as a property, not on one example - [x] `wiki/index.md` regenerates clean; `frontmatterlint` reports **0 broken across the whole brain** - [x] Same helper used everywhere — fixed at the helper, not the call site ### It was four call sites, not one ```go if len(line) > 100 { return line[:100] + "…" } ``` `len()` counts bytes, the slice cuts bytes, so on `å ä ö` (2 bytes) or an em dash (3) the cut lands mid-character: | Site | Feeds | |---|---| | `internal/wiki/index.go:66` | `wiki/index.md` on disk | | `internal/mcp/tools_context.go:176` | `brain_context` excerpts | | `internal/mcp/tools_context.go:197` | `brain_context` excerpts | | `internal/search/search.go:431` | `brain_query` excerpts | **Only the first one was visible**, and that is the part worth keeping. `wiki/index.md` is written raw to disk, so a strict decoder rejects the whole file — which is how the frontmatter lint found it. The other three feed MCP responses, where `encoding/json` silently substitutes U+FFFD. Those have been quietly eating the last character of every truncated Swedish excerpt, with no error, no log line, and nothing to notice — for however long they have existed. So the loud bug was the lucky one. Fixing only `index.go` would have closed this issue and left three live instances feeding models slightly-corrupted text. ### The test is the property, not the example ```go s := strings.Repeat("räv — ål — ökäd ", 20) for limit := 0; limit <= len(s)+5; limit++ { if !utf8.ValidString(textutil.Truncate(s, limit)) { t.Fatalf(...) } } ``` Walking every offset matters here specifically: a single example passes or fails on where that one cut happens to land, so an off-by-one could ship by picking a lucky boundary. Three more cases pin behaviour a naive rune fix gets wrong — the marker is appended **only** when something was actually dropped (an ellipsis on complete text is a lie about the content), the limit counts characters rather than bytes, and a non-positive limit returns empty instead of panicking. ### The repair 8 entries in the index had an orphaned lead byte where an em dash was sliced in half: ``` - …Local Model Agentic Coding Capability �… + …Local Model Agentic Coding Capability — … ``` Regenerated with the fixed writer rather than patched by hand — the file is generated, so hand-patching the bytes would have lasted until the next rebuild. Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agent/ready
Meets the agent-ready contract; authorizes unattended execution
 blocked/client
Waiting on the client
 blocked/supplier
Waiting on a supplier
 blocked/upstream
Waiting on an upstream project or vendor
 risk/high
Irreversible, security-relevant, or control plane
 risk/low
Reversible, local, test-covered
 risk/medium
Touches shared state, deploys or data
 size/l
Too big - split it
 size/m
Under two days
 size/s
Under half a day
 size/xs
Under an hour
 status:dispatch
status:dispatched
status:review
status:reviewing
type/chore
Maintenance, no behaviour change
 type/decision
Human decision required; never agent/ready
 type/docs
Documentation only
 type/feat
New capability
 type/fix
Corrects broken behaviour
 type/ops
Infrastructure, deploy, runtime
 type/research
Investigate; output is a finding, not code
 type/test
Test coverage only
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
renovate-bot mathias
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/hyperguild#93