Extraction/canonicalization mass-attaches spurious [[mission]] link to unrelated tunnel-candidates notes #88

Closed
opened 2026-07-26 21:12:31 +00:00 by mathias · 1 comment
Owner

Problem

4 separate, topically-unrelated notes each got a [[mission]] link (→ wiki/telos/decisions/mission.md) added with zero real connection:

  • wiki/sources/tunnel-candidates-2026-07-19.md, -07-20.md, -07-22.md, -07-25.md — SSO decisions, BGP tunnel notes, security allowlist comparisons, none about TELOS mission.

Local-LLM semantic-relevance check flagged all 4 independently as hallucinated/no topical connection. Same target, same source-note type, recurring — one systemic cause, not 4 independent hallucinations.

Where to look

Two candidate mechanisms in ingestion/internal/pipeline/:

  1. links.go's CanonicalizeLinks — if the LLM output a bare [[mission]] (plain-link form) as a generic/filler mention, canonicalizeContent's title-lookup would resolve it against the inventory's "Mission" title and silently canonicalize it into a real link, rather than questioning whether the mention belongs there at all.
  2. The extraction prompt itself (schema.md / BuildPrompt) — if the schema's example output or instructions make "mission" look like a default/expected related-concept slot, the LLM may be pattern-matching it in from few-shot bias on tunnel-candidates-shaped inputs specifically.

Scope

  1. Reproduce against one of the 4 known inputs (or a fixture matching their shape) to confirm which mechanism is responsible.
  2. Fix the over-trigger (tighten schema/prompt example, or add a relevance check before canonicalizing a bare mention into a link).
  3. Remove the 4 existing bad links.

Acceptance criteria

  • Regression test: a tunnel-candidates-*-shaped fixture with no mission-related content does not get [[mission]] attached.
  • The 4 existing bad links removed.
## Problem 4 separate, topically-unrelated notes each got a `[[mission]]` link (→ `wiki/telos/decisions/mission.md`) added with zero real connection: - `wiki/sources/tunnel-candidates-2026-07-19.md`, `-07-20.md`, `-07-22.md`, `-07-25.md` — SSO decisions, BGP tunnel notes, security allowlist comparisons, none about TELOS mission. Local-LLM semantic-relevance check flagged all 4 independently as hallucinated/no topical connection. Same target, same source-note type, recurring — one systemic cause, not 4 independent hallucinations. ## Where to look Two candidate mechanisms in `ingestion/internal/pipeline/`: 1. `links.go`'s `CanonicalizeLinks` — if the LLM output a bare `[[mission]]` (plain-link form) as a generic/filler mention, `canonicalizeContent`'s title-lookup would resolve it against the inventory's "Mission" title and silently canonicalize it into a real link, rather than questioning whether the mention belongs there at all. 2. The extraction prompt itself (`schema.md` / `BuildPrompt`) — if the schema's example output or instructions make "mission" look like a default/expected related-concept slot, the LLM may be pattern-matching it in from few-shot bias on `tunnel-candidates`-shaped inputs specifically. ## Scope 1. Reproduce against one of the 4 known inputs (or a fixture matching their shape) to confirm which mechanism is responsible. 2. Fix the over-trigger (tighten schema/prompt example, or add a relevance check before canonicalizing a bare mention into a link). 3. Remove the 4 existing bad links. ## Acceptance criteria - Regression test: a `tunnel-candidates-*`-shaped fixture with no mission-related content does not get `[[mission]]` attached. - The 4 existing bad links removed.
Author
Owner

Root cause was neither of the two mechanisms this issue named. Investigated both directly:

  1. CanonicalizeLinks — correctly resolved [[Mission]] because wiki/concepts/mission.md genuinely exists as a concept page titled "Mission." Not a bug; it did its job.
  2. Extraction prompt/schema — no "mission" bias found in systemPrompt/BuildPrompt.

Actual mechanism, found by tracing where these tunnel-candidates-*.md notes come from: they're not normal ingested sources — they're internal/brain/tunnel.go's AutoTunnel/DetectTunnels output. DetectTunnels does a naive case-insensitive whole-word substring match of every wiki note's title against source content; "Mission" (a generic English word) fuzzy-matches unrelated content constantly, and those fuzzy (non-exact) matches get logged to brain/raw/tunnel-candidates-YYYY-MM-DD.md as a human-review queue (brain_tunnel to promote).

The bug: internal/watcher/watcher.go's processDir walks brain/raw/ and feeds every .md file to the LLM extraction pipeline — with no exclusion for tunnel-candidates-*.md. So the watcher re-ingested AutoTunnel's own review-queue file as if it were external source material. The LLM, reading a log that literally contains (term: "mission") entries, reasonably surfaced "mission" as a term in its summary — which CanonicalizeLinks then correctly turned into a real link, because it correctly is a real page, just not a relevant one.

Fixed in hyperguild@f005548: processDir now skips tunnel-candidates-* filenames, mirroring the exclusion api.ListPending already applied. Regression test reproduces the original bug (asserts the LLM Complete func is never called for a tunnel-candidates file) and confirms the fix.

Removed the 4 existing bad links in mathias/brain@95af3b7 — de-linked "mission" to plain text with a note explaining the fuzzy match, and trimmed one derivative hallucinated sentence in the 07-25 note that asserted a "broader mission" connection built on the bad link.

Root cause was **neither** of the two mechanisms this issue named. Investigated both directly: 1. **CanonicalizeLinks** — correctly resolved `[[Mission]]` because `wiki/concepts/mission.md` genuinely exists as a concept page titled "Mission." Not a bug; it did its job. 2. **Extraction prompt/schema** — no "mission" bias found in `systemPrompt`/`BuildPrompt`. Actual mechanism, found by tracing where these `tunnel-candidates-*.md` notes come from: they're not normal ingested sources — they're `internal/brain/tunnel.go`'s `AutoTunnel`/`DetectTunnels` output. `DetectTunnels` does a naive case-insensitive whole-word substring match of every wiki note's title against source content; "Mission" (a generic English word) fuzzy-matches unrelated content constantly, and those fuzzy (non-exact) matches get logged to `brain/raw/tunnel-candidates-YYYY-MM-DD.md` as a **human-review queue** (`brain_tunnel` to promote). The bug: `internal/watcher/watcher.go`'s `processDir` walks `brain/raw/` and feeds every `.md` file to the LLM extraction pipeline — with no exclusion for `tunnel-candidates-*.md`. So the watcher re-ingested AutoTunnel's own review-queue file as if it were external source material. The LLM, reading a log that literally contains `(term: "mission")` entries, reasonably surfaced "mission" as a term in its summary — which CanonicalizeLinks then correctly turned into a real link, because it correctly *is* a real page, just not a relevant one. Fixed in hyperguild@f005548: `processDir` now skips `tunnel-candidates-*` filenames, mirroring the exclusion `api.ListPending` already applied. Regression test reproduces the original bug (asserts the LLM `Complete` func is never called for a tunnel-candidates file) and confirms the fix. Removed the 4 existing bad links in `mathias/brain@95af3b7` — de-linked "mission" to plain text with a note explaining the fuzzy match, and trimmed one derivative hallucinated sentence in the 07-25 note that asserted a "broader mission" connection built on the bad link.
Sign in to join this conversation.
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/hyperguild#88