mathias/hyperguild
1
0
Fork 0
Code Issues 30 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

Brain interface: intent↔interface mismatch analysis (problem statement, evidence-backed) #43

New Issue
Closed
opened 2026-06-15 17:57:14 +00:00 by mathias · 2 comments
mathias commented 2026-06-15 17:57:14 +00:00
Owner
Copy Link
Copy Source

What this is

An evidence-backed problem statement for the brain interface, produced by a two-column intent analysis of real usage (autonomous agents + human-in-the-loop). It is deliberately not a redesign — the verb signatures, consumer split, and layer model are a separate, deliberate step left to the human. This issue exists so the finding doesn't evaporate before that step.

Evidence

  • Two-column analysis doc: mathias/brain branch analysis/intent-two-column, analysis/2026-06-15-brain-intent-two-column-analysis.md (commit 2ea5644).
  • Agent column: koala session logs, brain/sessions/analysis/agent-intent-column.{jsonl,md} (commit e8dbcf6).
  • Human column: Claude.ai conversation history, May–Jun 2026.

Method: classify each brain call by what it was trying to do (knowledge-act), not which tool it called; find intent↔interface mismatches.

Headline finding

Both columns, independently, ~37–38% mismatch with 0% genuine intent_unclear. Usage is not chaotic — the muddiness is in the interface, not the behaviour. The brain today is append + keyword-search: good at create and lexical read, missing update, missing promotion-as-operation, and serving semantic intent with a lexical (BM25) tool.

The structural root: the interface was never split by consumer, so the mechanism-named tools (brain_write, brain_query, brain_ingest, …) read as purpose-less — each serves whichever consumer happens to call it, with no statement of which knowledge-act it's for. And each consumer is missing a different verb.

The gaps (by consumer)

Shared — highest priority:

  • update / supersede — the seam. Agents 5/5 mismatch (blind-rewrite same slug → duplicates/contradictions). Humans 5+ (manual cherry-pick; brain PR #1 closed because content existed at two paths with no reconcile; re-ingest overwrites wholesale). The one act both consumers need and neither has. Highest-value gap. Note: distinct from #23 (embedding re-sync on edit) — this is a missing write verb, not an embedding-staleness fix.
  • Semantic vs lexical read. Both bend BM25 toward meaning-based intent. A true semantic read is missing.
  • Layer routing. knowledge/ vs wiki/ is a curation-state distinction (unreviewed vs promoted) with no verb to move a note across it — a noun with no verb. Both columns flag brain_ingest_raw ignoring the path prefix.

Agent-only:

  • verify_write_landed — 4/4 mismatch. No read-after-write; agents lexical-re-query their own just-written note to confirm it landed. Humans never hit this.
  • synthesized_answer. brain_answer distrusted — replaced by parallel brain_query.

Human-only:

  • curate / promote raw→wiki. Top human friction. Already tracked — see #38 (brain_pending + brain_promote). Not re-specced here.

Relationship to existing issues

  • #38 — specs the promote half (human curation gap). This issue does not duplicate it; it frames the broader interface problem it's one part of. (#37 was a duplicate of #38 — now closed.)
  • #45 — brain_update supersede-by-slug write verb + read-after-write (brain_get, brain_write handle). The implementation issue for the two highest-priority gaps named here.
  • #23 (closed) — embedding re-sync on edit; adjacent but not the update/supersede write verb.
  • #25 — skills/MCP-boundary overhaul; the MCP-vs-CLI verdict per server should account for the verb gaps named here.
  • #28 (closed) — shipped brain_context.

What's NOT in this issue

No verb signatures. No consumer-split design. No layer-model redesign. Those are the deliberate next step (notebook-first). This issue is the durable handle for the problem, with evidence attached.

Suggested done-when (for the problem-framing, not the fix)

  • Human reviews the two-column evidence doc and confirms/adjusts the gap list. (2026-06-22 — gap list confirmed; no adjustments.)
  • Decision recorded on whether update/supersede and verify_write_landed become tracked implementation issues. (2026-06-22 — both YES. update/supersede → #45 as supersede-by-slug whole-note replace, git as history layer. verify_write_landed → folded into #45 as brain_get + brain_write handle rather than its own issue.)
  • #37/#38 deduped and confirmed as the promote half. (2026-06-22 — #37 closed as duplicate; #38 retained.)
  • Redesign step scheduled (separate issue / ADR) once the verb set and consumer split are decided. (2026-08-22 — DECIDED, not merely scheduled. ADR-0009 Decision filled: no consumer split; split by authority instead. Verb set designed on #80. See the closing comment below.)

Decision log — 2026-06-22 review session

Worked the done-when sequence end to end:

  1. #37/#38 deduped — #37 closed (strictly weaker), #38 kept as canonical promote-half.
  2. update/supersede tracked — decision: YES. Shape = supersede-by-slug, whole-note replace (not patch, not append-tombstone) — agents regenerate whole notes, so patch-addressing solves a non-problem. Provenance = git is the history mechanism (no sidecar, no tombstone); superseded content recoverable but not queryable; frontmatter stamps the supersession.
  3. verify_write_landed tracked — decision: YES, but folded into the same issue (#45) as brain_get(id|path) + extending brain_write to return a handle. Not its own issue — it's a return-value change plus one getter.
  4. #45 filed with the above as the implementation target.

Still open: the redesign step (consumer-split / layer-model) below.

## What this is An evidence-backed **problem statement** for the brain interface, produced by a two-column intent analysis of real usage (autonomous agents + human-in-the-loop). It is deliberately *not* a redesign — the verb signatures, consumer split, and layer model are a separate, deliberate step left to the human. This issue exists so the finding doesn't evaporate before that step. ## Evidence - Two-column analysis doc: `mathias/brain` branch `analysis/intent-two-column`, `analysis/2026-06-15-brain-intent-two-column-analysis.md` (commit `2ea5644`). - Agent column: koala session logs, `brain/sessions/analysis/agent-intent-column.{jsonl,md}` (commit `e8dbcf6`). - Human column: Claude.ai conversation history, May–Jun 2026. Method: classify each brain call by *what it was trying to do* (knowledge-act), not which tool it called; find intent↔interface mismatches. ## Headline finding Both columns, independently, ~37–38% mismatch with **0% genuine `intent_unclear`**. Usage is not chaotic — the muddiness is in the interface, not the behaviour. The brain today is **append + keyword-search**: good at create and lexical read, missing update, missing promotion-as-operation, and serving semantic intent with a lexical (BM25) tool. The structural root: the interface was never **split by consumer**, so the mechanism-named tools (`brain_write`, `brain_query`, `brain_ingest`, …) read as purpose-less — each serves whichever consumer happens to call it, with no statement of which knowledge-act it's for. And each consumer is missing a *different* verb. ## The gaps (by consumer) **Shared — highest priority:** - **`update / supersede` — the seam.** Agents 5/5 mismatch (blind-rewrite same slug → duplicates/contradictions). Humans 5+ (manual cherry-pick; brain PR #1 closed because content existed at two paths with no reconcile; re-ingest overwrites wholesale). The one act both consumers need and neither has. **Highest-value gap.** Note: distinct from #23 (embedding re-sync on edit) — this is a missing *write verb*, not an embedding-staleness fix. - **Semantic vs lexical read.** Both bend BM25 toward meaning-based intent. A true semantic read is missing. - **Layer routing.** knowledge/ vs wiki/ is a *curation-state* distinction (unreviewed vs promoted) with **no verb to move a note across it** — a noun with no verb. Both columns flag `brain_ingest_raw` ignoring the path prefix. **Agent-only:** - **`verify_write_landed` — 4/4 mismatch.** No read-after-write; agents lexical-re-query their own just-written note to confirm it landed. Humans never hit this. - **`synthesized_answer`.** `brain_answer` distrusted — replaced by parallel `brain_query`. **Human-only:** - **`curate / promote raw→wiki`.** Top human friction. **Already tracked — see #38** (`brain_pending` + `brain_promote`). Not re-specced here. ## Relationship to existing issues - **#38** — specs the *promote* half (human curation gap). This issue does **not** duplicate it; it frames the broader interface problem it's one part of. (#37 was a duplicate of #38 — now closed.) - **#45** — `brain_update` supersede-by-slug write verb + read-after-write (`brain_get`, `brain_write` handle). The implementation issue for the two highest-priority gaps named here. - **#23** (closed) — embedding re-sync on edit; adjacent but not the `update/supersede` write verb. - **#25** — skills/MCP-boundary overhaul; the MCP-vs-CLI verdict per server should account for the verb gaps named here. - **#28** (closed) — shipped `brain_context`. ## What's NOT in this issue No verb signatures. No consumer-split design. No layer-model redesign. Those are the deliberate next step (notebook-first). This issue is the durable handle for the problem, with evidence attached. ## Suggested done-when (for the problem-framing, not the fix) - [x] Human reviews the two-column evidence doc and confirms/adjusts the gap list. *(2026-06-22 — gap list confirmed; no adjustments.)* - [x] Decision recorded on whether `update/supersede` and `verify_write_landed` become tracked implementation issues. *(2026-06-22 — both YES. `update/supersede` → #45 as supersede-by-slug whole-note replace, git as history layer. `verify_write_landed` → folded into #45 as `brain_get` + `brain_write` handle rather than its own issue.)* - [x] #37/#38 deduped and confirmed as the promote half. *(2026-06-22 — #37 closed as duplicate; #38 retained.)* - [x] Redesign step scheduled (separate issue / ADR) once the verb set and consumer split are decided. *(2026-08-22 — DECIDED, not merely scheduled. ADR-0009 Decision filled: no consumer split; split by authority instead. Verb set designed on #80. See the closing comment below.)* --- ## Decision log — 2026-06-22 review session Worked the done-when sequence end to end: 1. **#37/#38 deduped** — #37 closed (strictly weaker), #38 kept as canonical promote-half. 2. **`update/supersede` tracked** — decision: YES. Shape = supersede-by-slug, whole-note replace (not patch, not append-tombstone) — agents regenerate whole notes, so patch-addressing solves a non-problem. Provenance = git is the history mechanism (no sidecar, no tombstone); superseded content recoverable but not queryable; frontmatter stamps the supersession. 3. **`verify_write_landed` tracked** — decision: YES, but folded into the same issue (#45) as `brain_get(id|path)` + extending `brain_write` to return a handle. Not its own issue — it's a return-value change plus one getter. 4. **#45 filed** with the above as the implementation target. Still open: the redesign step (consumer-split / layer-model) below.
mathias commented 2026-08-22 06:41:57 +00:00
Author
Owner
Copy Link
Copy Source

Last box closed — the redesign step is decided, not just scheduled

This issue's final done-when read "redesign step scheduled (separate issue / ADR) once the verb set and consumer split are decided." Ticked, and stronger than written: the decision is made rather than merely calendared.

ADR-0009 Decision is filled (infra PR #296): no consumer split — split by authority instead.

The interesting part is that this issue's own evidence argued against the redesign it pointed at. #43/#44 is naturally read as "two consumers, ~37–38% mismatch each, therefore split the surface by consumer." Re-read for which acts each column performs rather than which acts each column fails at, four of the five gaps are shared — update/supersede (the highest-value one) is shared, semantic-vs-lexical is shared, write-schema confusion is shared. Only promote raw→wiki is genuinely consumer-specific. So the consumer axis explains where each side hurts, not what each side does, and splitting the verb set on it would duplicate the shared surface to express one act.

verify_write_landed — this issue's cleanest agent-only finding, 4/4 mismatch — is what settled it. It is not a missing verb; it is a missing return value, already solved by #45's handle plus brain_get. Had the consumer split been done in June as framed, that gap would plausibly have become a permanent agent-only verify verb encoding another verb's temporary deficiency. The framing was right to be captured and right to be left open.

What replaces it: the knowledge/ → wiki/ seam and instruction-carrying files require a human principal regardless of who initiates, enforced server-side — reusing the brain_edit_root gate shipped in #96/#97. A model that has just read an untrusted page must not be able to promote that page's claims into curated space where later readers treat them as reviewed. That is a capability boundary; consumer-splitting the names would not have provided it.

Also worth recording against this issue's own diagnosis: the authority gate deciding the layer is what finally turns "layer routing is a noun with no verb" into a verb (promote/reject), and it removes the brain_ingest_raw-ignores-the-path-prefix class you flagged, because no caller states a layer any longer.

Not fixed, and not claimed to be — the other two clusters this issue named:

  • semantic vs lexical retrieval — answered structurally (one recall verb with a strategy hint) but the ranking quality gap is untouched. Renaming a lexical tool does not make it semantic.
  • brain_answer distrust — orthogonal. Agents re-queried within a minute because synthesis was not good enough, not because the tool was misnamed. recall's strategy hint is deliberately escapable so this cannot be papered over.

Both stay open on their own merits and should not be considered addressed by the redesign.

Verb-by-verb design: #80. Implementation is not started and is gated on review of ADR-0009 + #80.

With this box ticked, every done-when on this issue is complete. Closing is your call — the problem statement it holds is still the best single description of why the surface is shaped the way it is, so there may be value in leaving it as the durable handle it was filed to be.

## Last box closed — the redesign step is decided, not just scheduled This issue's final done-when read *"redesign step scheduled (separate issue / ADR) once the verb set and consumer split are decided."* Ticked, and stronger than written: the decision is made rather than merely calendared. **ADR-0009 Decision is filled** (infra PR #296): *no consumer split — split by authority instead.* The interesting part is that this issue's own evidence argued **against** the redesign it pointed at. #43/#44 is naturally read as "two consumers, ~37–38% mismatch each, therefore split the surface by consumer." Re-read for *which acts each column performs* rather than *which acts each column fails at*, four of the five gaps are shared — update/supersede (the highest-value one) is shared, semantic-vs-lexical is shared, write-schema confusion is shared. Only promote raw→wiki is genuinely consumer-specific. So the consumer axis explains where each side hurts, not what each side does, and splitting the verb set on it would duplicate the shared surface to express one act. `verify_write_landed` — this issue's cleanest agent-only finding, 4/4 mismatch — is what settled it. It is not a missing verb; it is a missing **return value**, already solved by #45's handle plus `brain_get`. Had the consumer split been done in June as framed, that gap would plausibly have become a permanent agent-only `verify` verb encoding another verb's temporary deficiency. The framing was right to be captured and right to be left open. What replaces it: the `knowledge/` → `wiki/` seam and instruction-carrying files require a **human principal regardless of who initiates**, enforced server-side — reusing the `brain_edit_root` gate shipped in #96/#97. A model that has just read an untrusted page must not be able to promote that page's claims into curated space where later readers treat them as reviewed. That is a capability boundary; consumer-splitting the names would not have provided it. **Also worth recording against this issue's own diagnosis:** the authority gate deciding the *layer* is what finally turns "layer routing is a noun with no verb" into a verb (`promote`/`reject`), and it removes the `brain_ingest_raw`-ignores-the-path-prefix class you flagged, because no caller states a layer any longer. **Not fixed, and not claimed to be** — the other two clusters this issue named: - **semantic vs lexical retrieval** — answered structurally (one `recall` verb with a strategy hint) but the ranking quality gap is untouched. Renaming a lexical tool does not make it semantic. - **`brain_answer` distrust** — orthogonal. Agents re-queried within a minute because synthesis was not good enough, not because the tool was misnamed. `recall`'s strategy hint is deliberately escapable so this cannot be papered over. Both stay open on their own merits and should not be considered addressed by the redesign. Verb-by-verb design: #80. Implementation is not started and is gated on review of ADR-0009 + #80. With this box ticked, every done-when on this issue is complete. Closing is your call — the problem statement it holds is still the best single description of why the surface is shaped the way it is, so there may be value in leaving it as the durable handle it was filed to be.
mathias added the risk/lowsize/mtype/research labels 2026-08-27 05:25:41 +00:00
mathias commented 2026-10-04 20:52:21 +00:00
Author
Owner
Copy Link
Copy Source

Closing — verified live 2026-10-04: all four done-when boxes in the body are checked. The last one was ticked 2026-08-22, when ADR-0009's Decision chose to split by authority, not by consumer, with the verb set designed on #80. The two clusters explicitly left unaddressed (semantic ranking quality, brain_answer distrust) stand on their own merits and are not part of this problem-framing issue. Closed issues stay searchable, so this still works as the durable handle for why the surface has its current shape.

Created via git-mcp on behalf of @mathias

Closing — verified live 2026-10-04: all four done-when boxes in the body are checked. The last one was ticked 2026-08-22, when ADR-0009's Decision chose to split by authority, not by consumer, with the verb set designed on #80. The two clusters explicitly left unaddressed (semantic ranking quality, `brain_answer` distrust) stand on their own merits and are not part of this problem-framing issue. Closed issues stay searchable, so this still works as the durable handle for why the surface has its current shape. --- _Created via git-mcp on behalf of @mathias_
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agent/ready
Meets the agent-ready contract; authorizes unattended execution
 blocked/client
Waiting on the client
 blocked/supplier
Waiting on a supplier
 blocked/upstream
Waiting on an upstream project or vendor
 risk/high
Irreversible, security-relevant, or control plane
 risk/low
Reversible, local, test-covered
 risk/medium
Touches shared state, deploys or data
 size/l
Too big - split it
 size/m
Under two days
 size/s
Under half a day
 size/xs
Under an hour
 status:dispatch
status:dispatched
status:review
status:reviewing
type/chore
Maintenance, no behaviour change
 type/decision
Human decision required; never agent/ready
 type/docs
Documentation only
 type/feat
New capability
 type/fix
Corrects broken behaviour
 type/ops
Infrastructure, deploy, runtime
 type/research
Investigate; output is a finding, not code
 type/test
Test coverage only
No labels risk/low size/m type/research
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
renovate-bot mathias
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/hyperguild#43