docs(brain): add agent-consumer brain-MCP intent analysis
Agent-consumer column of the two-part brain intent↔interface study. Reconstructs the knowledge-act behind every brain MCP call in the Claude Code agent transcripts on koala (the only corpus with brain calls; brain/sessions and agentsquad eval logs carry none). 46 distinct knowledge-acts, 37% interface mismatch, 0% intent_unclear. Headline gap: no update/supersede verb → agents blind re-write same slug (5x); no read-after-write → lexical re-query of own note (4x); lexical-only reads → semantic-as-keyword-stuffing chains (3x); brain_answer hedged with parallel brain_query. Canonical schema brain-intent-extraction.md absent on host; vocab reconstructed, every row tagged schema_source=reconstructed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.8
@@ -0,0 +1,48 @@
|
||||
{"_meta":true,"note":"Agent-consumer column of brain-MCP intent analysis. consumer_type fixed=autonomous_agent. CAVEAT: canonical schema file brain-intent-extraction.md is NOT present on this host (koala) — only this session's own task prompt references it. The closed intent vocabulary below was RECONSTRUCTED from the task prompt's framing + brain/schema.md. Re-map intent labels if the canonical vocab differs. schema_source=reconstructed on every row.","closed_intent_vocab":["semantic_retrieval","lexical_lookup","check_prior_art","synthesized_answer","store_new_knowledge","update_or_supersede","ingest_raw_source","verify_write_landed","discover_capability","intent_unclear"],"intent_tool_match_values":["match","mismatch","partial"],"corpus":"~/.claude/projects/*/*.jsonl (Claude Code agent transcripts on koala). brain/sessions/*.jsonl empty. agentsquad docs/eval/*.jsonl are code-review eval results, NOT brain calls. No separate Crush logs found. Zero brain calls appear under any mcp__ name with a human typing the call — all brain acts are agent-initiated (CLAUDE.md reflex), so all qualify as autonomous_agent."}
|
||||
{"id":"a01","session":"tapir-c","ts":"2026-06-?T15:01:51","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"single pre-task query 'YouTube Data API captions download ownership limitation timedtext adapter Go' — named-entity lexical lookup, fit BM25 well, no reformulation."}
|
||||
{"id":"a02","session":"tapir","ts":"2026-06-05T21:44:10","tool":"brain_ingest","intent":"ingest_raw_source","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_ingest 14s earlier — tool not ambient, had to be discovered/loaded first.","evidence":"source=tapir-scheduled-discovery-session-2026-06-05, a session learnings dump."}
|
||||
{"id":"a03","session":"tapir","ts":"2026-06-?T14:00:59","tool":"brain_ingest","intent":"ingest_raw_source","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"source=tapir-rls-identity-bootstrapping, first write of RLS lesson."}
|
||||
{"id":"a04","session":"tapir","ts":"2026-06-?T14:02:08","tool":"brain_ingest","intent":"update_or_supersede","intent_tool_match":"mismatch","workaround":"RE-INGESTED same source name 'tapir-rls-identity-bootstrapping' 69s later with edited/condensed body. No update/patch/supersede verb exists, so the agent overwrote-by-re-ingest. Whether this dedups or creates a v2 duplicate is opaque to the agent.","observed_friction":"agent revised content within 70s of first write — classic edit-after-write with no edit primitive.","evidence":"two brain_ingest, identical source string, divergent content."}
|
||||
{"id":"a05","session":"tapir","ts":"2026-06-?T14:56:27","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"postgres-cascade-skips-tables-without-fk.md — distinct new lesson."}
|
||||
{"id":"a06","session":"tapir","ts":"2026-06-?T21:08:57","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_query — discovery tax again.","evidence":"'Dex passwords.dex.coreos.com CRD ...' keyword-rich, single shot."}
|
||||
{"id":"a07","session":"AI-infra","ts":"2026-05-?T15:38:03","tool":"brain_query(HTTP-curl)","intent":"discover_capability","intent_tool_match":"mismatch","workaround":"raw `curl -X POST` to brain-mcp endpoint instead of MCP tool. Preceded by two ToolSearch ('brain knowledge memory' then 'brain') that did not yield a usable loaded tool, so agent fell back to HTTP.","observed_friction":"3-step ladder: ToolSearch 'brain knowledge memory' -> ToolSearch 'brain' -> curl. Agent did not know which act maps to which tool name.","evidence":"curl -s -o /tmp/brain-init.txt -w code:%{http_code} -X POST ..."}
|
||||
{"id":"a08","session":"AI-infra","ts":"2026-05-?T04:46:13","tool":"brain_query(HTTP-curl)","intent":"discover_capability","intent_tool_match":"mismatch","workaround":"hand-set TOKEN=... then curl brain-test endpoint — probing whether the HTTP brain path is reachable/authed at all. MCP path not used.","observed_friction":"agent testing connectivity by hand; MCP auth/availability not trusted.","evidence":"TOKEN=...; curl -s -o /tmp/brain-test ..."}
|
||||
{"id":"a09","session":"AI-infra","ts":"2026-05-?T05:23:35","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_query (after an earlier 05:03 ToolSearch 'brain ingestion knowledge wiki' that explored layers).","evidence":"'koala machine state RTX 5070 llama-swap' — named-entity recall, fits lexical."}
|
||||
{"id":"a10","session":"AI-infra","ts":"2026-05-?T05:27:36","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"batch of 3 in ~1s (k3s/flux gitops; llama-swap ai-stack GPU; MCP Dex OAuth claude.ai) — parallel prior-art sweep, all named-entity."}
|
||||
{"id":"a11","session":"AI-infra","ts":"2026-05-?T07:13:50","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"batch of 4 in ~2s before a debugging session (flux healthCheck; exit 255 restart loop; NVML mismatch; mirror rebase). Named symptoms, lexical fit OK on first pass."}
|
||||
{"id":"a12","session":"AI-infra","ts":"2026-05-?T07:14:26","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"4 writes in ~35s (flux-healthcheck-stale; exit-255-unknown-reason-not-oom; nvidia-nvml-mismatch; mcp-static-bearer) — answers to the 4 queries just run, captured as lessons. Healthy query->fix->write loop."}
|
||||
{"id":"a13","session":"AI-infra","ts":"2026-05-?T07:15:25","tool":"brain_query","intent":"verify_write_landed","intent_tool_match":"mismatch","workaround":"25s after writing exit-255-unknown-reason-not-oom.md, re-queried 'exit 255 unknown SIGKILL containerd' — reformulated terms (SIGKILL/containerd not in original query 'exit 255 unknown reason restart loop diagnosis'). Either confirming the fresh write is retrievable or re-searching because first lexical query missed. No read-after-write / get-by-id act exists.","observed_friction":"reformulation chain: 'exit 255 unknown reason restart loop diagnosis' -> 'exit 255 unknown SIGKILL containerd'. Same need, different keywords.","evidence":"query at 07:13:51 vs 07:15:25 bracketing the 07:14:37 write."}
|
||||
{"id":"a14","session":"AI-infra","ts":"2026-05-?T07:22:55","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"batch of 5 in ~20s before a homelab security audit (piguard/iguana tailscale; unifi UCG firewall; SOPS age; ingress TLS cert-manager; koala UFW iptables). Named-entity sweep."}
|
||||
{"id":"a15","session":"AI-infra","ts":"2026-05-?T09:27:53","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"audit-shortcut-tls-blocks-zero; policy-audit-mode-blocks-nothing — distinct new audit lessons."}
|
||||
{"id":"a16","session":"AI-infra","ts":"2026-05-?T09:28:22","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"homelab-security-chains-not-bugs.md FIRST write (worked example: koala 2026-05-13)."}
|
||||
{"id":"a17","session":"AI-infra","ts":"2026-05-?T10:32:49","tool":"brain_write","intent":"update_or_supersede","intent_tool_match":"mismatch","workaround":"RE-WROTE homelab-security-chains-not-bugs.md ~64min later with a different/expanded worked example (host-user dotfile, over-broad ClusterRole). Same filename, additive revision, no patch/append/supersede verb — agent overwrites and hopes the index replaces rather than duplicates.","observed_friction":"the in-between hour of audit work produced a better example; only way to fold it in was a full re-write of the same slug.","evidence":"two brain_write same filename at 09:28:22 and 10:32:49, divergent worked examples."}
|
||||
{"id":"a18","session":"AI-infra","ts":"2026-05-?T10:18:25","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"homelab-document-accepted-risk-to-break-audit-cycle.md — distinct."}
|
||||
{"id":"a19","session":"AI-infra","ts":"2026-05-?T10:32:55","tool":"brain_query","intent":"verify_write_landed","intent_tool_match":"mismatch","workaround":"6s after the homelab-chains re-write, queried 'RBAC MCP cluster pods log chain' — checking the chain reasoning is retrievable / finding the related entry. Read-after-write done via lexical search.","observed_friction":null,"evidence":"query immediately follows the 10:32:49 write."}
|
||||
{"id":"a20","session":"AI-infra","ts":"2026-05-?T18:57:34","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_write,brain_query — re-discovered tools this session.","evidence":"'extension build pinned version major version upgrade postgres pgvector'."}
|
||||
{"id":"a21","session":"AI-infra","ts":"2026-05-?T18:57:58","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"extension-version-lags-platform-major-upgrade.md."}
|
||||
{"id":"a22","session":"AI-infra","ts":"2026-05-?T18:58:06","tool":"brain_query","intent":"verify_write_landed","intent_tool_match":"mismatch","workaround":"8s after writing extension-version-lags, re-queried 'pgvector postgres extension version compile error bump' — reformulated from the 18:57:34 query ('extension build pinned version...'). Lexical re-search to confirm the just-written lesson is findable, with different keyword guess.","observed_friction":"reformulation: 'extension build pinned version major version upgrade postgres pgvector' -> 'pgvector postgres extension version compile error bump'.","evidence":"write at 18:57:58 bracketed by queries 18:57:34 and 18:58:06."}
|
||||
{"id":"a23","session":"AI-infra","ts":"2026-05-?T18:31:23","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"webfetch-readme-when-image-or-flag-uncertain.md FIRST write."}
|
||||
{"id":"a24","session":"AI-infra","ts":"2026-05-?T18:34:23","tool":"brain_write","intent":"update_or_supersede","intent_tool_match":"mismatch","workaround":"RE-WROTE webfetch-readme-when-image-or-flag-uncertain.md 3min later, near-identical body. Looks like a retry/overwrite (uncertain the first landed, or minor edit). No idempotent upsert with confirmation, so agent re-fires the write.","observed_friction":"followed 7s later by a brain_query on the same topic ('OSS tool image registry CLI flag webhook path schema drift README pre-flight') — write-write-query, i.e. overwrite then verify-by-search.","evidence":"two brain_write same filename 18:31:23 / 18:34:23, then query 18:34:31."}
|
||||
{"id":"a25","session":"AI-infra","ts":"2026-05-?T18:34:31","tool":"brain_query","intent":"verify_write_landed","intent_tool_match":"mismatch","workaround":"keyword-stuffed lexical query 'OSS tool image registry CLI flag webhook path schema drift README pre-flight' fired right after the webfetch-readme write — agent dumps every concept token hoping BM25 surfaces its own fresh note. This is semantic intent (find that conceptual lesson) coerced into a bag-of-keywords.","observed_friction":"query is a concatenation of the note's section headings — a tell that the agent is groping lexically for content it knows by meaning.","evidence":"query text mirrors the just-written note's bullet topics."}
|
||||
{"id":"a26","session":"dev","ts":"2026-06-?T21:18:26","tool":"brain_answer","intent":"synthesized_answer","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_answer.","evidence":"'tapir transcript persistence shared cross-user dedup table RLS isolation ADR-021 ...' -> 22min later a brain_write (acted on the answer). Answer consumed, not re-queried. Healthy."}
|
||||
{"id":"a27","session":"dev","ts":"2026-06-?T21:40:20","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"tapir-migration-and-rls-test-infra-gotchas, with wing/hall absent here (flat) — see schema-confusion note a40."}
|
||||
{"id":"a28","session":"dev","ts":"2026-06-?T05:35:04","tool":"brain_answer","intent":"synthesized_answer","intent_tool_match":"partial","workaround":"asked 'gitea MCP not working workaround file issue via API which token ... how to authenticate gitea API' — a how-do-I question. Next brain act (05:39 query) is a different topic (tapir transcript), so the answer was apparently sufficient OR abandoned; ambiguous.","observed_friction":null,"evidence":"brain_answer then unrelated brain_query 4min later."}
|
||||
{"id":"a29","session":"dev","ts":"2026-06-?T05:39:54","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"'tapir transcript persistence ADR-021 shared non-RLS'."}
|
||||
{"id":"a30","session":"dev","ts":"2026-06-?T05:57:37","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"gitea-mcp-per-repo-tools-404-and-rest-fallback FIRST write."}
|
||||
{"id":"a31","session":"dev","ts":"2026-06-?T05:57:55","tool":"brain_write","intent":"update_or_supersede","intent_tool_match":"mismatch","workaround":"RE-WROTE gitea-mcp-per-repo-tools-404-and-rest-fallback 18s later — overwrite/retry of same slug, no upsert confirmation.","observed_friction":"sub-20s gap = almost certainly a content tweak the agent could not express as an edit.","evidence":"two brain_write same filename 05:57:37 / 05:57:55."}
|
||||
{"id":"a32","session":"dev","ts":"2026-05-?T11:51:47","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"infra-litellm-absorption-2026-05-16.md."}
|
||||
{"id":"a33","session":"dev","ts":"2026-05-?T12:07:04","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"batch of 3 (litellm rebuild time piguard; docker compose orphaned volumes; prometheus_client ModuleNotFoundError) — lexical, error-string driven."}
|
||||
{"id":"a34","session":"dev","ts":"2026-05-?T15:08:15","tool":"brain_answer","intent":"synthesized_answer","intent_tool_match":"mismatch","workaround":"THREE brain_answer at 15:08 (moved compose volumes? / pi rebuild time? / litellm ModuleNotFound prometheus) — the SAME three topics queried lexically an hour earlier (12:07) — were IMMEDIATELY followed at 15:09 by THREE brain_query on the same three topics. The agent asked the synthesizer, was unsatisfied, and fell straight back to raw lexical search. Strongest answer->query fallback in the corpus.","observed_friction":"answer/query duplication across one intent: agent hedges by firing both interfaces, trusting neither.","evidence":"15:08 answers vs 15:09 queries, topic-for-topic aligned."}
|
||||
{"id":"a35","session":"dev","ts":"2026-05-?T15:09:16","tool":"brain_query","intent":"semantic_retrieval","intent_tool_match":"mismatch","workaround":"after the 3 brain_answer calls failed to satisfy, re-issued as lexical brain_query ('moved compose stack to new directory volumes disappeared empty'; 'raspberry pi docker build time arm slow'; 'how to enable prometheus metrics on litellm proxy callback'). The want is meaning-based ('did my volumes move?') but the only retrieval that 'worked' was keyword search — and these are full natural-language sentences crammed into a BM25 box.","observed_friction":"natural-language questions ('how to enable...', 'moved ... disappeared') passed to a lexical query tool — semantic intent, lexical interface.","evidence":"3 queries at 15:09 mirror the 3 answers at 15:08."}
|
||||
{"id":"a36","session":"dev","ts":"2026-05-?T20:31:50","tool":"brain_answer","intent":"synthesized_answer","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"'What happened with the litellm migration on 2026-05-16?' — episodic recall question, answer fit; no re-query followed."}
|
||||
{"id":"a37","session":"dev","ts":"2026-05-?T21:07:17","tool":"brain_query","intent":"semantic_retrieval","intent_tool_match":"mismatch","workaround":"FOUR-step reformulation chain over one Go bug: 'bytes.Buffer Bytes Reset aliasing slice sharing' -> 'go buffer reuse map backing array bug' -> [write go-bytes-buffer-bytes-reset-aliasing-trap.md] -> 'go map values all show same content after loop' -> 'bytes.Buffer Bytes returns same data every iteration'. The agent knows the SYMPTOM (all map values identical) and the CAUSE (Bytes() aliasing) but cannot phrase a single lexical query that bridges them — it wants concept retrieval and is forced to brute-force keyword variants.","observed_friction":"4 distinct phrasings of the same bug, two before and two after writing the lesson — also doubles as verify_write_landed on the trailing queries.","evidence":"21:07:17, 21:07:17, (write 21:08:01), 21:08:10, 21:08:19."}
|
||||
{"id":"a38","session":"dev","ts":"2026-05-?T21:08:01","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"go-bytes-buffer-bytes-reset-aliasing-trap.md."}
|
||||
{"id":"a39","session":"dev","ts":"2026-05-?T07:54:26","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"mcp-tool-design-get-needs-list-partner.md — a design principle."}
|
||||
{"id":"a40","session":"dev","ts":"2026-06-?T06:25:00","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"'Dex to Authentik migration auth.d-ma.be issuer cutover OIDC subject ...'."}
|
||||
{"id":"a41","session":"dev","ts":"2026-06-?T06:25:08","tool":"brain_answer","intent":"synthesized_answer","intent_tool_match":"mismatch","workaround":"brain_query (a40) and brain_answer (a41) fired ~8s apart on the SAME intent (Dex->Authentik subject-keyed token orphan). Agent runs lexical search AND synthesized answer in parallel for one question rather than choosing — it cannot predict which interface will return usable knowledge, so it pays both.","observed_friction":"query+answer doublet on one need.","evidence":"06:25:00 query then 06:25:08 answer, same topic."}
|
||||
{"id":"a42","session":"dev","ts":"2026-06-?T13:48:49","tool":"brain_query","intent":"verify_write_landed","intent_tool_match":"mismatch","workaround":"'authentik cutover validation probe' then 6min later 'authentik cutover post-flip validation' — reformulated pair, likely searching for the agent's own earlier cutover notes / confirming validation steps are recorded. Lexical re-search standing in for recall-my-recent-context.","observed_friction":"reformulation: 'validation probe' -> 'post-flip validation'.","evidence":"13:48:49 and 13:54:56."}
|
||||
{"id":"a43","session":"dev","ts":"2026-06-?T13:59:17","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_write.","evidence":"oidc-issuer-host-change-vs-idp-swap-subject."}
|
||||
{"id":"a44","session":"dev","ts":"2026-06-?T13:59:50","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"cannot-move-ingress-host-across-namespaces-flux-dryrun FIRST write."}
|
||||
{"id":"a45","session":"dev","ts":"2026-06-?T14:00:13","tool":"brain_write","intent":"update_or_supersede","intent_tool_match":"mismatch","workaround":"RE-WROTE cannot-move-ingress-host-across-namespaces-flux-dryrun 23s later — overwrite of same slug, no edit/upsert primitive.","observed_friction":"sub-30s gap = content correction expressed as a full re-write.","evidence":"two brain_write same filename 13:59:50 / 14:00:13."}
|
||||
{"id":"a46","session":"dev","ts":"2026-06-?T13:53:54","tool":"brain_write(HTTP-staged)","intent":"store_new_knowledge","intent_tool_match":"mismatch","workaround":"after `ToolSearch select:mcp__claude_ai_brain__authenticate` (MCP auth flow), the agent staged the entry as `cat > /tmp/brain_entry.json` ({filename:'postgres-force-rls-cross-u...', content}) for a curl write rather than calling brain_write directly — MCP write path was not usable (auth/loading), so it dropped to the HTTP bodge.","observed_friction":"reached for an 'authenticate' tool, then abandoned MCP for hand-built JSON + curl. Matches known pattern: brain/op MCP auth lapses often.","evidence":"ToolSearch authenticate 13:53:27 -> cat /tmp/brain_entry.json 13:53:54."}
|
||||
{"id":"a47","session":"template-go-agent","ts":"2026-05-?T18:46:26","tool":"BASH(not-a-brain-act)","intent":"intent_unclear","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"'brain_' substring was inside a git commit message body ('agent boundaries, network policy, agent s...'), NOT a brain call. Excluded from knowledge-act analysis; logged for audit completeness."}
|
||||
@@ -0,0 +1,148 @@
|
||||
# Agent-Consumer Brain Intent Analysis — koala column
|
||||
|
||||
**Consumer:** `autonomous_agent` (all rows). **Host:** koala. **Date:** 2026-06-15.
|
||||
**Raw rows:** `agent-intent-column.jsonl` (46 real knowledge-acts + 1 excluded false-positive).
|
||||
|
||||
## Caveat — canonical schema not on this host
|
||||
|
||||
The shared closed-vocabulary file `brain-intent-extraction.md` **does not exist on
|
||||
koala** — the only reference to it is inside *this task's own prompt*. The intent
|
||||
vocabulary below was **reconstructed** from the prompt's framing + `brain/schema.md`.
|
||||
Every row carries `schema_source: reconstructed`. If the canonical vocab differs,
|
||||
re-map the `intent` field; the `intent_tool_match` / `workaround` / `observed_friction`
|
||||
evidence stands regardless of label names.
|
||||
|
||||
**Reconstructed closed vocab:** `semantic_retrieval`, `lexical_lookup`,
|
||||
`check_prior_art`, `synthesized_answer`, `store_new_knowledge`, `update_or_supersede`,
|
||||
`ingest_raw_source`, `verify_write_landed`, `discover_capability`, `intent_unclear`.
|
||||
|
||||
## Corpus
|
||||
|
||||
- `~/.claude/projects/*/*.jsonl` — Claude Code agent transcripts (98 files). **The only
|
||||
source with brain calls.**
|
||||
- `brain/sessions/*.jsonl` — empty (only `.gitkeep`).
|
||||
- `agentsquad docs/eval/*.jsonl` — code-review eval results, **not** brain calls.
|
||||
- No separate Crush session logs on this host.
|
||||
- **Zero** brain calls were human-typed. Every brain act is agent-initiated (the
|
||||
CLAUDE.md "query as reflex / close-the-loop write" behaviour), so all qualify as
|
||||
`autonomous_agent`. The human gave the top-level task; the agent chose every brain act.
|
||||
|
||||
## 1. Intent histogram, split by `intent_tool_match`
|
||||
|
||||
| intent | match | mismatch | partial | total |
|
||||
|---|---|---|---|---|
|
||||
| check_prior_art | 10 | 0 | 0 | 10 |
|
||||
| store_new_knowledge | 14 | 1 | 0 | 15 |
|
||||
| update_or_supersede | 0 | 5 | 0 | 5 |
|
||||
| synthesized_answer | 2 | 2 | 1 | 5 |
|
||||
| verify_write_landed | 0 | 4 | 0 | 4 |
|
||||
| semantic_retrieval | 0 | 3 | 0 | 3 |
|
||||
| ingest_raw_source | 2 | 0 | 0 | 2 |
|
||||
| discover_capability | 0 | 2 | 0 | 2 |
|
||||
| **total** | **28** | **17** | **1** | **46** |
|
||||
|
||||
> Batch note: several rows collapse a same-second fan-out of identical-intent calls
|
||||
> (a10=3, a11=4, a14=5, a33=3, a34=3, a35=3). Call-level the corpus is ~62 brain calls;
|
||||
> the table counts the 46 distinct knowledge-acts. Frequency is deliberately *not* the
|
||||
> point — the mismatch column is.
|
||||
|
||||
**37% of agent knowledge-acts (17/46) are interface mismatches.** Every mismatch falls
|
||||
into one of four intents: `update_or_supersede`, `verify_write_landed`,
|
||||
`semantic_retrieval`, `discover_capability` — plus one `store` that had to use HTTP.
|
||||
|
||||
## 2. Mismatch list, grouped by intent (primary deliverable)
|
||||
|
||||
### update_or_supersede → re-write same slug (5/5 mismatch) — HIGHEST VALUE
|
||||
There is **no update / patch / append / supersede verb**. When an agent improves a note
|
||||
it already wrote, the only move is to call `brain_write`/`brain_ingest` **again with the
|
||||
same filename/source** and hope the index replaces rather than duplicates. Observed:
|
||||
|
||||
| slug | 1st write | 2nd write | gap | what changed |
|
||||
|---|---|---|---|---|
|
||||
| `tapir-rls-identity-bootstrapping` (ingest) | 14:00:59 | 14:02:08 | 69s | condensed body |
|
||||
| `homelab-security-chains-not-bugs.md` | 09:28:22 | 10:32:49 | 64m | new worked example |
|
||||
| `webfetch-readme-when-image-or-flag-uncertain.md` | 18:31:23 | 18:34:23 | 3m | near-identical (retry) |
|
||||
| `gitea-mcp-per-repo-tools-404-and-rest-fallback` | 05:57:37 | 05:57:55 | 18s | content tweak |
|
||||
| `cannot-move-ingress-host-across-namespaces-flux-dryrun` | 13:59:50 | 14:00:13 | 23s | content tweak |
|
||||
|
||||
Sub-30s gaps (3 of 5) read as "I wanted to edit but can only overwrite." The agent has
|
||||
no way to know whether the second write deduped or created a contradictory v2 — opacity
|
||||
the brain's own design principle (`mcp-tool-design-get-needs-list-partner.md`, written
|
||||
*by one of these very agents*) would flag: every `_write` needs a `_get`/`_update` partner.
|
||||
|
||||
### verify_write_landed → lexical re-query (4/4 mismatch)
|
||||
No read-after-write / get-by-id confirmation. After every substantive write, agents
|
||||
re-query lexically to check the note is retrievable — and *reformulate the keywords*
|
||||
because they can't predict what BM25 indexed:
|
||||
- `exit-255` lesson: query `exit 255 unknown reason restart loop diagnosis` → write →
|
||||
query `exit 255 unknown SIGKILL containerd`.
|
||||
- `extension-version-lags`: query `extension build pinned version...pgvector` → write →
|
||||
query `pgvector postgres extension version compile error bump`.
|
||||
- `webfetch-readme`: write → write → query stuffed with the note's own section headings.
|
||||
|
||||
### semantic_retrieval → BM25 keyword-stuffing (3/3 mismatch)
|
||||
Agent knows the *meaning* but not the *indexed words*, so it brute-forces phrasings of
|
||||
one need against a lexical tool:
|
||||
- **4-step chain on one Go bug:** `bytes.Buffer Bytes Reset aliasing slice sharing` →
|
||||
`go buffer reuse map backing array bug` → (write) → `go map values all show same
|
||||
content after loop` → `bytes.Buffer Bytes returns same data every iteration`. Symptom
|
||||
and cause both known; no single lexical query bridges them.
|
||||
- Natural-language questions (`how to enable prometheus metrics on litellm proxy
|
||||
callback`, `moved compose stack to new directory volumes disappeared empty`) shoved
|
||||
into `brain_query`.
|
||||
|
||||
### synthesized_answer → fall back to / hedge with brain_query (2 mismatch + 1 partial)
|
||||
`brain_answer` is frequently **not trusted as terminal**:
|
||||
- **Strongest signal:** 3× `brain_answer` at 15:08 (compose volumes / pi rebuild time /
|
||||
litellm ModuleNotFound) → 3× `brain_query` at 15:09 on the *same three topics*. The
|
||||
agent asked the synthesizer, was unsatisfied, and immediately re-ran raw search.
|
||||
- Dex→Authentik: `brain_query` and `brain_answer` fired **8s apart on one question** —
|
||||
the agent pays both interfaces because it can't predict which returns usable knowledge.
|
||||
- (Counter-examples exist: `brain_answer` for episodic recall — "what happened with the
|
||||
litellm migration on 2026-05-16?" — was consumed and not re-queried. So `answer`
|
||||
works for *episodic/temporal* recall, fails for *how-do-I / does-X-hold* reasoning.)
|
||||
|
||||
### discover_capability + store-via-HTTP (3 mismatch)
|
||||
brain tools are **not ambient** — they are deferred and must be `ToolSearch`-loaded each
|
||||
session. Agents fumble the discovery (`ToolSearch 'brain knowledge memory'` →
|
||||
`'brain'` → `'brain ingestion knowledge wiki'`) and, when MCP load/auth fails, drop to
|
||||
**raw `curl` against `brain-mcp` / hand-built `/tmp/brain_entry.json`**. One agent even
|
||||
`ToolSearch`-ed an `authenticate` tool, then abandoned MCP for the HTTP bodge — matching
|
||||
the known "brain/op MCP auth lapses too often" footgun.
|
||||
|
||||
### Write-interface / layer schema confusion (cross-cutting)
|
||||
`brain_write` was called with **three different param shapes** in the same corpus:
|
||||
`{filename, type:"lesson", content}`, `{filename, content}` (no type), and
|
||||
`{wing:"tapir", hall:"failures", filename, content}` — plus `brain_ingest {source,
|
||||
content}`. Agents are unsure which verb and which layer (flat slug vs `wing`/`hall`
|
||||
knowledge routing vs raw ingest) a given knowledge-act maps to. This is the
|
||||
`knowledge/ vs wiki/` confusion expressed at the parameter level.
|
||||
|
||||
## 3. `intent_unclear` rate
|
||||
|
||||
**0 / 46 genuine brain acts (0%).** Agent intent is unusually legible because these are
|
||||
Claude Code transcripts: the surrounding task, the query/filename strings, and the
|
||||
write content all disambiguate. One row (`a47`) was tagged `intent_unclear` and
|
||||
**excluded** — its `brain_` substring was inside a git commit message, not a brain call.
|
||||
Example of the only ambiguity that arose: a `brain_answer` on "gitea MCP not working...
|
||||
how to authenticate" followed by an unrelated query — can't tell if the answer satisfied
|
||||
or was abandoned (`partial`, row a28).
|
||||
|
||||
## 4. The single biggest intent↔interface gap
|
||||
|
||||
**The brain offers one write verb and one lexical read verb, but autonomous agents
|
||||
perform four distinct knowledge-acts against them — and three of the four have no fitting
|
||||
interface.** The deepest gap is the **missing update/supersede path**: agents close every
|
||||
task by writing a lesson (the CLAUDE.md ritual), routinely improve it minutes-to-an-hour
|
||||
later, and — having no edit primitive — re-write the same slug blind, unable to tell
|
||||
whether they corrected the entry or forked a contradiction into the index. This compounds
|
||||
with the lexical-only read side: because there is no `get-by-id` or semantic retrieval,
|
||||
agents can't even reliably *find their own just-written note* to check it, so they
|
||||
keyword-stuff reformulated queries and hedge `brain_answer` with parallel `brain_query`.
|
||||
The interface is built for *append-and-keyword-search*; the agents are trying to
|
||||
*curate a living, deduplicated knowledge base*, and the seam between those two shows up
|
||||
as the 5 blind re-writes, 4 read-after-write re-queries, and 3 semantic-as-lexical chains
|
||||
that dominate the mismatch column.
|
||||
|
||||
---
|
||||
*Evidence-only per task scope — no redesign proposed.*
|
||||
Reference in New Issue
Block a user