feat: structured brain — Hall taxonomy, Obsidian-compatible layout, and filtered query #1

New Issue

2026-05-07T15:02:19Z

gitea-mcp-bot commented

2026-05-07 15:02:19 +00:00

Context

The brain currently stores notes as a flat list of .md files under brain/knowledge/ and brain/wiki/ with a shallow three-way taxonomy (concepts/, entities/, sources/). Retrieval is term-frequency full-text search across all files.

Two problems emerge as the brain grows:

Retrieval precision degrades — "what decisions were made about jepa-fx?" and "what facts do I know about jepa-fx?" return the same result set. There is no way to query by memory type.
Human curation is hard — the flat layout does not map naturally to an Obsidian vault. A human auditing the brain has no navigational structure, no backlinks between related notes across domains, and no clear entry point.

This issue introduces a Hall dimension borrowed from the MemPalace architecture, restructures the wiki directory layout to be natively Obsidian-compatible, and adds a hall filter to brain_query.

Design

Concept: Wings and Halls

Every note lives at a two-dimensional address:

Wing — the topic domain (e.g. jepa-fx, hyperguild, bathroom-plumbing). Maps directly to a subdirectory.
Hall — the memory type within any Wing. Fixed vocabulary:

Hall	What goes here
`facts`	Established truths, reference data, definitions
`decisions`	Architectural choices, tradeoffs accepted, options rejected
`failures`	Bugs, wrong assumptions, dead ends — and why they failed
`hypotheses`	Unverified ideas staged for future validation
`sources`	Ingested external references (replaces `wiki/sources/`)

Directory layout (after migration)

brain/
├── wiki/
│   ├── jepa-fx/
│   │   ├── facts/
│   │   │   └── lejpa-architecture.md
│   │   ├── decisions/
│   │   │   └── val-vol-r2-as-optimization-metric.md
│   │   └── failures/
│   │       └── ts-jepa-discarded.md
│   ├── hyperguild/
│   │   ├── decisions/
│   │   │   └── routing-pod-local-floor-0.9.md
│   │   └── facts/
│   │       └── tier-detection-logic.md
│   └── _index.md        ← Obsidian vault entry point (auto-generated)
├── knowledge/           ← unchanged: staging area for human review
├── raw/                 ← unchanged: retrospective output
└── sessions/            ← unchanged: JSONL logs

This layout is a valid Obsidian vault. Open brain/ as a vault and you get Wing-level folders in the file explorer, Hall subfolders, and Obsidian's graph view will show natural clusters by Wing.

Obsidian integration properties

Every note uses YAML frontmatter with wing, hall, created_at, and optional tags
Wikilinks ([[other-note]]) written by retrospective or brain_write will resolve natively in Obsidian
_index.md files at Wing level act as MOC (Map of Content) — auto-generated, listing all notes in that Wing with their Hall
No Obsidian plugins required — the layout works with vanilla Obsidian

Implementation

1. New path resolver: `ingestion/internal/brain/path.go`

package brain

// ValidHalls is the closed vocabulary of hall names.
var ValidHalls = map[string]bool{
    "facts": true, "decisions": true, "failures": true,
    "hypotheses": true, "sources": true,
}

// NotePath resolves a canonical filesystem path for a note given wing, hall,
// and slug. Returns error if hall is not in ValidHalls.
func NotePath(brainDir, wing, hall, slug string) (string, error)

Wing and slug are sanitised (lowercase, alphanumeric + hyphens, no path separators). Hall is validated against ValidHalls.

2. Extend `WriteNote` in `ingestion/internal/api/handler.go`

Add optional wing and hall fields to writeRequest:

type writeRequest struct {
    Content  string `json:"content"`
    Filename string `json:"filename,omitempty"`
    Type     string `json:"type,omitempty"`   // kept for backwards compat
    Domain   string `json:"domain,omitempty"` // kept for backwards compat
    Wing     string `json:"wing,omitempty"`
    Hall     string `json:"hall,omitempty"`
}

When wing + hall are provided, route to brain/wiki/<wing>/<hall>/<filename> via brain.NotePath. When absent, fall back to current brain/knowledge/ behaviour. This is fully backwards-compatible — no existing callers break.

Inject YAML frontmatter automatically:

---
wing: jepa-fx
hall: decisions
created_at: 2026-05-07T10:00:00Z
---

3. Extend `search.Query` in `ingestion/internal/search/search.go`

Add hall and wing filter parameters:

type QueryOptions struct {
    Query string
    Limit int
    Wing  string // optional: restrict to brain/wiki/<wing>/
    Hall  string // optional: restrict to brain/wiki/<wing>/<hall>/
}

func Query(brainDir string, opts QueryOptions) ([]Result, error)

Path-based filtering: when wing is set, only walk brain/wiki/<wing>/. When hall is additionally set, only walk brain/wiki/<wing>/<hall>/. This is O(1) directory scoping — no change to the scoring logic.

Update Result to include wing and hall fields extracted from frontmatter or path segments.

4. Update MCP tool schemas

In ingestion/internal/mcp/:

brain_write — add optional wing (string) and hall (enum: facts/decisions/failures/hypotheses/sources) parameters.

brain_query — add optional wing (string) and hall (enum) parameters.

5. Index generation: `brain_index`

New MCP tool (and REST endpoint POST /index) that walks brain/wiki/ and regenerates _index.md at each Wing directory:

# jepa-fx

| Hall | Note | Created |
|------|------|---------|
| decisions | [val-vol-r2-as-optimization-metric](decisions/val-vol-r2-as-optimization-metric.md) | 2026-05-06 |
| facts | [lejpa-architecture](facts/lejpa-architecture.md) | 2026-05-04 |

Called automatically after every brain_write with wing+hall, and available as a manual trigger.

6. Migration script: `scripts/migrate-brain-halls.sh`

One-time script to migrate existing brain/wiki/concepts/ and brain/wiki/entities/ files into the new layout. Reads existing frontmatter type: and domain: fields to infer Wing and Hall. Falls back to wing=general, hall=facts for unmapped files. Non-destructive: leaves originals in place until --commit flag is passed.

Acceptance criteria

POST /write with {"wing":"jepa-fx","hall":"decisions","content":"...","filename":"my-decision"} writes to brain/wiki/jepa-fx/decisions/my-decision.md with correct frontmatter
POST /write without wing/hall continues to write to brain/knowledge/ unchanged
POST /query with {"query":"lejpa","wing":"jepa-fx","hall":"facts"} only searches brain/wiki/jepa-fx/facts/
brain_write MCP tool accepts and passes through wing and hall
brain_query MCP tool accepts and passes through wing and hall
Opening brain/ as an Obsidian vault shows Wing folders in the file explorer with Hall subfolders
_index.md at each Wing directory lists all notes in that Wing with Hall column
Migration script runs without error on an empty brain and on the existing brain fixture
All existing tests pass (backwards-compatible path)
New tests cover: valid wing+hall write, invalid hall rejected, wing-scoped query returns only wing results

Branch

feat/brain-halls from main

Out of scope

Vector/embedding search (separate issue)
Tunnel links between Wings (follow-up after Halls are stable)
Obsidian plugin development
Changing brain/sessions/ or brain/raw/ layout

Created via git-mcp on behalf of @mathiasbq

## Context The brain currently stores notes as a flat list of `.md` files under `brain/knowledge/` and `brain/wiki/` with a shallow three-way taxonomy (`concepts/`, `entities/`, `sources/`). Retrieval is term-frequency full-text search across all files. Two problems emerge as the brain grows: 1. **Retrieval precision degrades** — "what decisions were made about jepa-fx?" and "what facts do I know about jepa-fx?" return the same result set. There is no way to query by memory type. 2. **Human curation is hard** — the flat layout does not map naturally to an Obsidian vault. A human auditing the brain has no navigational structure, no backlinks between related notes across domains, and no clear entry point. This issue introduces a **Hall dimension** borrowed from the MemPalace architecture, restructures the wiki directory layout to be natively Obsidian-compatible, and adds a `hall` filter to `brain_query`. --- ## Design ### Concept: Wings and Halls Every note lives at a two-dimensional address: - **Wing** — the topic domain (e.g. `jepa-fx`, `hyperguild`, `bathroom-plumbing`). Maps directly to a subdirectory. - **Hall** — the memory type within any Wing. Fixed vocabulary: | Hall | What goes here | |------|---------------| | `facts` | Established truths, reference data, definitions | | `decisions` | Architectural choices, tradeoffs accepted, options rejected | | `failures` | Bugs, wrong assumptions, dead ends — and why they failed | | `hypotheses` | Unverified ideas staged for future validation | | `sources` | Ingested external references (replaces `wiki/sources/`) | ### Directory layout (after migration) ``` brain/ ├── wiki/ │ ├── jepa-fx/ │ │ ├── facts/ │ │ │ └── lejpa-architecture.md │ │ ├── decisions/ │ │ │ └── val-vol-r2-as-optimization-metric.md │ │ └── failures/ │ │ └── ts-jepa-discarded.md │ ├── hyperguild/ │ │ ├── decisions/ │ │ │ └── routing-pod-local-floor-0.9.md │ │ └── facts/ │ │ └── tier-detection-logic.md │ └── _index.md ← Obsidian vault entry point (auto-generated) ├── knowledge/ ← unchanged: staging area for human review ├── raw/ ← unchanged: retrospective output └── sessions/ ← unchanged: JSONL logs ``` This layout is a valid Obsidian vault. Open `brain/` as a vault and you get Wing-level folders in the file explorer, Hall subfolders, and Obsidian's graph view will show natural clusters by Wing. ### Obsidian integration properties - Every note uses YAML frontmatter with `wing`, `hall`, `created_at`, and optional `tags` - Wikilinks (`[[other-note]]`) written by `retrospective` or `brain_write` will resolve natively in Obsidian - `_index.md` files at Wing level act as MOC (Map of Content) — auto-generated, listing all notes in that Wing with their Hall - No Obsidian plugins required — the layout works with vanilla Obsidian --- ## Implementation ### 1. New path resolver: `ingestion/internal/brain/path.go` ```go package brain // ValidHalls is the closed vocabulary of hall names. var ValidHalls = map[string]bool{ "facts": true, "decisions": true, "failures": true, "hypotheses": true, "sources": true, } // NotePath resolves a canonical filesystem path for a note given wing, hall, // and slug. Returns error if hall is not in ValidHalls. func NotePath(brainDir, wing, hall, slug string) (string, error) ``` Wing and slug are sanitised (lowercase, alphanumeric + hyphens, no path separators). Hall is validated against `ValidHalls`. ### 2. Extend `WriteNote` in `ingestion/internal/api/handler.go` Add optional `wing` and `hall` fields to `writeRequest`: ```go type writeRequest struct { Content string `json:"content"` Filename string `json:"filename,omitempty"` Type string `json:"type,omitempty"` // kept for backwards compat Domain string `json:"domain,omitempty"` // kept for backwards compat Wing string `json:"wing,omitempty"` Hall string `json:"hall,omitempty"` } ``` When `wing` + `hall` are provided, route to `brain/wiki/<wing>/<hall>/<filename>` via `brain.NotePath`. When absent, fall back to current `brain/knowledge/` behaviour. This is fully backwards-compatible — no existing callers break. Inject YAML frontmatter automatically: ```yaml --- wing: jepa-fx hall: decisions created_at: 2026-05-07T10:00:00Z --- ``` ### 3. Extend `search.Query` in `ingestion/internal/search/search.go` Add `hall` and `wing` filter parameters: ```go type QueryOptions struct { Query string Limit int Wing string // optional: restrict to brain/wiki/<wing>/ Hall string // optional: restrict to brain/wiki/<wing>/<hall>/ } func Query(brainDir string, opts QueryOptions) ([]Result, error) ``` Path-based filtering: when `wing` is set, only walk `brain/wiki/<wing>/`. When `hall` is additionally set, only walk `brain/wiki/<wing>/<hall>/`. This is O(1) directory scoping — no change to the scoring logic. Update `Result` to include `wing` and `hall` fields extracted from frontmatter or path segments. ### 4. Update MCP tool schemas In `ingestion/internal/mcp/`: `brain_write` — add optional `wing` (string) and `hall` (enum: facts/decisions/failures/hypotheses/sources) parameters. `brain_query` — add optional `wing` (string) and `hall` (enum) parameters. ### 5. Index generation: `brain_index` New MCP tool (and REST endpoint `POST /index`) that walks `brain/wiki/` and regenerates `_index.md` at each Wing directory: ```markdown # jepa-fx | Hall | Note | Created | |------|------|---------| | decisions | [val-vol-r2-as-optimization-metric](decisions/val-vol-r2-as-optimization-metric.md) | 2026-05-06 | | facts | [lejpa-architecture](facts/lejpa-architecture.md) | 2026-05-04 | ``` Called automatically after every `brain_write` with `wing`+`hall`, and available as a manual trigger. ### 6. Migration script: `scripts/migrate-brain-halls.sh` One-time script to migrate existing `brain/wiki/concepts/` and `brain/wiki/entities/` files into the new layout. Reads existing frontmatter `type:` and `domain:` fields to infer Wing and Hall. Falls back to `wing=general`, `hall=facts` for unmapped files. Non-destructive: leaves originals in place until `--commit` flag is passed. --- ## Acceptance criteria - `POST /write` with `{"wing":"jepa-fx","hall":"decisions","content":"...","filename":"my-decision"}` writes to `brain/wiki/jepa-fx/decisions/my-decision.md` with correct frontmatter - `POST /write` without `wing`/`hall` continues to write to `brain/knowledge/` unchanged - `POST /query` with `{"query":"lejpa","wing":"jepa-fx","hall":"facts"}` only searches `brain/wiki/jepa-fx/facts/` - `brain_write` MCP tool accepts and passes through `wing` and `hall` - `brain_query` MCP tool accepts and passes through `wing` and `hall` - Opening `brain/` as an Obsidian vault shows Wing folders in the file explorer with Hall subfolders - `_index.md` at each Wing directory lists all notes in that Wing with Hall column - Migration script runs without error on an empty brain and on the existing brain fixture - All existing tests pass (backwards-compatible path) - New tests cover: valid wing+hall write, invalid hall rejected, wing-scoped query returns only wing results ## Branch `feat/brain-halls` from `main` ## Out of scope - Vector/embedding search (separate issue) - Tunnel links between Wings (follow-up after Halls are stable) - Obsidian plugin development - Changing `brain/sessions/` or `brain/raw/` layout --- _Created via git-mcp on behalf of @mathiasbq_

gitea-mcp-bot referenced this issue

2026-05-07 15:44:54 +00:00

feat: brain tunnels — cross-wing concept links and embedding-based retrieval #2

feat: structured brain — Hall taxonomy, Obsidian-compatible layout, and filtered query #1

Context

Design

Concept: Wings and Halls

Directory layout (after migration)

Obsidian integration properties

Implementation

1. New path resolver: ingestion/internal/brain/path.go

2. Extend WriteNote in ingestion/internal/api/handler.go

3. Extend search.Query in ingestion/internal/search/search.go

4. Update MCP tool schemas

5. Index generation: brain_index

6. Migration script: scripts/migrate-brain-halls.sh

Acceptance criteria

Branch

Out of scope

What landed

Acceptance criteria

Out of scope

1. New path resolver: `ingestion/internal/brain/path.go`

2. Extend `WriteNote` in `ingestion/internal/api/handler.go`

3. Extend `search.Query` in `ingestion/internal/search/search.go`

5. Index generation: `brain_index`

6. Migration script: `scripts/migrate-brain-halls.sh`