Compare commits

...
14 Commits
Author SHA1 Message Date
mathiasandClaude Sonnet 5 9dcd60931a chore: retrigger CI after gitea-runner RBAC fix (round 6 of the CI/CD outage chain)
CI / Lint / Test / Vet (push) Successful in 30s
CI / Mirror to GitHub (push) Has been skipped
cd.yml's build+push+ssh chain fully worked last round (infra 707a3c0
landed via CD Bot). Final gap was RBAC: kubectl (new in baked-v6) had
no ServiceAccount permissions. Fixed in infra@949b333 (dedicated SA +
scoped Role/RoleBinding for flux-system + supervisor, verified live
via kubectl auth can-i before this retrigger). No code change here.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 22:29:57 +02:00
mathiasandClaude Sonnet 5 1fac90ed2a fix(cd): pass git.d-ma.be SSH override as -o flags, not an appended ~/.ssh/config line
CI / Lint / Test / Vet (push) Successful in 29s
CI / Mirror to GitHub (push) Successful in 4s
cd.yml's own $HOME (/data) is a PVC that persists across job runs on
this runner. The previous fix (cb9c251) appended the corrected
HostName to ~/.ssh/config, but a stale 127.0.0.1:30022 entry from an
earlier failed run was already there — ssh_config is first-match-wins,
so the stale entry silently shadowed the fix and the very next run
failed identically. Cleaned the persisted stale entry on the runner
directly; this commit removes the append entirely so the step no
longer depends on file state surviving (or not surviving) between
runs — CLI -o overrides always win. Verified end-to-end against the
real infra repo from inside the runner pod (git ls-remote, correctly
reached gitea and got a clean auth rejection, not a connection error).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 22:16:00 +02:00
mathiasandClaude Sonnet 5 cb9c2513a4 fix(cd): use in-cluster DNS for git.d-ma.be SSH, not 127.0.0.1:30022
CI / Lint / Test / Vet (push) Successful in 28s
CI / Mirror to GitHub (push) Successful in 4s
act_runner moved from koala's bare host network into a k3s pod
(2026-07-26); loopback inside that pod's own network namespace never
reaches the host, so the git.d-ma.be SSH override (used to push the
infra repo update) failed with "Connection refused". Verified
TCP+SSH-handshake reachability to gitea-ssh-nodeport.gitea.svc.cluster.local:22
from inside the actual runner pod before committing this.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 22:09:52 +02:00
mathiasandClaude Sonnet 5 3617a6c386 chore: retrigger CI after gitea-runner baked-v6 (ssh/skopeo/buildctl/kubectl)
CI / Lint / Test / Vet (push) Successful in 28s
CI / Mirror to GitHub (push) Successful in 4s
check job passed clean on baked-v5; mirror job then exposed missing
ssh/ssh-keyscan. Fixed in infra@707a3c0, which also preemptively adds
skopeo/buildctl/kubectl for cd.yml's deploy job. No code change here —
retriggering for the full CI->CD end-to-end proof.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 22:03:07 +02:00
mathiasandClaude Sonnet 5 1938170131 chore: retrigger CI after gitea-runner gcc/libc6-dev fix (baked-v5)
CI / Lint / Test / Vet (push) Successful in 1m12s
CI / Mirror to GitHub (push) Failing after 1s
Runner was missing gcc/libc6-dev too (-race requires cgo). Added in
infra@0f88aea. No code change here — retriggering CI.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 21:55:20 +02:00
mathiasandClaude Sonnet 5 b34717e6b8 chore: retrigger CI after gitea-runner image fix (infra#task-baked-v4)
CI / Lint / Test / Vet (push) Failing after 34s
CI / Mirror to GitHub (push) Has been skipped
Runner was missing task (taskfile.dev), added in infra@686c90f. No
code change here — retriggering CI to confirm the fix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 21:49:33 +02:00
mathiasandClaude Sonnet 5 d8d7e9a307 chore(context): re-sync derived adapters (Python fallback stack, error-handling note)
CI / Lint / Test / Vet (push) Failing after 1s
CI / Mirror to GitHub (push) Has been skipped
Canonical ~/dev/.context/AGENT.md drifted (Python fallback stack table
row, Python error-handling convention) without a hyperguild commit
re-running task context:sync since — task check's drift gate has been
failing CI on every push since the sprint-1 commits landed, blocking
CD entirely (unrelated to those commits' own content: lint/test/vet
all pass clean, including -race).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 15:18:38 +02:00
mathiasandClaude Sonnet 5 f0055483a3 fix(watcher): never re-ingest AutoTunnel's own tunnel-candidates log (#88)
CI / Lint / Test / Vet (push) Failing after 1s
CI / Mirror to GitHub (push) Has been skipped
Root cause was neither of the two mechanisms the issue named
(CanonicalizeLinks over-canonicalizing, or a schema/prompt bias) —
CanonicalizeLinks correctly resolved [[Mission]] because
wiki/concepts/mission.md genuinely exists; the extraction prompt has
no "mission" bias either. The real bug: watcher.processDir walks
brain/raw/ and feeds every .md/.txt/.pdf file to pipeline.Run as
source material to extract, with no exclusion for
tunnel-candidates-*.md — AutoTunnel's own fuzzy-match human-review
queue (logFuzzyCandidates writes it into that same raw/ directory).
The watcher's next poll re-ingested the raw candidate log itself, and
the LLM naturally surfaced "mission" as a wikilink because the log
literally lists `(term: "mission")` entries in its content — a
generic-word title match DetectTunnels correctly flagged as fuzzy
(not auto-tunneled) but which still leaked downstream once treated as
real source material.

api.ListPending already excludes tunnel-candidates-* when listing
raw/ for promotion; processDir gets the same exclusion.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 14:37:50 +02:00
mathiasandClaude Sonnet 5 6d014c1d0f fix(ingest): CanonicalizeLinks strips wing:/wiki/ prefix and repairs path-style links (#87)
CI / Mirror to GitHub (push) Has been skipped
CI / Lint / Test / Vet (push) Failing after 2s
plainLinkRE matches any [[...]] without a pipe, including path-prefixed
forms like [[wing:homelab/failures/foo]] or [[wiki/agentsquad/decisions/bar]]
that the LLM extraction step occasionally emits. Title-lookup against
titleToSlug always fails for these (they're paths, not titles), so they
were silently left broken.

Before falling to the unknown-wikilink warning, strip a known wing:/wiki/
root prefix and emit the clean wing/hall/slug path directly — matching
the brain-graph path-style link convention.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 14:32:31 +02:00
mathiasandClaude Sonnet 5 1001acfb44 fix(ingest): merge existing frontmatter in writeHallNote instead of stacking (#86)
CI / Lint / Test / Vet (push) Failing after 1s
CI / Mirror to GitHub (push) Has been skipped
opts.Content can already carry its own "---\n...\n---" block (e.g. from
an upstream extraction/promotion step). writeHallNote used to prepend a
second block unconditionally, making a standard frontmatter parser blind
to everything the first block declared (title, tags, ...).

splitFrontmatter now pulls the existing block's fields out first;
wing/hall/created_at are always fresh-injected, and type/domain/
source_type prefer the note's own existing value over the opts
fallback. Remaining existing fields are carried through verbatim into
the single merged block.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-26 23:45:36 +02:00
mathiasandClaude Sonnet 5 6ad275b505 fix(ingest): thread source/author/published frontmatter through extraction (#85)
CI / Lint / Test / Vet (push) Failing after 1s
CI / Mirror to GitHub (push) Has been skipped
pipeline.Run parses source/author/published from the raw content's own
frontmatter and applies them deterministically to source-type RawPages
after LLM extraction — never relies on the LLM to copy them through.
buildFrontmatter now emits all three (source pages only) when present.

Backfill of the 46 already-affected wiki/sources/*.md notes is a
separate concern per the issue, not done here.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-26 23:42:33 +02:00
mathiasandClaude Sonnet 5 b600cc986c chore(module): rename module github.com/mathiasbq/supervisor -> git.d-ma.be/mathias/hyperguild (#42)
CI / Lint / Test / Vet (push) Failing after 5s
CI / Mirror to GitHub (push) Has been skipped
Aligns module path with canonical Gitea source (infra ADR-0004) and
repo name. Binary only, no downstream consumers.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-26 23:37:11 +02:00
mathiasandClaude Opus 4.8 9bdab1c48c fix(ingest): parse docmark's SSE-framed response, not bare JSON
CI / Lint / Test / Vet (push) Successful in 16s
CI / Mirror to GitHub (push) Has been skipped
Found via a real live-cluster end-to-end test (POST /ingest-path with a real
.docx against the deployed ingestion + docmark): 'decode docmark response:
invalid character e looking for beginning of value'. docmark's Streamable-HTTP
transport frames every response as SSE (event: message\r\ndata: {...}\r\n\r\n,
Content-Type: text/event-stream) -- stateless_http=True removes the need for
an initialize handshake/session ID, it does NOT change the wire framing to
bare JSON. My original client + its own test mock both assumed bare JSON,
so the unit tests passed while the real call failed.

sseDataPayload() extracts the data: line before JSON-unmarshaling; falls back
to the raw body if no SSE framing is present (forward-compatible). Test mock
(docmark_test.go) now emits the SAME framing the live server actually sends,
confirmed by directly probing docmark's live response before writing the fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 08:53:24 +02:00
mathiasandClaude Opus 4.8 6520c2fc4b feat(ingest): route DOCX/XLSX/PPTX/images through docmark (ADR-0013)
CI / Lint / Test / Vet (push) Successful in 16s
CI / Mirror to GitHub (push) Successful in 4s
Wires the sovereign docmark MCP server (git.d-ma.be/mathias/docmark) as the
converter for formats ingestion previously couldn't handle at all -- DOCX,
XLSX, PPTX, PNG/JPG. A single self-contained tools/call POST (docmark runs
stateless_http, no session handshake needed) does the conversion.

- internal/extract/docmark.go: extractViaDocmark(path) -- reads DOCMARK_URL +
  DOCMARK_BEARER_TOKEN from env, base64-encodes the file, calls docmark's
  convert_to_markdown tool, surfaces tool/HTTP errors with context.
- internal/extract/extract.go: routes .docx/.xlsx/.pptx/.png/.jpg/.jpeg to it.
- internal/api/handler.go: isSupportedExtension() replaces the static
  supportedExtensions map -- the new extensions are gated on DOCMARK_URL being
  set, checked per-request. When DOCMARK_URL is unset (e.g. this repo's
  current deployed state), behavior is byte-for-byte unchanged from today:
  same 400/silent-skip as before. Zero risk until the env var is actually
  wired in infra.

TDD throughout: docmark.go tested via httptest mock (success, not-configured,
tool-error, HTTP-error, all 6 new extensions route correctly); handler-level
test proves the DOCMARK_URL gate (docx 400 when unset, 200 when a working
mock is configured). Full task check green (fmt/vet/govulncheck/all tests).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 08:46:03 +02:00
42 changed files with 879 additions and 56 deletions
+7 -2
View File
@@ -88,7 +88,7 @@ These rules apply to every task across every project, regardless of harness.
| Containers | Docker Compose (dev), k3s (prod) | — | — |
| DB | PostgreSQL + sqlc | SQLite | — |
| Search | pgvector (vector), BM25 | Qdrant (when >1M vectors or hybrid retrieval) | — |
| Logging | slog (structured) | | — |
| Logging | slog (structured) | stdlib `logging` w/ structured `extra=` (or structlog) | — |
| Testing | Table-driven, testify | — | — |
| Agents (Go) | google.golang.org/adk + pkg/litellm adapter | — | — |
@@ -97,7 +97,12 @@ Exploratory: Rust, Zig — I'll tell you when I want these.
## Code conventions
- **Go style**: golines, gofumpt, golangci-lint
- **Errors**: `fmt.Errorf("operation: %w", err)` — never naked, never log-and-return
- **Python style** (fallback language): ruff (format+lint, one tool), mypy --strict (non-negotiable,
matches Go's static typing discipline), pytest + pytest-cov (table-driven via
`@pytest.mark.parametrize`), uv (venv+deps+lock, one tool), pydantic-settings (typed env-var config
— same principle as Go's typed structs), src-layout + `pyproject.toml` only (no `setup.py`)
- **Errors**: `fmt.Errorf("operation: %w", err)` — never naked, never log-and-return.
Python: `raise X from e` (exception chaining, same principle) — never bare `except`, never silent `pass`
- **Naming**: stdlib conventions, no stuttering
- **Architecture**: prefer stdlib over frameworks, constructor injection, env-var config parsed into typed structs
- **Git**: conventional commits (`feat:`, `fix:`, `chore:`), commit directly to main,
+17 -3
View File
@@ -48,9 +48,23 @@ jobs:
mkdir -p ~/.ssh
echo "${{ secrets.INFRA_DEPLOY_KEY }}" > ~/.ssh/infra_deploy_key
chmod 600 ~/.ssh/infra_deploy_key
printf 'Host git.d-ma.be\n HostName 127.0.0.1\n Port 30022\n StrictHostKeyChecking no\n' >> ~/.ssh/config
GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes" \
# In-cluster DNS to gitea's SSH NodePort service, not 127.0.0.1:30022
# (that only worked when act_runner ran on koala's bare host network;
# from inside the containerized runner's own pod netns, loopback
# never reaches the host — "Connection refused", found 2026-07-27).
#
# Pass as -o overrides on the ssh invocation itself, NOT appended to
# ~/.ssh/config: $HOME (/data) is a PVC that persists across job
# runs on this runner (same "workspace not ephemeral" class as
# brain: act-runner-host-executor-tmp-persists), so an appended
# line here would pile up duplicate `Host git.d-ma.be` blocks
# across every run — ssh_config is first-match-wins, so a stale
# entry from an earlier failed run would silently shadow this
# fix forever (exactly what happened once already: this fix's
# own first attempt got appended AFTER an already-stale entry
# and lost). CLI -o options always win regardless of file state,
# so this step is safe to re-run any number of times.
GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes -o HostName=gitea-ssh-nodeport.gitea.svc.cluster.local -o Port=22 -o StrictHostKeyChecking=no" \
git clone "${INFRA_REPO}" /tmp/infra-update
cd /tmp/infra-update
+7 -2
View File
@@ -83,7 +83,7 @@ These rules apply to every task across every project, regardless of harness.
| Containers | Docker Compose (dev), k3s (prod) | — | — |
| DB | PostgreSQL + sqlc | SQLite | — |
| Search | pgvector (vector), BM25 | Qdrant (when >1M vectors or hybrid retrieval) | — |
| Logging | slog (structured) | | — |
| Logging | slog (structured) | stdlib `logging` w/ structured `extra=` (or structlog) | — |
| Testing | Table-driven, testify | — | — |
| Agents (Go) | google.golang.org/adk + pkg/litellm adapter | — | — |
@@ -92,7 +92,12 @@ Exploratory: Rust, Zig — I'll tell you when I want these.
## Code conventions
- **Go style**: golines, gofumpt, golangci-lint
- **Errors**: `fmt.Errorf("operation: %w", err)` — never naked, never log-and-return
- **Python style** (fallback language): ruff (format+lint, one tool), mypy --strict (non-negotiable,
matches Go's static typing discipline), pytest + pytest-cov (table-driven via
`@pytest.mark.parametrize`), uv (venv+deps+lock, one tool), pydantic-settings (typed env-var config
— same principle as Go's typed structs), src-layout + `pyproject.toml` only (no `setup.py`)
- **Errors**: `fmt.Errorf("operation: %w", err)` — never naked, never log-and-return.
Python: `raise X from e` (exception chaining, same principle) — never bare `except`, never silent `pass`
- **Naming**: stdlib conventions, no stuttering
- **Architecture**: prefer stdlib over frameworks, constructor injection, env-var config parsed into typed structs
- **Git**: conventional commits (`feat:`, `fix:`, `chore:`), commit directly to main,
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"io"
"os"
"github.com/mathiasbq/supervisor/internal/tier"
"git.d-ma.be/mathias/hyperguild/internal/tier"
)
const defaultAnthropicProbe = "https://api.anthropic.com"
+1 -1
View File
@@ -1,4 +1,4 @@
module github.com/mathiasbq/supervisor
module git.d-ma.be/mathias/hyperguild
go 1.26.1
+96 -14
View File
@@ -156,17 +156,40 @@ func writeHallNote(brainDir string, opts WriteNoteOptions) (string, error) {
return "", fmt.Errorf("create hall dir: %w", err)
}
existingFields, body := splitFrontmatter(opts.Content)
existingByKey := make(map[string]frontmatterField, len(existingFields))
for _, f := range existingFields {
existingByKey[f.key] = f
}
emitted := make(map[string]bool, 6)
var fm strings.Builder
fm.WriteString("---\n")
fmt.Fprintf(&fm, "wing: %s\n", brain.Sanitise(opts.Wing))
fmt.Fprintf(&fm, "hall: %s\n", opts.Hall)
fmt.Fprintf(&fm, "created_at: %s\n", time.Now().UTC().Format(time.RFC3339))
if opts.Type != "" {
fmt.Fprintf(&fm, "type: %s\n", opts.Type)
emitted["wing"], emitted["hall"], emitted["created_at"] = true, true, true
// writeField merges one key: opts.Content's own value (if the note already
// carries this field in its own frontmatter) always wins over the fallback,
// so promotion/extraction-step metadata survives verbatim instead of being
// shadowed by a second, stacked frontmatter block (#86).
writeField := func(key, fallback string) {
emitted[key] = true
if f, ok := existingByKey[key]; ok {
for _, line := range f.lines {
fm.WriteString(line)
fm.WriteString("\n")
}
if opts.Domain != "" {
fmt.Fprintf(&fm, "domain: %s\n", opts.Domain)
return
}
if fallback != "" {
fmt.Fprintf(&fm, "%s: %s\n", key, fallback)
}
}
writeField("type", opts.Type)
writeField("domain", opts.Domain)
sourceType := opts.SourceType
if sourceType == "" && opts.Hall == "facts" {
// Most hall=facts entries are first-party (an eval/benchmark the
@@ -175,18 +198,69 @@ func writeHallNote(brainDir string, opts WriteNoteOptions) (string, error) {
// citation-needing entry (brain-gardener#7).
sourceType = "internal"
}
if sourceType != "" {
fmt.Fprintf(&fm, "source_type: %s\n", sourceType)
writeField("source_type", sourceType)
for _, f := range existingFields {
if emitted[f.key] {
continue
}
for _, line := range f.lines {
fm.WriteString(line)
fm.WriteString("\n")
}
}
fm.WriteString("---\n")
if err := os.WriteFile(dest, []byte(fm.String()+opts.Content), 0o644); err != nil {
if err := os.WriteFile(dest, []byte(fm.String()+body), 0o644); err != nil {
return "", fmt.Errorf("write: %w", err)
}
rel, _ := filepath.Rel(brainDir, dest)
return filepath.ToSlash(rel), nil
}
// frontmatterField is one top-level YAML key from a frontmatter block,
// along with its raw line and any indented continuation lines (e.g. a
// bulleted list value spanning multiple lines).
type frontmatterField struct {
key string
lines []string
}
// splitFrontmatter splits a leading "---\n...\n---\n" YAML block out of
// content, returning its top-level fields in original order and the
// remaining body. If content has no leading frontmatter block, fields is
// nil and body is content unchanged.
func splitFrontmatter(content string) (fields []frontmatterField, body string) {
if !strings.HasPrefix(content, "---\n") {
return nil, content
}
lines := strings.Split(content, "\n")
i := 1
var cur *frontmatterField
for ; i < len(lines); i++ {
line := lines[i]
if strings.TrimSpace(line) == "---" {
i++
break
}
if line != "" && !strings.HasPrefix(line, " ") && !strings.HasPrefix(line, "\t") {
if cur != nil {
fields = append(fields, *cur)
}
key, _, _ := strings.Cut(line, ":")
cur = &frontmatterField{key: strings.TrimSpace(key), lines: []string{line}}
} else if cur != nil {
cur.lines = append(cur.lines, line)
}
}
if cur != nil {
fields = append(fields, *cur)
}
body = strings.Join(lines[i:], "\n")
return fields, body
}
// writeLegacyNote preserves the original brain/knowledge/ behaviour for
// callers that have not adopted the wing/hall taxonomy.
func writeLegacyNote(brainDir string, opts WriteNoteOptions) (string, error) {
@@ -345,11 +419,19 @@ func (h *Handler) Ingest(w http.ResponseWriter, r *http.Request) {
writeJSON(w, ingestResponse{Pages: pages, Warnings: warnings})
}
// supportedExtensions lists file extensions that IngestPath will process.
var supportedExtensions = map[string]bool{
".md": true,
".txt": true,
".pdf": true,
// isSupportedExtension reports whether IngestPath will process ext.
// .docx/.xlsx/.pptx/.png/.jpg/.jpeg require docmark (ADR-0013) and are only
// supported when DOCMARK_URL is configured — checked per-call (not cached at
// package init) so it reflects the environment at request time.
func isSupportedExtension(ext string) bool {
switch ext {
case ".md", ".txt", ".pdf":
return true
case ".docx", ".xlsx", ".pptx", ".png", ".jpg", ".jpeg":
return os.Getenv("DOCMARK_URL") != ""
default:
return false
}
}
// IngestPath handles POST /ingest-path — ingest a file or directory.
@@ -382,7 +464,7 @@ func (h *Handler) IngestPath(w http.ResponseWriter, r *http.Request) {
return nil
}
ext := strings.ToLower(filepath.Ext(path))
if !supportedExtensions[ext] {
if !isSupportedExtension(ext) {
return nil
}
content, readErr := extract.Text(path)
@@ -410,7 +492,7 @@ func (h *Handler) IngestPath(w http.ResponseWriter, r *http.Request) {
}
} else {
ext := strings.ToLower(filepath.Ext(req.Path))
if !supportedExtensions[ext] {
if !isSupportedExtension(ext) {
writeError(w, http.StatusBadRequest, fmt.Sprintf("unsupported file extension: %s", ext))
return
}
+42
View File
@@ -186,6 +186,48 @@ func TestWriteNote_HallRouteOmitsSourceTypeForNonFactsHalls(t *testing.T) {
assert.NotContains(t, string(got), "source_type")
}
func TestWriteNote_HallRouteMergesExistingFrontmatterInsteadOfStacking(t *testing.T) {
dir := t.TempDir()
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "---\ntitle: act_runner host-executor\ntags: [gitea-actions, act_runner]\n---\n\n# Body\n\nSome content.\n",
Filename: "act-runner-host-executor",
Wing: "homelab",
Hall: "failures",
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
body := string(got)
// exactly one frontmatter block: only two "---" delimiter lines total
assert.Equal(t, 2, strings.Count(body, "---\n"), "expected a single merged frontmatter block, not stacked blocks")
assert.Contains(t, body, "wing: homelab")
assert.Contains(t, body, "hall: failures")
assert.Contains(t, body, "title: act_runner host-executor")
assert.Contains(t, body, "tags: [gitea-actions, act_runner]")
assert.Contains(t, body, "# Body")
}
func TestWriteNote_HallRouteExistingTypeWinsOverOptsType(t *testing.T) {
dir := t.TempDir()
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "---\ntype: hypothesis\n---\n\nBody.\n",
Filename: "note",
Wing: "agentsquad",
Hall: "decisions",
Type: "decision", // should lose to content's own "type: hypothesis"
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.Contains(t, string(got), "type: hypothesis")
assert.NotContains(t, string(got), "type: decision")
}
func TestWrite_GeneratesFilenameIfAbsent(t *testing.T) {
dir, h := setup(t)
body, _ := json.Marshal(map[string]any{"content": "auto name"})
@@ -0,0 +1,61 @@
// ingestion/internal/api/ingestpath_docmark_test.go
package api_test
import (
"bytes"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestIngestPath_DocxUnsupportedWhenDocmarkNotConfigured(t *testing.T) {
t.Setenv("DOCMARK_URL", "")
_, h := setup(t)
dir := t.TempDir()
f := filepath.Join(dir, "doc.docx")
require.NoError(t, os.WriteFile(f, []byte("fake docx"), 0o644))
body, _ := json.Marshal(map[string]any{"path": f, "source": "test-doc", "dry_run": true})
req := httptest.NewRequest(http.MethodPost, "/ingest-path", bytes.NewReader(body))
rec := httptest.NewRecorder()
h.IngestPath(rec, req)
assert.Equal(t, http.StatusBadRequest, rec.Code, rec.Body.String())
}
func TestIngestPath_DocxSupportedWhenDocmarkConfigured(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"jsonrpc":"2.0","id":1,"result":{"content":[{"type":"text","text":"# Converted Doc"}]}}`))
}))
defer srv.Close()
t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
t.Setenv("DOCMARK_BEARER_TOKEN", "tok")
_, h := setup(t)
dir := t.TempDir()
f := filepath.Join(dir, "doc.docx")
require.NoError(t, os.WriteFile(f, []byte("fake docx"), 0o644))
body, _ := json.Marshal(map[string]any{"path": f, "source": "test-doc", "dry_run": true})
req := httptest.NewRequest(http.MethodPost, "/ingest-path", bytes.NewReader(body))
rec := httptest.NewRecorder()
h.IngestPath(rec, req)
require.Equal(t, http.StatusOK, rec.Code, rec.Body.String())
var resp map[string]any
require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &resp))
pages, ok := resp["pages"].([]any)
require.True(t, ok)
assert.NotEmpty(t, pages)
}
+148
View File
@@ -0,0 +1,148 @@
// ingestion/internal/extract/docmark.go
package extract
import (
"bytes"
"encoding/base64"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"time"
)
// docmarkRequest is a JSON-RPC 2.0 tools/call request for docmark's
// convert_to_markdown tool.
type docmarkRequest struct {
JSONRPC string `json:"jsonrpc"`
ID int `json:"id"`
Method string `json:"method"`
Params struct {
Name string `json:"name"`
Arguments struct {
ContentBase64 string `json:"content_base64"`
Filename string `json:"filename"`
} `json:"arguments"`
} `json:"params"`
}
type docmarkResponse struct {
Result *struct {
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
} `json:"content"`
IsError bool `json:"isError"`
} `json:"result"`
Error *struct {
Message string `json:"message"`
} `json:"error"`
}
// extractViaDocmark converts path (PDF/DOCX/XLSX/PPTX/image) to Markdown by
// calling the docmark MCP server with a single self-contained tools/call
// request (docmark runs stateless_http -- no initialize handshake or session
// ID needed). DOCMARK_URL must be set (e.g.
// http://docmark.docmark.svc.cluster.local:3001/mcp); DOCMARK_BEARER_TOKEN
// is docmark's static bearer (network is docmark's primary auth boundary,
// this is defense-in-depth — ADR-0013).
func extractViaDocmark(path string) (string, error) {
url := os.Getenv("DOCMARK_URL")
if url == "" {
return "", fmt.Errorf("extractViaDocmark: DOCMARK_URL is not set")
}
raw, err := os.ReadFile(path)
if err != nil {
return "", fmt.Errorf("read %s: %w", path, err)
}
var reqBody docmarkRequest
reqBody.JSONRPC = "2.0"
reqBody.ID = 1
reqBody.Method = "tools/call"
reqBody.Params.Name = "convert_to_markdown"
reqBody.Params.Arguments.ContentBase64 = base64.StdEncoding.EncodeToString(raw)
reqBody.Params.Arguments.Filename = fileBase(path)
payload, err := json.Marshal(reqBody)
if err != nil {
return "", fmt.Errorf("marshal docmark request: %w", err)
}
httpReq, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(payload))
if err != nil {
return "", fmt.Errorf("build docmark request: %w", err)
}
httpReq.Header.Set("Content-Type", "application/json")
httpReq.Header.Set("Accept", "application/json, text/event-stream")
if tok := os.Getenv("DOCMARK_BEARER_TOKEN"); tok != "" {
httpReq.Header.Set("Authorization", "Bearer "+tok)
}
client := &http.Client{Timeout: 60 * time.Second}
resp, err := client.Do(httpReq)
if err != nil {
return "", fmt.Errorf("call docmark: %w", err)
}
defer func() { _ = resp.Body.Close() }()
body, err := io.ReadAll(resp.Body)
if err != nil {
return "", fmt.Errorf("read docmark response: %w", err)
}
if resp.StatusCode != http.StatusOK {
return "", fmt.Errorf("docmark: HTTP %d: %s", resp.StatusCode, string(body))
}
jsonBody := body
if data := sseDataPayload(body); data != nil {
jsonBody = data
}
var out docmarkResponse
if err := json.Unmarshal(jsonBody, &out); err != nil {
return "", fmt.Errorf("decode docmark response: %w", err)
}
if out.Error != nil {
return "", fmt.Errorf("docmark: %s", out.Error.Message)
}
if out.Result == nil || len(out.Result.Content) == 0 {
return "", fmt.Errorf("docmark: empty response")
}
text := out.Result.Content[0].Text
if out.Result.IsError {
return "", fmt.Errorf("docmark: %s", text)
}
return text, nil
}
// sseDataPayload extracts the JSON payload from an SSE-framed response body
// ("event: message\r\ndata: {...}\r\n\r\n"). docmark's Streamable-HTTP
// transport frames every response this way (Content-Type: text/event-stream)
// regardless of stateless_http — that flag removes the session/initialize
// requirement, not the SSE wire framing. Returns nil if body isn't SSE-framed
// (e.g. a plain-JSON response, kept as a fallback for forward-compatibility).
func sseDataPayload(body []byte) []byte {
const prefix = "data: "
for _, line := range bytes.Split(body, []byte("\n")) {
line = bytes.TrimRight(line, "\r")
if bytes.HasPrefix(line, []byte(prefix)) {
return bytes.TrimPrefix(line, []byte(prefix))
}
}
return nil
}
// fileBase returns the final path segment (like filepath.Base, kept local to
// avoid importing path/filepath just for this one call).
func fileBase(path string) string {
for i := len(path) - 1; i >= 0; i-- {
if path[i] == '/' || path[i] == '\\' {
return path[i+1:]
}
}
return path
}
+189
View File
@@ -0,0 +1,189 @@
// ingestion/internal/extract/docmark_test.go
package extract
import (
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// mcpToolResult mirrors the shape of docmark's JSON-RPC tools/call response.
type mcpToolResult struct {
JSONRPC string `json:"jsonrpc"`
ID int `json:"id"`
Result *struct {
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
} `json:"content"`
IsError bool `json:"isError"`
} `json:"result,omitempty"`
Error *struct {
Message string `json:"message"`
} `json:"error,omitempty"`
}
// writeMCPResponse mirrors docmark's REAL response framing (empirically
// confirmed against the live server): Content-Type: text/event-stream,
// body is SSE-framed ("event: message\r\ndata: {...}\r\n\r\n"), not bare
// JSON -- inherent to MCP Streamable-HTTP, independent of stateless_http.
func writeMCPResponse(w http.ResponseWriter, body mcpToolResult) {
payload, _ := json.Marshal(body)
w.Header().Set("Content-Type", "text/event-stream")
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte("event: message\r\ndata: "))
_, _ = w.Write(payload)
_, _ = w.Write([]byte("\r\n\r\n"))
}
func TestExtractViaDocmark_Success(t *testing.T) {
var gotAuth, gotAccept string
var gotBody map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotAuth = r.Header.Get("Authorization")
gotAccept = r.Header.Get("Accept")
b, _ := io.ReadAll(r.Body)
_ = json.Unmarshal(b, &gotBody)
writeMCPResponse(w, mcpToolResult{
JSONRPC: "2.0", ID: 1,
Result: &struct {
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
} `json:"content"`
IsError bool `json:"isError"`
}{
Content: []struct {
Type string `json:"type"`
Text string `json:"text"`
}{{Type: "text", Text: "# Converted\n\nhello"}},
},
})
}))
defer srv.Close()
t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
t.Setenv("DOCMARK_BEARER_TOKEN", "test-token-123")
dir := t.TempDir()
path := filepath.Join(dir, "doc.docx")
require.NoError(t, os.WriteFile(path, []byte("fake docx bytes"), 0o644))
got, err := extractViaDocmark(path)
require.NoError(t, err)
assert.Equal(t, "# Converted\n\nhello", got)
assert.Equal(t, "Bearer test-token-123", gotAuth)
assert.Contains(t, gotAccept, "application/json")
params, _ := gotBody["params"].(map[string]any)
require.NotNil(t, params)
assert.Equal(t, "convert_to_markdown", params["name"])
args, _ := params["arguments"].(map[string]any)
require.NotNil(t, args)
assert.Equal(t, "doc.docx", args["filename"])
assert.NotEmpty(t, args["content_base64"])
}
func TestExtractViaDocmark_NotConfigured(t *testing.T) {
t.Setenv("DOCMARK_URL", "")
dir := t.TempDir()
path := filepath.Join(dir, "doc.docx")
require.NoError(t, os.WriteFile(path, []byte("x"), 0o644))
_, err := extractViaDocmark(path)
require.Error(t, err)
assert.Contains(t, err.Error(), "DOCMARK_URL")
}
func TestExtractViaDocmark_ToolErrorSurfacesMessage(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
writeMCPResponse(w, mcpToolResult{
JSONRPC: "2.0", ID: 1,
Result: &struct {
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
} `json:"content"`
IsError bool `json:"isError"`
}{
Content: []struct {
Type string `json:"type"`
Text string `json:"text"`
}{{Type: "text", Text: "unsupported format for 'doc.docx'"}},
IsError: true,
},
})
}))
defer srv.Close()
t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
t.Setenv("DOCMARK_BEARER_TOKEN", "tok")
dir := t.TempDir()
path := filepath.Join(dir, "doc.docx")
require.NoError(t, os.WriteFile(path, []byte("x"), 0o644))
_, err := extractViaDocmark(path)
require.Error(t, err)
assert.Contains(t, err.Error(), "unsupported format")
}
func TestExtractViaDocmark_HTTPErrorSurfaces(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusUnauthorized)
_, _ = w.Write([]byte("unauthorized"))
}))
defer srv.Close()
t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
t.Setenv("DOCMARK_BEARER_TOKEN", "wrong")
dir := t.TempDir()
path := filepath.Join(dir, "doc.docx")
require.NoError(t, os.WriteFile(path, []byte("x"), 0o644))
_, err := extractViaDocmark(path)
require.Error(t, err)
assert.Contains(t, err.Error(), "401")
}
func TestText_RoutesDocxXlsxPptxImagesToDocmark(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
writeMCPResponse(w, mcpToolResult{
JSONRPC: "2.0", ID: 1,
Result: &struct {
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
} `json:"content"`
IsError bool `json:"isError"`
}{
Content: []struct {
Type string `json:"type"`
Text string `json:"text"`
}{{Type: "text", Text: "converted"}},
},
})
}))
defer srv.Close()
t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
t.Setenv("DOCMARK_BEARER_TOKEN", "tok")
for _, ext := range []string{".docx", ".xlsx", ".pptx", ".png", ".jpg", ".jpeg"} {
t.Run(ext, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "f"+ext)
require.NoError(t, os.WriteFile(path, []byte("x"), 0o644))
got, err := Text(path)
require.NoError(t, err)
assert.Equal(t, "converted", got)
})
}
}
+5 -1
View File
@@ -8,7 +8,9 @@ import (
)
// Text reads the file at path and returns its plain-text content.
// Supported extensions: .md, .txt (passthrough), .pdf (via pdftotext).
// Supported extensions: .md, .txt (passthrough), .pdf (via pdftotext),
// .docx/.xlsx/.pptx/.png/.jpg/.jpeg (via docmark, ADR-0013 -- requires
// DOCMARK_URL to be set; see docmark.go).
func Text(path string) (string, error) {
ext := strings.ToLower(fileExt(path))
switch ext {
@@ -20,6 +22,8 @@ func Text(path string) (string, error) {
return string(b), nil
case ".pdf":
return extractPDF(path)
case ".docx", ".xlsx", ".pptx", ".png", ".jpg", ".jpeg":
return extractViaDocmark(path)
default:
return "", fmt.Errorf("unsupported file extension: %s", ext)
}
+9
View File
@@ -76,6 +76,15 @@ func buildFrontmatter(rp RawPage, date string) string {
}
fmt.Fprintf(&sb, "date_ingested: %s\n", date)
fmt.Fprintf(&sb, "last_updated: %s\n", date)
if rp.Source != "" {
fmt.Fprintf(&sb, "source: %s\n", yamlScalar(rp.Source))
}
if rp.Author != "" {
fmt.Fprintf(&sb, "author: %s\n", yamlScalar(rp.Author))
}
if rp.Published != "" {
fmt.Fprintf(&sb, "published: %s\n", yamlScalar(rp.Published))
}
case "concept":
if rp.Domain != "" {
fmt.Fprintf(&sb, "domain: %s\n", yamlScalar(rp.Domain))
+46
View File
@@ -154,6 +154,52 @@ func TestBuildPages_EntityNoSubtype(t *testing.T) {
assert.Contains(t, pages[0].Content, "title: 'Basecamp'")
}
func TestBuildPages_SourcePageCarriesSourceAuthorPublished(t *testing.T) {
raw := []RawPage{
{
Title: "Ornith",
Type: "source",
Subtype: "article",
Content: "## Summary\n\nAn agentic coding model.\n",
Source: "https://example.com/ornith",
Author: "Jane Doe",
Published: "2026-07-20",
},
}
pages, warnings := BuildPages(raw, "ornith", "2026-07-26")
require.Len(t, pages, 1)
assert.Empty(t, warnings)
p := pages[0]
assert.Contains(t, p.Content, "source: 'https://example.com/ornith'")
assert.Contains(t, p.Content, "author: 'Jane Doe'")
assert.Contains(t, p.Content, "published: '2026-07-20'")
}
func TestBuildPages_SourcePageOmitsSourceAuthorPublishedWhenEmpty(t *testing.T) {
raw := []RawPage{
{Title: "Shape Up", Type: "source", Subtype: "book", Content: "## Summary\n\nA book.\n"},
}
pages, _ := BuildPages(raw, "shape-up", "2026-04-23")
require.Len(t, pages, 1)
assert.NotContains(t, pages[0].Content, "source:")
assert.NotContains(t, pages[0].Content, "author:")
assert.NotContains(t, pages[0].Content, "published:")
}
func TestBuildPages_ConceptPageIgnoresSourceAuthorPublished(t *testing.T) {
// source/author/published are source-note-only metadata; a concept page
// shouldn't carry them even if somehow set on the RawPage.
raw := []RawPage{
{Title: "Betting", Type: "concept", Content: "## Definition\n\nFoo.\n", Source: "x", Author: "y", Published: "z"},
}
pages, _ := BuildPages(raw, "src", "2026-04-23")
require.Len(t, pages, 1)
assert.NotContains(t, pages[0].Content, "source:")
assert.NotContains(t, pages[0].Content, "author:")
assert.NotContains(t, pages[0].Content, "published:")
}
func TestBuildPages_EmptyTitleSkippedWithWarning(t *testing.T) {
raw := []RawPage{
{Title: "", Type: "concept", Content: "## Definition\n\nFoo.\n"},
+24 -4
View File
@@ -51,6 +51,21 @@ func buildTitleMap(pages []wiki.Page, inventory map[wiki.PageType][]wiki.Entry)
return m
}
// pathStylePrefixes are known root prefixes the LLM extraction step
// sometimes bakes into a wikilink target instead of emitting a clean
// wing/hall/slug path (or bare title). Stripping them repairs the link
// in place — see hyperguild#87.
var pathStylePrefixes = []string{"wing:", "wiki/"}
func stripPathStylePrefix(displayName string) (string, bool) {
for _, prefix := range pathStylePrefixes {
if stripped, ok := strings.CutPrefix(displayName, prefix); ok {
return stripped, true
}
}
return displayName, false
}
func canonicalizeContent(content string, titleToSlug map[string]string) (string, []string) {
var warnings []string
result := plainLinkRE.ReplaceAllStringFunc(content, func(match string) string {
@@ -59,12 +74,17 @@ func canonicalizeContent(content string, titleToSlug map[string]string) (string,
return match
}
displayName := sub[1]
slug, ok := titleToSlug[strings.ToLower(displayName)]
if !ok {
if slug, ok := titleToSlug[strings.ToLower(displayName)]; ok {
return "[[" + slug + "|" + displayName + "]]"
}
if stripped, hadPrefix := stripPathStylePrefix(displayName); hadPrefix {
return "[[" + stripped + "]]"
}
warnings = append(warnings, fmt.Sprintf("unknown wikilink: [[%s]]", displayName))
return match
}
return "[[" + slug + "|" + displayName + "]]"
})
return result, warnings
}
+47
View File
@@ -102,6 +102,53 @@ func TestCanonicalizeLinks_CurrentBatchPagesResolved(t *testing.T) {
assert.Contains(t, got[0].Content, "[[betting|Betting]]")
}
func TestCanonicalizeLinks_StripsWingColonPrefix(t *testing.T) {
pages := []wiki.Page{
{
Path: "wiki/homelab/failures/act-runner-host-mode-container-needs-node-and-libatomic.md",
Content: "---\ntitle: 'act_runner host-mode'\n---\n\nSee [[wing:homelab/failures/rootless-buildah-act-runner-run-containers-denied]].\n",
},
}
got, warnings := CanonicalizeLinks(pages, map[wiki.PageType][]wiki.Entry{})
require.Len(t, got, 1)
assert.Empty(t, warnings)
assert.Contains(t, got[0].Content, "[[homelab/failures/rootless-buildah-act-runner-run-containers-denied]]")
assert.NotContains(t, got[0].Content, "wing:")
}
func TestCanonicalizeLinks_StripsWikiSlashPrefix(t *testing.T) {
pages := []wiki.Page{
{
Path: "wiki/agentsquad/hypotheses/council-consolidation-standalone-deliberation-service.md",
Content: "---\ntitle: 'council consolidation'\n---\n\nSee [[wiki/agentsquad/decisions/autoresearch-council-sibling-pipe]].\n",
},
}
got, warnings := CanonicalizeLinks(pages, map[wiki.PageType][]wiki.Entry{})
require.Len(t, got, 1)
assert.Empty(t, warnings)
assert.Contains(t, got[0].Content, "[[agentsquad/decisions/autoresearch-council-sibling-pipe]]")
assert.NotContains(t, got[0].Content, "wiki/agentsquad/decisions/autoresearch-council-sibling-pipe]]\n\n") // no leftover wiki/ prefix
assert.NotContains(t, got[0].Content, "[[wiki/")
}
func TestCanonicalizeLinks_TitleLookupStillTakesPriorityOverPrefixStrip(t *testing.T) {
// A plain link that resolves via the title map must still use the
// normal slug|Display form, not fall through to prefix-strip repair.
pages := []wiki.Page{
{
Path: "wiki/sources/shape-up.md",
Content: "---\ntitle: 'Shape Up'\n---\n\nSee [[Betting]].\n",
},
}
inventory := map[wiki.PageType][]wiki.Entry{
wiki.PageTypeConcept: {{Slug: "betting", Title: "Betting"}},
}
got, warnings := CanonicalizeLinks(pages, inventory)
require.Len(t, got, 1)
assert.Empty(t, warnings)
assert.Contains(t, got[0].Content, "[[betting|Betting]]")
}
func TestCanonicalizeLinks_MultipleLinksInOnePage(t *testing.T) {
pages := []wiki.Page{
{
+62
View File
@@ -15,6 +15,14 @@ type RawPage struct {
Subtype string `json:"subtype"` // entity: person|company|tool|model|framework|technology; source: article|pdf|book|video|note|project
Domain string `json:"domain"`
Content string `json:"content"` // Markdown body only — no frontmatter
// Source, Author, Published are deterministic passthrough from the raw
// ingested content's own frontmatter (see parseContentFrontmatter) — never
// set by the LLM. json:"-" keeps them immune to same-named keys the LLM
// might emit. Only meaningful for Type == "source".
Source string `json:"-"`
Author string `json:"-"`
Published string `json:"-"`
}
// ParseRawPages parses LLM output as a JSON array of RawPage objects.
@@ -98,6 +106,60 @@ func repairJSON(s string) string {
return b.String()
}
// sourceMeta is source/author/published pulled from the raw ingested
// content's own frontmatter — deterministic passthrough, never LLM output.
type sourceMeta struct {
Source string
Author string
Published string
}
// parseContentFrontmatter extracts source/author/published from a leading
// "---\n...\n---" YAML block in raw ingested content. Only these three flat
// scalar keys are recognised; anything else in the block is ignored. Returns
// a zero-value sourceMeta if content has no frontmatter block.
func parseContentFrontmatter(content string) sourceMeta {
var meta sourceMeta
if !strings.HasPrefix(content, "---\n") && !strings.HasPrefix(content, "---\r\n") {
return meta
}
lines := strings.Split(content, "\n")
for _, line := range lines[1:] {
if strings.TrimSpace(line) == "---" {
break
}
key, val, ok := strings.Cut(line, ":")
if !ok {
continue
}
key = strings.TrimSpace(key)
val = strings.Trim(strings.TrimSpace(val), `"'`)
switch key {
case "source":
meta.Source = val
case "author":
meta.Author = val
case "published":
meta.Published = val
}
}
return meta
}
// applySourceMeta deterministically overwrites Source/Author/Published on
// every "source"-type page with meta — the LLM never controls these fields.
func applySourceMeta(pages []RawPage, meta sourceMeta) {
for i := range pages {
if pages[i].Type != "source" {
continue
}
pages[i].Source = meta.Source
pages[i].Author = meta.Author
pages[i].Published = meta.Published
}
}
func stripFences(s string) string {
for _, prefix := range []string{"```json\n", "```json\r\n", "```\n", "```\r\n"} {
if strings.HasPrefix(s, prefix) {
+2
View File
@@ -59,6 +59,8 @@ func Run(ctx context.Context, cfg Config, brainDir, content, source string, dryR
allWarnings = append(allWarnings, warnings...)
}
applySourceMeta(allRaw, parseContentFrontmatter(content))
return buildAndWrite(allRaw, sourceSlug, date, brainDir, source, inventory, allWarnings, dryRun)
}
@@ -130,6 +130,40 @@ func TestRun_MergesDuplicatePaths(t *testing.T) {
assert.Contains(t, string(content), "[[Baz]]")
}
func TestRun_ThreadsSourceAuthorPublishedFromContentFrontmatter(t *testing.T) {
brainDir := t.TempDir()
for _, sub := range []string{"wiki/concepts", "wiki/entities", "wiki/sources"} {
require.NoError(t, os.MkdirAll(filepath.Join(brainDir, sub), 0o755))
}
llmResponse := mustJSON([]RawPage{{
Title: "Ornith",
Type: "source",
Subtype: "article",
Content: "## Summary\n\nAn agentic coding model.\n",
}})
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_ = json.NewEncoder(w).Encode(map[string]any{
"choices": []map[string]any{{"message": map[string]any{"content": llmResponse}}},
})
}))
defer srv.Close()
cfg := Config{Complete: llm.New(srv.URL, "", "m", 30*time.Second).Complete}
rawContent := "---\nsource: https://example.com/ornith\nauthor: Jane Doe\npublished: 2026-07-20\n---\n\nAn agentic coding model that runs on your laptop.\n"
result, err := Run(context.Background(), cfg, brainDir, rawContent, "ornith", false)
require.NoError(t, err)
require.Len(t, result.Pages, 1)
content, err := os.ReadFile(filepath.Join(brainDir, "wiki", "sources", "ornith.md"))
require.NoError(t, err)
assert.Contains(t, string(content), "source: 'https://example.com/ornith'")
assert.Contains(t, string(content), "author: 'Jane Doe'")
assert.Contains(t, string(content), "published: '2026-07-20'")
}
func mustJSON(v any) string {
b, err := json.Marshal(v)
if err != nil {
+7
View File
@@ -74,6 +74,13 @@ func processDir(ctx context.Context, cfg Config, date string) []error {
return nil
}
// AutoTunnel's own fuzzy-match human-review queue, not source
// material to extract (hyperguild#88) — same exclusion as
// api.ListPending already applies when listing raw/ for promotion.
if strings.HasPrefix(d.Name(), "tunnel-candidates-") {
return nil
}
// Skip files that have already been processed or permanently failed.
if _, err := os.Stat(path + ".processed"); err == nil {
return nil
@@ -229,3 +229,49 @@ func TestProcessDir_SkipsSubdirs(t *testing.T) {
_, err = os.Stat(failedFile)
assert.NoError(t, err, "failed subdir file should be untouched")
}
// TestProcessDir_SkipsTunnelCandidateFiles guards against hyperguild#88:
// AutoTunnel's own human-review queue (brain/raw/tunnel-candidates-*.md)
// must never be re-ingested as if it were external source material — doing
// so fed the raw "(term: X)" log entries back through the LLM extraction
// pipeline, which then legitimately (from its own perspective) surfaced
// [[X]] as a wikilink for any term that happened to match a real page
// title, however generic (e.g. "mission").
func TestProcessDir_SkipsTunnelCandidateFiles(t *testing.T) {
brainDir := setupBrainDir(t)
tunnelFile := filepath.Join(brainDir, "raw", "tunnel-candidates-2026-07-19.md")
require.NoError(t, os.WriteFile(tunnelFile, []byte(
"# Tunnel candidates 2026-07-19\n\n- `wiki/homelab/decisions/foo.md` ↔ `wiki/telos/decisions/mission.md` (term: \"mission\")\n",
), 0o644))
var completeCalls int
completeFn := func(ctx context.Context, system, user string) (string, error) {
completeCalls++
raw := pipeline.RawPage{Title: "Should not be written", Type: "source", Subtype: "article", Content: "## Summary\n\nx.\n"}
b, _ := json.Marshal([]pipeline.RawPage{raw})
return string(b), nil
}
cfg := Config{
BrainDir: brainDir,
Interval: time.Hour, // not used; we call processDir directly
Pipeline: pipeline.Config{
Complete: completeFn,
ChunkSize: 0,
Schema: "# Schema\nThree page types.",
},
}
date := time.Now().UTC().Format("2006-01-02")
errs := processDir(context.Background(), cfg, date)
assert.Empty(t, errs)
assert.Zero(t, completeCalls, "tunnel-candidates file must never reach the LLM extraction pipeline")
// File must be left alone in raw/ — not moved to processed/, no marker written.
_, err := os.Stat(tunnelFile + ".processed")
assert.True(t, os.IsNotExist(err), "tunnel-candidates file should not get a .processed marker")
_, err = os.Stat(filepath.Join(brainDir, "raw", "processed", date, "tunnel-candidates-2026-07-19.md"))
assert.True(t, os.IsNotExist(err), "tunnel-candidates file should not be copied to processed/")
}
+1 -1
View File
@@ -13,7 +13,7 @@ import (
"github.com/lestrrat-go/jwx/v2/jwa"
"github.com/lestrrat-go/jwx/v2/jwk"
"github.com/lestrrat-go/jwx/v2/jwt"
"github.com/mathiasbq/supervisor/internal/auth"
"git.d-ma.be/mathias/hyperguild/internal/auth"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -6,7 +6,7 @@ import (
"net/http/httptest"
"testing"
"github.com/mathiasbq/supervisor/internal/auth"
"git.d-ma.be/mathias/hyperguild/internal/auth"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -7,7 +7,7 @@ import (
"net/http/httptest"
"testing"
"github.com/mathiasbq/supervisor/internal/brain"
"git.d-ma.be/mathias/hyperguild/internal/brain"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -3,7 +3,7 @@ package config_test
import (
"testing"
"github.com/mathiasbq/supervisor/internal/config"
"git.d-ma.be/mathias/hyperguild/internal/config"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -5,7 +5,7 @@ import (
"path/filepath"
"testing"
"github.com/mathiasbq/supervisor/internal/config"
"git.d-ma.be/mathias/hyperguild/internal/config"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -3,7 +3,7 @@ package config_test
import (
"testing"
"github.com/mathiasbq/supervisor/internal/config"
"git.d-ma.be/mathias/hyperguild/internal/config"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"testing"
"time"
iexec "github.com/mathiasbq/supervisor/internal/exec"
iexec "git.d-ma.be/mathias/hyperguild/internal/exec"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -9,7 +9,7 @@ import (
"net/http/httptest"
"testing"
"github.com/mathiasbq/supervisor/internal/githubclient"
"git.d-ma.be/mathias/hyperguild/internal/githubclient"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+2 -2
View File
@@ -8,8 +8,8 @@ import (
"net/http"
"strings"
"github.com/mathiasbq/supervisor/internal/auth"
"github.com/mathiasbq/supervisor/internal/registry"
"git.d-ma.be/mathias/hyperguild/internal/auth"
"git.d-ma.be/mathias/hyperguild/internal/registry"
)
type request struct {
+2 -2
View File
@@ -8,8 +8,8 @@ import (
"strings"
"testing"
"github.com/mathiasbq/supervisor/internal/mcp"
"github.com/mathiasbq/supervisor/internal/registry"
"git.d-ma.be/mathias/hyperguild/internal/mcp"
"git.d-ma.be/mathias/hyperguild/internal/registry"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -9,7 +9,7 @@ import (
"net/http/httptest"
"testing"
"github.com/mathiasbq/supervisor/internal/mcpclient"
"git.d-ma.be/mathias/hyperguild/internal/mcpclient"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -5,7 +5,7 @@ import (
"encoding/json"
"testing"
"github.com/mathiasbq/supervisor/internal/registry"
"git.d-ma.be/mathias/hyperguild/internal/registry"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -6,7 +6,7 @@ import (
"testing"
"time"
"github.com/mathiasbq/supervisor/internal/session"
"git.d-ma.be/mathias/hyperguild/internal/session"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"testing"
"time"
"github.com/mathiasbq/supervisor/internal/session"
"git.d-ma.be/mathias/hyperguild/internal/session"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"net/http/httptest"
"testing"
"github.com/mathiasbq/supervisor/internal/skills/brain"
"git.d-ma.be/mathias/hyperguild/internal/skills/brain"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -4,7 +4,7 @@ package brain
import (
"encoding/json"
"github.com/mathiasbq/supervisor/internal/registry"
"git.d-ma.be/mathias/hyperguild/internal/registry"
)
// Config holds brain skill configuration.
+2 -2
View File
@@ -6,8 +6,8 @@ import (
"encoding/json"
"testing"
"github.com/mathiasbq/supervisor/internal/skills/org"
"github.com/mathiasbq/supervisor/internal/tier"
"git.d-ma.be/mathias/hyperguild/internal/skills/org"
"git.d-ma.be/mathias/hyperguild/internal/tier"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+2 -2
View File
@@ -5,8 +5,8 @@ import (
"context"
"encoding/json"
"github.com/mathiasbq/supervisor/internal/registry"
"github.com/mathiasbq/supervisor/internal/tier"
"git.d-ma.be/mathias/hyperguild/internal/registry"
"git.d-ma.be/mathias/hyperguild/internal/tier"
)
// TierFn returns the current tier. Injected for testability.
+1 -1
View File
@@ -7,7 +7,7 @@ import (
"fmt"
"time"
"github.com/mathiasbq/supervisor/internal/session"
"git.d-ma.be/mathias/hyperguild/internal/session"
)
type logArgs struct {
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"path/filepath"
"testing"
"github.com/mathiasbq/supervisor/internal/skills/sessionlog"
"git.d-ma.be/mathias/hyperguild/internal/skills/sessionlog"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
+1 -1
View File
@@ -4,7 +4,7 @@ package sessionlog
import (
"encoding/json"
"github.com/mathiasbq/supervisor/internal/registry"
"git.d-ma.be/mathias/hyperguild/internal/registry"
)
// Config holds sessionlog skill configuration.
+1 -1
View File
@@ -7,7 +7,7 @@ import (
"net/http/httptest"
"testing"
"github.com/mathiasbq/supervisor/internal/tier"
"git.d-ma.be/mathias/hyperguild/internal/tier"
"github.com/stretchr/testify/assert"
)