docs: add 2026-05-28 decisions — harness boundary + field benchmark definition
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 3s

This commit is contained in:
mathias
2026-05-28 10:31:41 +00:00
parent 525811bc1a
commit 2beaac2feb
+61
View File
@@ -4,6 +4,67 @@ Record *why* things are the way they are. Future-you will thank present-you.
---
## 2026-05-28 — hyperguild and agentsquad are separate harnesses, not competing implementations
**Context:** After a grill-me architecture review, the question arose whether `hyperguild`
and `agentsquad` should be consolidated — they both involve agent orchestration and both
use LiteLLM routing.
**Decision:** Keep separate. Different harnesses targeting different orchestration models.
- **hyperguild** = Claude Code + MCP. Supervisor pattern. Skills as SKILL.md files invoked
inside Claude Code sessions. Routing pod routes skill calls to local models. Brain MCP
for knowledge accumulation. Target: disciplined solo coding sessions.
- **agentsquad** = OpenCode + LiteLLM. Flat peer-to-peer executor/reviewer agents.
Per-agent model routing via piguard virtual model names, risk-tiered approval middleware.
Target: multi-agent task execution pipelines.
Skills (`mathias/skills`) are shared — both harnesses consume the same SKILL.md files.
**Consequences:** No consolidation work. Both READMEs should state their harness explicitly.
---
## 2026-05-28 — "field benchmark" for local models = pass-rate at scale (supersedes GOTTH eval suite)
**Context:** The GOTTH eval suite (45 offline prompts across 5 categories) was replaced by
a "field benchmark" in May 2026, but the replacement was never defined concretely.
**Decision:** The field benchmark is per-skill pass rate over real routing pod usage,
collected automatically by `internal/routing/passrate.go` and exposed at:
```
GET /pass-rate?skill=<name>&window=<duration>
```
No separate eval suite. No synthetic prompts. The benchmark runs itself once the routing
pod receives real traffic. Target: 30-day rolling window per skill, reviewed monthly.
**Bootstrap note:** With no session history, `passrate.go` returns `nil` and the router
defaults to the thinking model for every call. The fast-model path activates only after
real pass-rate data accumulates. Seed with real usage — do not pre-populate.
**Consequences:** Zero maintenance overhead for the benchmark. The tradeoff is that results
are only meaningful after ~2 weeks of real usage, and skills that are rarely invoked will
have statistically thin pass-rate data. Revisit if a skill has fewer than 20 calls in 30 days.
---
## 2026-05-28 — brain injection in skill handlers: review is done, others unverified
**Context:** The April 2026 scope reset listed "brain_query injection into skill handlers"
as the top priority. As of 2026-05-28, `internal/skills/review/handlers.go` calls
`brain.Query(ctx, ...)` before dispatching to the LLM — confirmed in code review.
Status of debug, retrospective, and trainer handlers is unverified.
**Decision:** Treat review as the reference implementation. Verify debug, retrospective,
trainer against the same pattern before shipping new skill work.
**Consequences:** The April concern may be stale for review. A one-pass audit of the other
three skill handlers closes this fully.
---
## 2026-04-08 — AGENTS.md as cross-tool standard, not CLAUDE.md
**Context**: Multiple tools (Crush, Pi, Antigravity) read `AGENTS.md` natively. Claude Code reads `CLAUDE.md`. Building on `CLAUDE.md` as the primary format locks into one vendor.