docs: add 2026-05-28 decisions — harness boundary + field benchmark definition
This commit is contained in:
@@ -4,6 +4,67 @@ Record *why* things are the way they are. Future-you will thank present-you.
|
||||
|
||||
---
|
||||
|
||||
## 2026-05-28 — hyperguild and agentsquad are separate harnesses, not competing implementations
|
||||
|
||||
**Context:** After a grill-me architecture review, the question arose whether `hyperguild`
|
||||
and `agentsquad` should be consolidated — they both involve agent orchestration and both
|
||||
use LiteLLM routing.
|
||||
|
||||
**Decision:** Keep separate. Different harnesses targeting different orchestration models.
|
||||
|
||||
- **hyperguild** = Claude Code + MCP. Supervisor pattern. Skills as SKILL.md files invoked
|
||||
inside Claude Code sessions. Routing pod routes skill calls to local models. Brain MCP
|
||||
for knowledge accumulation. Target: disciplined solo coding sessions.
|
||||
- **agentsquad** = OpenCode + LiteLLM. Flat peer-to-peer executor/reviewer agents.
|
||||
Per-agent model routing via piguard virtual model names, risk-tiered approval middleware.
|
||||
Target: multi-agent task execution pipelines.
|
||||
|
||||
Skills (`mathias/skills`) are shared — both harnesses consume the same SKILL.md files.
|
||||
|
||||
**Consequences:** No consolidation work. Both READMEs should state their harness explicitly.
|
||||
|
||||
---
|
||||
|
||||
## 2026-05-28 — "field benchmark" for local models = pass-rate at scale (supersedes GOTTH eval suite)
|
||||
|
||||
**Context:** The GOTTH eval suite (45 offline prompts across 5 categories) was replaced by
|
||||
a "field benchmark" in May 2026, but the replacement was never defined concretely.
|
||||
|
||||
**Decision:** The field benchmark is per-skill pass rate over real routing pod usage,
|
||||
collected automatically by `internal/routing/passrate.go` and exposed at:
|
||||
|
||||
```
|
||||
GET /pass-rate?skill=<name>&window=<duration>
|
||||
```
|
||||
|
||||
No separate eval suite. No synthetic prompts. The benchmark runs itself once the routing
|
||||
pod receives real traffic. Target: 30-day rolling window per skill, reviewed monthly.
|
||||
|
||||
**Bootstrap note:** With no session history, `passrate.go` returns `nil` and the router
|
||||
defaults to the thinking model for every call. The fast-model path activates only after
|
||||
real pass-rate data accumulates. Seed with real usage — do not pre-populate.
|
||||
|
||||
**Consequences:** Zero maintenance overhead for the benchmark. The tradeoff is that results
|
||||
are only meaningful after ~2 weeks of real usage, and skills that are rarely invoked will
|
||||
have statistically thin pass-rate data. Revisit if a skill has fewer than 20 calls in 30 days.
|
||||
|
||||
---
|
||||
|
||||
## 2026-05-28 — brain injection in skill handlers: review is done, others unverified
|
||||
|
||||
**Context:** The April 2026 scope reset listed "brain_query injection into skill handlers"
|
||||
as the top priority. As of 2026-05-28, `internal/skills/review/handlers.go` calls
|
||||
`brain.Query(ctx, ...)` before dispatching to the LLM — confirmed in code review.
|
||||
Status of debug, retrospective, and trainer handlers is unverified.
|
||||
|
||||
**Decision:** Treat review as the reference implementation. Verify debug, retrospective,
|
||||
trainer against the same pattern before shipping new skill work.
|
||||
|
||||
**Consequences:** The April concern may be stale for review. A one-pass audit of the other
|
||||
three skill handlers closes this fully.
|
||||
|
||||
---
|
||||
|
||||
## 2026-04-08 — AGENTS.md as cross-tool standard, not CLAUDE.md
|
||||
|
||||
**Context**: Multiple tools (Crush, Pi, Antigravity) read `AGENTS.md` natively. Claude Code reads `CLAUDE.md`. Building on `CLAUDE.md` as the primary format locks into one vendor.
|
||||
|
||||
Reference in New Issue
Block a user