105 lines
4.5 KiB
Markdown
105 lines
4.5 KiB
Markdown
# Runbook: exercising review/debug traffic to fill the pass-rate dataset
|
||
|
||
**Why this exists:** the routing pod's local-vs-cloud decision is gated on a
|
||
pass-rate history that only accrues from real `review`/`debug` invocations
|
||
**through the pod**. Until the dataset has data, the fast (local) path never
|
||
activates and the core hypothesis (hyperguild #35) can't be validated. This
|
||
runbook is how you spin that flywheel.
|
||
|
||
## The one trap
|
||
|
||
Pass-rate accrues **only** when a skill tool is called via the routing pod's MCP
|
||
endpoint. These look like they should count but **do not**:
|
||
|
||
- **Crush** — talks to LiteLLM directly, bypasses the pod. No log.
|
||
- **claude.ai web / Claude Desktop without the connector** — no log.
|
||
- **Running the local `code-review` / `debug` skills** (`~/dev/.skills`) inline in
|
||
a Claude Code session — those are local skills, not the pod's MCP tools. No log.
|
||
|
||
Only a `tools/call` to the routing pod records a pass/fail.
|
||
|
||
## Endpoints
|
||
|
||
| Purpose | URL | Auth |
|
||
|---------|-----|------|
|
||
| Routing MCP (local, Tailscale) | `http://koala:30310/mcp` | Bearer `ROUTING_MCP_TOKEN` |
|
||
| Routing MCP (remote) | `https://routing-mcp.d-ma.be/mcp` | OAuth via `auth.d-ma.be` (audience `claude-ai`) |
|
||
| Pass-rate readout | `http://koala:30330/pass-rate?skill=<name>` | Bearer `BRAIN_MCP_TOKEN` (since hyperguild#149) |
|
||
|
||
Tools advertised: **`review`**, **`debug`** (the two the #35 gate measures),
|
||
plus `session_log`, `retrospective`, `trainer`.
|
||
|
||
## Step 1 — connect the routing pod as an MCP server
|
||
|
||
**Local** (needs the bearer token; keep it out of argv via 1Password):
|
||
|
||
```bash
|
||
op run --env-file ~/.op-env -- \
|
||
claude mcp add routing --transport http http://koala:30310/mcp \
|
||
--header "Authorization: Bearer $ROUTING_MCP_TOKEN"
|
||
```
|
||
|
||
**Remote** (claude.ai / Claude Desktop): add a custom connector pointing at
|
||
`https://routing-mcp.d-ma.be/mcp`; it completes OAuth against `auth.d-ma.be`,
|
||
no static token.
|
||
|
||
Verify: a `tools/list` should return `review`, `debug`, `session_log`,
|
||
`retrospective`, `trainer`.
|
||
|
||
## Step 2 — route real work through it
|
||
|
||
In normal sessions, invoke the pod's tools instead of reviewing/debugging inline:
|
||
|
||
- *"Use the **routing** `review` tool on this diff."*
|
||
- *"**debug** this failure through the routing pod."*
|
||
|
||
Each call logs an outcome to ingestion → `/pass-rate` ticks up.
|
||
|
||
## Step 3 — how routing actually picks the model
|
||
|
||
Per `internal/routing/policy.go`:
|
||
|
||
1. pass-rate `nil` (cold) → **local** fast tier. The router defaults to local
|
||
from invocation #1, not to cloud — so the fast tier is exercised immediately.
|
||
2. pass-rate `>= 0.90` (floor) → **local**; `< 0.70` (ceil) → **cloud/thinking**;
|
||
in the `[0.70, 0.90)` band a request-hash bit samples 50/50.
|
||
3. On a local execution error the router falls open to the thinking model for
|
||
that one call (logged `thinking_fallback`).
|
||
|
||
So you are not "paying in on cloud" — cold calls already run on the (validated)
|
||
local fast tier **`koala/qwen36-35b-a3b`** (Qwen3.6-35B-A3B MTP, promoted
|
||
2026-06-29, infra `c66a195`, `HYPERGUILD_FAST_MODEL`). Accumulating passes just
|
||
keeps it there once real pass-rate is computed.
|
||
|
||
> **Instrumentation note (#73, fixed 2026-06-30):** until v0.11.1 the pod logged
|
||
> successes as `"skip"` (not `"pass"`), under `skill:"_routing"`, via an
|
||
> unauthenticated POST that silently 401'd — so `/pass-rate` stayed at zero no
|
||
> matter how much you used it. That's fixed and verified (a real review call now
|
||
> moves `/pass-rate?skill=review` 0→1). If you see traffic not registering,
|
||
> re-check #73's three failure modes first.
|
||
|
||
## Target & verification
|
||
|
||
- **50 logged invocations** across `review` + `debug` within the 14-day window.
|
||
The clock restarts **2026-06-30** (the day instrumentation was verified working;
|
||
the original 2026-06-26→07-10 window measured broken plumbing) → **kill-date
|
||
2026-07-14**, ~4 calls/day (1 already logged from the #73 smoke test).
|
||
- Check progress anytime:
|
||
|
||
```bash
|
||
curl -s "http://koala:30330/pass-rate?skill=review"
|
||
curl -s "http://koala:30330/pass-rate?skill=debug"
|
||
```
|
||
|
||
- If ~4–5/day isn't realistic alongside Crush, that is **not** a failure — per
|
||
#35 deliverable #1 it's the signal hyperguild isn't on the work critical path,
|
||
and the pre-decided **Berget fallback** (`gpt-oss-120b` / `qwen3-32b`) carries
|
||
the fast tier instead.
|
||
|
||
## Refs
|
||
|
||
- hyperguild #35 — the validation issue (data gate = deliverable #1)
|
||
- `docs/multi-model-routing.md` — routing policy
|
||
- brain: `wiki/homelab/hypotheses/qwen36-35b-a3b-fast-model-experiment-2026-05-28.md`
|
||
- infra `c66a195` — qwen36 promotion; `models.yml` / `llama-swap-configmap.yaml`
|