hyperguild/docs/runbooks/exercising-pass-rate-traffic.md

Runbook: exercising review/debug traffic to fill the pass-rate dataset

Why this exists: the routing pod's local-vs-cloud decision is gated on a pass-rate history that only accrues from real review/debug invocations through the pod. Until the dataset has data, the fast (local) path never activates and the core hypothesis (hyperguild #35) can't be validated. This runbook is how you spin that flywheel.

The one trap

Pass-rate accrues only when a skill tool is called via the routing pod's MCP endpoint. These look like they should count but do not:

• Crush — talks to LiteLLM directly, bypasses the pod. No log.
• claude.ai web / Claude Desktop without the connector — no log.
• Running the local code-review / debug skills (~/dev/.skills) inline in a Claude Code session — those are local skills, not the pod's MCP tools. No log.

Only a tools/call to the routing pod records a pass/fail.

Endpoints

Purpose	URL	Auth
Routing MCP (local, Tailscale)	http://koala:30310/mcp	Bearer ROUTING_MCP_TOKEN
Routing MCP (remote)	https://routing-mcp.d-ma.be/mcp	OAuth via auth.d-ma.be (audience claude-ai)
Pass-rate readout	http://koala:30330/pass-rate?skill=<name>	none (read-only)

Tools advertised: review, debug (the two the #35 gate measures), plus session_log, retrospective, trainer.

Step 1 — connect the routing pod as an MCP server

Local (needs the bearer token; keep it out of argv via 1Password):

op run --env-file ~/.op-env -- \
  claude mcp add routing --transport http http://koala:30310/mcp \
    --header "Authorization: Bearer $ROUTING_MCP_TOKEN"

Remote (claude.ai / Claude Desktop): add a custom connector pointing at https://routing-mcp.d-ma.be/mcp; it completes OAuth against auth.d-ma.be, no static token.

Verify: a tools/list should return review, debug, session_log, retrospective, trainer.

Step 2 — route real work through it

In normal sessions, invoke the pod's tools instead of reviewing/debugging inline:

• "Use the routing review tool on this diff."
• "debug this failure through the routing pod."

Each call logs an outcome to ingestion → /pass-rate ticks up.

Step 3 — the flywheel (cold-start behavior is the gate)

The router's cold start is the data gate — same loop:

1. With pass-rate nil (cold), the policy routes to the thinking / cloud model — it won't trust the local fast model without history.
2. As review / debug accumulate passes, pass-rate climbs past the floor (HYPERGUILD_ROUTE_LOCAL_FLOOR=0.90) and the router starts sending those skills to the local fast tier.
3. So early calls "pay in" on cloud to build the record; then the local tier activates automatically. Exercising the traffic is what spins it up.

The fast tier is now koala/qwen36-35b-a3b (Qwen3.6-35B-A3B MTP), promoted 2026-06-29 after clearing the tool-call gate (infra c66a195, HYPERGUILD_FAST_MODEL).

Target & verification

• 50 logged invocations across review + debug within the 14-day window (opened 2026-06-26, kill-date 2026-07-10) → ~4–5 calls/day.

• Check progress anytime:

  curl -s "http://koala:30330/pass-rate?skill=review"
  curl -s "http://koala:30330/pass-rate?skill=debug"
  

• If ~4–5/day isn't realistic alongside Crush, that is not a failure — per #35 deliverable #1 it's the signal hyperguild isn't on the work critical path, and the pre-decided Berget fallback (gpt-oss-120b / qwen3-32b) carries the fast tier instead.

Refs

• hyperguild #35 — the validation issue (data gate = deliverable #1)
• docs/multi-model-routing.md — routing policy
• brain: wiki/homelab/hypotheses/qwen36-35b-a3b-fast-model-experiment-2026-05-28.md
• infra c66a195 — qwen36 promotion; models.yml / llama-swap-configmap.yaml