From dcb9ff4a56b6d919d8c8794d94916fc095c148c8 Mon Sep 17 00:00:00 2001 From: Mathias Date: Mon, 29 Jun 2026 23:41:07 +0200 Subject: [PATCH] docs(runbook): how to exercise review/debug traffic for the pass-rate gate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The #35 data gate stays at zero because pass-rate only accrues from review/ debug calls *through the routing pod* — Crush, cloud chat, and the local .skills all bypass it. Document the connect → route → flywheel steps, the endpoints (koala:30310/mcp + routing-mcp.d-ma.be), the cold-start behavior (nil pass-rate routes cloud until passes accrue past the 0.90 floor, then local qwen36 activates), the 50-invocation / 2026-07-10 target, and the Berget fallback. Refs #35. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/runbooks/exercising-pass-rate-traffic.md | 95 +++++++++++++++++++ 1 file changed, 95 insertions(+) create mode 100644 docs/runbooks/exercising-pass-rate-traffic.md diff --git a/docs/runbooks/exercising-pass-rate-traffic.md b/docs/runbooks/exercising-pass-rate-traffic.md new file mode 100644 index 0000000..1d3bb1d --- /dev/null +++ b/docs/runbooks/exercising-pass-rate-traffic.md @@ -0,0 +1,95 @@ +# Runbook: exercising review/debug traffic to fill the pass-rate dataset + +**Why this exists:** the routing pod's local-vs-cloud decision is gated on a +pass-rate history that only accrues from real `review`/`debug` invocations +**through the pod**. Until the dataset has data, the fast (local) path never +activates and the core hypothesis (hyperguild #35) can't be validated. This +runbook is how you spin that flywheel. + +## The one trap + +Pass-rate accrues **only** when a skill tool is called via the routing pod's MCP +endpoint. These look like they should count but **do not**: + +- **Crush** — talks to LiteLLM directly, bypasses the pod. No log. +- **claude.ai web / Claude Desktop without the connector** — no log. +- **Running the local `code-review` / `debug` skills** (`~/dev/.skills`) inline in + a Claude Code session — those are local skills, not the pod's MCP tools. No log. + +Only a `tools/call` to the routing pod records a pass/fail. + +## Endpoints + +| Purpose | URL | Auth | +|---------|-----|------| +| Routing MCP (local, Tailscale) | `http://koala:30310/mcp` | Bearer `ROUTING_MCP_TOKEN` | +| Routing MCP (remote) | `https://routing-mcp.d-ma.be/mcp` | OAuth via `auth.d-ma.be` (audience `claude-ai`) | +| Pass-rate readout | `http://koala:30330/pass-rate?skill=` | none (read-only) | + +Tools advertised: **`review`**, **`debug`** (the two the #35 gate measures), +plus `session_log`, `retrospective`, `trainer`. + +## Step 1 — connect the routing pod as an MCP server + +**Local** (needs the bearer token; keep it out of argv via 1Password): + +```bash +op run --env-file ~/.op-env -- \ + claude mcp add routing --transport http http://koala:30310/mcp \ + --header "Authorization: Bearer $ROUTING_MCP_TOKEN" +``` + +**Remote** (claude.ai / Claude Desktop): add a custom connector pointing at +`https://routing-mcp.d-ma.be/mcp`; it completes OAuth against `auth.d-ma.be`, +no static token. + +Verify: a `tools/list` should return `review`, `debug`, `session_log`, +`retrospective`, `trainer`. + +## Step 2 — route real work through it + +In normal sessions, invoke the pod's tools instead of reviewing/debugging inline: + +- *"Use the **routing** `review` tool on this diff."* +- *"**debug** this failure through the routing pod."* + +Each call logs an outcome to ingestion → `/pass-rate` ticks up. + +## Step 3 — the flywheel (cold-start behavior is the gate) + +The router's cold start **is** the data gate — same loop: + +1. With pass-rate `nil` (cold), the policy routes to the **thinking / cloud** + model — it won't trust the local fast model without history. +2. As `review` / `debug` accumulate **passes**, pass-rate climbs past the floor + (`HYPERGUILD_ROUTE_LOCAL_FLOOR=0.90`) and the router starts sending those + skills to the **local fast tier**. +3. So early calls "pay in" on cloud to build the record; then the local tier + activates automatically. Exercising the traffic is what spins it up. + +The fast tier is now **`koala/qwen36-35b-a3b`** (Qwen3.6-35B-A3B MTP), promoted +2026-06-29 after clearing the tool-call gate (infra `c66a195`, +`HYPERGUILD_FAST_MODEL`). + +## Target & verification + +- **50 logged invocations** across `review` + `debug` within the 14-day window + (opened 2026-06-26, **kill-date 2026-07-10**) → ~4–5 calls/day. +- Check progress anytime: + + ```bash + curl -s "http://koala:30330/pass-rate?skill=review" + curl -s "http://koala:30330/pass-rate?skill=debug" + ``` + +- If ~4–5/day isn't realistic alongside Crush, that is **not** a failure — per + #35 deliverable #1 it's the signal hyperguild isn't on the work critical path, + and the pre-decided **Berget fallback** (`gpt-oss-120b` / `qwen3-32b`) carries + the fast tier instead. + +## Refs + +- hyperguild #35 — the validation issue (data gate = deliverable #1) +- `docs/multi-model-routing.md` — routing policy +- brain: `wiki/homelab/hypotheses/qwen36-35b-a3b-fast-model-experiment-2026-05-28.md` +- infra `c66a195` — qwen36 promotion; `models.yml` / `llama-swap-configmap.yaml`