From 00e5f62c8eb66d735195f095582a33738a12257b Mon Sep 17 00:00:00 2001 From: Mathias Date: Tue, 30 Jun 2026 22:16:03 +0200 Subject: [PATCH] =?UTF-8?q?docs(runbook):=20correct=20flywheel=20(nil?= =?UTF-8?q?=E2=86=92local)=20+=20instrumentation-fixed=20note?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two corrections after the #73 investigation: (1) the router routes cold (nil pass-rate) calls to the LOCAL fast tier, not cloud — the earlier "pay in on cloud" description was wrong (per policy.go). (2) Note that pass-rate logging was broken until v0.11.1 (#73) and is now verified; reset the window to 2026-06-30→07-14. Refs #73, #35. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/runbooks/exercising-pass-rate-traffic.md | 37 ++++++++++++------- 1 file changed, 23 insertions(+), 14 deletions(-) diff --git a/docs/runbooks/exercising-pass-rate-traffic.md b/docs/runbooks/exercising-pass-rate-traffic.md index 1d3bb1d..2f13e76 100644 --- a/docs/runbooks/exercising-pass-rate-traffic.md +++ b/docs/runbooks/exercising-pass-rate-traffic.md @@ -55,26 +55,35 @@ In normal sessions, invoke the pod's tools instead of reviewing/debugging inline Each call logs an outcome to ingestion → `/pass-rate` ticks up. -## Step 3 — the flywheel (cold-start behavior is the gate) +## Step 3 — how routing actually picks the model -The router's cold start **is** the data gate — same loop: +Per `internal/routing/policy.go`: -1. With pass-rate `nil` (cold), the policy routes to the **thinking / cloud** - model — it won't trust the local fast model without history. -2. As `review` / `debug` accumulate **passes**, pass-rate climbs past the floor - (`HYPERGUILD_ROUTE_LOCAL_FLOOR=0.90`) and the router starts sending those - skills to the **local fast tier**. -3. So early calls "pay in" on cloud to build the record; then the local tier - activates automatically. Exercising the traffic is what spins it up. +1. pass-rate `nil` (cold) → **local** fast tier. The router defaults to local + from invocation #1, not to cloud — so the fast tier is exercised immediately. +2. pass-rate `>= 0.90` (floor) → **local**; `< 0.70` (ceil) → **cloud/thinking**; + in the `[0.70, 0.90)` band a request-hash bit samples 50/50. +3. On a local execution error the router falls open to the thinking model for + that one call (logged `thinking_fallback`). -The fast tier is now **`koala/qwen36-35b-a3b`** (Qwen3.6-35B-A3B MTP), promoted -2026-06-29 after clearing the tool-call gate (infra `c66a195`, -`HYPERGUILD_FAST_MODEL`). +So you are not "paying in on cloud" — cold calls already run on the (validated) +local fast tier **`koala/qwen36-35b-a3b`** (Qwen3.6-35B-A3B MTP, promoted +2026-06-29, infra `c66a195`, `HYPERGUILD_FAST_MODEL`). Accumulating passes just +keeps it there once real pass-rate is computed. + +> **Instrumentation note (#73, fixed 2026-06-30):** until v0.11.1 the pod logged +> successes as `"skip"` (not `"pass"`), under `skill:"_routing"`, via an +> unauthenticated POST that silently 401'd — so `/pass-rate` stayed at zero no +> matter how much you used it. That's fixed and verified (a real review call now +> moves `/pass-rate?skill=review` 0→1). If you see traffic not registering, +> re-check #73's three failure modes first. ## Target & verification -- **50 logged invocations** across `review` + `debug` within the 14-day window - (opened 2026-06-26, **kill-date 2026-07-10**) → ~4–5 calls/day. +- **50 logged invocations** across `review` + `debug` within the 14-day window. + The clock restarts **2026-06-30** (the day instrumentation was verified working; + the original 2026-06-26→07-10 window measured broken plumbing) → **kill-date + 2026-07-14**, ~4 calls/day (1 already logged from the #73 smoke test). - Check progress anytime: ```bash