mathias/hyperguild

Fork 0

Files

T

mathiasandClaude Opus 5.5 b5c227179f

CI / Lint / Test / Vet (pull_request) Successful in 35s

Details

CI / Mirror to GitHub (pull_request) Skipped

Details

fix(ingestion): close the PR #152 review findings on REST auth

- main.go's wiring was untested (the test built its own copy of the
  router, so dropping auth in main or re-registering POST /query as
  public stayed green). Routing is now one function, buildMux, which
  main calls and the tests call: it constructs the bearer middleware
  itself, registers public handlers only for patterns on a fixed
  allowlist (a public REST pattern would beat the authenticated
  catch-all by specificity), refuses an empty token, and returns an
  http.Handler so nothing can add a route to the result.
- hyperguild CLI: PassRate sent no token and would 401. Every request
  now goes through brainClient.do, which adds it; the test covers
  Query, Write and PassRate and is named for that.
- ResolveIngestPath returns one refusal for every bad path; the raw
  EvalSymlinks error let a caller probe which server files exist and
  leaked the brain directory.
- PROJECT.md (and derived adapters) and the pass-rate runbook state
  that the REST API needs BRAIN_MCP_TOKEN and is tailnet/in-cluster.

Refs #149

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-10-03 16:32:20 +02:00

4.5 KiB

Raw Permalink Blame History

Runbook: exercising review/debug traffic to fill the pass-rate dataset

Why this exists: the routing pod's local-vs-cloud decision is gated on a pass-rate history that only accrues from real review/debug invocations through the pod. Until the dataset has data, the fast (local) path never activates and the core hypothesis (hyperguild #35) can't be validated. This runbook is how you spin that flywheel.

The one trap

Pass-rate accrues only when a skill tool is called via the routing pod's MCP endpoint. These look like they should count but do not:

Crush — talks to LiteLLM directly, bypasses the pod. No log.
claude.ai web / Claude Desktop without the connector — no log.
Running the local code-review / debug skills (~/dev/.skills) inline in a Claude Code session — those are local skills, not the pod's MCP tools. No log.

Only a tools/call to the routing pod records a pass/fail.

Endpoints

Purpose	URL	Auth
Routing MCP (local, Tailscale)	`http://koala:30310/mcp`	Bearer `ROUTING_MCP_TOKEN`
Routing MCP (remote)	`https://routing-mcp.d-ma.be/mcp`	OAuth via `auth.d-ma.be` (audience `claude-ai`)
Pass-rate readout	`http://koala:30330/pass-rate?skill=<name>`	Bearer `BRAIN_MCP_TOKEN` (since hyperguild#149)

Tools advertised: review, debug (the two the #35 gate measures), plus session_log, retrospective, trainer.

Step 1 — connect the routing pod as an MCP server

Local (needs the bearer token; keep it out of argv via 1Password):

op run --env-file ~/.op-env -- \
  claude mcp add routing --transport http http://koala:30310/mcp \
    --header "Authorization: Bearer $ROUTING_MCP_TOKEN"

Remote (claude.ai / Claude Desktop): add a custom connector pointing at https://routing-mcp.d-ma.be/mcp; it completes OAuth against auth.d-ma.be, no static token.

Verify: a tools/list should return review, debug, session_log, retrospective, trainer.

Step 2 — route real work through it

In normal sessions, invoke the pod's tools instead of reviewing/debugging inline:

"Use the routing review tool on this diff."
"debug this failure through the routing pod."

Each call logs an outcome to ingestion → /pass-rate ticks up.

Step 3 — how routing actually picks the model

Per internal/routing/policy.go:

pass-rate nil (cold) → local fast tier. The router defaults to local from invocation #1, not to cloud — so the fast tier is exercised immediately.
pass-rate >= 0.90 (floor) → local; < 0.70 (ceil) → cloud/thinking; in the [0.70, 0.90) band a request-hash bit samples 50/50.
On a local execution error the router falls open to the thinking model for that one call (logged thinking_fallback).

So you are not "paying in on cloud" — cold calls already run on the (validated) local fast tier koala/qwen36-35b-a3b (Qwen3.6-35B-A3B MTP, promoted 2026-06-29, infra c66a195, HYPERGUILD_FAST_MODEL). Accumulating passes just keeps it there once real pass-rate is computed.

Instrumentation note (#73, fixed 2026-06-30): until v0.11.1 the pod logged successes as "skip" (not "pass"), under skill:"_routing", via an unauthenticated POST that silently 401'd — so /pass-rate stayed at zero no matter how much you used it. That's fixed and verified (a real review call now moves /pass-rate?skill=review 0→1). If you see traffic not registering, re-check #73's three failure modes first.

Target & verification

50 logged invocations across review + debug within the 14-day window. The clock restarts 2026-06-30 (the day instrumentation was verified working; the original 2026-06-26→07-10 window measured broken plumbing) → kill-date 2026-07-14, ~4 calls/day (1 already logged from the #73 smoke test).

Check progress anytime:

curl -s "http://koala:30330/pass-rate?skill=review"
curl -s "http://koala:30330/pass-rate?skill=debug"

If ~4–5/day isn't realistic alongside Crush, that is not a failure — per #35 deliverable #1 it's the signal hyperguild isn't on the work critical path, and the pre-decided Berget fallback (gpt-oss-120b / qwen3-32b) carries the fast tier instead.

Refs

hyperguild #35 — the validation issue (data gate = deliverable #1)
docs/multi-model-routing.md — routing policy
brain: wiki/homelab/hypotheses/qwen36-35b-a3b-fast-model-experiment-2026-05-28.md
infra c66a195 — qwen36 promotion; models.yml / llama-swap-configmap.yaml

4.5 KiB Raw Permalink Blame History Unescape Escape