mathias/hyperguild
1
0
Fork 0
Code Issues 30 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

chore: gate #25 and #29 on routing bootstrap + fast-model validation (30-day hold) #34

New Issue
Closed
opened 2026-05-29 08:30:41 +00:00 by mathias · 1 comment
mathias commented 2026-05-29 08:30:41 +00:00
Owner
Copy Link
Copy Source

Context

Issues #25 (skills platform overhaul) and #29 (skill-from-session generator) are both significant scopes. This issue captures the sequencing constraint surfaced in a Full Grill session (2026-05-29):

Neither #25 nor #29 should start until:

  1. Brain injection is confirmed in all four skill handlers (#32 closed). The feedback loop is broken until this is done — investing in skill quality or skill generation before the brain is actually influencing outputs is premature.

  2. At least one skill hits the 0.90 fast-model threshold in real routing traffic. The core hypothesis of the routing pod is that a fast local model can substitute for the thinking model on high-pass-rate skills. This has not been validated — no fast-model routing traffic has occurred yet (bootstrap note: routing defaults to thinking model until pass-rate data accumulates). If a dense 14B model on koala cannot reach 0.90 on any skill after 30 days of real usage, the routing architecture's value proposition is unproven and the scope of #25 should be reconsidered.

Gate criteria (must all be true before starting #25 or #29)

  • #32 closed (brain injection confirmed in all four handlers)
  • Routing pod has received ≥20 real skill invocations per active skill over a 30-day window
  • At least one skill has a GET /pass-rate?skill=<name>&window=30d score ≥ 0.90 using the fast local model (koala dense 14B or equivalent)

Canary check (run at day 30)

# From flamingo over Tailscale
curl "http://koala:30310/pass-rate?skill=review&window=30d"
curl "http://koala:30310/pass-rate?skill=debug&window=30d"
curl "http://koala:30310/pass-rate?skill=retrospective&window=30d"
curl "http://koala:30310/pass-rate?skill=trainer&window=30d"

Also run brain_answer on a known question to verify retrieval quality is non-trivial.

If gate is not met at day 30

Related

  • #25 — skills platform overhaul (blocked)
  • #29 — skill-from-session generator (blocked)
  • #32 — brain injection audit (must close first)
  • #24 — lazy skill loading (may be prerequisite if call volume is low)
  • DECISIONS.md 2026-05-28 — field benchmark definition
## Context Issues #25 (skills platform overhaul) and #29 (skill-from-session generator) are both significant scopes. This issue captures the sequencing constraint surfaced in a Full Grill session (2026-05-29): **Neither #25 nor #29 should start until:** 1. **Brain injection is confirmed in all four skill handlers** (#32 closed). The feedback loop is broken until this is done — investing in skill quality or skill generation before the brain is actually influencing outputs is premature. 2. **At least one skill hits the 0.90 fast-model threshold** in real routing traffic. The core hypothesis of the routing pod is that a fast local model can substitute for the thinking model on high-pass-rate skills. This has not been validated — no fast-model routing traffic has occurred yet (bootstrap note: routing defaults to thinking model until pass-rate data accumulates). If a dense 14B model on koala cannot reach 0.90 on any skill after 30 days of real usage, the routing architecture's value proposition is unproven and the scope of #25 should be reconsidered. ## Gate criteria (must all be true before starting #25 or #29) - [ ] #32 closed (brain injection confirmed in all four handlers) - [ ] Routing pod has received ≥20 real skill invocations per active skill over a 30-day window - [ ] At least one skill has a `GET /pass-rate?skill=<name>&window=30d` score ≥ 0.90 using the fast local model (koala dense 14B or equivalent) ## Canary check (run at day 30) ```bash # From flamingo over Tailscale curl "http://koala:30310/pass-rate?skill=review&window=30d" curl "http://koala:30310/pass-rate?skill=debug&window=30d" curl "http://koala:30310/pass-rate?skill=retrospective&window=30d" curl "http://koala:30310/pass-rate?skill=trainer&window=30d" ``` Also run `brain_answer` on a known question to verify retrieval quality is non-trivial. ## If gate is not met at day 30 - Fewer than 20 calls/skill → skills are not being triggered enough; reconsider lazy-loading (#24) before #25 - No skill at ≥0.90 → evaluate dense 14B model on koala; if no model reaches threshold, routing pod fast-path may not be viable — revisit #25 scope accordingly - Brain retrieval is weak → prioritise brain corpus quality before skill overhaul ## Related - #25 — skills platform overhaul (blocked) - #29 — skill-from-session generator (blocked) - #32 — brain injection audit (must close first) - #24 — lazy skill loading (may be prerequisite if call volume is low) - DECISIONS.md 2026-05-28 — field benchmark definition
mathias commented 2026-08-19 18:19:27 +00:00
Author
Owner
Copy Link
Copy Source

Obsolete per #75 (closed 2026-08-19): this gate's entire premise — validate the routing pod's fast-model hypothesis via real /pass-rate traffic before starting #25/#29 — no longer applies. #75 didn't validate or fix the routing/pass-rate signal, it retired it (icebox, "fresh start, not resurrection" per infra#174's finding that the signal is survivorship-biased and unusable). There is no koala:30310/pass-rate endpoint running to canary-check anymore. #25 no longer has this gate to wait on — see the re-scope note left there. Closing. Cascade re-triage per hyperguild#75, 2026-08-19.

Obsolete per #75 (closed 2026-08-19): this gate's entire premise — validate the routing pod's fast-model hypothesis via real `/pass-rate` traffic before starting #25/#29 — no longer applies. #75 didn't validate or fix the routing/pass-rate signal, it retired it (icebox, "fresh start, not resurrection" per infra#174's finding that the signal is survivorship-biased and unusable). There is no `koala:30310/pass-rate` endpoint running to canary-check anymore. #25 no longer has this gate to wait on — see the re-scope note left there. Closing. Cascade re-triage per hyperguild#75, 2026-08-19.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agent/ready
Meets the agent-ready contract; authorizes unattended execution
 blocked/client
Waiting on the client
 blocked/supplier
Waiting on a supplier
 blocked/upstream
Waiting on an upstream project or vendor
 risk/high
Irreversible, security-relevant, or control plane
 risk/low
Reversible, local, test-covered
 risk/medium
Touches shared state, deploys or data
 size/l
Too big - split it
 size/m
Under two days
 size/s
Under half a day
 size/xs
Under an hour
 status:dispatch
status:dispatched
status:review
status:reviewing
type/chore
Maintenance, no behaviour change
 type/decision
Human decision required; never agent/ready
 type/docs
Documentation only
 type/feat
New capability
 type/fix
Corrects broken behaviour
 type/ops
Infrastructure, deploy, runtime
 type/research
Investigate; output is a finding, not code
 type/test
Test coverage only
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
renovate-bot mathias
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/hyperguild#34