config/models.yaml is not read by the deployed service, and names two models that no longer exist #116

New Issue

2026-09-16T21:28:02Z

mathias commented

2026-09-16 21:28:02 +00:00

Found while retiring iguana/qwen3-coder-next estate-wide (infra#416) and sweeping every repo for references.

config/models.yaml declares:

default_chain:
  - iguana/qwen3-coder-next
skills:
  tdd:          { chain: [koala/qwen3-coder-30b] }
  review:       { chain: [iguana/devstral] }
  debug:        { chain: [iguana/deepseek-r1-14b] }
  spec:         { chain: [koala/phi4-14b] }
  retrospective:{ chain: [iguana/qwen3-coder-next] }
  trainer:      { chain: [iguana/qwen3-coder-next] }

Two problems

1. The deployed service does not read this file. The live ingestion deployment in the supervisor namespace names its models entirely by env:

INGEST_LLM_MODEL=koala/qwen35-9b-fast
BRAIN_LLM_PRIMARY_MODEL=google/gemma-4-31B-it
BRAIN_LLM_FALLBACK_MODEL=gemma4:31b
BRAIN_LLM_TERTIARY_MODEL=gemini-flash-latest
BRAIN_EMBED_MODEL=nomic-embed-text:latest
BRAIN_RERANKER_MODEL=dengcao/Qwen3-Reranker-0.6B:F16

Not one name from config/models.yaml appears. The file's own header says "first entry per skill is used" — nothing uses it. Editing it to change routing would change nothing, silently, which is worse than having no file: it reads as the place routing is configured.

2. Two of the six models it names no longer exist, checked against infra k3s/apps/ai-stack/litellm-config.yaml:

Model	In catalogue
`iguana/qwen3-coder-next`	❌ retired 2026-09-16, infra#416
`koala/qwen3-coder-30b`	❌ absent
`iguana/devstral`	✅
`iguana/deepseek-r1-14b`	✅
`koala/phi4-14b`	✅

koala/qwen3-coder-30b was already gone before today — infra's llm-consumers.yaml documents that exact name as the dead entry which broke agentsquad's LiteLLM grant. So the tdd skill has pointed at a nonexistent model for some time, and the file being unread is the only reason nobody noticed.

Why this is worth fixing rather than ignoring

This is the same defect class as three others found in the estate this week: a capability that exists, is correct-looking, and is connected to nothing (resolve_label_id never called, AGENTSQUAD_TIERS pointing at an unmounted path, agent/ready labels with no consumer). Each was invisible precisely because the artifact looked right.

Options

A. Delete the file. If env is the real configuration surface, the file is a decoy. Cheapest and most honest.

B. Make it load-bearing. Have the service read it and fail on startup when a named model is absent from LiteLLM's /v1/models. That turns the second problem into a boot-time error instead of a silent wrong answer.

C. Keep it as documentation — but then it has to be accurate, and something has to keep it that way, which is how it got here.

A or B. C is what it already was.

If B, the check is cheap: GET /v1/models on the gateway and set-difference the chains. infra now has the analogous gate for its canary — tests/canary-tiers-vs-catalogue_test.sh — which asserts every probed model exists in the catalogue, after exactly this failure mode kept a 51 GB model resident for nothing.

Created via git-mcp on behalf of @mathias

Found while retiring `iguana/qwen3-coder-next` estate-wide (infra#416) and sweeping every repo for references. `config/models.yaml` declares: ```yaml default_chain: - iguana/qwen3-coder-next skills: tdd: { chain: [koala/qwen3-coder-30b] } review: { chain: [iguana/devstral] } debug: { chain: [iguana/deepseek-r1-14b] } spec: { chain: [koala/phi4-14b] } retrospective:{ chain: [iguana/qwen3-coder-next] } trainer: { chain: [iguana/qwen3-coder-next] } ``` ## Two problems **1. The deployed service does not read this file.** The live `ingestion` deployment in the `supervisor` namespace names its models entirely by env: ``` INGEST_LLM_MODEL=koala/qwen35-9b-fast BRAIN_LLM_PRIMARY_MODEL=google/gemma-4-31B-it BRAIN_LLM_FALLBACK_MODEL=gemma4:31b BRAIN_LLM_TERTIARY_MODEL=gemini-flash-latest BRAIN_EMBED_MODEL=nomic-embed-text:latest BRAIN_RERANKER_MODEL=dengcao/Qwen3-Reranker-0.6B:F16 ``` Not one name from `config/models.yaml` appears. The file's own header says *"first entry per skill is used"* — nothing uses it. Editing it to change routing would change nothing, silently, which is worse than having no file: it reads as the place routing is configured. **2. Two of the six models it names no longer exist**, checked against `infra` `k3s/apps/ai-stack/litellm-config.yaml`: | Model | In catalogue | |---|---| | `iguana/qwen3-coder-next` | ❌ retired 2026-09-16, infra#416 | | `koala/qwen3-coder-30b` | ❌ absent | | `iguana/devstral` | ✅ | | `iguana/deepseek-r1-14b` | ✅ | | `koala/phi4-14b` | ✅ | `koala/qwen3-coder-30b` was already gone before today — infra's `llm-consumers.yaml` documents that exact name as the dead entry which broke agentsquad's LiteLLM grant. So the `tdd` skill has pointed at a nonexistent model for some time, and the file being unread is the only reason nobody noticed. ## Why this is worth fixing rather than ignoring This is the same defect class as three others found in the estate this week: a capability that exists, is correct-looking, and is connected to nothing (`resolve_label_id` never called, `AGENTSQUAD_TIERS` pointing at an unmounted path, `agent/ready` labels with no consumer). Each was invisible precisely because the artifact looked right. ## Options **A. Delete the file.** If env is the real configuration surface, the file is a decoy. Cheapest and most honest. **B. Make it load-bearing.** Have the service read it and fail on startup when a named model is absent from LiteLLM's `/v1/models`. That turns the second problem into a boot-time error instead of a silent wrong answer. **C. Keep it as documentation** — but then it has to be accurate, and something has to keep it that way, which is how it got here. **A or B.** C is what it already was. If B, the check is cheap: `GET /v1/models` on the gateway and set-difference the chains. infra now has the analogous gate for its canary — `tests/canary-tiers-vs-catalogue_test.sh` — which asserts every probed model exists in the catalogue, after exactly this failure mode kept a 51 GB model resident for nothing. --- _Created via git-mcp on behalf of @mathias_

mathias referenced this issue from a commit

2026-09-16 22:21:16 +00:00

fix(config): delete the model-routing chain — nothing has called it since April

mathias referenced a pull request that will close this issue

2026-09-16 22:21:31 +00:00

fix(config): delete the model-routing chain — nothing has called it since April #119

mathias added the risk/medium size/s type/fix labels 2026-09-16 22:23:51 +00:00

mathias closed this issue

2026-09-16 22:24:47 +00:00

mathias referenced this issue from a commit

2026-09-16 22:24:47 +00:00

fix(config): delete the model-routing chain — nothing has called it since April (#119)

mathias commented

2026-09-16 22:25:17 +00:00

Resolved via Option A — PR #119 merged (squash).

Merge-gate evidence

The PR's case rested entirely on a negative claim ("nothing reads this file"), so I re-established it independently rather than re-running the author's grep.

Searches performed (all against origin/main, i.e. pre-merge state):

git grep -n -I -E 'models\.yaml|ModelsFile|SUPERVISOR_MODELS_FILE|LoadModels|ModelFor|ModelsConfig' origin/main — searches the committed tree, not the working dir, so a dirty or shared checkout cannot hide a reference.
Same pattern over git ls-files | grep -v '^docs/' — catches every non-Go file type: shell, Taskfile, Procfile, Dockerfile, CI workflow, manifests.
internal/config/ scanned for reflect., go:generate, and yaml:" / json:" struct tags. Config carries no struct tags at all — so no YAML/JSON decoder could populate ModelsFile by name. This is the failure mode a plain symbol grep misses.
Interface / method-value indirection: grepped for ModelFor, config.Models, Models{ across all *.go. Zero hits outside the deleted files. ModelFor was a method on a concrete unexported-data struct satisfying no interface in the repo.
Second Go module. This repo has two — ./go.mod and ./ingestion/go.mod, and no go.work. Grepped ingestion/ separately: no references. (task check genuinely covers both — lint, test and vet each run once at root and once with cd ingestion. Verified by reading the Taskfile, not by trusting the desc: string.)
infra repo (/home/mathias/dev/AI/infra), grepped for SUPERVISOR_MODELS_FILE|models\.yaml|MODELS_FILE: only two hits, both in docs/plans/ prose describing this very issue. No ConfigMap, no env var, no volume mount. k3s/apps/supervisor/ contains a Deployment for ingestion only — there is no supervisor Deployment to mount anything.
Deployed image contents. The only Dockerfile in the repo is ingestion/Dockerfile, and cd.yml builds it with --local context=ingestion. config/ is outside that build context, so config/models.yaml was never baked into any deployed image. The ENV SUPERVISOR_MODELS_FILE=/app/config/models.yaml line in docs/superpowers/plans/2026-04-20-cd-pipeline.md is a plan that was never implemented — a naive grep hit that reads alarming and is inert.

Corroborating evidence for "superseded, not just unused": internal/config/routing.go configures model selection entirely by env — HYPERGUILD_FAST_MODEL (default koala/qwen35-9b-fast) and HYPERGUILD_THINKING_MODEL. That matches the INGEST_LLM_MODEL=koala/qwen35-9b-fast you quoted from the live Deployment. The env surface is the real one; the YAML was the decoy, exactly as the issue argued.

Gate

go build ./... — exit 0 (root) and exit 0 (ingestion)
CI=1 task check — exit 0, grep -c FAIL = 0 (lint + test -race + vet + govulncheck + gitleaks, both modules)
CI run 7374 — event pull_request, head_sha 351b187, conclusion success

Collateral damage check

Diff is git diff --numstat 0 additions / 131 deletions across all five files — a pure subtraction, so the line-filter edits to .env.example and config.go could not have mangled anything. Confirmed anyway: the removed .env.example line was the last of its block, leaving the blank line and the # LiteLLM gateway heading intact; config.go still parses with the struct and Load() intact. internal/config/config_test.go never referenced ModelsFile, so deleting models_test.go removed coverage of the deleted code only.

One finding the PR did not address (not a blocker)

Procfile:2 still sets SUPERVISOR_MODELS_FILE=./config/models.yaml. I checked whether this is a live reference and it is not: that line's command is go run ./cmd/supervisor/, and cmd/supervisor was deleted in c117634 ("retire supervisor pod"). The line was already non-functional on main before this PR, and setting an env var nothing reads breaks nothing. Pre-existing rot, not regression — filed separately along with the same class of staleness in Taskfile.yml and scripts/smoke-routing.sh.

Created via git-mcp on behalf of @mathias

Resolved via Option A — PR #119 merged (squash). ## Merge-gate evidence The PR's case rested entirely on a negative claim ("nothing reads this file"), so I re-established it independently rather than re-running the author's grep. **Searches performed** (all against `origin/main`, i.e. pre-merge state): 1. `git grep -n -I -E 'models\.yaml|ModelsFile|SUPERVISOR_MODELS_FILE|LoadModels|ModelFor|ModelsConfig' origin/main` — searches the *committed tree*, not the working dir, so a dirty or shared checkout cannot hide a reference. 2. Same pattern over `git ls-files | grep -v '^docs/'` — catches every non-Go file type: shell, Taskfile, Procfile, Dockerfile, CI workflow, manifests. 3. `internal/config/` scanned for `reflect.`, `go:generate`, and `yaml:"` / `json:"` struct tags. **`Config` carries no struct tags at all** — so no YAML/JSON decoder could populate `ModelsFile` by name. This is the failure mode a plain symbol grep misses. 4. Interface / method-value indirection: grepped for `ModelFor`, `config.Models`, `Models{` across all `*.go`. Zero hits outside the deleted files. `ModelFor` was a method on a concrete unexported-data struct satisfying no interface in the repo. 5. **Second Go module.** This repo has two — `./go.mod` and `./ingestion/go.mod`, and no `go.work`. Grepped `ingestion/` separately: no references. (`task check` genuinely covers both — `lint`, `test` and `vet` each run once at root and once with `cd ingestion`. Verified by reading the Taskfile, not by trusting the `desc:` string.) 6. **infra repo** (`/home/mathias/dev/AI/infra`), grepped for `SUPERVISOR_MODELS_FILE|models\.yaml|MODELS_FILE`: only two hits, both in `docs/plans/` prose describing *this very issue*. No ConfigMap, no env var, no volume mount. `k3s/apps/supervisor/` contains a Deployment for `ingestion` only — there is no supervisor Deployment to mount anything. 7. **Deployed image contents.** The only Dockerfile in the repo is `ingestion/Dockerfile`, and `cd.yml` builds it with `--local context=ingestion`. `config/` is outside that build context, so `config/models.yaml` was never baked into any deployed image. The `ENV SUPERVISOR_MODELS_FILE=/app/config/models.yaml` line in `docs/superpowers/plans/2026-04-20-cd-pipeline.md` is a *plan* that was never implemented — a naive grep hit that reads alarming and is inert. **Corroborating evidence for "superseded, not just unused":** `internal/config/routing.go` configures model selection entirely by env — `HYPERGUILD_FAST_MODEL` (default `koala/qwen35-9b-fast`) and `HYPERGUILD_THINKING_MODEL`. That matches the `INGEST_LLM_MODEL=koala/qwen35-9b-fast` you quoted from the live Deployment. The env surface is the real one; the YAML was the decoy, exactly as the issue argued. ## Gate - `go build ./...` — exit 0 (root) and exit 0 (ingestion) - `CI=1 task check` — **exit 0**, `grep -c FAIL` = **0** (lint + test `-race` + vet + govulncheck + gitleaks, both modules) - CI run [7374](https://git.d-ma.be/mathias/hyperguild/actions/runs/7374) — event `pull_request`, head_sha `351b187`, conclusion **success** ## Collateral damage check Diff is `git diff --numstat` **0 additions / 131 deletions** across all five files — a pure subtraction, so the line-filter edits to `.env.example` and `config.go` could not have mangled anything. Confirmed anyway: the removed `.env.example` line was the last of its block, leaving the blank line and the `# LiteLLM gateway` heading intact; `config.go` still parses with the struct and `Load()` intact. `internal/config/config_test.go` never referenced `ModelsFile`, so deleting `models_test.go` removed coverage of the deleted code only. ## One finding the PR did not address (not a blocker) `Procfile:2` still sets `SUPERVISOR_MODELS_FILE=./config/models.yaml`. I checked whether this is a live reference and it is not: that line's command is `go run ./cmd/supervisor/`, and `cmd/supervisor` was deleted in c117634 ("retire supervisor pod"). The line was already non-functional on `main` before this PR, and setting an env var nothing reads breaks nothing. Pre-existing rot, not regression — filed separately along with the same class of staleness in `Taskfile.yml` and `scripts/smoke-routing.sh`. --- _Created via git-mcp on behalf of @mathias_

mathias referenced this issue

2026-09-16 22:25:40 +00:00

Procfile, Taskfile and smoke-routing.sh still invoke cmd/supervisor and cmd/routing, neither of which exists #120

mathias referenced this issue from a commit

2026-09-16 22:32:33 +00:00

fix(taskfile): remove build targets for binaries deleted months ago, and make check build

mathias referenced this issue

2026-09-16 22:32:46 +00:00

fix(taskfile): remove build targets for binaries deleted months ago, and make check build #121

Sign in to join this conversation.