loop.py treats agent-call and train failures as non-fatal (logs, continue) and always sys.exit(0) at the end. So a run where all iterations fail still reports Job succeeded.
Observed in run #54 (commit 8539ec3, before the model fix): all 3 agent calls returned HTTP 400, the loop did zero work (best == baseline, delta +0.0000), yet CI was green. Without reading the log you'd believe the autoresearch loop ran successfully.
Why it matters
A green CI on a no-op run masks regressions — a broken LLM endpoint, a bad model name, or a malformed prompt all pass silently. The signal "the loop optimized something" is exactly what should gate green.
Suggested fix
Make the loop exit non-zero (or at least emit a CI-visible warning / ::warning::) when no iteration produced a valid train run — i.e. every iteration was agent-fail or train-fail. A baseline-only run with 0 successful iterations is not a success.
Source: Autoresearch Loop bring-up, 2026-06-29 (runs #54/#56). Fixed separately: the empty-LOOP_MODEL 400 cause (6fb77f5). This issue is the green-on-no-op masking that hid it.
## Problem
`loop.py` treats agent-call and train failures as non-fatal (logs, `continue`) and always `sys.exit(0)` at the end. So a run where **all** iterations fail still reports `Job succeeded`.
Observed in run #54 (commit 8539ec3, before the model fix): all 3 agent calls returned `HTTP 400`, the loop did zero work (`best == baseline`, `delta +0.0000`), yet CI was green. Without reading the log you'd believe the autoresearch loop ran successfully.
## Why it matters
A green CI on a no-op run masks regressions — a broken LLM endpoint, a bad model name, or a malformed prompt all pass silently. The signal "the loop optimized something" is exactly what should gate green.
## Suggested fix
Make the loop exit non-zero (or at least emit a CI-visible warning / `::warning::`) when **no iteration produced a valid train run** — i.e. every iteration was `agent-fail` or `train-fail`. A baseline-only run with 0 successful iterations is not a success.
Source: Autoresearch Loop bring-up, 2026-06-29 (runs #54/#56). Fixed separately: the empty-`LOOP_MODEL` 400 cause (6fb77f5). This issue is the green-on-no-op masking that hid it.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Problem
loop.pytreats agent-call and train failures as non-fatal (logs,continue) and alwayssys.exit(0)at the end. So a run where all iterations fail still reportsJob succeeded.Observed in run #54 (commit
8539ec3, before the model fix): all 3 agent calls returnedHTTP 400, the loop did zero work (best == baseline,delta +0.0000), yet CI was green. Without reading the log you'd believe the autoresearch loop ran successfully.Why it matters
A green CI on a no-op run masks regressions — a broken LLM endpoint, a bad model name, or a malformed prompt all pass silently. The signal "the loop optimized something" is exactly what should gate green.
Suggested fix
Make the loop exit non-zero (or at least emit a CI-visible warning /
::warning::) when no iteration produced a valid train run — i.e. every iteration wasagent-failortrain-fail. A baseline-only run with 0 successful iterations is not a success.Source: Autoresearch Loop bring-up, 2026-06-29 (runs #54/#56). Fixed separately: the empty-
LOOP_MODEL400 cause (6fb77f5). This issue is the green-on-no-op masking that hid it.