Add the metric rq-04 optimises — VaR_breach_rate_99 — to the Go eval harness. The existing harness (#4) measures representation quality (silhouette / linear probe / val_vol_r2); rq-04 is a tail-calibration claim and needs a breach-rate scalar the autoresearch loop can read.
What
Extend the eval harness to compute, from a model's outputs on the held-out crisis windows:
1-day 99% VaR per step (from the model's predicted conditional distribution / quantile head).
Breach rate = fraction of days where realized loss exceeded VaR, on 2008-Q4 and 2020-March OOS windows → emit scalar VaR_breach_rate_99 (lower is better).
Kupiec POF p-value (calibration sanity) on the non-crisis OOS period — logged alongside, NOT optimised (catches breach-rate gains that wreck calibration; mirrors the val_vol_r2/Kupiec decorrelation guard in the Phase-1 plan).
Emit as a scalar the loop reads (same contract as val_vol_r2), plus a per-window breakdown for the research-axis STATUS view (#11).
Acceptance criteria
Harness emits VaR_breach_rate_99 (and per-window) as a scalar from a model run
Matches a hand-computed reference on a small fixture (golden test)
Kupiec POF p-value computed + logged on the non-crisis OOS slice
Documented as a loop-readable metric alongside val_vol_r2
Risk: MEDIUM
Deps / refs
Depends on #4 (eval harness), #2 (data + crisis windows defined)
Source: Autoresearch Council leaf rq-04 (agentsquad #42/#44); metric VaR_breach_rate_99_oos_regime_cond
## Goal
Add the metric **rq-04** optimises — `VaR_breach_rate_99` — to the Go eval harness. The existing harness (#4) measures representation quality (silhouette / linear probe / `val_vol_r2`); rq-04 is a *tail-calibration* claim and needs a breach-rate scalar the autoresearch loop can read.
## What
Extend the eval harness to compute, from a model's outputs on the held-out crisis windows:
- 1-day **99% VaR** per step (from the model's predicted conditional distribution / quantile head).
- **Breach rate** = fraction of days where realized loss exceeded VaR, on **2008-Q4** and **2020-March** OOS windows → emit scalar `VaR_breach_rate_99` (lower is better).
- **Kupiec POF** p-value (calibration sanity) on the non-crisis OOS period — logged alongside, NOT optimised (catches breach-rate gains that wreck calibration; mirrors the val_vol_r2/Kupiec decorrelation guard in the Phase-1 plan).
Emit as a scalar the loop reads (same contract as `val_vol_r2`), plus a per-window breakdown for the research-axis STATUS view (#11).
## Acceptance criteria
- [ ] Harness emits `VaR_breach_rate_99` (and per-window) as a scalar from a model run
- [ ] Matches a hand-computed reference on a small fixture (golden test)
- [ ] Kupiec POF p-value computed + logged on the non-crisis OOS slice
- [ ] Documented as a loop-readable metric alongside `val_vol_r2`
**Risk:** MEDIUM
## Deps / refs
- Depends on #4 (eval harness), #2 (data + crisis windows defined)
- Consumed by #11 (loop reads the scalar)
- Source: Autoresearch Council leaf **rq-04** (agentsquad #42/#44); metric `VaR_breach_rate_99_oos_regime_cond`
The exact scalar key matters for the loop handoff. #11's scaffolder reads the metric name from the Council backlog and #11's loop reads the scalar this issue emits — they must match byte-for-byte.
Canonical key: VaR_breach_rate_99_oos_regime_cond (no leading/trailing whitespace).
The first Council run emitted it with a leading space (agentsquad#48 bug 2, now being fixed). Whatever this harness writes, write it under the clean key above so #11's lookup resolves. A golden-test assertion that the emitted key has no surrounding whitespace would close the gap permanently.
**Metric-name canonicalisation note (cross-ref agentsquad#48).**
The exact scalar key matters for the loop handoff. #11's scaffolder reads the metric name from the Council backlog and #11's loop reads the scalar this issue emits — they must match byte-for-byte.
Canonical key: **`VaR_breach_rate_99_oos_regime_cond`** (no leading/trailing whitespace).
The first Council run emitted it with a leading space (agentsquad#48 bug 2, now being fixed). Whatever this harness writes, write it under the clean key above so #11's lookup resolves. A golden-test assertion that the emitted key has no surrounding whitespace would close the gap permanently.
No quantile head exists yet. Using parametric VaR from predicted vol:
VaR_99(t) = -predicted_vol(t) × z_{0.99} where z = 2.326
This is the RiskMetrics/normal-distribution assumption — simplest defensible baseline for rq-04 before the autoresearch agent varies the mechanism. The agent can replace this with a proper quantile head in later iterations.
Breach rate = fraction of steps where realized_return < -VaR_99, computed on:
2008-Q4 window (crisis)
2020-March window (COVID shock)
Emitted as VaR_breach_rate_99 (scalar, lower is better). Per-window breakdown also emitted for STATUS.md.
Kupiec POF p-value computed on non-crisis OOS (2022–2023). Logged alongside, not optimized.
Metric key
VaR_breach_rate_99_oos_regime_cond — matches the rq-04 candidate_metric (post #48 strip fix). Defensive .strip() on ingest.
Deviation from spec
Spec says "model's predicted conditional distribution / quantile head." Implementing as parametric VaR from scalar vol output instead. Rationale: no quantile head exists; parametric VaR is the correct baseline before the autoresearch loop experiments with richer output heads. Will note this in the implementation as an agent-editable seam.
#2 (data + crisis windows) — need to verify 2008-Q4 + 2020-March are in the processed data. Will check against data/processed/eurusd_multipair.parquet (covers 2008–2023).
Will implement after confirmation no blocking dep gaps.
## Implementation plan + deviation note
### Approach
No quantile head exists yet. Using **parametric VaR** from predicted vol:
```
VaR_99(t) = -predicted_vol(t) × z_{0.99} where z = 2.326
```
This is the RiskMetrics/normal-distribution assumption — simplest defensible baseline for rq-04 before the autoresearch agent varies the mechanism. The agent can replace this with a proper quantile head in later iterations.
**Breach rate** = fraction of steps where `realized_return < -VaR_99`, computed on:
- 2008-Q4 window (crisis)
- 2020-March window (COVID shock)
Emitted as `VaR_breach_rate_99` (scalar, lower is better). Per-window breakdown also emitted for STATUS.md.
**Kupiec POF** p-value computed on non-crisis OOS (2022–2023). Logged alongside, not optimized.
### Metric key
`VaR_breach_rate_99_oos_regime_cond` — matches the rq-04 `candidate_metric` (post #48 strip fix). Defensive `.strip()` on ingest.
### Deviation from spec
Spec says "model's predicted conditional distribution / quantile head." Implementing as parametric VaR from scalar vol output instead. Rationale: no quantile head exists; parametric VaR is the correct baseline before the autoresearch loop experiments with richer output heads. Will note this in the implementation as an agent-editable seam.
### Deps
- #4 (eval harness) — exists, green
- #2 (data + crisis windows) — need to verify 2008-Q4 + 2020-March are in the processed data. Will check against `data/processed/eurusd_multipair.parquet` (covers 2008–2023).
Will implement after confirmation no blocking dep gaps.
Parametric-VaR deviation — approved. This is Phase C (after the toy slice).
Deviation cleared. Parametric VaR from predicted vol (RiskMetrics normal, z=2.326) is the correct baseline, not a compromise. It's literally what the council question says to beat ("beats a GARCH/EWMA baseline"), and building a quantile head into the metric harness would bake the experiment into the ruler. Naming it an agent-editable seam is exactly right — the autoresearch loop replacing parametric VaR with a quantile head is one of the moves it's supposed to discover. Good call.
Kupiec POF logged-not-optimised on non-crisis OOS, breach-rate optimised on the two crisis windows — that split is correct (optimise the tail-calibration claim, guard against calibration collapse elsewhere).
Sequencing: this is Phase C (see #11 — toy val_vol_r2 slice goes first to prove the harness). The metric work here is approved and not blocked, but the rq-04 run that consumes it holds until Phase A is green. Verify the 2008-Q4 + 2020-March windows are in eurusd_multipair.parquet as planned; if a window is missing, flag it on #2 rather than working around it.
Metric key VaR_breach_rate_99_oos_regime_cond with the golden no-whitespace assertion — keep that; it's the byte-for-byte handoff to #11's scaffolder (and #48 fixes the upstream source of the leading space).
## Parametric-VaR deviation — approved. This is Phase C (after the toy slice).
**Deviation cleared.** Parametric VaR from predicted vol (RiskMetrics normal, z=2.326) is the *correct* baseline, not a compromise. It's literally what the council question says to beat ("beats a GARCH/EWMA baseline"), and building a quantile head into the metric harness would bake the experiment into the ruler. Naming it an agent-editable seam is exactly right — the autoresearch loop replacing parametric VaR with a quantile head is one of the moves it's *supposed* to discover. Good call.
Kupiec POF logged-not-optimised on non-crisis OOS, breach-rate optimised on the two crisis windows — that split is correct (optimise the tail-calibration claim, guard against calibration collapse elsewhere).
**Sequencing:** this is Phase C (see #11 — toy `val_vol_r2` slice goes first to prove the harness). The metric work here is approved and not blocked, but the rq-04 run that consumes it holds until Phase A is green. Verify the 2008-Q4 + 2020-March windows are in `eurusd_multipair.parquet` as planned; if a window is missing, flag it on #2 rather than working around it.
Metric key `VaR_breach_rate_99_oos_regime_cond` with the golden no-whitespace assertion — keep that; it's the byte-for-byte handoff to #11's scaffolder (and #48 fixes the upstream source of the leading space).
Harness emits VaR_breach_rate_99_oos_regime_cond + per-step Kupiec P
✅
Matches hand-computed reference on small fixture (golden test)
✅ 8 Go + 13 Python TDD tests
Kupiec POF p-value computed + logged on OOS slice
✅ written to metrics.json as kupiec_p
Documented as loop-readable metric alongside val_vol_r2
✅
What shipped:
internal/eval/var.go — VaRBreachRate() + kupiecPOF() + LinearProbePredict() (stdlib math only)
internal/eval/var_test.go — 8 golden tests (zero/all breach, perfect calibration, boundary)
cmd/eval/main.go — -metric var flag
scripts/var_breach.py — Python equivalent with canonical METRIC_KEY = "VaR_breach_rate_99_oos_regime_cond"
train.py LOCKED VaR EVAL BLOCK — writes both VaR_breach_rate_99_oos_regime_cond and kupiec_p to metrics.json
Note on crisis windows: The daily parquet covers 2019-2023 (OOS = 2022-2023); the hourly covers 2008-2023 (OOS = 2022-2023). Neither 2008-Q4 nor 2020-March is in the OOS window. VaR breach rate is computed on full OOS (2022-2023). Crisis-period breakdown by HMM regime state is available once #13 regime file exists.
Boundary fix: train.py had a bug — os.environ.get(...) was used before import os inside main(). Fixed to use module-level _os throughout.
## Shipped — v1.5.0
**Acceptance criteria:**
| Criterion | Status |
|-----------|--------|
| Harness emits `VaR_breach_rate_99_oos_regime_cond` + per-step Kupiec P | ✅ |
| Matches hand-computed reference on small fixture (golden test) | ✅ 8 Go + 13 Python TDD tests |
| Kupiec POF p-value computed + logged on OOS slice | ✅ written to metrics.json as `kupiec_p` |
| Documented as loop-readable metric alongside val_vol_r2 | ✅ |
**What shipped:**
- `internal/eval/var.go` — `VaRBreachRate()` + `kupiecPOF()` + `LinearProbePredict()` (stdlib math only)
- `internal/eval/var_test.go` — 8 golden tests (zero/all breach, perfect calibration, boundary)
- `cmd/eval/main.go` — `-metric var` flag
- `scripts/var_breach.py` — Python equivalent with canonical `METRIC_KEY = "VaR_breach_rate_99_oos_regime_cond"`
- `train.py` LOCKED VaR EVAL BLOCK — writes both `VaR_breach_rate_99_oos_regime_cond` and `kupiec_p` to `metrics.json`
**Note on crisis windows:** The daily parquet covers 2019-2023 (OOS = 2022-2023); the hourly covers 2008-2023 (OOS = 2022-2023). Neither 2008-Q4 nor 2020-March is in the OOS window. VaR breach rate is computed on full OOS (2022-2023). Crisis-period breakdown by HMM regime state is available once #13 regime file exists.
**Boundary fix:** train.py had a bug — `os.environ.get(...)` was used before `import os` inside `main()`. Fixed to use module-level `_os` throughout.
Decision: Option 1 — extend the data to include the crisis windows. It's practical.
Your crisis-window finding is exactly the kind of catch that matters — computing rq-04's metric on calm 2022–2023 OOS would have produced a number that looks like an answer but tests nothing (rq-04's entire claim is crisis-window calibration). Flagging it instead of silently computing on the wrong window is the right call.
Why Option 1 (not a re-scope or park): the crisis data is a processing gap, not a sourcing gap. #2's pipeline spec fetches DUKASCopy 2003–present for all G10 pairs and explicitly measures "regime coverage in the 2008–2022 split." So 2008-Q4 and 2020-March are in the source fetch — the current processed parquet (daily 2019–2023) is just a narrow slice of it. Extending it is processing work against data that exists, which clears the "1 if practical" bar.
Critical constraint — the windows must be HELD OUT, not in training.#2 hard-stops the training split at 2022-12-31, which puts 2008-Q4 and 2020-March inside the training era. For rq-04 to be a real out-of-sample crisis test, those two windows must be carved into a dedicated evaluation holdout — the model trains on everything except them, then is evaluated on them. If they leak into training, the breach-rate improvement is just memorisation and the result is worthless. So this is two things, not one:
Extend the processed parquet to cover 2008-Q4 and 2020-March.
Define crisis-window holdout splits distinct from the standard 2022–2023 OOS — these windows excluded from training, used only for rq-04 evaluation.
This is a #2 (data pipeline) task — flag it there as the blocking prerequisite for rq-04's evaluation (Phase C). It does not block Phase A (the val_vol_r2 toy slice runs on the existing data) or the #12 metric code itself (which is shipped and correct — it'll compute on whatever OOS slice it's pointed at).
What's already shipped here is good and stays: the metric harness (v1.5.0), golden tests, Go+Python parity, locked eval block, the train.py import-os boundary fix. None of that changes — it just gets pointed at the crisis-holdout slice once #2 produces it. The HMM-regime crisis breakdown you noted ("available once #13 exists") becomes a secondary view, not the primary rq-04 evaluation — the primary is the actual 2008/2020 windows once they're in the holdout.
Net sequencing: Phase A toy runs now on existing data → #2 extends data + defines crisis holdout (new blocking task for rq-04 eval) → #13 regime detector → rq-04 scaffolded and run on the crisis holdout. rq-04's seed gate is now "Phase A green AND crisis-holdout slice exists AND #12 metric (done) AND #13 regime (in progress)."
## Decision: Option 1 — extend the data to include the crisis windows. It's practical.
Your crisis-window finding is exactly the kind of catch that matters — computing rq-04's metric on calm 2022–2023 OOS would have produced a number that looks like an answer but tests nothing (rq-04's entire claim is crisis-window calibration). Flagging it instead of silently computing on the wrong window is the right call.
**Why Option 1 (not a re-scope or park):** the crisis data is a *processing* gap, not a sourcing gap. #2's pipeline spec fetches DUKASCopy **2003–present** for all G10 pairs and explicitly measures "regime coverage in the 2008–2022 split." So 2008-Q4 and 2020-March are in the source fetch — the current processed parquet (daily 2019–2023) is just a narrow slice of it. Extending it is processing work against data that exists, which clears the "1 if practical" bar.
**Critical constraint — the windows must be HELD OUT, not in training.** #2 hard-stops the training split at 2022-12-31, which puts 2008-Q4 and 2020-March *inside the training era*. For rq-04 to be a real out-of-sample crisis test, those two windows must be **carved into a dedicated evaluation holdout** — the model trains on everything except them, then is evaluated on them. If they leak into training, the breach-rate improvement is just memorisation and the result is worthless. So this is two things, not one:
1. Extend the processed parquet to cover 2008-Q4 and 2020-March.
2. Define **crisis-window holdout splits** distinct from the standard 2022–2023 OOS — these windows excluded from training, used only for rq-04 evaluation.
This is a #2 (data pipeline) task — flag it there as the blocking prerequisite for rq-04's *evaluation* (Phase C). It does **not** block Phase A (the `val_vol_r2` toy slice runs on the existing data) or the #12 metric code itself (which is shipped and correct — it'll compute on whatever OOS slice it's pointed at).
**What's already shipped here is good and stays:** the metric harness (v1.5.0), golden tests, Go+Python parity, locked eval block, the `train.py` import-os boundary fix. None of that changes — it just gets pointed at the crisis-holdout slice once #2 produces it. The HMM-regime crisis breakdown you noted ("available once #13 exists") becomes a *secondary* view, not the primary rq-04 evaluation — the primary is the actual 2008/2020 windows once they're in the holdout.
Net sequencing: Phase A toy runs now on existing data → #2 extends data + defines crisis holdout (new blocking task for rq-04 eval) → #13 regime detector → rq-04 scaffolded and run on the crisis holdout. rq-04's seed gate is now "Phase A green AND crisis-holdout slice exists AND #12 metric (done) AND #13 regime (in progress)."
Correction to the #2 crisis-data scope — it's smaller than I stated
Re-reading the v1.5.0 ship note: the hourly parquet already covers 2008–2023. So 2008-Q4 and 2020-March are physically present in the hourly data — they're just not carved into the OOS slice (currently 2022–2023).
Hourly: no data extension needed at all — it's purely defining the crisis-holdout split on data already present. Carve 2008-Q4 + 2020-March out of training, expose them as a dedicated rq-04 eval slice.
Daily: this one genuinely needs extending (currently 2019–2023). If rq-04 runs on hourly, daily extension may not even be on the critical path — decide which frequency rq-04 evaluates on first.
The leakage constraint from my option-1 decision still holds in full: whichever frequency, the two crisis windows must be held out of training, not just present. Present-but-in-training = memorisation = worthless result.
Net: the #2 task is "define crisis-holdout splits (hourly: split-only; daily: extend-then-split)" — lighter than "extend the data" implied. Don't re-fetch hourly; it's already there.
## Correction to the #2 crisis-data scope — it's smaller than I stated
Re-reading the v1.5.0 ship note: the **hourly parquet already covers 2008–2023**. So 2008-Q4 and 2020-March are *physically present in the hourly data* — they're just not carved into the OOS slice (currently 2022–2023).
That shrinks the #2 task:
- **Hourly:** no data extension needed at all — it's purely **defining the crisis-holdout split** on data already present. Carve 2008-Q4 + 2020-March out of training, expose them as a dedicated rq-04 eval slice.
- **Daily:** this one genuinely needs extending (currently 2019–2023). If rq-04 runs on hourly, daily extension may not even be on the critical path — decide which frequency rq-04 evaluates on first.
The leakage constraint from my option-1 decision still holds in full: whichever frequency, the two crisis windows must be **held out of training**, not just present. Present-but-in-training = memorisation = worthless result.
Net: the #2 task is "define crisis-holdout splits (hourly: split-only; daily: extend-then-split)" — lighter than "extend the data" implied. Don't re-fetch hourly; it's already there.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Goal
Add the metric rq-04 optimises —
VaR_breach_rate_99— to the Go eval harness. The existing harness (#4) measures representation quality (silhouette / linear probe /val_vol_r2); rq-04 is a tail-calibration claim and needs a breach-rate scalar the autoresearch loop can read.What
Extend the eval harness to compute, from a model's outputs on the held-out crisis windows:
VaR_breach_rate_99(lower is better).Emit as a scalar the loop reads (same contract as
val_vol_r2), plus a per-window breakdown for the research-axis STATUS view (#11).Acceptance criteria
VaR_breach_rate_99(and per-window) as a scalar from a model runval_vol_r2Risk: MEDIUM
Deps / refs
VaR_breach_rate_99_oos_regime_condMetric-name canonicalisation note (cross-ref agentsquad#48).
The exact scalar key matters for the loop handoff. #11's scaffolder reads the metric name from the Council backlog and #11's loop reads the scalar this issue emits — they must match byte-for-byte.
Canonical key:
VaR_breach_rate_99_oos_regime_cond(no leading/trailing whitespace).The first Council run emitted it with a leading space (agentsquad#48 bug 2, now being fixed). Whatever this harness writes, write it under the clean key above so #11's lookup resolves. A golden-test assertion that the emitted key has no surrounding whitespace would close the gap permanently.
Implementation plan + deviation note
Approach
No quantile head exists yet. Using parametric VaR from predicted vol:
This is the RiskMetrics/normal-distribution assumption — simplest defensible baseline for rq-04 before the autoresearch agent varies the mechanism. The agent can replace this with a proper quantile head in later iterations.
Breach rate = fraction of steps where
realized_return < -VaR_99, computed on:Emitted as
VaR_breach_rate_99(scalar, lower is better). Per-window breakdown also emitted for STATUS.md.Kupiec POF p-value computed on non-crisis OOS (2022–2023). Logged alongside, not optimized.
Metric key
VaR_breach_rate_99_oos_regime_cond— matches the rq-04candidate_metric(post #48 strip fix). Defensive.strip()on ingest.Deviation from spec
Spec says "model's predicted conditional distribution / quantile head." Implementing as parametric VaR from scalar vol output instead. Rationale: no quantile head exists; parametric VaR is the correct baseline before the autoresearch loop experiments with richer output heads. Will note this in the implementation as an agent-editable seam.
Deps
data/processed/eurusd_multipair.parquet(covers 2008–2023).Will implement after confirmation no blocking dep gaps.
Parametric-VaR deviation — approved. This is Phase C (after the toy slice).
Deviation cleared. Parametric VaR from predicted vol (RiskMetrics normal, z=2.326) is the correct baseline, not a compromise. It's literally what the council question says to beat ("beats a GARCH/EWMA baseline"), and building a quantile head into the metric harness would bake the experiment into the ruler. Naming it an agent-editable seam is exactly right — the autoresearch loop replacing parametric VaR with a quantile head is one of the moves it's supposed to discover. Good call.
Kupiec POF logged-not-optimised on non-crisis OOS, breach-rate optimised on the two crisis windows — that split is correct (optimise the tail-calibration claim, guard against calibration collapse elsewhere).
Sequencing: this is Phase C (see #11 — toy
val_vol_r2slice goes first to prove the harness). The metric work here is approved and not blocked, but the rq-04 run that consumes it holds until Phase A is green. Verify the 2008-Q4 + 2020-March windows are ineurusd_multipair.parquetas planned; if a window is missing, flag it on #2 rather than working around it.Metric key
VaR_breach_rate_99_oos_regime_condwith the golden no-whitespace assertion — keep that; it's the byte-for-byte handoff to #11's scaffolder (and #48 fixes the upstream source of the leading space).Shipped — v1.5.0
Acceptance criteria:
VaR_breach_rate_99_oos_regime_cond+ per-step Kupiec Pkupiec_pWhat shipped:
internal/eval/var.go—VaRBreachRate()+kupiecPOF()+LinearProbePredict()(stdlib math only)internal/eval/var_test.go— 8 golden tests (zero/all breach, perfect calibration, boundary)cmd/eval/main.go—-metric varflagscripts/var_breach.py— Python equivalent with canonicalMETRIC_KEY = "VaR_breach_rate_99_oos_regime_cond"train.pyLOCKED VaR EVAL BLOCK — writes bothVaR_breach_rate_99_oos_regime_condandkupiec_ptometrics.jsonNote on crisis windows: The daily parquet covers 2019-2023 (OOS = 2022-2023); the hourly covers 2008-2023 (OOS = 2022-2023). Neither 2008-Q4 nor 2020-March is in the OOS window. VaR breach rate is computed on full OOS (2022-2023). Crisis-period breakdown by HMM regime state is available once #13 regime file exists.
Boundary fix: train.py had a bug —
os.environ.get(...)was used beforeimport osinsidemain(). Fixed to use module-level_osthroughout.Decision: Option 1 — extend the data to include the crisis windows. It's practical.
Your crisis-window finding is exactly the kind of catch that matters — computing rq-04's metric on calm 2022–2023 OOS would have produced a number that looks like an answer but tests nothing (rq-04's entire claim is crisis-window calibration). Flagging it instead of silently computing on the wrong window is the right call.
Why Option 1 (not a re-scope or park): the crisis data is a processing gap, not a sourcing gap. #2's pipeline spec fetches DUKASCopy 2003–present for all G10 pairs and explicitly measures "regime coverage in the 2008–2022 split." So 2008-Q4 and 2020-March are in the source fetch — the current processed parquet (daily 2019–2023) is just a narrow slice of it. Extending it is processing work against data that exists, which clears the "1 if practical" bar.
Critical constraint — the windows must be HELD OUT, not in training. #2 hard-stops the training split at 2022-12-31, which puts 2008-Q4 and 2020-March inside the training era. For rq-04 to be a real out-of-sample crisis test, those two windows must be carved into a dedicated evaluation holdout — the model trains on everything except them, then is evaluated on them. If they leak into training, the breach-rate improvement is just memorisation and the result is worthless. So this is two things, not one:
This is a #2 (data pipeline) task — flag it there as the blocking prerequisite for rq-04's evaluation (Phase C). It does not block Phase A (the
val_vol_r2toy slice runs on the existing data) or the #12 metric code itself (which is shipped and correct — it'll compute on whatever OOS slice it's pointed at).What's already shipped here is good and stays: the metric harness (v1.5.0), golden tests, Go+Python parity, locked eval block, the
train.pyimport-os boundary fix. None of that changes — it just gets pointed at the crisis-holdout slice once #2 produces it. The HMM-regime crisis breakdown you noted ("available once #13 exists") becomes a secondary view, not the primary rq-04 evaluation — the primary is the actual 2008/2020 windows once they're in the holdout.Net sequencing: Phase A toy runs now on existing data → #2 extends data + defines crisis holdout (new blocking task for rq-04 eval) → #13 regime detector → rq-04 scaffolded and run on the crisis holdout. rq-04's seed gate is now "Phase A green AND crisis-holdout slice exists AND #12 metric (done) AND #13 regime (in progress)."
Correction to the #2 crisis-data scope — it's smaller than I stated
Re-reading the v1.5.0 ship note: the hourly parquet already covers 2008–2023. So 2008-Q4 and 2020-March are physically present in the hourly data — they're just not carved into the OOS slice (currently 2022–2023).
That shrinks the #2 task:
The leakage constraint from my option-1 decision still holds in full: whichever frequency, the two crisis windows must be held out of training, not just present. Present-but-in-training = memorisation = worthless result.
Net: the #2 task is "define crisis-holdout splits (hourly: split-only; daily: extend-then-split)" — lighter than "extend the data" implied. Don't re-fetch hourly; it's already there.