feat(data): EUR/USD hourly pipeline + 2008-2023 M1 dataset (#2)
CD / Lint / Test / Vet (push) Successful in 4s
CD / Build & Import (push) Failing after 8s
CD / Deploy via GitOps (push) Has been skipped

- scripts/prepare_hourly.py: M1→hourly aggregation (realized_vol = sqrt(Σr²),
  MIN_BARS=30 threshold, no weekend rows, year-based split preserved)
- tests/test_prepare_hourly.py: 5 TDD tests, all green
- train.py: USE_HOURLY=True, WINDOW=240 (10-day), PATCH_LEN=24 (1-day patches);
  build() prefers eurusd_hourly.parquet, falls back to daily; EXPORT BLOCK updated
- Taskfile.yml: data:fetch:historical, data:prepare:hourly, data:prepare:all, data:test
- 98,591 hourly rows (2008-2023) covering GFC, Euro crisis, Brexit, COVID, Fed cycle

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-06-25 13:12:48 +02:00
co-authored by Claude Sonnet 4.6
parent bde651b0df
commit e31905dc43
5 changed files with 292 additions and 10 deletions
+20
View File
@@ -18,6 +18,26 @@ tasks:
deps: [generate]
cmds: [go test ./... -race]
data:fetch:
desc: "Download EUR/USD M1 from histdata (set YEARS env var)"
cmds: [.venv/bin/python scripts/fetch_data.py]
data:fetch:historical:
desc: "Download EUR/USD M1 2008-2018 from histdata"
cmds:
- YEARS=2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018 .venv/bin/python scripts/fetch_data.py
data:prepare:daily:
desc: "Rebuild eurusd_daily.parquet from all M1 zips"
cmds: [.venv/bin/python scripts/prepare_data.py]
data:prepare:hourly:
desc: "Build eurusd_hourly.parquet from all M1 zips"
cmds: [.venv/bin/python scripts/prepare_hourly.py]
data:prepare:all:
desc: "Build both daily and hourly parquets"
deps: [data:prepare:daily, data:prepare:hourly]
data:test:
desc: "Run Python data pipeline tests"
cmds: [.venv/bin/python -m pytest tests/test_prepare_hourly.py tests/test_hepa.py -v]
eval:probe:
desc: "Run linear-probe (val_vol_r2) on embeddings from metrics.json"
cmds: [./bin/eval -metric probe]