Implement the Go data pipeline for downloading and processing G10 FX tick data from DUKASCopy into hourly OHLCV + feature format ready for model training.
Acceptance criteria
task data:fetch downloads all 10 G10 pairs, 2003–present, to data/raw/
task data:process resamples ticks → hourly OHLCV, computes log-return and rolling 20-period HV, merges VIX (daily → hourly interpolation), outputs to data/processed/
task data:validate reports: gap count per pair, outlier ticks (>5σ), regime coverage (% high-vol periods in 2008–2022 train split)
Hard stop at 2022-12-31 for training split; 2023 window handled separately
All pipeline logic has Go unit tests (task test passes)
Data schema documented in data/README.md
Notes
DUKASCopy API is HTTP-based; tick data is free but rate-limited — implement respectful backoff
Weekend gaps (FX market closed) must be handled explicitly — do not interpolate across them
This is Phase 0 prerequisite — nothing else can start without this
Spec
Write a feature-spec for this component before implementation. Load .skills/feature-spec/SKILL.md.
## What
Implement the Go data pipeline for downloading and processing G10 FX tick data from DUKASCopy into hourly OHLCV + feature format ready for model training.
## Acceptance criteria
- [ ] `task data:fetch` downloads all 10 G10 pairs, 2003–present, to `data/raw/`
- [ ] `task data:process` resamples ticks → hourly OHLCV, computes log-return and rolling 20-period HV, merges VIX (daily → hourly interpolation), outputs to `data/processed/`
- [ ] `task data:validate` reports: gap count per pair, outlier ticks (>5σ), regime coverage (% high-vol periods in 2008–2022 train split)
- [ ] Hard stop at 2022-12-31 for training split; 2023 window handled separately
- [ ] All pipeline logic has Go unit tests (`task test` passes)
- [ ] Data schema documented in `data/README.md`
## Notes
- DUKASCopy API is HTTP-based; tick data is free but rate-limited — implement respectful backoff
- Weekend gaps (FX market closed) must be handled explicitly — do not interpolate across them
- This is Phase 0 prerequisite — nothing else can start without this
## Spec
Write a `feature-spec` for this component before implementation. Load `.skills/feature-spec/SKILL.md`.
New blocking sub-task for rq-04 evaluation: crisis-window holdout splits
Surfaced by #12 (VaR-breach metric shipped) and decided on #12: rq-04's hypothesis is specifically about VaR-breach calibration during 2008-Q4 and 2020-March, but the current processed parquet (daily 2019–2023) doesn't include those windows — OOS is calm 2022–2023. rq-04 can't be honestly evaluated until they're present and held out.
This pipeline already fetches the right source data (spec: DUKASCopy 2003–present, regime coverage measured on 2008–2022), so this is processing + splitting, not new sourcing.
Task:
Extend the processed parquet (daily + hourly) to cover 2008-Q4 and 2020-March from the existing 2003–present raw fetch.
Define crisis-window holdout splits — 2008-Q4 and 2020-March excluded from the training split and from the standard 2022–2023 OOS; used only as a dedicated rq-04 crisis-evaluation slice.
Critical — leakage: the current spec hard-stops training at 2022-12-31, which places both crisis windows inside the training era. They MUST be carved out of training, or the rq-04 breach-rate improvement is memorisation, not generalisation. The holdout must be a true out-of-sample-on-the-crisis construction: train on all-except-the-two-windows, evaluate on the two windows.
Blocking relationship: this gates rq-04's evaluation (jepa-fx Phase C). It does NOT gate Phase A (the val_vol_r2 toy slice, which runs on existing data) or the #12 metric code (shipped, correct, slice-agnostic). Suggest a data:crisis-holdout task target (or extend data:process with a crisis-eval split flag) + the data:validate regime-coverage report extended to confirm the two windows are present and non-empty.
rq-04 seed gate is now: Phase A green AND this crisis-holdout slice exists AND #13 regime detector done (#12 metric already done).
## New blocking sub-task for rq-04 evaluation: crisis-window holdout splits
Surfaced by #12 (VaR-breach metric shipped) and decided on #12: rq-04's hypothesis is specifically about VaR-breach calibration **during 2008-Q4 and 2020-March**, but the current processed parquet (daily 2019–2023) doesn't include those windows — OOS is calm 2022–2023. rq-04 can't be honestly evaluated until they're present and held out.
This pipeline already fetches the right source data (spec: DUKASCopy 2003–present, regime coverage measured on 2008–2022), so this is processing + splitting, not new sourcing.
**Task:**
1. **Extend the processed parquet** (daily + hourly) to cover 2008-Q4 and 2020-March from the existing 2003–present raw fetch.
2. **Define crisis-window holdout splits** — 2008-Q4 and 2020-March excluded from the training split and from the standard 2022–2023 OOS; used *only* as a dedicated rq-04 crisis-evaluation slice.
**Critical — leakage:** the current spec hard-stops training at 2022-12-31, which places both crisis windows *inside the training era*. They MUST be carved out of training, or the rq-04 breach-rate improvement is memorisation, not generalisation. The holdout must be a true out-of-sample-on-the-crisis construction: train on all-except-the-two-windows, evaluate on the two windows.
**Blocking relationship:** this gates rq-04's *evaluation* (jepa-fx Phase C). It does NOT gate Phase A (the `val_vol_r2` toy slice, which runs on existing data) or the #12 metric code (shipped, correct, slice-agnostic). Suggest a `data:crisis-holdout` task target (or extend `data:process` with a crisis-eval split flag) + the `data:validate` regime-coverage report extended to confirm the two windows are present and non-empty.
rq-04 seed gate is now: Phase A green AND this crisis-holdout slice exists AND #13 regime detector done (#12 metric already done).
Crisis-holdout split task — blocking prerequisite for rq-04 (Phase C)
Surfaced during #12/#13 review. Scope smaller than the "extend the data" framing implied:
Hourly (2008-2023 — data already present):
No data extension needed
Task: define crisis-holdout split — carve 2008-Q4 (Oct–Dec 2008) and 2020-March (Mar 2020) out of the training set, expose as a dedicated rq-04 eval slice
These windows must be held out of training entirely — present-but-in-training = memorisation = worthless result
Daily (2019-2023 — short):
Needs extension if rq-04 runs on daily frequency
Decide which frequency rq-04 evaluates on first; if hourly, daily extension is off the critical path
rq-04 Phase C gate: crisis-holdout slice exists AND eurusd_regime.parquet regenerated against it (so HMM state labels cover the two crisis windows) AND Phase A green.
Nothing here blocks Phase A (toy val_vol_r2 slice on existing data). Metric harness v1.5.0 already correct — it'll point at whatever slice it's given.
## Crisis-holdout split task — blocking prerequisite for rq-04 (Phase C)
Surfaced during #12/#13 review. Scope smaller than the "extend the data" framing implied:
**Hourly (2008-2023 — data already present):**
- No data extension needed
- Task: define crisis-holdout split — carve 2008-Q4 (Oct–Dec 2008) and 2020-March (Mar 2020) out of the training set, expose as a dedicated rq-04 eval slice
- These windows must be **held out of training entirely** — present-but-in-training = memorisation = worthless result
**Daily (2019-2023 — short):**
- Needs extension if rq-04 runs on daily frequency
- Decide which frequency rq-04 evaluates on first; if hourly, daily extension is off the critical path
**rq-04 Phase C gate:** crisis-holdout slice exists AND `eurusd_regime.parquet` regenerated against it (so HMM state labels cover the two crisis windows) AND Phase A green.
Nothing here blocks Phase A (toy `val_vol_r2` slice on existing data). Metric harness v1.5.0 already correct — it'll point at whatever slice it's given.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What
Implement the Go data pipeline for downloading and processing G10 FX tick data from DUKASCopy into hourly OHLCV + feature format ready for model training.
Acceptance criteria
task data:fetchdownloads all 10 G10 pairs, 2003–present, todata/raw/task data:processresamples ticks → hourly OHLCV, computes log-return and rolling 20-period HV, merges VIX (daily → hourly interpolation), outputs todata/processed/task data:validatereports: gap count per pair, outlier ticks (>5σ), regime coverage (% high-vol periods in 2008–2022 train split)task testpasses)data/README.mdNotes
Spec
Write a
feature-specfor this component before implementation. Load.skills/feature-spec/SKILL.md.New blocking sub-task for rq-04 evaluation: crisis-window holdout splits
Surfaced by #12 (VaR-breach metric shipped) and decided on #12: rq-04's hypothesis is specifically about VaR-breach calibration during 2008-Q4 and 2020-March, but the current processed parquet (daily 2019–2023) doesn't include those windows — OOS is calm 2022–2023. rq-04 can't be honestly evaluated until they're present and held out.
This pipeline already fetches the right source data (spec: DUKASCopy 2003–present, regime coverage measured on 2008–2022), so this is processing + splitting, not new sourcing.
Task:
Critical — leakage: the current spec hard-stops training at 2022-12-31, which places both crisis windows inside the training era. They MUST be carved out of training, or the rq-04 breach-rate improvement is memorisation, not generalisation. The holdout must be a true out-of-sample-on-the-crisis construction: train on all-except-the-two-windows, evaluate on the two windows.
Blocking relationship: this gates rq-04's evaluation (jepa-fx Phase C). It does NOT gate Phase A (the
val_vol_r2toy slice, which runs on existing data) or the #12 metric code (shipped, correct, slice-agnostic). Suggest adata:crisis-holdouttask target (or extenddata:processwith a crisis-eval split flag) + thedata:validateregime-coverage report extended to confirm the two windows are present and non-empty.rq-04 seed gate is now: Phase A green AND this crisis-holdout slice exists AND #13 regime detector done (#12 metric already done).
Crisis-holdout split task — blocking prerequisite for rq-04 (Phase C)
Surfaced during #12/#13 review. Scope smaller than the "extend the data" framing implied:
Hourly (2008-2023 — data already present):
Daily (2019-2023 — short):
rq-04 Phase C gate: crisis-holdout slice exists AND
eurusd_regime.parquetregenerated against it (so HMM state labels cover the two crisis windows) AND Phase A green.Nothing here blocks Phase A (toy
val_vol_r2slice on existing data). Metric harness v1.5.0 already correct — it'll point at whatever slice it's given.