docs: add .skills/README.md listing active skills

.context/PROJECT.md

@@ -4,101 +4,77 @@
 
 - **Name**: jepa-fx-risk
 - **Owner**: Mathias
-- **Client**: personal research
+- **Client**: personal / research
 - **Repo**: gitea.d-ma.be/mathias/jepa-fx-risk
-- **Status**: active — Phase 0
+- **Status**: active
+- **Brain wing**: `jepa-fx` (`wiki/jepa-fx/`)
 
 ## Purpose
 
-Research project: apply JEPA-based self-supervised representation learning to
-FX risk management for a corporate bank with an internal global FX trading desk.
-
-Primary tasks: FX volatility forecasting and VaR/CVaR estimation.
-Target output: internal PoC for the trading desk.
-
-## Architecture decision (see DECISIONS.md ADR-001)
-
-**TS-JEPA + SIGReg** — TS-JEPA temporal patchwise architecture (Ennadir et al.,
-arXiv:2509.25449) with EMA replaced by Sketched Isotropic Gaussian Regularization
-(SIGReg, Balestriero & LeCun, arXiv:2511.08544). Single search axis: λ ∈ [0.01, 1.0].
-
-Phase 2 hypothesis: MTS-JEPA multi-resolution objective (arXiv:2602.04643).
+Research project. Not a product. The "user" is future-self and research readers. Success is a reproducible, publishable result — not deployment.
 
 ## Stack
 
-**Go** (`src/`): data pipeline (DUKASCopy fetch + hourly processing), evaluation
-harness (silhouette, linear probe R², collapse diagnostic, Kupiec/Christoffersen),
-results dashboard (Templ + HTMX + CDN Tailwind).
+- **Primary language**: Go 1.24+ (pipeline, eval harness, CLI, experiment runner)
+- **ML layer**: Python 3.12 + PyTorch (TS-JEPA training loop only — isolated in `model/`)
+- **Build**: Task (Taskfile.yml)
+- **Target infra**: koala (Arch + Blackwell GPU, training), iguana (Mac Studio M2, dev)
+- **MCP**: brain (knowledge), gitea (version control)
 
-**Python** (`model/`): all training and embedding export only. PyTorch cu130
-(Blackwell sm_120 compatible). No evaluation logic in Python.
+## Research context
 
-**Infra**: koala (Arch Linux, Blackwell GPU 12 GB VRAM) for training.
-iguana (Mac Studio M2 Ultra) + LiteLLM on piguard for autoresearch agent LLM.
+**Primary hypothesis:**
+> JEPA embeddings trained on FX time-series will produce latent market-state representations structurally separable by regime without explicit labels, measurable by silhouette score > 0.35 on k-means clusters vs. realised-volatility regime labels, on held-out data including at least one structural break.
 
-## Repository layout
+**Current phase:** Phase 0 — SSL Feasibility Gate
 
-```
-jepa-fx-risk/
-├── src/                    # Go — data pipeline + eval harness + dashboard
-│   ├── data/               # DUKASCopy fetch, hourly processing, validation
-│   └── eval/               # silhouette, linear probe, collapse, backtest
-├── model/                  # Python — training only
-│   ├── train.py            # TS-JEPA + SIGReg backbone (autoresearch edits this)
-│   ├── prepare.py          # LOCKED — data loading, tokenization, export
-│   └── requirements.txt
-├── specs/                  # Research specs (one per phase/experiment type)
-├── experiments/            # Per-run outputs: embeddings, metrics.json, git tag
-├── results/summaries/      # Human-readable outcome per experiment
-├── program.md              # Autoresearch agenda — researcher edits this
-├── DECISIONS.md            # Architecture Decision Records
-└── Taskfile.yml            # task data:fetch, task experiment:run, task eval:*
-```
+**Training cutoff:** 2023-01-01 (hard — never look at post-2023 data during development)
 
-## Phase structure
+**Out of scope:** options pricing, directional alpha, live/paper trading, exotic pairs (Phases 1–3)
 
-- **Phase 0** (current): MAE baseline on EUR/USD hourly 2008–2022.
-  Gate: silhouette > 0.20, MAE > PCA, ±10% over 3 reruns.
-- **Phase 1**: TS-JEPA + SIGReg autoresearch sweep, 50 experiments.
-  Gate: val_vol_r2 > GARCH baseline AND Kupiec p > 0.05.
-- **Phase 2**: MTS-JEPA multi-resolution hypothesis.
-- **Phase 3**: Internal bank tick/position data (future, out of current scope).
+**Brain wing for prior decisions and failure modes:** `brain_query wing=jepa-fx`
 
-## Data
+## Conventions
 
-DUKASCopy hourly OHLCV, 10 G10 pairs, 2008–2022 train / 2023 val / 2024 test.
-Features: log-return, log rolling-20-period HV, VIX (daily interpolated).
-Weekend gaps handled explicitly. See ADR-002.
+### Scientific discipline
+- Every experiment has a spec in `specs/` before any code runs
+- Hypotheses are falsifiable and have quantitative acceptance criteria
+- Null results are recorded and published — not discarded
+- Training cutoff is sacred — post-2023 data never informs any design decision
+- Results reported with baselines; no cherry-picking
 
-## Key conventions
+### Code style
+- Go: `gofumpt`, `golangci-lint` with project config; table-driven tests; `testify`
+- Errors: `fmt.Errorf("context: %w", err)` — no naked returns
+- Python: `ruff` for lint; type hints throughout; `pytest` for tests
+- No Jupyter notebooks for anything reproducible — notebooks are EDA scratch only
 
-- `prepare.py` is LOCKED — never modified by agents or autoresearch
-- Evaluation metrics are always computed by the Go harness, never in Python
-- Every experiment gets a git tag: `exp/YYYYMMDD-description`
-- Null results are recorded explicitly in `results/summaries/` — do not iterate silently
-- `program.md` is the only file the researcher edits to steer autoresearch
-- CI runs `task check` (lint + vet + test) only — no GPU, no training
+### Git
+- Conventional commits: `feat:`, `fix:`, `chore:`, `docs:`, `experiment:`, `result:`
+- Branch: `feat/`, `experiment/phase-N-description`, `fix/`
+- Every experiment run gets a git tag: `exp/YYYYMMDD-short-description`
+- PRs: one concern per PR; description explains *why* not *what*
 
-## Evaluation metrics
+### Experiment discipline
+- One spec per phase in `specs/` — written before any implementation
+- Each run recorded in `experiments/YYYYMMDD-HHMMSS-description/`
+- Metric summaries committed to `results/summaries/` — large outputs gitignored
+- `task check` must pass before any commit
 
-Primary (autoresearch optimizes): `val_vol_r2` — linear probe R² on 1-day
-realized volatility from frozen embeddings, computed by Go harness.
+### Security / data
+- No raw FX data committed (gitignored) — see `data/README.md` for reproducible download
+- No API keys or tokens in code — env vars only
+- Training data and results stay local — nothing to cloud unless explicitly decided
 
-Secondary (logged, not optimized): Kupiec p-value (VaR 99% backtest on EUR/USD).
-Must co-move with val_vol_r2 — checked from experiment 1.
+## Agent instructions
 
-Diagnostics: silhouette score (regime clustering), PC1/HV correlation (collapse check).
+When acting as a coding agent on this project:
 
-## Benchmarks to beat (trading desk comparison)
-
-- GARCH(1,1) — volatility forecasting baseline
-- Historical Simulation VaR (250-day rolling) — Basel default
-- EWMA RiskMetrics (λ=0.94)
-
-## Agent guidance
-
-Read `DECISIONS.md` before making architecture suggestions.
-Do not modify `prepare.py` or `src/eval/`.
-Do not suggest changing the Python/Go separation.
-Training runs are always manual via `task experiment:run` — never triggered by CI.
-When implementing, follow Go conventions in `.skills/go-patterns/SKILL.md`.
+1. Read this file and all `SKILL.md` files in `.skills/` before starting work
+2. Run `brain_query wing=jepa-fx` to load current decisions and failure modes
+3. Run `task check` before every commit (lint + vet + test)
+4. Check `DECISIONS.md` before making any architecture or methodology choice
+5. Every experiment needs a spec in `specs/` — no specless experiments
+6. Never touch post-2023 data during development; it is sealed
+7. Record null results honestly — do not iterate until metrics pass without noting it
+8. When adding a Python dependency, justify it; prefer pure Go alternatives

.context/mcp.json

@@ -1,8 +1,18 @@
 {
   "mcpServers": {
-    "knowledge": {
-      "url": "http://localhost:3100/mcp",
-      "description": "Project knowledge base — vector + graph retrieval"
+    "brain": {
+      "type": "http",
+      "url": "https://brain-mcp.d-ma.be/mcp",
+      "headers": {
+        "Authorization": "Bearer ${BRAIN_MCP_TOKEN}"
+      }
+    },
+    "gitea": {
+      "type": "http",
+      "url": "https://git-mcp.d-ma.be/mcp",
+      "headers": {
+        "Authorization": "Bearer ${GITEA_MCP_TOKEN}"
+      }
     }
   }
 }

.gitignore

@@ -1,36 +1,39 @@
-# ---> Go
-# If you prefer the allow list template instead of the deny list, see community template:
-# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
-#
-# Binaries for programs and plugins
-*.exe
-*.exe~
-*.dll
-*.so
-*.dylib
-
-# Test binary, built with `go test -c`
+# Go build artifacts
+bin/
 *.test
-
-# Output of the go coverage tool, specifically when used with LiteIDE
 *.out
 
-# Dependency directories (remove the comment below to include it)
-# vendor/
-
-# Go workspace file
+# Go workspace
 go.work
 go.work.sum
 
-# env file
+# Environment
 .env
 
-# Project-specific
-bin/
-*.templ.go
+# Python
+model/.venv/
+model/__pycache__/
+model/**/__pycache__/
+model/**/*.pyc
+model/.pytest_cache/
+model/**/.pytest_cache/
+model/.ruff_cache/
 
-# python venv (autoresearch loop)
-.venv/
+# Data — never commit raw or processed FX data
+data/raw/
+data/processed/
+data/cache/
 
-# downloaded + processed market data (track via DVC/MinIO, #10 — not git)
-data/
+# Experiment outputs — commit summaries only (results/summaries/)
+experiments/*/embeddings/
+experiments/*/checkpoints/
+experiments/*/logs/
+
+# Large results — commit metric tables and figures only
+results/raw/
+
+# Notebooks — never commit outputs
+notebooks/**/.ipynb_checkpoints/
+
+# OS
+.DS_Store

.skills/README.md

@@ -0,0 +1,13 @@
+This directory contains skill symlinks for this project.
+
+Agents: load all SKILL.md files in subdirectories before starting work.
+
+Active skills:
+- experiment-spec  — write experiment specs before running any phase
+- feature-spec     — write component specs before implementing within a phase
+- tdd              — acceptance criteria map to tests; red-green-refactor
+- grill-me         — stress-test specs and hypotheses before committing
+- session-retrospective — after each phase, surface learnings for the brain
+- clean-code       — Go and Python style conventions
+- planning         — break phases into trackable tasks
+- debug            — systematic debugging approach

AGENTS.md

@@ -1,13 +1,5 @@
-# hostexecutor
+# jepa-fx-risk — Agent context
+# Auto-generated from .context/PROJECT.md by `task context:sync`
+# Do not edit directly.
 
-## Identity
-
-- **Name**: hostexecutor
-- **Owner**: Mathias
-- **Client**: personal
-- **Repo**: gitea.d-ma.be/mathias/hostexecutor
-- **Status**: active
-
-## Stack
-
-Go + Templ + HTMX + CDN Tailwind. See `~/dev/.context/AGENT.md` for cross-project conventions.
+See .context/PROJECT.md for the canonical source.

CLAUDE.md

@@ -1,13 +1,5 @@
-# hostexecutor
+# jepa-fx-risk — Agent context
+# Auto-generated from .context/PROJECT.md by `task context:sync`
+# Do not edit directly.
 
-## Identity
-
-- **Name**: hostexecutor
-- **Owner**: Mathias
-- **Client**: personal
-- **Repo**: gitea.d-ma.be/mathias/hostexecutor
-- **Status**: active
-
-## Stack
-
-Go + Templ + HTMX + CDN Tailwind. See `~/dev/.context/AGENT.md` for cross-project conventions.
+See .context/PROJECT.md for the canonical source.

DECISIONS.md

@@ -1,215 +1,120 @@
-# Architecture Decision Records
+# DECISIONS.md
 
-This file records significant technical and research decisions for `jepa-fx-risk`.
-Each record is immutable once merged — append new records rather than editing old ones.
-Format: ID · Date · Status · Context · Decision · Rationale · Consequences.
+Architecture and methodology decisions for `jepa-fx-risk`. Every non-obvious choice lives here with its rationale. Agents: read this before making any design decision.
+
+Last updated: 2026-05-27
 
 ---
 
-## ADR-001 · Architecture: TS-JEPA + SIGReg as Phase 1 backbone
+## Language split: Go-first, Python-minimal
 
-**Date:** 2026-05-28
-**Status:** Accepted
-**Supersedes:** informal decision to use TS-JEPA standalone (pre-ADR)
+**Decision:** Data pipeline, evaluation harness, CLI, and experiment runner in Go 1.24+. Python 3.12 + PyTorch only for the TS-JEPA model training loop, isolated in `model/`.
 
-### Context
+**Rationale:** Python's dependency ecosystem is a reliability risk for a multi-year project (conflicting CUDA versions, transitive breakage, environment drift). Go produces a single static binary, has excellent CSV/Parquet support, is fast enough for all non-training workloads, and keeps the reproducible parts of the project dependency-free. The training loop genuinely requires PyTorch — that is the one place Python is unavoidable.
 
-Four JEPA variants were evaluated for FX volatility forecasting and VaR/CVaR estimation:
-
-| Variant | Origin | Key property |
-|---|---|---|
-| TS-JEPA | Ennadir et al., Sep 2025 | Time-series native; EMA collapse prevention |
-| LeJEPA | Balestriero & LeCun, Nov 2025 | Proven optimal embeddings (isotropic Gaussian); SIGReg |
-| MTS-JEPA | He et al., Feb 2026 | Multi-resolution + codebook; no public code |
-| Var-JEPA | Multiple, Mar 2026 | ELBO-based UQ; no public code |
-
-Key constraints: 12 GB VRAM (Blackwell, koala), hourly DUKASCopy data, internal PoC target,
-autoresearch loop requires a clean single-scalar search space, trading desk requires an
-explainable theoretical story.
-
-### Decision
-
-Use **TS-JEPA architecture with SIGReg replacing EMA** as the Phase 1 backbone.
-
-Concretely:
-- Start from the TS-JEPA open-source implementation (arXiv:2509.25449, GitHub)
-- Remove the EMA target-network mechanism
-- Replace it with Sketched Isotropic Gaussian Regularization (SIGReg) from LeJEPA
-  (arXiv:2511.08544), controlled by a single λ hyperparameter
-- Keep TS-JEPA's temporal patchwise masking and Transformer encoder unchanged
-
-### Rationale
-
-**Why not pure TS-JEPA:** EMA is a heuristic; λ interacts with EMA momentum and
-learning rate, creating a three-way search space that is hard to navigate with autoresearch.
-EMA also has no theoretical non-stationarity guarantee.
-
-**Why not pure LeJEPA:** The reference implementation targets vision (multi-crop views).
-Adapting it to temporal patchwise masking requires non-trivial surgery and moves away from
-open code. TS-JEPA's masking is already the right inductive bias for time series.
-
-**Why the hybrid:** SIGReg is architecture-agnostic — it operates on the embedding
-distribution, not the encoder structure. Swapping EMA for SIGReg is a ~20-line change to
-TS-JEPA's training loop. The result is:
-- Time-series native (TS-JEPA masking + patch structure)
-- Provably collapse-free without heuristics (SIGReg)
-- Single search axis for autoresearch (λ ∈ [0.01, 1.0])
-- Non-stationarity robustness proven formally (arXiv:2602.19373 extends LeJEPA
-  guarantees to non-stationary target distributions — directly relevant to FX)
-- Explainable to a model validation team: "embeddings are provably optimal for
-  downstream prediction under distributional uncertainty"
-
-**Why not MTS-JEPA or Var-JEPA now:** Both lack public code (as of May 2026).
-MTS-JEPA's multi-resolution objective is the right next hypothesis (see ADR-003).
-Var-JEPA's ELBO-based UQ is a compelling future direction for CVaR estimation.
-
-### Consequences
-
-- Phase 0 (MAE baseline) is unaffected — it precedes the JEPA architecture choice
-- Issue #3 (TS-JEPA reproduction) is still the right first step; SIGReg is added after
-  reproduction is confirmed
-- The autoresearch `program.md` primary search axis is λ (SIGReg weight)
-- Secondary axes: masking block size, patch stride, encoder depth
-- `model/requirements.txt` must include the SIGReg implementation (≈20 lines,
-  can be vendored directly)
+**Boundary:** `model/` is the Python perimeter. Nothing outside it imports Python.
 
 ---
 
-## ADR-002 · Data: DUKASCopy hourly G10 FX as primary training data
+## Primary hypothesis
 
-**Date:** 2026-05-28
-**Status:** Accepted
+**Decision:** Regime detection via embedding structural separability is the primary hypothesis. Distributional VaR forecasting is secondary and only pursued if Phase 1 and 2 succeed.
 
-### Context
+**Rationale:** Regime detection is testable in Phase 1 without regulatory-quality outputs. If embeddings don't show regime structure, distributional VaR will also fail. Choosing a primary hypothesis prevents the project from retreating from one to the other on failure.
 
-Data scale is the most dangerous assumption for any SSL/JEPA approach. Daily FX data
-(~5,000 samples over 20 years) is insufficient for self-supervised pretraining.
-Two alternatives were considered: daily public data (yfinance) vs. hourly tick data
-(DUKASCopy, free, rate-limited HTTP API).
-
-### Decision
-
-Use **DUKASCopy hourly OHLCV** as the primary data source.
-
-- 10 G10 pairs: EURUSD, GBPUSD, USDJPY, USDCHF, AUDUSD, NZDUSD, USDCAD,
-  EURGBP, EURJPY, GBPJPY
-- Training window: 2008-01-01 – 2022-12-31 (~175,000 samples per pair)
-- Validation window: 2023-01-01 – 2023-12-31 (~2,600 samples)
-- Test window: 2024-01-01 – 2024-12-31 (held out, never seen during development)
-- Features per bar: log-return, log rolling-20-period HV, VIX (daily interpolated)
-- Weekend gaps handled explicitly — no interpolation across market close
-
-### Rationale
-
-Hourly data gives ~35× more samples than daily. This is the minimum threshold for
-JEPA-style SSL to show a training signal within 10-minute autoresearch experiments.
-DUKASCopy is free, reliable, and provides consistent tick-level source data back to 2003.
-
-### Consequences
-
-- The Go data pipeline (Issue #2) is the critical path for everything else
-- Phase 0 MAE baseline trains on the same 2008-2022 window
-- Daily data (yfinance) may still be used for VIX and rate differentials as auxiliary features
+**Success criterion (Phase 1):** Silhouette score > 0.35 on k-means clusters (k=3–5) vs. realised-volatility regime label (rolling 30-day HV percentile, high/low), computed on held-out test data including at least one structural break.
 
 ---
 
-## ADR-003 · Research roadmap: Phase structure and JEPA variant progression
+## Training data: 2008–2022, all G10 pairs
 
-**Date:** 2026-05-28
-**Status:** Accepted
+**Decision:** Train on 15 years of all G10 FX pairs from DUKASCopy, 2008–2022. Do not start with a single 5-year EUR/USD window.
 
-### Decision
+**Rationale:** A single 5-year window (e.g. 2019–2024) is dominated by one or two regimes and gives the encoder insufficient regime diversity to learn regime-sensitive representations. Training on 2008–2022 ensures the encoder sees: GFC (2008), EUR sovereign debt (2011–2012), SNB cap removal (2015), COVID (2020), USD rate cycle (2022). Multi-pair training also allows cross-currency transfer (Phase 4).
 
-Three-phase research roadmap:
-
-**Phase 0 — SSL feasibility gate (MAE baseline)**
-Implement a 1D temporal MAE (not JEPA) on EUR/USD hourly 2008-2022.
-Gate criteria: silhouette > 0.20 on 2023 held-out, MAE > PCA baseline, ±10% over 3 reruns.
-Purpose: validate that the data and eval harness work before committing to JEPA complexity.
-If gate fails: follow null result protocol in `specs/phase-0-ssl-feasibility.md`.
-
-**Phase 1 — TS-JEPA + SIGReg autoresearch sweep**
-Primary architecture per ADR-001.
-Autoresearch loop: `program.md`-driven, 10-min experiments, 50-experiment budget.
-Primary metric: `val_vol_r2` (linear probe R² on 1-day realized volatility).
-Gate criteria: `val_vol_r2` > GARCH-implied baseline AND Kupiec p-value > 0.05 on
-EUR/USD VaR 99%.
-Kupiec is logged from experiment 1 to verify it co-moves with `val_vol_r2`.
-
-**Phase 2 — MTS-JEPA multi-resolution hypothesis**
-Introduce parallel multi-scale predictive pathways (1h, 8h, 24h context windows)
-adapted from MTS-JEPA (arXiv:2602.04643).
-Hypothesis: multi-scale representations improve regime detection (silhouette) and
-reduce VaR exceedance clustering (Christoffersen test).
-Prerequisite: Phase 1 gate passed AND MTS-JEPA code available or reproducible from paper.
-Time-box: if MTS-JEPA code not available within 4 weeks of Phase 2 start, implement
-multi-resolution masking from scratch using Phase 1 backbone as base.
-
-**Phase 3 — Internal bank data (future)**
-Replace DUKASCopy pipeline with internal tick feed adapter.
-Fine-tune heads only; backbone frozen or lightly fine-tuned.
-Out of scope for current PoC cycle.
-
-### Consequences
-
-- Issue #5 (Phase 0 MAE) is the unblocked next executable step
-- Phase 1 autoresearch is blocked until Phase 0 passes its gate
-- Var-JEPA (ELBO-based UQ) is a named future hypothesis for CVaR estimation in Phase 2+
-  but not on the critical path
+**Alternative considered:** Start simple with EUR/USD only, expand later. Rejected because regime diversity in training is a structural requirement, not a nice-to-have. Retrofitting it in Phase 2 would require retraining from scratch.
 
 ---
 
-## ADR-004 · Evaluation: Go harness + Python training separation
+## Hard training cutoff: 2023-01-01
 
-**Date:** 2026-05-28
-**Status:** Accepted
+**Decision:** All data from 2023-01-01 onward is sealed. No architecture, hyperparameter, or methodology decision may be informed by post-2023 data. Post-2023 test set opened only once, for final evaluation.
 
-### Decision
+**Rationale:** Out-of-sample integrity is essential for publishability and honest self-assessment. The held-out window (2023–2026) includes: 2023 US regional bank stress, 2024 JPY intervention episodes. These are the test of genuine generalisation.
 
-Hard separation between training (Python) and evaluation (Go):
-
-- **Python** (`model/`): all training, embedding export, model checkpointing
-- **Go** (`src/eval/`): all evaluation metrics — silhouette, linear probe R², collapse
-  diagnostic, Kupiec/Christoffersen backtests
-- Interface: Python exports embedding matrices + labels to `experiments/RUNID/` as
-  `.npy` files; Go eval harness reads them and writes `metrics.json`
-
-### Rationale
-
-Go evaluation gives deterministic, fast, auditable metric computation with proper
-unit tests. It decouples the experimental loop from the training framework, making
-it possible to re-evaluate any past experiment without re-running training.
-The Go layer also serves as the foundation for the eventual trading desk dashboard.
-
-### Consequences
-
-- All acceptance criteria in Issues #4 and #5 are specified in terms of Go eval outputs
-- `val_vol_r2` (the autoresearch optimization metric) is computed by the Go harness,
-  not inside the Python training loop
-- Python training loop calls `task eval:probe` as a subprocess after each experiment
-  to get the scalar fed back to autoresearch
+**Enforcement:** `data/raw/` is gitignored. The download script hard-stops at 2022-12-31 for training splits. Any deviation requires a DECISIONS.md entry explaining why.
 
 ---
 
-## ADR-005 · Compute: Blackwell GPU on koala, PyTorch cu130
+## Phase 0: SSL feasibility gate before JEPA work
 
-**Date:** 2026-05-28
-**Status:** Accepted
+**Decision:** Before any JEPA-specific implementation, run a Phase 0 experiment: masked autoencoder (MAE) baseline on EUR/USD hourly data. If MAE silhouette < 0.20, SSL-based regime detection is likely not feasible on this data — stop and investigate before proceeding to JEPA.
 
-### Decision
+**Rationale:** JEPA's complexity is only justified if the core SSL premise (that latent representations capture regime structure) holds for FX data. A failed MAE experiment tells us this in 2 weeks rather than 4 months. Added after Full Grill session (2026-05-27).
 
-All GPU training runs on koala (Arch Linux, Blackwell GPU, 12 GB VRAM).
-PyTorch install: `pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu130`
-(CUDA 13.0 wheel — required for sm_120 Blackwell support; stable as of May 2026).
-Driver requirement: NVIDIA R570+, CUDA toolkit 12.8+.
+**Go/no-go threshold:** MAE silhouette < 0.20 on held-out 2023 data → pause, investigate, do not proceed to Phase 1.
 
-Ollama on iguana (Mac Studio M2 Ultra) serves the autoresearch agent LLM via the
-existing LiteLLM proxy on piguard. Agent calls never hit koala directly.
+---
 
-### Consequences
+## Architecture: TS-JEPA as starting implementation
 
-- `model/requirements.txt` must NOT pin torch to a cu124 or earlier wheel
-- CI (Issue #7) must NOT run GPU tests — CPU-only for unit tests, GPU only via
-  `task experiment:run` on koala
-- 12 GB VRAM is sufficient for <5M parameter models at batch=64; monitor if
-  autoresearch explores larger architectures
+**Decision:** Use TS-JEPA (Ennadir et al., 2025) as the starting JEPA implementation. MTS-JEPA (He et al., 2026) is the upgrade path if multi-resolution proves necessary.
+
+**Rationale:** TS-JEPA is simpler. Validate the concept before adding multi-resolution complexity. If Phase 1 succeeds with TS-JEPA, MTS-JEPA is an ablation, not a prerequisite.
+
+**Risk:** Both are preprints. Code reproducibility is unconfirmed. First task of Phase 0 is reproducing TS-JEPA on the paper's own benchmark — if this takes > 2 weeks, contact authors or fall back to implementing JEPA masking from scratch using V-JEPA codebase as reference.
+
+---
+
+## Input features: minimal for Phase 0/1
+
+**Decision:** Phase 0 and Phase 1 use three features only: log-return (hourly), rolling 20-period realised volatility (hourly), VIX (daily, interpolated to hourly).
+
+**Rationale:** Too many input features in early phases makes it impossible to distinguish "JEPA learned regime structure" from "JEPA learned to encode a feature that correlates with regime." Minimal features reduce confounding.
+
+**Expansion path:** Add DXY, G10 vol surface, yield spreads in Phase 2 if Phase 1 succeeds.
+
+---
+
+## Collapse diagnostic
+
+**Decision:** If PC1 of JEPA embeddings correlates > 0.85 with rolling 30-day HV, treat Phase 1 as a partial failure — the encoder learned volatility level, not regime structure. This is a useful finding but not the hypothesis.
+
+**Rationale:** EUR/USD hourly returns are strongly heteroskedastic. A JEPA encoder trained to predict future embeddings will strongly tend to encode current volatility as its primary latent dimension. This is predictable, not regime-sensitive. The linear probe and PC1 diagnostic together distinguish "learned volatility" from "learned regime."
+
+---
+
+## Linear probe as mandatory Phase 1 exit gate
+
+**Decision:** Train a linear model on frozen JEPA embeddings to predict realised volatility decile. R² < 0.4 → embeddings are not encoding useful risk structure → do not proceed to Phase 2.
+
+**Rationale:** If a simple linear model cannot extract volatility regime from the embeddings, the representations are not useful for risk management purposes regardless of their silhouette score. Interpretability-by-linear-probe is the minimum bar for any downstream use.
+
+---
+
+## Explicit out-of-scope (Phases 1–3)
+
+The following are explicitly out of scope for this research programme and go to a parking lot if they arise:
+
+- Options / derivatives pricing
+- Directional alpha generation
+- Live or paper trading
+- Exotic pairs beyond G10
+- Institutional deployment
+- Real-time inference systems
+
+---
+
+## Experiment spec required before any experiment
+
+**Decision:** Every experiment phase must have a written spec in `specs/` before any code runs. No specless experiments.
+
+**Rationale:** Consistent with spec-driven-dev way of working. Specs force falsifiable hypothesis statement, quantitative acceptance criteria, and explicit null-result protocol before results are known — preventing post-hoc rationalisation.
+
+---
+
+## Null results are results
+
+**Decision:** Null results (hypothesis rejected) are recorded in `results/summaries/` and treated as valid research outputs, not failures to be iterated away silently.
+
+**Rationale:** A confirmed null result (e.g. "SSL cannot find regime structure in FX hourly data") is publishable and scientifically valuable. Iterating hyperparameters until metrics pass without recording the failed attempts is p-hacking. Humble attitude; scientific approach.

README.md

@@ -1,13 +1,66 @@
-# hostexecutor
+# jepa-fx-risk
 
-> Generated from `mathias/template-go-web`.
+Research project exploring JEPA (Joint Embedding Predictive Architecture) as a framework for latent representation learning applied to FX trading risk management.
 
-## Bootstrap
+**Primary hypothesis:** JEPA embeddings trained on FX time-series will produce latent market-state representations that are structurally separable by regime without explicit regime labels, measurable by silhouette score on k-means clusters validated against held-out realised-volatility regime labels.
 
-After creating from template, run:
+**Current phase:** Phase 0 — SSL Feasibility Gate (not yet started)
+
+**Brain wing:** `jepa-fx` — query via `brain_query wing=jepa-fx`
+
+---
+
+## Architecture
+
+This is a **Go-first, Python-minimal** research project.
+
+| Layer | Language | Rationale |
+|---|---|---|
+| Data pipeline | Go | Type-safe, fast, no dependency hell |
+| Experiment runner | Go | CLI tooling, reproducible invocations |
+| Evaluation harness | Go | Silhouette, linear probe, collapse diagnostics |
+| Model training | Python + PyTorch | TS-JEPA requires it; kept minimal and isolated |
+
+## Structure
+
+```
+specs/          experiment specs (one per phase)
+src/            Go packages: pipeline, eval, cmd
+model/          Python: TS-JEPA training loop only
+experiments/    one directory per run (gitignored except summaries)
+results/        tracked: metric tables, key figures
+data/           gitignored; see data/README.md for download instructions
+notebooks/      EDA only; outputs stripped before commit
+docs/           ADRs and research notes
+```
+
+## Quick start
 
 ```bash
-go mod tidy       # regenerate go.sum with real module path
-task generate     # generate templ files
-task build        # build the binary
+task data:fetch          # download DUKASCopy G10 tick data
+task pipeline:build      # build Go data pipeline
+task check               # lint + vet + test
+task experiment:run -- --spec specs/phase-0-ssl-feasibility.md
 ```
+
+## Phases
+
+| Phase | Goal | Status |
+|---|---|---|
+| 0 | SSL feasibility gate — MAE baseline on FX data | 🔲 Not started |
+| 1 | JEPA representation PoC — silhouette > 0.35 | 🔲 Not started |
+| 2 | Regime detection validation — recall > 70%, lead > 5d | 🔲 Not started |
+| 3 | Distributional forecasting — VaR passes Kupiec | 🔲 Not started |
+| 4 | Multi-pair transfer — exotic pair improvement | 🔲 Not started |
+
+## Key constraints
+
+- **Training cutoff:** 2023-01-01 — all post-2023 data is sealed for final evaluation
+- **Training data:** 2008–2022, all G10 pairs, DUKASCopy
+- **Out of scope:** options pricing, directional alpha, live trading, exotic pairs (Phases 1–3)
+- **End-state:** publishable research result — not institutional deployment
+
+## Related
+
+- Brain wing: `wiki/jepa-fx/` — decisions, hypotheses, failure modes
+- Synthesis document: `JEPA_FX_Risk_Synthesis.docx` (session 2026-05-27)

STATUS.md

@@ -1,22 +0,0 @@
-# Autoresearch STATUS
-
-| iter | val_vol_r2 | delta | action | secs | gpu | change |
-|------|-----------|-------|--------|------|-----|--------|
-| 1 | 0.3749 | +0.0928 | KEEP | 2s | gpu=0% vram=10054/12227MiB temp=34°C | iter1 |
-| 1 | 0.3011 | +0.0776 | KEEP | 2s | gpu=0% vram=10054/12227MiB temp=34°C | iter1 |
-| 2 | 0.3032 | +0.0021 | KEEP | 2s | gpu=0% vram=10054/12227MiB temp=35°C | iter2 |
-| 1 | 0.2759 | -0.0273 | revert | 2s | gpu=0% vram=10054/12227MiB temp=34°C | iter1 |
-| 2 | 0.3442 | +0.0410 | KEEP | 2s | gpu=0% vram=10054/12227MiB temp=34°C | iter2 |
-| 3 | 0.3371 | -0.0071 | revert | 2s | gpu=0% vram=10054/12227MiB temp=34°C | iter3 |
-| 4 | 0.3143 | -0.0299 | revert | 2s | gpu=0% vram=10054/12227MiB temp=34°C | iter4 |
-| 5 | 0.3355 | -0.0087 | revert | 2s | gpu=0% vram=10054/12227MiB temp=34°C | iter5 |
-| 6 | 0.2377 | -0.1065 | revert | 2s | gpu=0% vram=10054/12227MiB temp=35°C | iter6 |
-| 1 | -0.1247 | +0.0296 | KEEP | 4s | gpu=0% vram=10054/12227MiB temp=35°C | iter1 |
-| 2 | -0.1203 | +0.0044 | KEEP | 4s | gpu=0% vram=10054/12227MiB temp=35°C | iter2 |
-| 3 | -0.0716 | +0.0487 | KEEP | 4s | gpu=0% vram=10054/12227MiB temp=36°C | iter3 |
-| 4 | 0.0590 | +0.1306 | KEEP | 5s | gpu=0% vram=10054/12227MiB temp=36°C | iter4 |
-| 5 | 0.0599 | +0.0009 | KEEP | 5s | gpu=0% vram=10054/12227MiB temp=37°C | iter5 |
-| 1 | 0.0563 | -0.0036 | revert | 5s | gpu=0% vram=10054/12227MiB temp=35°C | iter1 |
-| 2 | 0.0577 | -0.0022 | revert | 5s | gpu=0% vram=10054/12227MiB temp=36°C | iter2 |
-| 3 | 0.0563 | -0.0036 | revert | 5s | gpu=0% vram=10054/12227MiB temp=36°C | iter3 |
-| 4 | -0.1613 | -0.2212 | revert | 5s | gpu=0% vram=10054/12227MiB temp=37°C | iter4 |

Taskfile.yml

@@ -1,84 +1,79 @@
 version: '3'
 
 tasks:
-  generate:
-    desc: Run templ generate
-    cmds: [templ generate]
-  build:
-    desc: Build all binaries
-    deps: [generate]
+
+  # ── Quality gate ──────────────────────────────────────────────────────────
+
+  check:
+    desc: "Full quality gate: lint + vet + test (run before every commit)"
     cmds:
-      - go build -o bin/jepa-fx-risk ./cmd/jepa-fx-risk
-      - go build -o bin/eval ./cmd/eval
-  run:
-    deps: [build]
-    cmds: [./bin/jepa-fx-risk]
+      - golangci-lint run ./src/...
+      - go vet ./src/...
+      - go test ./src/... -race -count=1
+      - cd model && python -m pytest tests/ -q
+
   test:
-    desc: Run all tests
-    deps: [generate]
-    cmds: [go test ./... -race]
+    desc: Run Go tests only
+    cmds: [go test ./src/... -race]
+
+  lint:
+    desc: Lint Go code
+    cmds: [golangci-lint run ./src/...]
+
+  # ── Data ─────────────────────────────────────────────────────────────────
 
   data:fetch:
-    desc: "Download EUR/USD M1 from histdata (set YEARS env var)"
-    cmds: [.venv/bin/python scripts/fetch_data.py]
-  data:fetch:historical:
-    desc: "Download EUR/USD M1 2008-2018 from histdata"
-    cmds:
-      - YEARS=2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018 .venv/bin/python scripts/fetch_data.py
-  data:prepare:daily:
-    desc: "Rebuild eurusd_daily.parquet from all M1 zips"
-    cmds: [.venv/bin/python scripts/prepare_data.py]
-  data:prepare:hourly:
-    desc: "Build eurusd_hourly.parquet from all M1 zips"
-    cmds: [.venv/bin/python scripts/prepare_hourly.py]
-  data:prepare:all:
-    desc: "Build both daily and hourly parquets"
-    deps: [data:prepare:daily, data:prepare:hourly]
-  train:multipair:
-    desc: "Train 5-pair G10 HEPA (D=256, best config, phase1_r2≈0.44)"
-    cmds: [JEPA_USE_MULTIPAIR=1 JEPA_D_MODEL=256 .venv/bin/python train.py]
+    desc: "Download G10 tick data from DUKASCopy (2003–2022)"
+    cmds: [go run ./src/cmd/fetch --config data/config.yaml]
 
-  data:fetch:multipair:
-    desc: "Download G10 M1 data (GBPUSD/USDJPY/USDCHF/AUDUSD) 2008-2023 from histdata"
-    cmds: [.venv/bin/python scripts/fetch_multipair.py]
-  data:prepare:pair:
-    desc: "Build {PAIR}_hourly.parquet from data/raw/{PAIR}/ (e.g. PAIR=gbpusd)"
-    cmds: [PAIR={{.PAIR}} .venv/bin/python scripts/prepare_hourly.py {{.EXTRA_ARGS}}]
-    vars:
-      PAIR: '{{default "eurusd" .PAIR}}'
-  data:prepare:multipair:
-    desc: "Merge 5-pair hourly parquets into eurusd_multipair.parquet"
-    cmds: [.venv/bin/python scripts/prepare_multipair.py]
-  data:test:
-    desc: "Run Python data pipeline tests"
-    cmds: [.venv/bin/python -m pytest tests/test_prepare_hourly.py tests/test_hepa.py tests/test_multipair.py -v]
+  data:process:
+    desc: "Resample ticks → hourly OHLCV + features (log-return, rolling HV)"
+    cmds: [go run ./src/cmd/process --input data/raw --output data/processed]
+
+  data:validate:
+    desc: "Validate processed data: gap detection, outlier report, regime coverage"
+    cmds: [go run ./src/cmd/validate --input data/processed]
+
+  # ── Experiment ───────────────────────────────────────────────────────────
+
+  experiment:run:
+    desc: "Run an experiment from a spec file. Usage: task experiment:run -- --spec specs/phase-0-ssl-feasibility.md"
+    cmds: [go run ./src/cmd/experiment {{.CLI_ARGS}}]
+
+  experiment:list:
+    desc: List all recorded experiment runs
+    cmds: [ls -lt experiments/ | head -20]
+
+  # ── Evaluation ───────────────────────────────────────────────────────────
+
+  eval:silhouette:
+    desc: "Compute silhouette score on embedding output. Usage: task eval:silhouette -- --run experiments/RUNID"
+    cmds: [go run ./src/cmd/eval silhouette {{.CLI_ARGS}}]
 
   eval:probe:
-    desc: "Run linear-probe (val_vol_r2) on embeddings from metrics.json"
-    cmds: [./bin/eval -metric probe]
-  eval:silhouette:
-    desc: "Run silhouette on embeddings vs binary HV labels"
-    cmds: [./bin/eval -metric silhouette]
+    desc: "Run linear probe on frozen embeddings vs. realised-vol decile"
+    cmds: [go run ./src/cmd/eval probe {{.CLI_ARGS}}]
+
   eval:collapse:
-    desc: "Run effective-rank collapse diagnostic"
-    cmds: [./bin/eval -metric erank]
-  lint:
-    cmds: [golangci-lint run ./...]
-  check:
-    desc: Lint, vet, and test (used by CI)
-    deps: [generate]
+    desc: "Check collapse diagnostic: PC1 correlation with rolling HV"
+    cmds: [go run ./src/cmd/eval collapse {{.CLI_ARGS}}]
+
+  # ── Model (Python) ───────────────────────────────────────────────────────
+
+  model:train:
+    desc: "Train TS-JEPA model. Usage: task model:train -- --config model/configs/phase0.yaml"
+    dir: model
+    cmds: [python train.py {{.CLI_ARGS}}]
+
+  model:setup:
+    desc: "Create Python venv and install model dependencies (uv)"
+    dir: model
     cmds:
-      - golangci-lint run ./...
-      - go vet ./...
-      - go test ./... -race -count=1
+      - uv venv .venv
+      - uv pip install -r requirements.txt
+
+  # ── Context ──────────────────────────────────────────────────────────────
 
   context:sync:
-    desc: Regenerate all harness-specific context files
-    cmds:
-      - bash scripts/context-sync.sh
-  context:sync:claude:
-    cmds: [bash scripts/context-sync.sh claude]
-  context:sync:agents:
-    cmds: [bash scripts/context-sync.sh agents]
-  context:sync:cursor:
-    cmds: [bash scripts/context-sync.sh cursor]
+    desc: "Regenerate CLAUDE.md and AGENTS.md from .context/PROJECT.md"
+    cmds: [bash scripts/context-sync.sh]

cmd/eval/main.go

@@ -1,194 +0,0 @@
-// cmd/eval — CLI driver for the jepa-fx-risk evaluation harness.
-//
-//	./bin/eval -metric probe|silhouette|erank [-emb embeddings.json]
-//
-// embeddings.json format (from train.py EXPORT_EMBEDDINGS=1):
-//
-//	{
-//	  "embeddings":         [[...], ...],  // OOS frozen embeddings
-//	  "realized_vol":       [...],         // OOS target (next-day RV)
-//	  "hv_label":           [...],         // binary HV label (top-33%)
-//	  "train_embeddings":   [[...], ...],  // train-set frozen embeddings
-//	  "train_realized_vol": [...]          // train-set RV targets
-//	}
-//
-// eval:probe standardises both sets using train statistics (no leakage).
-// Falls back to internal 70/30 split of OOS if train_embeddings absent.
-package main
-
-import (
-	"encoding/json"
-	"flag"
-	"fmt"
-	"log/slog"
-	"math"
-	"os"
-
-	"gitea.d-ma.be/mathias/jepa-fx-risk/internal/eval"
-)
-
-func main() {
-	metric := flag.String("metric", "probe", "probe | silhouette | erank")
-	embFile := flag.String("emb", "embeddings.json", "path to embeddings JSON")
-	flag.Parse()
-
-	log := slog.New(slog.NewJSONHandler(os.Stdout, nil))
-
-	d, err := readJSON(*embFile)
-	if err != nil {
-		log.Error("load embeddings", "err", err)
-		os.Exit(1)
-	}
-	log.Info("loaded", "oos", len(d.Embeddings), "dim", len(d.Embeddings[0]),
-		"train", len(d.TrainEmbeddings), "metric", *metric)
-
-	switch *metric {
-	case "probe":
-		var r2 float64
-		if len(d.TrainEmbeddings) > 0 {
-			// standardise both sets using train statistics to prevent leakage
-			trEmb, mu, sd := standardiseCompute(d.TrainEmbeddings)
-			oosEmb := applyStandardise(d.Embeddings, mu, sd)
-			r2 = eval.LinearProbeTrainTest(trEmb, d.TrainRealizedVol, oosEmb, d.RealizedVol, 1e-3)
-			log.Info("probe mode", "fit_on", "train_embeddings", "eval_on", "oos")
-		} else {
-			// fallback: internal 70/30 split of OOS embeddings
-			oosEmb, mu, sd := standardiseCompute(d.Embeddings)
-			n70 := int(float64(len(oosEmb)) * 0.7)
-			oos70 := applyStandardise(d.Embeddings[n70:], mu, sd)
-			r2 = eval.LinearProbeTrainTest(oosEmb[:n70], d.RealizedVol[:n70],
-				oos70, d.RealizedVol[n70:], 1e-3)
-			log.Info("probe mode", "fit_on", "oos[0:70%]", "eval_on", "oos[70%:]")
-		}
-		fmt.Printf(`{"metric":"val_vol_r2","value":%.6f}`+"\n", r2)
-		log.Info("linear probe", "val_vol_r2", fmt.Sprintf("%.4f", r2))
-
-	case "silhouette":
-		if len(d.HVLabel) == 0 {
-			log.Error("silhouette requires hv_label in embeddings.json")
-			os.Exit(1)
-		}
-		oosEmb := standardise(d.Embeddings)
-		sil, err := eval.Silhouette(oosEmb, d.HVLabel)
-		if err != nil {
-			log.Error("silhouette", "err", err)
-			os.Exit(1)
-		}
-		fmt.Printf(`{"metric":"silhouette","value":%.6f}`+"\n", sil)
-		log.Info("silhouette", "score", fmt.Sprintf("%.4f", sil))
-
-	case "erank":
-		oosEmb := standardise(d.Embeddings)
-		er := eval.EffectiveRank(oosEmb)
-		fmt.Printf(`{"metric":"effective_rank","value":%.6f}`+"\n", er)
-		log.Info("effective rank", "erank", fmt.Sprintf("%.2f", er))
-
-	case "var":
-		// Parametric 99% VaR breach rate from probe predictions vs actual realized vol.
-		// Requires train_embeddings (for no-leakage probe fit) and realized_vol (OOS).
-		if len(d.RealizedVol) == 0 {
-			log.Error("var requires realized_vol in embeddings.json")
-			os.Exit(1)
-		}
-		var predVol []float64
-		if len(d.TrainEmbeddings) > 0 {
-			trEmb, mu, sd := standardiseCompute(d.TrainEmbeddings)
-			oosEmb := applyStandardise(d.Embeddings, mu, sd)
-			predVol = eval.LinearProbePredict(trEmb, d.TrainRealizedVol, oosEmb, 1e-3)
-		} else {
-			oosEmb, mu, sd := standardiseCompute(d.Embeddings)
-			n70 := int(float64(len(oosEmb)) * 0.7)
-			oos70 := applyStandardise(d.Embeddings[n70:], mu, sd)
-			predVol = eval.LinearProbePredict(oosEmb[:n70], d.RealizedVol[:n70], oos70, 1e-3)
-			d.RealizedVol = d.RealizedVol[n70:]
-		}
-		const z99 = 2.326
-		breachRate, kupiecP := eval.VaRBreachRate(predVol, d.RealizedVol, z99)
-		fmt.Printf(`{"metric":"VaR_breach_rate_99_oos_regime_cond","value":%.6f,"kupiec_p":%.6f}`+"\n",
-			breachRate, kupiecP)
-		log.Info("VaR breach rate 99%", "breach_rate", fmt.Sprintf("%.4f", breachRate),
-			"kupiec_p", fmt.Sprintf("%.4f", kupiecP))
-
-	default:
-		log.Error("unknown metric", "metric", *metric)
-		os.Exit(1)
-	}
-}
-
-type embJSON struct {
-	Embeddings       [][]float64 `json:"embeddings"`
-	Dates            []string    `json:"dates"`
-	RealizedVol      []float64   `json:"realized_vol"`
-	HVLabel          []int       `json:"hv_label"`
-	TrainEmbeddings  [][]float64 `json:"train_embeddings"`
-	TrainRealizedVol []float64   `json:"train_realized_vol"`
-}
-
-func readJSON(path string) (*embJSON, error) {
-	f, err := os.Open(path)
-	if err != nil {
-		return nil, fmt.Errorf("open %s: %w", path, err)
-	}
-	defer func() { _ = f.Close() }()
-	var d embJSON
-	if err := json.NewDecoder(f).Decode(&d); err != nil {
-		return nil, fmt.Errorf("decode: %w", err)
-	}
-	if len(d.Embeddings) == 0 {
-		return nil, fmt.Errorf("empty embeddings in %s", path)
-	}
-	return &d, nil
-}
-
-// standardise centres + scales to zero mean / unit std; returns normalised rows.
-func standardise(rows [][]float64) [][]float64 {
-	out, _, _ := standardiseCompute(rows)
-	return out
-}
-
-// standardiseCompute centres + scales and returns (normalised, mu, sd) for reuse.
-func standardiseCompute(rows [][]float64) ([][]float64, []float64, []float64) {
-	if len(rows) == 0 {
-		return rows, nil, nil
-	}
-	n, dim := len(rows), len(rows[0])
-	mu := make([]float64, dim)
-	for _, r := range rows {
-		for j, v := range r {
-			mu[j] += v
-		}
-	}
-	for j := range mu {
-		mu[j] /= float64(n)
-	}
-	sd := make([]float64, dim)
-	for _, r := range rows {
-		for j, v := range r {
-			diff := v - mu[j]
-			sd[j] += diff * diff
-		}
-	}
-	for j := range sd {
-		sd[j] = math.Sqrt(sd[j]/float64(n)) + 1e-8
-	}
-	out := make([][]float64, n)
-	for i, r := range rows {
-		out[i] = make([]float64, dim)
-		for j, v := range r {
-			out[i][j] = (v - mu[j]) / sd[j]
-		}
-	}
-	return out, mu, sd
-}
-
-// applyStandardise normalises rows using pre-computed mu and sd.
-func applyStandardise(rows [][]float64, mu, sd []float64) [][]float64 {
-	out := make([][]float64, len(rows))
-	for i, r := range rows {
-		out[i] = make([]float64, len(r))
-		for j, v := range r {
-			out[i][j] = (v - mu[j]) / sd[j]
-		}
-	}
-	return out
-}

cmd/hostexecutor/main.go

@@ -5,7 +5,7 @@ import (
 	"net/http"
 	"os"
 
-	"gitea.d-ma.be/mathias/jepa-fx-risk/internal/web"
+	"gitea.d-ma.be/mathias/hostexecutor/internal/web"
 )
 
 func main() {

data/README.md

@@ -0,0 +1,36 @@
+# Data
+
+Raw and processed FX data is **gitignored** — never committed.
+
+## Download: DUKASCopy G10 tick data
+
+Use the provided fetch task:
+
+```bash
+task data:fetch    # downloads raw tick data → data/raw/
+task data:process  # resamples to hourly OHLCV + features → data/processed/
+task data:validate # gap detection, outlier report, regime coverage check
+```
+
+## Expected structure (local only)
+
+```
+data/
+  raw/          G10 pairs, tick OHLCV, 2003–present (gitignored)
+  processed/    Hourly log-returns, rolling HV, VIX-merged (gitignored)
+  cache/        Intermediate artefacts (gitignored)
+  config.yaml   Download configuration (committed)
+```
+
+## Pairs
+
+EUR/USD, GBP/USD, USD/JPY, USD/CHF, AUD/USD, USD/CAD, NZD/USD, EUR/GBP, EUR/JPY, EUR/CHF
+
+## Splits
+
+| Split | Date range | Purpose |
+|---|---|---|
+| Train | 2008-01-01 – 2022-12-31 | Model training only |
+| Held-out test | 2023-01-01 – 2023-12-31 | Final evaluation — do not open until evaluation |
+
+**Training cutoff is hard: 2023-01-01. No post-2022 data informs any design decision.**

experiments/README.md

@@ -0,0 +1,32 @@
+# Experiments
+
+One directory per experiment run. Large outputs (embeddings, checkpoints, logs) are gitignored. Only metric summaries are committed to `results/summaries/`.
+
+## Naming convention
+
+```
+YYYYMMDD-HHMMSS-phase-N-short-description/
+  config.yaml       parameters used for this run (committed via results/summaries/)
+  metrics.json      final metric snapshot (committed via results/summaries/)
+  embeddings/       (gitignored — large)
+  checkpoints/      (gitignored — large)
+  logs/             (gitignored — large)
+```
+
+## Git tag convention
+
+Every run that produces reportable metrics gets a tag:
+
+```
+exp/YYYYMMDD-phase-N-description
+```
+
+Example: `exp/20260601-phase-0-mae-baseline`
+
+## Starting a run
+
+```bash
+task experiment:run -- --spec specs/phase-0-ssl-feasibility.md --tag my-run-description
+```
+
+The runner creates the directory, writes config, runs the experiment, and outputs metrics.json.

go.mod

@@ -1,5 +1,7 @@
-module gitea.d-ma.be/mathias/jepa-fx-risk
+module gitea.d-ma.be/mathias/hostexecutor
 
 go 1.26
 
-require github.com/a-h/templ v0.3.1020
+require (
+	github.com/a-h/templ v0.2.778
+)

go.sum

@@ -1,4 +0,0 @@
-github.com/a-h/templ v0.3.1020 h1:ypAT/L5ySWEnZ6Zft/5yfoWXYYkhFNvEFOeeqecg4tw=
-github.com/a-h/templ v0.3.1020/go.mod h1:A2DlK61v+K+NRoGnhmYbNYVmtYHcFO5/AisMvBdDxTM=
-github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
-github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=

internal/eval/eval.go

@@ -1,383 +0,0 @@
-// Package eval implements the Go evaluation harness for jepa-fx-risk (#4).
-// Three diagnostics on frozen embeddings exported from train.py:
-//   - LinearProbe  — val_vol_r2: OOS R² of a ridge probe predicting next-day realized vol
-//   - Silhouette   — mean silhouette score of embeddings vs a binary label (HV regime)
-//   - EffectiveRank — Roy's effective rank: exp(H(σ²)) where H is entropy of normalised singular values
-package eval
-
-import (
-	"errors"
-	"math"
-)
-
-// LinearProbeTrainTest fits ridge regression on (trainEmb, trainY) and evaluates
-// on (testEmb, testY). Returns OOS R². Use this for proper held-out evaluation.
-func LinearProbeTrainTest(trainEmb [][]float64, trainY []float64,
-	testEmb [][]float64, testY []float64, lambda float64) float64 {
-	n := len(trainEmb)
-	if n == 0 || len(testEmb) == 0 {
-		return 0
-	}
-	d := len(trainEmb[0])
-	p := d + 1
-
-	A := make([][]float64, n)
-	for i, e := range trainEmb {
-		row := make([]float64, p)
-		copy(row, e)
-		row[d] = 1.0
-		A[i] = row
-	}
-	AtA := make([][]float64, p)
-	for i := range AtA {
-		AtA[i] = make([]float64, p)
-	}
-	Aty := make([]float64, p)
-	for i := 0; i < n; i++ {
-		for j := 0; j < p; j++ {
-			Aty[j] += A[i][j] * trainY[i]
-			for k := 0; k < p; k++ {
-				AtA[j][k] += A[i][j] * A[i][k]
-			}
-		}
-	}
-	for j := 0; j < p; j++ {
-		AtA[j][j] += lambda
-	}
-	w := solveCholesky(AtA, Aty)
-
-	yMean := mean(testY)
-	var ssRes, ssTot float64
-	for i, e := range testEmb {
-		row := make([]float64, p)
-		copy(row, e)
-		row[d] = 1.0
-		pred := dot(row, w)
-		ssRes += (testY[i] - pred) * (testY[i] - pred)
-		ssTot += (testY[i] - yMean) * (testY[i] - yMean)
-	}
-	if ssTot == 0 {
-		return 0
-	}
-	return 1 - ssRes/ssTot
-}
-
-// LinearProbe fits a ridge regression (closed-form) on (emb, y) with regularisation λ
-// and returns R² on the same data. Call with train embeddings; probe on held-out by
-// splitting before calling.
-//
-// emb[i] is the embedding vector for sample i; y[i] is the scalar target.
-func LinearProbe(emb [][]float64, y []float64, lambda float64) float64 {
-	n := len(emb)
-	if n == 0 {
-		return 0
-	}
-	d := len(emb[0])
-
-	// Build augmented design matrix A = [emb | 1] (n × d+1)
-	A := make([][]float64, n)
-	for i, e := range emb {
-		row := make([]float64, d+1)
-		copy(row, e)
-		row[d] = 1.0
-		A[i] = row
-	}
-
-	// Normal equations: (AᵀA + λI) w = Aᵀy  (ridge)
-	p := d + 1
-	AtA := make([][]float64, p)
-	for i := range AtA {
-		AtA[i] = make([]float64, p)
-	}
-	Aty := make([]float64, p)
-
-	for i := 0; i < n; i++ {
-		for j := 0; j < p; j++ {
-			Aty[j] += A[i][j] * y[i]
-			for k := 0; k < p; k++ {
-				AtA[j][k] += A[i][j] * A[i][k]
-			}
-		}
-	}
-	for j := 0; j < p; j++ {
-		AtA[j][j] += lambda
-	}
-
-	w := solveCholesky(AtA, Aty)
-
-	// R² = 1 - SS_res / SS_tot
-	yMean := mean(y)
-	var ssRes, ssTot float64
-	for i := 0; i < n; i++ {
-		pred := dot(A[i], w)
-		ssRes += (y[i] - pred) * (y[i] - pred)
-		ssTot += (y[i] - yMean) * (y[i] - yMean)
-	}
-	if ssTot == 0 {
-		return 0
-	}
-	return 1 - ssRes/ssTot
-}
-
-// Silhouette returns the mean silhouette coefficient of the embeddings with respect
-// to the given integer labels. Distances are Euclidean. Returns an error if fewer
-// than 2 distinct labels are present.
-func Silhouette(emb [][]float64, labels []int) (float64, error) {
-	n := len(emb)
-	if n == 0 {
-		return 0, errors.New("eval: empty embeddings")
-	}
-
-	// count distinct labels
-	labelSet := map[int]struct{}{}
-	for _, l := range labels {
-		labelSet[l] = struct{}{}
-	}
-	if len(labelSet) < 2 {
-		return 0, errors.New("eval: silhouette requires at least 2 distinct labels")
-	}
-
-	// group indices by label
-	groups := map[int][]int{}
-	for i, l := range labels {
-		groups[l] = append(groups[l], i)
-	}
-
-	var total float64
-	for i := 0; i < n; i++ {
-		li := labels[i]
-
-		// a(i) = mean intra-cluster distance
-		var aSum float64
-		inGroup := groups[li]
-		for _, j := range inGroup {
-			if j != i {
-				aSum += euclidean(emb[i], emb[j])
-			}
-		}
-		var a float64
-		if len(inGroup) > 1 {
-			a = aSum / float64(len(inGroup)-1)
-		}
-
-		// b(i) = min mean inter-cluster distance
-		b := math.MaxFloat64
-		for l, idxs := range groups {
-			if l == li {
-				continue
-			}
-			var dSum float64
-			for _, j := range idxs {
-				dSum += euclidean(emb[i], emb[j])
-			}
-			avg := dSum / float64(len(idxs))
-			if avg < b {
-				b = avg
-			}
-		}
-
-		s := (b - a) / math.Max(a, b)
-		total += s
-	}
-	return total / float64(n), nil
-}
-
-// EffectiveRank computes Roy's effective rank of the embedding matrix:
-// exp(H) where H = -∑ pᵢ log(pᵢ) is the Shannon entropy of the normalised
-// squared singular values. Returns 1 for a rank-1 matrix and ≈ dim for
-// a full-rank isotropic matrix.
-func EffectiveRank(emb [][]float64) float64 {
-	n := len(emb)
-	if n == 0 {
-		return 0
-	}
-	d := len(emb[0])
-
-	// Compute covariance-like matrix CᵀC where C is mean-centered embedding.
-	mu := make([]float64, d)
-	for _, e := range emb {
-		for j, v := range e {
-			mu[j] += v
-		}
-	}
-	for j := range mu {
-		mu[j] /= float64(n)
-	}
-
-	// C = emb - mu  (n × d); compute CᵀC (d × d)
-	CtC := make([][]float64, d)
-	for i := range CtC {
-		CtC[i] = make([]float64, d)
-	}
-	for _, e := range emb {
-		for j := 0; j < d; j++ {
-			cj := e[j] - mu[j]
-			for k := 0; k < d; k++ {
-				CtC[j][k] += cj * (e[k] - mu[k])
-			}
-		}
-	}
-
-	// Eigenvalues of CᵀC via power iteration approximation isn't great;
-	// use the Frobenius / trace approach: σᵢ² ∝ eigenvalues of CᵀC.
-	// For a pure-Go impl without LAPACK: use the fact that the normalised
-	// squared singular values equal normalised eigenvalues of CᵀC.
-	// Compute them via Jacobi iteration for small d, or use the analytical
-	// formula for 2×2, or use iterative QR for general d.
-	eigs := jacobiEigenvalues(CtC)
-
-	// normalise to sum-1 distribution
-	var sumEig float64
-	for _, v := range eigs {
-		if v > 0 {
-			sumEig += v
-		}
-	}
-	if sumEig == 0 {
-		return 1
-	}
-	var H float64
-	for _, v := range eigs {
-		if v > 0 {
-			p := v / sumEig
-			H -= p * math.Log(p)
-		}
-	}
-	return math.Exp(H)
-}
-
-// ── internal helpers ──────────────────────────────────────────────────────────
-
-func euclidean(a, b []float64) float64 {
-	var s float64
-	for i := range a {
-		d := a[i] - b[i]
-		s += d * d
-	}
-	return math.Sqrt(s)
-}
-
-func dot(a, b []float64) float64 {
-	var s float64
-	for i := range a {
-		s += a[i] * b[i]
-	}
-	return s
-}
-
-func mean(y []float64) float64 {
-	var s float64
-	for _, v := range y {
-		s += v
-	}
-	return s / float64(len(y))
-}
-
-// solveCholesky solves Ax = b for symmetric positive-definite A via
-// Cholesky decomposition. Falls back to pseudo-inverse on failure.
-func solveCholesky(A [][]float64, b []float64) []float64 {
-	n := len(A)
-	// Cholesky decomposition: A = LLᵀ
-	L := make([][]float64, n)
-	for i := range L {
-		L[i] = make([]float64, n)
-	}
-	for i := 0; i < n; i++ {
-		for j := 0; j <= i; j++ {
-			s := A[i][j]
-			for k := 0; k < j; k++ {
-				s -= L[i][k] * L[j][k]
-			}
-			if i == j {
-				if s <= 0 {
-					s = 1e-12
-				}
-				L[i][j] = math.Sqrt(s)
-			} else {
-				L[i][j] = s / L[j][j]
-			}
-		}
-	}
-	// Forward substitution Ly = b
-	y := make([]float64, n)
-	for i := 0; i < n; i++ {
-		s := b[i]
-		for k := 0; k < i; k++ {
-			s -= L[i][k] * y[k]
-		}
-		y[i] = s / L[i][i]
-	}
-	// Back substitution Lᵀx = y
-	x := make([]float64, n)
-	for i := n - 1; i >= 0; i-- {
-		s := y[i]
-		for k := i + 1; k < n; k++ {
-			s -= L[k][i] * x[k]
-		}
-		x[i] = s / L[i][i]
-	}
-	return x
-}
-
-// jacobiEigenvalues returns eigenvalues of a symmetric matrix via Jacobi iteration.
-func jacobiEigenvalues(A [][]float64) []float64 {
-	n := len(A)
-	// copy
-	a := make([][]float64, n)
-	for i := range a {
-		a[i] = make([]float64, n)
-		copy(a[i], A[i])
-	}
-
-	const maxIter = 100
-	const tol = 1e-10
-	for iter := 0; iter < maxIter; iter++ {
-		// find largest off-diagonal element
-		p, q, amax := 0, 1, 0.0
-		for i := 0; i < n; i++ {
-			for j := i + 1; j < n; j++ {
-				if v := math.Abs(a[i][j]); v > amax {
-					amax = v
-					p, q = i, j
-				}
-			}
-		}
-		if amax < tol {
-			break
-		}
-		// Jacobi rotation
-		theta := 0.5 * math.Atan2(2*a[p][q], a[q][q]-a[p][p])
-		c, s := math.Cos(theta), math.Sin(theta)
-		// apply rotation
-		newA := make([][]float64, n)
-		for i := range newA {
-			newA[i] = make([]float64, n)
-			copy(newA[i], a[i])
-		}
-		app := c*c*a[p][p] + 2*c*s*a[p][q] + s*s*a[q][q]
-		aqq := s*s*a[p][p] - 2*c*s*a[p][q] + c*c*a[q][q]
-		apq := 0.0
-		newA[p][p] = app
-		newA[q][q] = aqq
-		newA[p][q] = apq
-		newA[q][p] = apq
-		for r := 0; r < n; r++ {
-			if r == p || r == q {
-				continue
-			}
-			arp := c*a[r][p] + s*a[r][q]
-			arq := -s*a[r][p] + c*a[r][q]
-			newA[r][p] = arp
-			newA[p][r] = arp
-			newA[r][q] = arq
-			newA[q][r] = arq
-		}
-		a = newA
-	}
-
-	eigs := make([]float64, n)
-	for i := range eigs {
-		eigs[i] = a[i][i]
-	}
-	return eigs
-}

internal/eval/eval_test.go

@@ -1,138 +0,0 @@
-package eval_test
-
-import (
-	"math"
-	"math/rand"
-	"testing"
-
-	"gitea.d-ma.be/mathias/jepa-fx-risk/internal/eval"
-)
-
-func seededRNG(seed int64) *rand.Rand {
-	return rand.New(rand.NewSource(seed))
-}
-
-// ── LinearProbe (val_vol_r2) ──────────────────────────────────────────────────
-
-func TestLinearProbe_Perfect(t *testing.T) {
-	n := 50
-	emb := make([][]float64, n)
-	y := make([]float64, n)
-	for i := range emb {
-		emb[i] = []float64{float64(i)}
-		y[i] = float64(i)
-	}
-	r2 := eval.LinearProbe(emb, y, 1e-3)
-	if r2 < 0.99 {
-		t.Fatalf("perfect predictor: want R²≥0.99, got %.4f", r2)
-	}
-}
-
-func TestLinearProbe_ConstantTarget(t *testing.T) {
-	n := 40
-	emb := make([][]float64, n)
-	y := make([]float64, n)
-	for i := range emb {
-		emb[i] = []float64{float64(i), float64(i * i)}
-		y[i] = 3.0
-	}
-	r2 := eval.LinearProbe(emb, y, 1e-3)
-	if r2 > 0.01 {
-		t.Fatalf("constant target: want R²≤0.01, got %.4f", r2)
-	}
-}
-
-func TestLinearProbe_NoiseEmbedding(t *testing.T) {
-	rng := seededRNG(42)
-	n := 80
-	emb := make([][]float64, n)
-	y := make([]float64, n)
-	for i := range emb {
-		emb[i] = []float64{rng.NormFloat64(), rng.NormFloat64()}
-		y[i] = float64(i)
-	}
-	r2 := eval.LinearProbe(emb, y, 1e-3)
-	if r2 > 0.10 {
-		t.Fatalf("noise embedding: want R²<0.10, got %.4f", r2)
-	}
-}
-
-// ── Silhouette ────────────────────────────────────────────────────────────────
-
-func TestSilhouette_PerfectClusters(t *testing.T) {
-	emb := make([][]float64, 40)
-	labels := make([]int, 40)
-	for i := range emb {
-		if i < 20 {
-			emb[i] = []float64{0.0, 0.0}
-			labels[i] = 0
-		} else {
-			emb[i] = []float64{1000.0, 1000.0}
-			labels[i] = 1
-		}
-	}
-	sil, err := eval.Silhouette(emb, labels)
-	if err != nil {
-		t.Fatal(err)
-	}
-	if sil < 0.95 {
-		t.Fatalf("perfect clusters: want sil≥0.95, got %.4f", sil)
-	}
-}
-
-func TestSilhouette_SingleLabel(t *testing.T) {
-	emb := [][]float64{{1, 2}, {3, 4}, {5, 6}}
-	labels := []int{0, 0, 0}
-	_, err := eval.Silhouette(emb, labels)
-	if err == nil {
-		t.Fatal("expected error for single-label input")
-	}
-}
-
-func TestSilhouette_RandomClusters(t *testing.T) {
-	rng := seededRNG(7)
-	n := 60
-	emb := make([][]float64, n)
-	labels := make([]int, n)
-	for i := range emb {
-		emb[i] = []float64{rng.NormFloat64(), rng.NormFloat64()}
-		labels[i] = i % 2
-	}
-	sil, err := eval.Silhouette(emb, labels)
-	if err != nil {
-		t.Fatal(err)
-	}
-	if math.Abs(sil) > 0.30 {
-		t.Fatalf("random clusters: want |sil|≤0.30, got %.4f", sil)
-	}
-}
-
-// ── EffectiveRank ─────────────────────────────────────────────────────────────
-
-func TestEffectiveRank_Rank1(t *testing.T) {
-	emb := make([][]float64, 30)
-	for i := range emb {
-		emb[i] = []float64{1.0, 2.0, 3.0, 4.0}
-	}
-	er := eval.EffectiveRank(emb)
-	if er > 1.5 {
-		t.Fatalf("rank-1 matrix: want erank≤1.5, got %.4f", er)
-	}
-}
-
-func TestEffectiveRank_FullRank(t *testing.T) {
-	rng := seededRNG(99)
-	dim := 8
-	emb := make([][]float64, 200)
-	for i := range emb {
-		row := make([]float64, dim)
-		for j := range row {
-			row[j] = rng.NormFloat64()
-		}
-		emb[i] = row
-	}
-	er := eval.EffectiveRank(emb)
-	if er < float64(dim)*0.7 {
-		t.Fatalf("full-rank: want erank≥%.1f, got %.4f", float64(dim)*0.7, er)
-	}
-}

internal/eval/var.go

@@ -1,107 +0,0 @@
-package eval
-
-import "math"
-
-// VaRBreachRate computes the parametric 99% VaR breach rate and Kupiec POF p-value.
-//
-// VaR_99_t  = predVol[t] × z99   (z99 = 2.326 for 99% normal VaR)
-// breach_t  = actualVol[t] > VaR_99_t  (strict inequality)
-// breachRate = fraction of breaches over all steps
-// kupiecP   = Kupiec POF p-value: P(chi²(1) > LR) where LR is the likelihood ratio
-//             testing H0: true breach probability = 1%. High p = well-calibrated.
-//
-// Returns (0, 1) for empty or mismatched input.
-func VaRBreachRate(predVol, actualVol []float64, z99 float64) (breachRate, kupiecP float64) {
-	n := len(predVol)
-	if n == 0 || n != len(actualVol) {
-		return 0, 1
-	}
-
-	var n1 int
-	for i := 0; i < n; i++ {
-		if actualVol[i] > predVol[i]*z99 {
-			n1++
-		}
-	}
-
-	breachRate = float64(n1) / float64(n)
-	kupiecP = kupiecPOF(n, n1, 0.01)
-	return
-}
-
-// kupiecPOF returns the Kupiec Proportion-of-Failures p-value.
-// H0: true breach probability = p0 (e.g. 0.01 for 99% VaR).
-// Returns 1.0 for edge cases (n=0, p_hat=p0).
-func kupiecPOF(n, n1 int, p0 float64) float64 {
-	if n == 0 {
-		return 1.0
-	}
-	n0 := n - n1
-	phat := float64(n1) / float64(n)
-
-	var lr float64
-	switch {
-	case n1 == 0:
-		// 0 × ln(0/p0) = 0 by convention; only the n0 term contributes
-		lr = 2 * float64(n0) * math.Log((1-phat)/(1-p0))
-	case n1 == n:
-		// n0 term vanishes
-		lr = 2 * float64(n1) * math.Log(phat/p0)
-	default:
-		lr = 2 * (float64(n1)*math.Log(phat/p0) + float64(n0)*math.Log((1-phat)/(1-p0)))
-	}
-
-	if lr <= 0 {
-		return 1.0
-	}
-	// P(chi²(1) > LR) = erfc(sqrt(LR/2))  [chi²(1) = Z², Z~N(0,1)]
-	return math.Erfc(math.Sqrt(lr / 2))
-}
-
-// LinearProbePredict fits ridge regression on (trainEmb, trainY) and returns
-// predictions for testEmb. Complements LinearProbeTrainTest when the caller
-// needs the raw predictions (e.g. to compute VaR breach rate).
-// Returns nil when trainEmb is empty.
-func LinearProbePredict(trainEmb [][]float64, trainY []float64,
-	testEmb [][]float64, lambda float64) []float64 {
-	n := len(trainEmb)
-	if n == 0 || len(testEmb) == 0 {
-		return nil
-	}
-	d := len(trainEmb[0])
-	p := d + 1
-
-	A := make([][]float64, n)
-	for i, e := range trainEmb {
-		row := make([]float64, p)
-		copy(row, e)
-		row[d] = 1.0
-		A[i] = row
-	}
-	AtA := make([][]float64, p)
-	for i := range AtA {
-		AtA[i] = make([]float64, p)
-	}
-	Aty := make([]float64, p)
-	for i := 0; i < n; i++ {
-		for j := 0; j < p; j++ {
-			Aty[j] += A[i][j] * trainY[i]
-			for k := 0; k < p; k++ {
-				AtA[j][k] += A[i][j] * A[i][k]
-			}
-		}
-	}
-	for j := 0; j < p; j++ {
-		AtA[j][j] += lambda
-	}
-	w := solveCholesky(AtA, Aty)
-
-	preds := make([]float64, len(testEmb))
-	for i, e := range testEmb {
-		row := make([]float64, p)
-		copy(row, e)
-		row[d] = 1.0
-		preds[i] = dot(row, w)
-	}
-	return preds
-}

internal/eval/var_test.go

@@ -1,138 +0,0 @@
-package eval_test
-
-import (
-	"math"
-	"testing"
-
-	"gitea.d-ma.be/mathias/jepa-fx-risk/internal/eval"
-)
-
-// ── VaRBreachRate golden tests ──────────────────────────────────────────────
-//
-// VaR_99_t  = predVol[t] × z99   (parametric 99% normal VaR)
-// breach_t  = actualVol[t] > VaR_99_t
-// breachRate = mean(breach_t)
-// kupiecP   = Kupiec POF p-value (chi²(1) test, H0: breach rate = 1%)
-
-func TestVaRBreachRate_ZeroBreaches(t *testing.T) {
-	// 0.02 < 0.01×2.326=0.02326 → no breaches
-	pred := []float64{0.01, 0.01, 0.01}
-	act := []float64{0.02, 0.02, 0.02}
-	rate, _ := eval.VaRBreachRate(pred, act, 2.326)
-	if rate != 0 {
-		t.Fatalf("want rate=0, got %.4f", rate)
-	}
-}
-
-func TestVaRBreachRate_AllBreach(t *testing.T) {
-	// 0.03 > 0.02326 → all breach
-	pred := []float64{0.01, 0.01}
-	act := []float64{0.03, 0.03}
-	rate, _ := eval.VaRBreachRate(pred, act, 2.326)
-	if math.Abs(rate-1.0) > 1e-9 {
-		t.Fatalf("want rate=1.0, got %.4f", rate)
-	}
-}
-
-func TestVaRBreachRate_Golden(t *testing.T) {
-	// n=10, 2 breaches at indices 0 and 2 → rate=0.2
-	// Kupiec: p_hat=0.2 vs p0=0.01 → strongly reject H0 (p < 0.05)
-	pred := make([]float64, 10)
-	act := make([]float64, 10)
-	for i := range pred {
-		pred[i] = 0.01
-		act[i] = 0.01 // no breach: 0.01 < 0.02326
-	}
-	act[0] = 0.03 // breach
-	act[2] = 0.03 // breach
-
-	rate, kupiecP := eval.VaRBreachRate(pred, act, 2.326)
-
-	if math.Abs(rate-0.2) > 1e-9 {
-		t.Fatalf("breach rate: want 0.2, got %.4f", rate)
-	}
-	if kupiecP > 0.05 {
-		t.Fatalf("kupiec p-value: want <0.05 (strong reject H0), got %.4f", kupiecP)
-	}
-}
-
-func TestVaRBreachRate_PerfectCalibration(t *testing.T) {
-	// n=100, exactly 1 breach → p_hat=0.01=p0 → LR=0 → kupiecP≈1.0
-	n := 100
-	pred := make([]float64, n)
-	act := make([]float64, n)
-	for i := range pred {
-		pred[i] = 0.01
-		act[i] = 0.015 // < 0.02326, no breach
-	}
-	act[0] = 0.025 // > 0.02326, breach
-
-	rate, kupiecP := eval.VaRBreachRate(pred, act, 2.326)
-
-	if math.Abs(rate-0.01) > 1e-9 {
-		t.Fatalf("breach rate: want 0.01, got %.4f", rate)
-	}
-	if kupiecP < 0.9 {
-		t.Fatalf("kupiec p-value: want ≈1.0 (well calibrated), got %.4f", kupiecP)
-	}
-}
-
-func TestVaRBreachRate_EmptyInput(t *testing.T) {
-	rate, kupiecP := eval.VaRBreachRate(nil, nil, 2.326)
-	if rate != 0 || kupiecP != 1 {
-		t.Fatalf("empty: want (0,1), got (%.4f,%.4f)", rate, kupiecP)
-	}
-}
-
-func TestVaRBreachRate_LenMismatch(t *testing.T) {
-	rate, kupiecP := eval.VaRBreachRate([]float64{0.01}, []float64{0.01, 0.02}, 2.326)
-	if rate != 0 || kupiecP != 1 {
-		t.Fatalf("mismatch: want (0,1), got (%.4f,%.4f)", rate, kupiecP)
-	}
-}
-
-func TestVaRBreachRate_Z99Default(t *testing.T) {
-	// z99=2.326 is the canonical value; test that boundary case works
-	// VaR = 0.01 × 2.326 = 0.02326
-	// actual = 0.02326 → NOT a breach (strict >)
-	pred := []float64{0.01}
-	act := []float64{0.02326}
-	rate, _ := eval.VaRBreachRate(pred, act, 2.326)
-	if rate != 0 {
-		t.Fatalf("boundary: exactly at VaR is not a breach; want rate=0, got %.4f", rate)
-	}
-}
-
-// ── LinearProbePredict ──────────────────────────────────────────────────────
-
-func TestLinearProbePredict_PerfectLinear(t *testing.T) {
-	// y = x; predictions should match targets closely
-	n := 20
-	trainEmb := make([][]float64, n)
-	trainY := make([]float64, n)
-	testEmb := make([][]float64, 5)
-	testY := []float64{5, 10, 15, 20, 25}
-	for i := range trainEmb {
-		trainEmb[i] = []float64{float64(i)}
-		trainY[i] = float64(i)
-	}
-	for i := range testEmb {
-		testEmb[i] = []float64{testY[i]}
-	}
-	preds := eval.LinearProbePredict(trainEmb, trainY, testEmb, 1e-3)
-	if len(preds) != len(testEmb) {
-		t.Fatalf("len: want %d, got %d", len(testEmb), len(preds))
-	}
-	for i, p := range preds {
-		if math.Abs(p-testY[i]) > 1.0 {
-			t.Fatalf("pred[%d]: want ≈%.1f, got %.4f", i, testY[i], p)
-		}
-	}
-}
-
-func TestLinearProbePredict_EmptyTrain(t *testing.T) {
-	preds := eval.LinearProbePredict(nil, nil, [][]float64{{1.0}}, 1e-3)
-	if len(preds) != 0 {
-		t.Fatalf("empty train: want nil/empty preds, got len=%d", len(preds))
-	}
-}

internal/web/index_templ.go

@@ -1,100 +0,0 @@
-// Code generated by templ - DO NOT EDIT.
-
-// templ: version: v0.3.1020
-package web
-
-//lint:file-ignore SA4006 This context is only used if a nested component is present.
-
-import "github.com/a-h/templ"
-import templruntime "github.com/a-h/templ/runtime"
-
-func Index() templ.Component {
-	return templruntime.GeneratedTemplate(func(templ_7745c5c3_Input templruntime.GeneratedComponentInput) (templ_7745c5c3_Err error) {
-		templ_7745c5c3_W, ctx := templ_7745c5c3_Input.Writer, templ_7745c5c3_Input.Context
-		if templ_7745c5c3_CtxErr := ctx.Err(); templ_7745c5c3_CtxErr != nil {
-			return templ_7745c5c3_CtxErr
-		}
-		templ_7745c5c3_Buffer, templ_7745c5c3_IsBuffer := templruntime.GetBuffer(templ_7745c5c3_W)
-		if !templ_7745c5c3_IsBuffer {
-			defer func() {
-				templ_7745c5c3_BufErr := templruntime.ReleaseBuffer(templ_7745c5c3_Buffer)
-				if templ_7745c5c3_Err == nil {
-					templ_7745c5c3_Err = templ_7745c5c3_BufErr
-				}
-			}()
-		}
-		ctx = templ.InitializeContext(ctx)
-		templ_7745c5c3_Var1 := templ.GetChildren(ctx)
-		if templ_7745c5c3_Var1 == nil {
-			templ_7745c5c3_Var1 = templ.NopComponent
-		}
-		ctx = templ.ClearChildren(ctx)
-		templ_7745c5c3_Var2 := templruntime.GeneratedTemplate(func(templ_7745c5c3_Input templruntime.GeneratedComponentInput) (templ_7745c5c3_Err error) {
-			templ_7745c5c3_W, ctx := templ_7745c5c3_Input.Writer, templ_7745c5c3_Input.Context
-			templ_7745c5c3_Buffer, templ_7745c5c3_IsBuffer := templruntime.GetBuffer(templ_7745c5c3_W)
-			if !templ_7745c5c3_IsBuffer {
-				defer func() {
-					templ_7745c5c3_BufErr := templruntime.ReleaseBuffer(templ_7745c5c3_Buffer)
-					if templ_7745c5c3_Err == nil {
-						templ_7745c5c3_Err = templ_7745c5c3_BufErr
-					}
-				}()
-			}
-			ctx = templ.InitializeContext(ctx)
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 1, "<h1 class=\"text-3xl font-semibold mb-6\">hostexecutor</h1><button hx-get=\"/api/hello\" hx-target=\"#out\" class=\"px-4 py-2 bg-slate-900 text-white rounded-md hover:bg-slate-700\">Say hello</button><div id=\"out\" class=\"mt-6 text-slate-700\"></div>")
-			if templ_7745c5c3_Err != nil {
-				return templ_7745c5c3_Err
-			}
-			return nil
-		})
-		templ_7745c5c3_Err = Layout("hostexecutor").Render(templ.WithChildren(ctx, templ_7745c5c3_Var2), templ_7745c5c3_Buffer)
-		if templ_7745c5c3_Err != nil {
-			return templ_7745c5c3_Err
-		}
-		return nil
-	})
-}
-
-func Hello(name string) templ.Component {
-	return templruntime.GeneratedTemplate(func(templ_7745c5c3_Input templruntime.GeneratedComponentInput) (templ_7745c5c3_Err error) {
-		templ_7745c5c3_W, ctx := templ_7745c5c3_Input.Writer, templ_7745c5c3_Input.Context
-		if templ_7745c5c3_CtxErr := ctx.Err(); templ_7745c5c3_CtxErr != nil {
-			return templ_7745c5c3_CtxErr
-		}
-		templ_7745c5c3_Buffer, templ_7745c5c3_IsBuffer := templruntime.GetBuffer(templ_7745c5c3_W)
-		if !templ_7745c5c3_IsBuffer {
-			defer func() {
-				templ_7745c5c3_BufErr := templruntime.ReleaseBuffer(templ_7745c5c3_Buffer)
-				if templ_7745c5c3_Err == nil {
-					templ_7745c5c3_Err = templ_7745c5c3_BufErr
-				}
-			}()
-		}
-		ctx = templ.InitializeContext(ctx)
-		templ_7745c5c3_Var3 := templ.GetChildren(ctx)
-		if templ_7745c5c3_Var3 == nil {
-			templ_7745c5c3_Var3 = templ.NopComponent
-		}
-		ctx = templ.ClearChildren(ctx)
-		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 2, "<p>Hello, ")
-		if templ_7745c5c3_Err != nil {
-			return templ_7745c5c3_Err
-		}
-		var templ_7745c5c3_Var4 string
-		templ_7745c5c3_Var4, templ_7745c5c3_Err = templ.JoinStringErrs(name)
-		if templ_7745c5c3_Err != nil {
-			return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/index.templ`, Line: 15, Col: 17}
-		}
-		_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var4))
-		if templ_7745c5c3_Err != nil {
-			return templ_7745c5c3_Err
-		}
-		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 3, "!</p>")
-		if templ_7745c5c3_Err != nil {
-			return templ_7745c5c3_Err
-		}
-		return nil
-	})
-}
-
-var _ = templruntime.GeneratedTemplate

internal/web/layout_templ.go

@@ -1,61 +0,0 @@
-// Code generated by templ - DO NOT EDIT.
-
-// templ: version: v0.3.1020
-package web
-
-//lint:file-ignore SA4006 This context is only used if a nested component is present.
-
-import "github.com/a-h/templ"
-import templruntime "github.com/a-h/templ/runtime"
-
-func Layout(title string) templ.Component {
-	return templruntime.GeneratedTemplate(func(templ_7745c5c3_Input templruntime.GeneratedComponentInput) (templ_7745c5c3_Err error) {
-		templ_7745c5c3_W, ctx := templ_7745c5c3_Input.Writer, templ_7745c5c3_Input.Context
-		if templ_7745c5c3_CtxErr := ctx.Err(); templ_7745c5c3_CtxErr != nil {
-			return templ_7745c5c3_CtxErr
-		}
-		templ_7745c5c3_Buffer, templ_7745c5c3_IsBuffer := templruntime.GetBuffer(templ_7745c5c3_W)
-		if !templ_7745c5c3_IsBuffer {
-			defer func() {
-				templ_7745c5c3_BufErr := templruntime.ReleaseBuffer(templ_7745c5c3_Buffer)
-				if templ_7745c5c3_Err == nil {
-					templ_7745c5c3_Err = templ_7745c5c3_BufErr
-				}
-			}()
-		}
-		ctx = templ.InitializeContext(ctx)
-		templ_7745c5c3_Var1 := templ.GetChildren(ctx)
-		if templ_7745c5c3_Var1 == nil {
-			templ_7745c5c3_Var1 = templ.NopComponent
-		}
-		ctx = templ.ClearChildren(ctx)
-		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 1, "<!doctype html><html lang=\"en\"><head><meta charset=\"utf-8\"><meta name=\"viewport\" content=\"width=device-width,initial-scale=1\"><title>")
-		if templ_7745c5c3_Err != nil {
-			return templ_7745c5c3_Err
-		}
-		var templ_7745c5c3_Var2 string
-		templ_7745c5c3_Var2, templ_7745c5c3_Err = templ.JoinStringErrs(title)
-		if templ_7745c5c3_Err != nil {
-			return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/layout.templ`, Line: 9, Col: 17}
-		}
-		_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var2))
-		if templ_7745c5c3_Err != nil {
-			return templ_7745c5c3_Err
-		}
-		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 2, "</title><script src=\"https://unpkg.com/htmx.org@2.0.0\"></script><script src=\"https://cdn.tailwindcss.com\"></script></head><body class=\"min-h-screen bg-slate-50 text-slate-900 antialiased\"><main class=\"max-w-3xl mx-auto px-6 py-12\">")
-		if templ_7745c5c3_Err != nil {
-			return templ_7745c5c3_Err
-		}
-		templ_7745c5c3_Err = templ_7745c5c3_Var1.Render(ctx, templ_7745c5c3_Buffer)
-		if templ_7745c5c3_Err != nil {
-			return templ_7745c5c3_Err
-		}
-		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 3, "</main></body></html>")
-		if templ_7745c5c3_Err != nil {
-			return templ_7745c5c3_Err
-		}
-		return nil
-	})
-}
-
-var _ = templruntime.GeneratedTemplate

loop.py

@@ -1,283 +0,0 @@
-"""loop.py — Karpathy-style autoresearch loop for jepa-fx-risk.
-
-Agent (on iguana/berget — NOT koala, whose GPU is reserved for train.py) reads
-program.md + train.py + STATUS.md, proposes ONE change to train.py, we run it,
-keep if val_vol_r2 improved else git-revert. Appends per-iter record to STATUS.md.
-
-    LITELLM_KEY=xxx python loop.py [--iters N] [--model MODEL] [--run-dir runs/rq-04]
-
-Env:
-  LITELLM_KEY      — LiteLLM master key (required)
-  LITELLM_BASE     — default http://localhost:30401/v1
-  LOOP_MODEL       — default berget/gemma4-31b (non-thinking; iguana/berget only)
-  LOOP_ITERS       — default 3
-  TRAIN_TIMEOUT    — seconds per train.py run, default 120
-  NTFY_URL         — optional: POST crash/stall alerts here (e.g. ntfy.sh/<topic>)
-"""
-import argparse
-import json
-import os
-import subprocess
-import sys
-import time
-import textwrap
-from pathlib import Path
-
-import urllib.request
-
-LITELLM_BASE  = os.environ.get("LITELLM_BASE", "http://localhost:30401/v1")
-LITELLM_KEY   = os.environ.get("LITELLM_KEY", "")
-LOOP_MODEL    = os.environ.get("LOOP_MODEL", "berget/gemma4-31b")
-LOOP_ITERS    = int(os.environ.get("LOOP_ITERS", "3"))
-TRAIN_TIMEOUT = int(os.environ.get("TRAIN_TIMEOUT", "120"))
-NTFY_URL      = os.environ.get("NTFY_URL", "")
-
-# Resolved by main() once --run-dir is parsed.
-RUN_DIR      = Path(".")
-STATUS_MD    = Path("STATUS.md")
-METRICS_JSON = Path("metrics.json")
-TRAIN_PY     = Path("train.py")
-HEARTBEAT    = Path("HEARTBEAT")
-
-AGENT_SYSTEM = textwrap.dedent("""\
-    You are the autoresearch agent for jepa-fx-risk. Your job: propose ONE small,
-    targeted change to train.py to improve val_vol_r2 (OOS R² predicting 1-day
-    realized vol from frozen embeddings). Higher is better.
-
-    Rules:
-    - Return ONLY the full new content of train.py — nothing else, no explanation,
-      no markdown fence. Raw Python only.
-    - Change ONE thing at a time (one knob, one structural idea).
-    - Do NOT touch prepare_data.py, loop.py, or the data pipeline — only train.py.
-    - Do NOT add new data sources or new files.
-    - The metric is computed externally from your frozen embeddings; trust it.
-""")
-
-
-def read_file(p: Path) -> str:
-    return p.read_text() if p.exists() else ""
-
-
-def gpu_snapshot() -> str:
-    try:
-        out = subprocess.check_output(
-            ["nvidia-smi", "--query-gpu=utilization.gpu,memory.used,memory.total,temperature.gpu",
-             "--format=csv,noheader,nounits"], timeout=5, text=True
-        ).strip()
-        util, mem_used, mem_total, temp = [x.strip() for x in out.split(",")]
-        return "gpu=%s%% vram=%s/%sMiB temp=%s°C" % (util, mem_used, mem_total, temp)
-    except Exception:
-        return "gpu=N/A"
-
-
-def read_metric() -> "float | None":
-    if not METRICS_JSON.exists():
-        return None
-    try:
-        return float(json.loads(METRICS_JSON.read_text())["val_vol_r2"])
-    except Exception:
-        return None
-
-
-def run_train() -> "tuple[float | None, float, str]":
-    """Run train.py from project root with METRICS_OUT pointing into the run dir."""
-    t0 = time.time()
-    env = dict(os.environ)
-    env["METRICS_OUT"] = str(METRICS_JSON.resolve())
-    try:
-        r = subprocess.run(
-            [sys.executable, str(TRAIN_PY.resolve())],
-            capture_output=True, text=True, timeout=TRAIN_TIMEOUT, env=env,
-        )
-        elapsed = time.time() - t0
-        if r.returncode != 0:
-            return None, elapsed, (r.stderr or r.stdout)[-300:]
-        metric = read_metric()
-        return metric, elapsed, ""
-    except subprocess.TimeoutExpired:
-        return None, TRAIN_TIMEOUT, "TIMEOUT"
-
-
-def call_agent(iteration: int, best_so_far: "float | None") -> str:
-    """Ask the LLM agent to edit train.py. Returns new train.py content."""
-    context = "\n\n".join([
-        "# program.md\n" + read_file(RUN_DIR / "program.md"),
-        "# train.py (current)\n" + read_file(TRAIN_PY),
-        "# STATUS.md (history)\n" + read_file(STATUS_MD)[-2000:],
-        "# metrics.json (last run)\n" + read_file(METRICS_JSON),
-        "Iteration %d. Best val_vol_r2 so far: %s. Improve it." % (
-            iteration, "%.4f" % best_so_far if best_so_far is not None else "none yet"
-        ),
-    ])
-    payload = json.dumps({
-        "model": LOOP_MODEL,
-        "messages": [
-            {"role": "system", "content": AGENT_SYSTEM},
-            {"role": "user",   "content": context},
-        ],
-        "temperature": 0.7,
-        "max_tokens": 4096,
-    }).encode()
-    req = urllib.request.Request(
-        LITELLM_BASE + "/chat/completions",
-        data=payload,
-        headers={"Authorization": "Bearer " + LITELLM_KEY,
-                 "Content-Type": "application/json"},
-        method="POST",
-    )
-    resp = urllib.request.urlopen(req, timeout=60)
-    data = json.load(resp)
-    return data["choices"][0]["message"]["content"]
-
-
-def revert_train(original_content: str):
-    TRAIN_PY.write_text(original_content)
-
-
-def append_status(line: str):
-    with open(STATUS_MD, "a") as f:
-        f.write(line + "\n")
-
-
-def write_heartbeat(iteration: int, status: str = "alive"):
-    """Update HEARTBEAT so watchdogs can detect stalls."""
-    HEARTBEAT.write_text("%s  iter=%d  ts=%.0f\n" % (status, iteration, time.time()))
-
-
-def ntfy(msg: str):
-    """POST an alert to NTFY_URL (best-effort; silently ignored on any error)."""
-    if not NTFY_URL:
-        return
-    try:
-        req = urllib.request.Request(
-            NTFY_URL, data=msg.encode(), method="POST",
-            headers={"Content-Type": "text/plain"},
-        )
-        urllib.request.urlopen(req, timeout=5)
-    except Exception:
-        pass
-
-
-def main():
-    global RUN_DIR, STATUS_MD, METRICS_JSON, TRAIN_PY, HEARTBEAT
-
-    parser = argparse.ArgumentParser()
-    parser.add_argument("--iters", type=int, default=LOOP_ITERS)
-    parser.add_argument("--model", default=LOOP_MODEL)
-    parser.add_argument(
-        "--run-dir", default=None,
-        help="run dir scaffolded by autoresearch_start.py; "
-             "STATUS.md, metrics.json, HEARTBEAT, and train.py live here",
-    )
-    args = parser.parse_args()
-
-    loop_iters = args.iters
-    loop_model = args.model
-
-    if args.run_dir:
-        RUN_DIR = Path(args.run_dir)
-        if not RUN_DIR.is_dir():
-            print("ERROR: run dir not found:", RUN_DIR); sys.exit(1)
-
-    STATUS_MD    = RUN_DIR / "STATUS.md"
-    METRICS_JSON = RUN_DIR / "metrics.json"
-    TRAIN_PY     = RUN_DIR / "train.py"
-    HEARTBEAT    = RUN_DIR / "HEARTBEAT"
-
-    if not LITELLM_KEY:
-        print("ERROR: set LITELLM_KEY"); sys.exit(1)
-
-    if not STATUS_MD.exists():
-        STATUS_MD.write_text(
-            "# Autoresearch STATUS\n\n"
-            "| iter | val_vol_r2 | delta | action | secs | gpu | change |\n"
-            "|------|-----------|-------|--------|------|-----|--------|\n"
-        )
-
-    baseline = read_metric()
-    if baseline is None:
-        print("No metrics.json — running train.py for baseline...")
-        m, secs, err = run_train()
-        if m is None:
-            msg = "Baseline run failed: " + err
-            print(msg)
-            ntfy("[jepa-fx-risk] loop CRASH — " + msg)
-            sys.exit(1)
-        baseline = m
-        print("Baseline: val_vol_r2 = %.4f (%.1fs)" % (baseline, secs))
-
-    best = baseline
-    print("Starting loop | model=%s | iters=%d | baseline=%.4f" % (loop_model, loop_iters, best))
-    if args.run_dir:
-        print("  run-dir:", RUN_DIR)
-
-    iter_index = 0
-    try:
-        for i in range(1, loop_iters + 1):
-            iter_index = i
-            write_heartbeat(i, "agent-call")
-            print("\n--- iter %d/%d ---" % (i, loop_iters))
-            original = TRAIN_PY.read_text()
-
-            print("  calling agent (%s)..." % loop_model)
-            t_agent = time.time()
-            try:
-                new_code = call_agent(i, best)
-            except Exception as e:
-                msg = str(e)
-                print("  agent call failed:", msg)
-                append_status("| %d | ERR | — | agent-fail | — | — | %s |" % (i, msg[:60]))
-                write_heartbeat(i, "agent-fail")
-                ntfy("[jepa-fx-risk] iter %d agent FAIL — %s" % (i, msg[:80]))
-                continue
-            agent_secs = time.time() - t_agent
-            print("  agent replied in %.1fs" % agent_secs)
-
-            # strip accidental markdown fences
-            if new_code.strip().startswith("```"):
-                lines = new_code.strip().splitlines()
-                new_code = "\n".join(lines[1:-1] if lines[-1].strip() == "```" else lines[1:])
-
-            TRAIN_PY.write_text(new_code)
-
-            write_heartbeat(i, "training")
-            gpu = gpu_snapshot()
-            print("  running train.py [%s]..." % gpu)
-            metric, secs, err = run_train()
-
-            if metric is None:
-                print("  train.py FAILED — reverting. err:", err[:100])
-                revert_train(original)
-                append_status("| %d | FAIL | — | revert | %.0fs | %s | run error |" % (i, secs, gpu))
-                write_heartbeat(i, "train-fail")
-                ntfy("[jepa-fx-risk] iter %d train FAIL — %s" % (i, err[:80]))
-                continue
-
-            delta = metric - best
-            if metric > best:
-                best = metric
-                action = "KEEP"
-            else:
-                revert_train(original)
-                action = "revert"
-
-            summary = "| %d | %.4f | %+.4f | %s | %.0fs | %s | iter%d |" % (
-                i, metric, delta, action, secs, gpu, i)
-            append_status(summary)
-            write_heartbeat(i, "done")
-            print("  val_vol_r2=%.4f  delta=%+.4f  action=%s  [%.0fs]" % (metric, delta, action, secs))
-
-    except Exception as e:
-        msg = "loop CRASH at iter %d: %s" % (iter_index, e)
-        print("FATAL:", msg)
-        ntfy("[jepa-fx-risk] " + msg)
-        raise
-
-    print("\nDone. Best val_vol_r2 = %.4f (baseline was %.4f, delta %+.4f)" % (best, baseline, best - baseline))
-    print("STATUS.md updated.")
-    write_heartbeat(loop_iters, "done")
-    ntfy("[jepa-fx-risk] loop done. best val_vol_r2=%.4f (delta %+.4f)" % (best, best - baseline))
-
-
-if __name__ == "__main__":
-    main()

metrics.json

@@ -1,14 +0,0 @@
-{
-  "val_vol_r2": 0.3641397896593044,
-  "phase1_r2": 0.3908407688140869,
-  "n_test": 11641,
-  "knobs": {
-    "WINDOW": 120,
-    "PATCH_LEN": 24,
-    "D_MODEL": 128,
-    "DEPTH": 2,
-    "ALPHA": 0.1,
-    "DELTA_T_MAX": 3,
-    "EPOCHS": 300
-  }
-}

model/README.md

@@ -0,0 +1,31 @@
+# model
+
+Python + PyTorch perimeter. This is the only directory in the project that uses Python.
+
+## Contents
+
+```
+model/
+  train.py              TS-JEPA training entry point
+  configs/              YAML configs per phase
+    phase0-mae.yaml
+    phase1-tsjepa.yaml
+  tsjepa/               TS-JEPA model implementation (adapted from paper)
+  tests/                pytest tests for model components
+  requirements.txt      pinned Python dependencies
+  .venv/                (gitignored — created by `task model:setup`)
+```
+
+## Setup
+
+```bash
+task model:setup    # creates .venv and installs requirements via uv
+```
+
+## Dependency policy
+
+Every Python dependency must be justified in a comment in `requirements.txt`. Prefer Go implementations for anything outside the training loop. When adding a new dependency, add an entry to DECISIONS.md explaining why a Go alternative wasn't sufficient.
+
+## Python version
+
+3.12 (pinned in `.python-version`)

notebooks/README.md

@@ -0,0 +1,11 @@
+# Notebooks
+
+Exploratory Data Analysis (EDA) scratch space only.
+
+## Rules
+
+- **Never use notebooks for anything reproducible.** Training, evaluation, and metric computation belong in `src/` with tests.
+- **Strip all outputs before committing.** Use `nbstripout` or equivalent.
+- **Label every notebook with a phase prefix:** `phase0-eda-eurusd-distribution.ipynb`
+
+Notebooks are thinking tools, not research artifacts. If a finding from a notebook is worth keeping, it goes into a spec, a decision in `DECISIONS.md`, or a result in `results/summaries/` — not the notebook itself.

requirements.txt

@@ -1,10 +0,0 @@
-# Python deps for the autoresearch loop (train.py + scripts). Install torch from
-# the cu130 index FIRST (koala Blackwell sm_120, torch 2.12.1+cu130 verified):
-#   pip install torch --index-url https://download.pytorch.org/whl/cu130
-#   pip install -r requirements.txt
-numpy>=2.0
-pandas>=2.2
-pyarrow>=16
-histdata>=1.3        # histdata.com downloader (handles the tk token politely)
-hmmlearn>=0.3        # regime detector (prepare_regime.py, jepa-fx-risk#13)
-scikit-learn>=1.4    # HMM dependency

results/README.md

@@ -0,0 +1,32 @@
+# Results
+
+Tracked outputs from experiment runs. Large raw outputs are gitignored — only summaries are committed.
+
+## Structure
+
+```
+results/
+  summaries/    metric tables, key figures, per-phase result records (committed)
+  raw/          full embedding outputs, backtest CSVs (gitignored)
+```
+
+## Per-phase result record format
+
+Each concluded phase produces a result record in `summaries/`:
+
+```
+summaries/
+  phase-0-[pass|null].md
+  phase-1-[pass|null].md
+  ...
+```
+
+Each record must include:
+- Phase name and hypothesis
+- Key metrics (silhouette score, R², etc.) with confidence intervals where applicable
+- Baseline comparison
+- Verdict: PASS / NULL RESULT
+- If null: what was investigated, what was found, next step taken
+- Link to experiment git tag
+
+**Null results are valid research outputs and must be committed, not discarded.**

results/hpo/hpo_results.jsonl

@@ -1,18 +0,0 @@
-{"config": {"JEPA_D_MODEL": 64, "JEPA_DEPTH": 2, "JEPA_WINDOW": 120}, "val_vol_r2": 0.302411480667525, "phase1_r2": 0.35809940099716187, "stdout_last": "val_vol_r2 = 0.3024  (n_test=11641, dev=cuda)", "ts": "2026-06-26T10:37:37.028406"}
-{"config": {"JEPA_D_MODEL": 64, "JEPA_DEPTH": 2, "JEPA_WINDOW": 240}, "val_vol_r2": 0.29654798431244755, "phase1_r2": 0.35618388652801514, "stdout_last": "val_vol_r2 = 0.2965  (n_test=11641, dev=cuda)", "ts": "2026-06-26T10:37:49.822762"}
-{"config": {"JEPA_D_MODEL": 64, "JEPA_DEPTH": 2, "JEPA_WINDOW": 480}, "val_vol_r2": 0.31050360040290237, "phase1_r2": 0.36530405282974243, "stdout_last": "val_vol_r2 = 0.3105  (n_test=11641, dev=cuda)", "ts": "2026-06-26T10:38:02.966176"}
-{"config": {"JEPA_D_MODEL": 64, "JEPA_DEPTH": 4, "JEPA_WINDOW": 120}, "val_vol_r2": 0.2925057399716364, "phase1_r2": 0.3467639684677124, "stdout_last": "val_vol_r2 = 0.2925  (n_test=11641, dev=cuda)", "ts": "2026-06-26T10:38:16.195552"}
-{"config": {"JEPA_D_MODEL": 64, "JEPA_DEPTH": 4, "JEPA_WINDOW": 240}, "val_vol_r2": 0.29334667623516786, "phase1_r2": 0.35872191190719604, "stdout_last": "val_vol_r2 = 0.2933  (n_test=11641, dev=cuda)", "ts": "2026-06-26T10:38:31.372602"}
-{"config": {"JEPA_D_MODEL": 64, "JEPA_DEPTH": 4, "JEPA_WINDOW": 480}, "val_vol_r2": 0.3123527205416422, "phase1_r2": 0.3572431206703186, "stdout_last": "val_vol_r2 = 0.3124  (n_test=11641, dev=cuda)", "ts": "2026-06-26T10:38:46.672203"}
-{"config": {"JEPA_D_MODEL": 128, "JEPA_DEPTH": 2, "JEPA_WINDOW": 120}, "val_vol_r2": 0.3641397896593044, "phase1_r2": 0.3908407688140869, "stdout_last": "val_vol_r2 = 0.3641  (n_test=11641, dev=cuda)", "ts": "2026-06-26T10:39:00.793878"}
-{"config": {"JEPA_D_MODEL": 128, "JEPA_DEPTH": 2, "JEPA_WINDOW": 240}, "val_vol_r2": 0.35845865364171503, "phase1_r2": 0.3737195134162903, "stdout_last": "val_vol_r2 = 0.3585  (n_test=11641, dev=cuda)", "ts": "2026-06-26T10:39:13.321428"}
-{"config": {"JEPA_D_MODEL": 128, "JEPA_DEPTH": 2, "JEPA_WINDOW": 480}, "val_vol_r2": 0.35310115657814645, "phase1_r2": 0.35306859016418457, "stdout_last": "val_vol_r2 = 0.3531  (n_test=11641, dev=cuda)", "ts": "2026-06-26T10:39:26.402249"}
-{"config": {"JEPA_D_MODEL": 128, "JEPA_DEPTH": 4, "JEPA_WINDOW": 120}, "val_vol_r2": 0.3655629727960601, "phase1_r2": 0.371029257774353, "stdout_last": "val_vol_r2 = 0.3656  (n_test=11641, dev=cuda)", "ts": "2026-06-26T10:39:39.786248"}
-{"config": {"JEPA_D_MODEL": 128, "JEPA_DEPTH": 4, "JEPA_WINDOW": 240}, "val_vol_r2": 0.36109622605593217, "phase1_r2": 0.3666273355484009, "stdout_last": "val_vol_r2 = 0.3611  (n_test=11641, dev=cuda)", "ts": "2026-06-26T10:39:53.194111"}
-{"config": {"JEPA_D_MODEL": 128, "JEPA_DEPTH": 4, "JEPA_WINDOW": 480}, "val_vol_r2": 0.362228341965093, "phase1_r2": 0.3590735197067261, "stdout_last": "val_vol_r2 = 0.3622  (n_test=11641, dev=cuda)", "ts": "2026-06-26T10:40:06.991680"}
-{"config": {"JEPA_D_MODEL": 256, "JEPA_DEPTH": 2, "JEPA_WINDOW": 120}, "val_vol_r2": 0.3749483295047378, "phase1_r2": 0.3801569938659668, "stdout_last": "val_vol_r2 = 0.3749  (n_test=11641, dev=cuda)", "ts": "2026-06-26T10:40:21.512210"}
-{"config": {"JEPA_D_MODEL": 256, "JEPA_DEPTH": 2, "JEPA_WINDOW": 240}, "val_vol_r2": 0.3765593861479334, "phase1_r2": 0.38416117429733276, "stdout_last": "val_vol_r2 = 0.3766  (n_test=11641, dev=cuda)", "ts": "2026-06-26T10:40:34.959919"}
-{"config": {"JEPA_D_MODEL": 256, "JEPA_DEPTH": 2, "JEPA_WINDOW": 480}, "val_vol_r2": 0.3653399117639956, "phase1_r2": 0.3685130476951599, "stdout_last": "val_vol_r2 = 0.3653  (n_test=11641, dev=cuda)", "ts": "2026-06-26T10:40:48.806135"}
-{"config": {"JEPA_D_MODEL": 256, "JEPA_DEPTH": 4, "JEPA_WINDOW": 120}, "val_vol_r2": 0.375961424966925, "phase1_r2": 0.37861257791519165, "stdout_last": "val_vol_r2 = 0.3760  (n_test=11641, dev=cuda)", "ts": "2026-06-26T10:41:04.056939"}
-{"config": {"JEPA_D_MODEL": 256, "JEPA_DEPTH": 4, "JEPA_WINDOW": 240}, "val_vol_r2": 0.37841726893098504, "phase1_r2": 0.3781360387802124, "stdout_last": "val_vol_r2 = 0.3784  (n_test=11641, dev=cuda)", "ts": "2026-06-26T10:41:18.693263"}
-{"config": {"JEPA_D_MODEL": 256, "JEPA_DEPTH": 4, "JEPA_WINDOW": 480}, "val_vol_r2": 0.37118530199441635, "phase1_r2": 0.3651617765426636, "stdout_last": "val_vol_r2 = 0.3712  (n_test=11641, dev=cuda)", "ts": "2026-06-26T10:41:34.796157"}

results/summaries/phase-0-metrics.json

@@ -1,13 +0,0 @@
-{
-  "label": "null",
-  "mean_sil": 0.018206419112781685,
-  "pca_sil": 0.13589094579219818,
-  "spread": 0.8673340065023978,
-  "pc1_hv_corr": 0.525803392278542,
-  "per_seed": [
-    0.026790648698806763,
-    0.010999602265655994,
-    0.016829006373882294
-  ],
-  "passed": false
-}

results/summaries/phase-0-null.md

@@ -1,25 +0,0 @@
-# Phase-0 SSL feasibility gate — null
-
-**Date:** 2026-06-24
-**Path B deviation:** Daily 2019-2023 (not hourly 2008-2022); Python harness
-(not Go #4); gate metric adapted from silhouette-on-embedding to match
-available data. Go harness (#4) remains open for production experiments.
-
-## Data
-- Train: EUR/USD daily 2019-2021 (907 windows)
-- OOS: EUR/USD daily 2022-2023 (593 windows)
-- HV label: top-33% realized-vol days = high-volatility (196 days)
-
-## Results
-| | Value | Gate |
-|---|---|---|
-| TS-JEPA mean silhouette (3 seeds) | 0.0182 | > 0.20 → **False** |
-| Beats PCA baseline (0.1359) | 0.0182 | > PCA → **False** |
-| Seed stability (spread) | 86.73% | < 10% → **False** |
-| PC1/HV correlation | 0.5258 | < 0.95 → **True** |
-
-Per-seed: ['0.0268', '0.0110', '0.0168']
-
-## Verdict: **NULL**
-
-One or more gate criteria not met. See null result protocol in #5.

scripts/autoresearch_start.py

@@ -1,155 +0,0 @@
-"""autoresearch start — scaffold a run dir from an Autoresearch Council backlog leaf.
-
-Usage:
-    python scripts/autoresearch_start.py <backlog.json> <rq-id>
-
-Reads the Council backlog JSON (from agentsquad autoresearch_pipe.py Stage-3 output),
-finds the node by rq-id, validates it is autoresearch-ready (fail-closed), then
-scaffolds runs/<rq-id>/ with:
-
-  program.md  — hypothesis, single metric (stripped), agent search-space seam
-  run.json    — provenance (strategic_question + council_node) + config
-  train.py    — copy of project train.py (the loop edits this, keeps history clean)
-
-Launch:
-  LITELLM_KEY=xxx python loop.py --run-dir runs/<rq-id>
-
-Refs: jepa-fx-risk#11, agentsquad#44
-"""
-
-import json
-import shutil
-import sys
-from datetime import datetime, timezone
-from pathlib import Path
-
-
-def load_backlog(path: str) -> dict:
-    try:
-        with open(path) as f:
-            return json.load(f)
-    except FileNotFoundError:
-        print(f"error: backlog file not found: {path}", file=sys.stderr)
-        raise
-
-
-def scaffold_run(
-    backlog_path_or_dict,
-    rq_id: str,
-    run_dir: Path,
-    train_py_src: Path,
-) -> None:
-    """Scaffold a run dir. Raises SystemExit on any validation failure."""
-    if isinstance(backlog_path_or_dict, (str, Path)):
-        backlog = load_backlog(str(backlog_path_or_dict))
-    else:
-        backlog = backlog_path_or_dict
-
-    # Find node
-    nodes_by_id = {n["id"]: n for n in backlog.get("nodes", [])}
-    if rq_id not in nodes_by_id:
-        print(f"error: rq-id {rq_id!r} not found in backlog", file=sys.stderr)
-        sys.exit(1)
-    node = nodes_by_id[rq_id]
-
-    # Fail-closed: only autoresearch-ready nodes may be scaffolded
-    status = node.get("status", "")
-    if status != "autoresearch-ready":
-        print(
-            f"error: {rq_id} has status {status!r}, not 'autoresearch-ready' — refusing to scaffold",
-            file=sys.stderr,
-        )
-        sys.exit(1)
-
-    # Guard against overwriting an existing run
-    if run_dir.exists():
-        print(
-            f"error: {run_dir} already exists — remove it first to re-scaffold",
-            file=sys.stderr,
-        )
-        sys.exit(1)
-
-    metric = (node.get("candidate_metric") or "").strip()
-    strategic_q = backlog.get("strategic_question", "")
-    council_node = node["id"]
-    generated_at = datetime.now(timezone.utc).isoformat()
-
-    run_dir.mkdir(parents=True)
-
-    # --- program.md ---
-    program_md = f"""# program.md — {council_node}: {node.get("question", "")[:80]}
-
-## Provenance
-- strategic_question: {json.dumps(strategic_q)}
-- council_node: {council_node} (autoresearch-ready; Autoresearch Council backlog)
-- generated_at: {generated_at}
-
-## Hypothesis
-{node.get("question", "")}
-
-## Single validation metric (optimise this, nothing else)
-`{metric}` — see eval harness for the exact definition. Only this scalar drives
-keep/revert decisions. Report alongside but do NOT optimise:
-  - Kupiec POF p-value (calibration sanity)
-  - val_vol_r2 (representation quality guard)
-
-## What the agent MAY modify (the search space)
-- Hyperparameters in train.py (model size, LR, window, patch_len, epochs, etc.)
-- Conditioning mechanisms (e.g. JEPA_ENABLE_REGIME toggle)
-- Loss function weights and architecture depth
-
-## Frozen (do NOT touch — keeps the ablation clean)
-- Data pipeline and splits (train ≤2021, OOS ≥2022, test 2024 held out)
-- The metric definition and scoring code
-- loop.py, scripts/, tests/
-
-## Experiment loop (per Karpathy autoresearch)
-Each iter (≤ time-box): apply ONE change to train.py → run → read
-`{metric}` → keep if improved (and Kupiec p-value did not collapse), else revert.
-Stop on: target reached, max iters, or K consecutive iters with no improvement.
-"""
-    (run_dir / "program.md").write_text(program_md)
-
-    # --- run.json (provenance + config) ---
-    run_meta = {
-        "strategic_question": strategic_q,
-        "council_node": council_node,
-        "metric": metric,
-        "generated_at": generated_at,
-        "model_tier": "homelab",
-        "max_iters": 10,
-        "time_box_minutes": 5,
-    }
-    (run_dir / "run.json").write_text(json.dumps(run_meta, indent=2) + "\n")
-
-    # --- train.py (loop edits this copy; project root train.py is the template) ---
-    shutil.copy(train_py_src, run_dir / "train.py")
-
-
-def main() -> None:
-    if len(sys.argv) != 3:
-        print("usage: python scripts/autoresearch_start.py <backlog.json> <rq-id>")
-        sys.exit(1)
-
-    backlog_path, rq_id = sys.argv[1], sys.argv[2]
-
-    project_root = Path(__file__).parent.parent
-    run_dir = project_root / "runs" / rq_id
-    train_py_src = project_root / "train.py"
-
-    scaffold_run(backlog_path, rq_id, run_dir, train_py_src)
-
-    backlog = load_backlog(backlog_path)
-    nodes_by_id = {n["id"]: n for n in backlog.get("nodes", [])}
-    metric = (nodes_by_id[rq_id].get("candidate_metric") or "").strip()
-
-    print(f"✓ scaffolded {run_dir}")
-    print(f"  node:   {rq_id}")
-    print(f"  metric: {metric}")
-    print()
-    print("launch:")
-    print(f"  LITELLM_KEY=xxx python loop.py --run-dir runs/{rq_id}")
-
-
-if __name__ == "__main__":
-    main()

scripts/check_gpu.py

@@ -1,21 +0,0 @@
-"""Phase-0 compute gate (brain wiki/jepa-fx/facts/autoresearch-integration-phase1):
-PyTorch cu130 must see the koala Blackwell GPU and compute before any experiment.
-
-    python scripts/check_gpu.py   # exits 0 if the GPU is usable, 1 otherwise
-
-Note: koala shares this 12GB card with the llama-swap LLM stack. The autoresearch
-agent should run on iguana/berget models so koala's GPU stays free for train.py.
-"""
-import sys
-import torch
-
-print("torch", torch.__version__)
-if not torch.cuda.is_available():
-    print("CUDA NOT AVAILABLE — gate BLOCKED")
-    sys.exit(1)
-print("device:", torch.cuda.get_device_name(0))
-print("capability: sm_%d%d" % torch.cuda.get_device_capability(0))
-x = torch.randn(2000, 2000, device="cuda")
-(x @ x).sum().item()
-torch.cuda.synchronize()
-print("GPU matmul OK — Phase-0 compute gate GREEN")

scripts/fetch_data.py

@@ -1,31 +0,0 @@
-"""Fetch EUR/USD M1 bars from histdata.com (free, research use).
-
-Polite: one request per year, spaced; past years query month=None. Uses the
-maintained `histdata` package which handles histdata's anti-hotlink tk token.
-Output: data/raw/DAT_ASCII_EURUSD_M1_<year>.zip
-
-    YEARS=2019,2020,2021 python scripts/fetch_data.py
-"""
-import os
-import time
-
-from histdata import download_hist_data
-from histdata.api import Platform as P, TimeFrame as T
-
-YEARS = [y.strip() for y in os.environ.get("YEARS", "2019,2020,2021").split(",")]
-
-
-def main():
-    os.makedirs("data/raw", exist_ok=True)
-    for yr in YEARS:
-        f = download_hist_data(
-            year=yr, month=None, pair="eurusd",
-            platform=P.GENERIC_ASCII, time_frame=T.ONE_MINUTE,
-            output_directory="data/raw",
-        )
-        print("fetched", yr, "->", f)
-        time.sleep(2)  # be a good citizen
-
-
-if __name__ == "__main__":
-    main()

scripts/fetch_multipair.py

@@ -1,48 +0,0 @@
-"""Fetch G10 FX M1 data from histdata.com for all pairs except EURUSD (already fetched).
-
-Each pair's zips go into data/raw/{pair}/ to avoid collisions.
-Output: data/raw/gbpusd/DAT_ASCII_GBPUSD_M1_YYYY.zip etc.
-
-    python scripts/fetch_multipair.py
-    PAIRS=gbpusd,usdjpy YEARS=2020,2021 python scripts/fetch_multipair.py
-"""
-import os
-import time
-
-from histdata import download_hist_data
-from histdata.api import Platform as P, TimeFrame as T
-
-PAIRS_DEFAULT = ["gbpusd", "usdjpy", "usdchf", "audusd"]
-YEARS_DEFAULT = list(range(2008, 2024))
-
-
-def main():
-    pairs_env = os.environ.get("PAIRS", "")
-    pairs = [p.strip() for p in pairs_env.split(",")] if pairs_env else PAIRS_DEFAULT
-
-    years_env = os.environ.get("YEARS", "")
-    years = [int(y.strip()) for y in years_env.split(",")] if years_env else YEARS_DEFAULT
-
-    for pair in pairs:
-        out_dir = f"data/raw/{pair}"
-        os.makedirs(out_dir, exist_ok=True)
-        print(f"\n=== {pair.upper()} ===")
-        for yr in years:
-            out_path = os.path.join(out_dir, f"DAT_ASCII_{pair.upper()}_M1_{yr}.zip")
-            if os.path.exists(out_path):
-                print(f"  {yr} already present, skip")
-                continue
-            try:
-                f = download_hist_data(
-                    year=str(yr), month=None, pair=pair,
-                    platform=P.GENERIC_ASCII, time_frame=T.ONE_MINUTE,
-                    output_directory=out_dir,
-                )
-                print(f"  fetched {yr} → {f}")
-            except Exception as e:
-                print(f"  {yr} FAILED: {e}")
-            time.sleep(2)
-
-
-if __name__ == "__main__":
-    main()

scripts/hpo_sweep.py

@@ -1,97 +0,0 @@
-"""HPO sweep for jepa-fx-risk HEPA backbone.
-
-Runs train.py with different JEPA_* env overrides, logs results to
-results/hpo/hpo_results.jsonl. Each config writes its metrics.json then
-the result is appended to the JSONL.
-
-Usage:
-    python scripts/hpo_sweep.py
-    python scripts/hpo_sweep.py --dry-run   # print configs, don't train
-"""
-import argparse
-import json
-import os
-import subprocess
-import sys
-from datetime import datetime
-from itertools import product
-from pathlib import Path
-
-# ── Search space ──────────────────────────────────────────────────────────────
-SEARCH_SPACE = {
-    "JEPA_D_MODEL": [64, 128, 256],
-    "JEPA_DEPTH":   [2, 4],
-    "JEPA_WINDOW":  [120, 240, 480],
-}
-# Fixed: PATCH_LEN=24 (1-day patches), N_HEADS=4, EPOCHS=300, PHASE1_EPOCHS=200
-
-PYTHON = str(Path(sys.executable))
-OUT_DIR = Path("results/hpo")
-
-
-def configs():
-    """Yield all configs as dicts of JEPA_* env overrides."""
-    keys = list(SEARCH_SPACE.keys())
-    for vals in product(*SEARCH_SPACE.values()):
-        yield dict(zip(keys, vals))
-
-
-def run_config(cfg: dict, metrics_path: str = "metrics.json") -> dict:
-    env = {**os.environ, **{k: str(v) for k, v in cfg.items()}}
-    result = subprocess.run(
-        [PYTHON, "train.py"],
-        env=env,
-        capture_output=True,
-        text=True,
-    )
-    if result.returncode != 0:
-        return {"config": cfg, "error": result.stderr[-500:]}
-    stdout_last = result.stdout.strip().split("\n")[-1]
-    with open(metrics_path) as f:
-        m = json.load(f)
-    return {
-        "config": cfg,
-        "val_vol_r2":  m.get("val_vol_r2"),
-        "phase1_r2":   m.get("phase1_r2"),
-        "stdout_last": stdout_last,
-    }
-
-
-def main():
-    parser = argparse.ArgumentParser()
-    parser.add_argument("--dry-run", action="store_true")
-    args = parser.parse_args()
-
-    OUT_DIR.mkdir(parents=True, exist_ok=True)
-    out_file = OUT_DIR / "hpo_results.jsonl"
-
-    all_cfgs = list(configs())
-    print(f"HPO sweep: {len(all_cfgs)} configs")
-    for i, cfg in enumerate(all_cfgs):
-        label = " ".join(f"{k.replace('JEPA_','')}={v}" for k, v in cfg.items())
-        print(f"\n[{i+1}/{len(all_cfgs)}] {label}")
-        if args.dry_run:
-            continue
-        ts = datetime.utcnow().isoformat()
-        row = run_config(cfg)
-        row["ts"] = ts
-        with open(out_file, "a") as f:
-            f.write(json.dumps(row) + "\n")
-        if "error" in row:
-            print(f"  ERROR: {row['error'][:200]}")
-        else:
-            print(f"  val_vol_r2={row['val_vol_r2']:.4f}  phase1_r2={row['phase1_r2']:.4f}")
-
-    if not args.dry_run:
-        # Print leaderboard
-        rows = [json.loads(l) for l in open(out_file) if l.strip()]
-        rows = [r for r in rows if "error" not in r]
-        rows.sort(key=lambda r: r.get("phase1_r2", -999), reverse=True)
-        print("\n── Leaderboard (by phase1_r2) ─────────────────────────")
-        for r in rows[:5]:
-            cfg_str = " ".join(f"{k.replace('JEPA_','')}={v}" for k,v in r["config"].items())
-            print(f"  {r['phase1_r2']:.4f}  {cfg_str}")
-
-
-if __name__ == "__main__":
-    main()

scripts/phase0_gate.py

@@ -1,237 +0,0 @@
-"""Phase-0 SSL feasibility gate (path B — Python fast-close of #5).
-
-Spec deviation documented: original spec (#5) required hourly 2008-2022 data
-and a Go eval harness (#4). Path B uses daily 2019-2023 + Python harness to
-close the gate quickly, since val_vol_r2 > 0 already demonstrates SSL
-feasibility. The Go harness (#4) remains open for production experiments.
-
-Gate criteria (from #5):
-  - Silhouette > 0.20 on held-out 2022-2023 (binary HV label: top-33% RV days)
-  - TS-JEPA silhouette > PCA baseline silhouette
-  - Rerun x3 seeds within ±10% of mean silhouette
-  - PC1/HV correlation < 0.95 (sanity: not trivially memorising the label)
-
-    python scripts/phase0_gate.py
-"""
-import json
-import math
-import os
-import numpy as np
-import pandas as pd
-import torch
-import torch.nn as nn
-from sklearn.decomposition import PCA
-from sklearn.metrics import silhouette_score
-from sklearn.preprocessing import StandardScaler
-
-SEEDS     = [0, 1, 2]
-WINDOW    = 30
-PATCH_LEN = 5
-STRIDE    = 5
-D_MODEL   = 32
-DEPTH     = 2
-N_HEADS   = 4
-EPOCHS    = 400
-LR        = 3e-4
-SIGREG_LAM = 0.5
-HV_PERCENTILE = 67   # top-33% = "high volatility"
-
-dev = "cuda" if torch.cuda.is_available() else "cpu"
-
-
-# ── SIGReg ──────────────────────────────────────────────────────────────────
-
-def sigreg(tokens: torch.Tensor, knots: int = 17) -> torch.Tensor:
-    B, T, D = tokens.shape
-    z = tokens.reshape(B * T, D).float()
-    t = torch.linspace(0, 3, knots, device=z.device, dtype=z.dtype)
-    dt = 3.0 / (knots - 1)
-    w = torch.full((knots,), 2 * dt, device=z.device, dtype=z.dtype)
-    w[0] = dt; w[-1] = dt
-    phi = torch.exp(-t.square() / 2.0)
-    A = torch.randn(D, 256, device=z.device, dtype=z.dtype)
-    A = A / A.norm(p=2, dim=0)
-    x_t = (z @ A).unsqueeze(-1) * t
-    err = (x_t.cos().mean(0) - phi).square() + x_t.sin().mean(0).square()
-    return ((err @ (w * phi)) * z.shape[0]).mean()
-
-
-# ── Encoder ──────────────────────────────────────────────────────────────────
-
-class PatchEncoder(nn.Module):
-    def __init__(self, in_feats, patch_len, stride, d_model, depth, n_heads):
-        super().__init__()
-        self.patch_len = patch_len
-        self.stride    = stride
-        self.embed = nn.Linear(patch_len * in_feats, d_model)
-        layer = nn.TransformerEncoderLayer(d_model, n_heads, 2 * d_model,
-                                           dropout=0.0, batch_first=True)
-        self.tf = nn.TransformerEncoder(layer, num_layers=depth)
-        n_patches = (WINDOW - patch_len) // stride + 1
-        pos = torch.zeros(n_patches, d_model)
-        for p in range(n_patches):
-            for i in range(0, d_model, 2):
-                pos[p, i]     = math.sin(p / 10000 ** (i / d_model))
-                if i + 1 < d_model:
-                    pos[p, i+1] = math.cos(p / 10000 ** (i / d_model))
-        self.register_buffer("pos", pos)
-
-    def forward(self, x):
-        B, W, F = x.shape
-        n_patches = (W - self.patch_len) // self.stride + 1
-        patches = torch.stack([x[:, i*self.stride:i*self.stride+self.patch_len, :]
-                                .reshape(B, -1) for i in range(n_patches)], dim=1)
-        tokens = self.embed(patches) + self.pos[:n_patches]
-        return self.tf(tokens)
-
-
-# ── Data ─────────────────────────────────────────────────────────────────────
-
-def load_data():
-    df = pd.read_parquet("data/processed/eurusd_daily.parquet").reset_index(drop=True)
-    df["date"] = pd.to_datetime(df["date"])
-    train = df[df["date"].dt.year <= 2021].copy()
-    oos   = df[df["date"].dt.year >= 2022].copy()
-
-    feats_all  = df[["ret", "realized_vol"]].to_numpy(np.float32)
-    target_all = df["realized_vol"].to_numpy(np.float32)
-    dates_all  = df["date"].values
-
-    mu = feats_all[:len(train)].mean(0)
-    sd = feats_all[:len(train)].std(0) + 1e-8
-
-    def windows(df_subset, feats_norm, dates):
-        idx_start = df.index[df["date"].isin(df_subset["date"])][0]
-        X, oos_dates, oos_rv = [], [], []
-        for t in range(idx_start + WINDOW, idx_start + len(df_subset)):
-            X.append(feats_norm[t - WINDOW:t])
-            oos_dates.append(dates[t])
-            oos_rv.append(target_all[t])
-        return np.stack(X), np.array(oos_rv), np.array(oos_dates)
-
-    feats_norm = (feats_all - mu) / sd
-
-    Xtr, rvtr, _ = windows(train, feats_norm, dates_all)
-    Xte, rvte, te_dates = windows(oos, feats_norm, dates_all)
-
-    # binary HV label: top-33% realized vol days in OOS = "high volatility"
-    hv_threshold = np.percentile(rvte, HV_PERCENTILE)
-    hv_labels = (rvte >= hv_threshold).astype(int)
-
-    return Xtr, rvtr, Xte, rvte, hv_labels
-
-
-# ── Train + embed ─────────────────────────────────────────────────────────────
-
-def train_and_embed(Xtr, Xte, seed):
-    torch.manual_seed(seed)
-    np.random.seed(seed)
-    enc  = PatchEncoder(Xtr.shape[2], PATCH_LEN, STRIDE, D_MODEL, DEPTH, N_HEADS).to(dev)
-    pred = nn.Sequential(nn.Linear(D_MODEL, D_MODEL), nn.GELU(),
-                         nn.Linear(D_MODEL, D_MODEL)).to(dev)
-    opt  = torch.optim.AdamW(list(enc.parameters()) + list(pred.parameters()), lr=LR)
-
-    Xtr_t = torch.tensor(Xtr, device=dev)
-    n_patches = (WINDOW - PATCH_LEN) // STRIDE + 1
-    n_mask = max(1, int(0.30 * n_patches))
-
-    for ep in range(EPOCHS):
-        idx_mask = torch.randperm(n_patches)[:n_mask]
-        tokens_ctx    = enc(Xtr_t)
-        tokens_target = enc(Xtr_t).detach()
-        jepa_loss = ((pred(tokens_ctx[:, idx_mask, :]) -
-                      tokens_target[:, idx_mask, :]) ** 2).mean()
-        reg = sigreg(tokens_ctx)
-        loss = jepa_loss + SIGREG_LAM * reg
-        opt.zero_grad(); loss.backward(); opt.step()
-
-    enc.eval()
-    with torch.no_grad():
-        Ete = enc(torch.tensor(Xte, device=dev)).mean(1).cpu().numpy()
-
-    return Ete
-
-
-# ── Gate ─────────────────────────────────────────────────────────────────────
-
-def pca_baseline(Xte, hv_labels):
-    flat = Xte.reshape(len(Xte), -1)
-    sc = StandardScaler().fit(flat)
-    emb = PCA(n_components=8).fit_transform(sc.transform(flat))
-    return silhouette_score(emb, hv_labels), emb
-
-
-def main():
-    os.makedirs("results/summaries", exist_ok=True)
-    Xtr, rvtr, Xte, rvte, hv_labels = load_data()
-    print(f"train={len(Xtr)} OOS={len(Xte)} HV={hv_labels.sum()}/{len(hv_labels)}")
-
-    pca_sil, pca_emb = pca_baseline(Xte, hv_labels)
-    pc1 = pca_emb[:, 0]
-    pc1_hv_corr = abs(np.corrcoef(pc1, hv_labels)[0, 1])
-    print(f"PCA baseline silhouette = {pca_sil:.4f} | PC1/HV |r| = {pc1_hv_corr:.4f}")
-
-    sils = []
-    for seed in SEEDS:
-        emb = train_and_embed(Xtr, Xte, seed)
-        sc  = StandardScaler().fit(emb)
-        sil = silhouette_score(sc.transform(emb), hv_labels)
-        sils.append(sil)
-        print(f"  seed={seed}  silhouette={sil:.4f}")
-
-    mean_sil = np.mean(sils)
-    spread   = (max(sils) - min(sils)) / mean_sil if mean_sil != 0 else 99
-
-    # gate checks
-    g_sil    = mean_sil > 0.20
-    g_beats  = mean_sil > pca_sil
-    g_stable = spread < 0.10
-    g_corr   = pc1_hv_corr < 0.95
-    passed   = all([g_sil, g_beats, g_stable, g_corr])
-
-    label = "pass" if passed else "null"
-    print(f"\nsilhouette mean={mean_sil:.4f} spread={spread:.2%} PCA={pca_sil:.4f} PC1/HV={pc1_hv_corr:.4f}")
-    print(f"gate: sil>0.20={g_sil} beats_pca={g_beats} stable={g_stable} corr<0.95={g_corr}")
-    print(f"PHASE-0: {label.upper()}")
-
-    summary = f"""# Phase-0 SSL feasibility gate — {label}
-
-**Date:** 2026-06-24
-**Path B deviation:** Daily 2019-2023 (not hourly 2008-2022); Python harness
-(not Go #4); gate metric adapted from silhouette-on-embedding to match
-available data. Go harness (#4) remains open for production experiments.
-
-## Data
-- Train: EUR/USD daily 2019-2021 ({len(Xtr)} windows)
-- OOS: EUR/USD daily 2022-2023 ({len(Xte)} windows)
-- HV label: top-{100-HV_PERCENTILE}% realized-vol days = high-volatility ({hv_labels.sum()} days)
-
-## Results
-| | Value | Gate |
-|---|---|---|
-| TS-JEPA mean silhouette (3 seeds) | {mean_sil:.4f} | > 0.20 → **{g_sil}** |
-| Beats PCA baseline ({pca_sil:.4f}) | {mean_sil:.4f} | > PCA → **{g_beats}** |
-| Seed stability (spread) | {spread:.2%} | < 10% → **{g_stable}** |
-| PC1/HV correlation | {pc1_hv_corr:.4f} | < 0.95 → **{g_corr}** |
-
-Per-seed: {[f"{s:.4f}" for s in sils]}
-
-## Verdict: **{label.upper()}**
-
-{"All 4 gate criteria met. TS-JEPA embeddings separate HV regimes significantly above PCA baseline with stable reproducibility." if passed else "One or more gate criteria not met. See null result protocol in #5."}
-"""
-    path = f"results/summaries/phase-0-{label}.md"
-    with open(path, "w") as f:
-        f.write(summary)
-    print(f"Written: {path}")
-
-    result = {"label": label, "mean_sil": mean_sil, "pca_sil": pca_sil,
-              "spread": spread, "pc1_hv_corr": pc1_hv_corr,
-              "per_seed": sils, "passed": passed}
-    with open("results/summaries/phase-0-metrics.json", "w") as f:
-        json.dump(result, f, indent=2)
-    return 0 if passed else 1
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

scripts/prepare_data.py

@@ -1,57 +0,0 @@
-"""LOCKED data pipeline (toy) — agent must NOT edit (brain Phase-1 contract).
-
-Parses histdata EUR/USD M1 zips → daily series with realized volatility (the
-val_vol_r2 target = 1-day realized vol from intraday squared returns).
-Output: data/processed/eurusd_daily.parquet  [date, close, ret, realized_vol].
-"""
-import glob
-import os
-import zipfile
-
-import numpy as np
-import pandas as pd
-
-RAW = "data/raw"
-OUT = "data/processed/eurusd_daily.parquet"
-
-
-def load_m1() -> pd.DataFrame:
-    frames = []
-    for zp in sorted(glob.glob(os.path.join(RAW, "DAT_ASCII_EURUSD_M1_*.zip"))):
-        with zipfile.ZipFile(zp) as z:
-            csv = [n for n in z.namelist() if n.endswith(".csv")][0]
-            with z.open(csv) as f:
-                df = pd.read_csv(
-                    f, sep=";", header=None,
-                    names=["dt", "open", "high", "low", "close", "vol"],
-                )
-        df["ts"] = pd.to_datetime(df["dt"], format="%Y%m%d %H%M%S")
-        frames.append(df[["ts", "close"]])
-    out = pd.concat(frames).sort_values("ts").reset_index(drop=True)
-    return out
-
-
-def main():
-    m1 = load_m1()
-    m1["r"] = np.log(m1["close"]).diff()
-    m1["day"] = m1["ts"].dt.normalize()
-    daily = m1.groupby("day").agg(
-        close=("close", "last"),
-        realized_vol=("r", lambda x: np.sqrt(np.nansum(x.values ** 2))),
-        n_min=("r", "count"),
-    ).reset_index()
-    daily = daily[daily["n_min"] > 60]  # drop thin days (holidays)
-    daily["ret"] = np.log(daily["close"]).diff()
-    daily = daily.dropna().reset_index(drop=True)
-    os.makedirs(os.path.dirname(OUT), exist_ok=True)
-    daily[["day", "close", "ret", "realized_vol"]].rename(columns={"day": "date"}).to_parquet(OUT)
-    print("rows:", len(daily), "| dates:", daily["day"].min().date(), "→", daily["day"].max().date())
-    # sanity: the COVID crash (March 2020) must show a realized-vol spike
-    rv = daily.set_index("day")["realized_vol"]
-    mar20 = rv["2020-03-01":"2020-03-31"].max()
-    typ = rv["2019-01-01":"2019-12-31"].median()
-    print("median 2019 RV: %.5f | max Mar-2020 RV: %.5f | spike x%.1f" % (typ, mar20, mar20 / typ))
-
-
-if __name__ == "__main__":
-    main()

scripts/prepare_hourly.py

@@ -1,128 +0,0 @@
-"""Prepare EUR/USD hourly OHLCV + realized vol from histdata M1 zips.
-
-Aggregates all M1 bars in data/raw/DAT_ASCII_EURUSD_M1_*.zip to hourly.
-Realized vol per hour = sqrt(sum(log-return²)) over the constituent M1 bars.
-Weekend hours are naturally absent (FX market closed Sat/Sun); NO interpolation.
-Hours with fewer than MIN_BARS M1 bars are dropped (holidays, thin sessions).
-
-Output: data/processed/eurusd_hourly.parquet
-Columns: datetime (UTC, tz-naive), close, ret (log), realized_vol
-
-    python scripts/prepare_hourly.py
-    RAW=data/raw OUT=data/processed/eurusd_hourly.parquet python scripts/prepare_hourly.py
-"""
-import glob
-import os
-import zipfile
-
-import numpy as np
-import pandas as pd
-
-PAIR        = os.environ.get("PAIR", "EURUSD").upper()
-RAW_DEFAULT = "data/raw"
-OUT_DEFAULT = f"data/processed/{PAIR.lower()}_hourly.parquet"
-MIN_BARS    = 30   # drop hours thinner than this (holidays, DST boundary artefacts)
-
-
-# ── Core transformation ──────────────────────────────────────────────────────
-
-def resample_to_hourly(m1: pd.DataFrame) -> pd.DataFrame:
-    """Aggregate M1 DataFrame to hourly bars.
-
-    Args:
-        m1: DataFrame with columns ['ts', 'open', 'high', 'low', 'close']
-            ('open'/'high'/'low' optional — omit for close-only data).
-
-    Returns:
-        DataFrame with columns ['datetime', 'close', 'ret', 'realized_vol',
-        'hl_range', 'ret_intrabar'] sorted by datetime.
-        Hours with fewer than MIN_BARS M1 ticks are dropped.
-    """
-    m1 = m1.sort_values("ts").copy()
-    m1["log_r"] = np.log(m1["close"]).diff()
-    m1["hour"]  = m1["ts"].dt.floor("h")
-
-    has_ohlc = all(c in m1.columns for c in ("open", "high", "low"))
-
-    agg_dict = dict(
-        close        = ("close", "last"),
-        realized_vol = ("log_r", lambda x: np.sqrt(np.nansum(x.values ** 2))),
-        n_bars       = ("log_r", "count"),
-    )
-    if has_ohlc:
-        agg_dict["high"]  = ("high",  "max")
-        agg_dict["low"]   = ("low",   "min")
-        agg_dict["open_"] = ("open",  "first")
-
-    agg = m1.groupby("hour").agg(**agg_dict).reset_index()
-
-    agg = agg[agg["n_bars"] >= MIN_BARS].copy()
-    agg["ret"] = np.log(agg["close"]).diff()
-    agg = agg.dropna(subset=["ret"]).reset_index(drop=True)
-    agg = agg.rename(columns={"hour": "datetime"})
-
-    if has_ohlc:
-        agg["hl_range"]    = np.log(agg["high"] / agg["low"])
-        agg["ret_intrabar"]= np.log(agg["close"] / agg["open_"])
-        cols = ["datetime", "close", "ret", "realized_vol", "hl_range", "ret_intrabar"]
-    else:
-        cols = ["datetime", "close", "ret", "realized_vol"]
-
-    return agg[cols]
-
-
-def load_m1_from_zips(raw_dir: str, pair: str = None) -> pd.DataFrame:
-    """Load and concatenate all M1 zips from raw_dir (histdata format)."""
-    p = (pair or PAIR).upper()
-    pattern = os.path.join(raw_dir, f"DAT_ASCII_{p}_M1_*.zip")
-    zips = sorted(glob.glob(pattern))
-    if not zips:
-        raise FileNotFoundError(f"No M1 zips found at {pattern}")
-    frames = []
-    for zp in zips:
-        with zipfile.ZipFile(zp) as z:
-            csv = [n for n in z.namelist() if n.endswith(".csv")][0]
-            with z.open(csv) as f:
-                df = pd.read_csv(
-                    f, sep=";", header=None,
-                    names=["dt", "open", "high", "low", "close", "vol"],
-                )
-        df["ts"] = pd.to_datetime(df["dt"], format="%Y%m%d %H%M%S")
-        frames.append(df[["ts", "open", "high", "low", "close"]])
-        print(f"  loaded {os.path.basename(zp)}: {len(df):,} rows")
-    return pd.concat(frames).sort_values("ts").reset_index(drop=True)
-
-
-def build_hourly_parquet(
-    raw_dir: str = RAW_DEFAULT,
-    out_path: str = OUT_DEFAULT,
-) -> pd.DataFrame:
-    """Full pipeline: load all M1 zips → hourly parquet. Returns the DataFrame."""
-    print(f"Loading M1 zips from {raw_dir}...")
-    m1 = load_m1_from_zips(raw_dir)
-    print(f"Total M1 bars: {len(m1):,}  ({m1['ts'].min().date()} → {m1['ts'].max().date()})")
-
-    print("Resampling to hourly...")
-    hourly = resample_to_hourly(m1)
-    print(f"Hourly rows: {len(hourly):,}  ({hourly['datetime'].min()} → {hourly['datetime'].max()})")
-
-    # Sanity: COVID crash (Mar 2020) should show realized vol spike if data covers it
-    if hourly["datetime"].dt.year.isin([2020]).any():
-        rv = hourly.set_index("datetime")["realized_vol"]
-        try:
-            mar20 = rv["2020-03-01":"2020-03-31"].max()
-            typ   = rv["2019-01-01":"2019-12-31"].median()
-            print(f"Sanity — median 2019 RV: {typ:.6f} | max Mar-2020 RV: {mar20:.6f} | spike ×{mar20/typ:.1f}")
-        except Exception:
-            pass
-
-    os.makedirs(os.path.dirname(os.path.abspath(out_path)), exist_ok=True)
-    hourly.to_parquet(out_path, index=False)
-    print(f"Written: {out_path}")
-    return hourly
-
-
-if __name__ == "__main__":
-    raw_dir  = os.environ.get("RAW",  RAW_DEFAULT)
-    out_path = os.environ.get("OUT",  OUT_DEFAULT)
-    build_hourly_parquet(raw_dir=raw_dir, out_path=out_path)

scripts/prepare_multipair.py

@@ -1,72 +0,0 @@
-"""Merge per-pair hourly parquets into a single wide multipair parquet.
-
-Each pair contributes two features: {pair}_ret and {pair}_rv (realized vol).
-The merge is an INNER JOIN on datetime — only hours present in ALL pairs are kept.
-The target for train.py remains eurusd_rv.
-
-Output: data/processed/eurusd_multipair.parquet
-
-    python scripts/prepare_multipair.py
-    PROCESSED=data/processed python scripts/prepare_multipair.py
-"""
-import os
-import pandas as pd
-
-PAIRS = ["eurusd", "gbpusd", "usdjpy", "usdchf", "audusd"]
-
-PROCESSED_DEFAULT = "data/processed"
-OUT_DEFAULT       = "data/processed/eurusd_multipair.parquet"
-
-
-def merge_pair_parquets(pair_dfs: dict) -> pd.DataFrame:
-    """Inner-join hourly DataFrames from multiple pairs on datetime.
-
-    Args:
-        pair_dfs: dict mapping pair name (e.g. "eurusd") to hourly DataFrame
-                  with columns [datetime, close, ret, realized_vol, ...].
-    Returns:
-        Wide DataFrame with columns:
-          datetime, {pair}_ret, {pair}_rv  for each pair.
-    """
-    merged = None
-    for pair, df in pair_dfs.items():
-        sub = df[["datetime", "ret", "realized_vol"]].copy()
-        sub = sub.rename(columns={"ret": f"{pair}_ret", "realized_vol": f"{pair}_rv"})
-        sub = sub.set_index("datetime")
-        if merged is None:
-            merged = sub
-        else:
-            merged = merged.join(sub, how="inner")
-
-    return merged.reset_index()
-
-
-def build_multipair_parquet(
-    processed_dir: str = PROCESSED_DEFAULT,
-    out_path: str = OUT_DEFAULT,
-    pairs: list = None,
-) -> None:
-    if pairs is None:
-        pairs = PAIRS
-    pair_dfs = {}
-    for pair in pairs:
-        path = os.path.join(processed_dir, f"{pair}_hourly.parquet")
-        if not os.path.exists(path):
-            raise FileNotFoundError(
-                f"{pair}_hourly.parquet not found at {path} — run prepare_hourly.py for this pair first"
-            )
-        df = pd.read_parquet(path)
-        pair_dfs[pair] = df
-
-    merged = merge_pair_parquets(pair_dfs)
-    merged.to_parquet(out_path, index=False)
-    n_pairs = len(pairs)
-    n_ch    = n_pairs * 2
-    print(f"Multipair parquet: {len(merged):,} rows × {n_ch} feature channels ({n_pairs} pairs)")
-    print(f"Date range: {merged['datetime'].min()} → {merged['datetime'].max()}")
-    print(f"Written: {out_path}")
-
-
-if __name__ == "__main__":
-    processed_dir = os.environ.get("PROCESSED", PROCESSED_DEFAULT)
-    build_multipair_parquet(processed_dir=processed_dir)

scripts/prepare_regime.py

@@ -1,134 +0,0 @@
-"""HMM regime detector — 3-state Gaussian HMM on realized_vol.
-
-Fits on the FULL dataset (training + OOS) so the state sequence is globally
-consistent across all periods. States are sorted by mean realized vol (ascending):
-    0 = calm,   1 = stressed,   2 = crisis
-
-Output: data/processed/eurusd_regime.parquet
-Columns: datetime (or date), regime (int: 0/1/2)
-
-Deterministic: fixed random_state=42 throughout.
-Cached: if the parquet already exists, it is not re-computed.
-
-Usage:
-    python scripts/prepare_regime.py [--hourly] [--daily] [--force]
-
-jepa-fx-risk#13
-"""
-
-import argparse
-import os
-from pathlib import Path
-
-import numpy as np
-import pandas as pd
-from hmmlearn import hmm
-
-DATA_DIR = Path(__file__).parent.parent / "data" / "processed"
-HOURLY_PATH = DATA_DIR / "eurusd_hourly.parquet"
-DAILY_PATH = DATA_DIR / "eurusd_daily.parquet"
-OUTPUT_PATH = DATA_DIR / "eurusd_regime.parquet"
-
-N_STATES = 3
-RANDOM_STATE = 42
-
-
-def fit_regime_hmm(realized_vol: np.ndarray, n_states: int = 3, random_state: int = 42) -> np.ndarray:
-    """Fit a Gaussian HMM on realized_vol and return state labels (0=calm → n_states-1=crisis).
-
-    States are sorted by mean realized vol ascending so label 0 is always calm,
-    label n_states-1 is always crisis. This makes the labelling deterministic
-    across datasets with different vol levels.
-
-    Args:
-        realized_vol:  1-D array of realized vol values
-        n_states:      number of HMM hidden states (default 3)
-        random_state:  random seed for reproducibility
-
-    Returns:
-        Integer label array of shape (len(realized_vol),), dtype int64
-    """
-    X = realized_vol.reshape(-1, 1).astype(np.float64)
-    model = hmm.GaussianHMM(
-        n_components=n_states,
-        covariance_type="diag",
-        min_covar=1e-6,
-        n_iter=100,
-        random_state=random_state,
-        tol=1e-4,
-    )
-    model.fit(X)
-    raw_labels = model.predict(X)
-
-    # Sort states by mean realized vol (ascending: calm=0, crisis=n_states-1)
-    state_means = np.array([X[raw_labels == s].mean() if (raw_labels == s).any() else 0.0
-                            for s in range(n_states)])
-    rank = np.argsort(state_means)          # rank[0] = original state id of the calmest cluster
-    remap = np.empty(n_states, dtype=np.int64)
-    for new_label, old_label in enumerate(rank):
-        remap[old_label] = new_label
-    return remap[raw_labels].astype(np.int64)
-
-
-def prepare_regime_df(parquet_path: str, freq: str = "hourly") -> pd.DataFrame:
-    """Load parquet, fit HMM, return DataFrame with timestamp + regime columns.
-
-    Args:
-        parquet_path: path to input parquet (hourly or daily)
-        freq:         "hourly" | "daily" — determines timestamp column name
-
-    Returns:
-        DataFrame with columns: (datetime|date), regime
-    """
-    df = pd.read_parquet(parquet_path)
-    if freq == "hourly":
-        ts = pd.to_datetime(df["datetime"])
-    else:
-        ts = pd.to_datetime(df["date"])
-
-    rv = df["realized_vol"].to_numpy(np.float32)
-    labels = fit_regime_hmm(rv, n_states=N_STATES, random_state=RANDOM_STATE)
-    return pd.DataFrame({"datetime": ts.values, "regime": labels})
-
-
-def main():
-    parser = argparse.ArgumentParser(description="Fit HMM regime detector")
-    parser.add_argument("--hourly", action="store_true", default=True,
-                        help="use hourly parquet (default)")
-    parser.add_argument("--daily", action="store_true", default=False,
-                        help="use daily parquet instead of hourly")
-    parser.add_argument("--force", action="store_true", default=False,
-                        help="overwrite existing output")
-    parser.add_argument("--out", default=str(OUTPUT_PATH),
-                        help="output parquet path")
-    args = parser.parse_args()
-
-    out_path = Path(args.out)
-    if out_path.exists() and not args.force:
-        print("regime parquet already exists:", out_path, "(use --force to recompute)")
-        return
-
-    if args.daily and DAILY_PATH.exists():
-        src, freq = str(DAILY_PATH), "daily"
-    elif HOURLY_PATH.exists():
-        src, freq = str(HOURLY_PATH), "hourly"
-    elif DAILY_PATH.exists():
-        src, freq = str(DAILY_PATH), "daily"
-    else:
-        raise FileNotFoundError("no parquet found in data/processed/")
-
-    print(f"fitting HMM ({N_STATES} states) on {src} ...")
-    df = prepare_regime_df(src, freq=freq)
-
-    counts = df["regime"].value_counts().sort_index()
-    print("regime distribution:")
-    for state, count in counts.items():
-        label = {0: "calm", 1: "stressed", 2: "crisis"}.get(state, f"state{state}")
-        print(f"  {state} ({label}): {count} ({100*count/len(df):.1f}%)")
-
-    df.to_parquet(out_path, index=False)
-    print("wrote:", out_path)
-
-
-if __name__ == "__main__":
-    main()

scripts/var_breach.py

@@ -1,67 +0,0 @@
-"""Parametric 99% VaR breach rate + Kupiec POF p-value.
-
-Used by train.py's LOCKED VaR EVAL BLOCK to write VaR_breach_rate_99_oos_regime_cond
-to metrics.json so the autoresearch loop can optimise it.
-
-jepa-fx-risk#12
-"""
-
-import math
-
-# Canonical metric key — no surrounding whitespace, as required by the loop contract.
-METRIC_KEY = "VaR_breach_rate_99_oos_regime_cond"
-
-# Default normal 99th-percentile z-score.
-Z99 = 2.326
-
-
-def var_breach_rate(pred_vol, actual_vol, z99=Z99):
-    """Compute VaR breach rate and Kupiec POF p-value.
-
-    Args:
-        pred_vol:   iterable of predicted conditional vol forecasts
-        actual_vol: iterable of actual realized vol (same length)
-        z99:        99th-percentile z-score (default 2.326)
-
-    Returns:
-        (breach_rate, kupiec_p) where:
-            breach_rate — fraction of steps where actual_vol > pred_vol × z99
-            kupiec_p    — Kupiec POF p-value (H0: true breach rate = 1%)
-                          High p-value = well-calibrated; low = miscalibrated tail.
-    """
-    pred_v = list(pred_vol)
-    act_v = list(actual_vol)
-    n = len(pred_v)
-    if n == 0 or n != len(act_v):
-        return 0.0, 1.0
-
-    n1 = sum(1 for p, a in zip(pred_v, act_v) if a > p * z99)
-    breach_rate = n1 / n
-    p = kupiec_pvalue(n, n1)
-    return breach_rate, p
-
-
-def kupiec_pvalue(n, n1, p0=0.01):
-    """Kupiec Proportion-of-Failures likelihood ratio test.
-
-    H0: true breach probability = p0.
-    Returns P(chi²(1) > LR) using the identity P(chi²(1)>x) = erfc(sqrt(x/2)).
-    Returns 1.0 for n=0 or LR<=0 (well-calibrated / over-conservative).
-    """
-    if n == 0:
-        return 1.0
-    n0 = n - n1
-    phat = n1 / n
-
-    if n1 == 0:
-        # 0 × ln(0/p0) = 0 by convention; only n0 term contributes
-        lr = 2 * n0 * math.log((1 - phat) / (1 - p0))
-    elif n1 == n:
-        lr = 2 * n1 * math.log(phat / p0)
-    else:
-        lr = 2 * (n1 * math.log(phat / p0) + n0 * math.log((1 - phat) / (1 - p0)))
-
-    if lr <= 0:
-        return 1.0
-    # P(chi²(1) > LR) = erfc(sqrt(LR/2))
-    return math.erfc(math.sqrt(lr / 2))

specs/phase-0-ssl-feasibility.md

@@ -0,0 +1,84 @@
+# Experiment Spec: Phase 0 — SSL Feasibility Gate
+
+## Hypothesis
+
+> We believe that a masked autoencoder (MAE) trained on FX hourly time-series will
+> produce latent embeddings that show structural separability by volatility regime
+> without explicit regime labels, measurable by silhouette score > 0.20 on k-means
+> clusters evaluated against a held-out realised-volatility regime label on 2023 data.
+
+This hypothesis is FALSE if silhouette score ≤ 0.20 on the held-out evaluation.
+
+## Background
+
+Before investing in JEPA-specific machinery, we need to confirm that self-supervised
+representation learning can find regime structure in FX time-series at all. If the
+simplest SSL method (MAE) cannot find structure, JEPA will not either — and the root
+cause needs to be understood before proceeding.
+
+Also validates that TS-JEPA code is reproducible: first task is running TS-JEPA on
+the paper's own benchmark, not on FX data. If reproduction takes > 2 weeks, contact
+authors or fall back to implementing JEPA masking from V-JEPA codebase.
+
+Added post Full Grill (2026-05-27). See DECISIONS.md: "Phase 0: SSL feasibility gate."
+
+## Design
+
+### Data
+- **Source:** DUKASCopy, EUR/USD hourly OHLCV
+- **Train:** 2008-01-01 – 2022-12-31
+- **Held-out test:** 2023-01-01 – 2023-12-31 (sealed until final evaluation)
+- **Features:** log-return (hourly), rolling 20-period realised HV, VIX (daily → hourly interpolation)
+- **Regime label (evaluation only):** rolling 30-day HV percentile; binary: top 50% = high-vol, bottom 50% = low-vol
+
+### Model
+- **Architecture:** 1D temporal Masked Autoencoder
+  - Encoder: 3-layer 1D CNN + positional encoding
+  - Decoder: 2-layer MLP reconstructing masked segment
+- **Masking:** contiguous temporal block (target); context window = 120h (5 days)
+- **Loss:** MSE reconstruction on masked segment
+
+### TS-JEPA reproduction task (runs in parallel / before MAE)
+- Reproduce TS-JEPA paper results on the authors' benchmark dataset
+- Go/no-go: if reproduction fails within 2 weeks → contact authors or pivot to V-JEPA adaptation
+
+### Baseline
+- **PCA** on raw feature vectors (same 120h context window, flattened)
+- Tests whether any dimensionality reduction finds regime structure; confirms SSL is adding something
+
+### Ablations
+- Masking horizon K ∈ {8h, 24h, 72h} — does context length affect embedding quality?
+
+## Acceptance Criteria
+
+- [ ] MAE silhouette score > 0.20 on held-out 2023 data (k-means k=3, vs. binary HV regime label)
+- [ ] MAE silhouette exceeds PCA baseline silhouette
+- [ ] Rerun ×3 within ±10% of reported silhouette (reproducibility)
+- [ ] PC1 / rolling-HV correlation < 0.95 (encoder is not purely encoding volatility level)
+- [ ] TS-JEPA reproduced on paper's benchmark within 2 weeks of start
+
+## Out of Scope
+
+- JEPA implementation (Phase 1)
+- Multi-pair training (Phase 1+)
+- VaR, ES, or any risk metric computation
+- Any data after 2022-12-31 (training); 2023 test set opened only for final evaluation
+- Hyperparameter search beyond the three masking horizons defined above
+
+## Null Result Protocol
+
+If MAE silhouette ≤ 0.20 on held-out data:
+1. Conclude: SSL-based regime detection is not straightforwardly feasible on EUR/USD hourly data with these three features
+2. Investigate in order: (a) try daily resolution instead of hourly, (b) expand feature set to 6 features (add DXY, yield spread), (c) try 4-class regime label (HV quartiles) instead of binary
+3. If all three investigations fail: conclude SSL regime detection is not viable on FX data; document and consider pivoting the primary hypothesis to distributional forecasting directly
+4. Record result in `results/summaries/phase-0-null.md`
+5. Write failure mode to brain: `brain_write wing=jepa-fx hall=failures`
+
+## Risks
+
+| Risk | Canary | Mitigation |
+|---|---|---|
+| TS-JEPA code unreproducible | Benchmark result doesn't match paper within 2 weeks | Contact authors; fall back to V-JEPA adaptation |
+| MAE encoder collapses | Reconstruction loss plateau in first 10 epochs; all embeddings near-identical | Add batch normalisation; reduce LR; check masking ratio |
+| Regime label too coarse | Silhouette low even with visually structured embeddings | Also evaluate with 4-class HV quartile label |
+| PC1 is just volatility | PC1/HV correlation > 0.95 | Useful finding; record it; do not declare success |

specs/phase-1-representation-poc.md

@@ -0,0 +1,82 @@
+# Experiment Spec: Phase 1 — JEPA Representation PoC
+
+## Hypothesis
+
+> We believe that a TS-JEPA encoder trained on G10 FX hourly data (2008–2022) will
+> produce latent market-state embeddings that are structurally separable by volatility
+> regime without explicit regime labels, measurable by silhouette score > 0.35 on
+> k-means clusters evaluated against realised-volatility regime labels on held-out
+> 2023 data including at least one structural break.
+
+This hypothesis is FALSE if silhouette score ≤ 0.35 OR linear probe R² ≤ 0.40 on
+held-out evaluation.
+
+**Prerequisites:** Phase 0 passed (MAE silhouette > 0.20, TS-JEPA reproduced).
+
+## Background
+
+Phase 0 confirmed that SSL-based representation learning can find regime structure in
+FX time-series. Phase 1 tests whether JEPA's specific inductive bias (predict target
+embeddings from context embeddings, never reconstruct raw data) produces richer
+representations than a simple MAE — and whether those representations are useful for
+risk management tasks (measurable via linear probe).
+
+## Design
+
+### Data
+- **Source:** DUKASCopy, all G10 pairs (EUR/USD, GBP/USD, USD/JPY, USD/CHF, AUD/USD, USD/CAD, NZD/USD, EUR/GBP, EUR/JPY, EUR/CHF), hourly
+- **Train:** 2008-01-01 – 2022-12-31 (all 10 pairs, jointly)
+- **Held-out test:** 2023-01-01 – 2023-12-31 (sealed until final evaluation)
+- **Features:** log-return, rolling 20-period HV, VIX (daily → hourly)
+- **Regime label (evaluation only):** rolling 30-day HV percentile; binary + 4-class (quartiles)
+
+### Model
+- **Architecture:** TS-JEPA (Ennadir et al., 2025)
+  - Context encoder: maps observed window → latent embedding
+  - Target encoder: EMA of context encoder (momentum β ≈ 0.996); stop-gradient
+  - Predictor: shallow MLP bridging context → target embedding
+- **Masking horizons:** ablate K ∈ {1h, 8h, 24h}; context window = 120h
+- **Training:** multi-pair joint training (one model, all G10 pairs)
+
+### Baseline
+- Phase 0 MAE (best masking horizon from Phase 0)
+- PCA on raw features (Phase 0 baseline)
+
+### Ablations
+1. TS-JEPA vs MAE — is JEPA's no-reconstruction inductive bias adding value?
+2. Single-pair (EUR/USD only) vs. multi-pair — does joint training improve representations?
+3. Masking horizon K: {1h, 8h, 24h}
+
+## Acceptance Criteria
+
+- [ ] Silhouette score > 0.35 on held-out 2023 data (k-means k=3–5, binary HV label)
+- [ ] Linear probe R² > 0.40 on frozen embeddings vs. realised-vol decile
+- [ ] PC1 / rolling-HV correlation < 0.85 (encoder learning more than volatility level)
+- [ ] TS-JEPA silhouette exceeds Phase 0 MAE silhouette by > 5%
+- [ ] Rerun ×3 within ±10% of reported silhouette
+
+## Out of Scope
+
+- VaR, ES, distributional forecasting (Phase 3)
+- Exotic pairs beyond G10
+- Options pricing, alpha generation, live trading
+- Any data after 2022-12-31 for training; test set opened only for final evaluation
+- Regime detection backtesting (Phase 2 — embedding drift as early warning)
+
+## Null Result Protocol
+
+If primary criteria not met:
+1. If silhouette > 0.20 but ≤ 0.35: JEPA shows partial structure; not sufficient for Phase 2. Investigate whether multi-pair training, longer context window, or additional features close the gap. One retry permitted with documented rationale.
+2. If silhouette ≤ 0.20: regression from Phase 0; investigate JEPA training stability (collapse risk). Do not proceed.
+3. If linear probe R² ≤ 0.40 despite good silhouette: embeddings are structured but not encoding risk-relevant information. Record as a finding; reconsider feature set.
+4. Record all results in `results/summaries/phase-1-[pass|null].md`
+5. Write findings to brain: `brain_write wing=jepa-fx hall=failures`
+
+## Risks
+
+| Risk | Canary | Mitigation |
+|---|---|---|
+| EMA encoder collapses | All embeddings converge to near-zero; loss goes to ~0 early | Verify EMA momentum schedule; check stop-gradient implementation |
+| Encoder learns only EUR/USD vol | PC1 dominated by EUR/USD HV even in multi-pair model | Evaluate per-pair silhouette; if EUR/USD dominates, weight loss by pair |
+| Phase 0 silhouette was data-split artefact | Phase 1 MAE baseline doesn't reproduce Phase 0 numbers | Fix random seeds; document split methodology in Phase 0 |
+| Insufficient regime diversity (2008–2022) | Embedding clusters don't separate 2023 structural break | Verify 2023 test includes high-vol episode; add 4-class label as backup |

tests/test_autoresearch_start.py

@@ -1,215 +0,0 @@
-"""Tests for scripts/autoresearch_start.py — jepa-fx-risk#11 Phase A scaffold.
-
-Success criterion: `autoresearch start <backlog.json> <rq-id>` scaffolds a
-runnable run dir from a ready leaf; refuses non-ready nodes; strips
-candidate_metric; records provenance.
-"""
-
-import importlib.util
-import json
-import sys
-from pathlib import Path
-
-import pytest
-
-# Load the module without executing main()
-_SCRIPT = Path(__file__).parent.parent / "scripts" / "autoresearch_start.py"
-
-
-def _import():
-    spec = importlib.util.spec_from_file_location("autoresearch_start", _SCRIPT)
-    mod = importlib.util.module_from_spec(spec)
-    spec.loader.exec_module(mod)
-    return mod
-
-
-@pytest.fixture()
-def mod():
-    return _import()
-
-
-@pytest.fixture()
-def backlog(tmp_path):
-    data = {
-        "strategic_question": "Test strategic question?",
-        "generated_at": "2026-06-27T00:00:00Z",
-        "nodes": [
-            {
-                "id": "rq-01",
-                "question": "Does X improve Y?",
-                "case_type": "autoresearch-loop",
-                "data": "obtainable",
-                "method": "adjacent",
-                "falsifiable": "yes",
-                "candidate_metric": " val_vol_r2",  # leading space — bypass test
-                "depends_on": [],
-                "status": "autoresearch-ready",
-                "track": "autoresearch",
-                "converged": True,
-                "survived_review": True,
-            },
-            {
-                "id": "rq-02",
-                "question": "Not ready yet?",
-                "case_type": "empirical-study",
-                "data": "obtainable",
-                "method": "adjacent",
-                "falsifiable": "yes",
-                "candidate_metric": None,
-                "depends_on": [],
-                "status": "needs-metric",
-                "track": "study",
-                "converged": True,
-                "survived_review": True,
-            },
-            {
-                "id": "rq-03",
-                "question": "A spike.",
-                "case_type": "spike",
-                "data": "have",
-                "method": "yes-named",
-                "falsifiable": "yes",
-                "candidate_metric": None,
-                "depends_on": [],
-                "status": "spike-ready",
-                "track": "spike",
-                "converged": True,
-                "survived_review": True,
-            },
-        ],
-    }
-    p = tmp_path / "backlog.json"
-    p.write_text(json.dumps(data))
-    return p
-
-
-@pytest.fixture()
-def fake_train_py(tmp_path):
-    """Minimal train.py placeholder for scaffold tests."""
-    src = tmp_path / "train_template.py"
-    src.write_text("# train.py placeholder\n")
-    return src
-
-
-# ---------------------------------------------------------------------------
-# fail-closed: refuse non-autoresearch-ready nodes
-# ---------------------------------------------------------------------------
-
-class TestRefuseNonReady:
-    def test_refuses_needs_metric(self, mod, backlog, fake_train_py, tmp_path):
-        run_dir = tmp_path / "runs" / "rq-02"
-        with pytest.raises(SystemExit) as exc:
-            mod.scaffold_run(backlog, "rq-02", run_dir, fake_train_py)
-        assert exc.value.code != 0
-
-    def test_refuses_spike_ready(self, mod, backlog, fake_train_py, tmp_path):
-        run_dir = tmp_path / "runs" / "rq-03"
-        with pytest.raises(SystemExit) as exc:
-            mod.scaffold_run(backlog, "rq-03", run_dir, fake_train_py)
-        assert exc.value.code != 0
-
-    def test_refuses_missing_rq_id(self, mod, backlog, fake_train_py, tmp_path):
-        run_dir = tmp_path / "runs" / "rq-99"
-        with pytest.raises(SystemExit) as exc:
-            mod.scaffold_run(backlog, "rq-99", run_dir, fake_train_py)
-        assert exc.value.code != 0
-
-    def test_refuses_existing_run_dir(self, mod, backlog, fake_train_py, tmp_path):
-        run_dir = tmp_path / "runs" / "rq-01"
-        run_dir.mkdir(parents=True)
-        with pytest.raises(SystemExit) as exc:
-            mod.scaffold_run(backlog, "rq-01", run_dir, fake_train_py)
-        assert exc.value.code != 0
-
-
-# ---------------------------------------------------------------------------
-# scaffold structure: correct files created
-# ---------------------------------------------------------------------------
-
-class TestScaffoldStructure:
-    @pytest.fixture(autouse=True)
-    def _scaffold(self, mod, backlog, fake_train_py, tmp_path):
-        self.run_dir = tmp_path / "runs" / "rq-01"
-        mod.scaffold_run(backlog, "rq-01", self.run_dir, fake_train_py)
-
-    def test_run_dir_created(self):
-        assert self.run_dir.is_dir()
-
-    def test_program_md_created(self):
-        assert (self.run_dir / "program.md").exists()
-
-    def test_run_json_created(self):
-        assert (self.run_dir / "run.json").exists()
-
-    def test_train_py_copied(self):
-        assert (self.run_dir / "train.py").exists()
-        assert (self.run_dir / "train.py").read_text() == "# train.py placeholder\n"
-
-
-# ---------------------------------------------------------------------------
-# program.md content
-# ---------------------------------------------------------------------------
-
-class TestProgramMd:
-    @pytest.fixture(autouse=True)
-    def _scaffold(self, mod, backlog, fake_train_py, tmp_path):
-        self.run_dir = tmp_path / "runs" / "rq-01"
-        mod.scaffold_run(backlog, "rq-01", self.run_dir, fake_train_py)
-        self.content = (self.run_dir / "program.md").read_text()
-
-    def test_contains_hypothesis(self):
-        assert "Does X improve Y?" in self.content
-
-    def test_metric_key_stripped(self):
-        # candidate_metric had leading space " val_vol_r2" — must be stripped
-        assert "`val_vol_r2`" in self.content
-        assert "` val_vol_r2`" not in self.content
-
-    def test_contains_strategic_question(self):
-        assert "Test strategic question?" in self.content
-
-    def test_contains_council_node(self):
-        assert "rq-01" in self.content
-
-
-# ---------------------------------------------------------------------------
-# run.json provenance
-# ---------------------------------------------------------------------------
-
-class TestRunJson:
-    @pytest.fixture(autouse=True)
-    def _scaffold(self, mod, backlog, fake_train_py, tmp_path):
-        self.run_dir = tmp_path / "runs" / "rq-01"
-        mod.scaffold_run(backlog, "rq-01", self.run_dir, fake_train_py)
-        self.run = json.loads((self.run_dir / "run.json").read_text())
-
-    def test_strategic_question_in_provenance(self):
-        assert self.run["strategic_question"] == "Test strategic question?"
-
-    def test_council_node_in_provenance(self):
-        assert self.run["council_node"] == "rq-01"
-
-    def test_metric_stripped_in_provenance(self):
-        assert self.run["metric"] == "val_vol_r2"
-        assert self.run["metric"] == self.run["metric"].strip()
-
-    def test_generated_at_present(self):
-        assert "generated_at" in self.run
-
-    def test_max_iters_present(self):
-        assert "max_iters" in self.run
-
-
-# ---------------------------------------------------------------------------
-# load_backlog helper
-# ---------------------------------------------------------------------------
-
-class TestLoadBacklog:
-    def test_loads_json(self, mod, backlog):
-        data = mod.load_backlog(str(backlog))
-        assert data["strategic_question"] == "Test strategic question?"
-        assert len(data["nodes"]) == 3
-
-    def test_missing_file_raises(self, mod, tmp_path):
-        with pytest.raises((FileNotFoundError, SystemExit)):
-            mod.load_backlog(str(tmp_path / "nonexistent.json"))

tests/test_hepa.py

@@ -1,277 +0,0 @@
-"""Failing tests for HEPA backbone + Phase-1 supervised head + HPO in train.py.
-
-Run: cd ~/dev/AI/jepa-fx-risk && .venv/bin/python -m pytest tests/test_hepa.py -v
-These tests define what the backbone and head must satisfy BEFORE implementation.
-"""
-import math
-import os
-import torch
-import torch.nn as nn
-import pytest
-
-# ── Tests import the classes from train.py ────────────────────────────────────
-# They will fail until train.py implements: CausalEncoder, HorizonPredictor, vicreg_loss
-
-
-def _import(env_overrides=None):
-    import importlib.util, sys
-    saved = {}
-    if env_overrides:
-        for k, v in env_overrides.items():
-            saved[k] = os.environ.get(k)
-            os.environ[k] = str(v)
-    # Force fresh module load (env vars must be read at import time)
-    name = f"train_{id(env_overrides)}"
-    spec = importlib.util.spec_from_file_location(name, "train.py")
-    mod = importlib.util.module_from_spec(spec)
-    spec.loader.exec_module(mod)
-    if env_overrides:
-        for k, orig in saved.items():
-            if orig is None:
-                os.environ.pop(k, None)
-            else:
-                os.environ[k] = orig
-    return mod
-
-
-@pytest.fixture(scope="module")
-def train_mod():
-    return _import()
-
-
-# 1. CausalEncoder exists and has correct output shape
-def test_causal_encoder_shape(train_mod):
-    enc = train_mod.CausalEncoder(n_channels=2, patch_len=10, d_model=32, n_heads=4, depth=1)
-    x = torch.randn(4, 60, 2)
-    tokens = enc(x)          # should return all tokens (B, N, D) for JEPA pretraining
-    assert tokens.shape == (4, 6, 32), f"expected (4, 6, 32), got {tokens.shape}"
-
-
-# 2. CausalEncoder is actually causal: earlier token outputs don't change when later inputs change
-def test_causal_masking(train_mod):
-    enc = train_mod.CausalEncoder(n_channels=2, patch_len=10, d_model=32, n_heads=4, depth=2)
-    enc.eval()
-    torch.manual_seed(0)
-    x = torch.randn(1, 60, 2)
-    x_perturbed = x.clone()
-    # non-uniform noise (constant shift absorbed by per-patch LayerNorm; variance change is not)
-    torch.manual_seed(99)
-    x_perturbed[:, 30:, :] += torch.randn_like(x[:, 30:, :]) * 5.0
-
-    with torch.no_grad():
-        h1 = enc(x)
-        h2 = enc(x_perturbed)
-
-    # First 3 tokens must be identical (causal — don't see future patches)
-    assert torch.allclose(h1[:, :3, :], h2[:, :3, :], atol=1e-5), \
-        "causal masking broken: early tokens change when later input changes"
-    # Last token should differ (it can see the perturbed patches)
-    assert not torch.allclose(h1[:, -1, :], h2[:, -1, :], atol=1e-5), \
-        "last token should differ when later input changes"
-
-
-# 3. HorizonPredictor exists, takes (h, delta_t_float) → same shape as h
-def test_horizon_predictor_shape(train_mod):
-    pred = train_mod.HorizonPredictor(d_model=32)
-    h = torch.randn(4, 32)
-    dt = torch.tensor([1.0, 2.0, 3.0, 1.0])
-    out = pred(h, dt)
-    assert out.shape == (4, 32), f"expected (4, 32), got {out.shape}"
-
-
-# 4. vicreg_loss is a scalar and backward doesn't error
-def test_vicreg_loss_backward(train_mod):
-    h_pred = torch.randn(8, 32, requires_grad=True)
-    h_target = torch.randn(8, 32)
-    loss = train_mod.vicreg_loss(h_pred, h_target, alpha=0.1)
-    assert loss.shape == (), f"expected scalar, got {loss.shape}"
-    loss.backward()
-    assert h_pred.grad is not None
-
-
-# 5. Full JEPA step: encode context, predict future, compute loss, backward
-def test_jepa_step_end_to_end(train_mod):
-    enc = train_mod.CausalEncoder(n_channels=2, patch_len=10, d_model=32, n_heads=4, depth=1)
-    pred = train_mod.HorizonPredictor(d_model=32)
-    opt = torch.optim.SGD(list(enc.parameters()) + list(pred.parameters()), lr=1e-3)
-
-    x = torch.randn(4, 60, 2)
-    tokens = enc(x)          # (4, 6, 32)
-    c, dt = 2, 2             # context position 2, horizon 2
-    h_ctx = tokens[:, c, :]
-    h_tgt = tokens[:, c + dt, :].detach()
-    h_hat = pred(h_ctx, torch.full((4,), float(dt)))
-    loss = train_mod.vicreg_loss(h_hat, h_tgt, alpha=0.1)
-    opt.zero_grad(); loss.backward(); opt.step()
-    assert loss.item() < 100, "loss exploded"
-
-
-# 6. build() returns year-based OOS split (2022-2023); hourly gives many more windows
-def test_build_year_split(train_mod):
-    (Xtr, ytr), (Xte, yte) = train_mod.build()
-    assert Xtr.shape[1] == train_mod.WINDOW
-    assert Xte.shape[1] == train_mod.WINDOW
-    assert len(Xtr) > 0 and len(Xte) > 0
-    # OOS: daily ≈ 600; hourly ≈ 17,000 (2 years × ~8,500 trading hours/year)
-    assert len(Xte) > 400, f"OOS too small: {len(Xte)}"
-
-
-# 7. hourly build gives > 10× more training windows than daily
-def test_build_hourly_more_windows(train_mod):
-    import os
-    if not os.path.exists("data/processed/eurusd_hourly.parquet"):
-        pytest.skip("eurusd_hourly.parquet not present — run data:prepare:hourly first")
-    (Xtr, _), _ = train_mod.build()
-    # Daily had ~877 train windows; hourly with 2008-2021 should have > 50,000
-    assert len(Xtr) > 10_000, f"expected >10k hourly train windows, got {len(Xtr)}"
-
-
-# ── Phase-1: supervised head ──────────────────────────────────────────────────
-
-# 8. SupervisedHead exists and maps (B, D) → (B,)
-def test_supervised_head_shape(train_mod):
-    D = 128
-    head = train_mod.SupervisedHead(D)
-    x = torch.randn(16, D)
-    out = head(x)
-    assert out.shape == (16,), f"expected (16,), got {out.shape}"
-
-
-# 9. SupervisedHead gradient flows (not frozen)
-def test_supervised_head_backward(train_mod):
-    head = train_mod.SupervisedHead(64)
-    x = torch.randn(8, 64)
-    loss = head(x).mean()
-    loss.backward()
-    for name, p in head.named_parameters():
-        assert p.grad is not None, f"no grad on {name}"
-
-
-# 10. Phase-1 beats linear on nonlinear synthetic signal
-def test_phase1_beats_linear_on_nonlinear(train_mod):
-    """MLP head should outperform ridge regression on data with nonlinear structure."""
-    import numpy as np
-    torch.manual_seed(0); np.random.seed(0)
-    N, D = 1000, 32
-    # target = |h|² (quadratic — linear can't fit well)
-    Etr = np.random.randn(N, D).astype(np.float32)
-    ytr = (Etr ** 2).sum(axis=1)
-    Ete = np.random.randn(200, D).astype(np.float32)
-    yte = (Ete ** 2).sum(axis=1)
-
-    # Ridge baseline
-    A = np.hstack([Etr, np.ones((N, 1))])
-    w = np.linalg.solve(A.T @ A + 1e-3 * np.eye(A.shape[1]), A.T @ ytr)
-    pred_lin = np.hstack([Ete, np.ones((200, 1))]) @ w
-    r2_lin = float(1 - ((yte - pred_lin) ** 2).sum() / ((yte - yte.mean()) ** 2).sum())
-
-    # MLP head
-    head = train_mod.SupervisedHead(D)
-    opt  = torch.optim.Adam(head.parameters(), lr=1e-2)
-    Xtr_t = torch.tensor(Etr); ytr_t = torch.tensor(ytr)
-    for _ in range(300):
-        loss = nn.functional.mse_loss(head(Xtr_t), ytr_t)
-        opt.zero_grad(); loss.backward(); opt.step()
-
-    head.eval()
-    with torch.no_grad():
-        pred_mlp = head(torch.tensor(Ete)).numpy()
-    r2_mlp = float(1 - ((yte - pred_mlp) ** 2).sum() / ((yte - yte.mean()) ** 2).sum())
-
-    assert r2_mlp > r2_lin + 0.05, (
-        f"MLP R²={r2_mlp:.3f} should beat ridge R²={r2_lin:.3f} by >0.05 on quadratic target"
-    )
-
-
-# 11. main() returns phase1_r2 in metrics.json (integration — needs real data)
-def test_metrics_json_has_phase1_r2(train_mod):
-    import json
-    if not os.path.exists("metrics.json"):
-        pytest.skip("metrics.json not present — run train.py first")
-    with open("metrics.json") as f:
-        m = json.load(f)
-    assert "phase1_r2" in m, f"phase1_r2 missing from metrics.json: {list(m.keys())}"
-    assert m["phase1_r2"] > m["val_vol_r2"], (
-        f"MLP head phase1_r2={m['phase1_r2']:.4f} should beat linear probe "
-        f"val_vol_r2={m['val_vol_r2']:.4f}"
-    )
-
-
-# ── HPO: env-var knob overrides ───────────────────────────────────────────────
-
-# 12. JEPA_WINDOW env var overrides WINDOW at import time
-def test_env_override_window():
-    mod = _import({"JEPA_WINDOW": "48"})
-    assert mod.WINDOW == 48, f"expected WINDOW=48, got {mod.WINDOW}"
-
-
-# 13. JEPA_D_MODEL and JEPA_DEPTH env vars work
-def test_env_override_d_model_depth():
-    mod = _import({"JEPA_D_MODEL": "64", "JEPA_DEPTH": "4"})
-    assert mod.D_MODEL == 64, f"expected D_MODEL=64, got {mod.D_MODEL}"
-    assert mod.DEPTH == 4, f"expected DEPTH=4, got {mod.DEPTH}"
-
-
-# 14. hpo_sweep.py exists and generates correct config list
-def test_hpo_sweep_configs():
-    import importlib.util
-    sweep_path = "scripts/hpo_sweep.py"
-    if not os.path.exists(sweep_path):
-        pytest.fail(f"{sweep_path} not found — implement it")
-    spec = importlib.util.spec_from_file_location("hpo_sweep", sweep_path)
-    mod = importlib.util.module_from_spec(spec)
-    spec.loader.exec_module(mod)
-    cfgs = list(mod.configs())
-    assert len(cfgs) > 0, "configs() returned empty list"
-    # Every config must have at least D_MODEL, DEPTH, WINDOW keys
-    required = {"JEPA_D_MODEL", "JEPA_DEPTH", "JEPA_WINDOW"}
-    for cfg in cfgs:
-        assert required.issubset(cfg.keys()), f"config missing required keys: {cfg}"
-
-
-# ── Option B: joint encoder fine-tuning in phase-1 ───────────────────────────
-
-# 15. PHASE1_JOINT and PHASE1_ENCODER_LR knobs exist at module level
-def test_joint_phase1_knobs():
-    mod = _import({"JEPA_PHASE1_JOINT": "1", "JEPA_PHASE1_ENCODER_LR": "1e-5"})
-    assert hasattr(mod, "PHASE1_JOINT"), "PHASE1_JOINT knob missing from train.py"
-    assert hasattr(mod, "PHASE1_ENCODER_LR"), "PHASE1_ENCODER_LR knob missing from train.py"
-    assert mod.PHASE1_JOINT is True
-    assert abs(mod.PHASE1_ENCODER_LR - 1e-5) < 1e-12
-
-
-# 16. PHASE1_JOINT defaults to True (joint mode on by default)
-def test_joint_phase1_default_on():
-    mod = _import()
-    assert hasattr(mod, "PHASE1_JOINT"), "PHASE1_JOINT knob missing"
-    assert mod.PHASE1_JOINT is True, f"PHASE1_JOINT default should be True, got {mod.PHASE1_JOINT}"
-
-
-# 17. JEPA_PHASE1_JOINT=0 disables joint (env override works)
-def test_joint_phase1_can_disable():
-    mod = _import({"JEPA_PHASE1_JOINT": "0"})
-    assert mod.PHASE1_JOINT is False, f"expected False, got {mod.PHASE1_JOINT}"
-
-
-# 18. Encoder receives non-zero gradients when joint-training with the head
-def test_joint_encoder_grad_flows(train_mod):
-    """Gradient must flow into encoder when using two-param-group joint optimizer."""
-    import torch.nn.functional as F
-    enc  = train_mod.CausalEncoder(n_channels=2, patch_len=8, d_model=16, n_heads=2, depth=1)
-    head = train_mod.SupervisedHead(16)
-    enc.train(); head.train()
-    opt = torch.optim.Adam([
-        {"params": head.parameters(), "lr": 1e-3},
-        {"params": enc.parameters(), "lr": 1e-5},
-    ], weight_decay=1e-4)
-    # Tiny batch: 4 windows of length 16 (= 2 patches of patch_len=8)
-    X = torch.randn(4, 16, 2)
-    y = torch.randn(4)
-    tokens = enc(X)           # (4, 2, 16)
-    h      = tokens[:, -1, :] # (4, 16) — last token
-    pred   = head(h)
-    loss   = F.mse_loss(pred, y)
-    loss.backward()
-    enc_grads = [p.grad for p in enc.parameters() if p.grad is not None]
-    assert len(enc_grads) > 0, "no encoder params received gradients"
-    assert any(g.abs().max().item() > 0 for g in enc_grads), "all encoder grads are zero"

tests/test_multipair.py

@@ -1,126 +0,0 @@
-"""Tests for multi-pair G10 pipeline (Option C).
-
-Tests the prepare_multipair.py merge logic and train.py multipair build().
-Run: cd ~/dev/AI/jepa-fx-risk && .venv/bin/python -m pytest tests/test_multipair.py -v
-"""
-import importlib.util
-import numpy as np
-import pandas as pd
-import pytest
-import os
-
-
-def _import_mp():
-    spec = importlib.util.spec_from_file_location("prepare_multipair", "scripts/prepare_multipair.py")
-    mod = importlib.util.module_from_spec(spec)
-    spec.loader.exec_module(mod)
-    return mod
-
-
-@pytest.fixture(scope="module")
-def mp():
-    return _import_mp()
-
-
-def _pair_df(start: str, n_hours: int, seed: int) -> pd.DataFrame:
-    """Synthetic single-pair hourly parquet (same schema as prepare_hourly output)."""
-    rng = np.random.default_rng(seed)
-    dts = pd.date_range(start, periods=n_hours, freq="h")
-    closes = 1.1 + np.cumsum(rng.normal(0, 0.001, n_hours))
-    return pd.DataFrame({
-        "datetime":     dts,
-        "close":        closes,
-        "ret":          rng.normal(0, 0.001, n_hours),
-        "realized_vol": np.abs(rng.normal(0.0005, 0.0001, n_hours)),
-    })
-
-
-# 1. merge_pair_parquets returns inner join on datetime
-def test_merge_inner_join(mp):
-    eur = _pair_df("2020-01-01 00:00", 100, seed=1)  # t0 to t0+99h
-    gbp = _pair_df("2020-01-01 20:00", 60, seed=2)   # t0+20 to t0+79h → 60 common
-    result = mp.merge_pair_parquets({"eurusd": eur, "gbpusd": gbp})
-    assert len(result) == 60, f"expected 60 (inner join), got {len(result)}"
-
-
-# 2. merge_pair_parquets prefixes columns with pair name
-def test_merge_column_prefixes(mp):
-    eur = _pair_df("2020-01-01 00:00", 50, seed=1)
-    gbp = _pair_df("2020-01-01 00:00", 50, seed=2)
-    result = mp.merge_pair_parquets({"eurusd": eur, "gbpusd": gbp})
-    assert "datetime" in result.columns, "datetime column missing"
-    assert "eurusd_ret" in result.columns
-    assert "eurusd_rv" in result.columns
-    assert "gbpusd_ret" in result.columns
-    assert "gbpusd_rv" in result.columns
-    # raw pair columns should not leak through unprefixed
-    assert "ret" not in result.columns
-    assert "realized_vol" not in result.columns
-
-
-# 3. No NaN in merged output
-def test_merge_no_nan(mp):
-    eur = _pair_df("2020-01-01 00:00", 50, seed=1)
-    gbp = _pair_df("2020-01-01 00:00", 50, seed=2)
-    result = mp.merge_pair_parquets({"eurusd": eur, "gbpusd": gbp})
-    nan_count = result.isnull().sum().sum()
-    assert nan_count == 0, f"{nan_count} NaN values in merged output"
-
-
-# 4. PAIRS constant is a non-empty list starting with eurusd
-def test_pairs_constant(mp):
-    assert hasattr(mp, "PAIRS"), "PAIRS constant missing from prepare_multipair.py"
-    assert len(mp.PAIRS) >= 2, "PAIRS must have at least 2 pairs"
-    assert mp.PAIRS[0] == "eurusd", "first pair must be eurusd (target pair)"
-
-
-# 5. merge target column is eurusd_rv (for build() target selection)
-def test_merge_has_eurusd_rv_as_target(mp):
-    eur = _pair_df("2020-01-01 00:00", 50, seed=1)
-    gbp = _pair_df("2020-01-01 00:00", 50, seed=2)
-    result = mp.merge_pair_parquets({"eurusd": eur, "gbpusd": gbp})
-    assert "eurusd_rv" in result.columns, "eurusd_rv (target) missing from merged output"
-    assert (result["eurusd_rv"] > 0).all(), "eurusd_rv should be positive"
-
-
-# 6. train.py recognises JEPA_USE_MULTIPAIR env var
-def test_use_multipair_knob():
-    import importlib.util as ilu
-    spec = ilu.spec_from_file_location(f"train_mp_{id(None)}", "train.py")
-    mod = ilu.module_from_spec(spec)
-    saved = os.environ.get("JEPA_USE_MULTIPAIR")
-    os.environ["JEPA_USE_MULTIPAIR"] = "1"
-    try:
-        spec.loader.exec_module(mod)
-    finally:
-        if saved is None:
-            os.environ.pop("JEPA_USE_MULTIPAIR", None)
-        else:
-            os.environ["JEPA_USE_MULTIPAIR"] = saved
-    assert hasattr(mod, "USE_MULTIPAIR"), "USE_MULTIPAIR knob missing from train.py"
-    assert mod.USE_MULTIPAIR is True
-
-
-# 7. build() uses n_pairs*2 channels when multipair parquet present
-def test_build_uses_multipair_channels():
-    import importlib.util as ilu
-    multipair_path = "data/processed/eurusd_multipair.parquet"
-    if not os.path.exists(multipair_path):
-        pytest.skip("eurusd_multipair.parquet not present — run data:prepare:multipair first")
-    saved = os.environ.get("JEPA_USE_MULTIPAIR")
-    os.environ["JEPA_USE_MULTIPAIR"] = "1"
-    try:
-        spec = ilu.spec_from_file_location(f"train_mp2_{id(None)}", "train.py")
-        mod = ilu.module_from_spec(spec)
-        spec.loader.exec_module(mod)
-        (Xtr, _), _ = mod.build()
-    finally:
-        if saved is None:
-            os.environ.pop("JEPA_USE_MULTIPAIR", None)
-        else:
-            os.environ["JEPA_USE_MULTIPAIR"] = saved
-    mp = _import_mp()
-    expected_ch = len(mp.PAIRS) * 2
-    assert Xtr.shape[2] == expected_ch, (
-        f"expected {expected_ch} channels (n_pairs={len(mp.PAIRS)}×2), got {Xtr.shape[2]}"
-    )

tests/test_prepare_hourly.py

@@ -1,207 +0,0 @@
-"""Failing tests for scripts/prepare_hourly.py.
-
-Tests the M1 → hourly aggregation logic using synthetic data before touching
-real downloads.
-
-Run: cd ~/dev/AI/jepa-fx-risk && .venv/bin/python -m pytest tests/test_prepare_hourly.py -v
-"""
-import numpy as np
-import pandas as pd
-import pytest
-import importlib.util, sys, os
-
-
-def _import():
-    spec = importlib.util.spec_from_file_location(
-        "prepare_hourly", "scripts/prepare_hourly.py"
-    )
-    mod = importlib.util.module_from_spec(spec)
-    spec.loader.exec_module(mod)
-    return mod
-
-
-@pytest.fixture(scope="module")
-def ph():
-    return _import()
-
-
-def _make_m1(n_days: int = 3, price: float = 1.1000, noise: float = 0.0005) -> pd.DataFrame:
-    """Synthetic M1 DataFrame starting 2020-01-06 (Monday), 390 ticks/day."""
-    rng = np.random.default_rng(42)
-    # generate full trading hours: Mon-Fri 00:00-23:59 (FX is 24h weekday)
-    start = pd.Timestamp("2020-01-06 00:00:00")  # Monday
-    periods = n_days * 24 * 60
-    ts = pd.date_range(start, periods=periods, freq="min")
-    # remove weekends
-    ts = ts[ts.day_of_week < 5]
-    prices = price + np.cumsum(rng.normal(0, noise, len(ts)))
-    return pd.DataFrame({"ts": ts, "close": prices})
-
-
-# 1. resample_to_hourly: DataFrame has correct columns
-def test_columns(ph):
-    m1 = _make_m1()
-    hourly = ph.resample_to_hourly(m1)
-    assert set(["datetime", "close", "ret", "realized_vol"]).issubset(hourly.columns), \
-        f"missing columns: {hourly.columns.tolist()}"
-
-
-# 2. No cross-weekend interpolation: gap between Friday 23:xx and Sunday/Monday must remain
-def test_no_weekend_interpolation(ph):
-    # Make 2 days: Friday + Monday (skip Saturday/Sunday)
-    fri = pd.date_range("2020-01-10 00:00", "2020-01-10 23:59", freq="min")  # Friday
-    mon = pd.date_range("2020-01-13 00:00", "2020-01-13 23:59", freq="min")  # Monday
-    ts = fri.append(mon)
-    prices = 1.1 + np.cumsum(np.random.default_rng(0).normal(0, 0.0001, len(ts)))
-    m1 = pd.DataFrame({"ts": ts, "close": prices})
-    hourly = ph.resample_to_hourly(m1)
-    dates = pd.DatetimeIndex(hourly["datetime"]).date
-    import datetime
-    sat = datetime.date(2020, 1, 11)
-    sun = datetime.date(2020, 1, 12)
-    assert sat not in dates and sun not in dates, "weekend rows found in hourly output"
-
-
-# 3. Realized vol = sqrt(sum(r²)) over minute returns in each hour
-def test_realized_vol_formula(ph):
-    # Two hours: anchor gives 10:00 a valid ret; measurement hour has one known log-return.
-    ts0 = pd.date_range("2020-01-06 09:00", periods=60, freq="min")
-    ts1 = pd.date_range("2020-01-06 10:00", periods=60, freq="min")
-    prices0 = np.ones(60) * 1.0
-    # price jumps at minute 1 and STAYS (no reversion) → one non-zero log-return
-    prices1 = np.full(60, np.exp(0.01))
-    prices1[0] = 1.0  # only first tick is at 1.0; jump happens at tick 1
-    m1 = pd.DataFrame({
-        "ts": np.concatenate([ts0, ts1]),
-        "close": np.concatenate([prices0, prices1]),
-    })
-    hourly = ph.resample_to_hourly(m1)
-    assert len(hourly) >= 1, "no rows after resample"
-    rv = hourly.iloc[-1]["realized_vol"]
-    expected = np.sqrt(0.01 ** 2)
-    assert abs(rv - expected) < 1e-6, f"realized_vol={rv:.8f}, expected≈{expected:.8f}"
-
-
-# 4. Only hours with ≥ 30 M1 bars are kept (thin hours dropped)
-def test_thin_hours_dropped(ph):
-    # 4 hours: pre-anchor gives 09:00 a valid ret; full survives; thin (11:00) is dropped.
-    # pre-anchor (08:00): gives 09:00 a valid ret
-    # anchor (09:00): 60 bars, valid ret  → kept
-    # full  (10:00): 60 bars, valid ret  → kept
-    # thin  (11:00): 10 bars             → dropped
-    # Result: 3 hourly candidates, first (pre-anchor) gets NaN ret → dropped → 2 rows
-    pre   = pd.date_range("2020-01-06 08:00", periods=60, freq="min")
-    anchor= pd.date_range("2020-01-06 09:00", periods=60, freq="min")
-    full  = pd.date_range("2020-01-06 10:00", periods=60, freq="min")
-    thin  = pd.date_range("2020-01-06 11:00", periods=10, freq="min")
-    ts = pre.append(anchor).append(full).append(thin)
-    m1 = pd.DataFrame({"ts": ts, "close": np.ones(len(ts)) * 1.1})
-    hourly = ph.resample_to_hourly(m1)
-    assert len(hourly) == 2, f"expected 2 rows (pre-anchor NaN ret dropped + thin dropped), got {len(hourly)}"
-
-
-# 5. Output parquet path and schema (integration — reads actual M1 zips if present)
-def test_output_schema_from_zips(ph, tmp_path):
-    import zipfile, io
-    rows = []
-    for h in range(24):
-        for m in range(60):
-            rows.append(f"20200106 {h:02d}{m:02d}00;1.10000;1.10100;1.09900;1.10000;100")
-    csv_content = "\n".join(rows).encode()
-    zip_buf = io.BytesIO()
-    with zipfile.ZipFile(zip_buf, "w") as zf:
-        zf.writestr("DAT_ASCII_EURUSD_M1_2020.csv", csv_content)
-    zip_buf.seek(0)
-    raw_dir = tmp_path / "raw"
-    raw_dir.mkdir()
-    (raw_dir / "DAT_ASCII_EURUSD_M1_2020.zip").write_bytes(zip_buf.read())
-
-    out_path = str(tmp_path / "eurusd_hourly.parquet")
-    ph.build_hourly_parquet(raw_dir=str(raw_dir), out_path=out_path)
-    assert os.path.exists(out_path), "output parquet not created"
-    df = pd.read_parquet(out_path)
-    assert set(["datetime", "close", "ret", "realized_vol"]).issubset(df.columns)
-    assert len(df) > 0
-
-
-# ── New OHLCV-derived features ────────────────────────────────────────────────
-
-def _make_m1_ohlcv(n_hours: int = 4, price: float = 1.1) -> pd.DataFrame:
-    """Synthetic M1 with distinct O, H, L, C so hl_range and ret_intrabar are nonzero."""
-    rng = np.random.default_rng(7)
-    ts = pd.date_range("2020-01-06 00:00", periods=n_hours * 60, freq="min")
-    closes = price + np.cumsum(rng.normal(0, 0.0002, len(ts)))
-    highs  = closes + rng.uniform(0.0001, 0.0005, len(ts))
-    lows   = closes - rng.uniform(0.0001, 0.0005, len(ts))
-    opens  = np.roll(closes, 1); opens[0] = price
-    return pd.DataFrame({"ts": ts, "open": opens, "high": highs, "low": lows, "close": closes})
-
-
-# 6. resample_to_hourly produces hl_range column
-def test_hourly_has_hl_range(ph):
-    m1 = _make_m1_ohlcv()
-    hourly = ph.resample_to_hourly(m1)
-    assert "hl_range" in hourly.columns, f"missing hl_range; cols={hourly.columns.tolist()}"
-    assert (hourly["hl_range"] > 0).all(), "hl_range should be positive"
-
-
-# 7. resample_to_hourly produces ret_intrabar column
-def test_hourly_has_ret_intrabar(ph):
-    m1 = _make_m1_ohlcv()
-    hourly = ph.resample_to_hourly(m1)
-    assert "ret_intrabar" in hourly.columns, f"missing ret_intrabar; cols={hourly.columns.tolist()}"
-
-
-# 8. hl_range = log(hourly_high / hourly_low)
-def test_hl_range_formula(ph):
-    # Two hours; second has known H=1.105, L=1.095
-    ts0 = pd.date_range("2020-01-06 00:00", periods=60, freq="min")
-    ts1 = pd.date_range("2020-01-06 01:00", periods=60, freq="min")
-    closes = np.full(120, 1.1)
-    highs  = np.full(120, 1.1)
-    lows   = np.full(120, 1.1)
-    # second hour: known spread
-    highs[60:] = 1.105
-    lows[60:]  = 1.095
-    m1 = pd.DataFrame({
-        "ts": np.concatenate([ts0, ts1]),
-        "open": closes, "high": highs, "low": lows, "close": closes,
-    })
-    hourly = ph.resample_to_hourly(m1)
-    assert len(hourly) >= 1
-    hl = hourly.iloc[-1]["hl_range"]
-    expected = float(np.log(1.105 / 1.095))
-    assert abs(hl - expected) < 1e-6, f"hl_range={hl:.8f}, expected={expected:.8f}"
-
-
-# 9. ret_intrabar = log(hourly_last_close / hourly_first_open)
-def test_ret_intrabar_formula(ph):
-    ts0 = pd.date_range("2020-01-06 00:00", periods=60, freq="min")
-    ts1 = pd.date_range("2020-01-06 01:00", periods=60, freq="min")
-    closes = np.full(120, 1.1)
-    opens  = np.full(120, 1.1)
-    # second hour: open=1.09, close=1.11
-    opens[60]   = 1.09
-    closes[119] = 1.11
-    m1 = pd.DataFrame({
-        "ts": np.concatenate([ts0, ts1]),
-        "open": opens, "high": closes + 0.001, "low": closes - 0.001, "close": closes,
-    })
-    hourly = ph.resample_to_hourly(m1)
-    assert len(hourly) >= 1
-    rib = hourly.iloc[-1]["ret_intrabar"]
-    expected = float(np.log(1.11 / 1.09))
-    assert abs(rib - expected) < 1e-6, f"ret_intrabar={rib:.8f}, expected={expected:.8f}"
-
-
-# 10. build() in train.py uses 2 feature channels (HPO: hl_range/ret_intrabar redundant)
-def test_build_uses_2_channels(tmp_path):
-    import importlib.util, os
-    hourly_path = "data/processed/eurusd_hourly.parquet"
-    if not os.path.exists(hourly_path):
-        pytest.skip("eurusd_hourly.parquet not present")
-    spec = importlib.util.spec_from_file_location("train_2ch", "train.py")
-    mod = importlib.util.module_from_spec(spec)
-    spec.loader.exec_module(mod)
-    (Xtr, _), _ = mod.build()
-    assert Xtr.shape[2] == 2, f"expected 2 channels, got {Xtr.shape[2]}"

tests/test_regime.py

@@ -1,134 +0,0 @@
-"""Tests for scripts/prepare_regime.py — HMM regime detector (jepa-fx-risk#13).
-
-TDD: tests first, implementation follows.
-"""
-
-import importlib.util
-import os
-import shutil
-from pathlib import Path
-
-import numpy as np
-import pandas as pd
-import pytest
-
-_SCRIPT = Path(__file__).parent.parent / "scripts" / "prepare_regime.py"
-
-DATA_DIR = Path(__file__).parent.parent / "data" / "processed"
-HOURLY = DATA_DIR / "eurusd_hourly.parquet"
-DAILY = DATA_DIR / "eurusd_daily.parquet"
-
-
-def _import():
-    spec = importlib.util.spec_from_file_location("prepare_regime", _SCRIPT)
-    mod = importlib.util.module_from_spec(spec)
-    spec.loader.exec_module(mod)
-    return mod
-
-
-@pytest.fixture()
-def mod():
-    return _import()
-
-
-# ---------------------------------------------------------------------------
-# fit_regime_hmm — pure function (doesn't touch disk)
-# ---------------------------------------------------------------------------
-
-def _synthetic_rv(seed=42, size=500):
-    """Noisy 3-regime vol series: calm→stressed→crisis→calm interleaved."""
-    rng = np.random.default_rng(seed)
-    low  = np.abs(rng.normal(0.005, 0.001, size=size // 3))
-    mid  = np.abs(rng.normal(0.015, 0.003, size=size // 3))
-    high = np.abs(rng.normal(0.04,  0.008, size=size - 2 * (size // 3)))
-    return np.concatenate([low, mid, high])
-
-
-class TestFitRegimeHmm:
-    def test_returns_integer_labels(self, mod):
-        rv = _synthetic_rv(seed=0)
-        labels = mod.fit_regime_hmm(rv, n_states=3, random_state=42)
-        assert np.issubdtype(labels.dtype, np.integer), f"dtype={labels.dtype}"
-        assert len(labels) == len(rv)
-
-    def test_states_are_0_1_2(self, mod):
-        rv = _synthetic_rv(seed=1)
-        labels = mod.fit_regime_hmm(rv, n_states=3, random_state=42)
-        unique = set(labels.tolist())
-        assert unique.issubset({0, 1, 2}), f"unexpected states: {unique}"
-
-    def test_deterministic(self, mod):
-        rv = _synthetic_rv(seed=7)
-        a = mod.fit_regime_hmm(rv, n_states=3, random_state=42)
-        b = mod.fit_regime_hmm(rv, n_states=3, random_state=42)
-        assert np.array_equal(a, b), "HMM not deterministic with same random_state"
-
-    def test_sorted_by_vol_asc(self, mod):
-        # 3 clearly separated noisy clusters; state 0 should be calm, 2 should be crisis.
-        rng = np.random.default_rng(42)
-        n = 200
-        low  = np.abs(rng.normal(0.005, 0.001, n))
-        mid  = np.abs(rng.normal(0.015, 0.003, n))
-        high = np.abs(rng.normal(0.05,  0.008, n))
-        rv = np.concatenate([low, mid, high])
-        labels = mod.fit_regime_hmm(rv, n_states=3, random_state=42)
-        # Mean regime label in the high-vol section should exceed mean in the low-vol section.
-        assert labels[2*n:].mean() > labels[:n].mean(), \
-            "crisis section mean regime label should exceed calm section"
-        # The calm section should not be labeled as crisis (2) dominantly.
-        calm_modal = int(np.bincount(labels[:n]).argmax())
-        assert calm_modal < 2, f"calm section mostly labeled {calm_modal}, expected 0 or 1"
-
-    def test_two_states(self, mod):
-        rv = _synthetic_rv(seed=0)
-        labels = mod.fit_regime_hmm(rv, n_states=2, random_state=42)
-        unique = set(labels.tolist())
-        assert unique.issubset({0, 1})
-
-
-
-
-# ---------------------------------------------------------------------------
-# prepare_regime_df — reads parquet, fits HMM, returns DataFrame
-# ---------------------------------------------------------------------------
-
-class TestPrepareRegimeDf:
-    @pytest.mark.skipif(not HOURLY.exists(), reason="hourly parquet not available")
-    def test_output_columns(self, mod):
-        df = mod.prepare_regime_df(str(HOURLY), freq="hourly")
-        assert "datetime" in df.columns
-        assert "regime" in df.columns
-
-    @pytest.mark.skipif(not HOURLY.exists(), reason="hourly parquet not available")
-    def test_regime_values(self, mod):
-        df = mod.prepare_regime_df(str(HOURLY), freq="hourly")
-        unique = set(df["regime"].tolist())
-        assert unique.issubset({0, 1, 2}), f"unexpected regime values: {unique}"
-
-    @pytest.mark.skipif(not HOURLY.exists(), reason="hourly parquet not available")
-    def test_no_nulls(self, mod):
-        df = mod.prepare_regime_df(str(HOURLY), freq="hourly")
-        assert df["regime"].isna().sum() == 0
-
-    @pytest.mark.skipif(not DAILY.exists(), reason="daily parquet not available")
-    def test_daily_fallback(self, mod):
-        df = mod.prepare_regime_df(str(DAILY), freq="daily")
-        assert "regime" in df.columns
-        assert set(df["regime"].tolist()).issubset({0, 1, 2})
-
-
-# ---------------------------------------------------------------------------
-# Integration: check that train.py REGIME SEAM exists and is togglable
-# ---------------------------------------------------------------------------
-
-class TestTrainPyRegimeSeam:
-    def test_enable_regime_env_var_documented(self):
-        train_py = Path(__file__).parent.parent / "train.py"
-        content = train_py.read_text()
-        assert "JEPA_ENABLE_REGIME" in content, "JEPA_ENABLE_REGIME toggle not found in train.py"
-
-    def test_regime_seam_comment_present(self):
-        train_py = Path(__file__).parent.parent / "train.py"
-        content = train_py.read_text()
-        assert "REGIME" in content and "seam" in content.lower(), \
-            "agent-editable regime seam marker not found in train.py"

tests/test_var_metric.py

@@ -1,108 +0,0 @@
-"""Tests for scripts/var_breach.py — VaR breach rate + Kupiec POF (jepa-fx-risk#12).
-
-Golden tests first: verify the math before wiring it into train.py.
-"""
-
-import importlib.util
-import math
-from pathlib import Path
-
-import pytest
-
-_SCRIPT = Path(__file__).parent.parent / "scripts" / "var_breach.py"
-
-
-def _import():
-    spec = importlib.util.spec_from_file_location("var_breach", _SCRIPT)
-    mod = importlib.util.module_from_spec(spec)
-    spec.loader.exec_module(mod)
-    return mod
-
-
-@pytest.fixture()
-def mod():
-    return _import()
-
-
-# ---------------------------------------------------------------------------
-# var_breach_rate
-# ---------------------------------------------------------------------------
-
-class TestVarBreachRate:
-    def test_zero_breaches(self, mod):
-        # 0.02 < 0.01×2.326=0.02326 → no breach
-        rate, _ = mod.var_breach_rate([0.01, 0.01], [0.02, 0.02])
-        assert rate == 0.0
-
-    def test_all_breach(self, mod):
-        # 0.03 > 0.02326 → all breach
-        rate, _ = mod.var_breach_rate([0.01, 0.01], [0.03, 0.03])
-        assert rate == 1.0
-
-    def test_golden_two_of_ten(self, mod):
-        pred = [0.01] * 10
-        actual = [0.01] * 10
-        actual[0] = 0.03   # breach
-        actual[2] = 0.03   # breach
-        rate, kupiec_p = mod.var_breach_rate(pred, actual)
-        assert abs(rate - 0.2) < 1e-9, f"rate={rate}"
-        assert kupiec_p < 0.05, f"kupiec_p={kupiec_p}"  # strong reject
-
-    def test_perfect_calibration(self, mod):
-        # n=100, 1 breach → p_hat=0.01=p0=0.01 → LR=0 → kupiec_p≈1
-        pred = [0.01] * 100
-        actual = [0.015] * 100
-        actual[0] = 0.025  # 0.025 > 0.02326 → breach
-        rate, kupiec_p = mod.var_breach_rate(pred, actual)
-        assert abs(rate - 0.01) < 1e-9
-        assert kupiec_p > 0.9, f"kupiec_p={kupiec_p}"
-
-    def test_boundary_at_var_is_not_breach(self, mod):
-        # exactly at VaR_99 is NOT a breach (strict >)
-        z99 = 2.326
-        var = 0.01 * z99
-        rate, _ = mod.var_breach_rate([0.01], [var], z99=z99)
-        assert rate == 0.0
-
-    def test_empty_returns_zero_one(self, mod):
-        rate, kupiec_p = mod.var_breach_rate([], [])
-        assert rate == 0.0
-        assert kupiec_p == 1.0
-
-    def test_metric_key_no_whitespace(self, mod):
-        key = mod.METRIC_KEY
-        assert key == key.strip(), f"metric key has surrounding whitespace: {key!r}"
-        assert " " not in key, f"metric key contains space: {key!r}"
-
-    def test_metric_key_is_canonical(self, mod):
-        assert mod.METRIC_KEY == "VaR_breach_rate_99_oos_regime_cond"
-
-
-# ---------------------------------------------------------------------------
-# kupiec_pvalue
-# ---------------------------------------------------------------------------
-
-class TestKupiecPValue:
-    def test_perfectly_calibrated(self, mod):
-        # p_hat == p0 → LR=0 → p-value=1
-        p = mod.kupiec_pvalue(100, 1, p0=0.01)
-        assert p > 0.99, f"p={p}"
-
-    def test_strong_reject_high_breach(self, mod):
-        # 20% breach when 1% expected → p << 0.05
-        p = mod.kupiec_pvalue(100, 20, p0=0.01)
-        assert p < 0.001, f"p={p}"
-
-    def test_zero_breaches_not_nan(self, mod):
-        p = mod.kupiec_pvalue(100, 0, p0=0.01)
-        assert not math.isnan(p)
-        assert 0 <= p <= 1.0
-
-    def test_all_breaches_not_nan(self, mod):
-        p = mod.kupiec_pvalue(10, 10, p0=0.01)
-        assert not math.isnan(p)
-        assert p < 0.001  # extremely unlikely
-
-    def test_zero_observations(self, mod):
-        p = mod.kupiec_pvalue(0, 0)
-        assert p == 1.0

train.py

@@ -1,385 +0,0 @@
-"""train.py — autoresearch agent file (only this may be edited).
-
-HEPA backbone (Petersen et al., arXiv:2605.11130, ICML 2026 Spotlight):
-Causal Transformer pre-trained via horizon-conditioned JEPA. Predictor
-maps (h_t, Δt) → predicted future embedding; loss = VICReg (L1 alignment
-on L2-normalised reps + variance-covariance regulariser, no stop-gradient).
-Probe: ridge regression on the last-token embedding (true OOS split).
-
-Agent may tune: encoder depth/width, patch geometry, ALPHA, DELTA_T_MAX,
-optimizer, LR. Do NOT touch prepare_data.py, loop.py, or the data pipeline.
-"""
-import json
-import math
-import numpy as np
-import pandas as pd
-import torch
-import torch.nn as nn
-import torch.nn.functional as F
-
-# --- agent-tunable knobs (all overridable via JEPA_* env vars for HPO) ---
-import os as _os
-USE_HOURLY    = True
-WINDOW        = int(_os.environ.get("JEPA_WINDOW",        120))   # HPO winner: 5-day context
-PATCH_LEN     = int(_os.environ.get("JEPA_PATCH_LEN",     24))
-D_MODEL       = int(_os.environ.get("JEPA_D_MODEL",       128))
-DEPTH         = int(_os.environ.get("JEPA_DEPTH",         2))
-N_HEADS       = int(_os.environ.get("JEPA_N_HEADS",       4))
-ALPHA         = float(_os.environ.get("JEPA_ALPHA",       0.1))
-DELTA_T_MAX   = int(_os.environ.get("JEPA_DELTA_T_MAX",   3))
-BATCH_SIZE    = int(_os.environ.get("JEPA_BATCH_SIZE",    512))
-EPOCHS        = int(_os.environ.get("JEPA_EPOCHS",        300))
-LR            = float(_os.environ.get("JEPA_LR",          3e-4))
-PHASE1_EPOCHS      = int(_os.environ.get("JEPA_PHASE1_EPOCHS",       200))
-PHASE1_LR          = float(_os.environ.get("JEPA_PHASE1_LR",           1e-3))
-PHASE1_JOINT       = bool(int(_os.environ.get("JEPA_PHASE1_JOINT",     1)))
-PHASE1_JOINT_EPOCHS= int(_os.environ.get("JEPA_PHASE1_JOINT_EPOCHS",  30))
-PHASE1_ENCODER_LR  = float(_os.environ.get("JEPA_PHASE1_ENCODER_LR",  3e-6))
-USE_MULTIPAIR      = bool(int(_os.environ.get("JEPA_USE_MULTIPAIR",    0)))
-JEPA_ENABLE_REGIME = bool(int(_os.environ.get("JEPA_ENABLE_REGIME",    0)))
-SEED          = int(_os.environ.get("JEPA_SEED",          0))
-# ---------------------------
-
-torch.manual_seed(SEED)
-np.random.seed(SEED)
-dev = "cuda" if torch.cuda.is_available() else "cpu"
-
-
-# ── VICReg pretraining loss ──────────────────────────────────────────────────
-
-def vicreg_loss(h_pred: torch.Tensor, h_target: torch.Tensor, alpha: float = 0.1) -> torch.Tensor:
-    """L = (1-α)·L1(normalize(ĥ), normalize(h*)) + α·(L_var + L_cov).
-
-    Both encoders receive gradients (joint training — no stop-grad on h_target).
-    Variance-covariance terms prevent embedding collapse.
-    """
-    pred_n = F.normalize(h_pred, dim=-1)
-    targ_n = F.normalize(h_target, dim=-1)
-    l1 = F.l1_loss(pred_n, targ_n)
-    # variance hinge: push each feature std toward ≥ 1
-    std = h_pred.std(dim=0) + 1e-4
-    l_var = F.relu(1.0 - std).mean()
-    # covariance penalty: decorrelate features
-    B, D = h_pred.shape
-    h_c = h_pred - h_pred.mean(dim=0, keepdim=True)
-    cov = (h_c.t() @ h_c) / max(B - 1, 1)
-    off = cov - torch.diag(torch.diag(cov))
-    l_cov = (off ** 2).sum() / D
-    return (1 - alpha) * l1 + alpha * (l_var + l_cov)
-
-
-# ── CausalEncoder ─────────────────────────────────────────────────────────────
-
-class CausalEncoder(nn.Module):
-    """Non-overlapping patches → per-patch LayerNorm → causal Transformer → all tokens (B, N, D).
-
-    Per-patch LayerNorm instead of full-window RevIN: each patch is normalised
-    using only its own timesteps, so no future statistics leak into past tokens.
-    Use [:, -1, :] for probing (last token sees full context).
-    Use [:, c, :] for JEPA pretraining (context-at-c).
-    """
-    def __init__(self, n_channels: int, patch_len: int, d_model: int,
-                 n_heads: int, depth: int):
-        super().__init__()
-        self.patch_len = patch_len
-        self.d_model   = d_model
-        patch_dim = patch_len * n_channels
-        self.patch_norm = nn.LayerNorm(patch_dim)   # applied per-patch, no future leakage
-        self.embed     = nn.Linear(patch_dim, d_model)
-        layer = nn.TransformerEncoderLayer(d_model, n_heads, 2 * d_model,
-                                           dropout=0.0, batch_first=True)
-        self.tf   = nn.TransformerEncoder(layer, num_layers=depth)
-        self.norm = nn.LayerNorm(d_model)
-
-    def forward(self, x: torch.Tensor) -> torch.Tensor:
-        B, W, F = x.shape
-        P = self.patch_len
-        N = W // P
-        tokens = x[:, :N * P, :].reshape(B, N, P * F)
-        tokens = self.embed(self.patch_norm(tokens))
-        # sinusoidal PE
-        pos = torch.arange(N, device=x.device).float()
-        div = torch.exp(torch.arange(0, self.d_model, 2, device=x.device).float()
-                        * -(math.log(10000.0) / self.d_model))
-        pe = torch.zeros(N, self.d_model, device=x.device)
-        pe[:, 0::2] = torch.sin(pos.unsqueeze(1) * div)
-        pe[:, 1::2] = torch.cos(pos.unsqueeze(1) * div)
-        tokens = tokens + pe
-        # causal mask
-        mask = nn.Transformer.generate_square_subsequent_mask(N, device=x.device)
-        return self.norm(self.tf(tokens, mask=mask, is_causal=True))
-
-
-# ── HorizonPredictor ─────────────────────────────────────────────────────────
-
-class HorizonPredictor(nn.Module):
-    """MLP(cat(h_t, Δt)) → predicted future embedding."""
-    def __init__(self, d_model: int):
-        super().__init__()
-        self.net = nn.Sequential(
-            nn.Linear(d_model + 1, d_model), nn.GELU(),
-            nn.Linear(d_model, d_model),     nn.GELU(),
-            nn.Linear(d_model, d_model),
-        )
-
-    def forward(self, h: torch.Tensor, delta_t: torch.Tensor) -> torch.Tensor:
-        dt = delta_t.float().unsqueeze(-1)
-        return self.net(torch.cat([h, dt], dim=-1))
-
-
-# ── Phase-1 supervised head ──────────────────────────────────────────────────
-
-class SupervisedHead(nn.Module):
-    """Small MLP trained on frozen HEPA embeddings to predict next-period realized vol."""
-    def __init__(self, d_model: int):
-        super().__init__()
-        self.net = nn.Sequential(
-            nn.Linear(d_model, d_model // 2), nn.GELU(),
-            nn.Linear(d_model // 2, 1),
-        )
-
-    def forward(self, h: torch.Tensor) -> torch.Tensor:
-        return self.net(h).squeeze(-1)
-
-
-# ── Data ─────────────────────────────────────────────────────────────────────
-
-def build():
-    """Year-based split: encoder trains on ≤2021; probe evaluates on ≥2022 OOS.
-
-    Uses eurusd_hourly.parquet when USE_HOURLY=True and the file exists;
-    falls back to eurusd_daily.parquet otherwise.
-    """
-    import os
-    multipair_path = "data/processed/eurusd_multipair.parquet"
-    hourly_path    = "data/processed/eurusd_hourly.parquet"
-    daily_path     = "data/processed/eurusd_daily.parquet"
-    if USE_MULTIPAIR and os.path.exists(multipair_path):
-        df = pd.read_parquet(multipair_path).reset_index(drop=True)
-        df["date"] = pd.to_datetime(df["datetime"])
-        # All {pair}_ret + {pair}_rv columns as features; eurusd_rv as target
-        feat_cols = [c for c in df.columns if c.endswith("_ret") or c.endswith("_rv")]
-        FEAT_COLS = feat_cols
-        target_col = "eurusd_rv"
-    elif USE_HOURLY and os.path.exists(hourly_path):
-        df = pd.read_parquet(hourly_path).reset_index(drop=True)
-        df["date"] = pd.to_datetime(df["datetime"])
-        # 2-channel default (HPO: adding hl_range+ret_intrabar hurt — correlated with base feats)
-        FEAT_COLS = ["ret", "realized_vol"]
-        target_col = "realized_vol"
-    else:
-        df = pd.read_parquet(daily_path).reset_index(drop=True)
-        df["date"] = pd.to_datetime(df["date"])
-        FEAT_COLS = ["ret", "realized_vol"]
-        target_col = "realized_vol"
-    # ── REGIME CONDITIONING SEAM — agent may vary this mechanism ─────────────
-    # Baseline: concat regime flag as an additional feature channel (0=calm, 2=crisis).
-    # Agent may swap for FiLM conditioning, learned regime embedding, or gating.
-    _regime_path = "data/processed/eurusd_regime.parquet"
-    if JEPA_ENABLE_REGIME and os.path.exists(_regime_path):
-        _rdf = pd.read_parquet(_regime_path)
-        _ts_col = "datetime" if "datetime" in _rdf.columns else "date"
-        _rdf[_ts_col] = pd.to_datetime(_rdf[_ts_col])
-        df = df.copy()
-        df = df.merge(
-            _rdf.rename(columns={_ts_col: "date"})[["date", "regime"]],
-            on="date", how="left",
-        )
-        df["regime"] = df["regime"].fillna(0).astype(np.float32)
-        FEAT_COLS = list(FEAT_COLS) + ["regime"]
-    # ── END REGIME SEAM ───────────────────────────────────────────────────────
-    feats  = df[FEAT_COLS].to_numpy(np.float32)
-    target = df[target_col].to_numpy(np.float32)
-    tr_idx = df.index[df["date"].dt.year <= 2021].tolist()
-    te_idx = df.index[df["date"].dt.year >= 2022].tolist()
-    mu = feats[:tr_idx[-1]+1].mean(0)
-    sd = feats[:tr_idx[-1]+1].std(0) + 1e-8
-    fn = (feats - mu) / sd
-    def windows(idx):
-        X, y = [], []
-        for t in idx:
-            if t - WINDOW >= 0 and t + 1 < len(df):
-                X.append(fn[t - WINDOW:t]); y.append(target[t + 1])
-        return np.stack(X).astype(np.float32), np.array(y, np.float32)
-    return windows(tr_idx), windows(te_idx)
-
-
-# ── Training ──────────────────────────────────────────────────────────────────
-
-def main():
-    (Xtr, ytr), (Xte, yte) = build()
-    n_feats   = Xtr.shape[2]
-    n_patches = WINDOW // PATCH_LEN
-    N_tr      = len(Xtr)
-    bs        = min(BATCH_SIZE, N_tr)
-
-    enc  = CausalEncoder(n_feats, PATCH_LEN, D_MODEL, N_HEADS, DEPTH).to(dev)
-    pred = HorizonPredictor(D_MODEL).to(dev)
-    opt  = torch.optim.AdamW(list(enc.parameters()) + list(pred.parameters()), lr=LR)
-
-    for ep in range(EPOCHS):
-        # Random mini-batch (avoids OOM on large hourly dataset)
-        idx_b = torch.randperm(N_tr)[:bs]
-        Xb    = torch.tensor(Xtr[idx_b.numpy()], device=dev)
-
-        # Sample random context position and horizon
-        c  = torch.randint(0, n_patches - 1, ()).item()
-        dt = torch.randint(1, max(2, min(DELTA_T_MAX, n_patches - 1 - c) + 1), ()).item()
-
-        tokens = enc(Xb)                        # (bs, N, D)
-        h_ctx  = tokens[:, c, :]               # context embedding
-        h_tgt  = tokens[:, c + dt, :]          # target embedding (joint training)
-        h_hat  = pred(h_ctx, torch.full((bs,), float(dt), device=dev))
-        loss   = vicreg_loss(h_hat, h_tgt, alpha=ALPHA)
-        opt.zero_grad(); loss.backward(); opt.step()
-
-    enc.eval()
-    with torch.no_grad():
-        def embed(X_np):
-            chunks = []
-            for i in range(0, len(X_np), bs):
-                t = torch.tensor(X_np[i:i+bs], device=dev)
-                chunks.append(enc(t)[:, -1, :].cpu().numpy())
-            return np.concatenate(chunks, axis=0)
-
-        Etr = embed(Xtr)
-        Ete = embed(Xte)
-
-    # Ridge probe: fit on train, evaluate on OOS (true OOS R²)
-    mu_e = Etr.mean(0); sd_e = Etr.std(0) + 1e-8
-    Etr_n = (Etr - mu_e) / sd_e
-    Ete_n = (Ete - mu_e) / sd_e
-    A  = np.hstack([Etr_n, np.ones((len(Etr_n), 1))])
-    w  = np.linalg.solve(A.T @ A + 1e-3 * np.eye(A.shape[1]), A.T @ ytr)
-    pred_np = np.hstack([Ete_n, np.ones((len(Ete_n), 1))]) @ w
-    ss_res  = ((yte - pred_np) ** 2).sum()
-    ss_tot  = ((yte - yte.mean()) ** 2).sum()
-    val_vol_r2 = float(1 - ss_res / ss_tot)
-
-    # Phase-1: MLP supervised head — joint or frozen-encoder path
-    ytr_mu = float(ytr.mean()); ytr_sd = float(ytr.std()) + 1e-8
-    ytr_z  = (ytr - ytr_mu) / ytr_sd
-    head   = SupervisedHead(D_MODEL).to(dev)
-    p1_bs  = min(BATCH_SIZE, len(Etr_n))
-
-    # Shared tensors for the frozen-head warmup (used by both paths)
-    Etr_t   = torch.tensor(Etr_n, device=dev)
-    ytr_z_t = torch.tensor(ytr_z, device=dev)
-    Ete_t   = torch.tensor(Ete_n, device=dev)
-    N_tr_h  = len(Etr_t)
-
-    # Phase 1a: warm up head on frozen embeddings (both paths run this)
-    head_opt = torch.optim.Adam(head.parameters(), lr=PHASE1_LR, weight_decay=1e-4)
-    for _ in range(PHASE1_EPOCHS):
-        perm = torch.randperm(N_tr_h, device=dev)
-        for start in range(0, N_tr_h, p1_bs):
-            idx_h = perm[start:start + p1_bs]
-            loss_h = F.mse_loss(head(Etr_t[idx_h]), ytr_z_t[idx_h])
-            head_opt.zero_grad(); loss_h.backward(); head_opt.step()
-
-    if PHASE1_JOINT:
-        # Phase 1b: short joint fine-tuning — encoder nudged with tiny LR.
-        # Normalize live encoder output with FROZEN stats (mu_e, sd_e) so the
-        # head sees the same embedding distribution it was warmed up on.
-        enc.train()
-        mu_e_t = torch.tensor(mu_e, device=dev)
-        sd_e_t = torch.tensor(sd_e, device=dev)
-        Xtr_t  = torch.tensor(Xtr, device=dev)
-        joint_opt = torch.optim.Adam([
-            {"params": head.parameters(), "lr": PHASE1_LR * 0.1},
-            {"params": enc.parameters(),  "lr": PHASE1_ENCODER_LR},
-        ], weight_decay=1e-4)
-        for _ in range(PHASE1_JOINT_EPOCHS):
-            perm = torch.randperm(len(Xtr_t), device=dev)
-            for start in range(0, len(Xtr_t), p1_bs):
-                idx_j = perm[start:start + p1_bs]
-                h_raw = enc(Xtr_t[idx_j])[:, -1, :]
-                h_n   = (h_raw - mu_e_t) / sd_e_t  # frozen-stats normalisation
-                loss_j = F.mse_loss(head(h_n), ytr_z_t[idx_j])
-                joint_opt.zero_grad(); loss_j.backward(); joint_opt.step()
-        enc.eval()
-        # Re-extract test embeddings with fine-tuned encoder, same normalisation
-        with torch.no_grad():
-            chunks = []
-            for i in range(0, len(Xte), p1_bs):
-                t = torch.tensor(Xte[i:i+p1_bs], device=dev)
-                h = enc(t)[:, -1, :]
-                chunks.append(((h - mu_e_t) / sd_e_t).cpu().numpy())
-            Ete_t = torch.tensor(np.concatenate(chunks), device=dev)
-
-    head.eval()
-    with torch.no_grad():
-        pred_h_z = head(Ete_t).cpu().numpy()
-
-    pred_h    = pred_h_z * ytr_sd + ytr_mu          # de-standardise
-    phase1_r2 = float(1 - ((yte - pred_h) ** 2).sum() / ss_tot)
-    print("phase1_r2   = %.4f  (n_test=%d)" % (phase1_r2, len(yte)))
-
-    # ── VaR EVAL BLOCK — do NOT edit (agent boundary) ───────────────────────
-    import sys as _sys
-    _sys.path.insert(0, _os.path.dirname(_os.path.abspath(__file__)))
-    from scripts.var_breach import var_breach_rate as _var_breach_rate, METRIC_KEY as _VAR_KEY
-    _var_rate, _kupiec_p = _var_breach_rate(pred_np.tolist(), yte.tolist())
-    print("%s=%.4f  Kupiec_p=%.4f" % (_VAR_KEY, _var_rate, _kupiec_p))
-    # ── END VaR EVAL BLOCK ───────────────────────────────────────────────────
-
-    _metrics_out = _os.environ.get("METRICS_OUT", "metrics.json")
-    json.dump({
-        "val_vol_r2": val_vol_r2, "phase1_r2": phase1_r2, "n_test": len(yte),
-        _VAR_KEY: _var_rate, "kupiec_p": _kupiec_p,
-        "knobs": {"WINDOW": WINDOW, "PATCH_LEN": PATCH_LEN,
-                  "D_MODEL": D_MODEL, "DEPTH": DEPTH, "ALPHA": ALPHA,
-                  "DELTA_T_MAX": DELTA_T_MAX, "EPOCHS": EPOCHS},
-    }, open(_metrics_out, "w"), indent=2)
-    print("val_vol_r2 = %.4f  (n_test=%d, dev=%s)" % (val_vol_r2, len(yte), dev))
-
-    # ── EXPORT BLOCK — do NOT edit (agent boundary) ──────────────────────────
-    # Set EXPORT_EMBEDDINGS=1 to write embeddings.json for the Go eval harness.
-    import os
-    if os.environ.get("EXPORT_EMBEDDINGS") == "1":
-        hourly_path2 = "data/processed/eurusd_hourly.parquet"
-        daily_path2  = "data/processed/eurusd_daily.parquet"
-        if USE_HOURLY and os.path.exists(hourly_path2):
-            df2 = pd.read_parquet(hourly_path2).reset_index(drop=True)
-            df2["date"] = pd.to_datetime(df2["datetime"])
-        else:
-            df2 = pd.read_parquet(daily_path2).reset_index(drop=True)
-            df2["date"] = pd.to_datetime(df2["date"])
-        tr_mask = df2["date"].dt.year <= 2021
-        base2 = ["ret", "realized_vol"]
-        extra2 = [c for c in ["hl_range", "ret_intrabar"] if c in df2.columns]
-        feats2 = df2[base2 + extra2].to_numpy(np.float32)
-        mu2 = feats2[tr_mask].mean(0); sd2 = feats2[tr_mask].std(0) + 1e-8
-        fn2 = (feats2 - mu2) / sd2
-        def _export_windows(year_mask):
-            idx = df2.index[year_mask].tolist()
-            Xs, dates, rvs = [], [], []
-            for t in idx:
-                if t - WINDOW >= 0 and t + 1 < len(df2):
-                    Xs.append(fn2[t - WINDOW:t])
-                    dates.append(str(df2["date"].iloc[t].date()))
-                    rvs.append(float(df2["realized_vol"].iloc[t + 1]))
-            if not Xs:
-                return [], [], []
-            Xa = np.stack(Xs)
-            chunks = []
-            with torch.no_grad():
-                for i in range(0, len(Xa), bs):
-                    chunks.append(enc(torch.tensor(Xa[i:i+bs], device=dev))[:, -1, :].cpu().numpy())
-            E = np.concatenate(chunks, axis=0).tolist()
-            return E, dates, rvs
-        Etr2, dates_tr, rv_tr = _export_windows(tr_mask)
-        Eoos, dates_oos, rv_oos = _export_windows(df2["date"].dt.year >= 2022)
-        hv_thr = float(np.percentile(rv_oos, 67))
-        hv_label = [1 if v >= hv_thr else 0 for v in rv_oos]
-        json.dump({"embeddings": Eoos, "dates": dates_oos,
-                   "realized_vol": rv_oos, "hv_label": hv_label,
-                   "train_embeddings": Etr2, "train_realized_vol": rv_tr},
-                  open("embeddings.json", "w"))
-        print("exported embeddings.json  train=%d  oos=%d  HV=%d/%d" % (
-            len(Etr2), len(Eoos), sum(hv_label), len(hv_label)))
-    # ── END EXPORT BLOCK ─────────────────────────────────────────────────────
-
-
-if __name__ == "__main__":
-    main()