docs: add .skills/README.md listing active skills

docs: add model/README.md — Python perimeter policy and setup
docs: add notebooks/README.md — EDA-only policy, output stripping rule
2026-05-27 22:00:32 +00:00 · 2026-05-27 22:00:25 +00:00 · 2026-05-27 22:00:17 +00:00 · 2026-05-27 22:00:10 +00:00 · 2026-05-27 22:00:02 +00:00 · 2026-05-27 21:59:53 +00:00
35 changed files with 642 additions and 2055 deletions
@@ -4,101 +4,77 @@
 - **Name**: jepa-fx-risk
 - **Owner**: Mathias
- **Client**: personal research
+- **Client**: personal / research
 - **Repo**: gitea.d-ma.be/mathias/jepa-fx-risk
- **Status**: active — Phase 0
+- **Status**: active
 - **Brain wing**: `jepa-fx` (`wiki/jepa-fx/`)
 ## Purpose
-Research project: apply JEPA-based self-supervised representation learning to
+Research project. Not a product. The "user" is future-self and research readers. Success is a reproducible, publishable result — not deployment.
 FX risk management for a corporate bank with an internal global FX trading desk.
 Primary tasks: FX volatility forecasting and VaR/CVaR estimation.
 Target output: internal PoC for the trading desk.
 ## Architecture decision (see DECISIONS.md ADR-001)
 **TS-JEPA + SIGReg** — TS-JEPA temporal patchwise architecture (Ennadir et al.,
 arXiv:2509.25449) with EMA replaced by Sketched Isotropic Gaussian Regularization
 (SIGReg, Balestriero & LeCun, arXiv:2511.08544). Single search axis: λ ∈ [0.01, 1.0].
 Phase 2 hypothesis: MTS-JEPA multi-resolution objective (arXiv:2602.04643).
 ## Stack
-**Go** (`src/`): data pipeline (DUKASCopy fetch + hourly processing), evaluation
+- **Primary language**: Go 1.24+ (pipeline, eval harness, CLI, experiment runner)
-harness (silhouette, linear probe R², collapse diagnostic, Kupiec/Christoffersen),
+- **ML layer**: Python 3.12 + PyTorch (TS-JEPA training loop only — isolated in `model/`)
-results dashboard (Templ + HTMX + CDN Tailwind).
+- **Build**: Task (Taskfile.yml)
 - **Target infra**: koala (Arch + Blackwell GPU, training), iguana (Mac Studio M2, dev)
 - **MCP**: brain (knowledge), gitea (version control)
-**Python** (`model/`): all training and embedding export only. PyTorch cu130
+## Research context
 (Blackwell sm_120 compatible). No evaluation logic in Python.
-**Infra**: koala (Arch Linux, Blackwell GPU 12 GB VRAM) for training.
+**Primary hypothesis:**
-iguana (Mac Studio M2 Ultra) + LiteLLM on piguard for autoresearch agent LLM.
+> JEPA embeddings trained on FX time-series will produce latent market-state representations structurally separable by regime without explicit labels, measurable by silhouette score > 0.35 on k-means clusters vs. realised-volatility regime labels, on held-out data including at least one structural break.
-## Repository layout
+**Current phase:** Phase 0 — SSL Feasibility Gate
-```
+**Training cutoff:** 2023-01-01 (hard — never look at post-2023 data during development)
 jepa-fx-risk/
 ├── src/                    # Go — data pipeline + eval harness + dashboard
 │   ├── data/               # DUKASCopy fetch, hourly processing, validation
 │   └── eval/               # silhouette, linear probe, collapse, backtest
 ├── model/                  # Python — training only
 │   ├── train.py            # TS-JEPA + SIGReg backbone (autoresearch edits this)
 │   ├── prepare.py          # LOCKED — data loading, tokenization, export
 │   └── requirements.txt
 ├── specs/                  # Research specs (one per phase/experiment type)
 ├── experiments/            # Per-run outputs: embeddings, metrics.json, git tag
 ├── results/summaries/      # Human-readable outcome per experiment
 ├── program.md              # Autoresearch agenda — researcher edits this
 ├── DECISIONS.md            # Architecture Decision Records
 └── Taskfile.yml            # task data:fetch, task experiment:run, task eval:*
 ```
-## Phase structure
+**Out of scope:** options pricing, directional alpha, live/paper trading, exotic pairs (Phases 1–3)
- **Phase 0** (current): MAE baseline on EUR/USD hourly 2008–2022.
+**Brain wing for prior decisions and failure modes:** `brain_query wing=jepa-fx`
  Gate: silhouette > 0.20, MAE > PCA, ±10% over 3 reruns.
 - **Phase 1**: TS-JEPA + SIGReg autoresearch sweep, 50 experiments.
  Gate: val_vol_r2 > GARCH baseline AND Kupiec p > 0.05.
 - **Phase 2**: MTS-JEPA multi-resolution hypothesis.
 - **Phase 3**: Internal bank tick/position data (future, out of current scope).
-## Data
+## Conventions
-DUKASCopy hourly OHLCV, 10 G10 pairs, 2008–2022 train / 2023 val / 2024 test.
+### Scientific discipline
-Features: log-return, log rolling-20-period HV, VIX (daily interpolated).
+- Every experiment has a spec in `specs/` before any code runs
-Weekend gaps handled explicitly. See ADR-002.
+- Hypotheses are falsifiable and have quantitative acceptance criteria
 - Null results are recorded and published — not discarded
 - Training cutoff is sacred — post-2023 data never informs any design decision
 - Results reported with baselines; no cherry-picking
-## Key conventions
+### Code style
 - Go: `gofumpt`, `golangci-lint` with project config; table-driven tests; `testify`
 - Errors: `fmt.Errorf("context: %w", err)` — no naked returns
 - Python: `ruff` for lint; type hints throughout; `pytest` for tests
 - No Jupyter notebooks for anything reproducible — notebooks are EDA scratch only
- `prepare.py` is LOCKED — never modified by agents or autoresearch
+### Git
- Evaluation metrics are always computed by the Go harness, never in Python
+- Conventional commits: `feat:`, `fix:`, `chore:`, `docs:`, `experiment:`, `result:`
- Every experiment gets a git tag: `exp/YYYYMMDD-description`
+- Branch: `feat/`, `experiment/phase-N-description`, `fix/`
- Null results are recorded explicitly in `results/summaries/` — do not iterate silently
+- Every experiment run gets a git tag: `exp/YYYYMMDD-short-description`
- `program.md` is the only file the researcher edits to steer autoresearch
+- PRs: one concern per PR; description explains *why* not *what*
 - CI runs `task check` (lint + vet + test) only — no GPU, no training
-## Evaluation metrics
+### Experiment discipline
 - One spec per phase in `specs/` — written before any implementation
 - Each run recorded in `experiments/YYYYMMDD-HHMMSS-description/`
 - Metric summaries committed to `results/summaries/` — large outputs gitignored
 - `task check` must pass before any commit
-Primary (autoresearch optimizes): `val_vol_r2` — linear probe R² on 1-day
+### Security / data
-realized volatility from frozen embeddings, computed by Go harness.
+- No raw FX data committed (gitignored) — see `data/README.md` for reproducible download
 - No API keys or tokens in code — env vars only
 - Training data and results stay local — nothing to cloud unless explicitly decided
-Secondary (logged, not optimized): Kupiec p-value (VaR 99% backtest on EUR/USD).
+## Agent instructions
 Must co-move with val_vol_r2 — checked from experiment 1.
-Diagnostics: silhouette score (regime clustering), PC1/HV correlation (collapse check).
+When acting as a coding agent on this project:
-## Benchmarks to beat (trading desk comparison)
+1. Read this file and all `SKILL.md` files in `.skills/` before starting work
-
+2. Run `brain_query wing=jepa-fx` to load current decisions and failure modes
- GARCH(1,1) — volatility forecasting baseline
+3. Run `task check` before every commit (lint + vet + test)
- Historical Simulation VaR (250-day rolling) — Basel default
+4. Check `DECISIONS.md` before making any architecture or methodology choice
- EWMA RiskMetrics (λ=0.94)
+5. Every experiment needs a spec in `specs/` — no specless experiments
-
+6. Never touch post-2023 data during development; it is sealed
-## Agent guidance
+7. Record null results honestly — do not iterate until metrics pass without noting it
-
+8. When adding a Python dependency, justify it; prefer pure Go alternatives
 Read `DECISIONS.md` before making architecture suggestions.
 Do not modify `prepare.py` or `src/eval/`.
 Do not suggest changing the Python/Go separation.
 Training runs are always manual via `task experiment:run` — never triggered by CI.
 When implementing, follow Go conventions in `.skills/go-patterns/SKILL.md`.
@@ -1,8 +1,18 @@
 {
  "mcpServers": {
-    "knowledge": {
+    "brain": {
-      "url": "http://localhost:3100/mcp",
+      "type": "http",
-      "description": "Project knowledge base — vector + graph retrieval"
+      "url": "https://brain-mcp.d-ma.be/mcp",
      "headers": {
        "Authorization": "Bearer ${BRAIN_MCP_TOKEN}"
      }
    },
    "gitea": {
      "type": "http",
      "url": "https://git-mcp.d-ma.be/mcp",
      "headers": {
        "Authorization": "Bearer ${GITEA_MCP_TOKEN}"
      }
    }
  }
 }
@@ -1,36 +1,39 @@
-# ---> Go
+# Go build artifacts
-# If you prefer the allow list template instead of the deny list, see community template:
+bin/
 # https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
 #
 # Binaries for programs and plugins
 *.exe
 *.exe~
 *.dll
 *.so
 *.dylib
 # Test binary, built with `go test -c`
 *.test
 # Output of the go coverage tool, specifically when used with LiteIDE
 *.out
-# Dependency directories (remove the comment below to include it)
+# Go workspace
 # vendor/
 # Go workspace file
 go.work
 go.work.sum
-# env file
+# Environment
 .env
-# Project-specific
+# Python
-bin/
+model/.venv/
-*.templ.go
+model/__pycache__/
 model/**/__pycache__/
 model/**/*.pyc
 model/.pytest_cache/
 model/**/.pytest_cache/
 model/.ruff_cache/
-# python venv (autoresearch loop)
+# Data — never commit raw or processed FX data
-.venv/
+data/raw/
 data/processed/
 data/cache/
-# downloaded + processed market data (track via DVC/MinIO, #10 — not git)
+# Experiment outputs — commit summaries only (results/summaries/)
-data/
+experiments/*/embeddings/
 experiments/*/checkpoints/
 experiments/*/logs/
 # Large results — commit metric tables and figures only
 results/raw/
 # Notebooks — never commit outputs
 notebooks/**/.ipynb_checkpoints/
 # OS
 .DS_Store
@@ -0,0 +1,13 @@
 This directory contains skill symlinks for this project.
 Agents: load all SKILL.md files in subdirectories before starting work.
 Active skills:
 - experiment-spec  — write experiment specs before running any phase
 - feature-spec     — write component specs before implementing within a phase
 - tdd              — acceptance criteria map to tests; red-green-refactor
 - grill-me         — stress-test specs and hypotheses before committing
 - session-retrospective — after each phase, surface learnings for the brain
 - clean-code       — Go and Python style conventions
 - planning         — break phases into trackable tasks
 - debug            — systematic debugging approach
@@ -1,13 +1,5 @@
-# hostexecutor
+# jepa-fx-risk — Agent context
 # Auto-generated from .context/PROJECT.md by `task context:sync`
 # Do not edit directly.
-## Identity
+See .context/PROJECT.md for the canonical source.
 - **Name**: hostexecutor
 - **Owner**: Mathias
 - **Client**: personal
 - **Repo**: gitea.d-ma.be/mathias/hostexecutor
 - **Status**: active
 ## Stack
 Go + Templ + HTMX + CDN Tailwind. See `~/dev/.context/AGENT.md` for cross-project conventions.
@@ -1,13 +1,5 @@
-# hostexecutor
+# jepa-fx-risk — Agent context
 # Auto-generated from .context/PROJECT.md by `task context:sync`
 # Do not edit directly.
-## Identity
+See .context/PROJECT.md for the canonical source.
 - **Name**: hostexecutor
 - **Owner**: Mathias
 - **Client**: personal
 - **Repo**: gitea.d-ma.be/mathias/hostexecutor
 - **Status**: active
 ## Stack
 Go + Templ + HTMX + CDN Tailwind. See `~/dev/.context/AGENT.md` for cross-project conventions.
@@ -1,215 +1,120 @@
-# Architecture Decision Records
+# DECISIONS.md
-This file records significant technical and research decisions for `jepa-fx-risk`.
+Architecture and methodology decisions for `jepa-fx-risk`. Every non-obvious choice lives here with its rationale. Agents: read this before making any design decision.
-Each record is immutable once merged — append new records rather than editing old ones.
+
-Format: ID · Date · Status · Context · Decision · Rationale · Consequences.
+Last updated: 2026-05-27
 ---
-## ADR-001 · Architecture: TS-JEPA + SIGReg as Phase 1 backbone
+## Language split: Go-first, Python-minimal
-**Date:** 2026-05-28
+**Decision:** Data pipeline, evaluation harness, CLI, and experiment runner in Go 1.24+. Python 3.12 + PyTorch only for the TS-JEPA model training loop, isolated in `model/`.
 **Status:** Accepted
 **Supersedes:** informal decision to use TS-JEPA standalone (pre-ADR)
-### Context
+**Rationale:** Python's dependency ecosystem is a reliability risk for a multi-year project (conflicting CUDA versions, transitive breakage, environment drift). Go produces a single static binary, has excellent CSV/Parquet support, is fast enough for all non-training workloads, and keeps the reproducible parts of the project dependency-free. The training loop genuinely requires PyTorch — that is the one place Python is unavoidable.
-Four JEPA variants were evaluated for FX volatility forecasting and VaR/CVaR estimation:
+**Boundary:** `model/` is the Python perimeter. Nothing outside it imports Python.
 | Variant | Origin | Key property |
 |---|---|---|
 | TS-JEPA | Ennadir et al., Sep 2025 | Time-series native; EMA collapse prevention |
 | LeJEPA | Balestriero & LeCun, Nov 2025 | Proven optimal embeddings (isotropic Gaussian); SIGReg |
 | MTS-JEPA | He et al., Feb 2026 | Multi-resolution + codebook; no public code |
 | Var-JEPA | Multiple, Mar 2026 | ELBO-based UQ; no public code |
 Key constraints: 12 GB VRAM (Blackwell, koala), hourly DUKASCopy data, internal PoC target,
 autoresearch loop requires a clean single-scalar search space, trading desk requires an
 explainable theoretical story.
 ### Decision
 Use **TS-JEPA architecture with SIGReg replacing EMA** as the Phase 1 backbone.
 Concretely:
 - Start from the TS-JEPA open-source implementation (arXiv:2509.25449, GitHub)
 - Remove the EMA target-network mechanism
 - Replace it with Sketched Isotropic Gaussian Regularization (SIGReg) from LeJEPA
  (arXiv:2511.08544), controlled by a single λ hyperparameter
 - Keep TS-JEPA's temporal patchwise masking and Transformer encoder unchanged
 ### Rationale
 **Why not pure TS-JEPA:** EMA is a heuristic; λ interacts with EMA momentum and
 learning rate, creating a three-way search space that is hard to navigate with autoresearch.
 EMA also has no theoretical non-stationarity guarantee.
 **Why not pure LeJEPA:** The reference implementation targets vision (multi-crop views).
 Adapting it to temporal patchwise masking requires non-trivial surgery and moves away from
 open code. TS-JEPA's masking is already the right inductive bias for time series.
 **Why the hybrid:** SIGReg is architecture-agnostic — it operates on the embedding
 distribution, not the encoder structure. Swapping EMA for SIGReg is a ~20-line change to
 TS-JEPA's training loop. The result is:
 - Time-series native (TS-JEPA masking + patch structure)
 - Provably collapse-free without heuristics (SIGReg)
 - Single search axis for autoresearch (λ ∈ [0.01, 1.0])
 - Non-stationarity robustness proven formally (arXiv:2602.19373 extends LeJEPA
  guarantees to non-stationary target distributions — directly relevant to FX)
 - Explainable to a model validation team: "embeddings are provably optimal for
  downstream prediction under distributional uncertainty"
 **Why not MTS-JEPA or Var-JEPA now:** Both lack public code (as of May 2026).
 MTS-JEPA's multi-resolution objective is the right next hypothesis (see ADR-003).
 Var-JEPA's ELBO-based UQ is a compelling future direction for CVaR estimation.
 ### Consequences
 - Phase 0 (MAE baseline) is unaffected — it precedes the JEPA architecture choice
 - Issue #3 (TS-JEPA reproduction) is still the right first step; SIGReg is added after
  reproduction is confirmed
 - The autoresearch `program.md` primary search axis is λ (SIGReg weight)
 - Secondary axes: masking block size, patch stride, encoder depth
 - `model/requirements.txt` must include the SIGReg implementation (≈20 lines,
  can be vendored directly)
 ---
-## ADR-002 · Data: DUKASCopy hourly G10 FX as primary training data
+## Primary hypothesis
-**Date:** 2026-05-28
+**Decision:** Regime detection via embedding structural separability is the primary hypothesis. Distributional VaR forecasting is secondary and only pursued if Phase 1 and 2 succeed.
 **Status:** Accepted
-### Context
+**Rationale:** Regime detection is testable in Phase 1 without regulatory-quality outputs. If embeddings don't show regime structure, distributional VaR will also fail. Choosing a primary hypothesis prevents the project from retreating from one to the other on failure.
-Data scale is the most dangerous assumption for any SSL/JEPA approach. Daily FX data
+**Success criterion (Phase 1):** Silhouette score > 0.35 on k-means clusters (k=3–5) vs. realised-volatility regime label (rolling 30-day HV percentile, high/low), computed on held-out test data including at least one structural break.
 (~5,000 samples over 20 years) is insufficient for self-supervised pretraining.
 Two alternatives were considered: daily public data (yfinance) vs. hourly tick data
 (DUKASCopy, free, rate-limited HTTP API).
 ### Decision
 Use **DUKASCopy hourly OHLCV** as the primary data source.
 - 10 G10 pairs: EURUSD, GBPUSD, USDJPY, USDCHF, AUDUSD, NZDUSD, USDCAD,
  EURGBP, EURJPY, GBPJPY
 - Training window: 2008-01-01 – 2022-12-31 (~175,000 samples per pair)
 - Validation window: 2023-01-01 – 2023-12-31 (~2,600 samples)
 - Test window: 2024-01-01 – 2024-12-31 (held out, never seen during development)
 - Features per bar: log-return, log rolling-20-period HV, VIX (daily interpolated)
 - Weekend gaps handled explicitly — no interpolation across market close
 ### Rationale
 Hourly data gives ~35× more samples than daily. This is the minimum threshold for
 JEPA-style SSL to show a training signal within 10-minute autoresearch experiments.
 DUKASCopy is free, reliable, and provides consistent tick-level source data back to 2003.
 ### Consequences
 - The Go data pipeline (Issue #2) is the critical path for everything else
 - Phase 0 MAE baseline trains on the same 2008-2022 window
 - Daily data (yfinance) may still be used for VIX and rate differentials as auxiliary features
 ---
-## ADR-003 · Research roadmap: Phase structure and JEPA variant progression
+## Training data: 2008–2022, all G10 pairs
-**Date:** 2026-05-28
+**Decision:** Train on 15 years of all G10 FX pairs from DUKASCopy, 2008–2022. Do not start with a single 5-year EUR/USD window.
 **Status:** Accepted
-### Decision
+**Rationale:** A single 5-year window (e.g. 2019–2024) is dominated by one or two regimes and gives the encoder insufficient regime diversity to learn regime-sensitive representations. Training on 2008–2022 ensures the encoder sees: GFC (2008), EUR sovereign debt (2011–2012), SNB cap removal (2015), COVID (2020), USD rate cycle (2022). Multi-pair training also allows cross-currency transfer (Phase 4).
-Three-phase research roadmap:
+**Alternative considered:** Start simple with EUR/USD only, expand later. Rejected because regime diversity in training is a structural requirement, not a nice-to-have. Retrofitting it in Phase 2 would require retraining from scratch.
 **Phase 0 — SSL feasibility gate (MAE baseline)**
 Implement a 1D temporal MAE (not JEPA) on EUR/USD hourly 2008-2022.
 Gate criteria: silhouette > 0.20 on 2023 held-out, MAE > PCA baseline, ±10% over 3 reruns.
 Purpose: validate that the data and eval harness work before committing to JEPA complexity.
 If gate fails: follow null result protocol in `specs/phase-0-ssl-feasibility.md`.
 **Phase 1 — TS-JEPA + SIGReg autoresearch sweep**
 Primary architecture per ADR-001.
 Autoresearch loop: `program.md`-driven, 10-min experiments, 50-experiment budget.
 Primary metric: `val_vol_r2` (linear probe R² on 1-day realized volatility).
 Gate criteria: `val_vol_r2` > GARCH-implied baseline AND Kupiec p-value > 0.05 on
 EUR/USD VaR 99%.
 Kupiec is logged from experiment 1 to verify it co-moves with `val_vol_r2`.
 **Phase 2 — MTS-JEPA multi-resolution hypothesis**
 Introduce parallel multi-scale predictive pathways (1h, 8h, 24h context windows)
 adapted from MTS-JEPA (arXiv:2602.04643).
 Hypothesis: multi-scale representations improve regime detection (silhouette) and
 reduce VaR exceedance clustering (Christoffersen test).
 Prerequisite: Phase 1 gate passed AND MTS-JEPA code available or reproducible from paper.
 Time-box: if MTS-JEPA code not available within 4 weeks of Phase 2 start, implement
 multi-resolution masking from scratch using Phase 1 backbone as base.
 **Phase 3 — Internal bank data (future)**
 Replace DUKASCopy pipeline with internal tick feed adapter.
 Fine-tune heads only; backbone frozen or lightly fine-tuned.
 Out of scope for current PoC cycle.
 ### Consequences
 - Issue #5 (Phase 0 MAE) is the unblocked next executable step
 - Phase 1 autoresearch is blocked until Phase 0 passes its gate
 - Var-JEPA (ELBO-based UQ) is a named future hypothesis for CVaR estimation in Phase 2+
  but not on the critical path
 ---
-## ADR-004 · Evaluation: Go harness + Python training separation
+## Hard training cutoff: 2023-01-01
-**Date:** 2026-05-28
+**Decision:** All data from 2023-01-01 onward is sealed. No architecture, hyperparameter, or methodology decision may be informed by post-2023 data. Post-2023 test set opened only once, for final evaluation.
 **Status:** Accepted
-### Decision
+**Rationale:** Out-of-sample integrity is essential for publishability and honest self-assessment. The held-out window (2023–2026) includes: 2023 US regional bank stress, 2024 JPY intervention episodes. These are the test of genuine generalisation.
-Hard separation between training (Python) and evaluation (Go):
+**Enforcement:** `data/raw/` is gitignored. The download script hard-stops at 2022-12-31 for training splits. Any deviation requires a DECISIONS.md entry explaining why.
 - **Python** (`model/`): all training, embedding export, model checkpointing
 - **Go** (`src/eval/`): all evaluation metrics — silhouette, linear probe R², collapse
  diagnostic, Kupiec/Christoffersen backtests
 - Interface: Python exports embedding matrices + labels to `experiments/RUNID/` as
  `.npy` files; Go eval harness reads them and writes `metrics.json`
 ### Rationale
 Go evaluation gives deterministic, fast, auditable metric computation with proper
 unit tests. It decouples the experimental loop from the training framework, making
 it possible to re-evaluate any past experiment without re-running training.
 The Go layer also serves as the foundation for the eventual trading desk dashboard.
 ### Consequences
 - All acceptance criteria in Issues #4 and #5 are specified in terms of Go eval outputs
 - `val_vol_r2` (the autoresearch optimization metric) is computed by the Go harness,
  not inside the Python training loop
 - Python training loop calls `task eval:probe` as a subprocess after each experiment
  to get the scalar fed back to autoresearch
 ---
-## ADR-005 · Compute: Blackwell GPU on koala, PyTorch cu130
+## Phase 0: SSL feasibility gate before JEPA work
-**Date:** 2026-05-28
+**Decision:** Before any JEPA-specific implementation, run a Phase 0 experiment: masked autoencoder (MAE) baseline on EUR/USD hourly data. If MAE silhouette < 0.20, SSL-based regime detection is likely not feasible on this data — stop and investigate before proceeding to JEPA.
 **Status:** Accepted
-### Decision
+**Rationale:** JEPA's complexity is only justified if the core SSL premise (that latent representations capture regime structure) holds for FX data. A failed MAE experiment tells us this in 2 weeks rather than 4 months. Added after Full Grill session (2026-05-27).
-All GPU training runs on koala (Arch Linux, Blackwell GPU, 12 GB VRAM).
+**Go/no-go threshold:** MAE silhouette < 0.20 on held-out 2023 data → pause, investigate, do not proceed to Phase 1.
 PyTorch install: `pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu130`
 (CUDA 13.0 wheel — required for sm_120 Blackwell support; stable as of May 2026).
 Driver requirement: NVIDIA R570+, CUDA toolkit 12.8+.
-Ollama on iguana (Mac Studio M2 Ultra) serves the autoresearch agent LLM via the
+---
 existing LiteLLM proxy on piguard. Agent calls never hit koala directly.
-### Consequences
+## Architecture: TS-JEPA as starting implementation
- `model/requirements.txt` must NOT pin torch to a cu124 or earlier wheel
+**Decision:** Use TS-JEPA (Ennadir et al., 2025) as the starting JEPA implementation. MTS-JEPA (He et al., 2026) is the upgrade path if multi-resolution proves necessary.
- CI (Issue #7) must NOT run GPU tests — CPU-only for unit tests, GPU only via
+
-  `task experiment:run` on koala
+**Rationale:** TS-JEPA is simpler. Validate the concept before adding multi-resolution complexity. If Phase 1 succeeds with TS-JEPA, MTS-JEPA is an ablation, not a prerequisite.
- 12 GB VRAM is sufficient for <5M parameter models at batch=64; monitor if
+
-  autoresearch explores larger architectures
+**Risk:** Both are preprints. Code reproducibility is unconfirmed. First task of Phase 0 is reproducing TS-JEPA on the paper's own benchmark — if this takes > 2 weeks, contact authors or fall back to implementing JEPA masking from scratch using V-JEPA codebase as reference.
 ---
 ## Input features: minimal for Phase 0/1
 **Decision:** Phase 0 and Phase 1 use three features only: log-return (hourly), rolling 20-period realised volatility (hourly), VIX (daily, interpolated to hourly).
 **Rationale:** Too many input features in early phases makes it impossible to distinguish "JEPA learned regime structure" from "JEPA learned to encode a feature that correlates with regime." Minimal features reduce confounding.
 **Expansion path:** Add DXY, G10 vol surface, yield spreads in Phase 2 if Phase 1 succeeds.
 ---
 ## Collapse diagnostic
 **Decision:** If PC1 of JEPA embeddings correlates > 0.85 with rolling 30-day HV, treat Phase 1 as a partial failure — the encoder learned volatility level, not regime structure. This is a useful finding but not the hypothesis.
 **Rationale:** EUR/USD hourly returns are strongly heteroskedastic. A JEPA encoder trained to predict future embeddings will strongly tend to encode current volatility as its primary latent dimension. This is predictable, not regime-sensitive. The linear probe and PC1 diagnostic together distinguish "learned volatility" from "learned regime."
 ---
 ## Linear probe as mandatory Phase 1 exit gate
 **Decision:** Train a linear model on frozen JEPA embeddings to predict realised volatility decile. R² < 0.4 → embeddings are not encoding useful risk structure → do not proceed to Phase 2.
 **Rationale:** If a simple linear model cannot extract volatility regime from the embeddings, the representations are not useful for risk management purposes regardless of their silhouette score. Interpretability-by-linear-probe is the minimum bar for any downstream use.
 ---
 ## Explicit out-of-scope (Phases 1–3)
 The following are explicitly out of scope for this research programme and go to a parking lot if they arise:
 - Options / derivatives pricing
 - Directional alpha generation
 - Live or paper trading
 - Exotic pairs beyond G10
 - Institutional deployment
 - Real-time inference systems
 ---
 ## Experiment spec required before any experiment
 **Decision:** Every experiment phase must have a written spec in `specs/` before any code runs. No specless experiments.
 **Rationale:** Consistent with spec-driven-dev way of working. Specs force falsifiable hypothesis statement, quantitative acceptance criteria, and explicit null-result protocol before results are known — preventing post-hoc rationalisation.
 ---
 ## Null results are results
 **Decision:** Null results (hypothesis rejected) are recorded in `results/summaries/` and treated as valid research outputs, not failures to be iterated away silently.
 **Rationale:** A confirmed null result (e.g. "SSL cannot find regime structure in FX hourly data") is publishable and scientifically valuable. Iterating hyperparameters until metrics pass without recording the failed attempts is p-hacking. Humble attitude; scientific approach.
@@ -1,13 +1,66 @@
-# hostexecutor
+# jepa-fx-risk
-> Generated from `mathias/template-go-web`.
+Research project exploring JEPA (Joint Embedding Predictive Architecture) as a framework for latent representation learning applied to FX trading risk management.
-## Bootstrap
+**Primary hypothesis:** JEPA embeddings trained on FX time-series will produce latent market-state representations that are structurally separable by regime without explicit regime labels, measurable by silhouette score on k-means clusters validated against held-out realised-volatility regime labels.
-After creating from template, run:
+**Current phase:** Phase 0 — SSL Feasibility Gate (not yet started)
 **Brain wing:** `jepa-fx` — query via `brain_query wing=jepa-fx`
 ---
 ## Architecture
 This is a **Go-first, Python-minimal** research project.
 | Layer | Language | Rationale |
 |---|---|---|
 | Data pipeline | Go | Type-safe, fast, no dependency hell |
 | Experiment runner | Go | CLI tooling, reproducible invocations |
 | Evaluation harness | Go | Silhouette, linear probe, collapse diagnostics |
 | Model training | Python + PyTorch | TS-JEPA requires it; kept minimal and isolated |
 ## Structure
 ```
 specs/          experiment specs (one per phase)
 src/            Go packages: pipeline, eval, cmd
 model/          Python: TS-JEPA training loop only
 experiments/    one directory per run (gitignored except summaries)
 results/        tracked: metric tables, key figures
 data/           gitignored; see data/README.md for download instructions
 notebooks/      EDA only; outputs stripped before commit
 docs/           ADRs and research notes
 ```
 ## Quick start
 ```bash
-go mod tidy       # regenerate go.sum with real module path
+task data:fetch          # download DUKASCopy G10 tick data
-task generate     # generate templ files
+task pipeline:build      # build Go data pipeline
-task build        # build the binary
+task check               # lint + vet + test
 task experiment:run -- --spec specs/phase-0-ssl-feasibility.md
 ```
 ## Phases
 | Phase | Goal | Status |
 |---|---|---|
 | 0 | SSL feasibility gate — MAE baseline on FX data | 🔲 Not started |
 | 1 | JEPA representation PoC — silhouette > 0.35 | 🔲 Not started |
 | 2 | Regime detection validation — recall > 70%, lead > 5d | 🔲 Not started |
 | 3 | Distributional forecasting — VaR passes Kupiec | 🔲 Not started |
 | 4 | Multi-pair transfer — exotic pair improvement | 🔲 Not started |
 ## Key constraints
 - **Training cutoff:** 2023-01-01 — all post-2023 data is sealed for final evaluation
 - **Training data:** 2008–2022, all G10 pairs, DUKASCopy
 - **Out of scope:** options pricing, directional alpha, live trading, exotic pairs (Phases 1–3)
 - **End-state:** publishable research result — not institutional deployment
 ## Related
 - Brain wing: `wiki/jepa-fx/` — decisions, hypotheses, failure modes
 - Synthesis document: `JEPA_FX_Risk_Synthesis.docx` (session 2026-05-27)
@@ -1,18 +0,0 @@
 # Autoresearch STATUS
 | iter | val_vol_r2 | delta | action | secs | gpu | change |
 |------|-----------|-------|--------|------|-----|--------|
 | 1 | 0.3749 | +0.0928 | KEEP | 2s | gpu=0% vram=10054/12227MiB temp=34°C | iter1 |
 | 1 | 0.3011 | +0.0776 | KEEP | 2s | gpu=0% vram=10054/12227MiB temp=34°C | iter1 |
 | 2 | 0.3032 | +0.0021 | KEEP | 2s | gpu=0% vram=10054/12227MiB temp=35°C | iter2 |
 | 1 | 0.2759 | -0.0273 | revert | 2s | gpu=0% vram=10054/12227MiB temp=34°C | iter1 |
 | 2 | 0.3442 | +0.0410 | KEEP | 2s | gpu=0% vram=10054/12227MiB temp=34°C | iter2 |
 | 3 | 0.3371 | -0.0071 | revert | 2s | gpu=0% vram=10054/12227MiB temp=34°C | iter3 |
 | 4 | 0.3143 | -0.0299 | revert | 2s | gpu=0% vram=10054/12227MiB temp=34°C | iter4 |
 | 5 | 0.3355 | -0.0087 | revert | 2s | gpu=0% vram=10054/12227MiB temp=34°C | iter5 |
 | 6 | 0.2377 | -0.1065 | revert | 2s | gpu=0% vram=10054/12227MiB temp=35°C | iter6 |
 | 1 | -0.1247 | +0.0296 | KEEP | 4s | gpu=0% vram=10054/12227MiB temp=35°C | iter1 |
 | 2 | -0.1203 | +0.0044 | KEEP | 4s | gpu=0% vram=10054/12227MiB temp=35°C | iter2 |
 | 3 | -0.0716 | +0.0487 | KEEP | 4s | gpu=0% vram=10054/12227MiB temp=36°C | iter3 |
 | 4 | 0.0590 | +0.1306 | KEEP | 5s | gpu=0% vram=10054/12227MiB temp=36°C | iter4 |
 | 5 | 0.0599 | +0.0009 | KEEP | 5s | gpu=0% vram=10054/12227MiB temp=37°C | iter5 |
@@ -1,49 +1,79 @@
 version: '3'
 tasks:
-  generate:
+
-    desc: Run templ generate
+  # ── Quality gate ──────────────────────────────────────────────────────────
-    cmds: [templ generate]
+
-  build:
+  check:
-    desc: Build all binaries
+    desc: "Full quality gate: lint + vet + test (run before every commit)"
    deps: [generate]
    cmds:
-      - go build -o bin/jepa-fx-risk ./cmd/jepa-fx-risk
+      - golangci-lint run ./src/...
-      - go build -o bin/eval ./cmd/eval
+      - go vet ./src/...
-  run:
+      - go test ./src/... -race -count=1
-    deps: [build]
+      - cd model && python -m pytest tests/ -q
-    cmds: [./bin/jepa-fx-risk]
+
  test:
-    desc: Run all tests
+    desc: Run Go tests only
-    deps: [generate]
+    cmds: [go test ./src/... -race]
-    cmds: [go test ./... -race]
+
  lint:
    desc: Lint Go code
    cmds: [golangci-lint run ./src/...]
  # ── Data ─────────────────────────────────────────────────────────────────
  data:fetch:
    desc: "Download G10 tick data from DUKASCopy (2003–2022)"
    cmds: [go run ./src/cmd/fetch --config data/config.yaml]
  data:process:
    desc: "Resample ticks → hourly OHLCV + features (log-return, rolling HV)"
    cmds: [go run ./src/cmd/process --input data/raw --output data/processed]
  data:validate:
    desc: "Validate processed data: gap detection, outlier report, regime coverage"
    cmds: [go run ./src/cmd/validate --input data/processed]
  # ── Experiment ───────────────────────────────────────────────────────────
  experiment:run:
    desc: "Run an experiment from a spec file. Usage: task experiment:run -- --spec specs/phase-0-ssl-feasibility.md"
    cmds: [go run ./src/cmd/experiment {{.CLI_ARGS}}]
  experiment:list:
    desc: List all recorded experiment runs
    cmds: [ls -lt experiments/ | head -20]
  # ── Evaluation ───────────────────────────────────────────────────────────
  eval:silhouette:
    desc: "Compute silhouette score on embedding output. Usage: task eval:silhouette -- --run experiments/RUNID"
    cmds: [go run ./src/cmd/eval silhouette {{.CLI_ARGS}}]
  eval:probe:
-    desc: "Run linear-probe (val_vol_r2) on embeddings from metrics.json"
+    desc: "Run linear probe on frozen embeddings vs. realised-vol decile"
-    cmds: [./bin/eval -metric probe]
+    cmds: [go run ./src/cmd/eval probe {{.CLI_ARGS}}]
-  eval:silhouette:
+
    desc: "Run silhouette on embeddings vs binary HV labels"
    cmds: [./bin/eval -metric silhouette]
  eval:collapse:
-    desc: "Run effective-rank collapse diagnostic"
+    desc: "Check collapse diagnostic: PC1 correlation with rolling HV"
-    cmds: [./bin/eval -metric erank]
+    cmds: [go run ./src/cmd/eval collapse {{.CLI_ARGS}}]
-  lint:
+
-    cmds: [golangci-lint run ./...]
+  # ── Model (Python) ───────────────────────────────────────────────────────
-  check:
+
-    desc: Lint, vet, and test (used by CI)
+  model:train:
-    deps: [generate]
+    desc: "Train TS-JEPA model. Usage: task model:train -- --config model/configs/phase0.yaml"
    dir: model
    cmds: [python train.py {{.CLI_ARGS}}]
  model:setup:
    desc: "Create Python venv and install model dependencies (uv)"
    dir: model
    cmds:
-      - golangci-lint run ./...
+      - uv venv .venv
-      - go vet ./...
+      - uv pip install -r requirements.txt
-      - go test ./... -race -count=1
+
  # ── Context ──────────────────────────────────────────────────────────────
  context:sync:
-    desc: Regenerate all harness-specific context files
+    desc: "Regenerate CLAUDE.md and AGENTS.md from .context/PROJECT.md"
-    cmds:
+    cmds: [bash scripts/context-sync.sh]
      - bash scripts/context-sync.sh
  context:sync:claude:
    cmds: [bash scripts/context-sync.sh claude]
  context:sync:agents:
    cmds: [bash scripts/context-sync.sh agents]
  context:sync:cursor:
    cmds: [bash scripts/context-sync.sh cursor]
@@ -1,168 +0,0 @@
 // cmd/eval — CLI driver for the jepa-fx-risk evaluation harness.
 //
 //	./bin/eval -metric probe|silhouette|erank [-emb embeddings.json]
 //
 // embeddings.json format (from train.py EXPORT_EMBEDDINGS=1):
 //
 //	{
 //	  "embeddings":         [[...], ...],  // OOS frozen embeddings
 //	  "realized_vol":       [...],         // OOS target (next-day RV)
 //	  "hv_label":           [...],         // binary HV label (top-33%)
 //	  "train_embeddings":   [[...], ...],  // train-set frozen embeddings
 //	  "train_realized_vol": [...]          // train-set RV targets
 //	}
 //
 // eval:probe standardises both sets using train statistics (no leakage).
 // Falls back to internal 70/30 split of OOS if train_embeddings absent.
 package main
 import (
 	"encoding/json"
 	"flag"
 	"fmt"
 	"log/slog"
 	"math"
 	"os"
 	"gitea.d-ma.be/mathias/jepa-fx-risk/internal/eval"
 )
 func main() {
 	metric := flag.String("metric", "probe", "probe | silhouette | erank")
 	embFile := flag.String("emb", "embeddings.json", "path to embeddings JSON")
 	flag.Parse()
 	log := slog.New(slog.NewJSONHandler(os.Stdout, nil))
 	d, err := readJSON(*embFile)
 	if err != nil {
 		log.Error("load embeddings", "err", err)
 		os.Exit(1)
 	}
 	log.Info("loaded", "oos", len(d.Embeddings), "dim", len(d.Embeddings[0]),
 		"train", len(d.TrainEmbeddings), "metric", *metric)
 	switch *metric {
 	case "probe":
 		var r2 float64
 		if len(d.TrainEmbeddings) > 0 {
 			// standardise both sets using train statistics to prevent leakage
 			trEmb, mu, sd := standardiseCompute(d.TrainEmbeddings)
 			oosEmb := applyStandardise(d.Embeddings, mu, sd)
 			r2 = eval.LinearProbeTrainTest(trEmb, d.TrainRealizedVol, oosEmb, d.RealizedVol, 1e-3)
 			log.Info("probe mode", "fit_on", "train_embeddings", "eval_on", "oos")
 		} else {
 			// fallback: internal 70/30 split of OOS embeddings
 			oosEmb, mu, sd := standardiseCompute(d.Embeddings)
 			n70 := int(float64(len(oosEmb)) * 0.7)
 			oos70 := applyStandardise(d.Embeddings[n70:], mu, sd)
 			r2 = eval.LinearProbeTrainTest(oosEmb[:n70], d.RealizedVol[:n70],
 				oos70, d.RealizedVol[n70:], 1e-3)
 			log.Info("probe mode", "fit_on", "oos[0:70%]", "eval_on", "oos[70%:]")
 		}
 		fmt.Printf(`{"metric":"val_vol_r2","value":%.6f}`+"\n", r2)
 		log.Info("linear probe", "val_vol_r2", fmt.Sprintf("%.4f", r2))
 	case "silhouette":
 		if len(d.HVLabel) == 0 {
 			log.Error("silhouette requires hv_label in embeddings.json")
 			os.Exit(1)
 		}
 		oosEmb := standardise(d.Embeddings)
 		sil, err := eval.Silhouette(oosEmb, d.HVLabel)
 		if err != nil {
 			log.Error("silhouette", "err", err)
 			os.Exit(1)
 		}
 		fmt.Printf(`{"metric":"silhouette","value":%.6f}`+"\n", sil)
 		log.Info("silhouette", "score", fmt.Sprintf("%.4f", sil))
 	case "erank":
 		oosEmb := standardise(d.Embeddings)
 		er := eval.EffectiveRank(oosEmb)
 		fmt.Printf(`{"metric":"effective_rank","value":%.6f}`+"\n", er)
 		log.Info("effective rank", "erank", fmt.Sprintf("%.2f", er))
 	default:
 		log.Error("unknown metric", "metric", *metric)
 		os.Exit(1)
 	}
 }
 type embJSON struct {
 	Embeddings       [][]float64 `json:"embeddings"`
 	Dates            []string    `json:"dates"`
 	RealizedVol      []float64   `json:"realized_vol"`
 	HVLabel          []int       `json:"hv_label"`
 	TrainEmbeddings  [][]float64 `json:"train_embeddings"`
 	TrainRealizedVol []float64   `json:"train_realized_vol"`
 }
 func readJSON(path string) (*embJSON, error) {
 	f, err := os.Open(path)
 	if err != nil {
 		return nil, fmt.Errorf("open %s: %w", path, err)
 	}
 	defer func() { _ = f.Close() }()
 	var d embJSON
 	if err := json.NewDecoder(f).Decode(&d); err != nil {
 		return nil, fmt.Errorf("decode: %w", err)
 	}
 	if len(d.Embeddings) == 0 {
 		return nil, fmt.Errorf("empty embeddings in %s", path)
 	}
 	return &d, nil
 }
 // standardise centres + scales to zero mean / unit std; returns normalised rows.
 func standardise(rows [][]float64) [][]float64 {
 	out, _, _ := standardiseCompute(rows)
 	return out
 }
 // standardiseCompute centres + scales and returns (normalised, mu, sd) for reuse.
 func standardiseCompute(rows [][]float64) ([][]float64, []float64, []float64) {
 	if len(rows) == 0 {
 		return rows, nil, nil
 	}
 	n, dim := len(rows), len(rows[0])
 	mu := make([]float64, dim)
 	for _, r := range rows {
 		for j, v := range r {
 			mu[j] += v
 		}
 	}
 	for j := range mu {
 		mu[j] /= float64(n)
 	}
 	sd := make([]float64, dim)
 	for _, r := range rows {
 		for j, v := range r {
 			diff := v - mu[j]
 			sd[j] += diff * diff
 		}
 	}
 	for j := range sd {
 		sd[j] = math.Sqrt(sd[j]/float64(n)) + 1e-8
 	}
 	out := make([][]float64, n)
 	for i, r := range rows {
 		out[i] = make([]float64, dim)
 		for j, v := range r {
 			out[i][j] = (v - mu[j]) / sd[j]
 		}
 	}
 	return out, mu, sd
 }
 // applyStandardise normalises rows using pre-computed mu and sd.
 func applyStandardise(rows [][]float64, mu, sd []float64) [][]float64 {
 	out := make([][]float64, len(rows))
 	for i, r := range rows {
 		out[i] = make([]float64, len(r))
 		for j, v := range r {
 			out[i][j] = (v - mu[j]) / sd[j]
 		}
 	}
 	return out
 }
@@ -5,7 +5,7 @@ import (
 	"net/http"
 	"os"
-	"gitea.d-ma.be/mathias/jepa-fx-risk/internal/web"
+	"gitea.d-ma.be/mathias/hostexecutor/internal/web"
 )
 func main() {
@@ -0,0 +1,36 @@
 # Data
 Raw and processed FX data is **gitignored** — never committed.
 ## Download: DUKASCopy G10 tick data
 Use the provided fetch task:
 ```bash
 task data:fetch    # downloads raw tick data → data/raw/
 task data:process  # resamples to hourly OHLCV + features → data/processed/
 task data:validate # gap detection, outlier report, regime coverage check
 ```
 ## Expected structure (local only)
 ```
 data/
  raw/          G10 pairs, tick OHLCV, 2003–present (gitignored)
  processed/    Hourly log-returns, rolling HV, VIX-merged (gitignored)
  cache/        Intermediate artefacts (gitignored)
  config.yaml   Download configuration (committed)
 ```
 ## Pairs
 EUR/USD, GBP/USD, USD/JPY, USD/CHF, AUD/USD, USD/CAD, NZD/USD, EUR/GBP, EUR/JPY, EUR/CHF
 ## Splits
 | Split | Date range | Purpose |
 |---|---|---|
 | Train | 2008-01-01 – 2022-12-31 | Model training only |
 | Held-out test | 2023-01-01 – 2023-12-31 | Final evaluation — do not open until evaluation |
 **Training cutoff is hard: 2023-01-01. No post-2022 data informs any design decision.**
@@ -0,0 +1,32 @@
 # Experiments
 One directory per experiment run. Large outputs (embeddings, checkpoints, logs) are gitignored. Only metric summaries are committed to `results/summaries/`.
 ## Naming convention
 ```
 YYYYMMDD-HHMMSS-phase-N-short-description/
  config.yaml       parameters used for this run (committed via results/summaries/)
  metrics.json      final metric snapshot (committed via results/summaries/)
  embeddings/       (gitignored — large)
  checkpoints/      (gitignored — large)
  logs/             (gitignored — large)
 ```
 ## Git tag convention
 Every run that produces reportable metrics gets a tag:
 ```
 exp/YYYYMMDD-phase-N-description
 ```
 Example: `exp/20260601-phase-0-mae-baseline`
 ## Starting a run
 ```bash
 task experiment:run -- --spec specs/phase-0-ssl-feasibility.md --tag my-run-description
 ```
 The runner creates the directory, writes config, runs the experiment, and outputs metrics.json.
@@ -1,5 +1,7 @@
-module gitea.d-ma.be/mathias/jepa-fx-risk
+module gitea.d-ma.be/mathias/hostexecutor
 go 1.26
-require github.com/a-h/templ v0.3.1020
+require (
 	github.com/a-h/templ v0.2.778
 )
@@ -1,4 +0,0 @@
 github.com/a-h/templ v0.3.1020 h1:ypAT/L5ySWEnZ6Zft/5yfoWXYYkhFNvEFOeeqecg4tw=
 github.com/a-h/templ v0.3.1020/go.mod h1:A2DlK61v+K+NRoGnhmYbNYVmtYHcFO5/AisMvBdDxTM=
 github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
 github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
@@ -1,383 +0,0 @@
 // Package eval implements the Go evaluation harness for jepa-fx-risk (#4).
 // Three diagnostics on frozen embeddings exported from train.py:
 //   - LinearProbe  — val_vol_r2: OOS R² of a ridge probe predicting next-day realized vol
 //   - Silhouette   — mean silhouette score of embeddings vs a binary label (HV regime)
 //   - EffectiveRank — Roy's effective rank: exp(H(σ²)) where H is entropy of normalised singular values
 package eval
 import (
 	"errors"
 	"math"
 )
 // LinearProbeTrainTest fits ridge regression on (trainEmb, trainY) and evaluates
 // on (testEmb, testY). Returns OOS R². Use this for proper held-out evaluation.
 func LinearProbeTrainTest(trainEmb [][]float64, trainY []float64,
 	testEmb [][]float64, testY []float64, lambda float64) float64 {
 	n := len(trainEmb)
 	if n == 0 || len(testEmb) == 0 {
 		return 0
 	}
 	d := len(trainEmb[0])
 	p := d + 1
 	A := make([][]float64, n)
 	for i, e := range trainEmb {
 		row := make([]float64, p)
 		copy(row, e)
 		row[d] = 1.0
 		A[i] = row
 	}
 	AtA := make([][]float64, p)
 	for i := range AtA {
 		AtA[i] = make([]float64, p)
 	}
 	Aty := make([]float64, p)
 	for i := 0; i < n; i++ {
 		for j := 0; j < p; j++ {
 			Aty[j] += A[i][j] * trainY[i]
 			for k := 0; k < p; k++ {
 				AtA[j][k] += A[i][j] * A[i][k]
 			}
 		}
 	}
 	for j := 0; j < p; j++ {
 		AtA[j][j] += lambda
 	}
 	w := solveCholesky(AtA, Aty)
 	yMean := mean(testY)
 	var ssRes, ssTot float64
 	for i, e := range testEmb {
 		row := make([]float64, p)
 		copy(row, e)
 		row[d] = 1.0
 		pred := dot(row, w)
 		ssRes += (testY[i] - pred) * (testY[i] - pred)
 		ssTot += (testY[i] - yMean) * (testY[i] - yMean)
 	}
 	if ssTot == 0 {
 		return 0
 	}
 	return 1 - ssRes/ssTot
 }
 // LinearProbe fits a ridge regression (closed-form) on (emb, y) with regularisation λ
 // and returns R² on the same data. Call with train embeddings; probe on held-out by
 // splitting before calling.
 //
 // emb[i] is the embedding vector for sample i; y[i] is the scalar target.
 func LinearProbe(emb [][]float64, y []float64, lambda float64) float64 {
 	n := len(emb)
 	if n == 0 {
 		return 0
 	}
 	d := len(emb[0])
 	// Build augmented design matrix A = [emb | 1] (n × d+1)
 	A := make([][]float64, n)
 	for i, e := range emb {
 		row := make([]float64, d+1)
 		copy(row, e)
 		row[d] = 1.0
 		A[i] = row
 	}
 	// Normal equations: (AᵀA + λI) w = Aᵀy  (ridge)
 	p := d + 1
 	AtA := make([][]float64, p)
 	for i := range AtA {
 		AtA[i] = make([]float64, p)
 	}
 	Aty := make([]float64, p)
 	for i := 0; i < n; i++ {
 		for j := 0; j < p; j++ {
 			Aty[j] += A[i][j] * y[i]
 			for k := 0; k < p; k++ {
 				AtA[j][k] += A[i][j] * A[i][k]
 			}
 		}
 	}
 	for j := 0; j < p; j++ {
 		AtA[j][j] += lambda
 	}
 	w := solveCholesky(AtA, Aty)
 	// R² = 1 - SS_res / SS_tot
 	yMean := mean(y)
 	var ssRes, ssTot float64
 	for i := 0; i < n; i++ {
 		pred := dot(A[i], w)
 		ssRes += (y[i] - pred) * (y[i] - pred)
 		ssTot += (y[i] - yMean) * (y[i] - yMean)
 	}
 	if ssTot == 0 {
 		return 0
 	}
 	return 1 - ssRes/ssTot
 }
 // Silhouette returns the mean silhouette coefficient of the embeddings with respect
 // to the given integer labels. Distances are Euclidean. Returns an error if fewer
 // than 2 distinct labels are present.
 func Silhouette(emb [][]float64, labels []int) (float64, error) {
 	n := len(emb)
 	if n == 0 {
 		return 0, errors.New("eval: empty embeddings")
 	}
 	// count distinct labels
 	labelSet := map[int]struct{}{}
 	for _, l := range labels {
 		labelSet[l] = struct{}{}
 	}
 	if len(labelSet) < 2 {
 		return 0, errors.New("eval: silhouette requires at least 2 distinct labels")
 	}
 	// group indices by label
 	groups := map[int][]int{}
 	for i, l := range labels {
 		groups[l] = append(groups[l], i)
 	}
 	var total float64
 	for i := 0; i < n; i++ {
 		li := labels[i]
 		// a(i) = mean intra-cluster distance
 		var aSum float64
 		inGroup := groups[li]
 		for _, j := range inGroup {
 			if j != i {
 				aSum += euclidean(emb[i], emb[j])
 			}
 		}
 		var a float64
 		if len(inGroup) > 1 {
 			a = aSum / float64(len(inGroup)-1)
 		}
 		// b(i) = min mean inter-cluster distance
 		b := math.MaxFloat64
 		for l, idxs := range groups {
 			if l == li {
 				continue
 			}
 			var dSum float64
 			for _, j := range idxs {
 				dSum += euclidean(emb[i], emb[j])
 			}
 			avg := dSum / float64(len(idxs))
 			if avg < b {
 				b = avg
 			}
 		}
 		s := (b - a) / math.Max(a, b)
 		total += s
 	}
 	return total / float64(n), nil
 }
 // EffectiveRank computes Roy's effective rank of the embedding matrix:
 // exp(H) where H = -∑ pᵢ log(pᵢ) is the Shannon entropy of the normalised
 // squared singular values. Returns 1 for a rank-1 matrix and ≈ dim for
 // a full-rank isotropic matrix.
 func EffectiveRank(emb [][]float64) float64 {
 	n := len(emb)
 	if n == 0 {
 		return 0
 	}
 	d := len(emb[0])
 	// Compute covariance-like matrix CᵀC where C is mean-centered embedding.
 	mu := make([]float64, d)
 	for _, e := range emb {
 		for j, v := range e {
 			mu[j] += v
 		}
 	}
 	for j := range mu {
 		mu[j] /= float64(n)
 	}
 	// C = emb - mu  (n × d); compute CᵀC (d × d)
 	CtC := make([][]float64, d)
 	for i := range CtC {
 		CtC[i] = make([]float64, d)
 	}
 	for _, e := range emb {
 		for j := 0; j < d; j++ {
 			cj := e[j] - mu[j]
 			for k := 0; k < d; k++ {
 				CtC[j][k] += cj * (e[k] - mu[k])
 			}
 		}
 	}
 	// Eigenvalues of CᵀC via power iteration approximation isn't great;
 	// use the Frobenius / trace approach: σᵢ² ∝ eigenvalues of CᵀC.
 	// For a pure-Go impl without LAPACK: use the fact that the normalised
 	// squared singular values equal normalised eigenvalues of CᵀC.
 	// Compute them via Jacobi iteration for small d, or use the analytical
 	// formula for 2×2, or use iterative QR for general d.
 	eigs := jacobiEigenvalues(CtC)
 	// normalise to sum-1 distribution
 	var sumEig float64
 	for _, v := range eigs {
 		if v > 0 {
 			sumEig += v
 		}
 	}
 	if sumEig == 0 {
 		return 1
 	}
 	var H float64
 	for _, v := range eigs {
 		if v > 0 {
 			p := v / sumEig
 			H -= p * math.Log(p)
 		}
 	}
 	return math.Exp(H)
 }
 // ── internal helpers ──────────────────────────────────────────────────────────
 func euclidean(a, b []float64) float64 {
 	var s float64
 	for i := range a {
 		d := a[i] - b[i]
 		s += d * d
 	}
 	return math.Sqrt(s)
 }
 func dot(a, b []float64) float64 {
 	var s float64
 	for i := range a {
 		s += a[i] * b[i]
 	}
 	return s
 }
 func mean(y []float64) float64 {
 	var s float64
 	for _, v := range y {
 		s += v
 	}
 	return s / float64(len(y))
 }
 // solveCholesky solves Ax = b for symmetric positive-definite A via
 // Cholesky decomposition. Falls back to pseudo-inverse on failure.
 func solveCholesky(A [][]float64, b []float64) []float64 {
 	n := len(A)
 	// Cholesky decomposition: A = LLᵀ
 	L := make([][]float64, n)
 	for i := range L {
 		L[i] = make([]float64, n)
 	}
 	for i := 0; i < n; i++ {
 		for j := 0; j <= i; j++ {
 			s := A[i][j]
 			for k := 0; k < j; k++ {
 				s -= L[i][k] * L[j][k]
 			}
 			if i == j {
 				if s <= 0 {
 					s = 1e-12
 				}
 				L[i][j] = math.Sqrt(s)
 			} else {
 				L[i][j] = s / L[j][j]
 			}
 		}
 	}
 	// Forward substitution Ly = b
 	y := make([]float64, n)
 	for i := 0; i < n; i++ {
 		s := b[i]
 		for k := 0; k < i; k++ {
 			s -= L[i][k] * y[k]
 		}
 		y[i] = s / L[i][i]
 	}
 	// Back substitution Lᵀx = y
 	x := make([]float64, n)
 	for i := n - 1; i >= 0; i-- {
 		s := y[i]
 		for k := i + 1; k < n; k++ {
 			s -= L[k][i] * x[k]
 		}
 		x[i] = s / L[i][i]
 	}
 	return x
 }
 // jacobiEigenvalues returns eigenvalues of a symmetric matrix via Jacobi iteration.
 func jacobiEigenvalues(A [][]float64) []float64 {
 	n := len(A)
 	// copy
 	a := make([][]float64, n)
 	for i := range a {
 		a[i] = make([]float64, n)
 		copy(a[i], A[i])
 	}
 	const maxIter = 100
 	const tol = 1e-10
 	for iter := 0; iter < maxIter; iter++ {
 		// find largest off-diagonal element
 		p, q, amax := 0, 1, 0.0
 		for i := 0; i < n; i++ {
 			for j := i + 1; j < n; j++ {
 				if v := math.Abs(a[i][j]); v > amax {
 					amax = v
 					p, q = i, j
 				}
 			}
 		}
 		if amax < tol {
 			break
 		}
 		// Jacobi rotation
 		theta := 0.5 * math.Atan2(2*a[p][q], a[q][q]-a[p][p])
 		c, s := math.Cos(theta), math.Sin(theta)
 		// apply rotation
 		newA := make([][]float64, n)
 		for i := range newA {
 			newA[i] = make([]float64, n)
 			copy(newA[i], a[i])
 		}
 		app := c*c*a[p][p] + 2*c*s*a[p][q] + s*s*a[q][q]
 		aqq := s*s*a[p][p] - 2*c*s*a[p][q] + c*c*a[q][q]
 		apq := 0.0
 		newA[p][p] = app
 		newA[q][q] = aqq
 		newA[p][q] = apq
 		newA[q][p] = apq
 		for r := 0; r < n; r++ {
 			if r == p || r == q {
 				continue
 			}
 			arp := c*a[r][p] + s*a[r][q]
 			arq := -s*a[r][p] + c*a[r][q]
 			newA[r][p] = arp
 			newA[p][r] = arp
 			newA[r][q] = arq
 			newA[q][r] = arq
 		}
 		a = newA
 	}
 	eigs := make([]float64, n)
 	for i := range eigs {
 		eigs[i] = a[i][i]
 	}
 	return eigs
 }
@@ -1,138 +0,0 @@
 package eval_test
 import (
 	"math"
 	"math/rand"
 	"testing"
 	"gitea.d-ma.be/mathias/jepa-fx-risk/internal/eval"
 )
 func seededRNG(seed int64) *rand.Rand {
 	return rand.New(rand.NewSource(seed))
 }
 // ── LinearProbe (val_vol_r2) ──────────────────────────────────────────────────
 func TestLinearProbe_Perfect(t *testing.T) {
 	n := 50
 	emb := make([][]float64, n)
 	y := make([]float64, n)
 	for i := range emb {
 		emb[i] = []float64{float64(i)}
 		y[i] = float64(i)
 	}
 	r2 := eval.LinearProbe(emb, y, 1e-3)
 	if r2 < 0.99 {
 		t.Fatalf("perfect predictor: want R²≥0.99, got %.4f", r2)
 	}
 }
 func TestLinearProbe_ConstantTarget(t *testing.T) {
 	n := 40
 	emb := make([][]float64, n)
 	y := make([]float64, n)
 	for i := range emb {
 		emb[i] = []float64{float64(i), float64(i * i)}
 		y[i] = 3.0
 	}
 	r2 := eval.LinearProbe(emb, y, 1e-3)
 	if r2 > 0.01 {
 		t.Fatalf("constant target: want R²≤0.01, got %.4f", r2)
 	}
 }
 func TestLinearProbe_NoiseEmbedding(t *testing.T) {
 	rng := seededRNG(42)
 	n := 80
 	emb := make([][]float64, n)
 	y := make([]float64, n)
 	for i := range emb {
 		emb[i] = []float64{rng.NormFloat64(), rng.NormFloat64()}
 		y[i] = float64(i)
 	}
 	r2 := eval.LinearProbe(emb, y, 1e-3)
 	if r2 > 0.10 {
 		t.Fatalf("noise embedding: want R²<0.10, got %.4f", r2)
 	}
 }
 // ── Silhouette ────────────────────────────────────────────────────────────────
 func TestSilhouette_PerfectClusters(t *testing.T) {
 	emb := make([][]float64, 40)
 	labels := make([]int, 40)
 	for i := range emb {
 		if i < 20 {
 			emb[i] = []float64{0.0, 0.0}
 			labels[i] = 0
 		} else {
 			emb[i] = []float64{1000.0, 1000.0}
 			labels[i] = 1
 		}
 	}
 	sil, err := eval.Silhouette(emb, labels)
 	if err != nil {
 		t.Fatal(err)
 	}
 	if sil < 0.95 {
 		t.Fatalf("perfect clusters: want sil≥0.95, got %.4f", sil)
 	}
 }
 func TestSilhouette_SingleLabel(t *testing.T) {
 	emb := [][]float64{{1, 2}, {3, 4}, {5, 6}}
 	labels := []int{0, 0, 0}
 	_, err := eval.Silhouette(emb, labels)
 	if err == nil {
 		t.Fatal("expected error for single-label input")
 	}
 }
 func TestSilhouette_RandomClusters(t *testing.T) {
 	rng := seededRNG(7)
 	n := 60
 	emb := make([][]float64, n)
 	labels := make([]int, n)
 	for i := range emb {
 		emb[i] = []float64{rng.NormFloat64(), rng.NormFloat64()}
 		labels[i] = i % 2
 	}
 	sil, err := eval.Silhouette(emb, labels)
 	if err != nil {
 		t.Fatal(err)
 	}
 	if math.Abs(sil) > 0.30 {
 		t.Fatalf("random clusters: want |sil|≤0.30, got %.4f", sil)
 	}
 }
 // ── EffectiveRank ─────────────────────────────────────────────────────────────
 func TestEffectiveRank_Rank1(t *testing.T) {
 	emb := make([][]float64, 30)
 	for i := range emb {
 		emb[i] = []float64{1.0, 2.0, 3.0, 4.0}
 	}
 	er := eval.EffectiveRank(emb)
 	if er > 1.5 {
 		t.Fatalf("rank-1 matrix: want erank≤1.5, got %.4f", er)
 	}
 }
 func TestEffectiveRank_FullRank(t *testing.T) {
 	rng := seededRNG(99)
 	dim := 8
 	emb := make([][]float64, 200)
 	for i := range emb {
 		row := make([]float64, dim)
 		for j := range row {
 			row[j] = rng.NormFloat64()
 		}
 		emb[i] = row
 	}
 	er := eval.EffectiveRank(emb)
 	if er < float64(dim)*0.7 {
 		t.Fatalf("full-rank: want erank≥%.1f, got %.4f", float64(dim)*0.7, er)
 	}
 }
@@ -1,100 +0,0 @@
 // Code generated by templ - DO NOT EDIT.
 // templ: version: v0.3.1020
 package web
 //lint:file-ignore SA4006 This context is only used if a nested component is present.
 import "github.com/a-h/templ"
 import templruntime "github.com/a-h/templ/runtime"
 func Index() templ.Component {
 	return templruntime.GeneratedTemplate(func(templ_7745c5c3_Input templruntime.GeneratedComponentInput) (templ_7745c5c3_Err error) {
 		templ_7745c5c3_W, ctx := templ_7745c5c3_Input.Writer, templ_7745c5c3_Input.Context
 		if templ_7745c5c3_CtxErr := ctx.Err(); templ_7745c5c3_CtxErr != nil {
 			return templ_7745c5c3_CtxErr
 		}
 		templ_7745c5c3_Buffer, templ_7745c5c3_IsBuffer := templruntime.GetBuffer(templ_7745c5c3_W)
 		if !templ_7745c5c3_IsBuffer {
 			defer func() {
 				templ_7745c5c3_BufErr := templruntime.ReleaseBuffer(templ_7745c5c3_Buffer)
 				if templ_7745c5c3_Err == nil {
 					templ_7745c5c3_Err = templ_7745c5c3_BufErr
 				}
 			}()
 		}
 		ctx = templ.InitializeContext(ctx)
 		templ_7745c5c3_Var1 := templ.GetChildren(ctx)
 		if templ_7745c5c3_Var1 == nil {
 			templ_7745c5c3_Var1 = templ.NopComponent
 		}
 		ctx = templ.ClearChildren(ctx)
 		templ_7745c5c3_Var2 := templruntime.GeneratedTemplate(func(templ_7745c5c3_Input templruntime.GeneratedComponentInput) (templ_7745c5c3_Err error) {
 			templ_7745c5c3_W, ctx := templ_7745c5c3_Input.Writer, templ_7745c5c3_Input.Context
 			templ_7745c5c3_Buffer, templ_7745c5c3_IsBuffer := templruntime.GetBuffer(templ_7745c5c3_W)
 			if !templ_7745c5c3_IsBuffer {
 				defer func() {
 					templ_7745c5c3_BufErr := templruntime.ReleaseBuffer(templ_7745c5c3_Buffer)
 					if templ_7745c5c3_Err == nil {
 						templ_7745c5c3_Err = templ_7745c5c3_BufErr
 					}
 				}()
 			}
 			ctx = templ.InitializeContext(ctx)
 			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 1, "<h1 class=\"text-3xl font-semibold mb-6\">hostexecutor</h1><button hx-get=\"/api/hello\" hx-target=\"#out\" class=\"px-4 py-2 bg-slate-900 text-white rounded-md hover:bg-slate-700\">Say hello</button><div id=\"out\" class=\"mt-6 text-slate-700\"></div>")
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
 			return nil
 		})
 		templ_7745c5c3_Err = Layout("hostexecutor").Render(templ.WithChildren(ctx, templ_7745c5c3_Var2), templ_7745c5c3_Buffer)
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
 		return nil
 	})
 }
 func Hello(name string) templ.Component {
 	return templruntime.GeneratedTemplate(func(templ_7745c5c3_Input templruntime.GeneratedComponentInput) (templ_7745c5c3_Err error) {
 		templ_7745c5c3_W, ctx := templ_7745c5c3_Input.Writer, templ_7745c5c3_Input.Context
 		if templ_7745c5c3_CtxErr := ctx.Err(); templ_7745c5c3_CtxErr != nil {
 			return templ_7745c5c3_CtxErr
 		}
 		templ_7745c5c3_Buffer, templ_7745c5c3_IsBuffer := templruntime.GetBuffer(templ_7745c5c3_W)
 		if !templ_7745c5c3_IsBuffer {
 			defer func() {
 				templ_7745c5c3_BufErr := templruntime.ReleaseBuffer(templ_7745c5c3_Buffer)
 				if templ_7745c5c3_Err == nil {
 					templ_7745c5c3_Err = templ_7745c5c3_BufErr
 				}
 			}()
 		}
 		ctx = templ.InitializeContext(ctx)
 		templ_7745c5c3_Var3 := templ.GetChildren(ctx)
 		if templ_7745c5c3_Var3 == nil {
 			templ_7745c5c3_Var3 = templ.NopComponent
 		}
 		ctx = templ.ClearChildren(ctx)
 		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 2, "<p>Hello, ")
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
 		var templ_7745c5c3_Var4 string
 		templ_7745c5c3_Var4, templ_7745c5c3_Err = templ.JoinStringErrs(name)
 		if templ_7745c5c3_Err != nil {
 			return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/index.templ`, Line: 15, Col: 17}
 		}
 		_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var4))
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
 		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 3, "!</p>")
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
 		return nil
 	})
 }
 var _ = templruntime.GeneratedTemplate
@@ -1,61 +0,0 @@
 // Code generated by templ - DO NOT EDIT.
 // templ: version: v0.3.1020
 package web
 //lint:file-ignore SA4006 This context is only used if a nested component is present.
 import "github.com/a-h/templ"
 import templruntime "github.com/a-h/templ/runtime"
 func Layout(title string) templ.Component {
 	return templruntime.GeneratedTemplate(func(templ_7745c5c3_Input templruntime.GeneratedComponentInput) (templ_7745c5c3_Err error) {
 		templ_7745c5c3_W, ctx := templ_7745c5c3_Input.Writer, templ_7745c5c3_Input.Context
 		if templ_7745c5c3_CtxErr := ctx.Err(); templ_7745c5c3_CtxErr != nil {
 			return templ_7745c5c3_CtxErr
 		}
 		templ_7745c5c3_Buffer, templ_7745c5c3_IsBuffer := templruntime.GetBuffer(templ_7745c5c3_W)
 		if !templ_7745c5c3_IsBuffer {
 			defer func() {
 				templ_7745c5c3_BufErr := templruntime.ReleaseBuffer(templ_7745c5c3_Buffer)
 				if templ_7745c5c3_Err == nil {
 					templ_7745c5c3_Err = templ_7745c5c3_BufErr
 				}
 			}()
 		}
 		ctx = templ.InitializeContext(ctx)
 		templ_7745c5c3_Var1 := templ.GetChildren(ctx)
 		if templ_7745c5c3_Var1 == nil {
 			templ_7745c5c3_Var1 = templ.NopComponent
 		}
 		ctx = templ.ClearChildren(ctx)
 		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 1, "<!doctype html><html lang=\"en\"><head><meta charset=\"utf-8\"><meta name=\"viewport\" content=\"width=device-width,initial-scale=1\"><title>")
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
 		var templ_7745c5c3_Var2 string
 		templ_7745c5c3_Var2, templ_7745c5c3_Err = templ.JoinStringErrs(title)
 		if templ_7745c5c3_Err != nil {
 			return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/layout.templ`, Line: 9, Col: 17}
 		}
 		_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var2))
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
 		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 2, "</title><script src=\"https://unpkg.com/htmx.org@2.0.0\"></script><script src=\"https://cdn.tailwindcss.com\"></script></head><body class=\"min-h-screen bg-slate-50 text-slate-900 antialiased\"><main class=\"max-w-3xl mx-auto px-6 py-12\">")
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
 		templ_7745c5c3_Err = templ_7745c5c3_Var1.Render(ctx, templ_7745c5c3_Buffer)
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
 		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 3, "</main></body></html>")
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
 		return nil
 	})
 }
 var _ = templruntime.GeneratedTemplate
@@ -1,205 +0,0 @@
 """loop.py — Karpathy-style autoresearch loop for jepa-fx-risk.
 Agent (on iguana/berget — NOT koala, whose GPU is reserved for train.py) reads
 program.md + train.py + STATUS.md, proposes ONE change to train.py, we run it,
 keep if val_vol_r2 improved else git-revert. Appends per-iter record to STATUS.md.
    LITELLM_KEY=xxx python loop.py [--iters N] [--model MODEL]
 Env:
  LITELLM_KEY      — LiteLLM master key (required)
  LITELLM_BASE     — default http://localhost:30401/v1
  LOOP_MODEL       — default berget/gemma4-31b (non-thinking; iguana/berget only)
  LOOP_ITERS       — default 3
  TRAIN_TIMEOUT    — seconds per train.py run, default 120
 """
 import argparse
 import json
 import os
 import subprocess
 import sys
 import time
 import textwrap
 from pathlib import Path
 import urllib.request
 LITELLM_BASE = os.environ.get("LITELLM_BASE", "http://localhost:30401/v1")
 LITELLM_KEY  = os.environ.get("LITELLM_KEY", "")
 LOOP_MODEL   = os.environ.get("LOOP_MODEL", "berget/gemma4-31b")
 LOOP_ITERS   = int(os.environ.get("LOOP_ITERS", "3"))
 TRAIN_TIMEOUT = int(os.environ.get("TRAIN_TIMEOUT", "120"))
 STATUS_MD    = Path("STATUS.md")
 METRICS_JSON = Path("metrics.json")
 TRAIN_PY     = Path("train.py")
 AGENT_SYSTEM = textwrap.dedent("""\
    You are the autoresearch agent for jepa-fx-risk. Your job: propose ONE small,
    targeted change to train.py to improve val_vol_r2 (OOS R² predicting 1-day
    realized vol from frozen embeddings). Higher is better.
    Rules:
    - Return ONLY the full new content of train.py — nothing else, no explanation,
      no markdown fence. Raw Python only.
    - Change ONE thing at a time (one knob, one structural idea).
    - Do NOT touch prepare_data.py, loop.py, or the data pipeline — only train.py.
    - Do NOT add new data sources or new files.
    - The metric is computed externally from your frozen embeddings; trust it.
 """)
 def read_file(p: Path) -> str:
    return p.read_text() if p.exists() else ""
 def gpu_snapshot() -> str:
    try:
        out = subprocess.check_output(
            ["nvidia-smi", "--query-gpu=utilization.gpu,memory.used,memory.total,temperature.gpu",
             "--format=csv,noheader,nounits"], timeout=5, text=True
        ).strip()
        util, mem_used, mem_total, temp = [x.strip() for x in out.split(",")]
        return "gpu=%s%% vram=%s/%sMiB temp=%s°C" % (util, mem_used, mem_total, temp)
    except Exception:
        return "gpu=N/A"
 def read_metric() -> float | None:
    if not METRICS_JSON.exists():
        return None
    try:
        return float(json.loads(METRICS_JSON.read_text())["val_vol_r2"])
    except Exception:
        return None
 def run_train() -> tuple[float | None, float, str]:
    """Run train.py. Returns (val_vol_r2 or None, wall_secs, stderr_tail)."""
    t0 = time.time()
    gpu_before = gpu_snapshot()
    try:
        r = subprocess.run(
            [sys.executable, "train.py"],
            capture_output=True, text=True, timeout=TRAIN_TIMEOUT,
        )
        elapsed = time.time() - t0
        if r.returncode != 0:
            return None, elapsed, (r.stderr or r.stdout)[-300:]
        metric = read_metric()
        return metric, elapsed, ""
    except subprocess.TimeoutExpired:
        return None, TRAIN_TIMEOUT, "TIMEOUT"
 def call_agent(iteration: int, best_so_far: float | None) -> str:
    """Ask the LLM agent to edit train.py. Returns new train.py content."""
    context = "\n\n".join([
        "# program.md\n" + read_file(Path("program.md")),
        "# train.py (current)\n" + read_file(TRAIN_PY),
        "# STATUS.md (history)\n" + read_file(STATUS_MD)[-2000:],
        "# metrics.json (last run)\n" + read_file(METRICS_JSON),
        "Iteration %d. Best val_vol_r2 so far: %s. Improve it." % (
            iteration, "%.4f" % best_so_far if best_so_far is not None else "none yet"
        ),
    ])
    payload = json.dumps({
        "model": LOOP_MODEL,
        "messages": [
            {"role": "system", "content": AGENT_SYSTEM},
            {"role": "user",   "content": context},
        ],
        "temperature": 0.7,
        "max_tokens": 4096,
    }).encode()
    req = urllib.request.Request(
        LITELLM_BASE + "/chat/completions",
        data=payload,
        headers={"Authorization": "Bearer " + LITELLM_KEY,
                 "Content-Type": "application/json"},
        method="POST",
    )
    resp = urllib.request.urlopen(req, timeout=60)
    data = json.load(resp)
    return data["choices"][0]["message"]["content"]
 def revert_train(original_content: str):
    TRAIN_PY.write_text(original_content)
 def append_status(line: str):
    with open(STATUS_MD, "a") as f:
        f.write(line + "\n")
 def main():
    if not LITELLM_KEY:
        print("ERROR: set LITELLM_KEY"); sys.exit(1)
    if not STATUS_MD.exists():
        STATUS_MD.write_text("# Autoresearch STATUS\n\n| iter | val_vol_r2 | delta | action | secs | gpu | change |\n|------|-----------|-------|--------|------|-----|--------|\n")
    # establish baseline
    baseline = read_metric()
    if baseline is None:
        print("No metrics.json — running train.py for baseline...")
        m, secs, err = run_train()
        if m is None:
            print("Baseline run failed:", err); sys.exit(1)
        baseline = m
        print("Baseline: val_vol_r2 = %.4f (%.1fs)" % (baseline, secs))
    best = baseline
    print("Starting loop | model=%s | iters=%d | baseline=%.4f" % (LOOP_MODEL, LOOP_ITERS, best))
    for i in range(1, LOOP_ITERS + 1):
        print("\n--- iter %d/%d ---" % (i, LOOP_ITERS))
        original = TRAIN_PY.read_text()
        print("  calling agent (%s)..." % LOOP_MODEL)
        t_agent = time.time()
        try:
            new_code = call_agent(i, best)
        except Exception as e:
            print("  agent call failed:", e)
            append_status("| %d | ERR | — | agent-fail | — | — | %s |" % (i, str(e)[:60]))
            continue
        agent_secs = time.time() - t_agent
        print("  agent replied in %.1fs" % agent_secs)
        # strip accidental markdown fences
        if new_code.strip().startswith("```"):
            lines = new_code.strip().splitlines()
            new_code = "\n".join(lines[1:-1] if lines[-1].strip() == "```" else lines[1:])
        TRAIN_PY.write_text(new_code)
        gpu = gpu_snapshot()
        print("  running train.py [%s]..." % gpu)
        metric, secs, err = run_train()
        if metric is None:
            print("  train.py FAILED — reverting. err:", err[:100])
            revert_train(original)
            append_status("| %d | FAIL | — | revert | %.0fs | %s | run error |" % (i, secs, gpu))
            continue
        delta = metric - best
        if metric > best:
            best = metric
            action = "KEEP"
        else:
            revert_train(original)
            action = "revert"
        summary = "| %d | %.4f | %+.4f | %s | %.0fs | %s | iter%d |" % (
            i, metric, delta, action, secs, gpu, i)
        append_status(summary)
        print("  val_vol_r2=%.4f  delta=%+.4f  action=%s  [%.0fs]" % (metric, delta, action, secs))
    print("\nDone. Best val_vol_r2 = %.4f (baseline was %.4f, delta %+.4f)" % (best, baseline, best - baseline))
    print("STATUS.md updated.")
 if __name__ == "__main__":
    main()
@@ -1,14 +0,0 @@
 {
  "val_vol_r2": 0.05988483092470609,
  "n_test": 263,
  "knobs": {
    "WINDOW": 60,
    "PATCH_LEN": 5,
    "STRIDE": 5,
    "D_MODEL": 64,
    "DEPTH": 2,
    "MASK_FRAC": 0.5,
    "SIGREG_LAM": 0.01,
    "EPOCHS": 300
  }
 }
@@ -0,0 +1,31 @@
 # model
 Python + PyTorch perimeter. This is the only directory in the project that uses Python.
 ## Contents
 ```
 model/
  train.py              TS-JEPA training entry point
  configs/              YAML configs per phase
    phase0-mae.yaml
    phase1-tsjepa.yaml
  tsjepa/               TS-JEPA model implementation (adapted from paper)
  tests/                pytest tests for model components
  requirements.txt      pinned Python dependencies
  .venv/                (gitignored — created by `task model:setup`)
 ```
 ## Setup
 ```bash
 task model:setup    # creates .venv and installs requirements via uv
 ```
 ## Dependency policy
 Every Python dependency must be justified in a comment in `requirements.txt`. Prefer Go implementations for anything outside the training loop. When adding a new dependency, add an entry to DECISIONS.md explaining why a Go alternative wasn't sufficient.
 ## Python version
 3.12 (pinned in `.python-version`)
@@ -0,0 +1,11 @@
 # Notebooks
 Exploratory Data Analysis (EDA) scratch space only.
 ## Rules
 - **Never use notebooks for anything reproducible.** Training, evaluation, and metric computation belong in `src/` with tests.
 - **Strip all outputs before committing.** Use `nbstripout` or equivalent.
 - **Label every notebook with a phase prefix:** `phase0-eda-eurusd-distribution.ipynb`
 Notebooks are thinking tools, not research artifacts. If a finding from a notebook is worth keeping, it goes into a spec, a decision in `DECISIONS.md`, or a result in `results/summaries/` — not the notebook itself.
@@ -1,8 +0,0 @@
 # Python deps for the autoresearch loop (train.py + scripts). Install torch from
 # the cu130 index FIRST (koala Blackwell sm_120, torch 2.12.1+cu130 verified):
 #   pip install torch --index-url https://download.pytorch.org/whl/cu130
 #   pip install -r requirements.txt
 numpy>=2.0
 pandas>=2.2
 pyarrow>=16
 histdata>=1.3        # histdata.com downloader (handles the tk token politely)
@@ -0,0 +1,32 @@
 # Results
 Tracked outputs from experiment runs. Large raw outputs are gitignored — only summaries are committed.
 ## Structure
 ```
 results/
  summaries/    metric tables, key figures, per-phase result records (committed)
  raw/          full embedding outputs, backtest CSVs (gitignored)
 ```
 ## Per-phase result record format
 Each concluded phase produces a result record in `summaries/`:
 ```
 summaries/
  phase-0-[pass|null].md
  phase-1-[pass|null].md
  ...
 ```
 Each record must include:
 - Phase name and hypothesis
 - Key metrics (silhouette score, R², etc.) with confidence intervals where applicable
 - Baseline comparison
 - Verdict: PASS / NULL RESULT
 - If null: what was investigated, what was found, next step taken
 - Link to experiment git tag
 **Null results are valid research outputs and must be committed, not discarded.**
@@ -1,13 +0,0 @@
 {
  "label": "null",
  "mean_sil": 0.018206419112781685,
  "pca_sil": 0.13589094579219818,
  "spread": 0.8673340065023978,
  "pc1_hv_corr": 0.525803392278542,
  "per_seed": [
    0.026790648698806763,
    0.010999602265655994,
    0.016829006373882294
  ],
  "passed": false
 }
@@ -1,25 +0,0 @@
 # Phase-0 SSL feasibility gate — null
 **Date:** 2026-06-24
 **Path B deviation:** Daily 2019-2023 (not hourly 2008-2022); Python harness
 (not Go #4); gate metric adapted from silhouette-on-embedding to match
 available data. Go harness (#4) remains open for production experiments.
 ## Data
 - Train: EUR/USD daily 2019-2021 (907 windows)
 - OOS: EUR/USD daily 2022-2023 (593 windows)
 - HV label: top-33% realized-vol days = high-volatility (196 days)
 ## Results
 | | Value | Gate |
 |---|---|---|
 | TS-JEPA mean silhouette (3 seeds) | 0.0182 | > 0.20 → **False** |
 | Beats PCA baseline (0.1359) | 0.0182 | > PCA → **False** |
 | Seed stability (spread) | 86.73% | < 10% → **False** |
 | PC1/HV correlation | 0.5258 | < 0.95 → **True** |
 Per-seed: ['0.0268', '0.0110', '0.0168']
 ## Verdict: **NULL**
 One or more gate criteria not met. See null result protocol in #5.
@@ -1,21 +0,0 @@
 """Phase-0 compute gate (brain wiki/jepa-fx/facts/autoresearch-integration-phase1):
 PyTorch cu130 must see the koala Blackwell GPU and compute before any experiment.
    python scripts/check_gpu.py   # exits 0 if the GPU is usable, 1 otherwise
 Note: koala shares this 12GB card with the llama-swap LLM stack. The autoresearch
 agent should run on iguana/berget models so koala's GPU stays free for train.py.
 """
 import sys
 import torch
 print("torch", torch.__version__)
 if not torch.cuda.is_available():
    print("CUDA NOT AVAILABLE — gate BLOCKED")
    sys.exit(1)
 print("device:", torch.cuda.get_device_name(0))
 print("capability: sm_%d%d" % torch.cuda.get_device_capability(0))
 x = torch.randn(2000, 2000, device="cuda")
 (x @ x).sum().item()
 torch.cuda.synchronize()
 print("GPU matmul OK — Phase-0 compute gate GREEN")
@@ -1,31 +0,0 @@
 """Fetch EUR/USD M1 bars from histdata.com (free, research use).
 Polite: one request per year, spaced; past years query month=None. Uses the
 maintained `histdata` package which handles histdata's anti-hotlink tk token.
 Output: data/raw/DAT_ASCII_EURUSD_M1_<year>.zip
    YEARS=2019,2020,2021 python scripts/fetch_data.py
 """
 import os
 import time
 from histdata import download_hist_data
 from histdata.api import Platform as P, TimeFrame as T
 YEARS = [y.strip() for y in os.environ.get("YEARS", "2019,2020,2021").split(",")]
 def main():
    os.makedirs("data/raw", exist_ok=True)
    for yr in YEARS:
        f = download_hist_data(
            year=yr, month=None, pair="eurusd",
            platform=P.GENERIC_ASCII, time_frame=T.ONE_MINUTE,
            output_directory="data/raw",
        )
        print("fetched", yr, "->", f)
        time.sleep(2)  # be a good citizen
 if __name__ == "__main__":
    main()
@@ -1,237 +0,0 @@
 """Phase-0 SSL feasibility gate (path B — Python fast-close of #5).
 Spec deviation documented: original spec (#5) required hourly 2008-2022 data
 and a Go eval harness (#4). Path B uses daily 2019-2023 + Python harness to
 close the gate quickly, since val_vol_r2 > 0 already demonstrates SSL
 feasibility. The Go harness (#4) remains open for production experiments.
 Gate criteria (from #5):
  - Silhouette > 0.20 on held-out 2022-2023 (binary HV label: top-33% RV days)
  - TS-JEPA silhouette > PCA baseline silhouette
  - Rerun x3 seeds within ±10% of mean silhouette
  - PC1/HV correlation < 0.95 (sanity: not trivially memorising the label)
    python scripts/phase0_gate.py
 """
 import json
 import math
 import os
 import numpy as np
 import pandas as pd
 import torch
 import torch.nn as nn
 from sklearn.decomposition import PCA
 from sklearn.metrics import silhouette_score
 from sklearn.preprocessing import StandardScaler
 SEEDS     = [0, 1, 2]
 WINDOW    = 30
 PATCH_LEN = 5
 STRIDE    = 5
 D_MODEL   = 32
 DEPTH     = 2
 N_HEADS   = 4
 EPOCHS    = 400
 LR        = 3e-4
 SIGREG_LAM = 0.5
 HV_PERCENTILE = 67   # top-33% = "high volatility"
 dev = "cuda" if torch.cuda.is_available() else "cpu"
 # ── SIGReg ──────────────────────────────────────────────────────────────────
 def sigreg(tokens: torch.Tensor, knots: int = 17) -> torch.Tensor:
    B, T, D = tokens.shape
    z = tokens.reshape(B * T, D).float()
    t = torch.linspace(0, 3, knots, device=z.device, dtype=z.dtype)
    dt = 3.0 / (knots - 1)
    w = torch.full((knots,), 2 * dt, device=z.device, dtype=z.dtype)
    w[0] = dt; w[-1] = dt
    phi = torch.exp(-t.square() / 2.0)
    A = torch.randn(D, 256, device=z.device, dtype=z.dtype)
    A = A / A.norm(p=2, dim=0)
    x_t = (z @ A).unsqueeze(-1) * t
    err = (x_t.cos().mean(0) - phi).square() + x_t.sin().mean(0).square()
    return ((err @ (w * phi)) * z.shape[0]).mean()
 # ── Encoder ──────────────────────────────────────────────────────────────────
 class PatchEncoder(nn.Module):
    def __init__(self, in_feats, patch_len, stride, d_model, depth, n_heads):
        super().__init__()
        self.patch_len = patch_len
        self.stride    = stride
        self.embed = nn.Linear(patch_len * in_feats, d_model)
        layer = nn.TransformerEncoderLayer(d_model, n_heads, 2 * d_model,
                                           dropout=0.0, batch_first=True)
        self.tf = nn.TransformerEncoder(layer, num_layers=depth)
        n_patches = (WINDOW - patch_len) // stride + 1
        pos = torch.zeros(n_patches, d_model)
        for p in range(n_patches):
            for i in range(0, d_model, 2):
                pos[p, i]     = math.sin(p / 10000 ** (i / d_model))
                if i + 1 < d_model:
                    pos[p, i+1] = math.cos(p / 10000 ** (i / d_model))
        self.register_buffer("pos", pos)
    def forward(self, x):
        B, W, F = x.shape
        n_patches = (W - self.patch_len) // self.stride + 1
        patches = torch.stack([x[:, i*self.stride:i*self.stride+self.patch_len, :]
                                .reshape(B, -1) for i in range(n_patches)], dim=1)
        tokens = self.embed(patches) + self.pos[:n_patches]
        return self.tf(tokens)
 # ── Data ─────────────────────────────────────────────────────────────────────
 def load_data():
    df = pd.read_parquet("data/processed/eurusd_daily.parquet").reset_index(drop=True)
    df["date"] = pd.to_datetime(df["date"])
    train = df[df["date"].dt.year <= 2021].copy()
    oos   = df[df["date"].dt.year >= 2022].copy()
    feats_all  = df[["ret", "realized_vol"]].to_numpy(np.float32)
    target_all = df["realized_vol"].to_numpy(np.float32)
    dates_all  = df["date"].values
    mu = feats_all[:len(train)].mean(0)
    sd = feats_all[:len(train)].std(0) + 1e-8
    def windows(df_subset, feats_norm, dates):
        idx_start = df.index[df["date"].isin(df_subset["date"])][0]
        X, oos_dates, oos_rv = [], [], []
        for t in range(idx_start + WINDOW, idx_start + len(df_subset)):
            X.append(feats_norm[t - WINDOW:t])
            oos_dates.append(dates[t])
            oos_rv.append(target_all[t])
        return np.stack(X), np.array(oos_rv), np.array(oos_dates)
    feats_norm = (feats_all - mu) / sd
    Xtr, rvtr, _ = windows(train, feats_norm, dates_all)
    Xte, rvte, te_dates = windows(oos, feats_norm, dates_all)
    # binary HV label: top-33% realized vol days in OOS = "high volatility"
    hv_threshold = np.percentile(rvte, HV_PERCENTILE)
    hv_labels = (rvte >= hv_threshold).astype(int)
    return Xtr, rvtr, Xte, rvte, hv_labels
 # ── Train + embed ─────────────────────────────────────────────────────────────
 def train_and_embed(Xtr, Xte, seed):
    torch.manual_seed(seed)
    np.random.seed(seed)
    enc  = PatchEncoder(Xtr.shape[2], PATCH_LEN, STRIDE, D_MODEL, DEPTH, N_HEADS).to(dev)
    pred = nn.Sequential(nn.Linear(D_MODEL, D_MODEL), nn.GELU(),
                         nn.Linear(D_MODEL, D_MODEL)).to(dev)
    opt  = torch.optim.AdamW(list(enc.parameters()) + list(pred.parameters()), lr=LR)
    Xtr_t = torch.tensor(Xtr, device=dev)
    n_patches = (WINDOW - PATCH_LEN) // STRIDE + 1
    n_mask = max(1, int(0.30 * n_patches))
    for ep in range(EPOCHS):
        idx_mask = torch.randperm(n_patches)[:n_mask]
        tokens_ctx    = enc(Xtr_t)
        tokens_target = enc(Xtr_t).detach()
        jepa_loss = ((pred(tokens_ctx[:, idx_mask, :]) -
                      tokens_target[:, idx_mask, :]) ** 2).mean()
        reg = sigreg(tokens_ctx)
        loss = jepa_loss + SIGREG_LAM * reg
        opt.zero_grad(); loss.backward(); opt.step()
    enc.eval()
    with torch.no_grad():
        Ete = enc(torch.tensor(Xte, device=dev)).mean(1).cpu().numpy()
    return Ete
 # ── Gate ─────────────────────────────────────────────────────────────────────
 def pca_baseline(Xte, hv_labels):
    flat = Xte.reshape(len(Xte), -1)
    sc = StandardScaler().fit(flat)
    emb = PCA(n_components=8).fit_transform(sc.transform(flat))
    return silhouette_score(emb, hv_labels), emb
 def main():
    os.makedirs("results/summaries", exist_ok=True)
    Xtr, rvtr, Xte, rvte, hv_labels = load_data()
    print(f"train={len(Xtr)} OOS={len(Xte)} HV={hv_labels.sum()}/{len(hv_labels)}")
    pca_sil, pca_emb = pca_baseline(Xte, hv_labels)
    pc1 = pca_emb[:, 0]
    pc1_hv_corr = abs(np.corrcoef(pc1, hv_labels)[0, 1])
    print(f"PCA baseline silhouette = {pca_sil:.4f} | PC1/HV |r| = {pc1_hv_corr:.4f}")
    sils = []
    for seed in SEEDS:
        emb = train_and_embed(Xtr, Xte, seed)
        sc  = StandardScaler().fit(emb)
        sil = silhouette_score(sc.transform(emb), hv_labels)
        sils.append(sil)
        print(f"  seed={seed}  silhouette={sil:.4f}")
    mean_sil = np.mean(sils)
    spread   = (max(sils) - min(sils)) / mean_sil if mean_sil != 0 else 99
    # gate checks
    g_sil    = mean_sil > 0.20
    g_beats  = mean_sil > pca_sil
    g_stable = spread < 0.10
    g_corr   = pc1_hv_corr < 0.95
    passed   = all([g_sil, g_beats, g_stable, g_corr])
    label = "pass" if passed else "null"
    print(f"\nsilhouette mean={mean_sil:.4f} spread={spread:.2%} PCA={pca_sil:.4f} PC1/HV={pc1_hv_corr:.4f}")
    print(f"gate: sil>0.20={g_sil} beats_pca={g_beats} stable={g_stable} corr<0.95={g_corr}")
    print(f"PHASE-0: {label.upper()}")
    summary = f"""# Phase-0 SSL feasibility gate — {label}
 **Date:** 2026-06-24
 **Path B deviation:** Daily 2019-2023 (not hourly 2008-2022); Python harness
 (not Go #4); gate metric adapted from silhouette-on-embedding to match
 available data. Go harness (#4) remains open for production experiments.
 ## Data
 - Train: EUR/USD daily 2019-2021 ({len(Xtr)} windows)
 - OOS: EUR/USD daily 2022-2023 ({len(Xte)} windows)
 - HV label: top-{100-HV_PERCENTILE}% realized-vol days = high-volatility ({hv_labels.sum()} days)
 ## Results
 | | Value | Gate |
 |---|---|---|
 | TS-JEPA mean silhouette (3 seeds) | {mean_sil:.4f} | > 0.20 → **{g_sil}** |
 | Beats PCA baseline ({pca_sil:.4f}) | {mean_sil:.4f} | > PCA → **{g_beats}** |
 | Seed stability (spread) | {spread:.2%} | < 10% → **{g_stable}** |
 | PC1/HV correlation | {pc1_hv_corr:.4f} | < 0.95 → **{g_corr}** |
 Per-seed: {[f"{s:.4f}" for s in sils]}
 ## Verdict: **{label.upper()}**
 {"All 4 gate criteria met. TS-JEPA embeddings separate HV regimes significantly above PCA baseline with stable reproducibility." if passed else "One or more gate criteria not met. See null result protocol in #5."}
 """
    path = f"results/summaries/phase-0-{label}.md"
    with open(path, "w") as f:
        f.write(summary)
    print(f"Written: {path}")
    result = {"label": label, "mean_sil": mean_sil, "pca_sil": pca_sil,
              "spread": spread, "pc1_hv_corr": pc1_hv_corr,
              "per_seed": sils, "passed": passed}
    with open("results/summaries/phase-0-metrics.json", "w") as f:
        json.dump(result, f, indent=2)
    return 0 if passed else 1
 if __name__ == "__main__":
    raise SystemExit(main())
@@ -1,57 +0,0 @@
 """LOCKED data pipeline (toy) — agent must NOT edit (brain Phase-1 contract).
 Parses histdata EUR/USD M1 zips → daily series with realized volatility (the
 val_vol_r2 target = 1-day realized vol from intraday squared returns).
 Output: data/processed/eurusd_daily.parquet  [date, close, ret, realized_vol].
 """
 import glob
 import os
 import zipfile
 import numpy as np
 import pandas as pd
 RAW = "data/raw"
 OUT = "data/processed/eurusd_daily.parquet"
 def load_m1() -> pd.DataFrame:
    frames = []
    for zp in sorted(glob.glob(os.path.join(RAW, "DAT_ASCII_EURUSD_M1_*.zip"))):
        with zipfile.ZipFile(zp) as z:
            csv = [n for n in z.namelist() if n.endswith(".csv")][0]
            with z.open(csv) as f:
                df = pd.read_csv(
                    f, sep=";", header=None,
                    names=["dt", "open", "high", "low", "close", "vol"],
                )
        df["ts"] = pd.to_datetime(df["dt"], format="%Y%m%d %H%M%S")
        frames.append(df[["ts", "close"]])
    out = pd.concat(frames).sort_values("ts").reset_index(drop=True)
    return out
 def main():
    m1 = load_m1()
    m1["r"] = np.log(m1["close"]).diff()
    m1["day"] = m1["ts"].dt.normalize()
    daily = m1.groupby("day").agg(
        close=("close", "last"),
        realized_vol=("r", lambda x: np.sqrt(np.nansum(x.values ** 2))),
        n_min=("r", "count"),
    ).reset_index()
    daily = daily[daily["n_min"] > 60]  # drop thin days (holidays)
    daily["ret"] = np.log(daily["close"]).diff()
    daily = daily.dropna().reset_index(drop=True)
    os.makedirs(os.path.dirname(OUT), exist_ok=True)
    daily[["day", "close", "ret", "realized_vol"]].rename(columns={"day": "date"}).to_parquet(OUT)
    print("rows:", len(daily), "| dates:", daily["day"].min().date(), "→", daily["day"].max().date())
    # sanity: the COVID crash (March 2020) must show a realized-vol spike
    rv = daily.set_index("day")["realized_vol"]
    mar20 = rv["2020-03-01":"2020-03-31"].max()
    typ = rv["2019-01-01":"2019-12-31"].median()
    print("median 2019 RV: %.5f | max Mar-2020 RV: %.5f | spike x%.1f" % (typ, mar20, mar20 / typ))
 if __name__ == "__main__":
    main()
@@ -0,0 +1,84 @@
 # Experiment Spec: Phase 0 — SSL Feasibility Gate
 ## Hypothesis
 > We believe that a masked autoencoder (MAE) trained on FX hourly time-series will
 > produce latent embeddings that show structural separability by volatility regime
 > without explicit regime labels, measurable by silhouette score > 0.20 on k-means
 > clusters evaluated against a held-out realised-volatility regime label on 2023 data.
 This hypothesis is FALSE if silhouette score ≤ 0.20 on the held-out evaluation.
 ## Background
 Before investing in JEPA-specific machinery, we need to confirm that self-supervised
 representation learning can find regime structure in FX time-series at all. If the
 simplest SSL method (MAE) cannot find structure, JEPA will not either — and the root
 cause needs to be understood before proceeding.
 Also validates that TS-JEPA code is reproducible: first task is running TS-JEPA on
 the paper's own benchmark, not on FX data. If reproduction takes > 2 weeks, contact
 authors or fall back to implementing JEPA masking from V-JEPA codebase.
 Added post Full Grill (2026-05-27). See DECISIONS.md: "Phase 0: SSL feasibility gate."
 ## Design
 ### Data
 - **Source:** DUKASCopy, EUR/USD hourly OHLCV
 - **Train:** 2008-01-01 – 2022-12-31
 - **Held-out test:** 2023-01-01 – 2023-12-31 (sealed until final evaluation)
 - **Features:** log-return (hourly), rolling 20-period realised HV, VIX (daily → hourly interpolation)
 - **Regime label (evaluation only):** rolling 30-day HV percentile; binary: top 50% = high-vol, bottom 50% = low-vol
 ### Model
 - **Architecture:** 1D temporal Masked Autoencoder
  - Encoder: 3-layer 1D CNN + positional encoding
  - Decoder: 2-layer MLP reconstructing masked segment
 - **Masking:** contiguous temporal block (target); context window = 120h (5 days)
 - **Loss:** MSE reconstruction on masked segment
 ### TS-JEPA reproduction task (runs in parallel / before MAE)
 - Reproduce TS-JEPA paper results on the authors' benchmark dataset
 - Go/no-go: if reproduction fails within 2 weeks → contact authors or pivot to V-JEPA adaptation
 ### Baseline
 - **PCA** on raw feature vectors (same 120h context window, flattened)
 - Tests whether any dimensionality reduction finds regime structure; confirms SSL is adding something
 ### Ablations
 - Masking horizon K ∈ {8h, 24h, 72h} — does context length affect embedding quality?
 ## Acceptance Criteria
 - [ ] MAE silhouette score > 0.20 on held-out 2023 data (k-means k=3, vs. binary HV regime label)
 - [ ] MAE silhouette exceeds PCA baseline silhouette
 - [ ] Rerun ×3 within ±10% of reported silhouette (reproducibility)
 - [ ] PC1 / rolling-HV correlation < 0.95 (encoder is not purely encoding volatility level)
 - [ ] TS-JEPA reproduced on paper's benchmark within 2 weeks of start
 ## Out of Scope
 - JEPA implementation (Phase 1)
 - Multi-pair training (Phase 1+)
 - VaR, ES, or any risk metric computation
 - Any data after 2022-12-31 (training); 2023 test set opened only for final evaluation
 - Hyperparameter search beyond the three masking horizons defined above
 ## Null Result Protocol
 If MAE silhouette ≤ 0.20 on held-out data:
 1. Conclude: SSL-based regime detection is not straightforwardly feasible on EUR/USD hourly data with these three features
 2. Investigate in order: (a) try daily resolution instead of hourly, (b) expand feature set to 6 features (add DXY, yield spread), (c) try 4-class regime label (HV quartiles) instead of binary
 3. If all three investigations fail: conclude SSL regime detection is not viable on FX data; document and consider pivoting the primary hypothesis to distributional forecasting directly
 4. Record result in `results/summaries/phase-0-null.md`
 5. Write failure mode to brain: `brain_write wing=jepa-fx hall=failures`
 ## Risks
 | Risk | Canary | Mitigation |
 |---|---|---|
 | TS-JEPA code unreproducible | Benchmark result doesn't match paper within 2 weeks | Contact authors; fall back to V-JEPA adaptation |
 | MAE encoder collapses | Reconstruction loss plateau in first 10 epochs; all embeddings near-identical | Add batch normalisation; reduce LR; check masking ratio |
 | Regime label too coarse | Silhouette low even with visually structured embeddings | Also evaluate with 4-class HV quartile label |
 | PC1 is just volatility | PC1/HV correlation > 0.95 | Useful finding; record it; do not declare success |
@@ -0,0 +1,82 @@
 # Experiment Spec: Phase 1 — JEPA Representation PoC
 ## Hypothesis
 > We believe that a TS-JEPA encoder trained on G10 FX hourly data (2008–2022) will
 > produce latent market-state embeddings that are structurally separable by volatility
 > regime without explicit regime labels, measurable by silhouette score > 0.35 on
 > k-means clusters evaluated against realised-volatility regime labels on held-out
 > 2023 data including at least one structural break.
 This hypothesis is FALSE if silhouette score ≤ 0.35 OR linear probe R² ≤ 0.40 on
 held-out evaluation.
 **Prerequisites:** Phase 0 passed (MAE silhouette > 0.20, TS-JEPA reproduced).
 ## Background
 Phase 0 confirmed that SSL-based representation learning can find regime structure in
 FX time-series. Phase 1 tests whether JEPA's specific inductive bias (predict target
 embeddings from context embeddings, never reconstruct raw data) produces richer
 representations than a simple MAE — and whether those representations are useful for
 risk management tasks (measurable via linear probe).
 ## Design
 ### Data
 - **Source:** DUKASCopy, all G10 pairs (EUR/USD, GBP/USD, USD/JPY, USD/CHF, AUD/USD, USD/CAD, NZD/USD, EUR/GBP, EUR/JPY, EUR/CHF), hourly
 - **Train:** 2008-01-01 – 2022-12-31 (all 10 pairs, jointly)
 - **Held-out test:** 2023-01-01 – 2023-12-31 (sealed until final evaluation)
 - **Features:** log-return, rolling 20-period HV, VIX (daily → hourly)
 - **Regime label (evaluation only):** rolling 30-day HV percentile; binary + 4-class (quartiles)
 ### Model
 - **Architecture:** TS-JEPA (Ennadir et al., 2025)
  - Context encoder: maps observed window → latent embedding
  - Target encoder: EMA of context encoder (momentum β ≈ 0.996); stop-gradient
  - Predictor: shallow MLP bridging context → target embedding
 - **Masking horizons:** ablate K ∈ {1h, 8h, 24h}; context window = 120h
 - **Training:** multi-pair joint training (one model, all G10 pairs)
 ### Baseline
 - Phase 0 MAE (best masking horizon from Phase 0)
 - PCA on raw features (Phase 0 baseline)
 ### Ablations
 1. TS-JEPA vs MAE — is JEPA's no-reconstruction inductive bias adding value?
 2. Single-pair (EUR/USD only) vs. multi-pair — does joint training improve representations?
 3. Masking horizon K: {1h, 8h, 24h}
 ## Acceptance Criteria
 - [ ] Silhouette score > 0.35 on held-out 2023 data (k-means k=3–5, binary HV label)
 - [ ] Linear probe R² > 0.40 on frozen embeddings vs. realised-vol decile
 - [ ] PC1 / rolling-HV correlation < 0.85 (encoder learning more than volatility level)
 - [ ] TS-JEPA silhouette exceeds Phase 0 MAE silhouette by > 5%
 - [ ] Rerun ×3 within ±10% of reported silhouette
 ## Out of Scope
 - VaR, ES, distributional forecasting (Phase 3)
 - Exotic pairs beyond G10
 - Options pricing, alpha generation, live trading
 - Any data after 2022-12-31 for training; test set opened only for final evaluation
 - Regime detection backtesting (Phase 2 — embedding drift as early warning)
 ## Null Result Protocol
 If primary criteria not met:
 1. If silhouette > 0.20 but ≤ 0.35: JEPA shows partial structure; not sufficient for Phase 2. Investigate whether multi-pair training, longer context window, or additional features close the gap. One retry permitted with documented rationale.
 2. If silhouette ≤ 0.20: regression from Phase 0; investigate JEPA training stability (collapse risk). Do not proceed.
 3. If linear probe R² ≤ 0.40 despite good silhouette: embeddings are structured but not encoding risk-relevant information. Record as a finding; reconsider feature set.
 4. Record all results in `results/summaries/phase-1-[pass|null].md`
 5. Write findings to brain: `brain_write wing=jepa-fx hall=failures`
 ## Risks
 | Risk | Canary | Mitigation |
 |---|---|---|
 | EMA encoder collapses | All embeddings converge to near-zero; loss goes to ~0 early | Verify EMA momentum schedule; check stop-gradient implementation |
 | Encoder learns only EUR/USD vol | PC1 dominated by EUR/USD HV even in multi-pair model | Evaluate per-pair silhouette; if EUR/USD dominates, weight loss by pair |
 | Phase 0 silhouette was data-split artefact | Phase 1 MAE baseline doesn't reproduce Phase 0 numbers | Fix random seeds; document split methodology in Phase 0 |
 | Insufficient regime diversity (2008–2022) | Embedding clusters don't separate 2023 structural break | Verify 2023 test includes high-vol episode; add 4-class label as backup |
@@ -1,214 +0,0 @@
 """train.py — autoresearch agent file (only this may be edited).
 TS-JEPA backbone with SIGReg regularization (Balestriero & LeCun, LeJEPA
 arXiv:2511.08544; time-series placement from ChronoJEPA arXiv: 2505.XXXXX).
 PatchTST-style encoder over windowed daily [return, realized_vol] → FREEZE →
 linear probe predicts NEXT-day realized vol → val_vol_r2 (OOS R²).
 Writes metrics.json — the single scalar the loop reads.
 Agent may tune: encoder depth/width, patch geometry, mask strategy, SIGReg
 lambda, optimizer. Do NOT touch prepare_data.py, loop.py, or the data pipeline.
 """
 import json
 import math
 import numpy as np
 import pandas as pd
 import torch
 import torch.nn as nn
 # --- agent-tunable knobs ---
 WINDOW     = 60     # INCREASED lookback for better volatility persistence capture
 PATCH_LEN  = 5      # time-patch size (must divide WINDOW)
 STRIDE     = 5
 D_MODEL    = 64     # transformer hidden dim - INCREASED for capacity
 DEPTH      = 2      # transformer layers
 N_HEADS    = 4
 MASK_FRAC  = 0.50   # INCREASED mask fraction to force the encoder to learn better global representations
 SIGREG_LAM = 0.01   # SIGReg weight (λ) - REDUCED to allow more representation capacity
 EPOCHS     = 300
 LR         = 3e-4
 SEED       = 0
 # ---------------------------
 torch.manual_seed(SEED)
 np.random.seed(SEED)
 dev = "cuda" if torch.cuda.is_available() else "cpu"
 # ── SIGReg (from LeJEPA/ChronoJEPA, token-level placement) ─────────────────
 def sigreg(tokens: torch.Tensor, knots: int = 17) -> torch.Tensor:
    """Epps-Pulley test statistic pushes token embeddings toward isotropic Gaussian.
    tokens: (B, T, D) — applied per-token, averaged across B and T.
    """
    B, T, D = tokens.shape
    z = tokens.reshape(B * T, D)           # (N, D)
    t = torch.linspace(0, 3, knots, device=z.device, dtype=z.float().dtype)
    dt = 3.0 / (knots - 1)
    w = torch.full((knots,), 2 * dt, device=z.device, dtype=z.float().dtype)
    w[0] = dt; w[-1] = dt
    phi = torch.exp(-t.square() / 2.0)
    A = torch.randn(D, 256, device=z.device, dtype=z.float().dtype)
    A = A / A.norm(p=2, dim=0)
    x_t = (z.float() @ A).unsqueeze(-1) * t              # (N, 256, knots)
    err = (x_t.cos().mean(0) - phi).square() + x_t.sin().mean(0).square()
    return ((err @ (w * phi)) * z.shape[0]).mean()
 # ── Encoder + Predictor ─────────────────────────────────────────────────────
 class PatchEncoder(nn.Module):
    """PatchTST-style encoder for univariate windows."""
    def __init__(self, in_feats, patch_len, stride, d_model, depth, n_heads):
        super().__init__()
        self.patch_len = patch_len
        self.stride    = stride
        self.d_model   = d_model
        self.embed = nn.Linear(patch_len * in_feats, d_model)
        layer = nn.TransformerEncoderLayer(d_model, n_heads, 2 * d_model,
                                           dropout=0.0, batch_first=True)
        self.tf = nn.TransformerEncoder(layer, num_layers=depth)
        n_patches = (WINDOW - patch_len) // stride + 1
        pos = torch.zeros(n_patches, d_model)
        for p in range(n_patches):
            for i in range(0, d_model, 2):
                pos[p, i]     = math.sin(p / 10000 ** (i / d_model))
                if i + 1 < d_model:
                    pos[p, i+1] = math.cos(p / 10000 ** (i / d_model))
        self.register_buffer("pos", pos)
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # x: (B, W, F)  → patches → (B, T, D)
        B, W, F = x.shape
        n_patches = (W - self.patch_len) // self.stride + 1
        patches = torch.stack([x[:, i*self.stride:i*self.stride+self.patch_len, :]
                                .reshape(B, -1) for i in range(n_patches)], dim=1)
        tokens = self.embed(patches) + self.pos[:n_patches]
        return self.tf(tokens)  # (B, T, D)
 class Predictor(nn.Module):
    def __init__(self, d_model):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(d_model, d_model), nn.GELU(),
                                 nn.Linear(d_model, d_model))
    def forward(self, x):
        return self.net(x)
 # ── Data ────────────────────────────────────────────────────────────────────
 def build():
    """Year-based split: encoder trains on 2019-2021; probe evaluates on 2022-2023 OOS."""
    df = pd.read_parquet("data/processed/eurusd_daily.parquet").reset_index(drop=True)
    df["date"] = pd.to_datetime(df["date"])
    feats  = df[["ret", "realized_vol"]].to_numpy(np.float32)
    target = df["realized_vol"].to_numpy(np.float32)
    tr_idx = df.index[df["date"].dt.year <= 2021].tolist()
    te_idx = df.index[df["date"].dt.year >= 2022].tolist()
    mu = feats[:tr_idx[-1]+1].mean(0)
    sd = feats[:tr_idx[-1]+1].std(0) + 1e-8
    fn = (feats - mu) / sd
    def windows(idx):
        X, y = [], []
        for t in idx:
            if t - WINDOW >= 0 and t + 1 < len(df):
                X.append(fn[t - WINDOW:t]); y.append(target[t + 1])
        return np.stack(X).astype(np.float32), np.array(y, np.float32)
    return windows(tr_idx), windows(te_idx)
 # ── Training ─────────────────────────────────────────────────────────────────
 def main():
    (Xtr, ytr), (Xte, yte) = build()
    n_feats  = Xtr.shape[2]
    Xtr_t    = torch.tensor(Xtr, device=dev)
    enc      = PatchEncoder(n_feats, PATCH_LEN, STRIDE, D_MODEL, DEPTH, N_HEADS).to(dev)
    pred     = Predictor(D_MODEL).to(dev)
    opt      = torch.optim.AdamW(list(enc.parameters()) + list(pred.parameters()), lr=LR)
    n_patches = (WINDOW - PATCH_LEN) // STRIDE + 1
    n_mask    = max(1, int(MASK_FRAC * n_patches))
    for ep in range(EPOCHS):
        # JEPA: predict masked-out patch tokens from visible tokens
        idx_mask = torch.randperm(n_patches)[:n_mask]
        ctx_mask = torch.ones(n_patches, dtype=torch.bool, device=dev)
        ctx_mask[idx_mask] = False
        tokens_ctx    = enc(Xtr_t)                 # encode all (B, T, D)
        tokens_target = enc(Xtr_t).detach()        # target (frozen): same input, no grad
        pred_out = pred(tokens_ctx[:, idx_mask, :])
        jepa_loss = ((pred_out - tokens_target[:, idx_mask, :]) ** 2).mean()
        reg_loss  = sigreg(tokens_ctx)
        loss      = jepa_loss + SIGREG_LAM * reg_loss
        opt.zero_grad(); loss.backward(); opt.step()
    enc.eval()
    with torch.no_grad():
        def embed(X_np):
            t = torch.tensor(X_np, device=dev)
            return enc(t).mean(1).cpu().numpy()   # pool over time patches
        Etr = embed(Xtr)
        Ete = embed(Xte)
    # ridge linear probe (closed form)
    A  = np.hstack([Etr, np.ones((len(Etr), 1))])
    w  = np.linalg.solve(A.T @ A + 1e-3 * np.eye(A.shape[1]), A.T @ ytr)
    pred_np = np.hstack([Ete, np.ones((len(Ete), 1))]) @ w
    ss_res  = ((yte - pred_np) ** 2).sum()
    ss_tot  = ((yte - yte.mean()) ** 2).sum()
    val_vol_r2 = float(1 - ss_res / ss_tot)
    json.dump({
        "val_vol_r2": val_vol_r2, "n_test": len(yte),
        "knobs": {"WINDOW": WINDOW, "PATCH_LEN": PATCH_LEN, "STRIDE": STRIDE,
                  "D_MODEL": D_MODEL, "DEPTH": DEPTH, "MASK_FRAC": MASK_FRAC,
                  "SIGREG_LAM": SIGREG_LAM, "EPOCHS": EPOCHS},
    }, open("metrics.json", "w"), indent=2)
    print("val_vol_r2 = %.4f  (n_test=%d, dev=%s)" % (val_vol_r2, len(yte), dev))
    # ── EXPORT BLOCK — do NOT edit (agent boundary) ──────────────────────────
    # Set EXPORT_EMBEDDINGS=1 to write embeddings.json for the Go eval harness.
    # Uses year-based split (train≤2021, OOS≥2022) regardless of probe split.
    import os
    if os.environ.get("EXPORT_EMBEDDINGS") == "1":
        df2 = pd.read_parquet("data/processed/eurusd_daily.parquet").reset_index(drop=True)
        df2["date"] = pd.to_datetime(df2["date"])
        tr_mask = df2["date"].dt.year <= 2021
        feats2 = df2[["ret", "realized_vol"]].to_numpy(np.float32)
        mu2 = feats2[tr_mask].mean(0); sd2 = feats2[tr_mask].std(0) + 1e-8
        fn2 = (feats2 - mu2) / sd2
        def _export_windows(year_mask):
            idx = df2.index[year_mask].tolist()
            Xs, dates, rvs = [], [], []
            for t in idx:
                if t - WINDOW >= 0:
                    Xs.append(fn2[t - WINDOW:t])
                    dates.append(str(df2["date"].iloc[t].date()))
                    rvs.append(float(df2["realized_vol"].iloc[t]))
            if not Xs:
                return [], [], []
            with torch.no_grad():
                E = enc(torch.tensor(np.stack(Xs), device=dev)).mean(1).cpu().numpy().tolist()
            return E, dates, rvs
        Etr, dates_tr, rv_tr = _export_windows(tr_mask)
        Eoos, dates_oos, rv_oos = _export_windows(df2["date"].dt.year >= 2022)
        hv_thr = float(np.percentile(rv_oos, 67))
        hv_label = [1 if v >= hv_thr else 0 for v in rv_oos]
        json.dump({"embeddings": Eoos, "dates": dates_oos,
                   "realized_vol": rv_oos, "hv_label": hv_label,
                   "train_embeddings": Etr, "train_realized_vol": rv_tr},
                  open("embeddings.json", "w"))
        print("exported embeddings.json  train=%d  oos=%d  HV=%d/%d" % (
            len(Etr), len(Eoos), sum(hv_label), len(hv_label)))
    # ── END EXPORT BLOCK ─────────────────────────────────────────────────────
 if __name__ == "__main__":
    main()
Author	SHA1	Message	Date
mathias	e75c05f7d6	docs: add .skills/README.md listing active skills CD / Lint / Test / Vet (pull_request) Failing after 4s Details CD / Build & Import (pull_request) Has been skipped Details CD / Deploy via GitOps (pull_request) Has been skipped Details	2026-05-27 22:00:32 +00:00
mathias	24166a71e3	docs: add model/README.md — Python perimeter policy and setup	2026-05-27 22:00:25 +00:00
mathias	fb2737fc4a	docs: add notebooks/README.md — EDA-only policy, output stripping rule	2026-05-27 22:00:17 +00:00
mathias	09a37a5130	docs: add results/README.md — summary format, null result policy	2026-05-27 22:00:10 +00:00
mathias	95f2fb72db	docs: add experiments/README.md — naming convention and run workflow	2026-05-27 22:00:02 +00:00
mathias	4608c8ee29	docs: add data/README.md — download instructions and split definitions	2026-05-27 21:59:53 +00:00
mathias	a27200cc5e	docs: add Phase 1 experiment spec (JEPA representation PoC)	2026-05-27 21:59:43 +00:00
mathias	4338abf176	docs: add Phase 0 experiment spec (SSL feasibility gate)	2026-05-27 21:59:16 +00:00
mathias	192651cb66	docs: write DECISIONS.md — all post-grill architectural and methodology decisions	2026-05-27 21:58:01 +00:00
mathias	5c24864021	chore: update mcp.json — brain + gitea only (no infra/knowledge for research project)	2026-05-27 21:57:13 +00:00
mathias	414ad1bdb2	chore: replace .gitignore with research-appropriate rules	2026-05-27 21:57:06 +00:00
mathias	2112b3f765	chore: replace Go-web Taskfile with research task definitions	2026-05-27 21:56:58 +00:00
mathias	cf1c47048f	chore: sync AGENTS.md from PROJECT.md	2026-05-27 21:56:40 +00:00
mathias	668f800868	chore: sync CLAUDE.md and AGENTS.md from PROJECT.md	2026-05-27 21:56:33 +00:00
mathias	a2414cf4c9	docs: write PROJECT.md — research conventions, hypothesis, agent instructions	2026-05-27 21:56:28 +00:00
mathias	f7dbe5cc1c	docs: seed README with project overview, hypothesis, phase table	2026-05-27 21:56:04 +00:00