docs: add .skills/README.md listing active skills

.context/PROJECT.md

@@ -4,101 +4,77 @@
 
 - **Name**: jepa-fx-risk
 - **Owner**: Mathias
-- **Client**: personal research
+- **Client**: personal / research
 - **Repo**: gitea.d-ma.be/mathias/jepa-fx-risk
-- **Status**: active — Phase 0
+- **Status**: active
+- **Brain wing**: `jepa-fx` (`wiki/jepa-fx/`)
 
 ## Purpose
 
-Research project: apply JEPA-based self-supervised representation learning to
+Research project. Not a product. The "user" is future-self and research readers. Success is a reproducible, publishable result — not deployment.
-FX risk management for a corporate bank with an internal global FX trading desk.
-
-Primary tasks: FX volatility forecasting and VaR/CVaR estimation.
-Target output: internal PoC for the trading desk.
-
-## Architecture decision (see DECISIONS.md ADR-001)
-
-**TS-JEPA + SIGReg** — TS-JEPA temporal patchwise architecture (Ennadir et al.,
-arXiv:2509.25449) with EMA replaced by Sketched Isotropic Gaussian Regularization
-(SIGReg, Balestriero & LeCun, arXiv:2511.08544). Single search axis: λ ∈ [0.01, 1.0].
-
-Phase 2 hypothesis: MTS-JEPA multi-resolution objective (arXiv:2602.04643).
 
 ## Stack
 
-**Go** (`src/`): data pipeline (DUKASCopy fetch + hourly processing), evaluation
+- **Primary language**: Go 1.24+ (pipeline, eval harness, CLI, experiment runner)
-harness (silhouette, linear probe R², collapse diagnostic, Kupiec/Christoffersen),
+- **ML layer**: Python 3.12 + PyTorch (TS-JEPA training loop only — isolated in `model/`)
-results dashboard (Templ + HTMX + CDN Tailwind).
+- **Build**: Task (Taskfile.yml)
+- **Target infra**: koala (Arch + Blackwell GPU, training), iguana (Mac Studio M2, dev)
+- **MCP**: brain (knowledge), gitea (version control)
 
-**Python** (`model/`): all training and embedding export only. PyTorch cu130
+## Research context
-(Blackwell sm_120 compatible). No evaluation logic in Python.
 
-**Infra**: koala (Arch Linux, Blackwell GPU 12 GB VRAM) for training.
+**Primary hypothesis:**
-iguana (Mac Studio M2 Ultra) + LiteLLM on piguard for autoresearch agent LLM.
+> JEPA embeddings trained on FX time-series will produce latent market-state representations structurally separable by regime without explicit labels, measurable by silhouette score > 0.35 on k-means clusters vs. realised-volatility regime labels, on held-out data including at least one structural break.
 
-## Repository layout
+**Current phase:** Phase 0 — SSL Feasibility Gate
 
-```
+**Training cutoff:** 2023-01-01 (hard — never look at post-2023 data during development)
-jepa-fx-risk/
-├── src/                    # Go — data pipeline + eval harness + dashboard
-│   ├── data/               # DUKASCopy fetch, hourly processing, validation
-│   └── eval/               # silhouette, linear probe, collapse, backtest
-├── model/                  # Python — training only
-│   ├── train.py            # TS-JEPA + SIGReg backbone (autoresearch edits this)
-│   ├── prepare.py          # LOCKED — data loading, tokenization, export
-│   └── requirements.txt
-├── specs/                  # Research specs (one per phase/experiment type)
-├── experiments/            # Per-run outputs: embeddings, metrics.json, git tag
-├── results/summaries/      # Human-readable outcome per experiment
-├── program.md              # Autoresearch agenda — researcher edits this
-├── DECISIONS.md            # Architecture Decision Records
-└── Taskfile.yml            # task data:fetch, task experiment:run, task eval:*
-```
 
-## Phase structure
+**Out of scope:** options pricing, directional alpha, live/paper trading, exotic pairs (Phases 1–3)
 
-- **Phase 0** (current): MAE baseline on EUR/USD hourly 2008–2022.
+**Brain wing for prior decisions and failure modes:** `brain_query wing=jepa-fx`
-  Gate: silhouette > 0.20, MAE > PCA, ±10% over 3 reruns.
-- **Phase 1**: TS-JEPA + SIGReg autoresearch sweep, 50 experiments.
-  Gate: val_vol_r2 > GARCH baseline AND Kupiec p > 0.05.
-- **Phase 2**: MTS-JEPA multi-resolution hypothesis.
-- **Phase 3**: Internal bank tick/position data (future, out of current scope).
 
-## Data
+## Conventions
 
-DUKASCopy hourly OHLCV, 10 G10 pairs, 2008–2022 train / 2023 val / 2024 test.
+### Scientific discipline
-Features: log-return, log rolling-20-period HV, VIX (daily interpolated).
+- Every experiment has a spec in `specs/` before any code runs
-Weekend gaps handled explicitly. See ADR-002.
+- Hypotheses are falsifiable and have quantitative acceptance criteria
+- Null results are recorded and published — not discarded
+- Training cutoff is sacred — post-2023 data never informs any design decision
+- Results reported with baselines; no cherry-picking
 
-## Key conventions
+### Code style
+- Go: `gofumpt`, `golangci-lint` with project config; table-driven tests; `testify`
+- Errors: `fmt.Errorf("context: %w", err)` — no naked returns
+- Python: `ruff` for lint; type hints throughout; `pytest` for tests
+- No Jupyter notebooks for anything reproducible — notebooks are EDA scratch only
 
-- `prepare.py` is LOCKED — never modified by agents or autoresearch
+### Git
-- Evaluation metrics are always computed by the Go harness, never in Python
+- Conventional commits: `feat:`, `fix:`, `chore:`, `docs:`, `experiment:`, `result:`
-- Every experiment gets a git tag: `exp/YYYYMMDD-description`
+- Branch: `feat/`, `experiment/phase-N-description`, `fix/`
-- Null results are recorded explicitly in `results/summaries/` — do not iterate silently
+- Every experiment run gets a git tag: `exp/YYYYMMDD-short-description`
-- `program.md` is the only file the researcher edits to steer autoresearch
+- PRs: one concern per PR; description explains *why* not *what*
-- CI runs `task check` (lint + vet + test) only — no GPU, no training
 
-## Evaluation metrics
+### Experiment discipline
+- One spec per phase in `specs/` — written before any implementation
+- Each run recorded in `experiments/YYYYMMDD-HHMMSS-description/`
+- Metric summaries committed to `results/summaries/` — large outputs gitignored
+- `task check` must pass before any commit
 
-Primary (autoresearch optimizes): `val_vol_r2` — linear probe R² on 1-day
+### Security / data
-realized volatility from frozen embeddings, computed by Go harness.
+- No raw FX data committed (gitignored) — see `data/README.md` for reproducible download
+- No API keys or tokens in code — env vars only
+- Training data and results stay local — nothing to cloud unless explicitly decided
 
-Secondary (logged, not optimized): Kupiec p-value (VaR 99% backtest on EUR/USD).
+## Agent instructions
-Must co-move with val_vol_r2 — checked from experiment 1.
 
-Diagnostics: silhouette score (regime clustering), PC1/HV correlation (collapse check).
+When acting as a coding agent on this project:
 
-## Benchmarks to beat (trading desk comparison)
+1. Read this file and all `SKILL.md` files in `.skills/` before starting work
-
+2. Run `brain_query wing=jepa-fx` to load current decisions and failure modes
-- GARCH(1,1) — volatility forecasting baseline
+3. Run `task check` before every commit (lint + vet + test)
-- Historical Simulation VaR (250-day rolling) — Basel default
+4. Check `DECISIONS.md` before making any architecture or methodology choice
-- EWMA RiskMetrics (λ=0.94)
+5. Every experiment needs a spec in `specs/` — no specless experiments
-
+6. Never touch post-2023 data during development; it is sealed
-## Agent guidance
+7. Record null results honestly — do not iterate until metrics pass without noting it
-
+8. When adding a Python dependency, justify it; prefer pure Go alternatives
-Read `DECISIONS.md` before making architecture suggestions.
-Do not modify `prepare.py` or `src/eval/`.
-Do not suggest changing the Python/Go separation.
-Training runs are always manual via `task experiment:run` — never triggered by CI.
-When implementing, follow Go conventions in `.skills/go-patterns/SKILL.md`.

.context/mcp.json

@@ -1,8 +1,18 @@
 {
   "mcpServers": {
-    "knowledge": {
+    "brain": {
-      "url": "http://localhost:3100/mcp",
+      "type": "http",
-      "description": "Project knowledge base — vector + graph retrieval"
+      "url": "https://brain-mcp.d-ma.be/mcp",
+      "headers": {
+        "Authorization": "Bearer ${BRAIN_MCP_TOKEN}"
+      }
+    },
+    "gitea": {
+      "type": "http",
+      "url": "https://git-mcp.d-ma.be/mcp",
+      "headers": {
+        "Authorization": "Bearer ${GITEA_MCP_TOKEN}"
+      }
     }
   }
 }

.gitignore

@@ -1,36 +1,39 @@
-# ---> Go
+# Go build artifacts
-# If you prefer the allow list template instead of the deny list, see community template:
+bin/
-# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
-#
-# Binaries for programs and plugins
-*.exe
-*.exe~
-*.dll
-*.so
-*.dylib
-
-# Test binary, built with `go test -c`
 *.test
-
-# Output of the go coverage tool, specifically when used with LiteIDE
 *.out
 
-# Dependency directories (remove the comment below to include it)
+# Go workspace
-# vendor/
-
-# Go workspace file
 go.work
 go.work.sum
 
-# env file
+# Environment
 .env
 
-# Project-specific
+# Python
-bin/
+model/.venv/
-*.templ.go
+model/__pycache__/
+model/**/__pycache__/
+model/**/*.pyc
+model/.pytest_cache/
+model/**/.pytest_cache/
+model/.ruff_cache/
 
-# python venv (autoresearch loop)
+# Data — never commit raw or processed FX data
-.venv/
+data/raw/
+data/processed/
+data/cache/
 
-# downloaded + processed market data (track via DVC/MinIO, #10 — not git)
+# Experiment outputs — commit summaries only (results/summaries/)
-data/
+experiments/*/embeddings/
+experiments/*/checkpoints/
+experiments/*/logs/
+
+# Large results — commit metric tables and figures only
+results/raw/
+
+# Notebooks — never commit outputs
+notebooks/**/.ipynb_checkpoints/
+
+# OS
+.DS_Store

.skills/README.md

@@ -0,0 +1,13 @@
+This directory contains skill symlinks for this project.
+
+Agents: load all SKILL.md files in subdirectories before starting work.
+
+Active skills:
+- experiment-spec  — write experiment specs before running any phase
+- feature-spec     — write component specs before implementing within a phase
+- tdd              — acceptance criteria map to tests; red-green-refactor
+- grill-me         — stress-test specs and hypotheses before committing
+- session-retrospective — after each phase, surface learnings for the brain
+- clean-code       — Go and Python style conventions
+- planning         — break phases into trackable tasks
+- debug            — systematic debugging approach

AGENTS.md

@@ -1,13 +1,5 @@
-# hostexecutor
+# jepa-fx-risk — Agent context
+# Auto-generated from .context/PROJECT.md by `task context:sync`
+# Do not edit directly.
 
-## Identity
+See .context/PROJECT.md for the canonical source.
-
-- **Name**: hostexecutor
-- **Owner**: Mathias
-- **Client**: personal
-- **Repo**: gitea.d-ma.be/mathias/hostexecutor
-- **Status**: active
-
-## Stack
-
-Go + Templ + HTMX + CDN Tailwind. See `~/dev/.context/AGENT.md` for cross-project conventions.

CLAUDE.md

@@ -1,13 +1,5 @@
-# hostexecutor
+# jepa-fx-risk — Agent context
+# Auto-generated from .context/PROJECT.md by `task context:sync`
+# Do not edit directly.
 
-## Identity
+See .context/PROJECT.md for the canonical source.
-
-- **Name**: hostexecutor
-- **Owner**: Mathias
-- **Client**: personal
-- **Repo**: gitea.d-ma.be/mathias/hostexecutor
-- **Status**: active
-
-## Stack
-
-Go + Templ + HTMX + CDN Tailwind. See `~/dev/.context/AGENT.md` for cross-project conventions.

DECISIONS.md

@@ -1,215 +1,120 @@
-# Architecture Decision Records
+# DECISIONS.md
 
-This file records significant technical and research decisions for `jepa-fx-risk`.
+Architecture and methodology decisions for `jepa-fx-risk`. Every non-obvious choice lives here with its rationale. Agents: read this before making any design decision.
-Each record is immutable once merged — append new records rather than editing old ones.
+
-Format: ID · Date · Status · Context · Decision · Rationale · Consequences.
+Last updated: 2026-05-27
 
 ---
 
-## ADR-001 · Architecture: TS-JEPA + SIGReg as Phase 1 backbone
+## Language split: Go-first, Python-minimal
 
-**Date:** 2026-05-28
+**Decision:** Data pipeline, evaluation harness, CLI, and experiment runner in Go 1.24+. Python 3.12 + PyTorch only for the TS-JEPA model training loop, isolated in `model/`.
-**Status:** Accepted
-**Supersedes:** informal decision to use TS-JEPA standalone (pre-ADR)
 
-### Context
+**Rationale:** Python's dependency ecosystem is a reliability risk for a multi-year project (conflicting CUDA versions, transitive breakage, environment drift). Go produces a single static binary, has excellent CSV/Parquet support, is fast enough for all non-training workloads, and keeps the reproducible parts of the project dependency-free. The training loop genuinely requires PyTorch — that is the one place Python is unavoidable.
 
-Four JEPA variants were evaluated for FX volatility forecasting and VaR/CVaR estimation:
+**Boundary:** `model/` is the Python perimeter. Nothing outside it imports Python.
-
-| Variant | Origin | Key property |
-|---|---|---|
-| TS-JEPA | Ennadir et al., Sep 2025 | Time-series native; EMA collapse prevention |
-| LeJEPA | Balestriero & LeCun, Nov 2025 | Proven optimal embeddings (isotropic Gaussian); SIGReg |
-| MTS-JEPA | He et al., Feb 2026 | Multi-resolution + codebook; no public code |
-| Var-JEPA | Multiple, Mar 2026 | ELBO-based UQ; no public code |
-
-Key constraints: 12 GB VRAM (Blackwell, koala), hourly DUKASCopy data, internal PoC target,
-autoresearch loop requires a clean single-scalar search space, trading desk requires an
-explainable theoretical story.
-
-### Decision
-
-Use **TS-JEPA architecture with SIGReg replacing EMA** as the Phase 1 backbone.
-
-Concretely:
-- Start from the TS-JEPA open-source implementation (arXiv:2509.25449, GitHub)
-- Remove the EMA target-network mechanism
-- Replace it with Sketched Isotropic Gaussian Regularization (SIGReg) from LeJEPA
-  (arXiv:2511.08544), controlled by a single λ hyperparameter
-- Keep TS-JEPA's temporal patchwise masking and Transformer encoder unchanged
-
-### Rationale
-
-**Why not pure TS-JEPA:** EMA is a heuristic; λ interacts with EMA momentum and
-learning rate, creating a three-way search space that is hard to navigate with autoresearch.
-EMA also has no theoretical non-stationarity guarantee.
-
-**Why not pure LeJEPA:** The reference implementation targets vision (multi-crop views).
-Adapting it to temporal patchwise masking requires non-trivial surgery and moves away from
-open code. TS-JEPA's masking is already the right inductive bias for time series.
-
-**Why the hybrid:** SIGReg is architecture-agnostic — it operates on the embedding
-distribution, not the encoder structure. Swapping EMA for SIGReg is a ~20-line change to
-TS-JEPA's training loop. The result is:
-- Time-series native (TS-JEPA masking + patch structure)
-- Provably collapse-free without heuristics (SIGReg)
-- Single search axis for autoresearch (λ ∈ [0.01, 1.0])
-- Non-stationarity robustness proven formally (arXiv:2602.19373 extends LeJEPA
-  guarantees to non-stationary target distributions — directly relevant to FX)
-- Explainable to a model validation team: "embeddings are provably optimal for
-  downstream prediction under distributional uncertainty"
-
-**Why not MTS-JEPA or Var-JEPA now:** Both lack public code (as of May 2026).
-MTS-JEPA's multi-resolution objective is the right next hypothesis (see ADR-003).
-Var-JEPA's ELBO-based UQ is a compelling future direction for CVaR estimation.
-
-### Consequences
-
-- Phase 0 (MAE baseline) is unaffected — it precedes the JEPA architecture choice
-- Issue #3 (TS-JEPA reproduction) is still the right first step; SIGReg is added after
-  reproduction is confirmed
-- The autoresearch `program.md` primary search axis is λ (SIGReg weight)
-- Secondary axes: masking block size, patch stride, encoder depth
-- `model/requirements.txt` must include the SIGReg implementation (≈20 lines,
-  can be vendored directly)
 
 ---
 
-## ADR-002 · Data: DUKASCopy hourly G10 FX as primary training data
+## Primary hypothesis
 
-**Date:** 2026-05-28
+**Decision:** Regime detection via embedding structural separability is the primary hypothesis. Distributional VaR forecasting is secondary and only pursued if Phase 1 and 2 succeed.
-**Status:** Accepted
 
-### Context
+**Rationale:** Regime detection is testable in Phase 1 without regulatory-quality outputs. If embeddings don't show regime structure, distributional VaR will also fail. Choosing a primary hypothesis prevents the project from retreating from one to the other on failure.
 
-Data scale is the most dangerous assumption for any SSL/JEPA approach. Daily FX data
+**Success criterion (Phase 1):** Silhouette score > 0.35 on k-means clusters (k=3–5) vs. realised-volatility regime label (rolling 30-day HV percentile, high/low), computed on held-out test data including at least one structural break.
-(~5,000 samples over 20 years) is insufficient for self-supervised pretraining.
-Two alternatives were considered: daily public data (yfinance) vs. hourly tick data
-(DUKASCopy, free, rate-limited HTTP API).
-
-### Decision
-
-Use **DUKASCopy hourly OHLCV** as the primary data source.
-
-- 10 G10 pairs: EURUSD, GBPUSD, USDJPY, USDCHF, AUDUSD, NZDUSD, USDCAD,
-  EURGBP, EURJPY, GBPJPY
-- Training window: 2008-01-01 – 2022-12-31 (~175,000 samples per pair)
-- Validation window: 2023-01-01 – 2023-12-31 (~2,600 samples)
-- Test window: 2024-01-01 – 2024-12-31 (held out, never seen during development)
-- Features per bar: log-return, log rolling-20-period HV, VIX (daily interpolated)
-- Weekend gaps handled explicitly — no interpolation across market close
-
-### Rationale
-
-Hourly data gives ~35× more samples than daily. This is the minimum threshold for
-JEPA-style SSL to show a training signal within 10-minute autoresearch experiments.
-DUKASCopy is free, reliable, and provides consistent tick-level source data back to 2003.
-
-### Consequences
-
-- The Go data pipeline (Issue #2) is the critical path for everything else
-- Phase 0 MAE baseline trains on the same 2008-2022 window
-- Daily data (yfinance) may still be used for VIX and rate differentials as auxiliary features
 
 ---
 
-## ADR-003 · Research roadmap: Phase structure and JEPA variant progression
+## Training data: 2008–2022, all G10 pairs
 
-**Date:** 2026-05-28
+**Decision:** Train on 15 years of all G10 FX pairs from DUKASCopy, 2008–2022. Do not start with a single 5-year EUR/USD window.
-**Status:** Accepted
 
-### Decision
+**Rationale:** A single 5-year window (e.g. 2019–2024) is dominated by one or two regimes and gives the encoder insufficient regime diversity to learn regime-sensitive representations. Training on 2008–2022 ensures the encoder sees: GFC (2008), EUR sovereign debt (2011–2012), SNB cap removal (2015), COVID (2020), USD rate cycle (2022). Multi-pair training also allows cross-currency transfer (Phase 4).
 
-Three-phase research roadmap:
+**Alternative considered:** Start simple with EUR/USD only, expand later. Rejected because regime diversity in training is a structural requirement, not a nice-to-have. Retrofitting it in Phase 2 would require retraining from scratch.
-
-**Phase 0 — SSL feasibility gate (MAE baseline)**
-Implement a 1D temporal MAE (not JEPA) on EUR/USD hourly 2008-2022.
-Gate criteria: silhouette > 0.20 on 2023 held-out, MAE > PCA baseline, ±10% over 3 reruns.
-Purpose: validate that the data and eval harness work before committing to JEPA complexity.
-If gate fails: follow null result protocol in `specs/phase-0-ssl-feasibility.md`.
-
-**Phase 1 — TS-JEPA + SIGReg autoresearch sweep**
-Primary architecture per ADR-001.
-Autoresearch loop: `program.md`-driven, 10-min experiments, 50-experiment budget.
-Primary metric: `val_vol_r2` (linear probe R² on 1-day realized volatility).
-Gate criteria: `val_vol_r2` > GARCH-implied baseline AND Kupiec p-value > 0.05 on
-EUR/USD VaR 99%.
-Kupiec is logged from experiment 1 to verify it co-moves with `val_vol_r2`.
-
-**Phase 2 — MTS-JEPA multi-resolution hypothesis**
-Introduce parallel multi-scale predictive pathways (1h, 8h, 24h context windows)
-adapted from MTS-JEPA (arXiv:2602.04643).
-Hypothesis: multi-scale representations improve regime detection (silhouette) and
-reduce VaR exceedance clustering (Christoffersen test).
-Prerequisite: Phase 1 gate passed AND MTS-JEPA code available or reproducible from paper.
-Time-box: if MTS-JEPA code not available within 4 weeks of Phase 2 start, implement
-multi-resolution masking from scratch using Phase 1 backbone as base.
-
-**Phase 3 — Internal bank data (future)**
-Replace DUKASCopy pipeline with internal tick feed adapter.
-Fine-tune heads only; backbone frozen or lightly fine-tuned.
-Out of scope for current PoC cycle.
-
-### Consequences
-
-- Issue #5 (Phase 0 MAE) is the unblocked next executable step
-- Phase 1 autoresearch is blocked until Phase 0 passes its gate
-- Var-JEPA (ELBO-based UQ) is a named future hypothesis for CVaR estimation in Phase 2+
-  but not on the critical path
 
 ---
 
-## ADR-004 · Evaluation: Go harness + Python training separation
+## Hard training cutoff: 2023-01-01
 
-**Date:** 2026-05-28
+**Decision:** All data from 2023-01-01 onward is sealed. No architecture, hyperparameter, or methodology decision may be informed by post-2023 data. Post-2023 test set opened only once, for final evaluation.
-**Status:** Accepted
 
-### Decision
+**Rationale:** Out-of-sample integrity is essential for publishability and honest self-assessment. The held-out window (2023–2026) includes: 2023 US regional bank stress, 2024 JPY intervention episodes. These are the test of genuine generalisation.
 
-Hard separation between training (Python) and evaluation (Go):
+**Enforcement:** `data/raw/` is gitignored. The download script hard-stops at 2022-12-31 for training splits. Any deviation requires a DECISIONS.md entry explaining why.
-
-- **Python** (`model/`): all training, embedding export, model checkpointing
-- **Go** (`src/eval/`): all evaluation metrics — silhouette, linear probe R², collapse
-  diagnostic, Kupiec/Christoffersen backtests
-- Interface: Python exports embedding matrices + labels to `experiments/RUNID/` as
-  `.npy` files; Go eval harness reads them and writes `metrics.json`
-
-### Rationale
-
-Go evaluation gives deterministic, fast, auditable metric computation with proper
-unit tests. It decouples the experimental loop from the training framework, making
-it possible to re-evaluate any past experiment without re-running training.
-The Go layer also serves as the foundation for the eventual trading desk dashboard.
-
-### Consequences
-
-- All acceptance criteria in Issues #4 and #5 are specified in terms of Go eval outputs
-- `val_vol_r2` (the autoresearch optimization metric) is computed by the Go harness,
-  not inside the Python training loop
-- Python training loop calls `task eval:probe` as a subprocess after each experiment
-  to get the scalar fed back to autoresearch
 
 ---
 
-## ADR-005 · Compute: Blackwell GPU on koala, PyTorch cu130
+## Phase 0: SSL feasibility gate before JEPA work
 
-**Date:** 2026-05-28
+**Decision:** Before any JEPA-specific implementation, run a Phase 0 experiment: masked autoencoder (MAE) baseline on EUR/USD hourly data. If MAE silhouette < 0.20, SSL-based regime detection is likely not feasible on this data — stop and investigate before proceeding to JEPA.
-**Status:** Accepted
 
-### Decision
+**Rationale:** JEPA's complexity is only justified if the core SSL premise (that latent representations capture regime structure) holds for FX data. A failed MAE experiment tells us this in 2 weeks rather than 4 months. Added after Full Grill session (2026-05-27).
 
-All GPU training runs on koala (Arch Linux, Blackwell GPU, 12 GB VRAM).
+**Go/no-go threshold:** MAE silhouette < 0.20 on held-out 2023 data → pause, investigate, do not proceed to Phase 1.
-PyTorch install: `pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu130`
-(CUDA 13.0 wheel — required for sm_120 Blackwell support; stable as of May 2026).
-Driver requirement: NVIDIA R570+, CUDA toolkit 12.8+.
 
-Ollama on iguana (Mac Studio M2 Ultra) serves the autoresearch agent LLM via the
+---
-existing LiteLLM proxy on piguard. Agent calls never hit koala directly.
 
-### Consequences
+## Architecture: TS-JEPA as starting implementation
 
-- `model/requirements.txt` must NOT pin torch to a cu124 or earlier wheel
+**Decision:** Use TS-JEPA (Ennadir et al., 2025) as the starting JEPA implementation. MTS-JEPA (He et al., 2026) is the upgrade path if multi-resolution proves necessary.
-- CI (Issue #7) must NOT run GPU tests — CPU-only for unit tests, GPU only via
+
-  `task experiment:run` on koala
+**Rationale:** TS-JEPA is simpler. Validate the concept before adding multi-resolution complexity. If Phase 1 succeeds with TS-JEPA, MTS-JEPA is an ablation, not a prerequisite.
-- 12 GB VRAM is sufficient for <5M parameter models at batch=64; monitor if
+
-  autoresearch explores larger architectures
+**Risk:** Both are preprints. Code reproducibility is unconfirmed. First task of Phase 0 is reproducing TS-JEPA on the paper's own benchmark — if this takes > 2 weeks, contact authors or fall back to implementing JEPA masking from scratch using V-JEPA codebase as reference.
+
+---
+
+## Input features: minimal for Phase 0/1
+
+**Decision:** Phase 0 and Phase 1 use three features only: log-return (hourly), rolling 20-period realised volatility (hourly), VIX (daily, interpolated to hourly).
+
+**Rationale:** Too many input features in early phases makes it impossible to distinguish "JEPA learned regime structure" from "JEPA learned to encode a feature that correlates with regime." Minimal features reduce confounding.
+
+**Expansion path:** Add DXY, G10 vol surface, yield spreads in Phase 2 if Phase 1 succeeds.
+
+---
+
+## Collapse diagnostic
+
+**Decision:** If PC1 of JEPA embeddings correlates > 0.85 with rolling 30-day HV, treat Phase 1 as a partial failure — the encoder learned volatility level, not regime structure. This is a useful finding but not the hypothesis.
+
+**Rationale:** EUR/USD hourly returns are strongly heteroskedastic. A JEPA encoder trained to predict future embeddings will strongly tend to encode current volatility as its primary latent dimension. This is predictable, not regime-sensitive. The linear probe and PC1 diagnostic together distinguish "learned volatility" from "learned regime."
+
+---
+
+## Linear probe as mandatory Phase 1 exit gate
+
+**Decision:** Train a linear model on frozen JEPA embeddings to predict realised volatility decile. R² < 0.4 → embeddings are not encoding useful risk structure → do not proceed to Phase 2.
+
+**Rationale:** If a simple linear model cannot extract volatility regime from the embeddings, the representations are not useful for risk management purposes regardless of their silhouette score. Interpretability-by-linear-probe is the minimum bar for any downstream use.
+
+---
+
+## Explicit out-of-scope (Phases 1–3)
+
+The following are explicitly out of scope for this research programme and go to a parking lot if they arise:
+
+- Options / derivatives pricing
+- Directional alpha generation
+- Live or paper trading
+- Exotic pairs beyond G10
+- Institutional deployment
+- Real-time inference systems
+
+---
+
+## Experiment spec required before any experiment
+
+**Decision:** Every experiment phase must have a written spec in `specs/` before any code runs. No specless experiments.
+
+**Rationale:** Consistent with spec-driven-dev way of working. Specs force falsifiable hypothesis statement, quantitative acceptance criteria, and explicit null-result protocol before results are known — preventing post-hoc rationalisation.
+
+---
+
+## Null results are results
+
+**Decision:** Null results (hypothesis rejected) are recorded in `results/summaries/` and treated as valid research outputs, not failures to be iterated away silently.
+
+**Rationale:** A confirmed null result (e.g. "SSL cannot find regime structure in FX hourly data") is publishable and scientifically valuable. Iterating hyperparameters until metrics pass without recording the failed attempts is p-hacking. Humble attitude; scientific approach.

README.md

@@ -1,13 +1,66 @@
-# hostexecutor
+# jepa-fx-risk
 
-> Generated from `mathias/template-go-web`.
+Research project exploring JEPA (Joint Embedding Predictive Architecture) as a framework for latent representation learning applied to FX trading risk management.
 
-## Bootstrap
+**Primary hypothesis:** JEPA embeddings trained on FX time-series will produce latent market-state representations that are structurally separable by regime without explicit regime labels, measurable by silhouette score on k-means clusters validated against held-out realised-volatility regime labels.
 
-After creating from template, run:
+**Current phase:** Phase 0 — SSL Feasibility Gate (not yet started)
+
+**Brain wing:** `jepa-fx` — query via `brain_query wing=jepa-fx`
+
+---
+
+## Architecture
+
+This is a **Go-first, Python-minimal** research project.
+
+| Layer | Language | Rationale |
+|---|---|---|
+| Data pipeline | Go | Type-safe, fast, no dependency hell |
+| Experiment runner | Go | CLI tooling, reproducible invocations |
+| Evaluation harness | Go | Silhouette, linear probe, collapse diagnostics |
+| Model training | Python + PyTorch | TS-JEPA requires it; kept minimal and isolated |
+
+## Structure
+
+```
+specs/          experiment specs (one per phase)
+src/            Go packages: pipeline, eval, cmd
+model/          Python: TS-JEPA training loop only
+experiments/    one directory per run (gitignored except summaries)
+results/        tracked: metric tables, key figures
+data/           gitignored; see data/README.md for download instructions
+notebooks/      EDA only; outputs stripped before commit
+docs/           ADRs and research notes
+```
+
+## Quick start
 
 ```bash
-go mod tidy       # regenerate go.sum with real module path
+task data:fetch          # download DUKASCopy G10 tick data
-task generate     # generate templ files
+task pipeline:build      # build Go data pipeline
-task build        # build the binary
+task check               # lint + vet + test
+task experiment:run -- --spec specs/phase-0-ssl-feasibility.md
 ```
+
+## Phases
+
+| Phase | Goal | Status |
+|---|---|---|
+| 0 | SSL feasibility gate — MAE baseline on FX data | 🔲 Not started |
+| 1 | JEPA representation PoC — silhouette > 0.35 | 🔲 Not started |
+| 2 | Regime detection validation — recall > 70%, lead > 5d | 🔲 Not started |
+| 3 | Distributional forecasting — VaR passes Kupiec | 🔲 Not started |
+| 4 | Multi-pair transfer — exotic pair improvement | 🔲 Not started |
+
+## Key constraints
+
+- **Training cutoff:** 2023-01-01 — all post-2023 data is sealed for final evaluation
+- **Training data:** 2008–2022, all G10 pairs, DUKASCopy
+- **Out of scope:** options pricing, directional alpha, live trading, exotic pairs (Phases 1–3)
+- **End-state:** publishable research result — not institutional deployment
+
+## Related
+
+- Brain wing: `wiki/jepa-fx/` — decisions, hypotheses, failure modes
+- Synthesis document: `JEPA_FX_Risk_Synthesis.docx` (session 2026-05-27)

STATUS.md

@@ -1,7 +0,0 @@
-# Autoresearch STATUS
-
-| iter | val_vol_r2 | delta | action | secs | gpu | change |
-|------|-----------|-------|--------|------|-----|--------|
-| 1 | 0.3749 | +0.0928 | KEEP | 2s | gpu=0% vram=10054/12227MiB temp=34°C | iter1 |
-| 1 | 0.3011 | +0.0776 | KEEP | 2s | gpu=0% vram=10054/12227MiB temp=34°C | iter1 |
-| 2 | 0.3032 | +0.0021 | KEEP | 2s | gpu=0% vram=10054/12227MiB temp=35°C | iter2 |

Taskfile.yml

@@ -1,37 +1,79 @@
 version: '3'
 
 tasks:
-  generate:
+
-    desc: Run templ generate
+  # ── Quality gate ──────────────────────────────────────────────────────────
-    cmds: [templ generate]
+
-  build:
-    desc: Build the binary
-    deps: [generate]
-    cmds: [go build -o bin/hostexecutor ./cmd/hostexecutor]
-  run:
-    deps: [build]
-    cmds: [./bin/hostexecutor]
-  test:
-    desc: Run all tests
-    deps: [generate]
-    cmds: [go test ./... -race]
-  lint:
-    cmds: [golangci-lint run ./...]
   check:
-    desc: Lint, vet, and test (used by CI)
+    desc: "Full quality gate: lint + vet + test (run before every commit)"
-    deps: [generate]
     cmds:
-      - golangci-lint run ./...
+      - golangci-lint run ./src/...
-      - go vet ./...
+      - go vet ./src/...
-      - go test ./... -race -count=1
+      - go test ./src/... -race -count=1
+      - cd model && python -m pytest tests/ -q
+
+  test:
+    desc: Run Go tests only
+    cmds: [go test ./src/... -race]
+
+  lint:
+    desc: Lint Go code
+    cmds: [golangci-lint run ./src/...]
+
+  # ── Data ─────────────────────────────────────────────────────────────────
+
+  data:fetch:
+    desc: "Download G10 tick data from DUKASCopy (2003–2022)"
+    cmds: [go run ./src/cmd/fetch --config data/config.yaml]
+
+  data:process:
+    desc: "Resample ticks → hourly OHLCV + features (log-return, rolling HV)"
+    cmds: [go run ./src/cmd/process --input data/raw --output data/processed]
+
+  data:validate:
+    desc: "Validate processed data: gap detection, outlier report, regime coverage"
+    cmds: [go run ./src/cmd/validate --input data/processed]
+
+  # ── Experiment ───────────────────────────────────────────────────────────
+
+  experiment:run:
+    desc: "Run an experiment from a spec file. Usage: task experiment:run -- --spec specs/phase-0-ssl-feasibility.md"
+    cmds: [go run ./src/cmd/experiment {{.CLI_ARGS}}]
+
+  experiment:list:
+    desc: List all recorded experiment runs
+    cmds: [ls -lt experiments/ | head -20]
+
+  # ── Evaluation ───────────────────────────────────────────────────────────
+
+  eval:silhouette:
+    desc: "Compute silhouette score on embedding output. Usage: task eval:silhouette -- --run experiments/RUNID"
+    cmds: [go run ./src/cmd/eval silhouette {{.CLI_ARGS}}]
+
+  eval:probe:
+    desc: "Run linear probe on frozen embeddings vs. realised-vol decile"
+    cmds: [go run ./src/cmd/eval probe {{.CLI_ARGS}}]
+
+  eval:collapse:
+    desc: "Check collapse diagnostic: PC1 correlation with rolling HV"
+    cmds: [go run ./src/cmd/eval collapse {{.CLI_ARGS}}]
+
+  # ── Model (Python) ───────────────────────────────────────────────────────
+
+  model:train:
+    desc: "Train TS-JEPA model. Usage: task model:train -- --config model/configs/phase0.yaml"
+    dir: model
+    cmds: [python train.py {{.CLI_ARGS}}]
+
+  model:setup:
+    desc: "Create Python venv and install model dependencies (uv)"
+    dir: model
+    cmds:
+      - uv venv .venv
+      - uv pip install -r requirements.txt
+
+  # ── Context ──────────────────────────────────────────────────────────────
 
   context:sync:
-    desc: Regenerate all harness-specific context files
+    desc: "Regenerate CLAUDE.md and AGENTS.md from .context/PROJECT.md"
-    cmds:
+    cmds: [bash scripts/context-sync.sh]
-      - bash scripts/context-sync.sh
-  context:sync:claude:
-    cmds: [bash scripts/context-sync.sh claude]
-  context:sync:agents:
-    cmds: [bash scripts/context-sync.sh agents]
-  context:sync:cursor:
-    cmds: [bash scripts/context-sync.sh cursor]

data/README.md

@@ -0,0 +1,36 @@
+# Data
+
+Raw and processed FX data is **gitignored** — never committed.
+
+## Download: DUKASCopy G10 tick data
+
+Use the provided fetch task:
+
+```bash
+task data:fetch    # downloads raw tick data → data/raw/
+task data:process  # resamples to hourly OHLCV + features → data/processed/
+task data:validate # gap detection, outlier report, regime coverage check
+```
+
+## Expected structure (local only)
+
+```
+data/
+  raw/          G10 pairs, tick OHLCV, 2003–present (gitignored)
+  processed/    Hourly log-returns, rolling HV, VIX-merged (gitignored)
+  cache/        Intermediate artefacts (gitignored)
+  config.yaml   Download configuration (committed)
+```
+
+## Pairs
+
+EUR/USD, GBP/USD, USD/JPY, USD/CHF, AUD/USD, USD/CAD, NZD/USD, EUR/GBP, EUR/JPY, EUR/CHF
+
+## Splits
+
+| Split | Date range | Purpose |
+|---|---|---|
+| Train | 2008-01-01 – 2022-12-31 | Model training only |
+| Held-out test | 2023-01-01 – 2023-12-31 | Final evaluation — do not open until evaluation |
+
+**Training cutoff is hard: 2023-01-01. No post-2022 data informs any design decision.**

experiments/README.md

@@ -0,0 +1,32 @@
+# Experiments
+
+One directory per experiment run. Large outputs (embeddings, checkpoints, logs) are gitignored. Only metric summaries are committed to `results/summaries/`.
+
+## Naming convention
+
+```
+YYYYMMDD-HHMMSS-phase-N-short-description/
+  config.yaml       parameters used for this run (committed via results/summaries/)
+  metrics.json      final metric snapshot (committed via results/summaries/)
+  embeddings/       (gitignored — large)
+  checkpoints/      (gitignored — large)
+  logs/             (gitignored — large)
+```
+
+## Git tag convention
+
+Every run that produces reportable metrics gets a tag:
+
+```
+exp/YYYYMMDD-phase-N-description
+```
+
+Example: `exp/20260601-phase-0-mae-baseline`
+
+## Starting a run
+
+```bash
+task experiment:run -- --spec specs/phase-0-ssl-feasibility.md --tag my-run-description
+```
+
+The runner creates the directory, writes config, runs the experiment, and outputs metrics.json.

loop.py

@@ -1,205 +0,0 @@
-"""loop.py — Karpathy-style autoresearch loop for jepa-fx-risk.
-
-Agent (on iguana/berget — NOT koala, whose GPU is reserved for train.py) reads
-program.md + train.py + STATUS.md, proposes ONE change to train.py, we run it,
-keep if val_vol_r2 improved else git-revert. Appends per-iter record to STATUS.md.
-
-    LITELLM_KEY=xxx python loop.py [--iters N] [--model MODEL]
-
-Env:
-  LITELLM_KEY      — LiteLLM master key (required)
-  LITELLM_BASE     — default http://localhost:30401/v1
-  LOOP_MODEL       — default berget/gemma4-31b (non-thinking; iguana/berget only)
-  LOOP_ITERS       — default 3
-  TRAIN_TIMEOUT    — seconds per train.py run, default 120
-"""
-import argparse
-import json
-import os
-import subprocess
-import sys
-import time
-import textwrap
-from pathlib import Path
-
-import urllib.request
-
-LITELLM_BASE = os.environ.get("LITELLM_BASE", "http://localhost:30401/v1")
-LITELLM_KEY  = os.environ.get("LITELLM_KEY", "")
-LOOP_MODEL   = os.environ.get("LOOP_MODEL", "berget/gemma4-31b")
-LOOP_ITERS   = int(os.environ.get("LOOP_ITERS", "3"))
-TRAIN_TIMEOUT = int(os.environ.get("TRAIN_TIMEOUT", "120"))
-STATUS_MD    = Path("STATUS.md")
-METRICS_JSON = Path("metrics.json")
-TRAIN_PY     = Path("train.py")
-
-AGENT_SYSTEM = textwrap.dedent("""\
-    You are the autoresearch agent for jepa-fx-risk. Your job: propose ONE small,
-    targeted change to train.py to improve val_vol_r2 (OOS R² predicting 1-day
-    realized vol from frozen embeddings). Higher is better.
-
-    Rules:
-    - Return ONLY the full new content of train.py — nothing else, no explanation,
-      no markdown fence. Raw Python only.
-    - Change ONE thing at a time (one knob, one structural idea).
-    - Do NOT touch prepare_data.py, loop.py, or the data pipeline — only train.py.
-    - Do NOT add new data sources or new files.
-    - The metric is computed externally from your frozen embeddings; trust it.
-""")
-
-
-def read_file(p: Path) -> str:
-    return p.read_text() if p.exists() else ""
-
-
-def gpu_snapshot() -> str:
-    try:
-        out = subprocess.check_output(
-            ["nvidia-smi", "--query-gpu=utilization.gpu,memory.used,memory.total,temperature.gpu",
-             "--format=csv,noheader,nounits"], timeout=5, text=True
-        ).strip()
-        util, mem_used, mem_total, temp = [x.strip() for x in out.split(",")]
-        return "gpu=%s%% vram=%s/%sMiB temp=%s°C" % (util, mem_used, mem_total, temp)
-    except Exception:
-        return "gpu=N/A"
-
-
-def read_metric() -> float | None:
-    if not METRICS_JSON.exists():
-        return None
-    try:
-        return float(json.loads(METRICS_JSON.read_text())["val_vol_r2"])
-    except Exception:
-        return None
-
-
-def run_train() -> tuple[float | None, float, str]:
-    """Run train.py. Returns (val_vol_r2 or None, wall_secs, stderr_tail)."""
-    t0 = time.time()
-    gpu_before = gpu_snapshot()
-    try:
-        r = subprocess.run(
-            [sys.executable, "train.py"],
-            capture_output=True, text=True, timeout=TRAIN_TIMEOUT,
-        )
-        elapsed = time.time() - t0
-        if r.returncode != 0:
-            return None, elapsed, (r.stderr or r.stdout)[-300:]
-        metric = read_metric()
-        return metric, elapsed, ""
-    except subprocess.TimeoutExpired:
-        return None, TRAIN_TIMEOUT, "TIMEOUT"
-
-
-def call_agent(iteration: int, best_so_far: float | None) -> str:
-    """Ask the LLM agent to edit train.py. Returns new train.py content."""
-    context = "\n\n".join([
-        "# program.md\n" + read_file(Path("program.md")),
-        "# train.py (current)\n" + read_file(TRAIN_PY),
-        "# STATUS.md (history)\n" + read_file(STATUS_MD)[-2000:],
-        "# metrics.json (last run)\n" + read_file(METRICS_JSON),
-        "Iteration %d. Best val_vol_r2 so far: %s. Improve it." % (
-            iteration, "%.4f" % best_so_far if best_so_far is not None else "none yet"
-        ),
-    ])
-    payload = json.dumps({
-        "model": LOOP_MODEL,
-        "messages": [
-            {"role": "system", "content": AGENT_SYSTEM},
-            {"role": "user",   "content": context},
-        ],
-        "temperature": 0.7,
-        "max_tokens": 4096,
-    }).encode()
-    req = urllib.request.Request(
-        LITELLM_BASE + "/chat/completions",
-        data=payload,
-        headers={"Authorization": "Bearer " + LITELLM_KEY,
-                 "Content-Type": "application/json"},
-        method="POST",
-    )
-    resp = urllib.request.urlopen(req, timeout=60)
-    data = json.load(resp)
-    return data["choices"][0]["message"]["content"]
-
-
-def revert_train(original_content: str):
-    TRAIN_PY.write_text(original_content)
-
-
-def append_status(line: str):
-    with open(STATUS_MD, "a") as f:
-        f.write(line + "\n")
-
-
-def main():
-    if not LITELLM_KEY:
-        print("ERROR: set LITELLM_KEY"); sys.exit(1)
-
-    if not STATUS_MD.exists():
-        STATUS_MD.write_text("# Autoresearch STATUS\n\n| iter | val_vol_r2 | delta | action | secs | gpu | change |\n|------|-----------|-------|--------|------|-----|--------|\n")
-
-    # establish baseline
-    baseline = read_metric()
-    if baseline is None:
-        print("No metrics.json — running train.py for baseline...")
-        m, secs, err = run_train()
-        if m is None:
-            print("Baseline run failed:", err); sys.exit(1)
-        baseline = m
-        print("Baseline: val_vol_r2 = %.4f (%.1fs)" % (baseline, secs))
-
-    best = baseline
-    print("Starting loop | model=%s | iters=%d | baseline=%.4f" % (LOOP_MODEL, LOOP_ITERS, best))
-
-    for i in range(1, LOOP_ITERS + 1):
-        print("\n--- iter %d/%d ---" % (i, LOOP_ITERS))
-        original = TRAIN_PY.read_text()
-
-        print("  calling agent (%s)..." % LOOP_MODEL)
-        t_agent = time.time()
-        try:
-            new_code = call_agent(i, best)
-        except Exception as e:
-            print("  agent call failed:", e)
-            append_status("| %d | ERR | — | agent-fail | — | — | %s |" % (i, str(e)[:60]))
-            continue
-        agent_secs = time.time() - t_agent
-        print("  agent replied in %.1fs" % agent_secs)
-
-        # strip accidental markdown fences
-        if new_code.strip().startswith("```"):
-            lines = new_code.strip().splitlines()
-            new_code = "\n".join(lines[1:-1] if lines[-1].strip() == "```" else lines[1:])
-
-        TRAIN_PY.write_text(new_code)
-
-        gpu = gpu_snapshot()
-        print("  running train.py [%s]..." % gpu)
-        metric, secs, err = run_train()
-
-        if metric is None:
-            print("  train.py FAILED — reverting. err:", err[:100])
-            revert_train(original)
-            append_status("| %d | FAIL | — | revert | %.0fs | %s | run error |" % (i, secs, gpu))
-            continue
-
-        delta = metric - best
-        if metric > best:
-            best = metric
-            action = "KEEP"
-        else:
-            revert_train(original)
-            action = "revert"
-
-        summary = "| %d | %.4f | %+.4f | %s | %.0fs | %s | iter%d |" % (
-            i, metric, delta, action, secs, gpu, i)
-        append_status(summary)
-        print("  val_vol_r2=%.4f  delta=%+.4f  action=%s  [%.0fs]" % (metric, delta, action, secs))
-
-    print("\nDone. Best val_vol_r2 = %.4f (baseline was %.4f, delta %+.4f)" % (best, baseline, best - baseline))
-    print("STATUS.md updated.")
-
-
-if __name__ == "__main__":
-    main()

metrics.json

@@ -1,10 +0,0 @@
-{
-  "val_vol_r2": 0.30321519081159654,
-  "n_test": 275,
-  "knobs": {
-    "WINDOW": 20,
-    "EMBED_DIM": 64,
-    "MASK_FRAC": 0.4,
-    "EPOCHS": 200
-  }
-}

model/README.md

@@ -0,0 +1,31 @@
+# model
+
+Python + PyTorch perimeter. This is the only directory in the project that uses Python.
+
+## Contents
+
+```
+model/
+  train.py              TS-JEPA training entry point
+  configs/              YAML configs per phase
+    phase0-mae.yaml
+    phase1-tsjepa.yaml
+  tsjepa/               TS-JEPA model implementation (adapted from paper)
+  tests/                pytest tests for model components
+  requirements.txt      pinned Python dependencies
+  .venv/                (gitignored — created by `task model:setup`)
+```
+
+## Setup
+
+```bash
+task model:setup    # creates .venv and installs requirements via uv
+```
+
+## Dependency policy
+
+Every Python dependency must be justified in a comment in `requirements.txt`. Prefer Go implementations for anything outside the training loop. When adding a new dependency, add an entry to DECISIONS.md explaining why a Go alternative wasn't sufficient.
+
+## Python version
+
+3.12 (pinned in `.python-version`)

notebooks/README.md

@@ -0,0 +1,11 @@
+# Notebooks
+
+Exploratory Data Analysis (EDA) scratch space only.
+
+## Rules
+
+- **Never use notebooks for anything reproducible.** Training, evaluation, and metric computation belong in `src/` with tests.
+- **Strip all outputs before committing.** Use `nbstripout` or equivalent.
+- **Label every notebook with a phase prefix:** `phase0-eda-eurusd-distribution.ipynb`
+
+Notebooks are thinking tools, not research artifacts. If a finding from a notebook is worth keeping, it goes into a spec, a decision in `DECISIONS.md`, or a result in `results/summaries/` — not the notebook itself.

requirements.txt

@@ -1,8 +0,0 @@
-# Python deps for the autoresearch loop (train.py + scripts). Install torch from
-# the cu130 index FIRST (koala Blackwell sm_120, torch 2.12.1+cu130 verified):
-#   pip install torch --index-url https://download.pytorch.org/whl/cu130
-#   pip install -r requirements.txt
-numpy>=2.0
-pandas>=2.2
-pyarrow>=16
-histdata>=1.3        # histdata.com downloader (handles the tk token politely)

results/README.md

@@ -0,0 +1,32 @@
+# Results
+
+Tracked outputs from experiment runs. Large raw outputs are gitignored — only summaries are committed.
+
+## Structure
+
+```
+results/
+  summaries/    metric tables, key figures, per-phase result records (committed)
+  raw/          full embedding outputs, backtest CSVs (gitignored)
+```
+
+## Per-phase result record format
+
+Each concluded phase produces a result record in `summaries/`:
+
+```
+summaries/
+  phase-0-[pass|null].md
+  phase-1-[pass|null].md
+  ...
+```
+
+Each record must include:
+- Phase name and hypothesis
+- Key metrics (silhouette score, R², etc.) with confidence intervals where applicable
+- Baseline comparison
+- Verdict: PASS / NULL RESULT
+- If null: what was investigated, what was found, next step taken
+- Link to experiment git tag
+
+**Null results are valid research outputs and must be committed, not discarded.**

scripts/check_gpu.py

@@ -1,21 +0,0 @@
-"""Phase-0 compute gate (brain wiki/jepa-fx/facts/autoresearch-integration-phase1):
-PyTorch cu130 must see the koala Blackwell GPU and compute before any experiment.
-
-    python scripts/check_gpu.py   # exits 0 if the GPU is usable, 1 otherwise
-
-Note: koala shares this 12GB card with the llama-swap LLM stack. The autoresearch
-agent should run on iguana/berget models so koala's GPU stays free for train.py.
-"""
-import sys
-import torch
-
-print("torch", torch.__version__)
-if not torch.cuda.is_available():
-    print("CUDA NOT AVAILABLE — gate BLOCKED")
-    sys.exit(1)
-print("device:", torch.cuda.get_device_name(0))
-print("capability: sm_%d%d" % torch.cuda.get_device_capability(0))
-x = torch.randn(2000, 2000, device="cuda")
-(x @ x).sum().item()
-torch.cuda.synchronize()
-print("GPU matmul OK — Phase-0 compute gate GREEN")

scripts/fetch_data.py

@@ -1,31 +0,0 @@
-"""Fetch EUR/USD M1 bars from histdata.com (free, research use).
-
-Polite: one request per year, spaced; past years query month=None. Uses the
-maintained `histdata` package which handles histdata's anti-hotlink tk token.
-Output: data/raw/DAT_ASCII_EURUSD_M1_<year>.zip
-
-    YEARS=2019,2020,2021 python scripts/fetch_data.py
-"""
-import os
-import time
-
-from histdata import download_hist_data
-from histdata.api import Platform as P, TimeFrame as T
-
-YEARS = [y.strip() for y in os.environ.get("YEARS", "2019,2020,2021").split(",")]
-
-
-def main():
-    os.makedirs("data/raw", exist_ok=True)
-    for yr in YEARS:
-        f = download_hist_data(
-            year=yr, month=None, pair="eurusd",
-            platform=P.GENERIC_ASCII, time_frame=T.ONE_MINUTE,
-            output_directory="data/raw",
-        )
-        print("fetched", yr, "->", f)
-        time.sleep(2)  # be a good citizen
-
-
-if __name__ == "__main__":
-    main()

scripts/prepare_data.py

@@ -1,57 +0,0 @@
-"""LOCKED data pipeline (toy) — agent must NOT edit (brain Phase-1 contract).
-
-Parses histdata EUR/USD M1 zips → daily series with realized volatility (the
-val_vol_r2 target = 1-day realized vol from intraday squared returns).
-Output: data/processed/eurusd_daily.parquet  [date, close, ret, realized_vol].
-"""
-import glob
-import os
-import zipfile
-
-import numpy as np
-import pandas as pd
-
-RAW = "data/raw"
-OUT = "data/processed/eurusd_daily.parquet"
-
-
-def load_m1() -> pd.DataFrame:
-    frames = []
-    for zp in sorted(glob.glob(os.path.join(RAW, "DAT_ASCII_EURUSD_M1_*.zip"))):
-        with zipfile.ZipFile(zp) as z:
-            csv = [n for n in z.namelist() if n.endswith(".csv")][0]
-            with z.open(csv) as f:
-                df = pd.read_csv(
-                    f, sep=";", header=None,
-                    names=["dt", "open", "high", "low", "close", "vol"],
-                )
-        df["ts"] = pd.to_datetime(df["dt"], format="%Y%m%d %H%M%S")
-        frames.append(df[["ts", "close"]])
-    out = pd.concat(frames).sort_values("ts").reset_index(drop=True)
-    return out
-
-
-def main():
-    m1 = load_m1()
-    m1["r"] = np.log(m1["close"]).diff()
-    m1["day"] = m1["ts"].dt.normalize()
-    daily = m1.groupby("day").agg(
-        close=("close", "last"),
-        realized_vol=("r", lambda x: np.sqrt(np.nansum(x.values ** 2))),
-        n_min=("r", "count"),
-    ).reset_index()
-    daily = daily[daily["n_min"] > 60]  # drop thin days (holidays)
-    daily["ret"] = np.log(daily["close"]).diff()
-    daily = daily.dropna().reset_index(drop=True)
-    os.makedirs(os.path.dirname(OUT), exist_ok=True)
-    daily[["day", "close", "ret", "realized_vol"]].rename(columns={"day": "date"}).to_parquet(OUT)
-    print("rows:", len(daily), "| dates:", daily["day"].min().date(), "→", daily["day"].max().date())
-    # sanity: the COVID crash (March 2020) must show a realized-vol spike
-    rv = daily.set_index("day")["realized_vol"]
-    mar20 = rv["2020-03-01":"2020-03-31"].max()
-    typ = rv["2019-01-01":"2019-12-31"].median()
-    print("median 2019 RV: %.5f | max Mar-2020 RV: %.5f | spike x%.1f" % (typ, mar20, mar20 / typ))
-
-
-if __name__ == "__main__":
-    main()

specs/phase-0-ssl-feasibility.md

@@ -0,0 +1,84 @@
+# Experiment Spec: Phase 0 — SSL Feasibility Gate
+
+## Hypothesis
+
+> We believe that a masked autoencoder (MAE) trained on FX hourly time-series will
+> produce latent embeddings that show structural separability by volatility regime
+> without explicit regime labels, measurable by silhouette score > 0.20 on k-means
+> clusters evaluated against a held-out realised-volatility regime label on 2023 data.
+
+This hypothesis is FALSE if silhouette score ≤ 0.20 on the held-out evaluation.
+
+## Background
+
+Before investing in JEPA-specific machinery, we need to confirm that self-supervised
+representation learning can find regime structure in FX time-series at all. If the
+simplest SSL method (MAE) cannot find structure, JEPA will not either — and the root
+cause needs to be understood before proceeding.
+
+Also validates that TS-JEPA code is reproducible: first task is running TS-JEPA on
+the paper's own benchmark, not on FX data. If reproduction takes > 2 weeks, contact
+authors or fall back to implementing JEPA masking from V-JEPA codebase.
+
+Added post Full Grill (2026-05-27). See DECISIONS.md: "Phase 0: SSL feasibility gate."
+
+## Design
+
+### Data
+- **Source:** DUKASCopy, EUR/USD hourly OHLCV
+- **Train:** 2008-01-01 – 2022-12-31
+- **Held-out test:** 2023-01-01 – 2023-12-31 (sealed until final evaluation)
+- **Features:** log-return (hourly), rolling 20-period realised HV, VIX (daily → hourly interpolation)
+- **Regime label (evaluation only):** rolling 30-day HV percentile; binary: top 50% = high-vol, bottom 50% = low-vol
+
+### Model
+- **Architecture:** 1D temporal Masked Autoencoder
+  - Encoder: 3-layer 1D CNN + positional encoding
+  - Decoder: 2-layer MLP reconstructing masked segment
+- **Masking:** contiguous temporal block (target); context window = 120h (5 days)
+- **Loss:** MSE reconstruction on masked segment
+
+### TS-JEPA reproduction task (runs in parallel / before MAE)
+- Reproduce TS-JEPA paper results on the authors' benchmark dataset
+- Go/no-go: if reproduction fails within 2 weeks → contact authors or pivot to V-JEPA adaptation
+
+### Baseline
+- **PCA** on raw feature vectors (same 120h context window, flattened)
+- Tests whether any dimensionality reduction finds regime structure; confirms SSL is adding something
+
+### Ablations
+- Masking horizon K ∈ {8h, 24h, 72h} — does context length affect embedding quality?
+
+## Acceptance Criteria
+
+- [ ] MAE silhouette score > 0.20 on held-out 2023 data (k-means k=3, vs. binary HV regime label)
+- [ ] MAE silhouette exceeds PCA baseline silhouette
+- [ ] Rerun ×3 within ±10% of reported silhouette (reproducibility)
+- [ ] PC1 / rolling-HV correlation < 0.95 (encoder is not purely encoding volatility level)
+- [ ] TS-JEPA reproduced on paper's benchmark within 2 weeks of start
+
+## Out of Scope
+
+- JEPA implementation (Phase 1)
+- Multi-pair training (Phase 1+)
+- VaR, ES, or any risk metric computation
+- Any data after 2022-12-31 (training); 2023 test set opened only for final evaluation
+- Hyperparameter search beyond the three masking horizons defined above
+
+## Null Result Protocol
+
+If MAE silhouette ≤ 0.20 on held-out data:
+1. Conclude: SSL-based regime detection is not straightforwardly feasible on EUR/USD hourly data with these three features
+2. Investigate in order: (a) try daily resolution instead of hourly, (b) expand feature set to 6 features (add DXY, yield spread), (c) try 4-class regime label (HV quartiles) instead of binary
+3. If all three investigations fail: conclude SSL regime detection is not viable on FX data; document and consider pivoting the primary hypothesis to distributional forecasting directly
+4. Record result in `results/summaries/phase-0-null.md`
+5. Write failure mode to brain: `brain_write wing=jepa-fx hall=failures`
+
+## Risks
+
+| Risk | Canary | Mitigation |
+|---|---|---|
+| TS-JEPA code unreproducible | Benchmark result doesn't match paper within 2 weeks | Contact authors; fall back to V-JEPA adaptation |
+| MAE encoder collapses | Reconstruction loss plateau in first 10 epochs; all embeddings near-identical | Add batch normalisation; reduce LR; check masking ratio |
+| Regime label too coarse | Silhouette low even with visually structured embeddings | Also evaluate with 4-class HV quartile label |
+| PC1 is just volatility | PC1/HV correlation > 0.95 | Useful finding; record it; do not declare success |

specs/phase-1-representation-poc.md

@@ -0,0 +1,82 @@
+# Experiment Spec: Phase 1 — JEPA Representation PoC
+
+## Hypothesis
+
+> We believe that a TS-JEPA encoder trained on G10 FX hourly data (2008–2022) will
+> produce latent market-state embeddings that are structurally separable by volatility
+> regime without explicit regime labels, measurable by silhouette score > 0.35 on
+> k-means clusters evaluated against realised-volatility regime labels on held-out
+> 2023 data including at least one structural break.
+
+This hypothesis is FALSE if silhouette score ≤ 0.35 OR linear probe R² ≤ 0.40 on
+held-out evaluation.
+
+**Prerequisites:** Phase 0 passed (MAE silhouette > 0.20, TS-JEPA reproduced).
+
+## Background
+
+Phase 0 confirmed that SSL-based representation learning can find regime structure in
+FX time-series. Phase 1 tests whether JEPA's specific inductive bias (predict target
+embeddings from context embeddings, never reconstruct raw data) produces richer
+representations than a simple MAE — and whether those representations are useful for
+risk management tasks (measurable via linear probe).
+
+## Design
+
+### Data
+- **Source:** DUKASCopy, all G10 pairs (EUR/USD, GBP/USD, USD/JPY, USD/CHF, AUD/USD, USD/CAD, NZD/USD, EUR/GBP, EUR/JPY, EUR/CHF), hourly
+- **Train:** 2008-01-01 – 2022-12-31 (all 10 pairs, jointly)
+- **Held-out test:** 2023-01-01 – 2023-12-31 (sealed until final evaluation)
+- **Features:** log-return, rolling 20-period HV, VIX (daily → hourly)
+- **Regime label (evaluation only):** rolling 30-day HV percentile; binary + 4-class (quartiles)
+
+### Model
+- **Architecture:** TS-JEPA (Ennadir et al., 2025)
+  - Context encoder: maps observed window → latent embedding
+  - Target encoder: EMA of context encoder (momentum β ≈ 0.996); stop-gradient
+  - Predictor: shallow MLP bridging context → target embedding
+- **Masking horizons:** ablate K ∈ {1h, 8h, 24h}; context window = 120h
+- **Training:** multi-pair joint training (one model, all G10 pairs)
+
+### Baseline
+- Phase 0 MAE (best masking horizon from Phase 0)
+- PCA on raw features (Phase 0 baseline)
+
+### Ablations
+1. TS-JEPA vs MAE — is JEPA's no-reconstruction inductive bias adding value?
+2. Single-pair (EUR/USD only) vs. multi-pair — does joint training improve representations?
+3. Masking horizon K: {1h, 8h, 24h}
+
+## Acceptance Criteria
+
+- [ ] Silhouette score > 0.35 on held-out 2023 data (k-means k=3–5, binary HV label)
+- [ ] Linear probe R² > 0.40 on frozen embeddings vs. realised-vol decile
+- [ ] PC1 / rolling-HV correlation < 0.85 (encoder learning more than volatility level)
+- [ ] TS-JEPA silhouette exceeds Phase 0 MAE silhouette by > 5%
+- [ ] Rerun ×3 within ±10% of reported silhouette
+
+## Out of Scope
+
+- VaR, ES, distributional forecasting (Phase 3)
+- Exotic pairs beyond G10
+- Options pricing, alpha generation, live trading
+- Any data after 2022-12-31 for training; test set opened only for final evaluation
+- Regime detection backtesting (Phase 2 — embedding drift as early warning)
+
+## Null Result Protocol
+
+If primary criteria not met:
+1. If silhouette > 0.20 but ≤ 0.35: JEPA shows partial structure; not sufficient for Phase 2. Investigate whether multi-pair training, longer context window, or additional features close the gap. One retry permitted with documented rationale.
+2. If silhouette ≤ 0.20: regression from Phase 0; investigate JEPA training stability (collapse risk). Do not proceed.
+3. If linear probe R² ≤ 0.40 despite good silhouette: embeddings are structured but not encoding risk-relevant information. Record as a finding; reconsider feature set.
+4. Record all results in `results/summaries/phase-1-[pass|null].md`
+5. Write findings to brain: `brain_write wing=jepa-fx hall=failures`
+
+## Risks
+
+| Risk | Canary | Mitigation |
+|---|---|---|
+| EMA encoder collapses | All embeddings converge to near-zero; loss goes to ~0 early | Verify EMA momentum schedule; check stop-gradient implementation |
+| Encoder learns only EUR/USD vol | PC1 dominated by EUR/USD HV even in multi-pair model | Evaluate per-pair silhouette; if EUR/USD dominates, weight loss by pair |
+| Phase 0 silhouette was data-split artefact | Phase 1 MAE baseline doesn't reproduce Phase 0 numbers | Fix random seeds; document split methodology in Phase 0 |
+| Insufficient regime diversity (2008–2022) | Embedding clusters don't separate 2023 structural break | Verify 2023 test includes high-vol episode; add 4-class label as backup |

train.py

@@ -1,93 +0,0 @@
-"""train.py — the ONLY file the autoresearch agent may edit (Phase-1 contract).
-
-Toy slice: a tiny self-supervised encoder (masked reconstruction of windowed
-daily [return, realized_vol]) → FROZEN → linear probe predicts NEXT-day realized
-vol → val_vol_r2 = OOS R². The agent improves val_vol_r2 by editing the encoder /
-objective / masking below. Writes metrics.json (the scalar the loop reads).
-
-    python train.py
-"""
-import json
-import numpy as np
-import pandas as pd
-import torch
-import torch.nn as nn
-
-# --- agent-tunable knobs ---
-WINDOW = 20
-EMBED_DIM = 64
-MASK_FRAC = 0.40
-EPOCHS = 200
-LR = 1e-3
-SEED = 0
-# ---------------------------
-
-torch.manual_seed(SEED)
-np.random.seed(SEED)
-dev = "cuda" if torch.cuda.is_available() else "cpu"
-
-
-def build():
-    df = pd.read_parquet("data/processed/eurusd_daily.parquet").reset_index(drop=True)
-    feats = df[["ret", "realized_vol"]].to_numpy(np.float32)
-    target = df["realized_vol"].to_numpy(np.float32)  # predict NEXT-day RV
-    X, y = [], []
-    for t in range(WINDOW, len(df) - 1):
-        X.append(feats[t - WINDOW:t])
-        y.append(target[t + 1])
-    X = np.stack(X); y = np.array(y, np.float32)
-    n_tr = int(0.7 * len(X))                      # time-ordered OOS split
-    mu, sd = X[:n_tr].mean((0, 1)), X[:n_tr].std((0, 1)) + 1e-8  # train-only stats
-    X = (X - mu) / sd
-    return (X[:n_tr], y[:n_tr]), (X[n_tr:], y[n_tr:])
-
-
-class Encoder(nn.Module):
-    def __init__(self, win, emb):
-        super().__init__()
-        self.net = nn.Sequential(
-            nn.Flatten(), 
-            nn.Linear(win * 2, 128), 
-            nn.LayerNorm(128),
-            nn.GELU(), 
-            nn.Linear(128, emb)
-        )
-
-    def forward(self, x):
-        return self.net(x)
-
-
-def main():
-    (Xtr, ytr), (Xte, yte) = build()
-    Xtr_t = torch.tensor(Xtr, device=dev)
-    enc = Encoder(WINDOW, EMBED_DIM).to(dev)
-    dec = nn.Sequential(nn.Linear(EMBED_DIM, 128), nn.GELU(), nn.Linear(128, WINDOW * 2)).to(dev)
-    opt = torch.optim.Adam(list(enc.parameters()) + list(dec.parameters()), lr=LR)
-
-    for _ in range(EPOCHS):  # SSL: masked reconstruction of the window
-        mask = (torch.rand_like(Xtr_t) > MASK_FRAC).float()
-        rec = dec(enc((Xtr_t * mask)))
-        loss = (((rec - Xtr_t.flatten(1)) ** 2) * (1 - mask.flatten(1))).mean()
-        opt.zero_grad(); loss.backward(); opt.step()
-
-    enc.eval()
-    with torch.no_grad():  # FROZEN embeddings
-        Etr = enc(Xtr_t).cpu().numpy()
-        Ete = enc(torch.tensor(Xte, device=dev)).cpu().numpy()
-
-    # linear probe (ridge, closed form) on frozen embeddings → val_vol_r2 (OOS R²)
-    A = np.hstack([Etr, np.ones((len(Etr), 1))])
-    w = np.linalg.solve(A.T @ A + 1e-3 * np.eye(A.shape[1]), A.T @ ytr)
-    pred = np.hstack([Ete, np.ones((len(Ete), 1))]) @ w
-    ss_res = ((yte - pred) ** 2).sum()
-    ss_tot = ((yte - yte.mean()) ** 2).sum()
-    val_vol_r2 = float(1 - ss_res / ss_tot)
-
-    json.dump({"val_vol_r2": val_vol_r2, "n_test": len(yte),
-               "knobs": {"WINDOW": WINDOW, "EMBED_DIM": EMBED_DIM, "MASK_FRAC": MASK_FRAC, "EPOCHS": EPOCHS}},
-              open("metrics.json", "w"), indent=2)
-    print("val_vol_r2 = %.4f  (n_test=%d, dev=%s)" % (val_vol_r2, len(yte), dev))
-
-
-if __name__ == "__main__":
-    main()