Author SHA1 Message Date
mathias e75c05f7d6 docs: add .skills/README.md listing active skills
CD / Lint / Test / Vet (pull_request) Failing after 4s
CD / Build & Import (pull_request) Has been skipped
CD / Deploy via GitOps (pull_request) Has been skipped
2026-05-27 22:00:32 +00:00
mathias 24166a71e3 docs: add model/README.md — Python perimeter policy and setup 2026-05-27 22:00:25 +00:00
mathias fb2737fc4a docs: add notebooks/README.md — EDA-only policy, output stripping rule 2026-05-27 22:00:17 +00:00
mathias 09a37a5130 docs: add results/README.md — summary format, null result policy 2026-05-27 22:00:10 +00:00
mathias 95f2fb72db docs: add experiments/README.md — naming convention and run workflow 2026-05-27 22:00:02 +00:00
mathias 4608c8ee29 docs: add data/README.md — download instructions and split definitions 2026-05-27 21:59:53 +00:00
mathias a27200cc5e docs: add Phase 1 experiment spec (JEPA representation PoC) 2026-05-27 21:59:43 +00:00
mathias 4338abf176 docs: add Phase 0 experiment spec (SSL feasibility gate) 2026-05-27 21:59:16 +00:00
mathias 192651cb66 docs: write DECISIONS.md — all post-grill architectural and methodology decisions 2026-05-27 21:58:01 +00:00
mathias 5c24864021 chore: update mcp.json — brain + gitea only (no infra/knowledge for research project) 2026-05-27 21:57:13 +00:00
mathias 414ad1bdb2 chore: replace .gitignore with research-appropriate rules 2026-05-27 21:57:06 +00:00
mathias 2112b3f765 chore: replace Go-web Taskfile with research task definitions 2026-05-27 21:56:58 +00:00
mathias cf1c47048f chore: sync AGENTS.md from PROJECT.md 2026-05-27 21:56:40 +00:00
mathias 668f800868 chore: sync CLAUDE.md and AGENTS.md from PROJECT.md 2026-05-27 21:56:33 +00:00
mathias a2414cf4c9 docs: write PROJECT.md — research conventions, hypothesis, agent instructions 2026-05-27 21:56:28 +00:00
mathias f7dbe5cc1c docs: seed README with project overview, hypothesis, phase table 2026-05-27 21:56:04 +00:00
16 changed files with 643 additions and 343 deletions
+52 -76
View File
@@ -4,101 +4,77 @@
- **Name**: jepa-fx-risk
- **Owner**: Mathias
- **Client**: personal research
- **Client**: personal / research
- **Repo**: gitea.d-ma.be/mathias/jepa-fx-risk
- **Status**: active — Phase 0
- **Status**: active
- **Brain wing**: `jepa-fx` (`wiki/jepa-fx/`)
## Purpose
Research project: apply JEPA-based self-supervised representation learning to
FX risk management for a corporate bank with an internal global FX trading desk.
Primary tasks: FX volatility forecasting and VaR/CVaR estimation.
Target output: internal PoC for the trading desk.
## Architecture decision (see DECISIONS.md ADR-001)
**TS-JEPA + SIGReg** — TS-JEPA temporal patchwise architecture (Ennadir et al.,
arXiv:2509.25449) with EMA replaced by Sketched Isotropic Gaussian Regularization
(SIGReg, Balestriero & LeCun, arXiv:2511.08544). Single search axis: λ ∈ [0.01, 1.0].
Phase 2 hypothesis: MTS-JEPA multi-resolution objective (arXiv:2602.04643).
Research project. Not a product. The "user" is future-self and research readers. Success is a reproducible, publishable result — not deployment.
## Stack
**Go** (`src/`): data pipeline (DUKASCopy fetch + hourly processing), evaluation
harness (silhouette, linear probe R², collapse diagnostic, Kupiec/Christoffersen),
results dashboard (Templ + HTMX + CDN Tailwind).
- **Primary language**: Go 1.24+ (pipeline, eval harness, CLI, experiment runner)
- **ML layer**: Python 3.12 + PyTorch (TS-JEPA training loop only — isolated in `model/`)
- **Build**: Task (Taskfile.yml)
- **Target infra**: koala (Arch + Blackwell GPU, training), iguana (Mac Studio M2, dev)
- **MCP**: brain (knowledge), gitea (version control)
**Python** (`model/`): all training and embedding export only. PyTorch cu130
(Blackwell sm_120 compatible). No evaluation logic in Python.
## Research context
**Infra**: koala (Arch Linux, Blackwell GPU 12 GB VRAM) for training.
iguana (Mac Studio M2 Ultra) + LiteLLM on piguard for autoresearch agent LLM.
**Primary hypothesis:**
> JEPA embeddings trained on FX time-series will produce latent market-state representations structurally separable by regime without explicit labels, measurable by silhouette score > 0.35 on k-means clusters vs. realised-volatility regime labels, on held-out data including at least one structural break.
## Repository layout
**Current phase:** Phase 0 — SSL Feasibility Gate
```
jepa-fx-risk/
├── src/ # Go — data pipeline + eval harness + dashboard
│ ├── data/ # DUKASCopy fetch, hourly processing, validation
│ └── eval/ # silhouette, linear probe, collapse, backtest
├── model/ # Python — training only
│ ├── train.py # TS-JEPA + SIGReg backbone (autoresearch edits this)
│ ├── prepare.py # LOCKED — data loading, tokenization, export
│ └── requirements.txt
├── specs/ # Research specs (one per phase/experiment type)
├── experiments/ # Per-run outputs: embeddings, metrics.json, git tag
├── results/summaries/ # Human-readable outcome per experiment
├── program.md # Autoresearch agenda — researcher edits this
├── DECISIONS.md # Architecture Decision Records
└── Taskfile.yml # task data:fetch, task experiment:run, task eval:*
```
**Training cutoff:** 2023-01-01 (hard — never look at post-2023 data during development)
## Phase structure
**Out of scope:** options pricing, directional alpha, live/paper trading, exotic pairs (Phases 13)
- **Phase 0** (current): MAE baseline on EUR/USD hourly 20082022.
Gate: silhouette > 0.20, MAE > PCA, ±10% over 3 reruns.
- **Phase 1**: TS-JEPA + SIGReg autoresearch sweep, 50 experiments.
Gate: val_vol_r2 > GARCH baseline AND Kupiec p > 0.05.
- **Phase 2**: MTS-JEPA multi-resolution hypothesis.
- **Phase 3**: Internal bank tick/position data (future, out of current scope).
**Brain wing for prior decisions and failure modes:** `brain_query wing=jepa-fx`
## Data
## Conventions
DUKASCopy hourly OHLCV, 10 G10 pairs, 20082022 train / 2023 val / 2024 test.
Features: log-return, log rolling-20-period HV, VIX (daily interpolated).
Weekend gaps handled explicitly. See ADR-002.
### Scientific discipline
- Every experiment has a spec in `specs/` before any code runs
- Hypotheses are falsifiable and have quantitative acceptance criteria
- Null results are recorded and published — not discarded
- Training cutoff is sacred — post-2023 data never informs any design decision
- Results reported with baselines; no cherry-picking
## Key conventions
### Code style
- Go: `gofumpt`, `golangci-lint` with project config; table-driven tests; `testify`
- Errors: `fmt.Errorf("context: %w", err)` — no naked returns
- Python: `ruff` for lint; type hints throughout; `pytest` for tests
- No Jupyter notebooks for anything reproducible — notebooks are EDA scratch only
- `prepare.py` is LOCKED — never modified by agents or autoresearch
- Evaluation metrics are always computed by the Go harness, never in Python
- Every experiment gets a git tag: `exp/YYYYMMDD-description`
- Null results are recorded explicitly in `results/summaries/` — do not iterate silently
- `program.md` is the only file the researcher edits to steer autoresearch
- CI runs `task check` (lint + vet + test) only — no GPU, no training
### Git
- Conventional commits: `feat:`, `fix:`, `chore:`, `docs:`, `experiment:`, `result:`
- Branch: `feat/`, `experiment/phase-N-description`, `fix/`
- Every experiment run gets a git tag: `exp/YYYYMMDD-short-description`
- PRs: one concern per PR; description explains *why* not *what*
## Evaluation metrics
### Experiment discipline
- One spec per phase in `specs/` — written before any implementation
- Each run recorded in `experiments/YYYYMMDD-HHMMSS-description/`
- Metric summaries committed to `results/summaries/` — large outputs gitignored
- `task check` must pass before any commit
Primary (autoresearch optimizes): `val_vol_r2` — linear probe R² on 1-day
realized volatility from frozen embeddings, computed by Go harness.
### Security / data
- No raw FX data committed (gitignored) — see `data/README.md` for reproducible download
- No API keys or tokens in code — env vars only
- Training data and results stay local — nothing to cloud unless explicitly decided
Secondary (logged, not optimized): Kupiec p-value (VaR 99% backtest on EUR/USD).
Must co-move with val_vol_r2 — checked from experiment 1.
## Agent instructions
Diagnostics: silhouette score (regime clustering), PC1/HV correlation (collapse check).
When acting as a coding agent on this project:
## Benchmarks to beat (trading desk comparison)
- GARCH(1,1) — volatility forecasting baseline
- Historical Simulation VaR (250-day rolling) — Basel default
- EWMA RiskMetrics (λ=0.94)
## Agent guidance
Read `DECISIONS.md` before making architecture suggestions.
Do not modify `prepare.py` or `src/eval/`.
Do not suggest changing the Python/Go separation.
Training runs are always manual via `task experiment:run` — never triggered by CI.
When implementing, follow Go conventions in `.skills/go-patterns/SKILL.md`.
1. Read this file and all `SKILL.md` files in `.skills/` before starting work
2. Run `brain_query wing=jepa-fx` to load current decisions and failure modes
3. Run `task check` before every commit (lint + vet + test)
4. Check `DECISIONS.md` before making any architecture or methodology choice
5. Every experiment needs a spec in `specs/` — no specless experiments
6. Never touch post-2023 data during development; it is sealed
7. Record null results honestly — do not iterate until metrics pass without noting it
8. When adding a Python dependency, justify it; prefer pure Go alternatives
+13 -3
View File
@@ -1,8 +1,18 @@
{
"mcpServers": {
"knowledge": {
"url": "http://localhost:3100/mcp",
"description": "Project knowledge base — vector + graph retrieval"
"brain": {
"type": "http",
"url": "https://brain-mcp.d-ma.be/mcp",
"headers": {
"Authorization": "Bearer ${BRAIN_MCP_TOKEN}"
}
},
"gitea": {
"type": "http",
"url": "https://git-mcp.d-ma.be/mcp",
"headers": {
"Authorization": "Bearer ${GITEA_MCP_TOKEN}"
}
}
}
}
+31 -22
View File
@@ -1,30 +1,39 @@
# ---> Go
# If you prefer the allow list template instead of the deny list, see community template:
# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
#
# Binaries for programs and plugins
*.exe
*.exe~
*.dll
*.so
*.dylib
# Test binary, built with `go test -c`
# Go build artifacts
bin/
*.test
# Output of the go coverage tool, specifically when used with LiteIDE
*.out
# Dependency directories (remove the comment below to include it)
# vendor/
# Go workspace file
# Go workspace
go.work
go.work.sum
# env file
# Environment
.env
# Project-specific
bin/
*.templ.go
# Python
model/.venv/
model/__pycache__/
model/**/__pycache__/
model/**/*.pyc
model/.pytest_cache/
model/**/.pytest_cache/
model/.ruff_cache/
# Data — never commit raw or processed FX data
data/raw/
data/processed/
data/cache/
# Experiment outputs — commit summaries only (results/summaries/)
experiments/*/embeddings/
experiments/*/checkpoints/
experiments/*/logs/
# Large results — commit metric tables and figures only
results/raw/
# Notebooks — never commit outputs
notebooks/**/.ipynb_checkpoints/
# OS
.DS_Store
+13
View File
@@ -0,0 +1,13 @@
This directory contains skill symlinks for this project.
Agents: load all SKILL.md files in subdirectories before starting work.
Active skills:
- experiment-spec — write experiment specs before running any phase
- feature-spec — write component specs before implementing within a phase
- tdd — acceptance criteria map to tests; red-green-refactor
- grill-me — stress-test specs and hypotheses before committing
- session-retrospective — after each phase, surface learnings for the brain
- clean-code — Go and Python style conventions
- planning — break phases into trackable tasks
- debug — systematic debugging approach
+4 -12
View File
@@ -1,13 +1,5 @@
# hostexecutor
# jepa-fx-risk — Agent context
# Auto-generated from .context/PROJECT.md by `task context:sync`
# Do not edit directly.
## Identity
- **Name**: hostexecutor
- **Owner**: Mathias
- **Client**: personal
- **Repo**: gitea.d-ma.be/mathias/hostexecutor
- **Status**: active
## Stack
Go + Templ + HTMX + CDN Tailwind. See `~/dev/.context/AGENT.md` for cross-project conventions.
See .context/PROJECT.md for the canonical source.
+4 -12
View File
@@ -1,13 +1,5 @@
# hostexecutor
# jepa-fx-risk — Agent context
# Auto-generated from .context/PROJECT.md by `task context:sync`
# Do not edit directly.
## Identity
- **Name**: hostexecutor
- **Owner**: Mathias
- **Client**: personal
- **Repo**: gitea.d-ma.be/mathias/hostexecutor
- **Status**: active
## Stack
Go + Templ + HTMX + CDN Tailwind. See `~/dev/.context/AGENT.md` for cross-project conventions.
See .context/PROJECT.md for the canonical source.
+86 -181
View File
@@ -1,215 +1,120 @@
# Architecture Decision Records
# DECISIONS.md
This file records significant technical and research decisions for `jepa-fx-risk`.
Each record is immutable once merged — append new records rather than editing old ones.
Format: ID · Date · Status · Context · Decision · Rationale · Consequences.
Architecture and methodology decisions for `jepa-fx-risk`. Every non-obvious choice lives here with its rationale. Agents: read this before making any design decision.
Last updated: 2026-05-27
---
## ADR-001 · Architecture: TS-JEPA + SIGReg as Phase 1 backbone
## Language split: Go-first, Python-minimal
**Date:** 2026-05-28
**Status:** Accepted
**Supersedes:** informal decision to use TS-JEPA standalone (pre-ADR)
**Decision:** Data pipeline, evaluation harness, CLI, and experiment runner in Go 1.24+. Python 3.12 + PyTorch only for the TS-JEPA model training loop, isolated in `model/`.
### Context
**Rationale:** Python's dependency ecosystem is a reliability risk for a multi-year project (conflicting CUDA versions, transitive breakage, environment drift). Go produces a single static binary, has excellent CSV/Parquet support, is fast enough for all non-training workloads, and keeps the reproducible parts of the project dependency-free. The training loop genuinely requires PyTorch — that is the one place Python is unavoidable.
Four JEPA variants were evaluated for FX volatility forecasting and VaR/CVaR estimation:
| Variant | Origin | Key property |
|---|---|---|
| TS-JEPA | Ennadir et al., Sep 2025 | Time-series native; EMA collapse prevention |
| LeJEPA | Balestriero & LeCun, Nov 2025 | Proven optimal embeddings (isotropic Gaussian); SIGReg |
| MTS-JEPA | He et al., Feb 2026 | Multi-resolution + codebook; no public code |
| Var-JEPA | Multiple, Mar 2026 | ELBO-based UQ; no public code |
Key constraints: 12 GB VRAM (Blackwell, koala), hourly DUKASCopy data, internal PoC target,
autoresearch loop requires a clean single-scalar search space, trading desk requires an
explainable theoretical story.
### Decision
Use **TS-JEPA architecture with SIGReg replacing EMA** as the Phase 1 backbone.
Concretely:
- Start from the TS-JEPA open-source implementation (arXiv:2509.25449, GitHub)
- Remove the EMA target-network mechanism
- Replace it with Sketched Isotropic Gaussian Regularization (SIGReg) from LeJEPA
(arXiv:2511.08544), controlled by a single λ hyperparameter
- Keep TS-JEPA's temporal patchwise masking and Transformer encoder unchanged
### Rationale
**Why not pure TS-JEPA:** EMA is a heuristic; λ interacts with EMA momentum and
learning rate, creating a three-way search space that is hard to navigate with autoresearch.
EMA also has no theoretical non-stationarity guarantee.
**Why not pure LeJEPA:** The reference implementation targets vision (multi-crop views).
Adapting it to temporal patchwise masking requires non-trivial surgery and moves away from
open code. TS-JEPA's masking is already the right inductive bias for time series.
**Why the hybrid:** SIGReg is architecture-agnostic — it operates on the embedding
distribution, not the encoder structure. Swapping EMA for SIGReg is a ~20-line change to
TS-JEPA's training loop. The result is:
- Time-series native (TS-JEPA masking + patch structure)
- Provably collapse-free without heuristics (SIGReg)
- Single search axis for autoresearch (λ ∈ [0.01, 1.0])
- Non-stationarity robustness proven formally (arXiv:2602.19373 extends LeJEPA
guarantees to non-stationary target distributions — directly relevant to FX)
- Explainable to a model validation team: "embeddings are provably optimal for
downstream prediction under distributional uncertainty"
**Why not MTS-JEPA or Var-JEPA now:** Both lack public code (as of May 2026).
MTS-JEPA's multi-resolution objective is the right next hypothesis (see ADR-003).
Var-JEPA's ELBO-based UQ is a compelling future direction for CVaR estimation.
### Consequences
- Phase 0 (MAE baseline) is unaffected — it precedes the JEPA architecture choice
- Issue #3 (TS-JEPA reproduction) is still the right first step; SIGReg is added after
reproduction is confirmed
- The autoresearch `program.md` primary search axis is λ (SIGReg weight)
- Secondary axes: masking block size, patch stride, encoder depth
- `model/requirements.txt` must include the SIGReg implementation (≈20 lines,
can be vendored directly)
**Boundary:** `model/` is the Python perimeter. Nothing outside it imports Python.
---
## ADR-002 · Data: DUKASCopy hourly G10 FX as primary training data
## Primary hypothesis
**Date:** 2026-05-28
**Status:** Accepted
**Decision:** Regime detection via embedding structural separability is the primary hypothesis. Distributional VaR forecasting is secondary and only pursued if Phase 1 and 2 succeed.
### Context
**Rationale:** Regime detection is testable in Phase 1 without regulatory-quality outputs. If embeddings don't show regime structure, distributional VaR will also fail. Choosing a primary hypothesis prevents the project from retreating from one to the other on failure.
Data scale is the most dangerous assumption for any SSL/JEPA approach. Daily FX data
(~5,000 samples over 20 years) is insufficient for self-supervised pretraining.
Two alternatives were considered: daily public data (yfinance) vs. hourly tick data
(DUKASCopy, free, rate-limited HTTP API).
### Decision
Use **DUKASCopy hourly OHLCV** as the primary data source.
- 10 G10 pairs: EURUSD, GBPUSD, USDJPY, USDCHF, AUDUSD, NZDUSD, USDCAD,
EURGBP, EURJPY, GBPJPY
- Training window: 2008-01-01 2022-12-31 (~175,000 samples per pair)
- Validation window: 2023-01-01 2023-12-31 (~2,600 samples)
- Test window: 2024-01-01 2024-12-31 (held out, never seen during development)
- Features per bar: log-return, log rolling-20-period HV, VIX (daily interpolated)
- Weekend gaps handled explicitly — no interpolation across market close
### Rationale
Hourly data gives ~35× more samples than daily. This is the minimum threshold for
JEPA-style SSL to show a training signal within 10-minute autoresearch experiments.
DUKASCopy is free, reliable, and provides consistent tick-level source data back to 2003.
### Consequences
- The Go data pipeline (Issue #2) is the critical path for everything else
- Phase 0 MAE baseline trains on the same 2008-2022 window
- Daily data (yfinance) may still be used for VIX and rate differentials as auxiliary features
**Success criterion (Phase 1):** Silhouette score > 0.35 on k-means clusters (k=35) vs. realised-volatility regime label (rolling 30-day HV percentile, high/low), computed on held-out test data including at least one structural break.
---
## ADR-003 · Research roadmap: Phase structure and JEPA variant progression
## Training data: 20082022, all G10 pairs
**Date:** 2026-05-28
**Status:** Accepted
**Decision:** Train on 15 years of all G10 FX pairs from DUKASCopy, 20082022. Do not start with a single 5-year EUR/USD window.
### Decision
**Rationale:** A single 5-year window (e.g. 20192024) is dominated by one or two regimes and gives the encoder insufficient regime diversity to learn regime-sensitive representations. Training on 20082022 ensures the encoder sees: GFC (2008), EUR sovereign debt (20112012), SNB cap removal (2015), COVID (2020), USD rate cycle (2022). Multi-pair training also allows cross-currency transfer (Phase 4).
Three-phase research roadmap:
**Phase 0 — SSL feasibility gate (MAE baseline)**
Implement a 1D temporal MAE (not JEPA) on EUR/USD hourly 2008-2022.
Gate criteria: silhouette > 0.20 on 2023 held-out, MAE > PCA baseline, ±10% over 3 reruns.
Purpose: validate that the data and eval harness work before committing to JEPA complexity.
If gate fails: follow null result protocol in `specs/phase-0-ssl-feasibility.md`.
**Phase 1 — TS-JEPA + SIGReg autoresearch sweep**
Primary architecture per ADR-001.
Autoresearch loop: `program.md`-driven, 10-min experiments, 50-experiment budget.
Primary metric: `val_vol_r2` (linear probe R² on 1-day realized volatility).
Gate criteria: `val_vol_r2` > GARCH-implied baseline AND Kupiec p-value > 0.05 on
EUR/USD VaR 99%.
Kupiec is logged from experiment 1 to verify it co-moves with `val_vol_r2`.
**Phase 2 — MTS-JEPA multi-resolution hypothesis**
Introduce parallel multi-scale predictive pathways (1h, 8h, 24h context windows)
adapted from MTS-JEPA (arXiv:2602.04643).
Hypothesis: multi-scale representations improve regime detection (silhouette) and
reduce VaR exceedance clustering (Christoffersen test).
Prerequisite: Phase 1 gate passed AND MTS-JEPA code available or reproducible from paper.
Time-box: if MTS-JEPA code not available within 4 weeks of Phase 2 start, implement
multi-resolution masking from scratch using Phase 1 backbone as base.
**Phase 3 — Internal bank data (future)**
Replace DUKASCopy pipeline with internal tick feed adapter.
Fine-tune heads only; backbone frozen or lightly fine-tuned.
Out of scope for current PoC cycle.
### Consequences
- Issue #5 (Phase 0 MAE) is the unblocked next executable step
- Phase 1 autoresearch is blocked until Phase 0 passes its gate
- Var-JEPA (ELBO-based UQ) is a named future hypothesis for CVaR estimation in Phase 2+
but not on the critical path
**Alternative considered:** Start simple with EUR/USD only, expand later. Rejected because regime diversity in training is a structural requirement, not a nice-to-have. Retrofitting it in Phase 2 would require retraining from scratch.
---
## ADR-004 · Evaluation: Go harness + Python training separation
## Hard training cutoff: 2023-01-01
**Date:** 2026-05-28
**Status:** Accepted
**Decision:** All data from 2023-01-01 onward is sealed. No architecture, hyperparameter, or methodology decision may be informed by post-2023 data. Post-2023 test set opened only once, for final evaluation.
### Decision
**Rationale:** Out-of-sample integrity is essential for publishability and honest self-assessment. The held-out window (20232026) includes: 2023 US regional bank stress, 2024 JPY intervention episodes. These are the test of genuine generalisation.
Hard separation between training (Python) and evaluation (Go):
- **Python** (`model/`): all training, embedding export, model checkpointing
- **Go** (`src/eval/`): all evaluation metrics — silhouette, linear probe R², collapse
diagnostic, Kupiec/Christoffersen backtests
- Interface: Python exports embedding matrices + labels to `experiments/RUNID/` as
`.npy` files; Go eval harness reads them and writes `metrics.json`
### Rationale
Go evaluation gives deterministic, fast, auditable metric computation with proper
unit tests. It decouples the experimental loop from the training framework, making
it possible to re-evaluate any past experiment without re-running training.
The Go layer also serves as the foundation for the eventual trading desk dashboard.
### Consequences
- All acceptance criteria in Issues #4 and #5 are specified in terms of Go eval outputs
- `val_vol_r2` (the autoresearch optimization metric) is computed by the Go harness,
not inside the Python training loop
- Python training loop calls `task eval:probe` as a subprocess after each experiment
to get the scalar fed back to autoresearch
**Enforcement:** `data/raw/` is gitignored. The download script hard-stops at 2022-12-31 for training splits. Any deviation requires a DECISIONS.md entry explaining why.
---
## ADR-005 · Compute: Blackwell GPU on koala, PyTorch cu130
## Phase 0: SSL feasibility gate before JEPA work
**Date:** 2026-05-28
**Status:** Accepted
**Decision:** Before any JEPA-specific implementation, run a Phase 0 experiment: masked autoencoder (MAE) baseline on EUR/USD hourly data. If MAE silhouette < 0.20, SSL-based regime detection is likely not feasible on this data — stop and investigate before proceeding to JEPA.
### Decision
**Rationale:** JEPA's complexity is only justified if the core SSL premise (that latent representations capture regime structure) holds for FX data. A failed MAE experiment tells us this in 2 weeks rather than 4 months. Added after Full Grill session (2026-05-27).
All GPU training runs on koala (Arch Linux, Blackwell GPU, 12 GB VRAM).
PyTorch install: `pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu130`
(CUDA 13.0 wheel — required for sm_120 Blackwell support; stable as of May 2026).
Driver requirement: NVIDIA R570+, CUDA toolkit 12.8+.
**Go/no-go threshold:** MAE silhouette < 0.20 on held-out 2023 data → pause, investigate, do not proceed to Phase 1.
Ollama on iguana (Mac Studio M2 Ultra) serves the autoresearch agent LLM via the
existing LiteLLM proxy on piguard. Agent calls never hit koala directly.
---
### Consequences
## Architecture: TS-JEPA as starting implementation
- `model/requirements.txt` must NOT pin torch to a cu124 or earlier wheel
- CI (Issue #7) must NOT run GPU tests — CPU-only for unit tests, GPU only via
`task experiment:run` on koala
- 12 GB VRAM is sufficient for <5M parameter models at batch=64; monitor if
autoresearch explores larger architectures
**Decision:** Use TS-JEPA (Ennadir et al., 2025) as the starting JEPA implementation. MTS-JEPA (He et al., 2026) is the upgrade path if multi-resolution proves necessary.
**Rationale:** TS-JEPA is simpler. Validate the concept before adding multi-resolution complexity. If Phase 1 succeeds with TS-JEPA, MTS-JEPA is an ablation, not a prerequisite.
**Risk:** Both are preprints. Code reproducibility is unconfirmed. First task of Phase 0 is reproducing TS-JEPA on the paper's own benchmark — if this takes > 2 weeks, contact authors or fall back to implementing JEPA masking from scratch using V-JEPA codebase as reference.
---
## Input features: minimal for Phase 0/1
**Decision:** Phase 0 and Phase 1 use three features only: log-return (hourly), rolling 20-period realised volatility (hourly), VIX (daily, interpolated to hourly).
**Rationale:** Too many input features in early phases makes it impossible to distinguish "JEPA learned regime structure" from "JEPA learned to encode a feature that correlates with regime." Minimal features reduce confounding.
**Expansion path:** Add DXY, G10 vol surface, yield spreads in Phase 2 if Phase 1 succeeds.
---
## Collapse diagnostic
**Decision:** If PC1 of JEPA embeddings correlates > 0.85 with rolling 30-day HV, treat Phase 1 as a partial failure — the encoder learned volatility level, not regime structure. This is a useful finding but not the hypothesis.
**Rationale:** EUR/USD hourly returns are strongly heteroskedastic. A JEPA encoder trained to predict future embeddings will strongly tend to encode current volatility as its primary latent dimension. This is predictable, not regime-sensitive. The linear probe and PC1 diagnostic together distinguish "learned volatility" from "learned regime."
---
## Linear probe as mandatory Phase 1 exit gate
**Decision:** Train a linear model on frozen JEPA embeddings to predict realised volatility decile. R² < 0.4 → embeddings are not encoding useful risk structure → do not proceed to Phase 2.
**Rationale:** If a simple linear model cannot extract volatility regime from the embeddings, the representations are not useful for risk management purposes regardless of their silhouette score. Interpretability-by-linear-probe is the minimum bar for any downstream use.
---
## Explicit out-of-scope (Phases 13)
The following are explicitly out of scope for this research programme and go to a parking lot if they arise:
- Options / derivatives pricing
- Directional alpha generation
- Live or paper trading
- Exotic pairs beyond G10
- Institutional deployment
- Real-time inference systems
---
## Experiment spec required before any experiment
**Decision:** Every experiment phase must have a written spec in `specs/` before any code runs. No specless experiments.
**Rationale:** Consistent with spec-driven-dev way of working. Specs force falsifiable hypothesis statement, quantitative acceptance criteria, and explicit null-result protocol before results are known — preventing post-hoc rationalisation.
---
## Null results are results
**Decision:** Null results (hypothesis rejected) are recorded in `results/summaries/` and treated as valid research outputs, not failures to be iterated away silently.
**Rationale:** A confirmed null result (e.g. "SSL cannot find regime structure in FX hourly data") is publishable and scientifically valuable. Iterating hyperparameters until metrics pass without recording the failed attempts is p-hacking. Humble attitude; scientific approach.
+60 -7
View File
@@ -1,13 +1,66 @@
# hostexecutor
# jepa-fx-risk
> Generated from `mathias/template-go-web`.
Research project exploring JEPA (Joint Embedding Predictive Architecture) as a framework for latent representation learning applied to FX trading risk management.
## Bootstrap
**Primary hypothesis:** JEPA embeddings trained on FX time-series will produce latent market-state representations that are structurally separable by regime without explicit regime labels, measurable by silhouette score on k-means clusters validated against held-out realised-volatility regime labels.
After creating from template, run:
**Current phase:** Phase 0 — SSL Feasibility Gate (not yet started)
**Brain wing:** `jepa-fx` — query via `brain_query wing=jepa-fx`
---
## Architecture
This is a **Go-first, Python-minimal** research project.
| Layer | Language | Rationale |
|---|---|---|
| Data pipeline | Go | Type-safe, fast, no dependency hell |
| Experiment runner | Go | CLI tooling, reproducible invocations |
| Evaluation harness | Go | Silhouette, linear probe, collapse diagnostics |
| Model training | Python + PyTorch | TS-JEPA requires it; kept minimal and isolated |
## Structure
```
specs/ experiment specs (one per phase)
src/ Go packages: pipeline, eval, cmd
model/ Python: TS-JEPA training loop only
experiments/ one directory per run (gitignored except summaries)
results/ tracked: metric tables, key figures
data/ gitignored; see data/README.md for download instructions
notebooks/ EDA only; outputs stripped before commit
docs/ ADRs and research notes
```
## Quick start
```bash
go mod tidy # regenerate go.sum with real module path
task generate # generate templ files
task build # build the binary
task data:fetch # download DUKASCopy G10 tick data
task pipeline:build # build Go data pipeline
task check # lint + vet + test
task experiment:run -- --spec specs/phase-0-ssl-feasibility.md
```
## Phases
| Phase | Goal | Status |
|---|---|---|
| 0 | SSL feasibility gate — MAE baseline on FX data | 🔲 Not started |
| 1 | JEPA representation PoC — silhouette > 0.35 | 🔲 Not started |
| 2 | Regime detection validation — recall > 70%, lead > 5d | 🔲 Not started |
| 3 | Distributional forecasting — VaR passes Kupiec | 🔲 Not started |
| 4 | Multi-pair transfer — exotic pair improvement | 🔲 Not started |
## Key constraints
- **Training cutoff:** 2023-01-01 — all post-2023 data is sealed for final evaluation
- **Training data:** 20082022, all G10 pairs, DUKASCopy
- **Out of scope:** options pricing, directional alpha, live trading, exotic pairs (Phases 13)
- **End-state:** publishable research result — not institutional deployment
## Related
- Brain wing: `wiki/jepa-fx/` — decisions, hypotheses, failure modes
- Synthesis document: `JEPA_FX_Risk_Synthesis.docx` (session 2026-05-27)
+72 -30
View File
@@ -1,37 +1,79 @@
version: '3'
tasks:
generate:
desc: Run templ generate
cmds: [templ generate]
build:
desc: Build the binary
deps: [generate]
cmds: [go build -o bin/hostexecutor ./cmd/hostexecutor]
run:
deps: [build]
cmds: [./bin/hostexecutor]
test:
desc: Run all tests
deps: [generate]
cmds: [go test ./... -race]
lint:
cmds: [golangci-lint run ./...]
# ── Quality gate ──────────────────────────────────────────────────────────
check:
desc: Lint, vet, and test (used by CI)
deps: [generate]
desc: "Full quality gate: lint + vet + test (run before every commit)"
cmds:
- golangci-lint run ./...
- go vet ./...
- go test ./... -race -count=1
- golangci-lint run ./src/...
- go vet ./src/...
- go test ./src/... -race -count=1
- cd model && python -m pytest tests/ -q
test:
desc: Run Go tests only
cmds: [go test ./src/... -race]
lint:
desc: Lint Go code
cmds: [golangci-lint run ./src/...]
# ── Data ─────────────────────────────────────────────────────────────────
data:fetch:
desc: "Download G10 tick data from DUKASCopy (20032022)"
cmds: [go run ./src/cmd/fetch --config data/config.yaml]
data:process:
desc: "Resample ticks → hourly OHLCV + features (log-return, rolling HV)"
cmds: [go run ./src/cmd/process --input data/raw --output data/processed]
data:validate:
desc: "Validate processed data: gap detection, outlier report, regime coverage"
cmds: [go run ./src/cmd/validate --input data/processed]
# ── Experiment ───────────────────────────────────────────────────────────
experiment:run:
desc: "Run an experiment from a spec file. Usage: task experiment:run -- --spec specs/phase-0-ssl-feasibility.md"
cmds: [go run ./src/cmd/experiment {{.CLI_ARGS}}]
experiment:list:
desc: List all recorded experiment runs
cmds: [ls -lt experiments/ | head -20]
# ── Evaluation ───────────────────────────────────────────────────────────
eval:silhouette:
desc: "Compute silhouette score on embedding output. Usage: task eval:silhouette -- --run experiments/RUNID"
cmds: [go run ./src/cmd/eval silhouette {{.CLI_ARGS}}]
eval:probe:
desc: "Run linear probe on frozen embeddings vs. realised-vol decile"
cmds: [go run ./src/cmd/eval probe {{.CLI_ARGS}}]
eval:collapse:
desc: "Check collapse diagnostic: PC1 correlation with rolling HV"
cmds: [go run ./src/cmd/eval collapse {{.CLI_ARGS}}]
# ── Model (Python) ───────────────────────────────────────────────────────
model:train:
desc: "Train TS-JEPA model. Usage: task model:train -- --config model/configs/phase0.yaml"
dir: model
cmds: [python train.py {{.CLI_ARGS}}]
model:setup:
desc: "Create Python venv and install model dependencies (uv)"
dir: model
cmds:
- uv venv .venv
- uv pip install -r requirements.txt
# ── Context ──────────────────────────────────────────────────────────────
context:sync:
desc: Regenerate all harness-specific context files
cmds:
- bash scripts/context-sync.sh
context:sync:claude:
cmds: [bash scripts/context-sync.sh claude]
context:sync:agents:
cmds: [bash scripts/context-sync.sh agents]
context:sync:cursor:
cmds: [bash scripts/context-sync.sh cursor]
desc: "Regenerate CLAUDE.md and AGENTS.md from .context/PROJECT.md"
cmds: [bash scripts/context-sync.sh]
+36
View File
@@ -0,0 +1,36 @@
# Data
Raw and processed FX data is **gitignored** — never committed.
## Download: DUKASCopy G10 tick data
Use the provided fetch task:
```bash
task data:fetch # downloads raw tick data → data/raw/
task data:process # resamples to hourly OHLCV + features → data/processed/
task data:validate # gap detection, outlier report, regime coverage check
```
## Expected structure (local only)
```
data/
raw/ G10 pairs, tick OHLCV, 2003present (gitignored)
processed/ Hourly log-returns, rolling HV, VIX-merged (gitignored)
cache/ Intermediate artefacts (gitignored)
config.yaml Download configuration (committed)
```
## Pairs
EUR/USD, GBP/USD, USD/JPY, USD/CHF, AUD/USD, USD/CAD, NZD/USD, EUR/GBP, EUR/JPY, EUR/CHF
## Splits
| Split | Date range | Purpose |
|---|---|---|
| Train | 2008-01-01 2022-12-31 | Model training only |
| Held-out test | 2023-01-01 2023-12-31 | Final evaluation — do not open until evaluation |
**Training cutoff is hard: 2023-01-01. No post-2022 data informs any design decision.**
+32
View File
@@ -0,0 +1,32 @@
# Experiments
One directory per experiment run. Large outputs (embeddings, checkpoints, logs) are gitignored. Only metric summaries are committed to `results/summaries/`.
## Naming convention
```
YYYYMMDD-HHMMSS-phase-N-short-description/
config.yaml parameters used for this run (committed via results/summaries/)
metrics.json final metric snapshot (committed via results/summaries/)
embeddings/ (gitignored — large)
checkpoints/ (gitignored — large)
logs/ (gitignored — large)
```
## Git tag convention
Every run that produces reportable metrics gets a tag:
```
exp/YYYYMMDD-phase-N-description
```
Example: `exp/20260601-phase-0-mae-baseline`
## Starting a run
```bash
task experiment:run -- --spec specs/phase-0-ssl-feasibility.md --tag my-run-description
```
The runner creates the directory, writes config, runs the experiment, and outputs metrics.json.
+31
View File
@@ -0,0 +1,31 @@
# model
Python + PyTorch perimeter. This is the only directory in the project that uses Python.
## Contents
```
model/
train.py TS-JEPA training entry point
configs/ YAML configs per phase
phase0-mae.yaml
phase1-tsjepa.yaml
tsjepa/ TS-JEPA model implementation (adapted from paper)
tests/ pytest tests for model components
requirements.txt pinned Python dependencies
.venv/ (gitignored — created by `task model:setup`)
```
## Setup
```bash
task model:setup # creates .venv and installs requirements via uv
```
## Dependency policy
Every Python dependency must be justified in a comment in `requirements.txt`. Prefer Go implementations for anything outside the training loop. When adding a new dependency, add an entry to DECISIONS.md explaining why a Go alternative wasn't sufficient.
## Python version
3.12 (pinned in `.python-version`)
+11
View File
@@ -0,0 +1,11 @@
# Notebooks
Exploratory Data Analysis (EDA) scratch space only.
## Rules
- **Never use notebooks for anything reproducible.** Training, evaluation, and metric computation belong in `src/` with tests.
- **Strip all outputs before committing.** Use `nbstripout` or equivalent.
- **Label every notebook with a phase prefix:** `phase0-eda-eurusd-distribution.ipynb`
Notebooks are thinking tools, not research artifacts. If a finding from a notebook is worth keeping, it goes into a spec, a decision in `DECISIONS.md`, or a result in `results/summaries/` — not the notebook itself.
+32
View File
@@ -0,0 +1,32 @@
# Results
Tracked outputs from experiment runs. Large raw outputs are gitignored — only summaries are committed.
## Structure
```
results/
summaries/ metric tables, key figures, per-phase result records (committed)
raw/ full embedding outputs, backtest CSVs (gitignored)
```
## Per-phase result record format
Each concluded phase produces a result record in `summaries/`:
```
summaries/
phase-0-[pass|null].md
phase-1-[pass|null].md
...
```
Each record must include:
- Phase name and hypothesis
- Key metrics (silhouette score, R², etc.) with confidence intervals where applicable
- Baseline comparison
- Verdict: PASS / NULL RESULT
- If null: what was investigated, what was found, next step taken
- Link to experiment git tag
**Null results are valid research outputs and must be committed, not discarded.**
+84
View File
@@ -0,0 +1,84 @@
# Experiment Spec: Phase 0 — SSL Feasibility Gate
## Hypothesis
> We believe that a masked autoencoder (MAE) trained on FX hourly time-series will
> produce latent embeddings that show structural separability by volatility regime
> without explicit regime labels, measurable by silhouette score > 0.20 on k-means
> clusters evaluated against a held-out realised-volatility regime label on 2023 data.
This hypothesis is FALSE if silhouette score ≤ 0.20 on the held-out evaluation.
## Background
Before investing in JEPA-specific machinery, we need to confirm that self-supervised
representation learning can find regime structure in FX time-series at all. If the
simplest SSL method (MAE) cannot find structure, JEPA will not either — and the root
cause needs to be understood before proceeding.
Also validates that TS-JEPA code is reproducible: first task is running TS-JEPA on
the paper's own benchmark, not on FX data. If reproduction takes > 2 weeks, contact
authors or fall back to implementing JEPA masking from V-JEPA codebase.
Added post Full Grill (2026-05-27). See DECISIONS.md: "Phase 0: SSL feasibility gate."
## Design
### Data
- **Source:** DUKASCopy, EUR/USD hourly OHLCV
- **Train:** 2008-01-01 2022-12-31
- **Held-out test:** 2023-01-01 2023-12-31 (sealed until final evaluation)
- **Features:** log-return (hourly), rolling 20-period realised HV, VIX (daily → hourly interpolation)
- **Regime label (evaluation only):** rolling 30-day HV percentile; binary: top 50% = high-vol, bottom 50% = low-vol
### Model
- **Architecture:** 1D temporal Masked Autoencoder
- Encoder: 3-layer 1D CNN + positional encoding
- Decoder: 2-layer MLP reconstructing masked segment
- **Masking:** contiguous temporal block (target); context window = 120h (5 days)
- **Loss:** MSE reconstruction on masked segment
### TS-JEPA reproduction task (runs in parallel / before MAE)
- Reproduce TS-JEPA paper results on the authors' benchmark dataset
- Go/no-go: if reproduction fails within 2 weeks → contact authors or pivot to V-JEPA adaptation
### Baseline
- **PCA** on raw feature vectors (same 120h context window, flattened)
- Tests whether any dimensionality reduction finds regime structure; confirms SSL is adding something
### Ablations
- Masking horizon K ∈ {8h, 24h, 72h} — does context length affect embedding quality?
## Acceptance Criteria
- [ ] MAE silhouette score > 0.20 on held-out 2023 data (k-means k=3, vs. binary HV regime label)
- [ ] MAE silhouette exceeds PCA baseline silhouette
- [ ] Rerun ×3 within ±10% of reported silhouette (reproducibility)
- [ ] PC1 / rolling-HV correlation < 0.95 (encoder is not purely encoding volatility level)
- [ ] TS-JEPA reproduced on paper's benchmark within 2 weeks of start
## Out of Scope
- JEPA implementation (Phase 1)
- Multi-pair training (Phase 1+)
- VaR, ES, or any risk metric computation
- Any data after 2022-12-31 (training); 2023 test set opened only for final evaluation
- Hyperparameter search beyond the three masking horizons defined above
## Null Result Protocol
If MAE silhouette ≤ 0.20 on held-out data:
1. Conclude: SSL-based regime detection is not straightforwardly feasible on EUR/USD hourly data with these three features
2. Investigate in order: (a) try daily resolution instead of hourly, (b) expand feature set to 6 features (add DXY, yield spread), (c) try 4-class regime label (HV quartiles) instead of binary
3. If all three investigations fail: conclude SSL regime detection is not viable on FX data; document and consider pivoting the primary hypothesis to distributional forecasting directly
4. Record result in `results/summaries/phase-0-null.md`
5. Write failure mode to brain: `brain_write wing=jepa-fx hall=failures`
## Risks
| Risk | Canary | Mitigation |
|---|---|---|
| TS-JEPA code unreproducible | Benchmark result doesn't match paper within 2 weeks | Contact authors; fall back to V-JEPA adaptation |
| MAE encoder collapses | Reconstruction loss plateau in first 10 epochs; all embeddings near-identical | Add batch normalisation; reduce LR; check masking ratio |
| Regime label too coarse | Silhouette low even with visually structured embeddings | Also evaluate with 4-class HV quartile label |
| PC1 is just volatility | PC1/HV correlation > 0.95 | Useful finding; record it; do not declare success |
+82
View File
@@ -0,0 +1,82 @@
# Experiment Spec: Phase 1 — JEPA Representation PoC
## Hypothesis
> We believe that a TS-JEPA encoder trained on G10 FX hourly data (20082022) will
> produce latent market-state embeddings that are structurally separable by volatility
> regime without explicit regime labels, measurable by silhouette score > 0.35 on
> k-means clusters evaluated against realised-volatility regime labels on held-out
> 2023 data including at least one structural break.
This hypothesis is FALSE if silhouette score ≤ 0.35 OR linear probe R² ≤ 0.40 on
held-out evaluation.
**Prerequisites:** Phase 0 passed (MAE silhouette > 0.20, TS-JEPA reproduced).
## Background
Phase 0 confirmed that SSL-based representation learning can find regime structure in
FX time-series. Phase 1 tests whether JEPA's specific inductive bias (predict target
embeddings from context embeddings, never reconstruct raw data) produces richer
representations than a simple MAE — and whether those representations are useful for
risk management tasks (measurable via linear probe).
## Design
### Data
- **Source:** DUKASCopy, all G10 pairs (EUR/USD, GBP/USD, USD/JPY, USD/CHF, AUD/USD, USD/CAD, NZD/USD, EUR/GBP, EUR/JPY, EUR/CHF), hourly
- **Train:** 2008-01-01 2022-12-31 (all 10 pairs, jointly)
- **Held-out test:** 2023-01-01 2023-12-31 (sealed until final evaluation)
- **Features:** log-return, rolling 20-period HV, VIX (daily → hourly)
- **Regime label (evaluation only):** rolling 30-day HV percentile; binary + 4-class (quartiles)
### Model
- **Architecture:** TS-JEPA (Ennadir et al., 2025)
- Context encoder: maps observed window → latent embedding
- Target encoder: EMA of context encoder (momentum β ≈ 0.996); stop-gradient
- Predictor: shallow MLP bridging context → target embedding
- **Masking horizons:** ablate K ∈ {1h, 8h, 24h}; context window = 120h
- **Training:** multi-pair joint training (one model, all G10 pairs)
### Baseline
- Phase 0 MAE (best masking horizon from Phase 0)
- PCA on raw features (Phase 0 baseline)
### Ablations
1. TS-JEPA vs MAE — is JEPA's no-reconstruction inductive bias adding value?
2. Single-pair (EUR/USD only) vs. multi-pair — does joint training improve representations?
3. Masking horizon K: {1h, 8h, 24h}
## Acceptance Criteria
- [ ] Silhouette score > 0.35 on held-out 2023 data (k-means k=35, binary HV label)
- [ ] Linear probe R² > 0.40 on frozen embeddings vs. realised-vol decile
- [ ] PC1 / rolling-HV correlation < 0.85 (encoder learning more than volatility level)
- [ ] TS-JEPA silhouette exceeds Phase 0 MAE silhouette by > 5%
- [ ] Rerun ×3 within ±10% of reported silhouette
## Out of Scope
- VaR, ES, distributional forecasting (Phase 3)
- Exotic pairs beyond G10
- Options pricing, alpha generation, live trading
- Any data after 2022-12-31 for training; test set opened only for final evaluation
- Regime detection backtesting (Phase 2 — embedding drift as early warning)
## Null Result Protocol
If primary criteria not met:
1. If silhouette > 0.20 but ≤ 0.35: JEPA shows partial structure; not sufficient for Phase 2. Investigate whether multi-pair training, longer context window, or additional features close the gap. One retry permitted with documented rationale.
2. If silhouette ≤ 0.20: regression from Phase 0; investigate JEPA training stability (collapse risk). Do not proceed.
3. If linear probe R² ≤ 0.40 despite good silhouette: embeddings are structured but not encoding risk-relevant information. Record as a finding; reconsider feature set.
4. Record all results in `results/summaries/phase-1-[pass|null].md`
5. Write findings to brain: `brain_write wing=jepa-fx hall=failures`
## Risks
| Risk | Canary | Mitigation |
|---|---|---|
| EMA encoder collapses | All embeddings converge to near-zero; loss goes to ~0 early | Verify EMA momentum schedule; check stop-gradient implementation |
| Encoder learns only EUR/USD vol | PC1 dominated by EUR/USD HV even in multi-pair model | Evaluate per-pair silhouette; if EUR/USD dominates, weight loss by pair |
| Phase 0 silhouette was data-split artefact | Phase 1 MAE baseline doesn't reproduce Phase 0 numbers | Fix random seeds; document split methodology in Phase 0 |
| Insufficient regime diversity (20082022) | Embedding clusters don't separate 2023 structural break | Verify 2023 test includes high-vol episode; add 4-class label as backup |