feat: DVC + MinIO experiment tracking pipeline #10

New Issue

2026-06-05T18:48:13Z

mathias commented

2026-06-05 18:48:13 +00:00

Context

The FX/JEPA research project (jepa-fx-risk) currently has no experiment reproducibility layer. Model checkpoints, dataset versions, and metric histories are not tracked in a way that allows reproducing a specific run. This spec adds DVC backed by a self-hosted MinIO instance on koala.

Goals

Every experiment run is fully reproducible from a Git commit + DVC lock file
Dataset versions (yfinance pulls, synthetic FX paths) are content-addressed and stored outside Git
Model checkpoints and eval metrics are versioned alongside code
dvc repro triggered automatically on exp-* tags via Gitea Actions

Non-goals

Cloud storage (all artifacts stay on koala/LAN)
MLflow UI (metrics in DVC params/metrics files is sufficient for now)

Design

MinIO on koala

Deploy MinIO as a k3s workload in the ai-stack namespace (or standalone Docker — whichever matches current infra conventions). Single-node, single bucket dvc-artifacts. Expose on LAN only (no Tailscale egress needed for CI runner on koala).

bucket: dvc-artifacts
access: LAN only (10.0.1.20)
credentials: stored in Gitea Actions secrets

DVC remote config

dvc remote add -d koala-minio s3://dvc-artifacts
dvc remote modify koala-minio endpointurl http://10.0.1.20:9000
dvc remote modify koala-minio access_key_id $MINIO_ACCESS_KEY
dvc remote modify koala-minio secret_access_key $MINIO_SECRET_KEY

Remote config committed to repo (credentials via env only).

`dvc.yaml` pipeline stages

stages:
  prepare:
    cmd: python src/prepare.py
    deps: [src/prepare.py, data/raw/]
    outs: [data/processed/]
  train:
    cmd: python src/train.py
    deps: [src/train.py, data/processed/, configs/train.yaml]
    outs: [models/checkpoint.pt]
    metrics: [metrics/val_vol_r2.json]
  eval:
    cmd: python src/eval.py
    deps: [src/eval.py, models/checkpoint.pt]
    metrics: [metrics/eval.json]

Gitea Actions workflow

# .gitea/workflows/dvc-repro.yaml
on:
  push:
    tags: ['exp-*']

jobs:
  repro:
    runs-on: koala   # self-hosted runner with GPU
    steps:
      - uses: actions/checkout@v4
      - run: pip install dvc[s3]
      - run: dvc pull
      - run: dvc repro
      - run: dvc push
      - run: cat metrics/val_vol_r2.json

Acceptance criteria

MinIO running on koala, accessible at http://10.0.1.20:9000
dvc.yaml covers prepare → train → eval pipeline
dvc repro runs clean from a fresh checkout (after dvc pull)
Pushing exp-* tag triggers the Actions workflow and pushes artifacts to MinIO
metrics/val_vol_r2.json written and readable post-run

Open questions

MinIO as k3s workload vs standalone Docker — check infra conventions in mathias/infra
GPU memory constraints on koala (RTX 5070 12GB) — confirm JEPA batch size fits before wiring CI

## Context The FX/JEPA research project (`jepa-fx-risk`) currently has no experiment reproducibility layer. Model checkpoints, dataset versions, and metric histories are not tracked in a way that allows reproducing a specific run. This spec adds DVC backed by a self-hosted MinIO instance on koala. ## Goals - Every experiment run is fully reproducible from a Git commit + DVC lock file - Dataset versions (yfinance pulls, synthetic FX paths) are content-addressed and stored outside Git - Model checkpoints and eval metrics are versioned alongside code - `dvc repro` triggered automatically on `exp-*` tags via Gitea Actions ## Non-goals - Cloud storage (all artifacts stay on koala/LAN) - MLflow UI (metrics in DVC params/metrics files is sufficient for now) ## Design ### MinIO on koala Deploy MinIO as a k3s workload in the `ai-stack` namespace (or standalone Docker — whichever matches current infra conventions). Single-node, single bucket `dvc-artifacts`. Expose on LAN only (no Tailscale egress needed for CI runner on koala). ``` bucket: dvc-artifacts access: LAN only (10.0.1.20) credentials: stored in Gitea Actions secrets ``` ### DVC remote config ``` dvc remote add -d koala-minio s3://dvc-artifacts dvc remote modify koala-minio endpointurl http://10.0.1.20:9000 dvc remote modify koala-minio access_key_id $MINIO_ACCESS_KEY dvc remote modify koala-minio secret_access_key $MINIO_SECRET_KEY ``` Remote config committed to repo (credentials via env only). ### `dvc.yaml` pipeline stages ```yaml stages: prepare: cmd: python src/prepare.py deps: [src/prepare.py, data/raw/] outs: [data/processed/] train: cmd: python src/train.py deps: [src/train.py, data/processed/, configs/train.yaml] outs: [models/checkpoint.pt] metrics: [metrics/val_vol_r2.json] eval: cmd: python src/eval.py deps: [src/eval.py, models/checkpoint.pt] metrics: [metrics/eval.json] ``` ### Gitea Actions workflow ```yaml # .gitea/workflows/dvc-repro.yaml on: push: tags: ['exp-*'] jobs: repro: runs-on: koala # self-hosted runner with GPU steps: - uses: actions/checkout@v4 - run: pip install dvc[s3] - run: dvc pull - run: dvc repro - run: dvc push - run: cat metrics/val_vol_r2.json ``` ## Acceptance criteria - [ ] MinIO running on koala, accessible at `http://10.0.1.20:9000` - [ ] `dvc.yaml` covers prepare → train → eval pipeline - [ ] `dvc repro` runs clean from a fresh checkout (after `dvc pull`) - [ ] Pushing `exp-*` tag triggers the Actions workflow and pushes artifacts to MinIO - [ ] `metrics/val_vol_r2.json` written and readable post-run ## Open questions - MinIO as k3s workload vs standalone Docker — check infra conventions in `mathias/infra` - GPU memory constraints on koala (RTX 5070 12GB) — confirm JEPA batch size fits before wiring CI

mathias referenced this issue

2026-06-23 14:26:31 +00:00

feat: launch + monitor an autoresearch run from an Autoresearch Council backlog leaf #11

mathias referenced this issue from a commit

2026-06-23 14:50:03 +00:00

feat(data): EUR/USD M1 fetch + daily realized-vol prep (toy slice, #2/#11)

mathias referenced this issue

2026-06-27 15:20:33 +00:00

Phase B: autoresearch sandbox as k8s Job (GPU isolation + PVC artifacts) #15

mathias referenced this issue from a commit

2026-06-29 19:20:41 +00:00

chore(ci): commit 59KB toy eurusd_daily.parquet as CI data fixture