feat: implement Go evaluation harness (silhouette, linear probe, collapse diagnostic) #4

Closed
opened 2026-05-27 22:01:33 +00:00 by mathias · 0 comments
Owner

What

Implement the Go evaluation harness: the set of metrics and diagnostics used to assess embedding quality after any training run.

Components

Silhouette scorer

  • Input: embedding matrix (N × D), regime labels (binary/4-class HV)
  • Output: silhouette score per k ∈ {3, 4, 5}; best k selected by score
  • Test: synthetic embeddings with known cluster structure → score > 0.6

Linear probe

  • Input: frozen embedding matrix, realised-vol decile labels
  • Output: R² of linear regression on held-out fold
  • Test: embeddings generated from known linear function of vol → R² > 0.95

Collapse diagnostic

  • Input: embedding matrix
  • Output: PC1 correlation with rolling 30-day HV; fraction of variance in PC1
  • Test: embeddings = HV + noise(0.1) → correlation > 0.95 detected correctly

Visualisation export

  • Input: embedding matrix, regime labels
  • Output: t-SNE / UMAP 2D projection as CSV (plotted externally or in notebook)

Acceptance criteria

  • All three evaluators implemented as Go packages in src/eval/
  • All unit tests pass (task test)
  • task eval:silhouette, task eval:probe, task eval:collapse all work end-to-end on synthetic data
  • Evaluation outputs written to experiments/RUNID/metrics.json in standard schema

Notes

Silhouette computation requires pairwise distances — use an efficient Go implementation; do not shell out to Python for this.

## What Implement the Go evaluation harness: the set of metrics and diagnostics used to assess embedding quality after any training run. ## Components ### Silhouette scorer - Input: embedding matrix (N × D), regime labels (binary/4-class HV) - Output: silhouette score per k ∈ {3, 4, 5}; best k selected by score - Test: synthetic embeddings with known cluster structure → score > 0.6 ### Linear probe - Input: frozen embedding matrix, realised-vol decile labels - Output: R² of linear regression on held-out fold - Test: embeddings generated from known linear function of vol → R² > 0.95 ### Collapse diagnostic - Input: embedding matrix - Output: PC1 correlation with rolling 30-day HV; fraction of variance in PC1 - Test: embeddings = HV + noise(0.1) → correlation > 0.95 detected correctly ### Visualisation export - Input: embedding matrix, regime labels - Output: t-SNE / UMAP 2D projection as CSV (plotted externally or in notebook) ## Acceptance criteria - [ ] All three evaluators implemented as Go packages in `src/eval/` - [ ] All unit tests pass (`task test`) - [ ] `task eval:silhouette`, `task eval:probe`, `task eval:collapse` all work end-to-end on synthetic data - [ ] Evaluation outputs written to `experiments/RUNID/metrics.json` in standard schema ## Notes Silhouette computation requires pairwise distances — use an efficient Go implementation; do not shell out to Python for this.
Sign in to join this conversation.
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/jepa-fx-risk#4