feat: add experiment-spec skill

2026-05-27 21:58:50 +00:00
2 changed files with 192 additions and 168 deletions
@@ -1,168 +0,0 @@
 ---
 name: dream
 description: >
  Run a "dream" — a reflective memory consolidation pass over an agent's memory
  directory. Use this skill whenever the user says "dream", "run a dream", "consolidate
  my memory files", "clean up my MEMORY.md", or asks Claude to do a memory maintenance
  pass, prune stale notes, or reorganize topic files. Also trigger when the user wants
  to rebuild a memory index, merge duplicate facts, or convert relative dates in notes
  to absolute ones. This is an agentic, multi-phase workflow — always use this skill
  rather than improvising the steps.
 ---
 # Dream — Memory Consolidation Skill
 You are performing a **dream**: a reflective, agentic pass over a memory directory.
 Your goal is to synthesize recent signal into durable, well-organized memory so that
 future sessions can orient quickly.
 ---
 ## Pre-flight
 Before starting, confirm:
 1. **Where is the memory directory?** Ask the user if not obvious from context.  
   Common locations: `~/memory/`, `~/.agent/memory/`, `./memory/`, a path in an env var like `$MEMORY_DIR`.
 2. **Are there transcripts or daily logs to scan?** Ask if not obvious.
 3. **Any topics to skip or treat as sensitive?**
 Once confirmed, proceed through the four phases in order. Narrate each phase briefly as you go.
 ---
 ## Phase 1 — Orient
 **Goal**: Get a map of what exists before touching anything.
 ```bash
 ls -la <memory_dir>/
 cat <memory_dir>/MEMORY.md
 ```
 For each file listed (excluding MEMORY.md):
 - Read or skim it (first 40–60 lines is usually enough unless it's small).
 - Note: topic, approximate recency, any obvious staleness or duplication.
 Build a mental inventory:
 - Files present, rough line counts
 - Topics covered
 - Any files that look abandoned, mislabeled, or overlapping
 ---
 ## Phase 2 — Gather Recent Signal
 **Goal**: Find new facts, corrections, and drift since the last dream.
 Check in this order:
 1. **Daily logs** — read recent entries (last 7–14 days).  
   Look for: new decisions, changed preferences, completed projects, new relationships/tools.
 2. **Drifted facts** — scan existing topic files for statements that may now be false.  
   Examples: "currently evaluating X" (did they pick one?), "planning to do Y" (done or dropped?), relative dates like "last week" or "recently".
 3. **Transcripts** — only grep narrowly if there's a specific gap.  
   Avoid bulk-reading transcripts; it's slow and noisy. Use targeted patterns:
   ```bash
   grep -r "decided\|switched to\|no longer\|now using\|moved to" <transcripts_dir>/ | tail -40
   ```
 Collect a list of **updates to make**: new facts, corrections, removals.
 ---
 ## Phase 3 — Consolidate
 **Goal**: Apply the updates. Leave memory files cleaner and more accurate than you found them.
 For each topic file:
 - **Merge duplicates**: if the same fact appears in two files, keep it in the more specific one and remove from the general one.
 - **Convert relative dates**: replace "last week", "recently", "a few months ago" with an absolute date (use the current date as reference; estimate if necessary and note the uncertainty).
 - **Delete contradicted facts**: if a new fact supersedes an old one, remove the old one outright — don't leave both.
 - **Tighten language**: convert vague hedges ("probably uses", "might be") to definite statements where the evidence supports it, or remove if genuinely unknown.
 - **Add new facts** from Phase 2 to the appropriate topic file. Create a new topic file if no good home exists.
 After editing files, do a final pass:
 ```bash
 grep -n "last week\|recently\|a few months\|soon\|currently planning" <memory_dir>/*.md
 ```
 Clean up any remaining relative time references.
 ---
 ## Phase 4 — Prune and Index
 **Goal**: Rebuild MEMORY.md as a clean, navigable index under 200 lines.
 **MEMORY.md structure**:
 ```markdown
 # Memory Index
 _Last updated: YYYY-MM-DD_
 ## Overview
 One short paragraph: who this agent is, primary context, most important standing facts.
 ## Topic Files
 | File | Contents | Last updated |
 |------|----------|-------------|
 | person.md | Identity, preferences, background | YYYY-MM-DD |
 | projects.md | Active and recent projects | YYYY-MM-DD |
 | tools.md | Stack, infra, dev environment | YYYY-MM-DD |
 | ... | ... | ... |
 ## Quick Facts
 - Bullet list of the 10–15 most frequently-needed facts (role, location, key tools, etc.)
 ## Recent Changes
 - Bullet list of what changed in this dream (so the next session knows what's fresh)
 ```
 Rules:
 - Remove any pointers to files that no longer exist.
 - Add pointers for any new files created in Phase 3.
 - Keep Quick Facts ≤ 15 bullets — this is for speed, not completeness.
 - Recent Changes section replaces itself each dream (don't accumulate).
 - Total MEMORY.md length: **200 lines max**.
 ---
 ## Output
 After completing all four phases, return a **dream summary** to the user:
 ```
 ## Dream complete — YYYY-MM-DD
 ### What changed
 - [file]: [what was updated]
 - MEMORY.md: rebuilt index, N topic files indexed
 ### Facts added
 - ...
 ### Facts removed / corrected
 - ...
 ### Files created
 - ...
 ### Files deleted or merged
 - ...
 ### Still uncertain / needs follow-up
 - ...
 ```
 Keep it concise — a few bullets per section, not exhaustive diffs. The goal is for the user to quickly confirm the dream went well and catch any mistakes.
 ---
 ## Tips and Edge Cases
 - **No MEMORY.md exists yet**: create one from scratch using the structure above. Treat all existing files as "first time indexed".
 - **Memory dir is empty**: create MEMORY.md and a starter `scratch.md` noting the date and that the memory system is new.
 - **Conflicting facts with no clear resolution**: note both in the file with a date stamp and flag in "Still uncertain" section of the summary.
 - **Large transcript dumps**: resist reading them in full. Grep is your friend. If you must read, read the last N lines only.
 - **Files with sensitive content**: if the user flagged topics to skip, skip them entirely — don't even open them.
@@ -0,0 +1,192 @@
 ---
 name: experiment-spec
 description: Write a rigorous experiment spec for a research phase before any code or training runs. Use instead of feature-spec for scientific/ML research projects. Enforces falsifiable hypothesis, quantitative acceptance criteria, baseline comparison, and null-result protocol.
 ---
 # Experiment Spec
 ## Overview
 An experiment spec is the scientific contract for one research phase or experiment, written before any implementation or training begins. It is the research analogue of `feature-spec` — same discipline, different vocabulary.
 **Core principle:** If you cannot write a falsifiable hypothesis with a quantitative acceptance criterion, you do not understand the experiment well enough to run it.
 ## When to Use
 - Starting a new research phase (Phase 0, Phase 1, etc.)
 - Running any experiment that will produce metrics used to make a go/no-go decision
 - Any time the question is "does X work?" rather than "build X"
 - Before touching training code, data, or hyperparameters for a new question
 **When NOT to use:**
 - Implementing a specific component whose behaviour is already defined by a phase spec (use `feature-spec` instead)
 - Exploratory data analysis with no hypothesis (use a notebook; note it as EDA)
 - Bug fixes or refactors
 ## Iron Laws
 1. **The hypothesis must be falsifiable.** "JEPA is promising" is not a hypothesis. "JEPA embeddings will achieve silhouette > 0.35 on held-out data" is. If you cannot state conditions under which the hypothesis is false, it is not a hypothesis.
 2. **Acceptance criteria must be quantitative and pre-registered.** Write the number before you run the experiment. Moving the goalposts after seeing results is p-hacking.
 3. **A baseline is mandatory.** Every experiment must compare against at least one simpler baseline. "Better than nothing" is not a baseline.
 4. **A null-result protocol is mandatory.** State what you will conclude and do if the hypothesis is rejected. "Try harder" is not a protocol.
 5. **The training cutoff is sacred.** No post-cutoff data informs any decision in the spec or implementation.
 ## Spec Template
 ```markdown
 # Experiment Spec: [Phase N — Short Name]
 ## Hypothesis
 > "[Falsifiable claim]: We believe [X] will produce [Y], measurable by [Z]."
 State conditions under which this hypothesis is FALSE.
 ## Background
 Why this experiment? What does it build on? What prior result or decision motivates it?
 (2–4 sentences. Reference DECISIONS.md or brain wing entries where relevant.)
 ## Design
 ### Data
 - Source, date range, pairs/assets, features used
 - Train / validation / test split (respect training cutoff)
 ### Model / method
 - Architecture, configuration, key hyperparameters
 - What is being varied vs. held fixed
 ### Baseline
 - What simpler method is being compared against?
 - Why is this the right baseline?
 ### Ablations (if any)
 - What variants will be run to isolate the effect being studied?
 ## Acceptance Criteria
 - [ ] [Primary criterion — quantitative threshold on primary metric]
 - [ ] [Baseline comparison — e.g. "exceeds baseline by >X%"]
 - [ ] [Reproducibility — reruns within ±Y% of reported metric]
 - [ ] [Collapse/sanity check — e.g. "PC1/rolling-HV correlation < 0.85"]
 ## Out of Scope
 What this experiment explicitly does NOT answer, even if related.
 Anything plausibly in scope that is deferred goes here.
 ## Null Result Protocol
 If the primary acceptance criterion is NOT met:
 - What do we conclude?
 - What is the next step? (Investigate X, pivot to Y, terminate programme)
 - What gets written to the brain and results/summaries/?
 ## Risks
 What could go wrong, and how would it be detected?
 At least one risk must be listed.
 ```
 ## Worked Example
 ```markdown
 # Experiment Spec: Phase 0 — SSL Feasibility Gate
 ## Hypothesis
 > We believe that a masked autoencoder (MAE) trained on FX hourly data will produce
 > latent embeddings that show structural separability by volatility regime without
 > explicit regime labels, measurable by silhouette score > 0.20 on held-out 2023 data.
 This hypothesis is FALSE if silhouette score ≤ 0.20 on the held-out evaluation.
 ## Background
 Before investing in JEPA-specific machinery, we need to confirm that SSL-based
 representation learning can find regime structure in FX time-series at all. MAE is
 the simplest SSL baseline — if it cannot find structure, JEPA will not either.
 Added post Full Grill (2026-05-27). See DECISIONS.md: "Phase 0: SSL feasibility gate".
 ## Design
 ### Data
 - Source: DUKASCopy, EUR/USD hourly, 2008–2022 (train), 2023 (held-out test)
 - Features: log-return, rolling 20-period HV, VIX (daily interpolated to hourly)
 - Regime label (for evaluation only, not training): rolling 30-day HV percentile,
  binary high/low threshold at 50th percentile
 ### Model
 - Masked Autoencoder: 1D temporal masking (mask contiguous 24h window)
 - Encoder: 3-layer 1D CNN + positional encoding
 - Decoder: 2-layer MLP reconstructing masked segment
 - Context window: 120 hours (5 days)
 ### Baseline
 - PCA on raw feature vectors (same window) — tests whether any dimensionality
  reduction shows regime structure, not just SSL
 ### Ablations
 - Masking horizon: K ∈ {8h, 24h, 72h} — does horizon affect embedding quality?
 ## Acceptance Criteria
 - [ ] Silhouette score > 0.20 on held-out 2023 data (k-means, k=3, vs. HV regime label)
 - [ ] MAE silhouette exceeds PCA baseline silhouette
 - [ ] Rerun within ±10% of reported silhouette
 - [ ] PC1 / rolling-HV correlation < 0.95 (not purely encoding volatility level)
 ## Out of Scope
 - JEPA implementation (Phase 1)
 - Multi-pair training (Phase 1+)
 - VaR or ES computation
 - Any use of post-2023 data
 ## Null Result Protocol
 If silhouette ≤ 0.20:
 - Conclude: SSL cannot reliably find regime structure in EUR/USD hourly data with
  these features at this resolution
 - Next step: investigate whether (a) hourly resolution is too noisy (try daily),
  (b) 3 features are insufficient, or (c) regime label definition is too coarse
 - Record result in results/summaries/phase-0-null.md and brain wing jepa-fx/failures/
 ## Risks
 - Encoder collapses to near-constant output: detect via reconstruction loss plateau
  in first 10 epochs; mitigation: add batch norm, reduce learning rate
 - Regime label too coarse (binary HV): silhouette may be low even with good structure;
  mitigation: also evaluate with 4-class label (HV quartiles)
 ```
 ## Common Failure Modes
 | Failure mode | What it looks like | Fix |
 |---|---|---|
 | Non-falsifiable hypothesis | "JEPA shows promise" | Rewrite with a number |
 | Post-hoc criteria | Threshold chosen after seeing results | Write the number first, commit the spec |
 | No baseline | Silhouette of 0.30 sounds good until PCA achieves 0.35 | Always include a dumber method |
 | Missing null protocol | "We'll figure it out if it fails" | Write it now — it forces clarity about what you're actually betting on |
 | Cutoff violation | Architecture choice informed by 2024 data patterns | Never open the test set during development |
 ## Brain MCP Integration
 **At spec start:**
 - `brain_query wing=jepa-fx hall=decisions` — load current architectural decisions
 - `brain_query wing=jepa-fx hall=failures` — load known failure modes; address them in Risks section
 **After spec is approved:**
 - `brain_write` to `jepa-fx/hypotheses/` with the hypothesis and acceptance criteria
 **After experiment concludes:**
 - `brain_write` to `jepa-fx/failures/` with any new failure modes discovered
 - `session_log` with outcome
 ## Cross-References
 - Use `feature-spec` for implementing a specific component within an already-specced phase
 - Use `grill-me` on the spec before running the experiment if the hypothesis feels shaky
 - Use `tdd` once the spec is approved — each acceptance criterion maps to a test
 - Use `session-retrospective` after the experiment concludes