Phase B: autoresearch sandbox as k8s Job (GPU isolation + PVC artifacts) #15

Open
opened 2026-06-27 15:20:33 +00:00 by mathias · 0 comments
Owner

Context

Phase A uses a Gitea Actions self-hosted runner on koala (workflow_dispatch → scaffold → loop). That works but the runner shares GPU + filesystem with everything else on the node. Phase B promotes autoresearch to a proper k8s sandbox.

What this replaces

.gitea/workflows/autoresearch.yml triggers loop.py directly on the runner. Phase B replaces the run job with a k8s Job submission — the runner becomes an orchestrator, not an executor.

Proposed design

Trigger: same Gitea workflow_dispatch inputs (fixture, rq_id, iters, model)

Execution:

  • Runner renders a k8s Job manifest from the inputs and kubectl applys it
  • Job spec:
    • runtimeClassName: nvidia (koala GPU isolation — see brain: koala-gpu-setup)
    • resources.limits: nvidia.com/gpu: 1, appropriate CPU/memory
    • env: LITELLM_KEY (from k8s Secret, ESO-synced from 1Password)
    • env: LITELLM_BASE, NTFY_URL
    • activeDeadlineSeconds: time-box (90 min default)
  • Job mounts a PVC for the run dir (or writes to MinIO/DVC — see #10)
  • Job runs: python scripts/autoresearch_start.py ... && python loop.py ...
  • Job completion: STATUS.md + metrics.json available from PVC / uploaded to MinIO

GitOps path (alternative to runner-apply):

  • Runner patches k3s/apps/autoresearch-jobs/<rq_id>.yaml in mathias/infra
  • Flux reconciles → k8s Job created
  • Mirrors the hostexecutor deploy pattern in cd.yml

Acceptance criteria

  • k8s Job spec defined in mathias/infra k3s/apps/autoresearch/
  • LITELLM_KEY sourced from k8s Secret (ESO-synced, never in Job manifest)
  • GPU resource limits enforced (runtimeClassName: nvidia, nvidia.com/gpu: 1)
  • Run artifacts (STATUS.md, metrics.json) persisted to PVC or MinIO after Job completion
  • Gitea workflow submits Job instead of running loop directly
  • kubectl wait --for=condition=complete job/<rq_id> in workflow → artifact fetch

Gate

Phase A live run green (jepa-fx-risk#11 closed). Build the sandbox on proven loop mechanics, not aspirational ones.

Dep

  • #10 (DVC + MinIO for experiment artifacts) — if MinIO is the artifact store
  • mathias/infra for Job manifest + ESO secret sync
## Context Phase A uses a Gitea Actions self-hosted runner on koala (workflow_dispatch → scaffold → loop). That works but the runner shares GPU + filesystem with everything else on the node. Phase B promotes autoresearch to a proper k8s sandbox. ## What this replaces `.gitea/workflows/autoresearch.yml` triggers loop.py directly on the runner. Phase B replaces the `run` job with a k8s Job submission — the runner becomes an orchestrator, not an executor. ## Proposed design **Trigger:** same Gitea `workflow_dispatch` inputs (fixture, rq_id, iters, model) **Execution:** - Runner renders a k8s Job manifest from the inputs and `kubectl apply`s it - Job spec: - `runtimeClassName: nvidia` (koala GPU isolation — see brain: koala-gpu-setup) - `resources.limits`: `nvidia.com/gpu: 1`, appropriate CPU/memory - env: `LITELLM_KEY` (from k8s Secret, ESO-synced from 1Password) - env: `LITELLM_BASE`, `NTFY_URL` - `activeDeadlineSeconds`: time-box (90 min default) - Job mounts a PVC for the run dir (or writes to MinIO/DVC — see #10) - Job runs: `python scripts/autoresearch_start.py ... && python loop.py ...` - Job completion: STATUS.md + metrics.json available from PVC / uploaded to MinIO **GitOps path (alternative to runner-apply):** - Runner patches `k3s/apps/autoresearch-jobs/<rq_id>.yaml` in `mathias/infra` - Flux reconciles → k8s Job created - Mirrors the hostexecutor deploy pattern in `cd.yml` ## Acceptance criteria - [ ] k8s Job spec defined in `mathias/infra k3s/apps/autoresearch/` - [ ] `LITELLM_KEY` sourced from k8s Secret (ESO-synced, never in Job manifest) - [ ] GPU resource limits enforced (`runtimeClassName: nvidia`, `nvidia.com/gpu: 1`) - [ ] Run artifacts (STATUS.md, metrics.json) persisted to PVC or MinIO after Job completion - [ ] Gitea workflow submits Job instead of running loop directly - [ ] `kubectl wait --for=condition=complete job/<rq_id>` in workflow → artifact fetch ## Gate Phase A live run green (jepa-fx-risk#11 closed). Build the sandbox on proven loop mechanics, not aspirational ones. ## Dep - #10 (DVC + MinIO for experiment artifacts) — if MinIO is the artifact store - `mathias/infra` for Job manifest + ESO secret sync
Sign in to join this conversation.
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/jepa-fx-risk#15