mathias/hyperguild
1
0
Fork 0
Code Issues 30 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

test: project_create e2e findings + improvements needed for claude.ai project initiation #11

New Issue
Closed
opened 2026-05-18 12:33:41 +00:00 by mathias · 3 comments
mathias commented 2026-05-18 12:33:41 +00:00
Owner
Copy Link
Copy Source

Context

On 2026-05-14, project_create was tested end-to-end from a claude.ai session
using the gitea-mcp connector directly (orchestrating the four steps manually,
since the routing pod project_create tool is not yet exposed as a claude.ai connector).

Test args:

{
  "name":        "test-throwaway-001",
  "description": "Throwaway test for project_create verification",
  "hypothesis":  "We believe the project_create tool will provision all four artifacts correctly",
  "folder":      "AGENTS",
  "stack":       "go-web",
  "private":     true
}

Test results — four artifacts

# Artifact Status Finding
1 Gitea repo gitea.d-ma.be/mathias/test-throwaway-001 ✅ Created correctly, 6 files substituted
2 GitHub mirror github.com/mathiasb/test-throwaway-001 ❌ Mirror configured but PAT missing — repo never appeared on GitHub
3 Infra namespace k3s/staging/test-throwaway-001/namespace.yaml ✅ Committed to infra main, Flux will reconcile
4 Issue #1 with experiment brief ✅ Created at correct URL with full hypothesis

3 of 4 artifacts succeeded. GitHub mirror is the only blocker.

Finding 1 — GitHub PAT not configured in routing pod (BLOCKER)

repo_mirror_push accepts remote_password for GitHub auth, but the routing pod
config does not inject GITHUB_PAT into the mirror call. The mirror is registered
on Gitea but cannot authenticate to GitHub, so no push ever happens.

Fix: Set GITHUB_PAT in routing pod env (k8s secret on koala). The project.Config
already has a GitHubPAT field — wire it into the repo_mirror_push_add call as
remote_password.

Required GitHub PAT scopes: repo (create repos via mirror, push code).

Where: internal/skills/project/project.go → mirror step → pass cfg.GitHubPAT
as remote_password in the gitea-mcp repo_mirror_push call.

Finding 2 — template-go-agent not marked as template (BLOCKER for go-agent stack)

Calling create_project_from_template with template_name=template-go-agent returns:

repo mathias/template-go-agent is not marked as template: validation failed

template-go-web works correctly. template-go-agent exists as a repo but has
template: false on Gitea.

Fix (two parts):

  1. Mark template-go-agent as a template repo: repo_update(owner=mathias, name=template-go-agent, template=true)
  2. Populate template-go-agent with actual content (currently sparse) — see local-dev #5

Until both are done, go-agent stack falls back to go-web silently, which is wrong.
project_create should return an error if the requested template is not available,
not silently use a different one.

Finding 3 — routing pod project_create not reachable from claude.ai

The project_create tool lives in the routing pod (cmd/routing, internal/skills/project),
which runs on koala at http://koala:30310/mcp. This endpoint is NOT connected as
a claude.ai MCP connector — only gitea-mcp (git-mcp.d-ma.be) and brain-mcp
(brain-mcp.d-ma.be) are.

During the test, the four steps were orchestrated manually via gitea-mcp tools.
This works but requires 4 tool calls instead of 1 and has no atomicity or error handling.

Fix: Expose routing pod MCP endpoint publicly via NPM on piguard:

Domain:     routing.d-ma.be  (or mcp.d-ma.be)
Forward to: koala:30310
SSL:        Let's Encrypt
Auth:       existing Bearer token (ROUTING_MCP_TOKEN)

Then add as a claude.ai custom connector. Once connected, project_create is a
single tool call from any claude.ai session.

Finding 4 — infra namespace path inconsistency

Issue #10 spec said k8s/staging/<name>/namespace.yaml but the actual infra repo
uses k3s/staging/<name>/namespace.yaml. The test used k3s/ (correct) but the
routing pod implementation may use k8s/ (incorrect).

Fix: Verify internal/skills/project/project.go uses k3s/staging/ not k8s/staging/.
Update spec in PROJECT.md and issue #10 if needed.

Finding 5 — no cleanup/delete flow for throwaway projects

After testing, test-throwaway-001 exists on Gitea and in infra. There is no
project_delete tool or cleanup flow. Deleting requires:

  1. repo_delete on Gitea (gitea-mcp #11 — not yet implemented)
  2. Manual delete on GitHub (no tool)
  3. Remove k3s/staging/test-throwaway-001/ from infra repo manually
  4. Flux reconcile removes the namespace

Fix (v2): Add project_delete tool that reverses the four steps. Mark as HIGH risk,
require ntfy approval. For now: document manual cleanup steps.

Manual cleanup for test-throwaway-001:

# 1. Delete Gitea repo (web UI until repo_delete is implemented)
# 2. Delete GitHub repo (web UI — github.com/mathiasb/test-throwaway-001)
# 3. Remove infra namespace:
git -C ~/dev/infra rm -r k3s/staging/test-throwaway-001/
git -C ~/dev/infra commit -m "chore(staging): remove throwaway test namespace"
git -C ~/dev/infra push

Improvements needed before project initiation is production-ready

P0 — must fix before using for real projects

  • #F1 Set GITHUB_PAT in routing pod k8s secret, wire into mirror call
  • #F2 Mark template-go-agent as template + populate content (local-dev #5)
  • #F3 Expose routing pod via NPM → add as claude.ai connector

P1 — important but not blocking first real use

  • #F4 Verify infra path is k3s/staging/ not k8s/staging/ in project.go
  • #F5 project_create should error (not silently fallback) if template unavailable
  • #F6 Add project_delete cleanup tool (v2)
  • #F7 gitea-mcp #11 repo_delete — needed for cleanup flow

P2 — quality of life improvements

  • project_create should trigger immediate mirror sync after adding push mirror,
    not wait for next sync_on_commit event — Gitea API: POST /push_mirrors-sync
  • Return mirror sync status in output: {"mirror_synced": true/false, "mirror_lag_seconds": N}
  • Add project_create to a visible claude.ai connector so it appears in /tools
  • ntfy notification when project is fully provisioned (all 4 artifacts confirmed)
  • Validate name param matches Gitea naming rules before making any API calls
    (current regex ^[a-z][a-z0-9-]{1,38}[a-z0-9]$ is good — verify it's enforced)

Related

  • hyperguild #10 (project_create original spec — closed)
  • gitea-mcp #11 (repo_delete — needed for cleanup)
  • gitea-mcp #12 (repo_update — needed to mark template-go-agent as template)
  • gitea-mcp #19 (mirror flow e2e test)
  • local-dev #5 (populate template-go-agent)
  • infra #26 (ntfy — already done ✅)
  • brain: adr-new-project-gitea-first-github-mirror
## Context On 2026-05-14, `project_create` was tested end-to-end from a claude.ai session using the gitea-mcp connector directly (orchestrating the four steps manually, since the routing pod `project_create` tool is not yet exposed as a claude.ai connector). Test args: ```json { "name": "test-throwaway-001", "description": "Throwaway test for project_create verification", "hypothesis": "We believe the project_create tool will provision all four artifacts correctly", "folder": "AGENTS", "stack": "go-web", "private": true } ``` --- ## Test results — four artifacts | # | Artifact | Status | Finding | |---|---|---|---| | 1 | Gitea repo `gitea.d-ma.be/mathias/test-throwaway-001` | ✅ | Created correctly, 6 files substituted | | 2 | GitHub mirror `github.com/mathiasb/test-throwaway-001` | ❌ | Mirror configured but PAT missing — repo never appeared on GitHub | | 3 | Infra namespace `k3s/staging/test-throwaway-001/namespace.yaml` | ✅ | Committed to infra main, Flux will reconcile | | 4 | Issue #1 with experiment brief | ✅ | Created at correct URL with full hypothesis | **3 of 4 artifacts succeeded. GitHub mirror is the only blocker.** --- ## Finding 1 — GitHub PAT not configured in routing pod (BLOCKER) `repo_mirror_push` accepts `remote_password` for GitHub auth, but the routing pod config does not inject `GITHUB_PAT` into the mirror call. The mirror is registered on Gitea but cannot authenticate to GitHub, so no push ever happens. **Fix:** Set `GITHUB_PAT` in routing pod env (k8s secret on koala). The `project.Config` already has a `GitHubPAT` field — wire it into the `repo_mirror_push_add` call as `remote_password`. Required GitHub PAT scopes: `repo` (create repos via mirror, push code). **Where:** `internal/skills/project/project.go` → mirror step → pass `cfg.GitHubPAT` as `remote_password` in the gitea-mcp `repo_mirror_push` call. --- ## Finding 2 — template-go-agent not marked as template (BLOCKER for go-agent stack) Calling `create_project_from_template` with `template_name=template-go-agent` returns: ``` repo mathias/template-go-agent is not marked as template: validation failed ``` `template-go-web` works correctly. `template-go-agent` exists as a repo but has `template: false` on Gitea. **Fix (two parts):** 1. Mark `template-go-agent` as a template repo: `repo_update(owner=mathias, name=template-go-agent, template=true)` 2. Populate `template-go-agent` with actual content (currently sparse) — see local-dev #5 Until both are done, `go-agent` stack falls back to `go-web` silently, which is wrong. `project_create` should return an error if the requested template is not available, not silently use a different one. --- ## Finding 3 — routing pod project_create not reachable from claude.ai The `project_create` tool lives in the routing pod (`cmd/routing`, `internal/skills/project`), which runs on koala at `http://koala:30310/mcp`. This endpoint is NOT connected as a claude.ai MCP connector — only gitea-mcp (`git-mcp.d-ma.be`) and brain-mcp (`brain-mcp.d-ma.be`) are. During the test, the four steps were orchestrated manually via gitea-mcp tools. This works but requires 4 tool calls instead of 1 and has no atomicity or error handling. **Fix:** Expose routing pod MCP endpoint publicly via NPM on piguard: ``` Domain: routing.d-ma.be (or mcp.d-ma.be) Forward to: koala:30310 SSL: Let's Encrypt Auth: existing Bearer token (ROUTING_MCP_TOKEN) ``` Then add as a claude.ai custom connector. Once connected, `project_create` is a single tool call from any claude.ai session. --- ## Finding 4 — infra namespace path inconsistency Issue #10 spec said `k8s/staging/<name>/namespace.yaml` but the actual infra repo uses `k3s/staging/<name>/namespace.yaml`. The test used `k3s/` (correct) but the routing pod implementation may use `k8s/` (incorrect). **Fix:** Verify `internal/skills/project/project.go` uses `k3s/staging/` not `k8s/staging/`. Update spec in PROJECT.md and issue #10 if needed. --- ## Finding 5 — no cleanup/delete flow for throwaway projects After testing, `test-throwaway-001` exists on Gitea and in infra. There is no `project_delete` tool or cleanup flow. Deleting requires: 1. `repo_delete` on Gitea (gitea-mcp #11 — not yet implemented) 2. Manual delete on GitHub (no tool) 3. Remove `k3s/staging/test-throwaway-001/` from infra repo manually 4. Flux reconcile removes the namespace **Fix (v2):** Add `project_delete` tool that reverses the four steps. Mark as HIGH risk, require ntfy approval. For now: document manual cleanup steps. Manual cleanup for test-throwaway-001: ```bash # 1. Delete Gitea repo (web UI until repo_delete is implemented) # 2. Delete GitHub repo (web UI — github.com/mathiasb/test-throwaway-001) # 3. Remove infra namespace: git -C ~/dev/infra rm -r k3s/staging/test-throwaway-001/ git -C ~/dev/infra commit -m "chore(staging): remove throwaway test namespace" git -C ~/dev/infra push ``` --- ## Improvements needed before project initiation is production-ready ### P0 — must fix before using for real projects - [ ] **#F1** Set `GITHUB_PAT` in routing pod k8s secret, wire into mirror call - [ ] **#F2** Mark `template-go-agent` as template + populate content (local-dev #5) - [ ] **#F3** Expose routing pod via NPM → add as claude.ai connector ### P1 — important but not blocking first real use - [ ] **#F4** Verify infra path is `k3s/staging/` not `k8s/staging/` in project.go - [ ] **#F5** `project_create` should error (not silently fallback) if template unavailable - [ ] **#F6** Add `project_delete` cleanup tool (v2) - [ ] **#F7** gitea-mcp #11 `repo_delete` — needed for cleanup flow ### P2 — quality of life improvements - [ ] `project_create` should trigger immediate mirror sync after adding push mirror, not wait for next `sync_on_commit` event — Gitea API: `POST /push_mirrors-sync` - [ ] Return mirror sync status in output: `{"mirror_synced": true/false, "mirror_lag_seconds": N}` - [ ] Add `project_create` to a visible claude.ai connector so it appears in `/tools` - [ ] ntfy notification when project is fully provisioned (all 4 artifacts confirmed) - [ ] Validate `name` param matches Gitea naming rules before making any API calls (current regex `^[a-z][a-z0-9-]{1,38}[a-z0-9]$` is good — verify it's enforced) --- ## Related - hyperguild #10 (project_create original spec — closed) - gitea-mcp #11 (repo_delete — needed for cleanup) - gitea-mcp #12 (repo_update — needed to mark template-go-agent as template) - gitea-mcp #19 (mirror flow e2e test) - local-dev #5 (populate template-go-agent) - infra #26 (ntfy — already done ✅) - brain: adr-new-project-gitea-first-github-mirror
mathias commented 2026-05-18 13:14:52 +00:00
Author
Owner
Copy Link
Copy Source

Resolved across this session:

  • F1 — GITHUB_PAT added to SOPS-encrypted routing-secrets, routing pod restarted, log confirms github_pat_set=true.
  • F2 — template-go-agent flipped via repo_update(template=true). Both templates now generate.
  • F3 — routing-mcp.d-ma.be live via NPM on piguard → koala:30310. MCP_RESOURCE_URL advertised by /.well-known/oauth-protected-resource. claude.ai connector connected and tools callable.
  • F4 — path is k3s/staging/<name>/namespace.yaml in internal/skills/project/handlers.go.
  • F5 — templateFor maps go-agent→template-go-agent only; no silent fallback. Stack validation rejects anything outside go-agent/go-web.
  • F7 — repo_delete already in gitea-mcp since v0.2.4.

Plus this session shipped a 5th orchestration step (create_github_repo) since gitea push-mirror cannot push to a non-existent remote — see internal/githubclient/ + internal/skills/project/handlers.go step stepCreateGitHub. 422 already-exists treated as idempotent success.

Deferred to v2: F6 project_delete cleanup tool.

P2 wishlist items remain open as follow-up nice-to-haves but don't block project initiation use.

Resolved across this session: - **F1** — `GITHUB_PAT` added to SOPS-encrypted `routing-secrets`, routing pod restarted, log confirms `github_pat_set=true`. - **F2** — `template-go-agent` flipped via `repo_update(template=true)`. Both templates now generate. - **F3** — `routing-mcp.d-ma.be` live via NPM on piguard → koala:30310. `MCP_RESOURCE_URL` advertised by `/.well-known/oauth-protected-resource`. claude.ai connector connected and tools callable. - **F4** — path is `k3s/staging/<name>/namespace.yaml` in `internal/skills/project/handlers.go`. - **F5** — `templateFor` maps `go-agent`→`template-go-agent` only; no silent fallback. Stack validation rejects anything outside `go-agent`/`go-web`. - **F7** — `repo_delete` already in gitea-mcp since v0.2.4. Plus this session shipped a 5th orchestration step (`create_github_repo`) since gitea push-mirror cannot push to a non-existent remote — see `internal/githubclient/` + `internal/skills/project/handlers.go` step `stepCreateGitHub`. 422 already-exists treated as idempotent success. Deferred to v2: F6 `project_delete` cleanup tool. P2 wishlist items remain open as follow-up nice-to-haves but don't block project initiation use.
mathias commented 2026-05-18 13:31:24 +00:00
Author
Owner
Copy Link
Copy Source

Validation complete — 2026-05-18

All four artifacts verified after manual fix:

✅ Gitea repo: gitea.d-ma.be/mathias/test-throwaway-001
✅ GitHub repo: github.com/mathiasb/test-throwaway-001 (mirror synced, last_error: "")
✅ Infra namespace: k3s/staging/test-throwaway-001/namespace.yaml on main
✅ Issue #1: experiment brief on Gitea

Critical architecture finding — add to P0 list

F1 revised: project_create needs an explicit GitHub repo creation step BEFORE configuring the push mirror. Gitea push-mirror can only push to an existing remote — it cannot create repos on GitHub.

Correct flow:

1. create_project_from_template  → Gitea repo created
2. GitHub API: POST /user/repos  → GitHub repo created (empty)
3. repo_mirror_push add (with PAT as remote_password)  → mirror configured
4. push_mirrors-sync             → initial sync triggered
5. file_write_branch (infra)     → staging namespace
6. issue_create                  → experiment brief

This requires githubclient in internal/skills/project/project.go to call
POST /user/repos with {"name": <name>, "private": <private>} before the
mirror step. The githubclient.Client is already wired in cmd/routing/main.go
when GITHUB_PAT is set — it just needs to be used for repo creation.

Cleanup needed

test-throwaway-001 should be deleted after this issue is resolved:

  • Gitea: repo_delete (or web UI until #11/gitea-mcp is implemented)
  • GitHub: manual deletion at github.com/mathiasb/test-throwaway-001/settings
  • Infra: remove k3s/staging/test-throwaway-001/ from main
## Validation complete — 2026-05-18 All four artifacts verified after manual fix: ✅ Gitea repo: `gitea.d-ma.be/mathias/test-throwaway-001` ✅ GitHub repo: `github.com/mathiasb/test-throwaway-001` (mirror synced, last_error: "") ✅ Infra namespace: `k3s/staging/test-throwaway-001/namespace.yaml` on main ✅ Issue #1: experiment brief on Gitea ## Critical architecture finding — add to P0 list **F1 revised:** `project_create` needs an explicit GitHub repo creation step BEFORE configuring the push mirror. Gitea push-mirror can only push to an existing remote — it cannot create repos on GitHub. Correct flow: ``` 1. create_project_from_template → Gitea repo created 2. GitHub API: POST /user/repos → GitHub repo created (empty) 3. repo_mirror_push add (with PAT as remote_password) → mirror configured 4. push_mirrors-sync → initial sync triggered 5. file_write_branch (infra) → staging namespace 6. issue_create → experiment brief ``` This requires `githubclient` in `internal/skills/project/project.go` to call `POST /user/repos` with `{"name": <name>, "private": <private>}` before the mirror step. The `githubclient.Client` is already wired in `cmd/routing/main.go` when `GITHUB_PAT` is set — it just needs to be used for repo creation. ## Cleanup needed test-throwaway-001 should be deleted after this issue is resolved: - Gitea: repo_delete (or web UI until #11/gitea-mcp is implemented) - GitHub: manual deletion at github.com/mathiasb/test-throwaway-001/settings - Infra: remove k3s/staging/test-throwaway-001/ from main
mathias commented 2026-05-18 14:33:25 +00:00
Author
Owner
Copy Link
Copy Source

✅ RESOLVED — 2026-05-18

Full e2e test passed after fixing auth (issue #13):

{
  "reached": ["create_repo", "create_github_repo", "mirror", "infra_commit", "issue"],
  "gitea_url": "http://gitea.d-ma.be/mathias/test-throwaway-003",
  "github_url": "https://github.com/mathiasb/test-throwaway-003",
  "issue_url": "http://gitea.d-ma.be/mathias/test-throwaway-003/issues/1"
}

All five artifacts verified. project_create is production-ready from claude.ai.

One finding to note

Infra namespace is committed to a staging/<name> branch, not directly to main.
This requires a manual merge before Flux deploys. For TBD compliance, project_create
should commit directly to main — or we accept the branch as an intentional review gate
before staging is activated. Document this decision in PROJECT.md.

Cleanup done

  • test-throwaway-001: Gitea repo + infra namespace removed
  • test-throwaway-003: infra namespace removed (commit 170edf4)
  • Still pending manual deletion: test-throwaway-001/002/003 on Gitea web UI + GitHub
## ✅ RESOLVED — 2026-05-18 Full e2e test passed after fixing auth (issue #13): ```json { "reached": ["create_repo", "create_github_repo", "mirror", "infra_commit", "issue"], "gitea_url": "http://gitea.d-ma.be/mathias/test-throwaway-003", "github_url": "https://github.com/mathiasb/test-throwaway-003", "issue_url": "http://gitea.d-ma.be/mathias/test-throwaway-003/issues/1" } ``` All five artifacts verified. project_create is production-ready from claude.ai. ## One finding to note Infra namespace is committed to a `staging/<name>` branch, not directly to main. This requires a manual merge before Flux deploys. For TBD compliance, project_create should commit directly to main — or we accept the branch as an intentional review gate before staging is activated. Document this decision in PROJECT.md. ## Cleanup done - test-throwaway-001: Gitea repo + infra namespace removed - test-throwaway-003: infra namespace removed (commit 170edf4) - Still pending manual deletion: test-throwaway-001/002/003 on Gitea web UI + GitHub
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agent/ready
Meets the agent-ready contract; authorizes unattended execution
 blocked/client
Waiting on the client
 blocked/supplier
Waiting on a supplier
 blocked/upstream
Waiting on an upstream project or vendor
 risk/high
Irreversible, security-relevant, or control plane
 risk/low
Reversible, local, test-covered
 risk/medium
Touches shared state, deploys or data
 size/l
Too big - split it
 size/m
Under two days
 size/s
Under half a day
 size/xs
Under an hour
 status:dispatch
status:dispatched
status:review
status:reviewing
type/chore
Maintenance, no behaviour change
 type/decision
Human decision required; never agent/ready
 type/docs
Documentation only
 type/feat
New capability
 type/fix
Corrects broken behaviour
 type/ops
Infrastructure, deploy, runtime
 type/research
Investigate; output is a finding, not code
 type/test
Test coverage only
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
renovate-bot mathias
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/hyperguild#11