mathias/hyperguild
1
0
Fork 0
Code Issues 30 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

CD runs twice per push and the second run fails on "nothing to commit" — a red CI signal with nothing wrong #118

New Issue
Closed
opened 2026-09-16 22:09:17 +00:00 by mathias · 2 comments
mathias commented 2026-09-16 22:09:17 +00:00
Owner
Copy Link
Copy Source

What happens

Every push to main produces two workflow_run-triggered CD runs. The first does the work; the second finds the infra repo already bumped and dies:

Built and pushed git.d-ma.be/mathias/ingestion:eb2acfe...
Cloning into '/tmp/infra-update'...
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean
  ❌  Failure - Main Update infra repo
exit status 1

Observed on runs 417 (success) and 418 (failure), both for eb2acfeeb8888ba3936250424e4309c6d49c57b1. Same duplicate pairing on 413/414 and 415/416. The deploy itself was fine: mathias/infra carries chore(deploy): ingestion → eb2acfe… and the pod is Running on the new image.

Why it matters

A red run that means "already done" is indistinguishable at a glance from a red run that means "the deploy broke", and the estate now files a tracked issue off CI-red automatically (infra#366). So this manufactures a false issue on every push, and trains the reader to ignore the real ones.

Do

  1. Make the commit step idempotent — git diff --cached --quiet || git commit …, the same guard brain-audit's own worklist step already uses, so a no-op bump exits 0.
  2. Then find the duplicate trigger: two workflow_run runs per push means the CD workflow is listening to something that fires twice (two workflows: entries, or both push and workflow_run reaching it). Fixing (1) alone hides the waste — a second full image build per push — without removing it.

Found while verifying the deploy of v0.18.1.

Created via git-mcp on behalf of @mathias

## What happens Every push to main produces **two** `workflow_run`-triggered CD runs. The first does the work; the second finds the infra repo already bumped and dies: ``` Built and pushed git.d-ma.be/mathias/ingestion:eb2acfe... Cloning into '/tmp/infra-update'... On branch main Your branch is up to date with 'origin/main'. nothing to commit, working tree clean ❌ Failure - Main Update infra repo exit status 1 ``` Observed on runs 417 (success) and 418 (failure), both for `eb2acfeeb8888ba3936250424e4309c6d49c57b1`. Same duplicate pairing on 413/414 and 415/416. The deploy itself was fine: `mathias/infra` carries `chore(deploy): ingestion → eb2acfe…` and the pod is `Running` on the new image. ## Why it matters A red run that means "already done" is indistinguishable at a glance from a red run that means "the deploy broke", and the estate now files a tracked issue off CI-red automatically (infra#366). So this manufactures a false issue on every push, and trains the reader to ignore the real ones. ## Do 1. Make the commit step idempotent — `git diff --cached --quiet || git commit …`, the same guard `brain-audit`'s own worklist step already uses, so a no-op bump exits 0. 2. Then find the duplicate trigger: two `workflow_run` runs per push means the CD workflow is listening to something that fires twice (two `workflows:` entries, or both `push` and `workflow_run` reaching it). Fixing (1) alone hides the waste — a second full image build per push — without removing it. Found while verifying the deploy of v0.18.1. --- _Created via git-mcp on behalf of @mathias_
mathias added the risk/mediumsize/stype/fix labels 2026-09-16 22:23:47 +00:00
mathias commented 2026-09-17 05:07:39 +00:00
Author
Owner
Copy Link
Copy Source

CLAIMED — Claude Code session on koala, 2026-09-16 22:15Z, part of a 4-issue sprint (hyperguild#117, #118, brain-gardener#196, #197). Another agent is active on this estate: please do not pick this up in parallel.

Update to the report: the duplicate CD pair did not fail on the next push (runs 419/420/421 for 6e69698 all green). So the failure is intermittent, not every-push — which fits step 1 being a genuine race on "did the other run already commit the bump", rather than a deterministic second-run no-op. Doesn't change the fix; does mean a green run is not evidence the race is gone, so the idempotency guard goes in regardless.

Created via git-mcp on behalf of @mathias

**CLAIMED** — Claude Code session on koala, 2026-09-16 22:15Z, part of a 4-issue sprint (hyperguild#117, #118, brain-gardener#196, #197). Another agent is active on this estate: please do not pick this up in parallel. Update to the report: the duplicate CD pair did **not** fail on the next push (runs 419/420/421 for `6e69698` all green). So the failure is intermittent, not every-push — which fits step 1 being a genuine race on "did the other run already commit the bump", rather than a deterministic second-run no-op. Doesn't change the fix; does mean a green run is not evidence the race is gone, so the idempotency guard goes in regardless. --- _Created via git-mcp on behalf of @mathias_
mathias added risk/lowstatus:dispatched and removed risk/medium labels 2026-09-17 05:08:07 +00:00
mathias commented 2026-09-17 05:10:39 +00:00
Author
Owner
Copy Link
Copy Source

DONE — cc86a74, tagged v0.18.3.

The duplicate had a specific cause, not a flaky trigger: ci.yml ran on push to main and on v* tags, and cd.yml keys off "CI succeeded". So git push origin main followed by git push origin refs/tags/vX produced two CI runs for the same sha, and two full build+deploys behind them. The tag-triggered deploy rebuilt an identical image and then found the infra manifest already pointing at it — git commit on an empty index exits 1, and the job went red.

That also explains the intermittency noted above: the failure needs a tag push, so a commit pushed without one deploys exactly once and stays green. Runs 419-421 were green because the tag went up before the CD run got that far, not because the race had gone.

Both parts fixed:

  1. ci.yml no longer triggers on tags. Nothing consumed those runs — no release workflow exists, and cd.yml reads github.sha, not the tag. This removes the wasted second image build as well as the red.
  2. The infra bump is now git diff --cached --quiet || commit+push, so any no-op run (a manual re-run, anything else that arrives second) logs "Infra already at " and exits 0.

Verification note, since a green run is weak evidence here: the next push with a tag is the real test. v0.18.3 was pushed after this commit landed and produced no second CD run — that is the first push in the series where the tag did not trigger a build at all.

Created via git-mcp on behalf of @mathias

**DONE** — `cc86a74`, tagged `v0.18.3`. The duplicate had a specific cause, not a flaky trigger: `ci.yml` ran on `push` to main **and** on `v*` tags, and `cd.yml` keys off "CI succeeded". So `git push origin main` followed by `git push origin refs/tags/vX` produced two CI runs for the same sha, and two full build+deploys behind them. The tag-triggered deploy rebuilt an identical image and then found the infra manifest already pointing at it — `git commit` on an empty index exits 1, and the job went red. That also explains the intermittency noted above: the failure needs a tag push, so a commit pushed without one deploys exactly once and stays green. Runs 419-421 were green because the tag went up before the CD run got that far, not because the race had gone. Both parts fixed: 1. `ci.yml` no longer triggers on tags. Nothing consumed those runs — no release workflow exists, and `cd.yml` reads `github.sha`, not the tag. This removes the wasted second image build as well as the red. 2. The infra bump is now `git diff --cached --quiet || commit+push`, so any no-op run (a manual re-run, anything else that arrives second) logs "Infra already at <sha>" and exits 0. Verification note, since a green run is weak evidence here: the next push with a tag is the real test. `v0.18.3` was pushed after this commit landed and produced **no** second CD run — that is the first push in the series where the tag did not trigger a build at all. --- _Created via git-mcp on behalf of @mathias_
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agent/ready
Meets the agent-ready contract; authorizes unattended execution
 blocked/client
Waiting on the client
 blocked/supplier
Waiting on a supplier
 blocked/upstream
Waiting on an upstream project or vendor
 risk/high
Irreversible, security-relevant, or control plane
 risk/low
Reversible, local, test-covered
 risk/medium
Touches shared state, deploys or data
 size/l
Too big - split it
 size/m
Under two days
 size/s
Under half a day
 size/xs
Under an hour
 status:dispatch
status:dispatched
status:review
status:reviewing
type/chore
Maintenance, no behaviour change
 type/decision
Human decision required; never agent/ready
 type/docs
Documentation only
 type/feat
New capability
 type/fix
Corrects broken behaviour
 type/ops
Infrastructure, deploy, runtime
 type/research
Investigate; output is a finding, not code
 type/test
Test coverage only
No labels risk/low size/s status:dispatched type/fix
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
renovate-bot mathias
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/hyperguild#118