Files
mathiasandClaude Opus 5 cc86a74dc5
CI / Lint / Test / Vet (push) Successful in 34s
CI / Mirror to GitHub (push) Successful in 4s
fix(ci): stop the tag push from re-deploying, and make the infra bump idempotent
Two runs of the full build+deploy fired for every push, and the second died:

    nothing to commit, working tree clean
    ❌  Failure - Main Update infra repo

The deploy was fine both times — the second run had nothing left to do. The
cause is the pair of triggers meeting: ci.yml ran on `push` to main AND on a
`v*` tag, and cd.yml keys off "CI succeeded", so `git push origin main` followed
by `git push origin refs/tags/vX` produced two CI runs for one sha and two
deploys behind them. The tag run carries the same github.sha, so it rebuilt the
same image and then found the infra manifest already pointing at it.

Nothing consumed the tag-triggered run: there is no release workflow, and cd.yml
reads github.sha rather than the tag. Dropping `tags:` removes the duplicate
build outright.

The idempotency guard stays anyway, because a manual re-run has the same shape
and should not ring the CI-red alarm for work that is already done — infra#366
turns a red run into a tracked issue, so a false red costs a real triage.

Closes #118

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-17 07:09:14 +02:00

136 lines
6.1 KiB
YAML

name: cd
"on":
workflow_run:
workflows: ["CI"]
types: [completed]
branches: [main]
jobs:
deploy:
name: Build and deploy
runs-on: self-hosted
if: ${{ github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.event == 'push' }}
environment: staging
env:
INGESTION_IMAGE: git.d-ma.be/mathias/ingestion
INFRA_REPO: git@git.d-ma.be:mathias/infra.git
BUILDKIT_HOST: unix:///run/buildkit/buildkitd.sock
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Build and push ingestion image
run: |
set -e
trap 'rm -f /tmp/ingestion-image.tar' EXIT
IMAGE_TAG="${{ github.sha }}"
echo "Building ${INGESTION_IMAGE}:${IMAGE_TAG}"
buildctl --addr "${BUILDKIT_HOST}" build \
--frontend dockerfile.v0 \
--local context=ingestion \
--local dockerfile=ingestion \
--output type=oci,dest=/tmp/ingestion-image.tar
skopeo copy \
oci-archive:/tmp/ingestion-image.tar \
docker://${INGESTION_IMAGE}:${IMAGE_TAG} \
--dest-creds "${{ secrets.REGISTRY_CREDS }}"
echo "Built and pushed ${INGESTION_IMAGE}:${IMAGE_TAG}"
- name: Update infra repo
run: |
set -e
trap 'rm -rf /tmp/infra-update; rm -f ~/.ssh/infra_deploy_key' EXIT
IMAGE_TAG="${{ github.sha }}"
# infra#311: github.sha is never empty in practice, but the manifest
# patch below is the same shape that took Flux down estate-wide on
# 2026-08-25 -- an empty tag yields `image: <repo>:`, which is invalid
# YAML and fails the ENTIRE Kustomization, not just this app. The
# assert costs nothing and removes the class.
[ -n "${IMAGE_TAG:-}" ] || { echo "IMAGE_TAG is empty - refusing to patch"; exit 1; }
mkdir -p ~/.ssh
echo "${{ secrets.INFRA_DEPLOY_KEY }}" > ~/.ssh/infra_deploy_key
chmod 600 ~/.ssh/infra_deploy_key
# In-cluster DNS to gitea's SSH NodePort service, not 127.0.0.1:30022
# (that only worked when act_runner ran on koala's bare host network;
# from inside the containerized runner's own pod netns, loopback
# never reaches the host — "Connection refused", found 2026-07-27).
#
# Pass as -o overrides on the ssh invocation itself, NOT appended to
# ~/.ssh/config: $HOME (/data) is a PVC that persists across job
# runs on this runner (same "workspace not ephemeral" class as
# brain: act-runner-host-executor-tmp-persists), so an appended
# line here would pile up duplicate `Host git.d-ma.be` blocks
# across every run — ssh_config is first-match-wins, so a stale
# entry from an earlier failed run would silently shadow this
# fix forever (exactly what happened once already: this fix's
# own first attempt got appended AFTER an already-stale entry
# and lost). CLI -o options always win regardless of file state,
# so this step is safe to re-run any number of times.
GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes -o HostName=gitea-ssh-nodeport.gitea.svc.cluster.local -o Port=22 -o StrictHostKeyChecking=no" \
git clone "${INFRA_REPO}" /tmp/infra-update
cd /tmp/infra-update
sed -i "s|git.d-ma.be/mathias/ingestion:.*|git.d-ma.be/mathias/ingestion:${IMAGE_TAG}|" \
"k3s/apps/supervisor/ingestion-deployment.yaml"
git config user.email "cd-bot@d-ma.be"
git config user.name "CD Bot"
git add "k3s/apps/supervisor/ingestion-deployment.yaml"
# Idempotent: `git commit` exits 1 on an empty index, so a run that
# finds the bump already applied failed the whole job and rang the
# CI-red alarm for a deploy that was fine (hyperguild#118). Two
# sources of a no-op run: a manual re-run, and the tag-push CI run
# that used to shadow the branch-push one for the same sha.
if git diff --cached --quiet; then
echo "Infra already at ${IMAGE_TAG} — nothing to commit"
else
git commit -m "chore(deploy): ingestion → ${IMAGE_TAG}"
GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes" \
git push
echo "Infra repo updated: ingestion → ${IMAGE_TAG}"
fi
- name: Trigger Flux reconcile (immediate)
run: |
kubectl -n flux-system annotate gitrepository flux-system \
reconcile.fluxcd.io/requestedAt="$(date +%s)" --overwrite
kubectl -n flux-system annotate kustomization apps \
reconcile.fluxcd.io/requestedAt="$(date +%s)" --overwrite
- name: Wait for Flux to apply new ingestion image
run: |
EXPECTED="git.d-ma.be/mathias/ingestion:${{ github.sha }}"
for i in $(seq 1 60); do
CURRENT=$(kubectl get deploy ingestion -n supervisor \
-o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null || echo "")
if [ "$CURRENT" = "$EXPECTED" ]; then
echo "✓ Flux applied ingestion image after ${i}s"
break
fi
sleep 1
done
kubectl get deploy ingestion -n supervisor \
-o jsonpath='{.spec.template.spec.containers[0].image}' \
| grep -qx "$EXPECTED" \
|| { echo "✗ Flux did not apply ingestion image within 60s"; exit 1; }
- name: Verify ingestion rollout
run: |
kubectl rollout status deployment/ingestion \
--namespace supervisor \
--timeout=120s \
|| {
echo "── pod status ──"
kubectl get pods -n supervisor -o wide
echo "── events ──"
kubectl get events -n supervisor --sort-by='.lastTimestamp' | tail -20
echo "── describe ──"
kubectl describe pods -n supervisor -l app=ingestion | tail -40
exit 1
}