Two runs of the full build+deploy fired for every push, and the second died:
nothing to commit, working tree clean
❌ Failure - Main Update infra repo
The deploy was fine both times — the second run had nothing left to do. The
cause is the pair of triggers meeting: ci.yml ran on `push` to main AND on a
`v*` tag, and cd.yml keys off "CI succeeded", so `git push origin main` followed
by `git push origin refs/tags/vX` produced two CI runs for one sha and two
deploys behind them. The tag run carries the same github.sha, so it rebuilt the
same image and then found the infra manifest already pointing at it.
Nothing consumed the tag-triggered run: there is no release workflow, and cd.yml
reads github.sha rather than the tag. Dropping `tags:` removes the duplicate
build outright.
The idempotency guard stays anyway, because a manual re-run has the same shape
and should not ring the CI-red alarm for work that is already done — infra#366
turns a red run into a tracked issue, so a false red costs a real triage.
Closes #118
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
136 lines
6.1 KiB
YAML
136 lines
6.1 KiB
YAML
name: cd
|
|
|
|
"on":
|
|
workflow_run:
|
|
workflows: ["CI"]
|
|
types: [completed]
|
|
branches: [main]
|
|
|
|
jobs:
|
|
deploy:
|
|
name: Build and deploy
|
|
runs-on: self-hosted
|
|
if: ${{ github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.event == 'push' }}
|
|
environment: staging
|
|
env:
|
|
INGESTION_IMAGE: git.d-ma.be/mathias/ingestion
|
|
INFRA_REPO: git@git.d-ma.be:mathias/infra.git
|
|
BUILDKIT_HOST: unix:///run/buildkit/buildkitd.sock
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v4
|
|
|
|
- name: Build and push ingestion image
|
|
run: |
|
|
set -e
|
|
trap 'rm -f /tmp/ingestion-image.tar' EXIT
|
|
IMAGE_TAG="${{ github.sha }}"
|
|
echo "Building ${INGESTION_IMAGE}:${IMAGE_TAG}"
|
|
|
|
buildctl --addr "${BUILDKIT_HOST}" build \
|
|
--frontend dockerfile.v0 \
|
|
--local context=ingestion \
|
|
--local dockerfile=ingestion \
|
|
--output type=oci,dest=/tmp/ingestion-image.tar
|
|
|
|
skopeo copy \
|
|
oci-archive:/tmp/ingestion-image.tar \
|
|
docker://${INGESTION_IMAGE}:${IMAGE_TAG} \
|
|
--dest-creds "${{ secrets.REGISTRY_CREDS }}"
|
|
|
|
echo "Built and pushed ${INGESTION_IMAGE}:${IMAGE_TAG}"
|
|
|
|
- name: Update infra repo
|
|
run: |
|
|
set -e
|
|
trap 'rm -rf /tmp/infra-update; rm -f ~/.ssh/infra_deploy_key' EXIT
|
|
IMAGE_TAG="${{ github.sha }}"
|
|
# infra#311: github.sha is never empty in practice, but the manifest
|
|
# patch below is the same shape that took Flux down estate-wide on
|
|
# 2026-08-25 -- an empty tag yields `image: <repo>:`, which is invalid
|
|
# YAML and fails the ENTIRE Kustomization, not just this app. The
|
|
# assert costs nothing and removes the class.
|
|
[ -n "${IMAGE_TAG:-}" ] || { echo "IMAGE_TAG is empty - refusing to patch"; exit 1; }
|
|
mkdir -p ~/.ssh
|
|
echo "${{ secrets.INFRA_DEPLOY_KEY }}" > ~/.ssh/infra_deploy_key
|
|
chmod 600 ~/.ssh/infra_deploy_key
|
|
# In-cluster DNS to gitea's SSH NodePort service, not 127.0.0.1:30022
|
|
# (that only worked when act_runner ran on koala's bare host network;
|
|
# from inside the containerized runner's own pod netns, loopback
|
|
# never reaches the host — "Connection refused", found 2026-07-27).
|
|
#
|
|
# Pass as -o overrides on the ssh invocation itself, NOT appended to
|
|
# ~/.ssh/config: $HOME (/data) is a PVC that persists across job
|
|
# runs on this runner (same "workspace not ephemeral" class as
|
|
# brain: act-runner-host-executor-tmp-persists), so an appended
|
|
# line here would pile up duplicate `Host git.d-ma.be` blocks
|
|
# across every run — ssh_config is first-match-wins, so a stale
|
|
# entry from an earlier failed run would silently shadow this
|
|
# fix forever (exactly what happened once already: this fix's
|
|
# own first attempt got appended AFTER an already-stale entry
|
|
# and lost). CLI -o options always win regardless of file state,
|
|
# so this step is safe to re-run any number of times.
|
|
GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes -o HostName=gitea-ssh-nodeport.gitea.svc.cluster.local -o Port=22 -o StrictHostKeyChecking=no" \
|
|
git clone "${INFRA_REPO}" /tmp/infra-update
|
|
|
|
cd /tmp/infra-update
|
|
|
|
sed -i "s|git.d-ma.be/mathias/ingestion:.*|git.d-ma.be/mathias/ingestion:${IMAGE_TAG}|" \
|
|
"k3s/apps/supervisor/ingestion-deployment.yaml"
|
|
|
|
git config user.email "cd-bot@d-ma.be"
|
|
git config user.name "CD Bot"
|
|
git add "k3s/apps/supervisor/ingestion-deployment.yaml"
|
|
# Idempotent: `git commit` exits 1 on an empty index, so a run that
|
|
# finds the bump already applied failed the whole job and rang the
|
|
# CI-red alarm for a deploy that was fine (hyperguild#118). Two
|
|
# sources of a no-op run: a manual re-run, and the tag-push CI run
|
|
# that used to shadow the branch-push one for the same sha.
|
|
if git diff --cached --quiet; then
|
|
echo "Infra already at ${IMAGE_TAG} — nothing to commit"
|
|
else
|
|
git commit -m "chore(deploy): ingestion → ${IMAGE_TAG}"
|
|
GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes" \
|
|
git push
|
|
echo "Infra repo updated: ingestion → ${IMAGE_TAG}"
|
|
fi
|
|
|
|
- name: Trigger Flux reconcile (immediate)
|
|
run: |
|
|
kubectl -n flux-system annotate gitrepository flux-system \
|
|
reconcile.fluxcd.io/requestedAt="$(date +%s)" --overwrite
|
|
kubectl -n flux-system annotate kustomization apps \
|
|
reconcile.fluxcd.io/requestedAt="$(date +%s)" --overwrite
|
|
|
|
- name: Wait for Flux to apply new ingestion image
|
|
run: |
|
|
EXPECTED="git.d-ma.be/mathias/ingestion:${{ github.sha }}"
|
|
for i in $(seq 1 60); do
|
|
CURRENT=$(kubectl get deploy ingestion -n supervisor \
|
|
-o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null || echo "")
|
|
if [ "$CURRENT" = "$EXPECTED" ]; then
|
|
echo "✓ Flux applied ingestion image after ${i}s"
|
|
break
|
|
fi
|
|
sleep 1
|
|
done
|
|
kubectl get deploy ingestion -n supervisor \
|
|
-o jsonpath='{.spec.template.spec.containers[0].image}' \
|
|
| grep -qx "$EXPECTED" \
|
|
|| { echo "✗ Flux did not apply ingestion image within 60s"; exit 1; }
|
|
|
|
- name: Verify ingestion rollout
|
|
run: |
|
|
kubectl rollout status deployment/ingestion \
|
|
--namespace supervisor \
|
|
--timeout=120s \
|
|
|| {
|
|
echo "── pod status ──"
|
|
kubectl get pods -n supervisor -o wide
|
|
echo "── events ──"
|
|
kubectl get events -n supervisor --sort-by='.lastTimestamp' | tail -20
|
|
echo "── describe ──"
|
|
kubectl describe pods -n supervisor -l app=ingestion | tail -40
|
|
exit 1
|
|
}
|