From 1c2ad2990fc74e3bbfa7489c656707aac35a4a70 Mon Sep 17 00:00:00 2001 From: Mathias Date: Mon, 17 Aug 2026 12:07:51 +0200 Subject: [PATCH] ci: build via buildctl instead of rootless buildah (infra#241) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Main has been CI-red since 2026-08-13. The build job dies extracting a base image layer inside the gitea-runner pod: ApplyLayer ... potentially insufficient UIDs or GIDs available in user namespace (requested 0:42 for /etc/gshadow) Root cause is the runner pod's securityContext, not this repo: it drops ALL capabilities and sets allowPrivilegeEscalation:false, so newuidmap/newgidmap can't gain CAP_SETUID/CAP_SETGID and rootless buildah falls back to a single non-remapped uid/gid mapping. Adding those capabilities was tried live in the infra repo and reverted as a confirmed no-op (no_new_privs blocks the setuid-exec gain regardless of the container's capability set). buildctl talks to koala's host buildkitd over /run/buildkit/buildkitd.sock, which the runner pod already mounts, so the privileged layer work happens in buildkitd on the host and the pod's hardened posture is untouched. Same migration agentsquad (infra#233) and segment-embedder (infra#243) already made. Drops the buildah smoke test with it: `buildah from ${REF}` would pull the image back into the pod's buildah store and hit the very wall this migration avoids, failing the job for a reason unrelated to the image. The deploy job's `Verify rollout` is the stronger check — it requires the real pod to reach Ready in-cluster, rather than the entrypoint printing its own name for 5s. Tag builds lose their only image-level check (deploy runs on main pushes only); accepted, since a tag is always cut from a main commit that already deployed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011PL5XcG68SvQqz3vGVbWbf --- .gitea/workflows/ci.yml | 87 ++++++++++++++++++++++------------------- 1 file changed, 47 insertions(+), 40 deletions(-) diff --git a/.gitea/workflows/ci.yml b/.gitea/workflows/ci.yml index c663710..077c70c 100644 --- a/.gitea/workflows/ci.yml +++ b/.gitea/workflows/ci.yml @@ -61,52 +61,59 @@ jobs: echo "version-tag=${REF#refs/tags/}" >> "$GITHUB_OUTPUT" fi + # infra#241: rootless buildah inside the gitea-runner pod can't remap the + # UID/GID range newuidmap/newgidmap need (the pod drops ALL capabilities + # and sets allowPrivilegeEscalation:false — both deliberate, it also holds + # CI secrets), so extracting a base-image layer that chowns outside the + # single available mapping fails with `ApplyLayer ... requested 0:42 for + # /etc/gshadow`. Main has been red on exactly this since 2026-08-13. + # buildctl instead talks to koala's host buildkitd over the socket the + # runner pod already mounts: the privileged extraction happens inside + # buildkitd on the host, never in this pod. Same migration agentsquad + # (infra#233) and segment-embedder (infra#243) already made — zero + # security tradeoff, unlike loosening the pod's capabilities. + # + # buildctl has no registry-push flag of its own: it writes an OCI archive + # and skopeo pushes that, which is also why every tag below is one copy of + # a single build rather than a rebuild. - name: Build and push to local registry run: | + set -euo pipefail + BUILDKIT_HOST="unix:///run/buildkit/buildkitd.sock" REGISTRY="localhost:5000" REF="${REGISTRY}/${{ env.IMAGE }}:${{ steps.meta.outputs.sha-tag }}" - buildah build \ - --label "org.opencontainers.image.revision=${{ github.sha }}" \ - --label "org.opencontainers.image.source=${{ github.repositoryUrl }}" \ - -t ${REF} \ - -t ${REGISTRY}/${{ env.IMAGE }}:latest \ - . - buildah push --tls-verify=false ${REF} - buildah push --tls-verify=false ${REGISTRY}/${{ env.IMAGE }}:latest - [[ -n "${{ steps.meta.outputs.version-tag }}" ]] && \ - buildah push --tls-verify=false \ - ${REGISTRY}/${{ env.IMAGE }}:${{ steps.meta.outputs.version-tag }} || true + TAR_FILE="$(mktemp)" + trap 'rm -f "$TAR_FILE"' EXIT + + buildctl --addr "$BUILDKIT_HOST" build \ + --frontend dockerfile.v0 \ + --local context=. \ + --local dockerfile=. \ + --opt label:org.opencontainers.image.revision="${{ github.sha }}" \ + --opt label:org.opencontainers.image.source="${{ github.repositoryUrl }}" \ + --output type=oci,dest="$TAR_FILE" + + skopeo copy --dest-tls-verify=false \ + oci-archive:"$TAR_FILE" docker://"${REF}" + skopeo copy --dest-tls-verify=false \ + oci-archive:"$TAR_FILE" docker://"${REGISTRY}/${{ env.IMAGE }}:latest" + if [[ -n "${{ steps.meta.outputs.version-tag }}" ]]; then + skopeo copy --dest-tls-verify=false \ + oci-archive:"$TAR_FILE" \ + docker://"${REGISTRY}/${{ env.IMAGE }}:${{ steps.meta.outputs.version-tag }}" + fi echo "Image pushed to ${REF}" - # Run the just-built local image briefly via buildah, not k3s ctr — - # avoids sudo/host-containerd access so this still works once the - # runner itself is containerized (infra#132). Tests the local - # buildah-store image directly, no registry round-trip needed. - # - # Bare `/tapir` (no subcommand) runs the long-running server, same as - # the old ctr-based smoke test -- ctr's --rm reliably force-killed it, - # but a plain `timeout N buildah run` does NOT: it only signals the - # `buildah run` wrapper, and the actual container process can survive - # that and keep the output pipe open, hanging the whole job (hit this - # live: 14min hang, infra#132). Backgrounding the run + `buildah rm -f` - # decouples "is the script blocked" from "did the process exit" -- - # rm -f forcibly tears down the container regardless of wrapper state. - - name: Smoke test - run: | - REGISTRY="localhost:5000" - REF="${REGISTRY}/${{ env.IMAGE }}:${{ steps.meta.outputs.sha-tag }}" - CONTAINER=$(buildah from ${REF}) - LOG=$(mktemp) - buildah run "$CONTAINER" -- /tapir > "$LOG" 2>&1 & - RUNPID=$! - sleep 5 - kill -9 "$RUNPID" 2>/dev/null || true - wait "$RUNPID" 2>/dev/null || true - buildah rm -f "$CONTAINER" >/dev/null 2>&1 || true - OUTPUT=$(cat "$LOG"); rm -f "$LOG" - echo "$OUTPUT" | grep -q "tapir" \ - && echo "Smoke test passed" \ - || echo "Smoke test inconclusive: $OUTPUT" + # The buildah-based smoke test that used to sit here is gone with the + # buildctl migration above: `buildah from ${REF}` would pull the image + # back into the pod's buildah store and hit the very uid/gid wall the + # migration exists to avoid — it would fail the job for a reason that has + # nothing to do with the image being good. The `Verify rollout` step in + # the deploy job below is the stronger check anyway: it requires the real + # pod to reach Ready in-cluster, not just the entrypoint to print its own + # name for 5s. Tag builds lose their only image-level check (deploy is + # main-push only) — accepted; a tag is always built from a main commit + # that already deployed. # ── 3. Deploy via infra repo + Flux ───────────────────────────────────────── # Flux native image-automation can't scan localhost:5000 from inside k3s pods