Migrating a Runner Fleet to Dual-Stack IPv6: What Broke

TL;DR

TL;DR: An emptied env var turned an image ref into /node:… ("invalid reference format"). Job-token allowlists reset to self-only → 403 on cross-project pulls. And component includes resolve at pipeline creation, so retrying old pipelines reuses the broken resolution — you must trigger fresh ones.

What happened

Break 1 — the empty variable that became a bad ref. Release jobs started dying before running a single line of script:

release:semver

Using docker image /node:24-alpine@sha256:7fdd…
ERROR: failed to pull image "/node:24-alpine@…":
        invalid reference format

The image was written as ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/node:…. Mid-migration that dependency-proxy variable was empty, so the ref resolved to a leading-slash nonsense string. It didn't error as "unset" — it fell open into a syntactically-invalid reference. The fix was to stop depending on the proxy var for this and pin the mirror registry path explicitly. Rule of thumb earned: an unset CI variable interpolated into a URL or image ref is a latent malformed-string bug, not a missing-value error.

Break 2 — job-token allowlists reset to self. Cross-project pulls (a shared PHP runtime image from another project) started returning 403. The inbound job-token allowlist had reset to self-only during the reconfiguration, so tokens minted by one project were no longer trusted by the other:

re-grant the group allowlist

$ glab api --method POST \
    projects/713/job_token_scope/groups_allowlist \
    -F target_group_id=175
{ "source_project_id": 713, "target_group_id": 175 }

Break 3 — the subtle one: retrying doesn't re-resolve. After fixing the component, I hit "retry" on the failed pipelines. Still broken. GitLab resolves include: component@version at pipeline creation time and freezes it into that pipeline. Retrying a job re-runs it against the old, frozen resolution — the broken one. The only way to pick up a component fix is a brand-new pipeline:

trigger fresh, don't retry

$ glab api --method POST "projects/599/pipeline?ref=main"
# component@main re-resolves to the fixed commit — retry never would have

Underneath all three: reachability. The convenience mirror was IPv4-only while the pods were coming up IPv6-first, so anything pointed at it stalled until it was moved to the dual-stack mirror. Once you accept that a "network migration" quietly touches image resolution, token trust, and pipeline caching, the surprises stop being surprising.

The lesson

Infra migrations fail open in CI: empty vars become malformed refs, reset allowlists become 403s, and frozen component resolution makes "retry" a trap. When a component is fixed, trigger a fresh pipeline — never retry the old one.

Conclusion

A network migration rarely stays at the network layer — it touches image resolution, token trust, and pipeline caching in ways that only surface once you try it. All three breaks failed open instead of erroring clearly, which made them harder to diagnose than they needed to be.