I Converted a Kubernetes Cluster to Dual-Stack So git Could Go IPv6-Only — Then AWS Said No

TL;DR

TL;DR: A self-hosted GitLab can't go inbound-IPv6-only as long as it's your AWS IAM OIDC provider. AWS STS fetches the OIDC verification key (JWKS) exclusively over IPv4. No A record → no AssumeRoleWithWebIdentity → every CI pipeline that talks to AWS via OIDC breaks.

The plan: less public IPv4

Public IPv4 addresses now cost money on AWS — and every open v4 address is attack surface. So I removed IPv4 from the estate step by step: the CI worker fleet runs IPv6-only (egress via NAT66), registries and package sources run dual-stack, and a small dedicated pool keeps v4 only for the jobs that demonstrably need it.

The last piece: put the GitLab box, which serves both git. and registry., on inbound IPv6-only. The box itself has long been IPv6-capable — global address, default v6 route, working v6 egress, curl -6 returns 200. Should have been a one-liner: drop the A records, close v4 in the security group, done — it wasn't, twice over.

Blocker 1: the pod network was single-stack

First attempt, rolled back immediately. Production k3s ran single-stack IPv4 — without --cluster-cidr/--service-cidr flags, so with the defaults, and pods only got an IPv4 address. ArgoCD's repo-server pod simply couldn't reach an AAAA-only git (network is unreachable) — every GitOps sync stalled.

The crux: the service CIDR is immutable after cluster init. Retrofitting dual-stack means rebuilding the cluster. I did that as a blue/green rebuild — new node, dual-stack from the start, data moved over, DNS switched, old node left standing as a rollback. A nice side effect: a real disaster-recovery drill under controlled conditions, downtime around 15 minutes.

Now the pods had IPv6, ArgoCD synced over v6 — blocker 1 done. So git went back to IPv6-only, this time it had to work.

The twist: GitLab is the OIDC provider

Minutes after the switch, every tofu apply pipeline turned red:

Error: failed to retrieve credentials … operation error STS:
AssumeRoleWithWebIdentity … InvalidIdentityToken: Couldn't retrieve
verification key from your identity provider

And not just Terraform — the GitLab patch automation and container signing too. Everything that talked to AWS via OIDC was dead.

The reason, once you see it, is obvious: CI authenticates to AWS keylessly via GitLab as the OIDC provider. The flow:

1. The CI job gets a signed OIDC token from GitLab.
2. It calls sts:AssumeRoleWithWebIdentity with that token.
3. AWS STS validates the signature — and to do so, fetches the provider's public key from its JWKS endpoint: git.…/oauth/discovery/keys.

Step 3 is the catch. That fetch comes from AWS' infrastructure outbound — over IPv4. Once git. has no A record left, AWS can't load the key. The token is technically fine; AWS just can't verify it. Result: InvalidIdentityToken.

The nasty part: AWS caches the key for a while. The apply run right after the switch still went green — the next one, after the cache expired, didn't. Nine minutes apart. If you only see the first one, you'd think the migration worked.

How to pin it down

Two commands are enough:

# Does the name still resolve on v4?
dig +short A git.example.eu          # → empty

# Is the JWKS endpoint reachable over v4? (what STS does)
curl -4 git.example.eu/oauth/discovery/keys   # → 000 (fail)
curl -6 git.example.eu/oauth/discovery/keys   # → 200 (ok)

v6 answers, v4 doesn't, AWS can only do v4 — diagnosis done. The fix was the revert (A records and v4 ingress back), and since CI can't fix itself — the fix is an apply — I had to run it once outside the pipeline.

The lesson

It's the same lesson as blocker 1, just with a consumer that's easy to miss:

Before you drop an A record, verify for every consumer that it can reach the name over IPv6 — including external SaaS like AWS STS, not just the obvious clients on your own network.

“The host is IPv6-capable” isn't enough. What matters is whether everyone who talks to it can reach it over IPv6. An OIDC provider is, by definition, something validated from the outside — and that validator isn't yours.

The way out

git can go inbound-IPv6-only — but only once no external AWS caller needs to reach it over IPv4 anymore. Concretely: stop having CI jobs assume the apply role via GitLab OIDC, and instead use the CI runner's EC2 instance role — a caller inside AWS, with no external JWKS fetch at all. Until then, git stays dual-stack. Roughly €3.60 a month for an Elastic IP isn't a reason to break half of CI.

The dual-stack cluster was worth the effort anyway — just for a different reason than planned.

Conclusion

Two blockers, one lesson: “IPv6-capable” is a property of the host, not of the migration. What matters is whether every consumer — the obvious one on your own network as much as the easy-to-miss external SaaS validator — can reach the name over IPv6 before the A record goes. The blue/green cluster rebuild was the right call and delivered a real DR drill for free. git stays dual-stack for now, until CI assumes its AWS role via the instance role instead of GitLab OIDC — then the external JWKS fetch goes away, and IPv6-only becomes possible again.