Getting tofu init off GitHub: installing OpenTofu providers from my own OCI registry

01 — The setup: an IPv6-only fleet, and the one holdout

My GitLab CI workers are ephemeral AWS spot instances, and the general fleet is deliberately IPv6-only — the instances come up with no public IPv4 address at all and egress over IPv6 only. That's a cost lever and an attack-surface lever at once: the vast majority of ephemeral worker-hours carry no public v4 address, and the network deps that matter (docker.io, quay.io, my own registry, the language CDNs) are all dual-stack.

A handful of jobs genuinely still need IPv4 — anything that talks to GitHub, mostly — so those carry a special tag and land on a small IPv4 pool. The whole game is keeping that list short, and watching what sneaks back onto it. For a long time koh-infra — the OpenTofu repo that manages the AWS account itself — was on that list. The reason turned out to be dumber than I expected.

02 — The problem: “install from OpenTofu” means “download from GitHub”

When you write source = "hashicorp/aws", OpenTofu resolves that through its public registry at registry.opentofu.org. But that registry doesn't host the provider binaries — it hands back a redirect to the release assets, which live on GitHub. And GitHub, to this day, publishes no AAAA record: it is IPv4-only.

So every tofu init — including the one that runs immediately before apply on the account's own infrastructure — pinned that job to the IPv4 pool and put a third party's release CDN on the critical path of an apply. During a spot-reclaim wave in mid-2026, GitHub's asset endpoint started intermittently returning 504s under the extra load, and a couple of infra pipelines went red not because of anything I'd changed, but because a download three redirects away had timed out.

The actual failure

tofu init → registry.opentofu.org → 307 → objects.githubusercontent.com → 504 Gateway Timeout

Not my code, not my network, not retryable in any way that helped. Squarely on the path between “push to main” and “infrastructure changed.”

Two things were wrong here, and only one of them was IPv4. The other was that my most sensitive automation — the one that can change IAM and networking — depended, at apply time, on the availability of a public release CDN I don't control. Even on the IPv4 pool, that's a dependency I wanted gone.

03 — The options: where should the providers actually come from?

OpenTofu supports a few ways to install providers from somewhere other than the public registry. The two realistic ones for me:

Option B won on a single principle: don't stand up a second thing when the first thing already does the job. I run a container registry. It's dual-stack. My CI already authenticates to it on every pipeline. Serving provider artifacts from it adds no new surface — providers become just another set of tagged artifacts next to the container images. The cost is that providers have to be repackaged into OCI's format, and that's where it gets interesting.

04 — The format: what an OpenTofu provider looks like as an OCI artifact

This is the part that isn't in the quickstart. A provider version isn't a single blob — it's a small two-level manifest tree, because one provider version ships a different binary per OS/arch and OpenTofu has to pick the right one at init time.

• Per platform (linux/amd64, linux/arm64): an artifact of type application/vnd.opentofu.provider-target, whose single layer is the provider zip with media type archive/zip, carrying the OCI platform descriptor.
• Per version: an index of type application/vnd.opentofu.provider that references both per-platform manifests. This index is what gets tagged :6.52.0.

At init time, tofu pulls the version index, matches the runner's os/arch against the platform fields, and downloads only that one platform's zip. Both of my spot architectures — amd64 and arm64 — resolve from the same tag. I assemble the whole thing in a local OCI layout — push each platform, manifest index create to tie them together — then oras cp the finished tree to the registry in one shot, so only the version index gets tagged.

Building one version (mirror:tofu-providers)

# assemble everything in a LOCAL oci-layout — no registry contact yet
for plat in amd64 arm64; do
  oras push --artifact-type application/vnd.opentofu.provider-target \
    --artifact-platform "linux/${plat}" \
    --oci-layout "tmp-layout:linux_${plat}" "${zip}:archive/zip"
done

# the version index, built locally over the two platform manifests
oras manifest index create --artifact-type application/vnd.opentofu.provider \
  --oci-layout "tmp-layout:${ver}" linux_amd64 linux_arm64

# ship the finished tree — only the index gets a tag; platforms go by digest
oras cp --from-oci-layout "tmp-layout:${ver}" "${dest}:${ver}"

Assembling in a local layout and oras cp-ing the whole tree means only the version index is tagged — the per-platform manifests go up by digest, so the registry keeps exactly one clean tag per version.

05 — The producer: a job whose whole purpose is to touch GitHub once

The repackaging lives in my mirroring repo as a job called mirror:tofu-providers. Its input is a tiny, backend-less Tofu module that exists only to declare which providers and versions to mirror — Renovate keeps the pins current:

The mirrored set (tofu-providers/versions.tf)

terraform {
  required_providers {
    aws  = { source = "hashicorp/aws",  version = "6.52.0" }
    http = { source = "hashicorp/http", version = "3.6.0" }
    tls  = { source = "hashicorp/tls",  version = "4.3.0" }
  }
}

This set must stay a superset of what the consumer locks. More on that coupling below.

The job runs tofu providers mirror -platform=linux_amd64 -platform=linux_arm64 to pull every declared provider for both arches into a filesystem mirror, then walks the resulting zips and runs the oras assembly from the previous section. It is the only place GitHub gets touched — and it runs rarely, gated on changes to versions.tf or a schedule, not on every pipeline. One job pays the IPv4 tax so that nothing else has to.

06 — The consumer: two blocks, and a deliberate dead end

On the koh-infra side, the whole redirection is one config file. oci_mirror maps every registry.opentofu.org source onto my registry path; direct { exclude } then removes the GitHub install method for those same sources.

Consumer config (koh-infra/.tofurc)

provider_installation {
  oci_mirror {
    repository_template = "registry.ole-hartwig.eu/devops/ci-mirrors/${namespace}/${type}"
    include             = ["registry.opentofu.org/*/*"]
  }
  direct {
    exclude = ["registry.opentofu.org/*/*"]   # no github fallback — on purpose
  }
}

Active only when CI sets TF_CLI_CONFIG_FILE. Local dev without that env var uses the normal direct install, so nothing changes on my laptop.

That exclude line is the important one, and it's easy to read as a mistake. Why cut off the fallback? Because a silent fallback to GitHub would quietly reintroduce exactly the IPv4-and-flakiness I just removed — invisibly, working fine right up until the next GitHub wobble, at which point I'd be debugging the same outage I thought I'd fixed. I'd rather fail loud and early: if a locked version isn't in the mirror, init stops and names the missing artifact. That turns a rare mystery outage into a boring, immediate, obvious error.

The trade for that clarity is a hard coupling I have to keep honest:

koh-infra/.terraform.lock.hcl  ⊆  ci-mirrors/tofu-providers/versions.tf

Why it holds in practice

Renovate updates both repos. If they ever drift, the symptom is an immediate red tofu init naming the missing registry.opentofu.org/<ns>/<type>:<ver> — cheap to diagnose, cheap to fix: add the version, let the mirror job run, retry. A loud coupling beats a silent fallback.

07 — The gotcha: order of operations on first activation

The no-fallback coupling has one sharp edge, and it only bites once. Because the consumer has no way to reach GitHub anymore, the mirror has to exist before the consumer trusts it. So the one-time rollout order is strict:

1. Merge the producer first and let mirror:tofu-providers run once, so the registry is actually populated.
2. Make sure the consumer's CI job token is allowed to pull from the mirror registry.
3. Then merge the consumer's .tofurc and drop its IPv4 tag.

Merge the consumer first and every one of its pipelines goes red — mirror empty, or token not yet allowed. After that first activation, order stops mattering; only the version-set invariant does.

08 — The result: one fewer thing that can 504

With the mirror in place, tofu init, fmt, and validate pull entirely from my dual-stack registry — so I dropped the IPv4 tag off those jobs and they moved back onto the IPv6 fleet where the rest of the fleet lives. GitHub is off the provider-install path completely.

To be precise about what this did and didn't fix: the AWS apply itself can still reach non-dual-stack AWS endpoints, so koh-infra isn't fully off IPv4 yet — that's a separate problem for a separate day. But the provider-install half is done, and it was the flaky half. The next time registry.opentofu.org has a bad afternoon, my infra pipeline won't even notice.

09 — What fought back: three ways the design was wrong before it was right

The idea was simple; the rollout was not. Each of the next three things looked like the mirror was broken and turned out to be a gap between what I assumed a tool did and what it actually does. None of them were GitHub flaking — every time, the download and repackage ran clean and the failure was mine.

A flag from the future

First push died with oras push: unknown flag: --artifact-platform. I'd pinned oras 1.2.3 — a plausible-looking “current stable” I picked from memory — but the flag that stamps the platform onto an artifact only landed in 1.3.0. I was pinning a version and using a feature from a newer one in the same job. The lesson isn't “use the newest version”; it's verify the flags you rely on exist in the version you pin, not just that the version is real.

Two segments deep, no more

Then every push 401-denied — even a direct one, even though the same job token pushed my other mirror images fine. The path was ci-mirrors/tofu-providers/hashicorp/aws. A throwaway probe job settled it in one run: a GitLab CI job token can push registry images at most two path segments below the project. probe/a/b worked; probe/a/b/c got 401. My grouping prefix made it one level too deep. Dropping it to ci-mirrors/hashicorp/aws — which is the canonical OpenTofu shape anyway — fixed it. The depth had nothing to do with permissions to create repos; the token could do that. It just can't go three deep.

The credentials tofu wouldn't read

The mirror was full, the path was right, and tofu init still got 403. I'd handed tofu a ~/.docker/config.json with a username/password — the way you'd write it for the Docker CLI. But OpenTofu reads those files in the containers-auth.json format, which honours only the base64 auth field. It found nothing usable, fell back to anonymous, and my internal registry politely refused. The tell was maddening: the runner pulled tofu's own job image from that exact registry without a hiccup — because image pulls use the job-payload credentials, a different door. The fix was to stop hand-rolling the file and give tofu an explicit oci_credentials block, where username/password is supported and which wins over ambient discovery.

The pattern under all three

Every failure was a confident assumption about a tool's behaviour — the flag exists, the path depth is fine, the cred file is read — that only a run could disprove. The cheapest debugging tool here wasn't a smarter guess; it was a 40-second probe job that asked the registry directly instead of asking me.

The broader lesson is one I keep relearning: the cheapest supply-chain dependency to remove is the one you can serve yourself from infrastructure you already run. I wasn't missing a tool — I was missing the realization that a container registry and a provider registry are, at the artifact level, the same thing.

Conclusion

No retry loop, no longer timeout — just remove the dependency. OpenTofu's OCI provider format is a small, well-specified manifest tree once you've assembled one by hand; the actual work wasn't in the format, it was in three quiet assumptions about tool behaviour that only a real run disproved. The broader lesson: the cheapest supply-chain dependency to remove is the one you can serve yourself from infrastructure you already run — a container registry and a provider registry are, at the artifact level, the same thing.