Application code as an image volume instead of oras-init

01 — Where it started

All applications run on one shared, hardened runtime image with PHP and FrankenPHP. Each application's code comes separately as a signed OCI artifact from the registry: a tar with vendor/, configuration and the public directory, around 500 MB for a TYPO3 installation. The pipeline builds it for TYPO3, Sylius and Laravel along the same lines; only the list of paths in the tar differs.

Inside the pod an init container did the delivery. It pulled the artifact with oras pull, unpacked it into an emptyDir and created the directories the framework writes to at runtime. That works reliably, but it repeats the same work on every pod start: for the pilot tenant, 13 of 53 seconds of cold start went to the download. On top of that came an extra tool in the pod, a retry loop and a set of registry credentials.

The cost showed most clearly in the scheduler, a CronJob that starts every five minutes. So that it would not pull half a gigabyte on every tick, it got a code cache on a ReadWriteOnce volume plus a marker file per release. That saved the download, but it tied the scheduler to whichever node the volume had first been created on.

02 — Image volumes and the first attempt

An image volume mounts the content of an OCI image read-only into a pod without turning it into a container. The feature arrived as alpha in Kubernetes 1.31, has been on by default since 1.35 and stable since 1.36. On the runtime side containerd supports it from version 2.1, and lean distributions such as k3s now ship a suitable version.

volumes:
  - name: code
    image:
      reference: registry.example.com/shop/sources:2.38.15@sha256:…
      pullPolicy: IfNotPresent

The obvious first step was to point it at the existing artifact. The kubelet pulled it without complaint, 509 MB in just under seven seconds, and the pod started. The directory, however, was empty, and neither the events nor the kubelet log said a word about it.

The cause is in the manifest. An artifact that follows the OCI guidelines for non-images carries its own artifactType, an empty config and layers with their own media type. containerd only mounts what looks like an image, that is an image config whose rootfs.diff_ids describe the layers. Anything else is pulled, stored and then quietly ignored.

{
  "artifactType": "application/vnd.example.sources.v1",
  "config": { "mediaType": "application/vnd.oci.empty.v1+json" },
  "layers": [ { "mediaType": "application/vnd.example.sources.tar" } ]
}

The same test turned up a second hurdle. The kubelet pulls the volume itself, using the pod's imagePullSecrets or the node's credentials. The node's registry account is deliberately not allowed to read the tenants' source projects, so the first pull failed with insufficient_scope. Since then the tenant's deploy token, which the init container used before, sits on the pod as a pull secret.

03 — Same layer, different manifest

The answer was in the layer itself. The tar is uncompressed and contains only relative paths, which already makes it a valid image layer of type application/vnd.oci.image.layer.v1.tar. Because it is uncompressed, its diff_id, the hash of the unpacked content, equals the layer digest. All that was missing was an image config saying so, and a manifest pointing at both.

For the test I uploaded exactly those two files, a few hundred bytes together. The new manifest refers to the code layer that was already in the registry, so not a single byte of code was transferred again. With that manifest the kubelet mounted the code at /app, and the Composer autoloader found every class.

In the pipeline, only the push changes. Instead of an artifact type, oras push passes an image config, and the layer gets the media type of an image:

printf '{"architecture":"arm64","os":"linux","config":{},
  "rootfs":{"type":"layers","diff_ids":["sha256:%s"]}}' \
  "$(sha256sum /tmp/app.tar | cut -d' ' -f1)" > /tmp/config.json

oras push \
  --config /tmp/config.json:application/vnd.oci.image.config.v1+json \
  "$REGISTRY/sources:$VERSION" \
  /tmp/app.tar:application/vnd.oci.image.layer.v1.tar

The change lives in the shared pipeline component, so it applies to TYPO3, Sylius and Laravel at once. It still carried no risk, because the layer keeps its org.opencontainers.image.title annotation. That is how oras pull knows where to write the file, so the old init container unpacks the new format exactly like the old one. The pipeline could switch while every tenant still ran on ORAS, and each tenant moves over on its own afterwards.

04 — What has to stay writable

An image volume is always read-only. For the app containers that was nothing new: they already ran with a read-only /app, and for each framework it had been measured where it writes at runtime. TYPO3 needs var/ and public/typo3temp/, plus public/_assets/, which the setup publishes. Sylius writes under var/ and publishes its bundle assets to public/bundles/; Laravel needs storage/ and bootstrap/cache/. Each of these directories gets its own emptyDir, mounted on top of the code.

The first pod with this layout still did not start:

error mounting ".../volume-subpaths/..." to rootfs at "/app/var":
  make mountpoint "/app/var": mkdirat .../rootfs/app/var: read-only file system

The kubelet sorts mounts by path length, so /app is mounted before /app/var. If the target directory is missing, runc creates it, and on a read-only volume that fails. Until now the init container had created these directories while unpacking; they were never part of the tar. Now the pipeline appends them to the tar as empty directories, with a list per framework that also covers the mount points for object storage such as public/fileadmin/ or public/media/.

Two more safeguards sit in the chart. The image volume has a name of its own instead of replacing the previous emptyDir of the same name. Argo CD applies manifests with server-side apply, and a field still owned by another field manager can survive, leaving a volume that is both emptyDir and image, which the API rejects. On top of that, a small init container checks that the autoloader is present in the volume. A release in the old format would otherwise start quietly with an empty /app; this way it stops with a message that names the reason.

05 — The pilot and what it found

TYPO3 went first, because the pattern started there and its writable paths were the best measured. The pilot was my own website. A switch per tenant chooses between the old path through ORAS and the image volume, and the rollout only takes the old pod out of service once the new one is ready. Mistakes therefore only stop the rollout, and visitors never notice. That is exactly what happened on the first attempt.

The new pod never became ready. TYPO3 aborted at boot with Could not create directory, right in the middle of vendor/. The Content Blocks extension links every block's assets into Resources/Public/ContentBlocks/ of its host extension the first time it builds its registry, and the host extension simply lives under vendor/. I had measured writes on the app container only. That container had run with a read-only /app for a while and never failed, because the setup init container had created those links on its writable /app first. The lesson: measure writes per pod, not per container.

The fix belongs in the build. The pipeline runs content-blocks:assets:publish before packing. It needs no database, the links are relative and therefore correct under /app as well, and at runtime the publisher finds everything in place and writes nothing. The rollback came with a second trap: in image mode the chart had dropped the volume for the scheduler's code cache, Argo had deleted it, and on the way back it hung in Terminating because a waiting scheduler pod still pointed at it.

The second attempt went cleanly. The download at start is gone, the init container only takes a look into the volume instead of spending 13 to 14 seconds, and a pod is ready after 42 seconds instead of 54. During the switch, 258 requests against the website, the second domain and the backend returned not a single error. The other TYPO3 tenants followed the same afternoon after a read-only boot test of their own images, with a little over 2000 requests and no error related to the switch.

One tenant still had something to say. Its setup took almost five minutes, because extension:setup wanted to write two new extension defaults into config/system/settings.php and the script kept retrying until it gave up. Both values now sit in the file in the repository, so the setup has nothing left to write there.

Sylius and Laravel follow in their own steps. Their pipeline already publishes the code in the new format, but the Sylius setup still writes JWT keys and a preload bridge straight into /app. Those files need a writable home before the switch is released for Sylius, and until then the chart refuses the combination.

06 — What it achieved

No ORAS runs in the pod any more, and the download at start is gone. The kubelet keeps each release once per node for every pod that needs it, and a new version only fetches the layers that changed. The scheduler no longer needs its code cache, which removes the last ReadWriteOnce volume tying it to a particular node.

The separation of runtime and code stays fully intact. The runtime image is still updated independently of the application releases. The cosign signature check runs before start against the same digest the volume is pinned to, and because published tags are immutable, pullPolicy: IfNotPresent is safe here.

An obvious next step concerns the layers themselves. Today the whole code is a single layer, so every release ships it in full. Separate layers for vendor/ and the application code proper would keep the dependencies on the node across releases. The bigger chunk of the cold start is now the framework setup that runs on every pod start. With the code in an image volume, an Argo hook has access to the code for the first time, so the setup can run once per release instead of in every pod. A first attempt brought a pod down to 23 seconds and still had to be rolled back, because the setup also created the pod's public assets on the side. How that ends is the subject of a separate field note, together with caches pre-warmed in the build.

Conclusion

An image volume replaces the init container that downloads and unpacks code on every start, for any PHP application that ships its code separately from the runtime. All it takes is putting the existing layer under an image manifest and shipping the mount points of the writable directories with it. That was the easy part.

The real lesson is in the first attempt. The kubelet mounts an artifact that does not look like an image without complaint and hands over an empty directory. A pod that starts proves little during this kind of switch. What counts is a look into the volume, and the best place for that check is an init container on every start.