Kai Ole Hartwig
8 min read

TYPO3 media without a mount: from the FUSE volume to a FAL driver on S3

Part 8 of the series put fileadmin into object storage and took the convenient route: a CSI driver mounts the S3 bucket into the pod as a filesystem, and TYPO3 notices nothing. How that runs with three cache layers is in fileadmin on S3 with caching. It ran for a year. This post describes why the mount had to go anyway, how TYPO3 now talks to S3 itself, and what the switch delivered when measured. A supplement to the series “TYPO3 on Kubernetes under IT-Grundschutz”.

TL;DR

TL;DR: The mounted bucket was a privileged pod with the node's role, a single point of failure for every tenant and a volume that pinned every TYPO3 pod down. None of the existing S3 extensions runs on TYPO3 14, so a first-party FAL driver came into being: identifiers stay object keys, URLs stay /fileadmin/…, the storage is switched in place, every tenant gets its own role. The rollout kept a way back, first one tenant, then all, then the mount fell away. It was correct at once. Faster it was not, and that leads to the next post.

01 — What the mount cost

The CSI driver for S3 runs as a DaemonSet on every node. It needs privileged, the SYS_ADMIN capability, access to /dev/fuse and bidirectional mount propagation. That is not a misconfiguration but the nature of a FUSE filesystem. A fault in this pod is a fault on the node.

It authenticates with the node's role. That role has to read and write every media bucket, because the same process serves every tenant. The tenant isolation from part 6 therefore ends at the storage boundary: whoever has the node has all the media. The same holds for spot nodes, which need the same rights only because of the mount.

Three smaller items come on top. If the DaemonSet fails, every tenant loses its fileadmin at the same time; the driver is one point of failure for all. A file becomes visible in the bucket only when the writer closes it, which with two replicas leads to moments where a file exists and does not. And because the driver enforces a fixed user ID, the directories were world-writable so that TYPO3 could write into them at all.

Each point on its own was bearable. Together they described a privileged process in the request path of every website that may do more than a single website needs.

02 — Why no existing extension

TYPO3 has known the other route for years: a FAL driver talks to the object store directly, and the mount disappears. The two well-known extensions for it stopped at TYPO3 13, and both bring properties that get in the way here: they deliver files under S3 or CDN addresses instead of /fileadmin/…, they cannot exclude foreign prefixes in the same bucket, and they depend on the large AWS SDK.

So a first-party driver came into being, lean and only for TYPO3 14 on PHP 8.5. The decisions that set it apart from the existing ones:

The driver runs as a Composer package in each of the four installations; it is being released as open source.

03 — One identity per tenant

With the driver, the identity can go where it belongs. Every tenant gets its own role in the cloud, trusted by exactly one ServiceAccount and allowed to read and write exactly one bucket. The pod receives it through web identity, with no key in a secret. The same pattern the database backups and the mail delivery already used; media were the last item still hanging on the node role.

A small thing with consequences: a pod has exactly one role. Whoever already had one for mail delivery gets the bucket permission merged into that same role, not a second one. How identities are otherwise issued in the cluster is described in Pod certificates instead of shared secrets.

04 — Rollout with a way back

The chart got a switch with two positions, mount and fal, and a second one that keeps the mount in the pod during the transition. Driver and mount see the same objects, so the way back was always a switch: back to mount, the setup hook puts the storage back on the local driver, done.

The smallest tenant went first. Checks were made live, not from the manifest: driver online under its own role, write, read and delete through the driver in the pod, delivery at the edge with the right cache headers, no Caddy error, no restart, and in CloudTrail the pod's role assumption. Then the other three. After one clean day, fal without a mount became the class default in the profile, and no CMS pod carries a media volume any more.

Three things held up the pilot, all in Caddy. The file matcher in Caddy 2.11 has no option for a foreign filesystem; the Caddy did not start, without logging a line. A filesystem needs root * ., otherwise every path becomes absolute and every answer 400. And cache headers have to be set by path before file_server, because the file matcher never sees the S3 filesystem. Each case was reproducible locally, with the same image against a local S3 test server. Approval first, then the switch applies here too: none of it reached a visitor, because the mount was still there.

05 — Measured: correct, but slower

Every check was green. Nobody had looked at latency. When the question came, the edge proxy's access log answered it: a cache miss on /fileadmin took a median of 81 milliseconds before, 247 with the driver. A 900-byte image went from 4 to 259 milliseconds. The culprit was not the driver but the S3 filesystem in Caddy: three requests to S3 for every file, and nothing kept. The mount had had a disk cache.

The cause, the measurement in the pod and the module that came out of it are in Ops Log 09: Three S3 requests per image. The result up front: hot files now leave the pod in 3 to 5 milliseconds, a cache miss at the edge takes 1 to 12 instead of 70 to 88.

Relation to IT-Grundschutz

06 — What remains

The CSI driver is not gone. The nightly malware scan still mounts every bucket, and the shop class and an exchange folder for point-of-sale systems need a filesystem. As long as the scan mounts every bucket, the node role keeps every media grant. The gain of this rebuild is in the request path, not in the node role; that shrinks only once the scan runs without a mount. That is the next task, with a decision of its own.

Frequently asked questions

What happens on a rollback?+

One switch in the chart. The mount comes back, the setup hook puts the storage on the local driver. Driver and mount see the same objects, nothing has to be migrated.

Do the file URLs stay the same?+

Yes. The Caddy in the pod serves /fileadmin/… straight from the bucket. No link, no sitemap and no search index changes.

Why not keep the mount and cache it better?+

The cache was never the problem. The privileged pod, the node role for every bucket and the single point of failure remain with any cache.

Conclusion

The mount was the right first answer and the wrong second one. It brought fileadmin into object storage without touching TYPO3, and paid for it with a privileged process and one identity for all. The driver turns that around: the application talks to the storage itself, under its own role, and the pod holds on to nothing. What the rebuild cost in latency and how the Caddy module brought it back is in Ops Log 09; how the module got into the runtime image without GitHub, in Ops Log 10. The series starts at part 1; part 8 is the part this post continues.

Next step

Is your TYPO3 still hanging on a volume?

I check which state in your pods is really pinned down and build the route into object storage with a way back: driver, roles, chart switch, measurement.

Review the architecture →

Or email me directly: mail@ole-hartwig.eu

About the author

Photo of Kai Ole Hartwig.

Kai Ole Hartwig

Freelance DevSecOps consultant · OnlyOle Consulting

Programming since 2002 – self-taught, set up my own business with KO-Web in 2012. Over 100 projects, with a focus on security, performance, automation and quality. Today freelance: DevSecOps consulting, training and software development.