caddy-fs-s3cache — serve files from S3 without asking three times per image.

TL;DR — Does this fit my setup?

Use it if

• your website serves images or downloads from an S3 bucket and the file addresses should stay the same
• the servers should run without mounted network drives (FUSE mounts) and without a shared disk, such as pods in Kubernetes
• the same files are requested often; then they almost always come from local disk

Do not use it if

• you want to show folder contents as a listing; the module knows single files, not directories
• a replaced file has to be visible everywhere at once; the module asks S3 again after a minute at the earliest, and the interval is configurable
• there is no local storage at all; a small temporary area is enough

Effort

One line when building Caddy and one block in the configuration. Credentials for S3 come from the usual AWS chain, in Kubernetes from the pod's identity. No password sits in the configuration file.

Why three requests per file, and how the cache saves them

The problem

Caddy's file server treats S3 like a disk. For every file it first asks whether the file exists and how large it is. Then it opens the file. Finally it jumps back to the beginning to deliver it. A disk does all that in microseconds. S3 cannot jump back, so the file is fetched a second time. Three requests, each 30 to 40 milliseconds, for a tiny image. And the next visitor starts from scratch.

The fix

caddy-fs-s3cache remembers what it has already fetched. The first request downloads the file and puts it on local disk. Existence, size, opening and delivery come from there afterwards. After a configurable interval, one minute by default, the module asks S3 whether the file has changed. Unchanged means a short answer without a download.

If S3 is down, Caddy keeps serving what it already has and writes a warning to the log. Only a file that was never fetched reports a server error instead of a misleading “not found”.

Measured in production

Measured on a TYPO3 website in Kubernetes. Same servers, same files, once with the existing module and once with caddy-fs-s3cache. The times are the web server's own response time, without the path to the browser.

For scale: a single S3 access from the pod takes 28 to 42 milliseconds in the same region. Three of them made up the 70 to 100 milliseconds. The cache turns that into the price of a local file.

Serving media from S3 without a FUSE mount? I help with the Caddy build, the Caddyfile and the pod identity in Kubernetes. Get in touch →

Installation and configuration

For operators who build Caddy or FrankenPHP themselves. Two steps.

1. Link the module into the build

xcaddy build --with github.com/ohartwig/caddy-fs-s3cache@v1.0.1

In FrankenPHP the line goes in as a further --with next to github.com/dunglas/frankenphp.

2. Declare the filesystem in the Caddyfile

The first block names the bucket and the cache. The second serves everything under /media/ from it.

{
    filesystem media s3cache {
        bucket my-media-bucket
        region eu-north-1
        cache_dir /var/cache/caddy/media
        cache_max_size 512MiB
        revalidate_after 60s
        negative_ttl 30s
        keepalive 25s
    }
}

example.org {
    handle /media/* {
        route {
            uri strip_prefix /media
            root * .
            fs media
            file_server {
                pass_thru
            }
            respond "Not Found" 404
        }
    }
}

The options

Permissions

The module needs read access to the files in the bucket (s3:GetObject). With keepalive on, it also needs the right to see the bucket (s3:ListBucket). Credentials do not go into the Caddyfile. As with every AWS tool they come from environment variables, the AWS configuration file or, in Kubernetes, the pod's identity.

Common pitfalls

• The cache directory is emptied on start. The module keeps its index in memory, and that is empty after a restart. Old files on disk would be unreachable, so it clears them away. In Kubernetes a temporary volume per pod (emptyDir) is the right place, not a shared drive.
• A replaced file appears with a delay. Whoever swaps a file under the same name sees the new version only after the next check with S3, so at the latest after revalidate_after. Files with immutable names, such as image variants with a checksum in the name, do not have this problem.
• A cache in front does not replace the local one. An HTTP cache in front of the web server only helps when the same address is requested more than once. The first request for every address always reaches the web server. With many rarely requested images, the local cache is exactly where it pays off most.
• Invalid paths never reach S3. Paths with .. or with a leading or trailing slash are rejected before a request is made.