The TYPO3 backend is off until someone comes

01 — A backend nobody needs at night

Since the split, the TYPO3 backend has a Deployment of its own. It serves the editors' host: the backend itself and the MCP endpoint an AI assistant uses to maintain content. During the day it is a busy pod. At night and over the weekend it gets little more than the monitoring's checks.

Still, it ran all the time. For us a backend pod holds about 310 MiB of memory, and a running pod is also attack surface nobody needs. Across all tenants that adds up to more than a gigabyte doing nothing but waiting for hours.

Going to zero only pays off if starting up is fast enough, though. Until the end of September a TYPO3 pod took between 46 and 126 seconds to become ready here. Only since the code arrives as an image volume is it twelve. That is what makes a backend at zero bearable at all.

02 — Kubernetes 1.37 can go to zero, but only on someone else's metric

The feature is called HPAScaleToZero. In Kubernetes 1.37 it is beta and on by default, on our k3s cluster without a single flag. An HPA with minReplicas: 0 is simply accepted.

The catch shows only afterwards. An HPA scaling on CPU measures its pods' CPU. At zero pods there is nothing to measure, and it would never come back up. So it needs a metric that exists while no pod runs. Kubernetes calls that an external metric.

A second decision follows from it. Only AverageValue works as target type. For the other target types the HPA divides by the current replica count, and at zero it has nothing to divide by.

minReplicas: 0
maxReplicas: 1
metrics:
  - type: External
    external:
      metric:
        name: tenant_http_requests_in_flight
        selector:
          matchExpressions:
            - { key: wake_host, operator: In, values: [cms.example.org] }
      target:
        type: AverageValue
        averageValue: "1m"
behavior:
  scaleDown:
    stabilizationWindowSeconds: 1800

03 — The metric comes from the edge proxy

In front of every tenant runs an edge proxy, a Caddy, and it always runs. It counts requests per host, and Prometheus collects those figures for the backend host every 15 seconds. prometheus-adapter translates between Prometheus and Kubernetes and turns them into external metrics the HPA can read. So something that always runs measures whether anyone is looking for the switched-off backend.

The start must not depend on the request rate, the number of finished requests per second. A request the proxy holds for a switched-off backend is not finished yet and does not show up in the counter of completed requests. It would fail to trigger exactly what it is waiting for. So requests in flight trigger the start, the ones running or waiting right now. Prometheus sees them at its next scrape, however short they are.

Whether the backend keeps running afterwards, on the other hand, follows the rate. That rate also contains the monitoring that checks the host. So the adapter subtracts a floor of 0.15 requests per second, just above what the checks alone produce. Only above it does the backend count as used.

One trap sat in the label. Caddy records the host as the client sent it, sometimes with a port. A label selector's value may not contain a colon, though, and that is exactly where the first selector failed. Since then Prometheus adds a label wake_host of its own at scrape time, without the port.

04 — The first request waits instead of failing

Scale-to-zero means that without load not a single pod runs. For the first call after that, the most common answer is a waiting page: the proxy answers 503 and the browser reloads after a few seconds. For an editor that would just about do. For an AI assistant writing to the MCP endpoint by POST, it is an error it does not repeat.

So the edge proxy holds the request. If it finds no pod for the backend host, it retries every second for up to 90 seconds and forwards the request as soon as a pod answers. Even a POST survives the start this way.

reverse_proxy app-backend:443 {
    dynamic a {
        name app-backend-headless
        port 8443
        refresh 10s
    }
    lb_try_duration 90s
    lb_try_interval 1s
}

That leaves the monitoring. A check that calls the backend every minute keeps it running for good. So the edge proxy answers the backend host's check itself. It tests TLS, the allow list and routing, but starts nothing. Whether a switched-off backend would start is watched by an alert of its own instead.

05 — What the measurement shows

Before any tenant switched over, the whole control loop ran with a test HPA. After the load stopped, it went to zero within 26 seconds, in the test with a short idle window. When load returned, the HPA started a pod after 17 seconds. At zero the HPA reported ScaledToZero=True and ScalingActive=True: it was at zero and kept listening.

In operation it waits longer. Only after 30 minutes without an editor's request does the backend go to zero. An editor fetching a coffee notices nothing. Whoever comes in first in the morning waits once, and longer: at the first real start, on the evening of 2 October, the answer came after 35 seconds. That includes Prometheus's next scrape, one HPA cycle, the pod start and forwarding the held request.

My own site's backend went first, on 2 October. The other tenants were to follow only once every start there had stayed under 60 seconds for a week. By now the backends of all tenants are switched off this way. A single switch in the tenant file takes it back for any one tenant.

06 — Where it does not fit

Not every pod should be switched off. The consumer that works off mail and image sizes writes its figures itself. At zero nobody would write them, and nothing could start it. Even with another source, a password-reset mail would then wait up to two minutes, to save about 200 MiB. So the consumer keeps running.

Nor does the backend scale upwards with this design. The start metric is chosen so that any request means exactly one pod. Allow two pods, and a single request would already start the second. At a demand below 0.05 CPU cores the backend does not need one anyway.

And a switched-off pod must not raise an alert, while a broken one must. So the existing alert for an app without a pod watches only the website. Two new alerts watch the backend: BackendWakeStuck, when the HPA wants a pod and none answers, and BackendHPAInactive, when the HPA can no longer read its metric. Without that second alert, a switched-off backend would never start again, and nobody would notice before the next login.

Conclusion

Scale-to-zero with built-in means has arrived in Kubernetes 1.37, but the line minReplicas: 0 is the smallest part of it. The work lies in the question of who measures while no pod runs, and who holds the first request until one runs again. For us both are the edge proxy that stands in front of every tenant anyway.

Whether it pays off is decided by the pod start. With a twelve-second pod start, editors wait half a minute once in the morning and never again that day. At two minutes it would have been an imposition, and the memory would not have made up for it.