TYPO3 on Kubernetes under IT-Grundschutz, part 9: Autoscaling and probes

01 — Horizontal Pod Autoscaler: what to scale on

The Horizontal Pod Autoscaler (HPA) adjusts the number of pods to a metric. For TYPO3, CPU is usually the right signal. In worker mode, for example with FrankenPHP, every pod keeps a fixed number of PHP workers in memory. Memory use therefore barely rises with load, CPU does. With worker models, memory as a scaling metric tends to lead to wrong decisions.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: typo3
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: typo3
  minReplicas: 2
  maxReplicas: 6
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300

The target utilisation refers to the CPU requests. Without properly set requests, the HPA scales into the void. Requests and limits are also mandatory for security: a pod without limits can saturate an entire node. minReplicas: 2 is the floor for a service that should survive maintenance without downtime. The stabilisation window stops the HPA from scaling down right after a short spike.

One signal does not fit every host of a tenant, though. The editorial backend has a completely different load profile from the website: a few heavy requests during the day, none at night. If both share a pod, they slow each other down, and the HPA cannot tell their load apart. With a Deployment of its own for the backend, each side scales with its own load, the backend up as well as down to zero. How the split works and why both sides answered faster afterwards is in the field note on the backend in its own pod.

02 — Rolling updates and PodDisruptionBudget

A deployment replaces pods step by step during an update. With maxSurge: 1 and maxUnavailable: 0, a new pod starts first. Only once it is ready does an old one go. Capacity never drops during the update.

The silent rollout

This setting has a flip side. If the new pod never becomes ready, the rollout simply stalls. The old pods keep running, the website stays reachable, and nothing reports a problem. With progressDeadlineSeconds, Kubernetes marks such a rollout as failed after the configured time. An alert on exactly that condition belongs to every deployment. Otherwise a platform quietly runs on an old version for days, and the next security update never arrives.

Updates are not the only interruptions, though. Nodes get maintained, drained and rebooted. That is what the PodDisruptionBudget (PDB) is for. It defines how many pods must at least keep running during a planned interruption.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: typo3
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: typo3

A PDB only governs voluntary interruptions. If a node fails and all replicas run on it, they are all gone anyway. That is why topologySpreadConstraints or an anti-affinity deliberately spread the pods across nodes and, where available, across availability zones.

03 — Probes that find real failures

Kubernetes has three checks. The startup probe gives a pod time to boot. The readiness probe decides whether a pod receives requests. The liveness probe restarts a pod that hangs.

The most common weakness is probes that only check whether a port is open. The web server answers, PHP only serves errors, and Kubernetes considers the pod healthy. What works in practice is a readiness probe that calls the real application path, directly at the pod, with the site's Host header and, where the pod itself speaks TLS, over HTTPS. A response with status 5xx counts as a failure.

That has a flip side worth knowing. If the probe renders the homepage, a broken homepage blocks every deployment. New pods never become ready and the rollout stalls. For readiness, that is intended during a rollout, because a pod with a broken homepage should not get traffic. In normal operation it has a second flip side: a content error hits every pod at once. All become unready, and the whole site returns 503. The readiness probe is therefore better aimed at a lean path that goes through PHP, database and cache but does not depend on the homepage content. For liveness it is dangerous: a content error would send every pod into restart loops. So liveness only checks that the process is alive. Readiness checks the application.

Graceful shutdown belongs here too. A preStop hook with a short pause gives the ingress time to take the pod out of rotation before it stops accepting requests. Requests in flight still complete.

The hardened image from part 2 has neither a shell nor sleep. An exec hook running sleep 5 fails there silently. Since Kubernetes 1.30 the hook can wait by itself, without any program in the image:

lifecycle:
  preStop:
    sleep:
      seconds: 5

The pause and the completion of in-flight requests must fit together into the pod's terminationGracePeriodSeconds. Otherwise Kubernetes kills the container in the middle of a request.

04 — More nodes: autoscaler, spot, and what prevents several nodes

The HPA creates pods. When they no longer fit onto the existing nodes, a cluster autoscaler has to start new ones. Cheap spot nodes are a good fit for load peaks. The preconditions are a stateless application and a spread in which not all replicas sit on spot nodes that can disappear together.

In practice, what most often ties a TYPO3 cluster to one node is not TYPO3 itself but storage. ReadWriteOnce volumes are bound to one node, local storage even to one machine. If the database lives on such a volume, its pod cannot simply move. The ways out are network storage that moves between nodes of a zone, a database replicated across nodes, or a managed database service. More on that in part 10 and part 12.

Mapping to IT-Grundschutz

• SYS.1.6.A15 Limiting resources per container: requests and limits, also the basis for the HPA.
• SYS.1.6.A25 High availability of containerised applications: several replicas, PDB, spread across nodes.
• APP.4.4.A11 Monitoring of containers: probes that test the real application path.
• APP.4.4.A19 High availability of Kubernetes: pods and nodes spread so that one interruption does not stop the service.
• APP.4.4.A21 Regular restart of pods: rolling updates without capacity loss make restarts low-risk.

Conclusion

Autoscaling is switched on quickly. TYPO3 only becomes available once the pieces fit together. The HPA needs honest requests. The PDB only works when the pods sit on different nodes. And the probes must test the path visitors take. After that, the bottleneck is rarely TYPO3. Usually it is storage.

That is exactly what the next part is about: database, backup and restore. Back to part 8: making TYPO3 stateless.