TYPO3 on Kubernetes under IT-Grundschutz, part 7: Secrets without Git

01 — Why secrets do not belong in Git, not even encrypted

GitOps promises that everything running in the cluster is in the repository. For secrets, that is a trap. A plain-text password in the repository is obviously wrong. Less obvious is the encrypted secret, for example with SOPS. It works, but it only moves the problem. The decryption key has to live somewhere, often in the cluster itself. Whoever has it reads every secret in the entire Git history, including values rotated long ago.

In practice there is a second drawback. With SOPS, rotation means re-encrypt, commit, merge, roll out. That takes time. Which is exactly why rotation rarely happens.

The more robust split looks like this: Git only records which secret a tenant needs and where it lives. The value itself lives in a secret store outside the cluster, such as AWS Secrets Manager, HashiCorp Vault or a comparable service. The repository stays complete and reviewable, but contains nothing that could be misused.

02 — External Secrets Operator and workload identity

The External Secrets Operator (ESO) closes the gap. It reads an ExternalSecret from Git, fetches the values from the store and creates a regular Kubernetes secret from them. When the value in the store changes, ESO pulls it in at the configured interval.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: typo3-db
  namespace: tenant-a
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: tenant-a-store
    kind: SecretStore
  target:
    name: typo3-db
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: tenant-a/typo3/db
        property: password

What matters is how ESO authenticates against the store. A static access key in the cluster would just be another secret in the cluster. Workload identity is better: the tenant's service account receives a cloud role via OIDC, on AWS this is called IRSA. That role may read exactly the secrets under its tenant's prefix and nothing else. A compromised namespace therefore cannot reach another tenant's credentials.

Hence one SecretStore per namespace instead of a single ClusterSecretStore. A central store with broad rights would be more convenient, but it would undo the tenant separation from part 6 at precisely this point.

03 — Files instead of environment variables

A Kubernetes secret can reach the container as an environment variable or as a file. For credentials, files are the better choice. Environment variables are inherited by every child process. They end up in crash reports, debug output and the process listing. A mounted file under /run/secrets is only read by whoever opens it deliberately. File permissions can restrict it to one user. And Kubernetes updates the file on rotation, as long as it is not mounted via subPath.

TYPO3 reads its configuration from settings.php and additional.php, in many setups from environment variables. The bridge is a small resolver. The configuration refers to a name, and the resolver reads the matching file during bootstrap. The configuration in the repository stays free of values, and the application still sees the right credentials.

// config/system/additional.php
$read = static function (string $name): string {
    $file = '/run/secrets/' . $name;
    if (!is_readable($file)) {
        throw new \RuntimeException('Missing secret: ' . $name);
    }
    return trim((string) file_get_contents($file));
};

$GLOBALS['TYPO3_CONF_VARS']['DB']['Connections']['Default']['password'] = $read('db-password');

It is important that a missing secret fails loudly. A resolver that silently returns an empty string for a missing file produces errors that surface much later, and somewhere else entirely.

Worker mode: the file is new, the value is not yet

In worker mode, TYPO3 reads additional.php once when the worker starts. After that it keeps the values in memory. An updated file therefore only takes effect once the worker restarts. For rotation, that means: after the change in the store, the pods restart in a rolling fashion, for example triggered by a reloader that reacts to the changed secret. The per-worker request cap from part 2 is not enough for this. It says nothing about when the last worker reads the new value.

04 — Rotation, alerts and encryption inside the cluster

With an external store, rotation becomes a single-place operation. The value changes in the store, ESO pulls it in, the file in the pod updates, and a rolling restart brings the value into the application.

Database passwords need a transition so that no connection drops. Unlike MySQL, MariaDB has no dual-password feature (RETAIN CURRENT PASSWORD). Two users in alternation have therefore proven themselves. The one currently unused gets a new password, the application switches over to it in a rolling fashion, and then it is the other one's turn. That way valid credentials exist at every point in time.

An alert on failed synchronisation matters just as much. An ExternalSecret with status SecretSyncedError otherwise only surfaces when a pod restarts and its secret is missing. ESO exposes metrics for this that translate directly into an alert rule.

Inside the cluster, secrets live in etcd. They belong there encrypted. k3s ships this as secrets-encryption; other distributions use an EncryptionConfiguration for the API server. Finally, you need an emergency path. A break-glass procedure defines who gets credentials when and how if the store is unreachable. Every use is logged.

How it maps to IT-Grundschutz

• SYS.1.6.A8 Secure storage of credentials for containers: the values live in the store, not in the image and not in Git.
• SYS.1.6.A20 Protection of configuration data: the repository only holds references.
• APP.4.4.A3 Identity and authorisation management for Kubernetes: one role per tenant, least privilege.
• APP.4.4.A8 Protection of configuration files and APP.4.4.A9 Use of Kubernetes service accounts: one service account per tenant, its token only where it is needed.
• CON.1 Crypto concept: encryption in etcd and in the store, documented key management.

Conclusion

Secrets belong neither in the image nor in Git. An external store, one workload identity per tenant, and files instead of environment variables are enough to solve this cleanly. The real gain shows during rotation: it turns from a project into a routine step.

The next part is about making TYPO3 stateless. Sessions, fileadmin, caches and locks move out of the pod. Back to part 6: network and tenant separation.