Kai Ole Hartwig
14 min read
Low
By

Nothing Worth Stealing, Part 1: Certificate instead of password

With mTLS on every connection, an application has two factors for signing in to its database and cache: the client certificate and a password. The certificate is short-lived and issued through the kubelet. The password is static and sits as a file in the container. That is exactly where an attacker looks first after finding a flaw in the application.

This how-to shows how to drop the password entirely. Valkey maps each certificate to an ACL user, and MariaDB requires the subject and issuer of the pod certificate. The switch runs in stages in which nothing breaks.

Part 1 of 4 of the series "Nothing Worth Stealing". It extends part 5: mTLS on every connection of the series "TYPO3 on Kubernetes under IT-Grundschutz".

01 — The starting point: two factors, one of them static

The starting point from part 5 of the TYPO3 series: Valkey listens only on its TLS port and requires a client certificate with tls-auth-clients yes. MariaDB enforces TLS with require_secure_transport=ON. On top of that, the application signs in to both with a password.

The password adds less than it seems. Anyone who can run code in the PHP container reads the password file just as the application does. And it is static. It stays valid until someone changes it, which often means for years.

The pod certificate has neither weakness. It is valid for 24 hours, and the kubelet renews it automatically. Its subject carries the pod's identity: CN=<namespace>/<serviceaccount>, for the application for example CN=kunde-x/x-app. Once the database and cache require exactly this subject, the password is redundant.

The prerequisite is one signer per tenant, as described in the field notes Pod certificates instead of shared secrets and The whole network on pod certificates. A certificate from tenant A must never be accepted by the cache of tenant B.

02 — Valkey: one ACL user per certificate

Valkey 9 can use the common name of the client certificate directly as the user name. The setting for this is tls-auth-clients-user:

 

tls-auth-clients yes
tls-auth-clients-user CN

user default off
user kunde-x/x-app            on nopass ~* &* +@all -@admin -@dangerous +flushdb +keys +info
user kunde-x/x-cache-exporter on nopass +info +ping +select +dbsize +config|get +client|list +client|info +client|setname +command|info +commandlog|len +slowlog +latency +memory|stats +cluster|info
user kunde-x/x-cache          on nopass +ping +config|set +acl|log +acl|whoami

 

The application gets everything except administration and dangerous commands. On top come the two dangerous ones a cache backend really needs. The exporter only reads. The reload sidecar sets certificates and may read the ACL log as a check.

Three points matter here:

The command list from actual use

The list of permitted commands should come from how the clients really behave, not from the documentation. Start generously and narrow it down with ACL LOG. In practice, typical gaps show up:

valkey-cli --tls ... ACL LOG 10

 

The command shows which user tried which command. After every stage the log should stay empty.

The pitfall: Valkey does not check the issuer

Valkey takes the CN as soon as the handshake succeeds. Which CA issued the certificate plays no part in the mapping. A certificate with the CN kunde-x/x-app from another CA that the cache also trusts therefore signs in as the application. In testing, exactly that happened with a forged certificate from a transitional CA.

This becomes a problem when the cache's trust bundle contains several CAs. During a move from cert-manager to pod certificates that is the normal case. With a password this was covered. Without one, it would be the only protection.

The rule is therefore: before the password goes, the cache trusts only the current tenant CA. With pod certificates this works through a label selector on the ClusterTrustBundle:

 

- clusterTrustBundle:
    signerName: example.com/tenant-kunde-x
    labelSelector:
      matchLabels: { example.com/trust: current }
    path: ca.crt

 

Afterwards the CA file in the cache pod contains exactly one certificate. You can check that before the next step.

03 — Switching Valkey in three stages

Every stage changes the Valkey configuration and therefore restarts Valkey. A restart empties the cache and drops all connections. In practice this costs a few seconds in which individual requests fail with "went away". Plan each stage for a quiet window.

  1. Create the ACL users, default keeps the password. Clients that still send AUTH with the password land on default and have full rights. Clients that send no password become their certificate user straight away. Nothing can break.
  2. Move clients one by one to no password. First the application, then the exporter and the reload sidecar. After each step, ACL LOG shows whether a command is missing. A PHP client such as phpredis signs in over TLS without AUTH. The common cache libraries treat an empty password as "none".
  3. user default off, remove requirepass. Only now, once no client sends the password any more. After that the password file can disappear from all pods, including the cache pod itself.

04 — MariaDB: requiring subject and issuer

MariaDB can require a specific client certificate for a user:

 

ALTER USER 'x_app'@'%'
  REQUIRE SUBJECT '/CN=kunde-x\\/x-app'
  AND ISSUER '/CN=Mandanten-CA kunde-x 2026';

 

Pitfall 1: the escaped slash

The CN of a pod certificate contains a slash. MariaDB compares the subject in its one-line notation. There, a slash inside a value is escaped as \/. REQUIRE SUBJECT '/CN=kunde-x/x-app' therefore never matches. What you mean is /CN=kunde-x\/x-app. Because the backslash must itself be escaped in an SQL string, the statement contains \\/.

The RFC 2253 notation without a leading slash does not match either. If in doubt: with log_warnings=4, MariaDB writes the subject it actually saw to the log.

For reading it back, --raw helps. Otherwise the client's batch mode doubles the backslash in the output:

 

mariadb --batch --raw --skip-column-names -e \
  "SELECT CONCAT(ssl_type,'|',x509_subject,'|',x509_issuer)
     FROM mysql.user WHERE user='x_app' AND host='%'"

 

Pitfall 2: without ISSUER, every trusted CA counts

As with the cache: a REQUIRE SUBJECT on its own lets through a certificate with the same subject from any CA the server trusts. Only AND ISSUER restricts the sign-in to the tenant CA. In testing, the forged certificate from the transitional CA was rejected with ISSUER and accepted without it.

The issuer usually carries a year or a generation in its name. When the tenant CA changes, the REQUIRE ISSUER has to follow. That belongs in the CA rotation procedure, not in someone's memory.

05 — A dedicated account for the readiness probe

The typical stumbling block in the first step is not the application at all. It is the database's readiness probe. Many images check readiness by signing in over the local Unix socket with the application user and its password. No client certificate travels over the socket.

As soon as the application user requires a certificate, the probe fails. The database drops out of the Service although it is running. That is exactly what happens if you simply set REQUIRE.

There are two clean ways:

CREATE USER IF NOT EXISTS 'mariadb'@'localhost' IDENTIFIED VIA unix_socket;
-- SHOW GRANTS must show exactly this:
-- GRANT USAGE ON *.* TO `mariadb`@`localhost` IDENTIFIED VIA unix_socket

 

unix_socket authenticates via the user ID of the process at the other end of the socket. The probe runs in the database container as its own system user and is therefore exactly that user. Another process with a different uid is rejected, and over TCP the account does not exist at all. The probe then needs no password.

The order matters. Create the account in one deploy and switch the probe to it only in the next. In the same deploy, the restarted database would meet a probe whose account does not exist yet. It would never become ready. A hook that only runs after a successful rollout would never create the account. Restarting the database for the new probe means a brief moment without a database. That too belongs in a quiet window.

06 — Moving MariaDB to "no password" without downtime

Simply removing the password does not work when several pods use the database. Some start with the new configuration, others still run with the old one. MariaDB can, however, chain several authentication methods with OR. That makes the switch gradual:

  1. Certificate in addition to the password. Set REQUIRE SUBJECT … AND ISSUER … and keep the password. This only tightens things. Watch this stage across at least one certificate rotation.
  2. Old password or none.
    ALTER USER 'x_app'@'%'   IDENTIFIED VIA mysql_native_password USING PASSWORD('<old-password>')           OR mysql_native_password USING '';
    An ALTER USER … IDENTIFIED VIA without a REQUIRE clause leaves the existing REQUIRE SUBJECT … AND ISSUER … from stage 1 in place. Read it back after every stage rather than relying on that. Clients now sign in with and without the password, and a wrong password is rejected. The certificate stays mandatory in both cases.
  3. Remove the password file from all pods. Application, workers, scheduler, setup and maintenance jobs, in any order. PDO signs in with null or '' as the password plus the client certificate. Also think of helper scripts that read the password file directly, for a setup lock for instance. They must cope with a missing file.
  4. Empty only.ALTER USER 'x_app'@'%' IDENTIFIED VIA mysql_native_password USING ''; without the OR alternative. Careful: at this stage a client that still sends the old password is rejected, even with a valid certificate. So it only comes once stage 3 has reached every pod.
  5. Clean up. Delete the transitional user 'x_app'@'localhost' once the unix_socket probe has proved itself.

When reading back a statement like the one in stage 2, reveal nothing. As a check, read only whether the chain has two entries and exactly one of them carries a hash. Never print the hash itself.

07 — Proving certificate rotation before the password goes

Without a password, the sign-in depends on the certificate alone. If a client does not pick up a renewed certificate, it fails after 24 hours at the latest. You should therefore see this rotation once in live operation before you drop the password. Waiting a whole day for it is unnecessary.

Pod certificates can be requested with a shorter lifetime. In the pod's volume, maxExpirationSeconds caps the lifetime. The shortest permitted lifetime is one hour:

 

- podCertificate:
    signerName: example.com/tenant-kunde-x
    keyType: ECDSAP256
    maxExpirationSeconds: 3600
    keyPath: app.key
    certificateChainPath: app.crt

 

The kubelet then renews after about two thirds of the lifetime, so after roughly 40 minutes instead of 16 hours. Two checks show whether the rotation holds:

Two things make the observation harder than it sounds:

After the test, revert the short lifetime. The signer's policy then applies again.

Frequently asked questions

Is signing in without a password not weaker than with one?+

No. The certificate was already mandatory before. What has gone is the static factor that anyone able to read the file in the container could take away. What remains is a key that is worthless after 24 hours and is bound to subject and issuer.

Why is REQUIRE X509 not enough?+

REQUIRE X509 only asks for any valid certificate from a trusted CA. Every component of the tenant could then sign in as the application, including the reverse proxy or the cache. Only subject and issuer bind the sign-in to exactly one workload.

What happens if certificate renewal fails?+

Then the sign-in fails as soon as the certificate expires, with or without a password. With a 24-hour lifetime and renewal after 16 hours, there are eight hours of buffer. Monitoring the remaining lifetime, as described in part 5 of the TYPO3 series, raises an alert beforehand.

Does this also work with cert-manager instead of pod certificates?+

The principle works with any client certificate whose subject names the component unambiguously. Pod certificates have the advantage that the key never sits in a Secret. The kubelet derives the identity from the ServiceAccount.

How do I get back if something goes wrong?+

Every stage has a one-step way back. For MariaDB that is the OR chain with the old password or REQUIRE NONE, for Valkey an open default user with requirepass. So keep the password in the secret store until all stages have run stably for a while. Only then delete it.

Conclusion

Signing in with the pod certificate is not a major rebuild. It is a series of small steps, none of which can break anything on its own. The real work lies in the details no documentation mentions. Valkey does not check the issuer, and MariaDB escapes the slash. The readiness probe signs in over a path without a certificate, and the exporter needs commands nobody knew about beforehand. Knowing these spots, you can remove the last static password between application, database and cache without a minute of downtime. Along the way this meets an IT-Grundschutz requirement in the simplest possible way: the most securely stored password is one that no longer exists (SYS.1.6.A8).

How the key of the pod certificate itself gets out of the PHP process is shown in part 2: The key does not belong in the PHP process.

The series "Nothing Worth Stealing"

Four parts on how credentials and keys in a TYPO3 cluster end up where injected PHP code can no longer take them.

  1. Part 1: Certificate instead of password (this article)
  2. Part 2: The key does not belong in the PHP process
  3. Part 3: What really protects secrets in the container
  4. Part 4: Egress by hostname instead of IP

The series builds on "TYPO3 on Kubernetes under IT-Grundschutz", above all on its parts 5 (mTLS on every connection), 6 (network and tenant isolation) and 7 (secrets without Git).

I move your cluster's database and cache to certificate sign-in. No password in the container and no downtime.

ACL users per component for Valkey, subject and issuer for MariaDB, a dedicated probe account and a staged transition in which every stage has a way back.

If you wish, I run the certificate chain for you on an ongoing basis.

Book a call →

About the author

Photo of Kai Ole Hartwig.

Kai Ole Hartwig

Freelance DevSecOps consultant · OnlyOle Consulting

Programming since 2002 – self-taught, set up my own business with KO-Web in 2012. Over 100 projects, with a focus on security, performance, automation and quality. Today freelance: DevSecOps consulting, training and software development.