Nothing Worth Stealing, Part 1: Certificate instead of password
With mTLS on every connection, an application has two factors for signing in to its database and cache: the client certificate and a password. The certificate is short-lived and issued through the kubelet. The password is static and sits as a file in the container. That is exactly where an attacker looks first after finding a flaw in the application.
This how-to shows how to drop the password entirely. Valkey maps each certificate to an ACL user, and MariaDB requires the subject and issuer of the pod certificate. The switch runs in stages in which nothing breaks.
Part 1 of 4 of the series "Nothing Worth Stealing". It extends part 5: mTLS on every connection of the series "TYPO3 on Kubernetes under IT-Grundschutz".
01 — The starting point: two factors, one of them static
The starting point from part 5 of the TYPO3 series: Valkey listens only on its TLS port and requires a client certificate with tls-auth-clients yes. MariaDB enforces TLS with require_secure_transport=ON. On top of that, the application signs in to both with a password.
The password adds less than it seems. Anyone who can run code in the PHP container reads the password file just as the application does. And it is static. It stays valid until someone changes it, which often means for years.
The pod certificate has neither weakness. It is valid for 24 hours, and the kubelet renews it automatically. Its subject carries the pod's identity: CN=<namespace>/<serviceaccount>, for the application for example CN=kunde-x/x-app. Once the database and cache require exactly this subject, the password is redundant.
The prerequisite is one signer per tenant, as described in the field notes Pod certificates instead of shared secrets and The whole network on pod certificates. A certificate from tenant A must never be accepted by the cache of tenant B.
02 — Valkey: one ACL user per certificate
Valkey 9 can use the common name of the client certificate directly as the user name. The setting for this is tls-auth-clients-user:
tls-auth-clients yes
tls-auth-clients-user CN
user default off
user kunde-x/x-app on nopass ~* &* +@all -@admin -@dangerous +flushdb +keys +info
user kunde-x/x-cache-exporter on nopass +info +ping +select +dbsize +config|get +client|list +client|info +client|setname +command|info +commandlog|len +slowlog +latency +memory|stats +cluster|info
user kunde-x/x-cache on nopass +ping +config|set +acl|log +acl|whoami
The application gets everything except administration and dangerous commands. On top come the two dangerous ones a cache backend really needs. The exporter only reads. The reload sidecar sets certificates and may read the ACL log as a check.
Three points matter here:
user default offis the core. If Valkey finds no matching user for a certificate, the connection lands on thedefaultuser. If that user were open, every valid certificate of the tenant would get in without a password, including the database's or the reverse proxy's. Withoff, every unmapped connection is rejected.nopassis not careless here. Without a matching client certificate nobody even reaches the sign-in, because the TLS handshake fails first.- The slash in the user name works.
kunde-x/x-appis a valid ACL name.
The command list from actual use
The list of permitted commands should come from how the clients really behave, not from the documentation. Start generously and narrow it down with ACL LOG. In practice, typical gaps show up:
- A cache backend that sends
FLUSHDBwhen clearing needs+flushdb. The command belongs to the@dangerouscategory and is otherwise removed. The same goes forKEYSif a backend uses it to find keys by pattern. - The redis_exporter calls
CLIENT SETNAMEandCOMMAND INFOwhen it connects, and on Valkey 9 alsoCOMMANDLOG LENon every scrape. If one of them is missing, the metrics have gaps andACL LOGfills withNOPERMentries. - A sidecar that reloads renewed certificates via
CONFIG SETneeds exactly that permission and almost nothing else. If you also let it read the ACL log, you see after every reload whether anything was rejected.
valkey-cli --tls ... ACL LOG 10
The command shows which user tried which command. After every stage the log should stay empty.
The pitfall: Valkey does not check the issuer
Valkey takes the CN as soon as the handshake succeeds. Which CA issued the certificate plays no part in the mapping. A certificate with the CN kunde-x/x-app from another CA that the cache also trusts therefore signs in as the application. In testing, exactly that happened with a forged certificate from a transitional CA.
This becomes a problem when the cache's trust bundle contains several CAs. During a move from cert-manager to pod certificates that is the normal case. With a password this was covered. Without one, it would be the only protection.
The rule is therefore: before the password goes, the cache trusts only the current tenant CA. With pod certificates this works through a label selector on the ClusterTrustBundle:
- clusterTrustBundle:
signerName: example.com/tenant-kunde-x
labelSelector:
matchLabels: { example.com/trust: current }
path: ca.crt
Afterwards the CA file in the cache pod contains exactly one certificate. You can check that before the next step.
03 — Switching Valkey in three stages
Every stage changes the Valkey configuration and therefore restarts Valkey. A restart empties the cache and drops all connections. In practice this costs a few seconds in which individual requests fail with "went away". Plan each stage for a quiet window.
- Create the ACL users,
defaultkeeps the password. Clients that still sendAUTHwith the password land ondefaultand have full rights. Clients that send no password become their certificate user straight away. Nothing can break. - Move clients one by one to no password. First the application, then the exporter and the reload sidecar. After each step,
ACL LOGshows whether a command is missing. A PHP client such as phpredis signs in over TLS withoutAUTH. The common cache libraries treat an empty password as "none". user default off, removerequirepass. Only now, once no client sends the password any more. After that the password file can disappear from all pods, including the cache pod itself.
04 — MariaDB: requiring subject and issuer
MariaDB can require a specific client certificate for a user:
ALTER USER 'x_app'@'%'
REQUIRE SUBJECT '/CN=kunde-x\\/x-app'
AND ISSUER '/CN=Mandanten-CA kunde-x 2026';
Pitfall 1: the escaped slash
The CN of a pod certificate contains a slash. MariaDB compares the subject in its one-line notation. There, a slash inside a value is escaped as \/. REQUIRE SUBJECT '/CN=kunde-x/x-app' therefore never matches. What you mean is /CN=kunde-x\/x-app. Because the backslash must itself be escaped in an SQL string, the statement contains \\/.
The RFC 2253 notation without a leading slash does not match either. If in doubt: with log_warnings=4, MariaDB writes the subject it actually saw to the log.
For reading it back, --raw helps. Otherwise the client's batch mode doubles the backslash in the output:
mariadb --batch --raw --skip-column-names -e \
"SELECT CONCAT(ssl_type,'|',x509_subject,'|',x509_issuer)
FROM mysql.user WHERE user='x_app' AND host='%'"
Pitfall 2: without ISSUER, every trusted CA counts
As with the cache: a REQUIRE SUBJECT on its own lets through a certificate with the same subject from any CA the server trusts. Only AND ISSUER restricts the sign-in to the tenant CA. In testing, the forged certificate from the transitional CA was rejected with ISSUER and accepted without it.
The issuer usually carries a year or a generation in its name. When the tenant CA changes, the REQUIRE ISSUER has to follow. That belongs in the CA rotation procedure, not in someone's memory.
05 — A dedicated account for the readiness probe
The typical stumbling block in the first step is not the application at all. It is the database's readiness probe. Many images check readiness by signing in over the local Unix socket with the application user and its password. No client certificate travels over the socket.
As soon as the application user requires a certificate, the probe fails. The database drops out of the Service although it is running. That is exactly what happens if you simply set REQUIRE.
There are two clean ways:
- Transition: a separate user
'x_app'@'localhost'with no privilege exceptUSAGE. Socket connections count aslocalhost. The probe signs in there, and theREQUIREonly applies to'%'. - Target: a probe account that signs in via
unix_socket.
CREATE USER IF NOT EXISTS 'mariadb'@'localhost' IDENTIFIED VIA unix_socket;
-- SHOW GRANTS must show exactly this:
-- GRANT USAGE ON *.* TO `mariadb`@`localhost` IDENTIFIED VIA unix_socket
unix_socket authenticates via the user ID of the process at the other end of the socket. The probe runs in the database container as its own system user and is therefore exactly that user. Another process with a different uid is rejected, and over TCP the account does not exist at all. The probe then needs no password.
The order matters. Create the account in one deploy and switch the probe to it only in the next. In the same deploy, the restarted database would meet a probe whose account does not exist yet. It would never become ready. A hook that only runs after a successful rollout would never create the account. Restarting the database for the new probe means a brief moment without a database. That too belongs in a quiet window.
06 — Moving MariaDB to "no password" without downtime
Simply removing the password does not work when several pods use the database. Some start with the new configuration, others still run with the old one. MariaDB can, however, chain several authentication methods with OR. That makes the switch gradual:
- Certificate in addition to the password. Set
REQUIRE SUBJECT … AND ISSUER …and keep the password. This only tightens things. Watch this stage across at least one certificate rotation. - Old password or none.
AnALTER USER 'x_app'@'%' IDENTIFIED VIA mysql_native_password USING PASSWORD('<old-password>') OR mysql_native_password USING '';ALTER USER … IDENTIFIED VIAwithout aREQUIREclause leaves the existingREQUIRE SUBJECT … AND ISSUER …from stage 1 in place. Read it back after every stage rather than relying on that. Clients now sign in with and without the password, and a wrong password is rejected. The certificate stays mandatory in both cases. - Remove the password file from all pods. Application, workers, scheduler, setup and maintenance jobs, in any order. PDO signs in with
nullor''as the password plus the client certificate. Also think of helper scripts that read the password file directly, for a setup lock for instance. They must cope with a missing file. - Empty only.
ALTER USER 'x_app'@'%' IDENTIFIED VIA mysql_native_password USING '';without theORalternative. Careful: at this stage a client that still sends the old password is rejected, even with a valid certificate. So it only comes once stage 3 has reached every pod. - Clean up. Delete the transitional user
'x_app'@'localhost'once theunix_socketprobe has proved itself.
When reading back a statement like the one in stage 2, reveal nothing. As a check, read only whether the chain has two entries and exactly one of them carries a hash. Never print the hash itself.
07 — Proving certificate rotation before the password goes
Without a password, the sign-in depends on the certificate alone. If a client does not pick up a renewed certificate, it fails after 24 hours at the latest. You should therefore see this rotation once in live operation before you drop the password. Waiting a whole day for it is unnecessary.
Pod certificates can be requested with a shorter lifetime. In the pod's volume, maxExpirationSeconds caps the lifetime. The shortest permitted lifetime is one hour:
- podCertificate:
signerName: example.com/tenant-kunde-x
keyType: ECDSAP256
maxExpirationSeconds: 3600
keyPath: app.key
certificateChainPath: app.crt
The kubelet then renews after about two thirds of the lifetime, so after roughly 40 minutes instead of 16 hours. Two checks show whether the rotation holds:
- New certificate without a restart. Every pod receives a second certificate without any container restarting.
- New connections after the old one has expired. Only when the old certificate has expired and new connections to the database and cache still succeed is it certain: the client really presents the new one.
Two things make the observation harder than it sounds:
- PodCertificateRequests disappear. Kubernetes cleans up the request objects shortly after issuance. Anyone counting how many requests a pod had sees nothing after a few minutes. The signer's log is more reliable, as it records every issuance with the pod name.
- Every restart resets the clock. A new pod gets a fresh certificate. A deploy that restarts the pods during the observation pushes the rotation back by the full lifetime.
After the test, revert the short lifetime. The signer's policy then applies again.
Frequently asked questions
Is signing in without a password not weaker than with one?+
No. The certificate was already mandatory before. What has gone is the static factor that anyone able to read the file in the container could take away. What remains is a key that is worthless after 24 hours and is bound to subject and issuer.
Why is REQUIRE X509 not enough?+
REQUIRE X509 only asks for any valid certificate from a trusted CA. Every component of the tenant could then sign in as the application, including the reverse proxy or the cache. Only subject and issuer bind the sign-in to exactly one workload.
What happens if certificate renewal fails?+
Then the sign-in fails as soon as the certificate expires, with or without a password. With a 24-hour lifetime and renewal after 16 hours, there are eight hours of buffer. Monitoring the remaining lifetime, as described in part 5 of the TYPO3 series, raises an alert beforehand.
Does this also work with cert-manager instead of pod certificates?+
The principle works with any client certificate whose subject names the component unambiguously. Pod certificates have the advantage that the key never sits in a Secret. The kubelet derives the identity from the ServiceAccount.
How do I get back if something goes wrong?+
Every stage has a one-step way back. For MariaDB that is the OR chain with the old password or REQUIRE NONE, for Valkey an open default user with requirepass. So keep the password in the secret store until all stages have run stably for a while. Only then delete it.
Conclusion
Signing in with the pod certificate is not a major rebuild. It is a series of small steps, none of which can break anything on its own. The real work lies in the details no documentation mentions. Valkey does not check the issuer, and MariaDB escapes the slash. The readiness probe signs in over a path without a certificate, and the exporter needs commands nobody knew about beforehand. Knowing these spots, you can remove the last static password between application, database and cache without a minute of downtime. Along the way this meets an IT-Grundschutz requirement in the simplest possible way: the most securely stored password is one that no longer exists (SYS.1.6.A8).
How the key of the pod certificate itself gets out of the PHP process is shown in part 2: The key does not belong in the PHP process.
The series "Nothing Worth Stealing"
Four parts on how credentials and keys in a TYPO3 cluster end up where injected PHP code can no longer take them.
- Part 1: Certificate instead of password (this article)
- Part 2: The key does not belong in the PHP process
- Part 3: What really protects secrets in the container
- Part 4: Egress by hostname instead of IP
The series builds on "TYPO3 on Kubernetes under IT-Grundschutz", above all on its parts 5 (mTLS on every connection), 6 (network and tenant isolation) and 7 (secrets without Git).
I move your cluster's database and cache to certificate sign-in. No password in the container and no downtime.
ACL users per component for Valkey, subject and issuer for MariaDB, a dedicated probe account and a staged transition in which every stage has a way back.
If you wish, I run the certificate chain for you on an ongoing basis.
About the author

Kai Ole Hartwig
Programming since 2002 – self-taught, set up my own business with KO-Web in 2012. Over 100 projects, with a focus on security, performance, automation and quality. Today freelance: DevSecOps consulting, training and software development.
