Pod certificates instead of shared secrets

01 — Where it started

On a multi-tenant k3s platform, CrowdSec runs centrally. One LAPI decides on bans. Agents read logs, bouncers enforce the bans. Every participant identifies itself with a shared secret. The value sits at the client and at the LAPI, and both copies have to match.

Three times they did not, and three times nobody noticed. A bouncer got 403 and kept running open. A firewall bouncer exited and took its nftables rules with it. An agent reported nothing because its machine was missing. More careful syncing would not have fixed the cause. The cause is the second copy.

02 — What Kubernetes 1.37 provides

Pod certificates are stable in 1.37 (KEP-4317), and so are ClusterTrustBundles (KEP-3257). A pod requests its certificate through a projected volume:

volumes:
  - name: lapi-tls
    projected:
      sources:
        - podCertificate:
            signerName: koh.ole-hartwig.eu/workload
            keyType: ECDSAP256
            keyPath: tls.key
            certificateChainPath: tls.crt
            userAnnotations:
              koh.ole-hartwig.eu/dns-names: crowdsec-loki.monitoring.svc,crowdsec-loki.monitoring.svc.cluster.local
        - clusterTrustBundle:
            signerName: koh.ole-hartwig.eu/workload
            labelSelector: {}
            path: ca.crt

The kubelet generates the key itself. It files a PodCertificateRequest with the public key and writes certificate and key into the volume. No Secret ever holds the key. The pod does not start until the certificate is there. After about two thirds of the lifetime the kubelet fetches a new one and swaps the files.

The ClusterTrustBundle hands the CA to every pod that is meant to trust the signer. Who issues the certificate, Kubernetes leaves open. That is the signer's job.

03 — The signer

cert-manager cannot issue pod certificates yet (cert-manager#8378). So I wrote a signer of my own, a good 1,500 lines of Go without the tests. It watches the requests for its signer name, checks them against a policy and signs with a key in AWS KMS.

The policy is a file. Whatever it does not list is denied:

{
  "signerName": "koh.ole-hartwig.eu/workload",
  "trustDomain": "koh.ole-hartwig.eu",
  "dnsNamesAnnotation": "koh.ole-hartwig.eu/dns-names",
  "lifetime": "24h",
  "refreshAt": 0.66,
  "keyTypes": ["ECDSAP256", "ED25519"],
  "grants": [
    {"namespace": "monitoring", "serviceAccount": "crowdsec-loki", "usage": "server",
     "dnsNames": ["crowdsec-loki.monitoring.svc", "crowdsec-loki.monitoring.svc.cluster.local"]},
    {"namespace": "kube-system", "serviceAccount": "traefik", "usage": "client", "ou": "crowdsec-agent"}
  ]
}

The identity belongs to the ServiceAccount, not the pod. The CN reads namespace/serviceaccount, plus a SPIFFE URI. The OU tells the other side which role the client has. A pod asks for DNS names through an annotation. The signer only issues names the grant allows.

The CA key is a KMS key of type ECC_NIST_P256. It never leaves KMS. The IAM policy allows exactly one operation:

"Action": "kms:Sign",
"Condition": {
  "StringEquals": {
    "kms:SigningAlgorithm": "ECDSA_SHA_256",
    "kms:MessageType": "DIGEST"
  }
}

A subcommand creates the CA once, by hand. It carries critical name constraints: only svc, svc.cluster.local and the URI domain. Even a faulty signer could not issue a valid certificate for a public domain. The signer runs with two replicas. The status write is conditional, so exactly one replica answers each request. An alert fires as soon as a request has waited longer than five minutes.

04 — The API server's rules

It took three releases to get the first certificate. The first two issued nothing. The API server rejected every status write, and my code did not log the rejection. The rules live in the API server's validation, not in the documentation:

• notBefore and notAfter in the status must match the values in the certificate to the second.
• notBefore may lie at most five minutes before the server's clock. That is why the backdating is one minute.
• The lifetime lies between one hour and maxExpirationSeconds, counted from the backdated start.
• beginRefreshAt lies at least ten minutes after the start and ten minutes before the end.
• The public key in the certificate must be the one from the request.

After that, the first certificate arrived 28 milliseconds after startup. The fake API server in my tests now enforces each of these rules. For each there is a mutation that brings the old bug back and turns the test red.

05 — CrowdSec on mTLS

The LAPI gets its server certificate from the volume above. CrowdSec's start script knows the variables it needs:

USE_TLS=true
LAPI_CERT_FILE=/run/lapi-tls/tls.crt
LAPI_KEY_FILE=/run/lapi-tls/tls.key
CACERT_FILE=/run/lapi-tls/ca.crt
AGENTS_ALLOWED_OU=crowdsec-agent

The agents get a client certificate and, instead of AGENT_USERNAME and AGENT_PASSWORD, only CLIENT_CERT_FILE and CLIENT_KEY_FILE. CrowdSec names the machine after the certificate: kube-system/traefik@10.42.0.30. The tenant is now in every alert.

The LAPI has a single listener, so I had to measure first how the clients behave during the switch. A running client survives the other side moving to TLS. It reports one error per request and carries on. A client that starts with the wrong protocol exits. The switch therefore ran in two steps: the LAPI first, then all clients against a listener that already spoke TLS.

The second measurement mattered more. CrowdSec reads its certificate exactly once, at startup. Without a countermeasure the LAPI would have served an expired certificate after 24 hours. SIGHUP reloads server and agent in the running process, measured at about two seconds. My image's entrypoint sends the signal as soon as the files change:

fingerprint() { cat $tls_files | cksum; }
old=$(fingerprint)
while sleep 60; do
  new=$(fingerprint)
  [ "$new" = "$old" ] && continue
  [ "$(cat /proc/1/comm)" = crowdsec ] || continue
  kill -HUP 1 && old=$new
done

It compares the content, not the timestamp: the kubelet swaps the volume through a symlink. The signal only goes to a process named crowdsec. With tini or a shell as PID 1, SIGHUP would end the container.

06 — What it cost

One node ran for five minutes without a packet filter. In the first step, a {{- if }} in the Helm template swallowed a blank line in the firewall bouncer's ConfigMap. I had checked the value I meant to change. The file was different all the same. Reloader restarted the bouncer against a LAPI it could not yet talk to. The CI job for render diffs compared only the tenants, not the platform. Today it compares every object and, for every changed ConfigMap, names the workload that will restart.

Two findings were older than the migration. The TYPO3 integration had never reached the LAPI, because a network policy let only the proxy through. And the proxy ran under the default ServiceAccount of its namespace. A certificate for that account would have given every pod there an identity that can trigger bans. It has its own now.

07 — What remains

All seven agents log in with a certificate. Six passwords, three ExternalSecrets and the loop that registered them are deleted.

The bouncers keep their API keys, now over TLS. The firewall bouncer reads a certificate once and knows no SIGHUP. A restart every 16 hours would clear its rules each time. I sent the fix upstream: go-cs-bouncer#62 reloads the client certificate and closes open connections when it does. crowdsec#4709 does the same for the LAPI. Once both are released, the signal detour in the image goes away.

Certificates inside a tenant stay with cert-manager. A CA per tenant is the isolation there. With a shared CA, every connection would have to pin the other side individually. I use pod certificates for identities that cross namespaces.

The signer is open source under Apache-2.0: github.com/ohartwig/pod-cert-signer, with signed releases and an image on ghcr.io. The repository has the policy reference and example manifests, plus the decisions behind the design. What it does and who it fits is on the project page.

Conclusion

Pod certificates pay off where a shared secret is kept in two places today and crosses namespaces. The laborious part is not the signer. The laborious part is the question of what the software actually does with a renewed certificate. Only a measurement answers it.