Kai Ole Hartwig

pod-cert-signer — a signer for pod certificates, the CA key stays in KMS.

Since Kubernetes 1.37 a pod requests its X.509 certificate through a projected volume. The kubelet generates the key, a signer decides. pod-cert-signer checks every request against a policy that denies by default. It signs with a CA key that never leaves AWS KMS, and it never sees a private key, neither the pod's nor its own.

  • In use: since September 2026 in my cluster, for seven CrowdSec agents that log in with a certificate instead of a password
  • Image:ghcr.io/ohartwig/pod-cert-signer:0, amd64 and arm64, signed keyless
  • Requires: Kubernetes 1.37 and a KMS key of spec ECC_NIST_P256
  • License: Apache-2.0 · no client-go, the AWS SDK for KMS only

The second copy

Until now

  • The password or API key sits at the client and at the server, as a Secret in etcd and in the backup
  • When the copies drift apart the answer is 401: depending on the software it keeps running unprotected, exits, or silently stops reporting
  • Rotating means touching both sides at once, so nobody rotates
  • Who holds which key is written down nowhere in one place

With pod-cert-signer

  • The server trusts a CA; the client proves who it is with a certificate whose key only the kubelet on its node ever held
  • A wrong or expired certificate fails in the handshake, visible on both sides
  • The kubelet renews before expiry, at the share of the lifetime the policy sets; nobody rotates by hand
  • Who gets what is a policy file, changed by review

Does it fit my cluster?

Use it if …

  • your cluster runs Kubernetes 1.37 or later and can reach AWS KMS
  • services in the cluster log in with passwords or API keys, and the software can do client certificates
  • you need identities that cross namespaces, without a CA as a Secret in the cluster
  • you would rather read a small component than configure a large one

Don't use it if …

  • your cluster is older than 1.37
  • your CA key should live in Vault, an HSM or another KMS: only AWS KMS is supported, another backend needs its own crypto.Signer
  • you need certificates for public names: the name constraints allow cluster names only
  • your software reads its certificate once and cannot be made to reload it

Why a signer of my own

Kubernetes 1.37 made pod certificates stable but ships no signer. Support in cert-manager has been open since January (cert-manager#8378). I needed certificates for the CrowdSec agents in my cluster, which talk to the central LAPI from other namespaces, and no further shared CA in the cluster. How the migration went, obstacles included, is in the field note.

The signer is kept small because it is the most privileged component. It talks to the Kubernetes API in a few hundred lines on the standard library instead of client-go. It uses the AWS SDK for credentials and KMS only: authentication is not something to write yourself. The tests run against a fake API server that speaks the watch protocol and enforces the API's time rules. A mutation run removes ten rules one at a time, and every mutant has to turn the suite red.

How a pod gets its certificate

The signer creates its CA certificate once itself, with the key in KMS. It carries critical name constraints: only .svc, .svc.cluster.local and its own trust domain.

 

KMS_KEY_ID=alias/pod-cert-signer pod-cert-signer init \
  -cn "Example workload CA" -trust-domain example.com > ca.crt

 

A pod requests its certificate with a volume:

 

volumes:
  - name: tls
    projected:
      sources:
        - podCertificate:
            signerName: example.com/workload
            keyType: ECDSAP256
            keyPath: tls.key
            certificateChainPath: tls.crt
        - clusterTrustBundle:
            signerName: example.com/workload
            labelSelector: {}
            path: ca.crt

 

The pod does not start until the certificate is issued. Who gets what is a JSON policy, mounted from a ConfigMap:

 

{"namespace": "team-a", "serviceAccount": "api", "usage": "server",
 "dnsNames": ["api.team-a.svc", "api.team-a.svc.cluster.local"]}

 

Anything not granted is denied, with one of four reasons: NoGrant, UnsupportedKeyType, DNSNameNotGranted, InvalidStubPKCS10Request. Because of the name constraints, even a bypassed policy check cannot issue a certificate for a public name.

The documentation

Quick start and policy

KMS key, CA certificate, deployment, first grant: four steps. Plus every policy field, the certificate profile and the metrics.

Read →
Quick start and policy

Decisions

Why KMS, why no intermediate, why two replicas without leader election, why classical signatures. And the rules by which the API rejects a status.

Read →
Decisions

Example manifests

Namespace, RBAC, policy ConfigMap, ClusterTrustBundle, Deployment, a hardened client pod and the IAM policy for the KMS key.

View →
Example manifests

Container image

linux/amd64 and linux/arm64

ghcr.io/ohartwig/pod-cert-signer:0: Wolfi with CA certificates and the binary, nothing else. Built from the signed release binaries, scanned, signed keyless.

View the image →
Container image

GitHub

Linux, amd64 and arm64

Source, releases with binaries, SHA256SUMS and a cosign signature, issues. Apache-2.0.

View on GitHub →
GitHub

Services that share a password?

I help with the move to pod certificates: one pair of services first, with a way back, then the rest.

Get in touch →