Grant first, switch second

TL;DR

TL;DR: A PodCertificateRequest the signer denies stays denied; the kubelet does not ask again for that pod. When a merge request brings a new grant and the workload that uses it at the same time, the new pods ask before the signer has reloaded its policy and then hang in Init for good. So the grant goes in a merge request of its own, and a gate checks it.

What happened

Pod certificates in Kubernetes go through a signer, and mine decides by a policy. For every ServiceAccount it states whether it gets a certificate and what for. If an entry is missing, the signer denies the request. That is intended: what the policy does not allow should not exist.

Less obvious is what happens next. For the kubelet, a denied request is final. It files no new one for the same pod, and the pod stays in Init because its volume never completes. The event for it is short and clear:

MountVolume.SetUp failed for volume "tls":
  PodCertificateRequest was permanently denied: reason="NoGrant"

A race against your own signer

A new grant and the switch that moves a workload onto the pod certificate feel like a pair, so they tend to land in the same merge request. Argo CD then applies both in the same sync. The new pods start at once and file their request while the signer is still loading its new policy. They are faster than the grant, and their denial sticks.

That happened twice in one day. The first time it hit a new pod of the edge proxy, and because the old pods kept running, nobody but me noticed. The second time it hit the web server of an internal tool. That one does not put a second instance next to the first during a rollout, so the old one was already gone when the new one got stuck. The site answered 503 for ten minutes.

What helped at once

The repair was trivial. Once the signer knew the grant, I deleted the stuck pods. Their successors filed new requests, got their certificates and were running a few minutes later. That the cause was so simple did not make the ten minutes any better.

A gate before the merge

To keep it from happening again, a gate now checks every merge request. It renders all workloads and collects every combination of signer, namespace and ServiceAccount for which a pod asks for a certificate. Each of them must already be granted on the merge base, the state the merge request is built against. A grant that only arrives with the same merge request does not count.

A gate that was never red proves nothing. So I ran it against the real state from before the outage, the merge base of the merge request that caused the ten minutes. It reports exactly the pods that got stuck back then, with the hint to ship the grant on its own first. Against today's state it is green.

The lesson

Whatever another system has to load first should not land in the same rollout as the thing that uses it. A new grant therefore ships on its own and goes live before the switch follows. Where a denial is final, the order cannot be left to the luck of a sync, which is exactly why checking it belongs before the merge.

Conclusion

The signer made the right call both times. What was wrong was my assumption that two changes which belong together may also ship together. With pod certificates the order is part of correctness, because a denial is never taken back. The gate turns that experience into a rule, and running it against the old state shows it would have caught the outage.

If you missed the start, it is in part 6: when the intrusion detection locked out its own edge proxy. The whole move to pod certificates is described in the field note.