TL;DR
TL;DR: Once the cluster ran dual stack, the edge proxy also reached the tenant proxies over IPv6. They, however, trusted forwarded client addresses from the IPv4 pod network only. Over IPv6 every visitor therefore carried the edge proxy's address, the intrusion detection banned it, and its packet filter dropped all traffic of that pod. The fix was one line in the trust list.
What happened
The switch itself went quietly. The services in the cluster got an IPv6 address next to their IPv4 one, the endpoints showed up in both families, and for a while everything was green. Half an hour later the first errors arrived, not on one site but on all of them. Over IPv4 the edge proxy answered 502, over IPv6 there was often no connection at all, and always for only part of the requests: sometimes a quarter, sometimes half.
One pod out of two
The first clue was in the access log. The edge proxy runs with two pods, and every error came from exactly one of them; the other was clean. The failing requests shared something else as well: all of them went to IPv6 addresses of the backends and ran into a connect timeout after about three seconds.
The neighbour table on the node then showed the real picture. Every pod had a reachable entry there, except the IPv6 address of the broken edge pod, which sat at INCOMPLETE. It no longer answered neighbour discovery, so replies to its own connections never reached it, and incoming IPv6 connections that landed on it broke off.
A new pod with the same fault
The obvious repair was to replace the pod. The new one was broken straight away, with a fresh address, which settled that it was not about that one pod. Something was making new edge pods vanish from the network.
The answer was in the intrusion detection's ban list. It held the IPv6 addresses of both edge pods, the old one and the one that had just started. Its packet filter dropped all traffic of those addresses, neighbour discovery included, and that is why it looked like a network fault on the node.
Why its own address, of all things
The tenant proxies sit behind the edge proxy. Their direct peer is therefore always an edge pod, and the real client address travels in the X-Forwarded-For header. They only trust that header, though, when the connection comes from the pod network:
trusted_proxies static <ipv4-pod-cidr>
client_ip_headers X-Forwarded-For
Until the switch every connection came over IPv4, and that held. Afterwards part of them came over IPv6, from a network that was not on the list. For those connections the tenant proxies ignored the header and reported the edge pod as the client. Every suspicious request from the internet, every scanner and every failed attempt was booked to the edge proxy itself, until the intrusion detection banned it.
The comment right above that line had described the problem months earlier: without the trust list, it said, the client address would be the internal hop, and the intrusion detection would ban the hop. It had only been written with IPv4 in mind.
How it got back on its feet
First, all traffic went through the one edge pod that was not banned yet. Then the services went back to a single address family, so that no further pod could walk into the same trap. The actual fix was extending the trust list by the IPv6 pod network. Two guard rules further down, meant to recognise internal callers, had the same gap, and the IPv6 network was missing there too.
trusted_proxies static <ipv4-pod-cidr> <ipv6-pod-cidr>The lesson
Change the address families of a cluster and you quietly change every list that trusts an address range. That is why it pays to find all of those places before moving to dual stack, from forwarded client addresses and internal allowlists to rate limiting and intrusion-detection exceptions. Otherwise a list that knows only one family turns the other into a stranger.
Frequently asked questions
How do you find such places before the switch?+
The simplest way is a search for every address range that counts as trusted, across configuration, charts and policies. Each hit then needs both address families or a reason why one is enough. To keep it that way, a test helps that sends a request over IPv6 through the whole chain and checks which client address arrives at the end.
Why not just exempt the pod network in the intrusion detection?+
That would only have hidden the cause. The tenant proxies would have kept reporting every visitor as the edge proxy, so real attackers would have vanished behind that address. The fix therefore belongs where the real client address got lost, which is the trust list.
Why did it look like a network fault?+
The intrusion detection's packet filter dropped neighbour discovery as well. On the node the banned pod therefore looked like one with a broken interface, and only the ban list revealed that a security decision lay behind it.
Conclusion
The platform did not lock itself out through a network fault but through a trust list that did not know one address family. That is why the symptoms looked like networking, even though they came from a security decision. What stays with me most is that the comment in the config had predicted exactly this outage, only for the addresses that existed when it was written.
The story continues in part 7: grant first, switch second.
I check your trust lists before a switch to dual stack walks around them.
I go through every place that treats an address range as trusted, from forwarded client addresses and internal allowlists to rate limiting and intrusion-detection exceptions, and I do it for both address families.
Rather than advice on paper, I stay with the switch in operation until the first quiet day.
About the author
Kai Ole Hartwig
Programming since 2002 – self-taught, set up my own business with KO-Web in 2012. Over 100 projects, with a focus on security, performance, automation and quality. Today freelance: DevSecOps consulting, training and software development.