Our ingress saw every visitor as one of our own nodes and let them reach the admin pages

In June the access logs for our back office, the internal pages our operations team uses to adjust orders and refunds, showed a few thousand login attempts overnight from what looked like nine addresses. All nine were our own Kubernetes nodes. The attempts came from the internet, and our ingress had let every one of them reach the login page.

Those pages are protected first by an allowlist on the Ingress, the whitelist-source-range annotation in ingress-nginx, and then by single sign on. The allowlist held our office range and 10.0.0.0/8, because staff on the VPN reach the cluster from inside our private network. Two weeks earlier we had replaced the cloud's HTTP load balancer in front of ingress-nginx with a network load balancer, to get HTTP/2 through to the controller and to cut cost. The old balancer added X-Forwarded-For, and nginx was configured to trust it. The new one forwards TCP and passes the client's address through.

Kubernetes then took the address away. The controller's Service was type LoadBalancer with the default externalTrafficPolicy of Cluster. Under that policy any node can accept the traffic and forward it to an ingress pod on another node, and kube-proxy rewrites the source address to the node's own so that the reply comes back the same way. The documentation says so plainly: the client's address is not preserved. nginx saw a node address inside 10.0.0.0/8, matched the allowlist, and passed every visitor on the internet as one of us. Single sign on held and nobody got in, but the only control between the world and that login page had been switched off by a load balancer change.

We set externalTrafficPolicy to Local the same day. The load balancer then sends traffic only to nodes running an ingress pod, using the health check port Kubernetes provides for that purpose, and the pod sees the real client. The allowlist is now the VPN's own small range, not the whole private network. The back office has moved to an internal load balancer with no public address at all. A probe outside our network requests those pages every five minutes and alerts on anything but a 403. And our rate limits, which had quietly been limiting nine addresses, count real clients again.

An allowlist trusts an address. Ours trusted whatever address the last hop wrote on the packet, and that hop belonged to us.

– Sergey Shinder