Kubernetes networking for network engineers: what a CNI actually does

When a network engineer first reads about Kubernetes networking, the usual reaction is suspicion. Every pod gets its own IP. Pods on different nodes can reach each other without NAT. There are no VLANs, no trunks, no spanning tree. The documentation makes it sound like a new invention. It isn't. It's the same forwarding you already know, with a new management plane layered on top.

The Container Network Interface is that management plane. CNI is not a product — it's a specification. It defines a contract between the container runtime and whichever network plugin you've chosen. When Kubernetes creates a pod, it calls your CNI plugin with a single job: attach this container to the network and make it reachable. When it deletes a pod, it calls the plugin again: clean up.

What the plugin actually does

Strip away the marketing and a CNI plugin does four things. It creates a virtual ethernet pair (veth), puts one end inside the pod's network namespace and the other end on the host. It assigns an IP address from a pool it manages. It sets up routes so the host knows how to reach the pod. And it does something — the interesting part — to make sure traffic can flow between pods on different nodes.

That last step is where CNIs diverge. The simplest possible implementation is a flat L3 network with routes added to every node: node A gets 10.0.1.0/24, node B gets 10.0.2.0/24, and every node has a route to every other node's pod CIDR via the other node's IP. This works. Flannel in host-gw mode does exactly this.

# What a simple CNI leaves on the host routing table
ip route show
# 10.0.1.0/24 via 192.168.1.11 dev eth0  ← pods on node-2
# 10.0.2.0/24 via 192.168.1.12 dev eth0  ← pods on node-3
# 10.0.0.0/24 dev cni0                   ← pods on this node

This is not magic. This is a routing table. You have read thousands of these.

When the underlay doesn't cooperate

The flat L3 approach only works if your underlying network will route arbitrary pod CIDRs between nodes — which it often won't. Cloud VPCs, in particular, tend to know about instance IPs and nothing else. The solution is encapsulation: wrap the pod-to-pod packet in something the underlay understands, send it to the destination node, unwrap it.

Flannel in VXLAN mode does this. Calico with IP-in-IP does this. What they're really doing is running a distributed overlay — which is exactly what EVPN/VXLAN does at the fabric layer, just with a different control plane. If you've configured VTEP peers on a spine, the concept maps directly.

Network policy is just iptables

When you add a NetworkPolicy, your CNI translates it into iptables (or eBPF) rules on the host. There is no magic firewall — there's a chain, there are matches, there are jumps. The same mental model that explains an ACL on a Cisco ASA explains a NetworkPolicy, just expressed differently.

This is worth understanding because it changes how you debug. When a pod can't reach another pod and there's a NetworkPolicy involved, you go to the node and look at iptables. Not at the Kubernetes API. Not at a dashboard. The forwarding decision is happening in the kernel, and the kernel will tell you exactly what it decided if you ask it.

Choosing a CNI

For most clusters on managed cloud platforms — EKS, GKE, AKS — the platform's native CNI is the right choice. You give up some flexibility and gain a lot of integration: the VPC knows about your pods, security groups work, flow logs work. Go with it.

Where you have a choice, the deciding factors are usually: do you need network policy (most CNIs support this now), do you need encryption (Wireguard or IPSec overlay), and do you want eBPF or iptables for the dataplane. Cilium is worth understanding regardless of what you choose — its model for thinking about policy and observability is better than what came before it.