Most Kubernetes networking issues I debug in production rarely come from the application itself. They come from misunderstandings about how Services, Ingress, and NetworkPolicies actually interact at the packet level. A Deployment can be healthy, pods running and ready, yet traffic never reaches them because a Service selector is off by one label, an Ingress controller is silently dropping traffic due to a misconfigured annotation, or a NetworkPolicy is applying a default-deny you forgot you committed three months ago. After running clusters on both managed cloud and bare-metal for several years, I've learned that Kubernetes networking is less about memorizing YAML fields and more about understanding the data path — from kube-proxy or eBPF dataplane, through CoreDNS, to the Ingress controller and finally to the pod. This article walks through how each layer works in practice, where it breaks, and how to configure it so it holds up under real load.
Decoding Service Discovery: How ClusterIP, NodePort and LoadBalancer Differ Under the Hood
Every network interaction inside a cluster starts with a Service, even though a Service is not a process or a network device. It is a stable virtual IP and DNS name backed by iptables or IPVS rules that are programmed on every node. When you create a ClusterIP Service, Kubernetes allocates a virtual IP from the service CIDR (typically 10.96.0.0/12) and programs kube-proxy to intercept traffic destined for that IP. In my experience, the biggest confusion is treating ClusterIP as if it forwards traffic itself — it does not. The actual load balancing happens on the node where the client pod runs, before the packet ever leaves the host.
ClusterIP and DNS-Based Service Discovery
CoreDNS creates an A record for every Service in the form service-name.namespace.svc.cluster.local. A pod doing a lookup for payments-api will receive the ClusterIP, not the pod IPs. That decouples consumers from pod churn. I always set publishNotReadyAddresses: false and rely on readiness probes, because otherwise DNS will return endpoints that cannot serve traffic. For headless Services (clusterIP: None), DNS returns the pod IPs directly — essential for StatefulSets like Cassandra or Kafka where the client needs to discover individual members. If you rely on client-side load balancing or quorum logic, headless is the correct choice; otherwise ClusterIP gives you a stable load-balanced VIP.
When NodePort and LoadBalancer Create More Problems Than They Solve
NodePort opens a high port (30000-32767) on every node and forwards it to the Service. I have found that teams use it for quick debugging but leave it in production, which exposes every node to inbound traffic and complicates firewall rules. LoadBalancer builds on NodePort by provisioning an external load balancer from your cloud provider. On EKS, GKE, or AKS this is convenient, but each Service of type LoadBalancer creates a separate cloud LB with its own cost and quota. On bare metal, a LoadBalancer Service will stay in <pending> unless you run something like MetalLB or kube-vip. My rule is simple: use ClusterIP for everything internal, and expose only one or two entry points via Ingress. Reserve LoadBalancer for TCP/UDP services that cannot be HTTP-routed, such as databases or legacy protocols.
apiVersion: v1
kind: Service
metadata:
name: payments-api
namespace: production
labels:
app: payments-api
spec:
selector:
app: payments-api
version: v2
ports:
- name: http
port: 80
targetPort: 8080
protocol: TCP
type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
name: payments-api-headless
namespace: production
spec:
clusterIP: None
selector:
app: payments-api
ports:
- port: 8080
name: http
The example above shows both patterns. The ClusterIP Service provides a stable VIP for standard HTTP clients, while the headless Service is useful for a monitoring job that needs to scrape each pod individually. Notice the selector includes version: v2 — I have seen outages where a rolling deployment changed labels but the Service still selected both versions, sending traffic to pods that were not yet ready. Per the Kubernetes Documentation on Services, label selectors are the sole link between Service and endpoints, so mismatches fail silently without events.
kube-proxy, IPVS and eBPF: What Actually Handles Service Traffic
When a pod sends traffic to a ClusterIP, the packet traverses the node's netfilter pipeline where kube-proxy's rules rewrite the destination to a backend pod IP. Understanding which mode kube-proxy runs in explains many performance anomalies. The default iptables mode creates a chain of rules that are evaluated sequentially. For a Service with 500 backends, that is 500 rules to scan per packet. It works, but latency grows linearly and CPU usage becomes noticeable during churn.
iptables vs IPVS Mode in Large Clusters
I switched a 150-node cluster with several thousand Services from iptables to IPVS and saw conntrack contention drop immediately. IPVS uses a hash table in the kernel and supports multiple scheduling algorithms — rr (round-robin), lc (least connection), and sh (source hashing) are the most relevant. To enable it, you set mode: ipvs in the kube-proxy ConfigMap and ensure ip_vs kernel modules are loaded. In my experience, IPVS is mandatory if you run high-cardinality Services or frequent endpoint updates, such as with Kubernetes Autoscaling: HPA, VPA and KEDA for Event-Driven Scaling where pods scale up and down constantly. The tradeoff is slightly more operational complexity, as you need to monitor IPVS sync and ensure kube-proxy metrics are scraped.
Where Cilium and eBPF Change the Data Path
Cilium replaces kube-proxy entirely with eBPF programs attached to the kernel. Instead of relying on iptables, eBPF maps handle load balancing with O(1) lookups and can enforce NetworkPolicies at the same layer. In clusters where I have deployed Cilium, service latency dropped by 15-20% and we eliminated kube-proxy as a component to manage. The key advantage is not just performance — eBPF gives you visibility with cilium monitor and Hubble, which shows policy verdicts per flow. If you are considering a service mesh, understand that Cilium already provides L3/L4 policy and can handle L7 with Envoy; you may not need a full mesh. I covered the tradeoffs in detail when evaluating Service Mesh Comparison: Istio vs Linkerd vs Cilium for Production, and the decision often comes down to whether you need mutual TLS and advanced traffic splitting at the mesh layer.
| Data Plane | Mechanism | Performance Profile | When I Recommend It |
|---|---|---|---|
| kube-proxy iptables | Sequential iptables NAT rules | O(n) per Service endpoints, degrades past ~1000 Services | Small to medium clusters (<50 nodes), default for simplicity |
| kube-proxy IPVS | Kernel hash table (IP Virtual Server) | O(1) lookup, supports session affinity and multiple schedulers | Large clusters, high endpoint churn, cost-sensitive environments |
| Cilium eBPF | eBPF maps and programs in kernel | O(1), lowest latency, no kube-proxy required | Clusters needing NetworkPolicy enforcement, observability, or kube-proxy replacement |
| kube-proxy nftables (alpha) | nftables set-based NAT | O(1) with modern kernel, simpler rule set | Experimental, future replacement for iptables on newer kernels |
If you are unsure which mode you are running, check kubectl -n kube-system get configmap kube-proxy -o yaml and look at the mode field, or run iptables-save | grep KUBE-SVC on a node. If you see thousands of KUBE-SVC chains, you are likely paying an iptables tax you can avoid.
Ingress Controllers Compared: NGINX, Traefik, HAProxy and the Gateway API
Ingress is the declarative API for exposing HTTP/S routes, but the Ingress resource does nothing by itself. An Ingress controller — a pod (or DaemonSet) that watches Ingress objects and configures a reverse proxy — does the actual work. In my experience, the choice of controller matters less than understanding its operational model. NGINX Ingress (the community ingress-nginx) reloads NGINX on config changes, which can cause brief worker reloads. Traefik and HAProxy Ingress update dynamically without reloads. For most teams starting out, ingress-nginx remains the safest choice due to documentation and examples, even though its reload behavior needs tuning with worker-shutdown-timeout for long-lived connections.
Configuring NGINX Ingress for TLS Termination and Path Routing
A common production pattern is to terminate TLS at the Ingress and route by path and host. I always use cert-manager with Let's Encrypt rather than manually managing Secrets, and I set explicit ingressClassName to avoid the deprecated kubernetes.io/ingress.class annotation. One lesson I learned hard: NGINX's default proxy-body-size is 1m, so file uploads fail silently with 413 unless you increase it. Also, enable use-forwarded-headers and set proxy-real-ip-cidr correctly if you sit behind a cloud LB, otherwise X-Forwarded-For can be spoofed.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: payments-ingress
namespace: production
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "10m"
nginx.ingress.kubernetes.io/proxy-connect-timeout: "5"
nginx.ingress.kubernetes.io/proxy-read-timeout: "60"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
ingressClassName: nginx
tls:
- hosts:
- api.example.com
secretName: payments-tls
rules:
- host: api.example.com
http:
paths:
- path: /api/v2/payments
pathType: Prefix
backend:
service:
name: payments-api
port:
number: 80
- path: /api/v2/refunds
pathType: Prefix
backend:
service:
name: refunds-api
port:
number: 80
This manifest routes two prefixes to different Services over a single host with automated TLS. I prefer pathType: Prefix over ImplementationSpecific because the matching behavior is standardized. For canary releases, I use NGINX's canary annotations or, better, split the logic into Gateway API resources.
Moving from Ingress Resources to Gateway API
The Kubernetes Gateway API (Gateway, HTTPRoute, GRPCRoute) is the successor to Ingress. It separates roles: platform teams manage Gateway and GatewayClass, developers manage HTTPRoutes that attach to them. I have found this separation prevents the annotation sprawl that makes NGINX Ingress objects unreadable. Gateway API also natively supports header-based routing, traffic splitting, and backend TLS without vendor annotations. If you are templating Ingress in Helm, consider migrating to Gateway API — it composes better with Helm Chart Development: Templates, Values, Dependencies and Testing because you can template HTTPRoute without embedding controller-specific annotations. Cilium, Istio, and Traefik already implement Gateway API; ingress-nginx has experimental support. For greenfield clusters, I start with Gateway API directly.
Writing Network Policies That Enforce Real Isolation Without Breaking Your App
By default, Kubernetes allows all pod-to-pod traffic. A NetworkPolicy is a firewall rule expressed as selectors — it does not create network infrastructure, it tells the CNI plugin which flows to allow. No policy means allow-all; a single default-deny policy flips the namespace to deny-all. I have seen teams apply a default-deny and immediately break DNS, health checks, and metrics scraping because they forgot to allow egress to CoreDNS and kubelet.
Default-Deny Strategies for Namespace Isolation
My standard baseline for any production namespace is three policies: default-deny ingress, default-deny egress, then explicit allow rules. This is zero-trust within the cluster. Start by allowing DNS (UDP/TCP 53 to kube-system labeled k8s-app=kube-dns), then allow ingress from the Ingress controller namespace, then allow egress to only the Services the workload actually needs. I label namespaces with name: production and reference them via namespaceSelector rather than hardcoding IPs, which keeps policies portable across clusters with different pod CIDRs.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: production
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-payments-api
namespace: production
spec:
podSelector:
matchLabels:
app: payments-api
policyTypes:
- Ingress
- Egress
ingress:
- from:
- namespaceSelector:
matchLabels:
name: ingress-nginx
- podSelector:
matchLabels:
app: payments-frontend
ports:
- protocol: TCP
port: 8080
egress:
- to:
- namespaceSelector:
matchLabels:
name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
- to:
- podSelector:
matchLabels:
app: postgres
ports:
- protocol: TCP
port: 5432
The first policy denies all traffic to every pod in production. The second explicitly allows payments-api to receive traffic only from the Ingress namespace and frontend pods, and to egress only to DNS and Postgres. Without the DNS egress rule, the application cannot resolve postgres.production.svc.cluster.local and fails with obscure connection timeouts. As documented in the Kubernetes Documentation on NetworkPolicies, egress to the API server is not required for normal pod operation, so you can keep egress tight.
Debugging Silent Drops Caused by Misapplied Selectors
NetworkPolicies are additive and selectors are ANDed. If a policy has both namespaceSelector and podSelector under the same from block, traffic must match both — not either. I have spent hours debugging a policy that used two separate from entries when it should have used one. Another common mistake is applying a policy to the wrong namespace; policies are namespaced and only select pods in their own namespace. Validate with kubectl -n production describe networkpolicy and test with a temporary debug pod running curl or netcat. If your CNI is Calico or Cilium, use calicoctl or cilium policy get to see the compiled rules.
Troubleshooting Kubernetes Networking Like a Production Incident
When a Service is unreachable, I follow a fixed order: DNS, endpoints, kube-proxy rules, NetworkPolicy, then CNI. This eliminates guessing. Start with nslookup payments-api.production.svc.cluster.local from a debug pod in the same namespace. If DNS fails, CoreDNS is overloaded or a NetworkPolicy blocks port 53. If DNS succeeds but curl to the ClusterIP times out, check endpoints with kubectl get endpoints payments-api -n production. Empty endpoints means label mismatch between Service and pods — the most frequent root cause I see.
DNS Resolution Failures and CoreDNS Tuning
CoreDNS forwards .cluster.local queries to the Kubernetes plugin and everything else upstream. Under high QPS, the default 8 CoreDNS replicas may thrash, and ndots:5 in /etc/resolv.conf multiplies every lookup by up to five queries. I set ndots:2 or use single-request-reopen in applications that do not need search expansion, and I enable CoreDNS autoscaling via cluster-proportional-autoscaler. I also raise the CoreDNS cache TTL and enable ready plugin health checks so the Service only returns ready endpoints. Remember that headless Services bypass ClusterIP and return pod IPs directly, so a failing readiness probe removes the pod from DNS entirely — which is correct but surprising during rolling updates.
Tracing Packet Drops from Pod to Service
If endpoints exist and DNS works, exec onto the node and inspect iptables or IPVS. For iptables, iptables -t nat -L KUBE-SERVICES -n | grep payments-api shows the NAT chain; for IPVS, ipvsadm -Ln lists virtual servers. For eBPF, cilium bpf lb list or Hubble flows show verdicts. I also run kubectl run -it --rm debug --image=nicolaka/netshoot --restart=Never -- curl -v http://payments-api.production:80/health to test from a clean network namespace. If curl works from netshoot but not from the application pod, a NetworkPolicy is almost certainly blocking the application pod's egress. Add temporary allow-all policies to bisect the issue, then narrow to minimal rules. In one incident, a liveness probe using HTTP GET was blocked by egress policy, causing kubelet to restart pods that were actually healthy.
Exposing Workloads on Bare Metal, On-Prem and Managed Cloud Without Surprises
The final hop — getting traffic from the internet into the cluster — differs dramatically by environment. On managed cloud, a Service of type LoadBalancer provisions a cloud LB that forwards to NodePort. This is simple but each LB costs money and has provider limits. I prefer a single Ingress controller exposed via LoadBalancer and then routing all HTTP traffic through it. This consolidates TLS and WAF at one layer. On AWS, set service.beta.kubernetes.io/aws-load-balancer-type: nlb to get a Network Load Balancer with static IPs and preserve client IP via externalTrafficPolicy: Local.
MetalLB and ExternalIPs for On-Prem Load Balancing
On bare metal there is no cloud controller to provision LBs. MetalLB fills that gap in two modes: Layer 2 (ARP/NDP announcement) and BGP. I have used Layer 2 for small on-prem clusters where a single node announces the Service IP and failover takes a few seconds. For larger sites, BGP with a ToR switch gives true ECMP load balancing. Configure MetalLB with an IP pool that does not overlap DHCP and set externalTrafficPolicy: Local to avoid an extra hop. If MetalLB is not an option, consider NodePort behind an external HAProxy or keepalived VIP — it is less automated but entirely predictable.
Cloud Provider LoadBalancer Integration and Cost Control
On GKE, the default Ingress controller is Google Cloud Load Balancer via the GCE ingress, which creates global LBs with different semantics than NGINX. I disable it when I want NGINX or Traefik. On EKS, the AWS Load Balancer Controller provides both ALB (L7) and NLB (L4) support via Ingress or Gateway API. ALB Ingress can directly route to pods via IP mode, bypassing NodePort and preserving source IP — useful for audit logging. Watch for idle timeout settings; ALB defaults to 60 seconds, which terminates long-running gRPC streams unless you raise it. Finally, remember that exposing workloads via Ingress still respects NetworkPolicies, so allow the Ingress controller's namespace to reach backend pods or traffic will be dropped after the Ingress has already accepted it, resulting in 504s that look like backend failures.
Frequently Asked Questions
Do I need a NetworkPolicy for every namespace, or can I rely on Service isolation?
Services provide discovery, not isolation. Without a NetworkPolicy, any pod can reach any Service in the cluster, including kube-system. I recommend a default-deny policy in every production namespace and explicit allow rules for required flows. Start with DNS and Ingress controller access, then add egress to databases and external APIs. Test policies in audit mode if your CNI supports it, such as Cilium's audit or Calico's staged policies, before enforcing.
Should I use Ingress or Gateway API for new clusters?
For new clusters, I choose Gateway API if the controller supports it — Cilium, Traefik, Istio, and the AWS Load Balancer Controller all do. Gateway API gives you role-based separation, header and method matching, and traffic splitting without annotations. If you already operate ingress-nginx and have templated Helm charts, staying on Ingress is reasonable until you need those features. You can run both side-by-side during migration.
Why does my Service have no endpoints even though pods are running?
The Service selector does not match the pod labels. Run kubectl get pods --show-labels -n <namespace> and compare to kubectl get svc <name> -o yaml. A missing or mismatched label, such as app: payments-api vs app: payment-api, results in zero endpoints with no warning event. Also check that pods pass readiness probes; unready pods are removed from Endpoints by default.
How do I preserve the original client IP when traffic goes through a LoadBalancer and Ingress?
Set externalTrafficPolicy: Local on the Ingress controller Service so the node does not SNAT the packet, and configure the controller to use X-Forwarded-For or PROXY protocol. On cloud LBs, enable proxy protocol or target-group IP mode. On NGINX Ingress, set use-proxy-protocol: "true" if the LB sends PROXY protocol, or trust the LB's CIDR with proxy-real-ip-cidr. Without this, your application logs will show the LB or node IP instead of the client.