Skip to content

Day 26 of 100

The Kubernetes Service Has No Endpoints

KubernetesIntermediate15–20 min

🔥 Problem Statement

A Service that's existed for months, unchanged in behavior, suddenly stops routing traffic. Clients get connection errors — not slow responses, not errors from the application, just nothing answering at all. `kubectl get pods` shows the pods Running and Ready, which on the surface looks like everything should be fine.

A recent, seemingly unrelated config update touched some label metadata on the deployment as part of a broader cleanup.

🏗️ Environment

  • Kubernetes: Deployment + Service, recently modified labels
  • Pods: Running and Ready per kubectl get pods
  • Service: exists, selector recently changed in a config update
  • Symptom: connection refused / no route to host from clients

🤔 Your Challenge

If the pods are genuinely Running and Ready, why would the Service still be unable to route any traffic to them?

  • How does a Kubernetes Service actually decide which pods to send traffic to?
  • If pod labels changed, what would that do to whether a Service's selector still matches them?
  • Would kubectl get pods show anything different for a pod that's Ready but not matched by any Service selector?
  • What command would directly show you whether the Service currently has any healthy targets at all?

Solution Hidden

Think through the problem yourself before looking at the answer.

💡Solution

Step 1 — Understand the symptoms

A Service in Kubernetes doesn't route to pods by name or by deployment — it routes to whatever pods currently match its label selector, full stop. kubectl get pods showing Running and Ready says nothing about whether those pods are actually matched by the Service's selector; those are two entirely independent checks; a pod can be perfectly healthy and still be invisible to a Service whose selector no longer matches its labels.

Step 2 — Identify the likely bottleneck

The recent label cleanup is almost certainly the cause. If the deployment's pod template labels were changed — even something that looked cosmetic, like a naming convention update — and the Service's selector wasn't updated to match, the Service now has a selector that matches zero pods. It still exists as an object, it's still configured, it's just pointing at nothing, because nothing currently carries the labels it's looking for.

Step 3 — Investigation

Bash
kubectl get endpoints <service-name>

This is the single most direct check — it shows exactly which pod IPs the Service currently considers valid targets. An empty result, with healthy pods elsewhere in the cluster, is close to definitive confirmation of a selector mismatch.

From there:

  • Compare the Service's spec.selector against the actual labels on the running pods (kubectl get pods --show-labels) — a mismatch, even a single differing key or value, is enough to produce zero matches.
  • Check the diff on the recent label-cleanup change specifically for anything touching the deployment's pod template labels — this is very likely where the mismatch was introduced, even if the change looked unrelated to networking at the time.
  • Confirm pod readiness genuinely is fine — kubectl describe pod on one of the affected pods, to rule out a coincidental, unrelated readiness issue happening at the same time.

Align the Service's selector with the pods' actual current labels — either revert the label change on the deployment, or update the Service's selector to match the new labels, whichever is the intended long-term state. Redeploy and confirm with kubectl get endpoints that the Service now shows healthy targets before considering this resolved.

Going forward, changes to pod template labels are worth treating as networking-relevant changes, not purely cosmetic ones — any Service, NetworkPolicy, or other selector-based object referencing those labels is affected the moment they change, even if the change itself looks unrelated to traffic routing.

Step 5 — Engineering lesson

Kubernetes Services are label-matching machines, not name-based routers, and that indirection is exactly what makes this failure mode so easy to introduce accidentally — a label change made for entirely unrelated reasons (cleanup, standardization) can silently break every Service selector that depended on the old labels, with the pods themselves showing no sign anything is wrong. kubectl get endpoints is the fastest way to cut through pod-level health checks and see what actually matters: whether the Service has anyone to send traffic to at all.

🧠 Today’s Takeaway

A Service can exist perfectly while pointing to nothing.

🔔 Don’t Miss Tomorrow’s Challenge

A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.

Get the useful stuff, not the noise.

Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

No spam. Unsubscribe anytime.