Green pods, empty Service

Lab machine

A private machine with its own Kubernetes cluster. Starting takes about 30 seconds. Sessions last up to 60 minutes.

Sasha Lind opened INC-2380 at 09:40SEV-2

This morning every workload moved to the recommended app.kubernetes.io labels. Waybill's pods are Running and Ready. Its Service has no endpoints.

A Service finds its pods by label. Nothing connects them except that match, and changing pod labels does not change the selector.

"Pods are green, so it must be the network. Want me to restart CoreDNS?" (on-call)

Labelling running pods by hand would work until the next pod is created.

Your task

Make the Service select every Ready Waybill pod, including pods created later, by fixing manifests/ and applying it.

On the machine

kubectl get svc waybill -o yaml, kubectl get pods --show-labels
kubectl get endpointslices
manifests/

Timeline

09:00Platform relabels every workload to app.kubernetes.io/*.

09:02Tracking API: 503, no endpoints available for service "waybill".

09:40"Pods are green, so it must be the network. Restart CoreDNS?"

Done when

The Service's ready endpoints equal the Ready Waybill pods.
A replacement pod is selected automatically.
manifests/ matches the cluster.

How it is checked:The checker compares the Service's ready endpoints with the Ready pods, deletes a pod and checks its replacement is selected, requests `/healthz` through the Service, and compares the cluster with `manifests/`.

Passed. Here is how others fixed it

This lab has 2 versions. Reset for a fresh machine, and you may get a different one.

Break it for a friend. Pick the version they get, and the link is copied for you to send: version 1 · version 2

Other fixes that also pass:

Add the Service's old labels back to the Deployment's pod template (not to running pods), so new pods match too.
Select on both `app.kubernetes.io/name` and `app.kubernetes.io/component` for a tighter match.

Fixes that look right but are not:

Adding the old labels to running pods with kubectl label; new pods lack them.
Restarting CoreDNS or kube-proxy when the endpoints list is empty.

Why it broke. A Service finds pods through its label selector, and then reaches them on its targetPort. If the selector matches nothing, or a named targetPort exists on none of the pods, the endpoint list is empty and every request has nowhere to go. Both fail silently until traffic arrives.

How Mara worked it

kubectl get endpointslices -l kubernetes.io/service-name=waybill
No endpoints. Traffic has nowhere to go, though every pod is Ready.
kubectl get svc waybill -o yaml and kubectl get pods --show-labels
Does the selector match the pods' labels? Does the targetPort name a port the pods have?
kubectl get pod <one> -o jsonpath='{.spec.containers[0].ports}'
The container's port names, which a named targetPort must match.
Fix manifests/service.yaml, kubectl apply -f manifests/
The Service follows the convention the pods now use.
kubectl get endpointslices again, and delete one pod
Endpoints match the Ready pods, and a replacement is picked up automatically.

Hints

Hint 1

Pods being Ready says nothing about who selects them, or on which port.

Hint 2

Compare the Service selector with the pod template labels, and its targetPort with the container's port names.

Hint 3

A selector must match every key-value pair it lists. A named targetPort must exist on the pod.

Hint 4

Fix the Service in manifests/service.yaml; labels added to running pods vanish on replacement.

Show the solution

`kubectl get endpointslices` shows no endpoints. Compare the Service with the pods: either its selector still lists the old `app`/`tier` labels (change it to `app.kubernetes.io/name: waybill`), or the selector matches but its `targetPort` names a port the pods no longer have (`web`, now `http`). Fix `manifests/service.yaml`, apply it, and confirm with `kubectl get endpointslices` and a request through the service proxy.

Kubernetes Operations

Labels, Services, and discovery

Services find pods by label selectors. One mismatched label leaves healthy pods with no traffic.

Labels are the glue

Kubernetes objects find each other through labels and selectors, not by name. A Deployment's selector says which pods its ReplicaSets own. A Service's selector says which pods receive its traffic. A NetworkPolicy's selector says which pods a rule applies to. Kubernetes: Up & Running stresses how much flexibility this gives, because any object can be grouped any way. It also makes a single label mismatch silent. Nothing errors. The Service simply selects zero pods.

kubectl get svc waybill -o jsonpath='{.spec.selector}'; echo
kubectl get pods -n waybill --show-labels
kubectl get pods -n waybill -l app=waybill            # does the Service's selector match anything?
kubectl get endpointslices -n waybill -l kubernetes.io/service-name=waybill

If the selector query returns nothing, so does the Service, however green the pods look.

From Service to pods

A Service gives a set of pods a stable virtual IP (the ClusterIP) and DNS name. The control plane continuously maintains EndpointSlices listing the addresses of pods that match the selector, marked by readiness. On each node, kube-proxy (or an eBPF data plane) programs the rules that send ClusterIP traffic to ready endpoints.

Two conditions must hold for a pod to receive traffic: it matches the selector and it is ready. "No endpoints" means the first condition failed. Endpoints that are all marked not ready point at the second, which is the “Pods, probes, and configuration” notes’ readiness probes.

DNS names

Cluster DNS gives every Service a name, <service>.<namespace>.svc.<cluster-domain>, usually with cluster.local as the domain. Pods get search domains for their own namespace, so waybill works from inside the namespace and waybill.waybill from elsewhere. The Networking for Operators track's ndots discussion explains the extra queries you see in DNS logs. Test from inside the cluster, not from your laptop:

kubectl run -it --rm dnsprobe --image=busybox:1.36 --restart=Never -- nslookup waybill.waybill

Migrating labels safely

Label migrations, such as adopting the recommended app.kubernetes.io/* labels, are a common source of empty Services. The safe sequence is expand and contract once more:

Add the new labels to pod templates alongside the old ones, and roll out.
Switch selectors (Services, NetworkPolicies, monitors) to the new labels.
Remove the old labels in a later change.

Deployment selectors are immutable in apps/v1, so changing one means creating a new Deployment, a larger operation to plan separately.

Getting traffic in

ClusterIP (default): reachable only inside the cluster.
NodePort: also exposed on a port on every node.
LoadBalancer: also asks the cloud provider for an external load balancer.
Ingress / Gateway API: HTTP routing by host and path to Services, implemented by a controller (often a proxy such as those in the Networking for Operators track).

Whatever the entry point, the last hop is a Service and its endpoints, so the selector check above belongs in every "the app is unreachable" investigation.

Node labels and placement

Pods are matched to nodes the same way. A nodeSelector or a required node affinity in the pod template lists node labels that must be present, and the scheduler only considers nodes that have them. If no node matches, the pod stays Pending, and kubectl describe pod ends with an event such as "0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector". Nothing else warns you. Renaming node labels, during a migration or a new node pool, strands every workload that still asks for the old name. Fix the workload's selector in its manifest rather than putting the old label back on nodes, which quietly undoes the rename for everyone who already moved.

Key terms

Label: A key/value pair on an object, such as app.kubernetes.io/name=waybill, used for selection.
Selector: A query over labels. A Service's selector decides which pods receive its traffic.
EndpointSlice: The objects listing the ready pod addresses currently behind a Service.
ClusterIP: A stable virtual IP for a Service inside the cluster, with a matching DNS name.

Read further

Kubernetes Up & Running, 3rd edition, Ch. 6, "Labels and Annotations" (Purchase)
Label syntax, selector operators, and how labels connect objects. Note the difference between labels, which are for selection, and annotations, which hold arbitrary metadata.
Kubernetes Up & Running, 3rd edition, Ch. 7, "Service Discovery" (Purchase)
Service objects, cluster IPs, DNS names, readiness checks and endpoints, and the section on looking beyond the cluster.
Kubernetes Documentation, Concepts, Concepts → Services, Load Balancing, and Networking → "Service" and "DNS for Services and Pods" (Free online (CC BY 4.0))
Selectors and EndpointSlices, Service types, and the DNS naming scheme (<service>.<namespace>.svc.<cluster-domain>).

Green pods, empty Service INC-2380