Pending after the node migration

Lab machine

A private machine with its own Kubernetes cluster. Starting takes about 30 seconds. Sessions last up to 60 minutes.

Mara Okafor opened INC-2524 at 07:40SEV-2

Last night's node migration renamed the pool labels. Every node is Ready, but the edge copy of Waybill has been Pending for forty minutes and the depot scanners are queueing offline.

waybill-edge serves the depot scanners and must run on the edge pool, next to the depot VPN gateways. The migration moved pool labels to the northstar.example/ prefix, and other teams have already switched to the new name.

"Migration went fine, every node is Ready. I renamed the pool labels to the northstar.example/ prefix like we agreed. Didn't touch any workloads." (Sasha)

A Pending pod is waiting for the scheduler to find it a node. The scheduler writes down why it could not.

Your task

Get both waybill-edge pods running on the edge pool again by fixing the workload in manifests/ and applying it. Leave the node labels as the migration left them.

On the machine

kubectl get pods, kubectl describe pod <name>
kubectl get nodes --show-labels
manifests/

Timeline

01:00Node migration CHG-4455 renames pool labels to northstar.example/pool.

07:00waybill-edge is redeployed. Both pods stay Pending.

07:25"Every node is Ready. Didn't touch any workloads." (Sasha)

07:40INC-2524 lands with you.

Done when

Both waybill-edge pods are Running and Ready, and the Service answers.
The pods can still only run on edge-pool nodes.
No node carries the retired `pool` label.
manifests/ matches the cluster.

How it is checked:The checker rejects nodes that carry the retired `pool` label and a Deployment with no placement rule. It waits for the rollout, requires both pods Ready and `/readyz` through the Service, checks that the pods can only land on `northstar.example/pool=edge` nodes, and compares the placement in `manifests/` with the cluster.

Passed. Here is how others fixed it

This lab has 2 versions. Reset for a fresh machine, and you may get a different one.

Break it for a friend. Pick the version they get, and the link is copied for you to send: version 1 · version 2

Other fixes that also pass:

Replace the nodeSelector with a required node affinity on `northstar.example/pool In [edge]`.

Fixes that look right but are not:

Labelling the nodes `pool=edge` again, which brings back the label the migration retired.
Deleting the nodeSelector, which lets scanner traffic land on any pool.
Patching the live Deployment and leaving `manifests/` stale.

Why it broke. The Deployment still asked for `pool=edge`, a label no node has since the migration, so the scheduler had nowhere to put the pods. Nothing was broken about the nodes. The workload's placement rule pointed at a name that no longer exists, and Kubernetes does not warn about a selector that matches nothing until pods need a home.

How Mara worked it

kubectl get pods, then kubectl describe pod <pending pod>
The last event is the scheduler explaining why no node fits.
kubectl get deploy waybill-edge -o yaml | grep -n -A8 -E 'nodeSelector|affinity'
Find every rule that constrains placement, wherever it is written.
kubectl get nodes --show-labels
Compare what the pods ask for with what the nodes have now.
Change that rule in manifests/deployment.yaml to northstar.example/pool = edge
The workload follows the migration, not the other way round.
kubectl apply -f manifests/ and kubectl get pods -w
Watch the new pods schedule and turn Ready.

Hints

Hint 1

`kubectl describe pod` ends with the scheduler's reason. Read it before changing anything.

Hint 2

Compare every placement rule in the pod spec (nodeSelector and nodeAffinity) with `kubectl get nodes --show-labels`.

Hint 3

Putting the old label back on nodes undoes the migration other teams already rely on.

Hint 4

Change the placement rule in `manifests/deployment.yaml` to the new label key, then apply it.

Show the solution

`kubectl describe pod` says no node matches the pod's node selector or affinity. The pods still ask for the retired `pool=edge`, either in `nodeSelector` or in a required `nodeAffinity` term. Change that rule in `manifests/deployment.yaml` to the new key, `northstar.example/pool: edge`, apply it, and watch the new pods schedule with `kubectl get pods -w`.

Kubernetes Operations

Labels, Services, and discovery

Services find pods by label selectors. One mismatched label leaves healthy pods with no traffic.

Labels are the glue

Kubernetes objects find each other through labels and selectors, not by name. A Deployment's selector says which pods its ReplicaSets own. A Service's selector says which pods receive its traffic. A NetworkPolicy's selector says which pods a rule applies to. Kubernetes: Up & Running stresses how much flexibility this gives, because any object can be grouped any way. It also makes a single label mismatch silent. Nothing errors. The Service simply selects zero pods.

kubectl get svc waybill -o jsonpath='{.spec.selector}'; echo
kubectl get pods -n waybill --show-labels
kubectl get pods -n waybill -l app=waybill            # does the Service's selector match anything?
kubectl get endpointslices -n waybill -l kubernetes.io/service-name=waybill

If the selector query returns nothing, so does the Service, however green the pods look.

From Service to pods

A Service gives a set of pods a stable virtual IP (the ClusterIP) and DNS name. The control plane continuously maintains EndpointSlices listing the addresses of pods that match the selector, marked by readiness. On each node, kube-proxy (or an eBPF data plane) programs the rules that send ClusterIP traffic to ready endpoints.

Two conditions must hold for a pod to receive traffic: it matches the selector and it is ready. "No endpoints" means the first condition failed. Endpoints that are all marked not ready point at the second, which is the “Pods, probes, and configuration” notes’ readiness probes.

DNS names

Cluster DNS gives every Service a name, <service>.<namespace>.svc.<cluster-domain>, usually with cluster.local as the domain. Pods get search domains for their own namespace, so waybill works from inside the namespace and waybill.waybill from elsewhere. The Networking for Operators track's ndots discussion explains the extra queries you see in DNS logs. Test from inside the cluster, not from your laptop:

kubectl run -it --rm dnsprobe --image=busybox:1.36 --restart=Never -- nslookup waybill.waybill

Migrating labels safely

Label migrations, such as adopting the recommended app.kubernetes.io/* labels, are a common source of empty Services. The safe sequence is expand and contract once more:

Add the new labels to pod templates alongside the old ones, and roll out.
Switch selectors (Services, NetworkPolicies, monitors) to the new labels.
Remove the old labels in a later change.

Deployment selectors are immutable in apps/v1, so changing one means creating a new Deployment, a larger operation to plan separately.

Getting traffic in

ClusterIP (default): reachable only inside the cluster.
NodePort: also exposed on a port on every node.
LoadBalancer: also asks the cloud provider for an external load balancer.
Ingress / Gateway API: HTTP routing by host and path to Services, implemented by a controller (often a proxy such as those in the Networking for Operators track).

Whatever the entry point, the last hop is a Service and its endpoints, so the selector check above belongs in every "the app is unreachable" investigation.

Node labels and placement

Pods are matched to nodes the same way. A nodeSelector or a required node affinity in the pod template lists node labels that must be present, and the scheduler only considers nodes that have them. If no node matches, the pod stays Pending, and kubectl describe pod ends with an event such as "0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector". Nothing else warns you. Renaming node labels, during a migration or a new node pool, strands every workload that still asks for the old name. Fix the workload's selector in its manifest rather than putting the old label back on nodes, which quietly undoes the rename for everyone who already moved.

Key terms

Label: A key/value pair on an object, such as app.kubernetes.io/name=waybill, used for selection.
Selector: A query over labels. A Service's selector decides which pods receive its traffic.
EndpointSlice: The objects listing the ready pod addresses currently behind a Service.
ClusterIP: A stable virtual IP for a Service inside the cluster, with a matching DNS name.

Read further

Kubernetes Up & Running, 3rd edition, Ch. 6, "Labels and Annotations" (Purchase)
Label syntax, selector operators, and how labels connect objects. Note the difference between labels, which are for selection, and annotations, which hold arbitrary metadata.
Kubernetes Up & Running, 3rd edition, Ch. 7, "Service Discovery" (Purchase)
Service objects, cluster IPs, DNS names, readiness checks and endpoints, and the section on looking beyond the cluster.
Kubernetes Documentation, Concepts, Concepts → Services, Load Balancing, and Networking → "Service" and "DNS for Services and Pods" (Free online (CC BY 4.0))
Selectors and EndpointSlices, Service types, and the DNS naming scheme (<service>.<namespace>.svc.<cluster-domain>).

Pending after the node migration INC-2524