Kubernetes controllers and reconciliation

You declare desired state; controllers keep acting until reality matches, which shapes how things fail.

Kubernetes is a set of controllers watching declared objects and driving the cluster toward them. A Deployment declares a Pod template and a replica count. Its controller manages ReplicaSets, which create Pods, and the scheduler binds Pods to nodes. Nothing is "run once". Controllers keep reconciling, so deleting a Pod just makes a new one appear.

This model explains the common failures:

  • CrashLoopBackOff: the container starts and exits, and the kubelet restarts it with growing back-off. Use kubectl logs --previous to read the crashed run. A missing ConfigMap key or a bad command is typical.
  • ImagePullBackOff: the image name, tag, or registry credentials are wrong. kubectl describe pod shows the pull error in Events.
  • Pending: the scheduler cannot place the Pod because of insufficient CPU or memory requests, an unsatisfiable node selector, or an unbound PersistentVolumeClaim. Events explain which.
  • Running but no traffic: a Service selects Pods by labels. A selector that matches nothing leaves the Service with no endpoints (kubectl get endpointslices). A failing readiness probe removes Pods from endpoints on purpose.
  • OOMKilled: the container exceeded its memory limit, not the node's memory.

Label selectors are the glue. Services, Deployments, NetworkPolicies, and PodDisruptionBudgets all find their targets by labels, so a typo in one label silently disconnects them.

Change clusters declaratively. Edit the manifest, kubectl apply it, and watch the rollout (kubectl rollout status). Hand-edits with kubectl edit or kubectl patch drift from source control, just like manual changes to Terraform-managed resources.