Deleting the pods made it worse INC-2371

Kubernetes is a database of desired state plus controllers that keep making reality match it.

A database and some loops

Kubernetes: Up & Running introduces Kubernetes through three ideas: declarative configuration, self-healing, and immutability. The architecture follows from them:

• The API server is the front door. Every read and write goes through it, and it persists objects in etcd.
• Objects (Deployments, Services, ConfigMaps, and others) each have a spec, what you want, and a status, what was last observed.
• Controllers watch objects and act to make reality match the spec. The Deployment controller manages ReplicaSets, the ReplicaSet controller manages pods, and the endpoint controllers manage Service endpoints.
• The scheduler assigns pending pods to nodes.
• On each node, the kubelet runs the pods assigned to it, runs their probes, and reports status.

No component "runs your app" in the imperative sense. You write desired state, and a chain of controllers converges on it.

Reconciliation changes how you fix things

Because controllers continuously compare and correct, actions outside the spec are temporary:

• Delete a pod owned by a ReplicaSet, and a new one is created from the same template.
• Edit a pod directly, and the change is lost the next time the pod is replaced.
• Scale by hand under a GitOps or autoscaling controller, and the scale is corrected back.

The lesson is to change the input. For a crash loop after a configuration change, that means fixing the ConfigMap or Deployment template, then triggering a rollout if the pods only read the configuration at start. Deleting pods only replays the same failure.

Follow the owners

Every managed object records its owner references. A pod named waybill-7c9d8f-abcde belongs to ReplicaSet waybill-7c9d8f, which belongs to Deployment waybill. kubectl get pod <name> -o jsonpath='{.metadata.ownerReferences}' shows the chain. Fixes almost always belong at the top of the chain.

Evidence lives in the API

Before opening a shell on a node, read what the cluster already recorded:

kubectl get deploy,rs,pods -l app=waybill -o wide
kubectl describe pod <pod>              # conditions, container states, last termination reason, events
kubectl get events --sort-by=.lastTimestamp -n waybill
kubectl logs <pod> --previous           # output from the container instance that just crashed
kubectl get deploy waybill -o yaml      # full spec and status, including conditions

describe and events tell you which component acted and why: FailedScheduling, BackOff, Unhealthy, OOMKilled, FailedMount. logs --previous matters for crash loops, because the current container may have no output yet.

Declarative changes, reviewed

The book and the documentation both recommend declarative object configuration: manifests in version control, applied with kubectl apply or a GitOps controller. That makes the desired state reviewable, diffable (kubectl diff -f), and recoverable, as in the Infrastructure as Code track for infrastructure. Imperative commands remain useful for investigation and emergencies. Write any emergency change back to the source of truth, or the next reconcile will undo it.

Key terms

Desired state

What an object's spec asks for, such as three replicas of this pod template.

Controller

A control loop that watches objects and acts to move the cluster's actual state toward their spec.

Reconciliation

The continuous compare-and-correct cycle controllers perform. Changes you make outside the spec get reverted.

Owner reference

Metadata linking an object to the object that manages it, for example Pod → ReplicaSet → Deployment.

Read further

• Kubernetes Up & Running, 3rd edition, Ch. 1, "Introduction" (Purchase)
  The arguments for declarative configuration, self-healing systems, and immutability. These three ideas explain most of Kubernetes' behaviour.
• Kubernetes Documentation, Concepts, Concepts → Overview → "Kubernetes Components", and Concepts → Cluster Architecture → "Controllers" (Free online (CC BY 4.0))
  The control plane and node components, and the controller pattern (control loops that move current state toward desired state).
• Kubernetes Documentation, Concepts, Concepts → Overview → "Objects In Kubernetes" (Free online (CC BY 4.0))
  Object spec and status, and how the API represents both. Note the difference between imperative commands and declarative object configuration.

Why pods crash-loop, how configuration reaches a container, and what liveness, readiness, and startup probes each decide.

The pod is the unit

A pod is one or more containers that share a network namespace (one IP address, one localhost) and volumes, scheduled together onto one node. Most pods run one application container, sometimes with helpers. The pod manifest is where operational decisions live: image, command, environment, resources, probes, and volumes. Kubernetes: Up & Running's pods chapter covers each of them.

Reading a crash loop

CrashLoopBackOff is not an error itself. It means the container keeps exiting, so the kubelet waits longer before each restart (an exponential back-off, capped at a few minutes). The cause is in the container's last run:

kubectl describe pod <pod>     # Last State: Terminated, Reason: Error, Exit Code: 1
kubectl logs <pod> --previous  # e.g. "unknown template key depot_code in label template depot-7"

Exit codes narrow the search. 1 or another small number usually means the application rejected something, often configuration. 137 is SIGKILL, frequently an OOM kill (the “Resources, limits, and access control” notes). 143 is SIGTERM. When the loop began right after a ConfigMap change, the ConfigMap is the first suspect. Fix it, then restart the pods if they only read configuration at start. Deleting pods without fixing the input reproduces the loop, as the “Objects and the control loop” notes explained.

How configuration reaches a container

A ConfigMap or Secret can be consumed in three ways, and they differ in when changes apply:

Because env-based configuration needs a restart, a common pattern is to roll the Deployment whenever its configuration changes, for example by putting a hash of the ConfigMap in a pod template annotation. Every config change then becomes a normal, observable, roll-back-able rollout.

Three probes, three decisions

The documentation is precise about what the kubelet does with each probe:

• Liveness: on repeated failure, restart the container. Use it only for states a restart fixes, such as a deadlocked event loop or a wedged process. Keep it cheap and independent of dependencies.
• Readiness: on failure, remove the pod from Service endpoints. Nothing is restarted. Use it to say "do not send me traffic right now", for example while warming up, while overloaded, or while a critical dependency is unreachable.
• Startup: until it succeeds, liveness and readiness are not run. Use it for slow starters instead of a long initial delay.

The classic mistake is a liveness probe that checks the database. When the database has a problem, every pod restarts in a loop. The restarts cannot fix the database, they add connection storms, and they turn a degraded service into an outage. The Containers in Production track's health-check lesson applies directly. Probes are decisions other components act on, so each must test what that decision needs.

Probes and rollouts

Readiness also gates rollouts. A Deployment counts a new pod as available only when it is ready. A readiness probe that can never pass on the new version, for example because it points at a path the new version moved, stalls the rollout. The old pods keep serving, so users are fine while the deploy hangs. The “Deployments and rollouts” notes diagnose exactly that.

Key terms

CrashLoopBackOff

A status meaning the container keeps exiting and the kubelet waits, with exponentially increasing delay, before restarting it.

Liveness probe

"Is this container broken beyond recovery?" On failure the kubelet restarts the container.

Readiness probe

"Should this pod receive traffic now?" On failure the pod is removed from Service endpoints, and it is not restarted.

Startup probe

Holds off liveness and readiness checks until a slow-starting application has started.

Read further

• Kubernetes Up & Running, 3rd edition, Ch. 5, "Pods" (Purchase)
  The pod as the unit of scheduling, the pod manifest, health checks (liveness and readiness), resource requests and limits, and volumes.
• Kubernetes Up & Running, 3rd edition, Ch. 13, "ConfigMaps and Secrets" (Purchase)
  The three ways to consume a ConfigMap (environment, command-line arguments, volume), and how updates propagate in each case.
• Kubernetes Documentation, Concepts, Concepts → Configuration → "Liveness, Readiness, and Startup Probes" (Free online (CC BY 4.0))
  The precise semantics of each probe type, and what the kubelet does when each one fails.