Pods, probes, and configuration
Why pods crash-loop, how configuration reaches a container, and what liveness, readiness, and startup probes each decide.
The pod is the unit
A pod is one or more containers that share a network namespace (one IP address, one localhost) and volumes, scheduled together onto one node. Most pods run one application container, sometimes with helpers. The pod manifest is where operational decisions live: image, command, environment, resources, probes, and volumes. Kubernetes: Up & Running's pods chapter covers each of them.
Reading a crash loop
CrashLoopBackOff is not an error itself. It means the container keeps exiting, so the kubelet waits longer before each restart (an exponential back-off, capped at a few minutes). The cause is in the container's last run:
kubectl describe pod <pod> # Last State: Terminated, Reason: Error, Exit Code: 1
kubectl logs <pod> --previous # e.g. "unknown template key depot_code in label template depot-7"
Exit codes narrow the search. 1 or another small number usually means the application rejected something, often configuration. 137 is SIGKILL, frequently an OOM kill (the “Resources, limits, and access control” notes). 143 is SIGTERM. When the loop began right after a ConfigMap change, the ConfigMap is the first suspect. Fix it, then restart the pods if they only read configuration at start. Deleting pods without fixing the input reproduces the loop, as the “Objects and the control loop” notes explained.
How configuration reaches a container
A ConfigMap or Secret can be consumed in three ways, and they differ in when changes apply:
| Consumption | Takes effect on change |
|---|---|
Environment variables (env, envFrom) |
Only when the container restarts |
| Command-line arguments built from env | Only when the container restarts |
| Mounted volume | Files update eventually in place (not with subPath), and the app must re-read them |
Because env-based configuration needs a restart, a common pattern is to roll the Deployment whenever its configuration changes, for example by putting a hash of the ConfigMap in a pod template annotation. Every config change then becomes a normal, observable, roll-back-able rollout.
Three probes, three decisions
The documentation is precise about what the kubelet does with each probe:
- Liveness: on repeated failure, restart the container. Use it only for states a restart fixes, such as a deadlocked event loop or a wedged process. Keep it cheap and independent of dependencies.
- Readiness: on failure, remove the pod from Service endpoints. Nothing is restarted. Use it to say "do not send me traffic right now", for example while warming up, while overloaded, or while a critical dependency is unreachable.
- Startup: until it succeeds, liveness and readiness are not run. Use it for slow starters instead of a long initial delay.
The classic mistake is a liveness probe that checks the database. When the database has a problem, every pod restarts in a loop. The restarts cannot fix the database, they add connection storms, and they turn a degraded service into an outage. The Containers in Production track's health-check lesson applies directly. Probes are decisions other components act on, so each must test what that decision needs.
Probes and rollouts
Readiness also gates rollouts. A Deployment counts a new pod as available only when it is ready. A readiness probe that can never pass on the new version, for example because it points at a path the new version moved, stalls the rollout. The old pods keep serving, so users are fine while the deploy hangs. The “Deployments and rollouts” notes diagnose exactly that.
Key terms
- CrashLoopBackOff
- A status meaning the container keeps exiting and the kubelet waits, with exponentially increasing delay, before restarting it.
- Liveness probe
- "Is this container broken beyond recovery?" On failure the kubelet restarts the container.
- Readiness probe
- "Should this pod receive traffic now?" On failure the pod is removed from Service endpoints, and it is not restarted.
- Startup probe
- Holds off liveness and readiness checks until a slow-starting application has started.
Read further
- Kubernetes Up & Running, 3rd edition, Ch. 5, "Pods" (Purchase)
The pod as the unit of scheduling, the pod manifest, health checks (liveness and readiness), resource requests and limits, and volumes. - Kubernetes Up & Running, 3rd edition, Ch. 13, "ConfigMaps and Secrets" (Purchase)
The three ways to consume a ConfigMap (environment, command-line arguments, volume), and how updates propagate in each case. - Kubernetes Documentation, Concepts, Concepts → Configuration → "Liveness, Readiness, and Startup Probes" (Free online (CC BY 4.0))
The precise semantics of each probe type, and what the kubelet does when each one fails.