State, configuration, and health
Containers are disposable, so data must live elsewhere, configuration must come from outside, and health must mean "can serve".
Containers are cattle, data is not
Twelve-Factor's disposability and processes factors describe the ideal container. It starts fast, shuts down gracefully on SIGTERM, and holds no state that matters. Orchestrators lean on that. They recreate containers on every configuration change, image update, node drain, or failure. Anything stored in the container's writable layer disappears with it.
Recreate is not restart. docker restart keeps the same container and its writable layer. docker compose up after a change, or any orchestrator rollout, creates a new container. Data that "survived restarts for months" can vanish on the first release that changes the image tag. That is the lab's incident.
Where state belongs
- Named volumes have a lifecycle independent of containers. They survive recreate and
down/up, and are removed only bydocker volume rmordocker compose down -v. That last command should never appear in a routine runbook. - Bind mounts map a host path. They are useful for configuration and development and tie the container to one host.
- Backing services (a managed database, an object store, a queue) take state out of the container entirely, which is the Twelve-Factor preference.
Durability is a promise about acknowledgement
A volume is only half of durability. DDIA's discussion of ACID defines durability precisely. Once a write is acknowledged, it survives a crash. On one node that requires writing to non-volatile storage, typically by appending to a write-ahead log and calling fsync, before replying. A queue that keeps items in memory and writes them out every minute can lose up to a minute of acknowledged work, volume or not. When you assess a stateful container, ask both questions: does the data live outside the writable layer, and is it on disk before the client hears "OK"?
Configuration comes from outside
The config factor says to keep everything that varies between deploys (URLs, credentials, feature flags) out of the code and the image, and supply it at run time. Environment variables are the simplest channel. Mounted files work well for larger configuration and for secrets. Container Security notes that environment variables can leak through docker inspect, crash dumps, and child processes. Many platforms therefore mount secrets as files with tight permissions. Either way, nothing secret is ever built into an image layer.
Logs go to stdout
The logs factor says a process writes its event stream to stdout and stderr and lets the platform route it. In containers this is what makes docker logs, kubectl logs, and log collectors work. Logs written to files inside the container are lost on recreate and invisible to the platform.
Health checks are decisions, not decorations
A runtime health check (HEALTHCHECK in a Dockerfile, healthcheck: in Compose, probes in Kubernetes) produces a signal that other systems act on. Dependants wait for it, load balancers route by it, and orchestrators restart on it. A check that tests the wrong thing makes those systems wrong:
- Process exists (
pgrep), or port is open: true while the service fails every request. - Endpoint answers 200 without touching dependencies: true while the database is unreachable.
- Endpoint that checks the critical path cheaply: for example, "can I get a connection from the pool and run
SELECT 1?". This tells you whether the instance can serve.
Avoid the opposite failure too. A check that calls every downstream service turns one dependency's outage into every instance reporting unhealthy, and an orchestrator that restarts all of them makes things worse. The Kubernetes Operations track separates this into liveness ("restart me") and readiness ("send me traffic"), which is the cleanest way to express both intents.
Key terms
- Writable layer
- The container's private top layer. It is deleted when the container is removed or recreated.
- Named volume
- Storage managed by the runtime with a lifecycle independent of any container. It survives recreate and
down/upunless explicitly removed. - Disposability
- Twelve-Factor's demand that processes start fast and shut down gracefully, so they can be replaced at any time.
- Health check
- A command the runtime runs periodically. Its result marks the container healthy or unhealthy for orchestration and humans.
Read further
- The Twelve-Factor App, Factors III "Config", VI "Processes", IX "Disposability", and XI "Logs" (Free online)
Stateless processes that keep persistent data in backing services, fast start-up and graceful shutdown, configuration in the environment, and logs as event streams to stdout. - Designing Data-Intensive Applications, Ch. 7, "Transactions" (Purchase)
The section on the meaning of ACID, specifically durability. Note what a database must have done before it can promise that an acknowledged write survives a crash. - Container Security, Chapter "Passing Secrets to Containers" (Purchase)
The trade-offs between environment variables and mounted files for secrets, and why secrets must never be part of an image.