A 30 second crash became a four minute outage

Lab machine

A private Linux machine with the problem already set up. Sessions last up to 60 minutes.

Mara Okafor opened OPS-2331 at 11:00task

Ledger crashed for 30 seconds at 09:40. It came back, and then stayed down for four more minutes under a flood of retries from our own clients.

Two thousand Waybill clients call Ledger. When Ledger crashed, each one retried every 200 milliseconds, forever. When Ledger came back, it faced every request from the outage at once, failed them, and they were retried again.

"Ledger was down for 30 seconds. We were down for four minutes, and the other three and a half were us." (Mara)

Read clients/retry.json and the incident file before assuming what the clients do today: a policy that ends the storm by giving up early fails the depots in a different way.

bin/storm-sim replays the outage against a retry policy and draws the load Ledger saw. It is a model, not a load test, but its arithmetic is the same arithmetic that took Ledger down.

Your task

Change clients/retry.json so that, in the replay, Ledger recovers within a minute of coming back, fewer than 10% of failed requests are abandoned, and retries are bounded.

On the machine

incident/NW-330.md
clients/retry.json and clients/README.md
bin/storm-sim

Timeline

09:40:00Ledger crashes during compaction; its supervisor restarts it.

09:40:30Ledger is back. Error rate stays at 100%.

09:43:10Waybill clients blocked at the firewall. Ledger recovers in seconds.

11:00OPS-2331: fix the retry policy before the next blip.

Done when

In the replay, Ledger recovers within 60 seconds of coming back.
Fewer than 10% of the requests that failed are abandoned.
Retries are bounded: 2 to 12 attempts, and no single wait longer than 60 seconds.

How it is checked:The checker replays the outage against your policy with the same model as bin/storm-sim, and checks the recovery time, the share of abandoned requests, and the attempt and wait bounds.

Passed. Here is how others fixed it

This lab has 2 versions. Reset for a fresh machine, and you may get a different one.

Break it for a friend. Pick the version they get, and the link is copied for you to send: version 1 · version 2

Other fixes that also pass:

Fewer, longer waits: 7 attempts, 2000 ms base, multiplier 2, 30000 ms cap.
Equal jitter with 8 to 10 attempts. Jitter matters most when many clients fail at the same instant and would otherwise retry in lockstep.

Fixes that look right but are not:

Capping attempts at 3 with short waits, which recovers Ledger by abandoning most requests during the outage.
Adding backoff but keeping max_attempts at -1, so a dead dependency is retried forever.

Why it broke. Retrying every 200 ms forever makes offered load grow with every second of outage, so it never falls below capacity. Cutting retries to a few quick tries ends the storm by dropping the work instead. Exponential backoff lets the backlog drain while the waits add up to more than the outage, a cap keeps any one request from waiting forever, and a bounded attempt count means a truly dead dependency is not hammered indefinitely.

How Mara worked it

cat incident/NW-330.md and clients/retry.json
What the clients run today, and what it did the last time Ledger blinked.
bin/storm-sim
Reproduce the failure before changing anything: the storm, or the abandoned requests.
Exponential backoff with jitter, a cap on the wait, and a limited number of attempts
Clients spread out, wait long enough to outlast the outage, and give up eventually.
bin/storm-sim after each change
Ledger recovers within a minute, and few requests are abandoned.

Hints

Hint 1

Run bin/storm-sim before changing anything. What happens to load right after the outage?

Hint 2

A multiplier above 1 makes each wait longer than the last (exponential backoff).

Hint 3

Add up the waits: they need to cover more than the 30 second outage, or requests give up too early.

Hint 4

Something like 8 attempts, 1000 ms base, multiplier 2, 30000 ms cap works. Try jitter settings and compare.

Show the solution

Run bin/storm-sim on the current policy: retrying forever with no backoff keeps Ledger overloaded for minutes, and three quick tries abandon almost every request during the outage. Set max_attempts to about 8 to 10, base_delay_ms 1000, multiplier 2, max_delay_ms 30000, and a jitter setting. The waits grow to cover the 30 second outage, the backlog drains instead of compounding, and Ledger recovers within about half a minute.

Site Reliability Engineering

Stability patterns and overload

Integration points fail, and the system must not fail with them. Timeouts, circuit breakers, bulkheads, back pressure, and load shedding.

Integration points are where systems die

Release It! starts from production experience: the most dangerous parts of a system are its integration points, every call to another service, database, or API. Remote calls can fail fast, fail slowly, hang, return garbage, or succeed and take forever. The book's stability antipatterns describe how one bad integration point spreads:

Blocked threads: callers wait on a slow dependency until every worker is stuck.
Chain reactions: one instance fails, its load shifts to its peers, and they fail in turn.
Cascading failures: a failure in one layer propagates to the layers calling it.
Slow responses: worse than fast failures, because they tie up resources on both sides.
Unbounded result sets: a query that returned 100 rows in testing returns ten million in production.

The common thread is a small problem in one place consuming a shared resource (threads, connections, memory, capacity) that everything else needs.

Patterns that contain failure

Each stability pattern counters one or more antipatterns:

Timeouts. Every remote call needs one, sized to the caller's own deadline. Without timeouts, every other pattern fails, because a hung call holds its thread forever. Propagate deadlines downstream (the SRE book recommends it) so work abandoned by the client is not still being done three services deep.

Retries, carefully. Retrying a transient failure is fine. Retrying immediately, many times, at every layer multiplies load during exactly the moment the system is weakest. Use exponential back-off with jitter, a small maximum, a retry budget (for example, at most 10% extra requests), and retries at one layer only.

Circuit breaker. Track failures to a dependency. When they cross a threshold, open the circuit and fail calls immediately. After a cool-off, let a trial request through (half-open). If it succeeds, close the circuit. The caller saves its resources, and the dependency gets breathing room. Combine with a fallback, such as cached data or a degraded feature, where one exists.

Bulkheads. Partition resources so a failure stays local: separate pools for Ledger calls and for label rendering, separate instances for critical and batch traffic. A flood in one compartment does not sink the ship.

Fail fast. If a request cannot succeed, for example because a dependency's circuit is open or a required resource is missing, reject it at once instead of doing partial work.

Steady state. Anything that accumulates (logs, temp files, sessions, caches, queues) needs an automatic bound or purge. Otherwise the system eventually fails from its own debris, as in the Linux Administration track's notes on disk-full and inode incidents.

Overload and back pressure

The SRE book's chapters on overload and cascading failures add the capacity view. As load approaches capacity, latency climbs, requests time out, clients retry, and the effective load rises further, a positive feedback loop that can take down a healthy fleet. The defences:

Load shedding: when overloaded, reject excess requests early and cheaply, so the requests you accept still finish within deadline. Throughput of successful requests (goodput) stays high.
Back pressure: bounded queues that push "slow down" upstream instead of buffering indefinitely.
Criticality: tag requests by importance (label printing above analytics) and shed the least critical first.
Client-side throttling: clients that see many rejections reduce their own request rate.
Graceful degradation: serve a reduced but useful response, such as tracking without the map, rather than none.

Health checks are part of stability

The Containers in Production track and the Kubernetes Operations track warned against health checks that fail on dependency problems. In stability terms, such checks cause chain reactions. Every instance is marked unhealthy, restarted, or removed at once, and a partial outage becomes total. A good design checks what the instance itself controls, reports dependency trouble through readiness or degraded responses, and lets circuit breakers handle the dependency.

Reviewing a service

For each dependency of a service, ask: Is there a timeout, and is it shorter than our own deadline? What is the retry policy? Is there a circuit breaker and a fallback? Is the connection pool isolated? What happens to our users when this dependency is slow rather than down? Writing the answers down for Waybill, covering Ledger, the label renderer, and the database, is a worthwhile exercise. It is also the most effective reliability review most teams never do.

Key terms

Timeout: A bound on how long to wait for a remote call. Without one, a slow dependency holds your resources indefinitely.
Circuit breaker: Stops calling a failing dependency after repeated errors, fails fast for a while, then probes to see if it has recovered.
Bulkhead: Partitioning resources (thread pools, connection pools, instances) so a failure in one part cannot exhaust all of them.
Load shedding: Deliberately rejecting some work when overloaded, so the rest can succeed.

Read further

Release It!, 2nd edition, Ch. 4, "Stability Antipatterns", and Ch. 5, "Stability Patterns" (Purchase)
Integration points, chain reactions, cascading failures, blocked threads, slow responses, and unbounded result sets. Then timeouts, circuit breaker, bulkheads, steady state, fail fast, shed load, and create back pressure. For each pattern, note which antipattern it counters.
Site Reliability Engineering, Ch. 22, "Addressing Cascading Failures" (Free to read online (CC BY-NC-ND 4.0))
Causes of cascading failure (server overload, resource exhaustion, retries), preventing overload, and the guidance on retries and deadlines.
Site Reliability Engineering, Ch. 21, "Handling Overload" (Free to read online (CC BY-NC-ND 4.0))
Per-customer limits, client-side throttling, criticality, and why handling overload gracefully beats trying never to be overloaded.

A 30 second crash became a four minute outage OPS-2331