Incident Night shift: 3 incidents on one machine The playbook that always says changed (ansible, about 25 min) The timeout change never took effect (ansible, about 30 min) Two depots ignored the change (ansible, about 35 min) Advisory tests, "for a week", three months ago (cicd, about 30 min) Every outside contributor's PR fails (cicd, about 35 min) The fix passed its tests and did not ship (cicd, about 55 min) The fuel fee nobody can remove (cicd, about 40 min) Shipping labels anyone can download (cloud, about 30 min) The deploy role that can do anything (cloud, about 35 min) 3,112 events vanished in a release (containers, about 30 min) An 800 MB image on a depot link (containers, about 35 min) Docker said healthy for 14 minutes of 500s (containers, about 30 min) Waybill cannot find Ledger in the new stack (containers, about 25 min) Waybill runs as root in its container (containers, about 30 min) Every connection slot is taken (databases, about 30 min) The retention job that deleted depot-3 (databases, about 45 min) Friday evening, everything at once (final-production, about 60 min) Append without erasing (foundations, about 10 min) Clean up scratch without collateral damage (foundations, about 15 min) Count the failed dispatch jobs (foundations, about 15 min) Find the active runbook (foundations, about 15 min) Find the latest handoff (foundations, about 10 min) Fix a script that breaks on spaces (foundations, about 15 min) Free space without losing evidence (foundations, about 15 min) Let on-call run the script (foundations, about 10 min) Restart the right process (foundations, about 15 min) The job that only works by hand (foundations, about 15 min) The release entry point vanished (foundations, about 30 min) Trace the release path (foundations, about 10 min) Which depots are down? (foundations, about 20 min) A staging token in a pull request (git, about 35 min) Ivo's label fix vanished from the release clone (git, about 25 min) Take theirs, the release branch is newer (git, about 20 min) The hotfix that is running nowhere in Git (git, about 50 min) Undo one commit on a shared main (git, about 30 min) v2.5 is running, but which v2.5? (git, about 35 min) "It is just a refactor," says the PR (iac, about 40 min) Locked out by a job that no longer exists (iac, about 35 min) Removing one depot restarts three (iac, about 30 min) 2.0 waits forever to be ready (kubernetes, about 35 min) Deleting the pods made it worse (kubernetes, about 25 min) Green pods, empty Service (kubernetes, about 25 min) Killed at a limit nobody chose (kubernetes, about 35 min) Pending after the node migration (kubernetes, about 20 min) The deploy bot lost one verb (kubernetes, about 30 min) The rollout that never started (kubernetes, about 25 min) /var is full and nothing is in it (linux, about 25 min) CPU pinned since the deploy (linux, about 25 min) It works when Ivo runs it (linux, about 25 min) Killed on a host with 4 GiB free (linux, about 35 min) No space left, 41% used (linux, about 25 min) On-call locked out of the bastion (linux, about 35 min) Sal can read the file but not reach it (linux, about 20 min) Same job, two render nodes, one fails (linux, about 25 min) The 06:00 report never arrived (linux, about 25 min) The log says everything is fine (linux, about 25 min) The port change that did not stick (linux, about 20 min) "Your site has a certificate error" (networking, about 45 min) A VPN update stole the API route (networking, about 35 min) Gatehouse still dials the old port (networking, about 20 min) Healthy on localhost, 502 everywhere else (networking, about 20 min) The certificate expired 75 days early (networking, about 25 min) The network is up and nothing resolves (networking, about 25 min) Valid certificate, broken chain (networking, about 35 min) 22 minutes of failed labels, nobody paged (observability, about 35 min) The alert that could not see (observability, about 35 min) The average says 90 milliseconds (observability, about 40 min) A password on the public website (security, about 25 min) The deploy account is root in all but name (security, about 30 min) Who can still log in as deploy? (security, about 25 min) A 30 second crash became a four minute outage (sre, about 35 min) Monday SLO review (sre, about 25 min) The pager cries wolf (sre, about 35 min) Title Length, minutes Up to 40 players, each on their own machine for the length of the round. Players need no account.
Create the round 