Kubernetes City. Operate declarative workloads and diagnose cluster failures.
7 labs, easiest first0 / 7
Pending after the node migrationINC-2524Last night's node migration renamed the pool labels. Every node is Ready, but the edge copy of Waybill has been Pending for forty minutes and the depot scanners are queueing offline.EasyFix 20 min · Linux + Kubernetes
Deleting the pods made it worseINC-237110:02, a config change for depot-7. 10:05, the pods did not pick it up, so on-call deleted them. 10:06, every Waybill pod is crash-looping.EasyFix 25 min · Linux + Kubernetes
Green pods, empty ServiceINC-2380This morning every workload moved to the recommended app.kubernetes.io labels. Waybill's pods are Running and Ready. Its Service has no endpoints.EasyFix 25 min · Linux + Kubernetes
The rollout that never startedINC-2530Ivo applied Waybill 2.0.0 forty minutes ago. `kubectl rollout status` is still waiting, every pod is still 1.4.0, and no new pod has even been created.MediumFix 25 min · Linux + Kubernetes
The deploy bot lost one verbSEC-131Last week the deploy bot's cluster-wide edit role was replaced with a narrow Role. Since then every Waybill deploy fails. The release channel wants cluster-admin.MediumFix 30 min · Linux + Kubernetes
2.0 waits forever to be readyDEP-812The 2.0.0 deploy has waited on rollout status for 20 minutes and CI gave up. Old pods still serve. The release channel wants to delete the readiness probe.HardFix 35 min · Linux + Kubernetes
Killed at a limit nobody choseINC-2388Docklight moved to Kubernetes last week. Its worker restarts every few seconds with exit code 137, and on-call wants to shrink the batch size the carrier contract fixes.HardInvestigate 35 min · Linux + Kubernetes
