Every lab starts a private Linux machine with the problem already in place. Search by symptom, tool, or track.
73 labs
Count the failed dispatch jobsOPS-2215Shell basics · The carrier billed Northstar for every failed dispatch attempt. Finance wants to dispute it with the real number: distinct jobs that failed, not attempts.EasyDo 15 min · Linux
Find the latest handoffOPS-2207Shell basics · Your first morning. Mara is taking over from the night shift and needs the incident they were working on. Three handoff notes, numbered out of order.EasyDo 10 min · Linux
Fix a script that breaks on spacesOPS-2251Shell basics · The weekly package report says West Dock 2 handled 999 parcels. It handled 66. The script reads three files that are not the one you asked for.EasyDo 15 min · Linux
Let on-call run the scriptOPS-2234Shell basics · Every morning at 07:00 the on-call engineer rotates the label printer queues. This morning Mara got "Permission denied" on the script.EasyDo 10 min · Linux
Append without erasingOPS-2219Shell basics · Last week someone wiped three months of depot health history with one character. The auditors arrive on Thursday. Add today's line without losing anything.EasyDo 10 min · Linux
Trace the release pathOPS-2211Shell basics · Ivo found two files called deploy and nearly ran the wrong one. Before the 10:00 release he wants to know which file the runbook actually runs.EasyDo 10 min · Linux
Which depots are down?OPS-2262Shell basics · The public status page died in last night's upgrade. Customer service opens at 08:00 and needs to know which depots to warn callers about.EasyInvestigate 20 min · Linux
Free space without losing evidenceOPS-2230Shell basics · The release volume is at 91% and climbing. Waybill crashed at 04:47, and Ivo needs whatever it left behind to find out why.EasyInvestigate 15 min · Linux
Restart the right processOPS-2240Shell basics · Waybill on depot-west stopped serving at 09:44. Next to it, a batch job on the manifests database has been running for three hours and must not be interrupted.EasyInvestigate 15 min · Linux
The job that only works by handOPS-2246Shell basics · The overnight manifest sync has failed three nights in a row, so depots start the day with yesterday's manifests. When Ivo runs it by hand, it works.EasyInvestigate 15 min · Linux
Clean up scratch without collateral damageOPS-2255Shell basics · The import job left temp files everywhere. The last person to clean up deleted the only backup of a manifest along with them.EasyDo 15 min · Linux
Find the active runbookOPS-2223Shell basics · Last night's page sent Tomas to a runbook for a service that was retired a year ago. He spent twenty minutes restarting something that does not exist.EasyInvestigate 15 min · Linux
The release entry point vanishedINC-2420Shell basics · 17:40, end of your first day. The 18:00 release is blocked. `current/bin/deploy` says No such file or directory, the script is right there in releases/, and Dmitri swears nobody touched current.MediumBoss incident 30 min · Linux
Sal can read the file but not reach itOPS-771Linux · Month-end close starts tomorrow. Since last night's archive migration, finance cannot open a single depot report, even though every file is group-readable.EasyFix 20 min · Linux, root
The port change that did not stickINC-2512Linux · The depot-4 printers moved to port 9105 overnight. Tomas changed the spooler's port in its unit and restarted it twice. It still listens on 9100, and the 06:00 trucks need their labels.EasyFix 20 min · Linux
The 06:00 report never arrivedINC-2331Linux · Every morning at 06:30 the depot-6 manager reads the Docklight delivery summary at stand-up. Today there was nothing to read.MediumFix 25 min · Linux
/var is full and nothing is in itINC-2304Linux · depot-2 cannot print shipping labels. /var is at 100%, the night shift deleted the 3.9 GB log, and df still says 100%.MediumInvestigate 25 min · Linux
No space left, 41% usedINC-2317Linux · The depot-4 spooler rejects every new manifest with "No space left on device". df says the spool is 41% full.MediumInvestigate 25 min · Linux
The log says everything is fineINC-2412Linux · Scanners at depot-8 keep dropping their connection. Waybill's restart counter climbs, and its log file looks spotless.MediumInvestigate 25 min · Linux
Same job, two render nodes, one failsOPS-814Linux · render-a printed last night's labels. render-b failed on the same job with an option it no longer understands.MediumInvestigate 25 min · Linux, root
It works when Ivo runs itINC-2291Linux · depot-3 was patched and rebooted at 05:40. Waybill has not come back under the service manager, and the fix Ivo made by hand lasted twenty minutes.MediumFix 25 min · Linux
CPU pinned since the deployALERT-7730Linux · depot-1's CPU has been at 100% since the 09:10 Waybill deploy. Someone in chat wants to pkill -f waybill.MediumRecover 25 min · Linux
On-call locked out of the bastionOPS-802Linux · Since the directory migration, Mara and Teo cannot SSH into the depot bastion. The deploy robot still can. Mara is on call tonight.HardFix 35 min · Linux, root
Killed on a host with 4 GiB freeINC-2399Linux · Carrier batches have stopped leaving depot-6. The Docklight worker restarts every few seconds, on a host the dashboard calls idle.HardInvestigate 35 min · Linux
Healthy on localhost, 502 everywhere elseCHG-5120Networking · Waybill moved to a new host this morning and the change record says the health check passed. Every scanner request through the depot proxy gets 502.EasyFix 20 min · Linux
Gatehouse still dials the old portINC-2378Networking · At 14:28 Ivo cut Waybill over to its green build on a new port. At 14:32 every scanner request through Gatehouse started coming back 502. Waybill answers fine on its own, and the label portal behind the same Gatehouse still works.EasyFix 20 min · Linux
The network is up and nothing resolvesINC-2330Networking · Since this morning's scanner container restart, depot-5 cannot resolve the API name. The on-call wants to hardcode the IP.MediumInvestigate 25 min · Linux
The certificate expired 75 days earlyOPS-833Networking · depot-7's scanners stopped printing at 06:10: certificate expired. Every other depot accepts the same certificate. Platform wants to renew it early.MediumInvestigate 25 min · Linux, root
A VPN update stole the API routeNET-944Networking · Since this morning's partner VPN update, depot-7 cannot reach the Waybill API. DNS works, the API is up, and the partner link works.HardInvestigate 35 min · Linux + Docker
Valid certificate, broken chainINC-2519Networking · depot-3's gateway certificate was renewed yesterday. This morning every cold-storage scanner rejects it, yet the gateway opens fine in a browser on Ana's laptop.HardFix 35 min · Linux
"Your site has a certificate error"INC-2440Networking · Since 14:40 customers opening the tracking site get a certificate warning, then a 404. The tracking certificate is fine. Nobody touched it.HardIncident 45 min · Linux, root
Take theirs, the release branch is newerREL-250Git · The 2.5.0 release merge stopped on a conflict in the release notes. The deploy freeze lifts in 30 minutes, and the release channel says to take theirs.EasyFix 20 min · Linux
Ivo's label fix vanished from the release cloneREL-262Git · Dmitri tidied his clone of Waybill before going home, and Ivo's South Sorting label fix went with it. South Sorting opens tomorrow at 06:00, tonight's build needs the fix, and Ivo is on a plane until Thursday.MediumRecover 25 min · Linux
Undo one commit on a shared mainINC-2358Git · For three nights the customs portal has rejected Northstar's export, because a delimiter change meant for the printer feed went into the customs config. More work has landed on the shared main since, and people are using it. Depot-3 has fish in the held queue.MediumFix 30 min · Linux
A staging token in a pull requestSEC-118Git · Secret scanning flagged PR #412: a depot-auth token in .env.local. The branch was already pushed, and the author has more work staged locally.HardRecover 35 min · Linux
v2.5 is running, but which v2.5?INC-2374Git · Customers report label bugs that were fixed in 2.5. The fleet runs waybill:v2.5. The image behind that tag was built from someone's work-in-progress branch.HardInvestigate 35 min · Linux
The hotfix that is running nowhere in GitINC-2391Git · Production runs the 2.5.1 hotfix. Git has no record of it: a force-push yesterday wiped the commit from main, and v2.5.1 was never tagged.HardIncident 50 min · Linux
Waybill cannot find Ledger in the new stackINT-77Containers · In the new integration stack both containers run and every Waybill health check fails: Ledger unreachable. Ledger is listening, and nothing has changed in either image.EasyFix 25 min · Linux + Docker
Docker said healthy for 14 minutes of 500sINC-2342Containers · At 03:12 Ledger crashed. For 14 minutes every tracking request failed, and the load balancer kept sending traffic to Waybill because Docker said it was healthy.MediumInvestigate 30 min · Linux + Docker
An 800 MB image on a depot linkREL-118Containers · Pulling the Waybill image to 24 depots takes eleven minutes on the depot links, and every release waits for it. The budget is 200 MB.MediumTune 35 min · Linux + Docker
Waybill runs as root in its containerSEC-102Containers · The security scan flagged Waybill: root inside the container. Making it non-root broke the event store the first time someone tried.MediumFix 30 min · Linux + Docker
3,112 events vanished in a releaseINC-2355Containers · The 02:00 release recreated every container. Docklight's queue went from 3,112 events to zero, and none of them reached a worker. A restart drill last week found no data loss.MediumFix 30 min · Linux + Docker
Advisory tests, "for a week", three months agoOPS-861CI/CD · Yesterday's checkout change failed the integration tests and deployed anyway. One in five depot checkouts now errors.MediumFix 30 min · Linux
The fuel fee nobody can removeFIN-94CI/CD · Finance removed the fuel fee on October 1. The change merged, its pipeline was green, it deployed. Customers are still being charged.HardFix 40 min · Linux
Every outside contributor's PR failsOSS-31CI/CD · Kim sent a pull request from a fork; CI fails with 401 from deployctl. Someone suggests pull_request_target "so forks get the token".HardInvestigate 35 min · Linux
The fix passed its tests and did not shipINC-2445CI/CD · Paged at 14:32: hazmat checkouts failing since the 14:20 release. That release's pipeline was green and fixed the very test that failed at 13:40.HardIncident 55 min · Linux
Removing one depot restarts threeCHG-5188Infrastructure as code · depot-2 closed on Friday. The change that removes it from the depot list would also replace the scanner APIs of depot-3 and depot-4, mid-shift.MediumFix 30 min · Linux + Docker
"It is just a refactor," says the PRPR-88Infrastructure as code · PR #88 moves Ledger into a module. CI's plan destroys the volume that holds the settlement journal and creates a new one. The author says it is just a refactor, ship it.HardRecover 40 min · Linux + Docker
Locked out by a job that no longer existsDEP-844Infrastructure as code · The hazmat flag for depot-7 never shipped. An apply job stopped halfway, and every run since says the state is locked. Someone suggests -lock=false.HardIncident 35 min · Linux + Docker
The playbook that always says changedOPS-2302Configuration management · Waybill moved to port 8443 a week ago. The depots still listen on 8080, and the nightly playbook reports changed=5 on every host, every night.EasyFix 25 min · Linux
The timeout change never took effectOPS-2320Configuration management · CHG-5520 raised the request timeout from 5 to 30 seconds. Every waybill.conf says 30s. The running services still time out after 5.MediumFix 30 min · Linux
Two depots ignored the changeOPS-2311Configuration management · With the nightly playbook fixed (OPS-2302), Tuesday's run finally applied CHG-5510. depot-1 moved to 8443. depot-2 and depot-3 still render 8080, and the playbook reports nothing wrong.HardInvestigate 35 min · Linux
Deleting the pods made it worseINC-2371Kubernetes · 10:02, a config change for depot-7. 10:05, the pods did not pick it up, so on-call deleted them. 10:06, every Waybill pod is crash-looping.EasyFix 25 min · Linux + Kubernetes
Pending after the node migrationINC-2524Kubernetes · Last night's node migration renamed the pool labels. Every node is Ready, but the edge copy of Waybill has been Pending for forty minutes and the depot scanners are queueing offline.EasyFix 20 min · Linux + Kubernetes
Green pods, empty ServiceINC-2380Kubernetes · This morning every workload moved to the recommended app.kubernetes.io labels. Waybill's pods are Running and Ready. Its Service has no endpoints.EasyFix 25 min · Linux + Kubernetes
The deploy bot lost one verbSEC-131Kubernetes · Last week the deploy bot's cluster-wide edit role was replaced with a narrow Role. Since then every Waybill deploy fails. The release channel wants cluster-admin.MediumFix 30 min · Linux + Kubernetes
The rollout that never startedINC-2530Kubernetes · Ivo applied Waybill 2.0.0 forty minutes ago. `kubectl rollout status` is still waiting, every pod is still 1.4.0, and no new pod has even been created.MediumFix 25 min · Linux + Kubernetes
2.0 waits forever to be readyDEP-812Kubernetes · The 2.0.0 deploy has waited on rollout status for 20 minutes and CI gave up. Old pods still serve. The release channel wants to delete the readiness probe.HardFix 35 min · Linux + Kubernetes
Killed at a limit nobody choseINC-2388Kubernetes · Docklight moved to Kubernetes last week. Its worker restarts every few seconds with exit code 137, and on-call wants to shrink the batch size the carrier contract fixes.HardInvestigate 35 min · Linux + Kubernetes
22 minutes of failed labels, nobody pagedPM-2402Monitoring · Yesterday from 14:02 to 14:24 one in five label requests failed. Depot staff phoned it in. Monitoring stayed green, and nobody was paged.MediumFix 35 min · Linux + Docker
The average says 90 millisecondsCX-311Monitoring · Since release 2.4.3, depot-3 staff say some label prints freeze for two seconds. The latency dashboard shows /labels at 90 ms.HardInvestigate 40 min · Linux + Docker
The alert that could not seePM-2417Monitoring · After the Waybill 2.4.4 deploy, labels failed for one in three requests for eleven minutes. The error alert exists, promtool says it is valid, and nobody was paged.HardIncident 35 min · Linux + Docker
Monday SLO reviewOPS-2340Incident response · Product wants to ship the new label designer this week. Whether that is allowed depends on how much of Waybill's error budget the last 28 days used.EasyInvestigate 25 min · Linux
The pager cries wolfOPS-2350Incident response · Last week the on-call engineer was paged over two hundred times. Two of those pages were real incidents. She has asked to come off the rotation.MediumTune 35 min · Linux
A 30 second crash became a four minute outageOPS-2331Incident response · Ledger crashed for 30 seconds at 09:40. It came back, and then stayed down for four more minutes under a flood of retries from our own clients.HardFix 35 min · Linux
A password on the public websiteSEC-91Security · An external scanner reports that the depot's public website serves a file with a database password in it. The access log says someone already downloaded it.EasyIncident 25 min · Linux
Who can still log in as deploy?SEC-93Security · The access review for the deploy account is overdue. Its authorized_keys file has been edited by hand since 2025, and nobody is sure whose keys are in it.EasyFix 25 min · Linux
The deploy account is root in all but nameSEC-95Security · An audit found that the deploy account may run anything as root. It only ever needs to restart Waybill and check its status.MediumFix 30 min · Linux, root
Every connection slot is takenINC-2537Databases · The Ledger settlement batch has failed every run since 02:10. The database is up and idle, yet it refuses Ledger with "remaining connection slots are reserved".MediumFix 30 min · Linux
The retention job that deleted depot-3INC-2541Databases · Overnight a retention migration was meant to purge old cancelled shipments. This morning every depot-3 parcel from before today is "not found" in tracking. There is a backup from 01:00.HardRecover 45 min · Linux
Shipping labels anyone can downloadSEC-318Cloud · A researcher downloaded one of Northstar's shipping labels, with a customer's name and home address, from the label archive bucket without any credentials. The bucket holds every label since 2024.MediumFix 30 min · Linux
The deploy role that can do anythingSEC-322Cloud · The access review found that the CI role every Waybill deploy assumes can do far more than deploy Waybill. Its policy was "temporary, for the migration", and then "narrowed" once. Nobody has compared it with what the pipeline calls.HardFix 35 min · Linux
Friday evening, everything at onceNW-360Capstone · Waybill 2.6 went out at 16:40. Since 19:30 every depot's tracking page shows errors. The engineer before you rebooted the host once and went home sick.HardBoss incident 60 min · Linux
No labs match. Try a shorter search or clear a filter.
