CPU pinned since the deploy ALERT-7730

Find the process that is actually responsible, stop it with the least force, and read process states.

Finding the process that matters

When a host is slow, the first job is to name the consumer. top (or htop) sorted by CPU or memory gives candidates. Then establish who each candidate is before acting on it: ps -o pid,ppid,user,stat,etime,args -p <pid>. The full command line and the parent tell you whether it is the service, a helper, a cron job, or someone's interactive session. /proc/<pid>/ holds the rest: cwd, exe, fd/, and status.

The tempting shortcut is to blame the process whose name matches the alert. An alert says "Waybill is slow". The CPU is actually consumed by an indexer started by the same deploy. Measure first, then attribute.

Signals: asking, insisting, pausing

A signal is a small message the kernel delivers to a process. The ones operators use:

The order of escalation is TERM, wait, verify, then KILL only if needed. A program that catches SIGTERM flushes buffers, finishes the current item, releases locks, and exits cleanly. SIGKILL gives it no chance to do any of that. Files can be left half-written and locks left behind for the next run to trip over. Service managers follow the same pattern: systemctl stop sends SIGTERM, waits for a timeout, then sends SIGKILL.

After signalling, verify. Check that the PID is gone (kill -0 <pid> fails), the resource has recovered, and the service you meant to protect is still answering.

Reading process states

ps and top show a state letter:

• R running or runnable, S interruptible sleep (waiting for an event, normal for idle daemons).
• D uninterruptible sleep. The process is waiting inside the kernel, usually on I/O, and does not respond even to SIGKILL until the I/O completes. Many D processes point at storage trouble.
• T stopped, by SIGSTOP or a debugger.
• Z zombie. The process has exited and its parent has not collected the status. It costs a table slot, not memory. Fix the parent.

Load is not CPU

Linux's load average counts tasks that are runnable plus tasks in uninterruptible sleep. A load of 12 on four CPUs can therefore mean CPU saturation or a pile of processes stuck on a slow disk. Confirm with vmstat 1. r is the run queue, b is the blocked count, and us/sy/wa split the CPU time. Systems Performance names this idea the USE method: for each resource, check utilisation, saturation, and errors. It returns in the “Memory, limits, and a method for performance” notes.

Priority, not punishment

nice and renice lower a process's CPU priority without stopping it. It is a good choice for a batch job you cannot kill but must not let starve the service. ionice does the same for disk I/O on schedulers that support it. Both are mitigations. The incident note should still say why the job ran at the wrong time.

Key terms

Signal

An asynchronous notification sent to a process. Most can be caught and handled. SIGKILL and SIGSTOP cannot.

SIGTERM

Signal 15, a polite request to exit. Programs catch it to flush data and close connections. It is the default for kill.

Zombie

A process that has exited but whose parent has not yet collected its status. It uses no CPU or memory, only a process-table slot.

Load average

On Linux, the average number of tasks that are runnable or in uninterruptible sleep, over 1, 5, and 15 minutes.

Read further

• UNIX and Linux System Administration Handbook, 5th edition, Ch. 4, "Process Control" (Purchase)
  Process attributes (PID, PPID, UID, niceness), the signal table and which signals can be caught, the state codes shown by ps, and the sections on top and /proc.
• How Linux Works, 3rd edition, Ch. 8, "A Closer Look at Processes and Resource Utilization" (Purchase)
  Measuring CPU time, the load average discussion, and per-process resource monitoring with top, lsof, and strace.

Why a process is killed for memory on a host with gigabytes free, and how to investigate resources systematically.

Memory has more than one number

free -h prints total, used, free, buff/cache, and available. Beginners read "free". Operators read "available". Linux fills idle RAM with page cache, which holds file contents it may need again, and releases it when programs ask for memory. A healthy, busy host therefore often shows little "free" memory. "Available" estimates how much new work could get without swapping.

Per process, two figures appear. VSZ is the virtual address space reserved, which can be enormous and mostly untouched. RSS is the physical memory currently resident. Capacity planning and limits care about RSS and about the totals the kernel accounts per cgroup.

Limits live in cgroups

Modern Linux places every service and container in a control group. A cgroup has its own memory accounting and can have a hard limit: MemoryMax= in a systemd unit, --memory for a container, or a Kubernetes container limit. When processes in the group exceed that limit and the kernel cannot reclaim enough inside the group, it invokes the OOM killer within that group. Host-wide free memory does not matter.

That is the explanation for "OOM-killed on an idle host". The evidence is direct:

journalctl -k | grep -i -A5 "out of memory"   # "Memory cgroup out of memory: Killed process ..."
systemctl show docklight -p MemoryMax -p MemoryCurrent
cat /sys/fs/cgroup/system.slice/docklight.service/memory.events   # oom_kill counter

The decision that follows is about the workload, not the host. Is the limit too low for legitimate peak usage, or does the worker leak or load too much at once? Raising the limit without that answer just moves the next kill later.

The host-wide OOM killer

If the whole host runs out, with no limits and swap exhausted, the kernel picks a victim by its OOM score, which roughly tracks memory size and is adjusted by oom_score_adj. Critical daemons are often protected with a negative adjustment. Seeing an unrelated process killed during host-wide pressure is normal behaviour. The actual consumer is in the kernel log's task dump.

A method instead of a hunch

Performance investigations go wrong when people run familiar tools and stop at the first oddity. Systems Performance calls that the streetlight anti-method. The USE method is a checklist. For every resource (CPUs, memory, storage devices, network interfaces, and software resources such as file descriptors or connection pools), ask three questions:

• Utilisation: how busy is it? (mpstat, free, iostat -x %util, sar -n DEV)
• Saturation: is work queuing? (run queue r in vmstat, swapping si/so, disk aqu-sz, dropped packets)
• Errors: are operations failing? (dmesg, interface error counters, OOM events, EMFILE)

It is quick, it finds bottlenecks you were not looking for, and when everything is clean it tells you the problem is likely in the application, not the host. Start an incident with a short pass through USE before you dig into one resource.

Report what you can verify

A memory finding in an incident note should name the limit that was hit, the evidence (kernel log line, oom_kill counter), the workload's actual peak, and the change, with a measurement afterwards showing headroom.

Key terms

RSS

Resident set size, the physical memory a process currently occupies. Virtual size (VSZ) is address space reserved, not memory used.

Page cache

File contents the kernel keeps in RAM. It is reclaimable, which is why "used" memory on a healthy host looks high.

cgroup

A kernel group of processes with resource accounting and limits. Containers and systemd services each run in one.

OOM killer

The kernel mechanism that kills a process when memory cannot be reclaimed, either host-wide or inside one cgroup.

Read further

• Systems Performance, 2nd edition, Ch. 2, "Methodologies" (Purchase; the USE method summary is free on the author's site)
  The USE method, the anti-methods (especially the streetlight anti-method and random change), and the workload characterisation method. Write down the USE checklist for memory.
• Systems Performance, 2nd edition, Ch. 7, "Memory" (Purchase; the USE method summary is free on the author's site)
  Virtual memory, paging, the page cache, the out-of-memory killer, and the observability tools section (free, vmstat, /proc/meminfo).
• UNIX and Linux System Administration Handbook, 5th edition, Ch. 29, "Performance Analysis" (Purchase)
  The general performance tuning philosophy and the sections on memory and analysis tools. Compare its checklist with the USE method.