Killed at a limit nobody chose

Lab machine

A private machine with its own Kubernetes cluster. Starting takes about 30 seconds. Sessions last up to 60 minutes.

Teo Marin opened INC-2388 at 12:20SEV-2

Docklight moved to Kubernetes last week. Its worker restarts every few seconds with exit code 137, and on-call wants to shrink the batch size the carrier contract fixes.

"Node has gigabytes free, must be a bug in the worker. Lower BATCH_MB to 8?" (on-call)

The carrier contract fixes batches at 64 MiB. The platform budget for this namespace is in capacity.md.

Your task

Find why the worker is killed and size its memory limit (and request) for a 64 MiB batch plus the runtime, within the 256 MiB budget. Keep BATCH_MB at 64, fix manifests/, apply, and watch batches complete.

On the machine

kubectl describe pod (Last State, Reason, Exit Code, Limits)
kubectl logs of the worker
kubectl describe limitrange
manifests/, capacity.md

Timeline

Last weekDocklight worker migrated to Kubernetes; limits copied from a chart default.

12:00Carrier batches go back to 64 MiB after a contract update.

12:01Worker pod: OOMKilled, restarting.

12:20"Raise BATCH_MB down to 8?"

Done when

The worker processes 64 MiB batches continuously without being killed.
It keeps a memory limit of 256 MiB or less, with a request no larger than the limit.
manifests/ matches the cluster.

How it is checked:The checker requires BATCH_MB 64, a memory limit at or under 256 MiB with a request no larger than the limit, the cluster matching `manifests/`, a completed rollout, and several batches processed with no restart.

Passed. Here is how others fixed it

This lab has 2 versions. Reset for a fresh machine, and you may get a different one.

Break it for a friend. Pick the version they get, and the link is copied for you to send: version 1 · version 2

Other fixes that also pass:

Set the limit to the measured peak plus a margin (for example 192Mi) and a matching request, as long as it stays within the 256 MiB budget.

Fixes that look right but are not:

Shrinking the batch size that the carrier contract fixes.
Removing the memory limit, which lets one pod starve the node.

Why it broke. A container is killed when it exceeds its own memory limit, whatever the node has free. That limit may come from a chart default nobody revisited, or from a namespace default that applies to every container without one. Either way it was nobody's measurement. Size limits from the working set, within the budget, and write them where the next reader looks.

How Mara worked it

kubectl describe pod <worker>
Last State: OOMKilled, exit code 137, and the limit it ran under.
kubectl logs <worker> --previous and cat capacity.md
A 64 MiB batch plus the runtime, against a 64Mi limit.
grep -n -A4 resources manifests/worker.yaml and kubectl describe limitrange
Where the limit came from: the manifest or a namespace default.
Set an explicit limit within 256Mi, with a request no larger, and apply
Fit the limit to the work, inside the platform budget, in the source of truth.
kubectl get pods -w and the worker log
Batches complete and the restart count stays put.

Hints

Hint 1

The kill is recorded in the pod's container status, not in the node's free memory.

Hint 2

Find where the pod's limit comes from: the manifest, or a namespace default (`kubectl describe limitrange`).

Hint 3

The batch size is a contract; the limit is yours to size.

Hint 4

Set an explicit memory limit (and a sensible request) in the manifest, within the budget, and apply it.

Show the solution

`kubectl describe pod` shows OOMKilled, exit 137, at a 64Mi limit. That limit is either the chart default copied into `manifests/worker.yaml`, or, when the manifest sets none, the namespace LimitRange default (`kubectl describe limitrange`). Set an explicit memory limit in `manifests/worker.yaml` that covers a 64 MiB batch plus the Node runtime (for example 160Mi, request 128Mi), apply it, and watch several batches complete with no restarts.

Kubernetes Operations

Resources, limits, and access control

Requests place pods, limits contain them, OOM kills follow limits, and RBAC grants each identity exactly what it needs.

Requests place, limits contain

Each container can declare requests and limits for CPU and memory:

resources:
  requests: { cpu: 250m, memory: 384Mi }
  limits: { memory: 512Mi }

The scheduler places pods by requests. A node "fits" a pod if the sum of requests stays under its allocatable capacity, whatever is actually in use.
The kubelet and kernel enforce limits. They are cgroup limits, the same mechanism as the Linux Administration track and the Containers in Production track.

From the combination, Kubernetes assigns a QoS class. Guaranteed means requests equal limits for every container, Burstable means some requests are set, and BestEffort means none are. Under node memory pressure, BestEffort pods are evicted first and Guaranteed last.

Memory limits and OOMKilled

When a container exceeds its memory limit, the kernel's OOM killer kills a process in its cgroup. The container exits with 137 and reason OOMKilled, and the kubelet restarts it. The node's free memory is irrelevant. The limit is per container. The lab's "OOMKilled on a node with free memory" is therefore expected behaviour with a too-small limit or a too-greedy workload.

Size from evidence:

Find the real peak working set, from kubectl top pod over time or container memory metrics in your monitoring.
Ask why it peaks. A batch worker loading a whole file into memory may need a code or config change (smaller batches), not just a bigger limit.
Set the limit above the legitimate peak with headroom, and set the request near typical usage so scheduling stays honest.

CPU limits throttle

CPU is compressible. Exceeding a CPU limit does not kill anything. The container is throttled, allowed its quota per scheduling period (100 ms by default) and paused for the rest. A request-handling service that bursts can use its quota early in each period and stall, producing latency spikes on an idle node. The cgroup's cpu.stat shows nr_throttled and throttled_usec. Many teams set CPU requests for fair sharing and leave CPU limits off for latency-sensitive services. Whichever you choose, measure throttling rather than guess. That is the USE method's saturation check, applied per container.

RBAC: who may do what, where

Kubernetes authorisation is RBAC over API verbs:

A Role (namespaced) or ClusterRole (cluster-wide) lists rules: API groups, resources, and verbs.
A RoleBinding grants a Role or ClusterRole within one namespace. A ClusterRoleBinding grants a ClusterRole everywhere.
Subjects are users, groups, or service accounts. Bots and workloads use service accounts.

Denials are precise: "cannot patch resource deployments in API group apps in the namespace waybill". That is the specification of the grant you need:

kind: Role
metadata: { name: deployer, namespace: waybill }
rules:
  - apiGroups: ['apps']
    resources: ['deployments']
    verbs: ['get', 'list', 'watch', 'patch', 'update']

Bind it to system:serviceaccount:ci:deploy-bot with a RoleBinding in waybill, then verify with kubectl auth can-i patch deployments -n waybill --as=system:serviceaccount:ci:deploy-bot. Check each further denial the same way, rather than restoring cluster-admin.

The good-practices page lists permissions that are effectively escalation: escalate and bind on roles, create on pods in privileged namespaces, wildcard verbs, and read access to Secrets. Treat any grant of those as a high-risk change, as in the “Least privilege by design” notes.

Quotas are checked at admission

A ResourceQuota caps the total requests, limits, and object counts in a namespace. It is enforced when a pod is created. If the new pod would push the namespace over a hard limit, the API server rejects it. With a quota on requests.memory, it also rejects pods that set no memory request. The rejection lands on whoever creates the pod. For a Deployment that is the ReplicaSet controller, so the ReplicaSet records FailedCreate events while the Deployment only reports that it is waiting. A rolling update briefly runs extra pods (maxSurge), so leave room in the quota for one more pod than you run. Size requests from measured use, then make the rollout fit the budget. Do not grow the budget to fit a guess.

Key terms

Request: The resources the scheduler reserves for a container. It decides placement and CPU share.
Limit: The maximum a container may use. Exceeding the memory limit causes an OOM kill, and hitting the CPU limit causes throttling.
QoS class: Guaranteed, Burstable, or BestEffort, derived from requests and limits. It affects eviction order under node pressure.
RoleBinding: Grants a Role's permissions to users, groups, or service accounts within one namespace.

Read further

Kubernetes Documentation, Concepts, Concepts → Configuration → "Resource Management for Pods and Containers" (Free online (CC BY 4.0))
Requests and limits, how the scheduler uses requests, how memory and CPU limits are enforced, and the troubleshooting section for exceeded limits.
Kubernetes Up & Running, 3rd edition, Ch. 14, "Role-Based Access Control for Kubernetes" (Purchase)
Identities, Roles and ClusterRoles, RoleBindings and ClusterRoleBindings, built-in roles, and testing authorisation with can-i.
Kubernetes Documentation, Concepts, Concepts → Security → "Role Based Access Control Good Practices" (Free online (CC BY 4.0))
Least-privilege guidance, and the escalation risks of particular verbs and resources (for example escalate, bind, and access to Secrets).

Killed at a limit nobody chose INC-2388