Resources, limits, and access control
Requests place pods, limits contain them, OOM kills follow limits, and RBAC grants each identity exactly what it needs.
Requests place, limits contain
Each container can declare requests and limits for CPU and memory:
resources:
requests: { cpu: 250m, memory: 384Mi }
limits: { memory: 512Mi }
- The scheduler places pods by requests. A node "fits" a pod if the sum of requests stays under its allocatable capacity, whatever is actually in use.
- The kubelet and kernel enforce limits. They are cgroup limits, the same mechanism as the Linux Administration track and the Containers in Production track.
From the combination, Kubernetes assigns a QoS class. Guaranteed means requests equal limits for every container, Burstable means some requests are set, and BestEffort means none are. Under node memory pressure, BestEffort pods are evicted first and Guaranteed last.
Memory limits and OOMKilled
When a container exceeds its memory limit, the kernel's OOM killer kills a process in its cgroup. The container exits with 137 and reason OOMKilled, and the kubelet restarts it. The node's free memory is irrelevant. The limit is per container. The lab's "OOMKilled on a node with free memory" is therefore expected behaviour with a too-small limit or a too-greedy workload.
Size from evidence:
- Find the real peak working set, from
kubectl top podover time or container memory metrics in your monitoring. - Ask why it peaks. A batch worker loading a whole file into memory may need a code or config change (smaller batches), not just a bigger limit.
- Set the limit above the legitimate peak with headroom, and set the request near typical usage so scheduling stays honest.
CPU limits throttle
CPU is compressible. Exceeding a CPU limit does not kill anything. The container is throttled, allowed its quota per scheduling period (100 ms by default) and paused for the rest. A request-handling service that bursts can use its quota early in each period and stall, producing latency spikes on an idle node. The cgroup's cpu.stat shows nr_throttled and throttled_usec. Many teams set CPU requests for fair sharing and leave CPU limits off for latency-sensitive services. Whichever you choose, measure throttling rather than guess. That is the USE method's saturation check, applied per container.
RBAC: who may do what, where
Kubernetes authorisation is RBAC over API verbs:
- A Role (namespaced) or ClusterRole (cluster-wide) lists rules: API groups, resources, and verbs.
- A RoleBinding grants a Role or ClusterRole within one namespace. A ClusterRoleBinding grants a ClusterRole everywhere.
- Subjects are users, groups, or service accounts. Bots and workloads use service accounts.
Denials are precise: "cannot patch resource deployments in API group apps in the namespace waybill". That is the specification of the grant you need:
kind: Role
metadata: { name: deployer, namespace: waybill }
rules:
- apiGroups: ['apps']
resources: ['deployments']
verbs: ['get', 'list', 'watch', 'patch', 'update']
Bind it to system:serviceaccount:ci:deploy-bot with a RoleBinding in waybill, then verify with kubectl auth can-i patch deployments -n waybill --as=system:serviceaccount:ci:deploy-bot. Check each further denial the same way, rather than restoring cluster-admin.
The good-practices page lists permissions that are effectively escalation: escalate and bind on roles, create on pods in privileged namespaces, wildcard verbs, and read access to Secrets. Treat any grant of those as a high-risk change, as in the “Least privilege by design” notes.
Quotas are checked at admission
A ResourceQuota caps the total requests, limits, and object counts in a namespace. It is enforced when a pod is created. If the new pod would push the namespace over a hard limit, the API server rejects it. With a quota on requests.memory, it also rejects pods that set no memory request. The rejection lands on whoever creates the pod. For a Deployment that is the ReplicaSet controller, so the ReplicaSet records FailedCreate events while the Deployment only reports that it is waiting. A rolling update briefly runs extra pods (maxSurge), so leave room in the quota for one more pod than you run. Size requests from measured use, then make the rollout fit the budget. Do not grow the budget to fit a guess.
Key terms
- Request
- The resources the scheduler reserves for a container. It decides placement and CPU share.
- Limit
- The maximum a container may use. Exceeding the memory limit causes an OOM kill, and hitting the CPU limit causes throttling.
- QoS class
- Guaranteed, Burstable, or BestEffort, derived from requests and limits. It affects eviction order under node pressure.
- RoleBinding
- Grants a Role's permissions to users, groups, or service accounts within one namespace.
Read further
- Kubernetes Documentation, Concepts, Concepts → Configuration → "Resource Management for Pods and Containers" (Free online (CC BY 4.0))
Requests and limits, how the scheduler uses requests, how memory and CPU limits are enforced, and the troubleshooting section for exceeded limits. - Kubernetes Up & Running, 3rd edition, Ch. 14, "Role-Based Access Control for Kubernetes" (Purchase)
Identities, Roles and ClusterRoles, RoleBindings and ClusterRoleBindings, built-in roles, and testing authorisation withcan-i. - Kubernetes Documentation, Concepts, Concepts → Security → "Role Based Access Control Good Practices" (Free online (CC BY 4.0))
Least-privilege guidance, and the escalation risks of particular verbs and resources (for exampleescalate,bind, and access to Secrets).