The deploy bot lost one verb SEC-131

Requests place pods, limits contain them, OOM kills follow limits, and RBAC grants each identity exactly what it needs.

Requests place, limits contain

Each container can declare requests and limits for CPU and memory:

resources:
  requests: { cpu: 250m, memory: 384Mi }
  limits: { memory: 512Mi }

• The scheduler places pods by requests. A node "fits" a pod if the sum of requests stays under its allocatable capacity, whatever is actually in use.
• The kubelet and kernel enforce limits. They are cgroup limits, the same mechanism as the Linux Administration track and the Containers in Production track.

From the combination, Kubernetes assigns a QoS class. Guaranteed means requests equal limits for every container, Burstable means some requests are set, and BestEffort means none are. Under node memory pressure, BestEffort pods are evicted first and Guaranteed last.

Memory limits and OOMKilled

When a container exceeds its memory limit, the kernel's OOM killer kills a process in its cgroup. The container exits with 137 and reason OOMKilled, and the kubelet restarts it. The node's free memory is irrelevant. The limit is per container. The lab's "OOMKilled on a node with free memory" is therefore expected behaviour with a too-small limit or a too-greedy workload.

Size from evidence:

1. Find the real peak working set, from kubectl top pod over time or container memory metrics in your monitoring.
2. Ask why it peaks. A batch worker loading a whole file into memory may need a code or config change (smaller batches), not just a bigger limit.
3. Set the limit above the legitimate peak with headroom, and set the request near typical usage so scheduling stays honest.

CPU limits throttle

CPU is compressible. Exceeding a CPU limit does not kill anything. The container is throttled, allowed its quota per scheduling period (100 ms by default) and paused for the rest. A request-handling service that bursts can use its quota early in each period and stall, producing latency spikes on an idle node. The cgroup's cpu.stat shows nr_throttled and throttled_usec. Many teams set CPU requests for fair sharing and leave CPU limits off for latency-sensitive services. Whichever you choose, measure throttling rather than guess. That is the USE method's saturation check, applied per container.

RBAC: who may do what, where

Kubernetes authorisation is RBAC over API verbs:

• A Role (namespaced) or ClusterRole (cluster-wide) lists rules: API groups, resources, and verbs.
• A RoleBinding grants a Role or ClusterRole within one namespace. A ClusterRoleBinding grants a ClusterRole everywhere.
• Subjects are users, groups, or service accounts. Bots and workloads use service accounts.

Denials are precise: "cannot patch resource deployments in API group apps in the namespace waybill". That is the specification of the grant you need:

kind: Role
metadata: { name: deployer, namespace: waybill }
rules:
  - apiGroups: ['apps']
    resources: ['deployments']
    verbs: ['get', 'list', 'watch', 'patch', 'update']

Bind it to system:serviceaccount:ci:deploy-bot with a RoleBinding in waybill, then verify with kubectl auth can-i patch deployments -n waybill --as=system:serviceaccount:ci:deploy-bot. Check each further denial the same way, rather than restoring cluster-admin.

The good-practices page lists permissions that are effectively escalation: escalate and bind on roles, create on pods in privileged namespaces, wildcard verbs, and read access to Secrets. Treat any grant of those as a high-risk change, as in the “Least privilege by design” notes.

Quotas are checked at admission

A ResourceQuota caps the total requests, limits, and object counts in a namespace. It is enforced when a pod is created. If the new pod would push the namespace over a hard limit, the API server rejects it. With a quota on requests.memory, it also rejects pods that set no memory request. The rejection lands on whoever creates the pod. For a Deployment that is the ReplicaSet controller, so the ReplicaSet records FailedCreate events while the Deployment only reports that it is waiting. A rolling update briefly runs extra pods (maxSurge), so leave room in the quota for one more pod than you run. Size requests from measured use, then make the rollout fit the budget. Do not grow the budget to fit a guess.

Key terms

Request

The resources the scheduler reserves for a container. It decides placement and CPU share.

Limit

The maximum a container may use. Exceeding the memory limit causes an OOM kill, and hitting the CPU limit causes throttling.

QoS class

Guaranteed, Burstable, or BestEffort, derived from requests and limits. It affects eviction order under node pressure.

RoleBinding

Grants a Role's permissions to users, groups, or service accounts within one namespace.

Read further

• Kubernetes Documentation, Concepts, Concepts → Configuration → "Resource Management for Pods and Containers" (Free online (CC BY 4.0))
  Requests and limits, how the scheduler uses requests, how memory and CPU limits are enforced, and the troubleshooting section for exceeded limits.
• Kubernetes Up & Running, 3rd edition, Ch. 14, "Role-Based Access Control for Kubernetes" (Purchase)
  Identities, Roles and ClusterRoles, RoleBindings and ClusterRoleBindings, built-in roles, and testing authorisation with can-i.
• Kubernetes Documentation, Concepts, Concepts → Security → "Role Based Access Control Good Practices" (Free online (CC BY 4.0))
  Least-privilege guidance, and the escalation risks of particular verbs and resources (for example escalate, bind, and access to Secrets).

Classify access by risk, grant narrow and auditable permissions, and make denials easy to diagnose.

Access is a design decision

Building Secure and Reliable Systems argues that least privilege has to be designed in. Retrofitting it to a system where everyone has root rarely works. The chapter's first step is to classify access by risk. Reading public metrics is low risk. Deploying a service, reading customer data, or changing access control is high risk. Each class gets proportionate controls:

The same thinking applies to machine identities. A deploy bot, a CI runner, and a monitoring agent each get a role scoped to what they do. Shared "admin" identities do not survive this classification.

Small interfaces beat big hammers

The book prefers small functional APIs over general access. If on-call engineers need to restart Waybill, give them a tool or API that restarts Waybill, not an interactive root shell on every host. A narrow interface:

• exposes only intended actions,
• can be audited meaningfully, since the logs say "restart waybill by sal" and not a list of shell keystrokes,
• can enforce policy (rate limits, approvals) in one place,
• and limits damage from mistakes and compromised accounts.

Kubernetes RBAC, sudo rules for specific commands, and deployment pipelines that hold the only production credentials all apply the idea.

Advanced controls

For high-risk actions the book describes several controls:

• Multi-party authorisation: a second person approves the action. It stops both insiders and single compromised accounts.
• Temporary access: grants that expire automatically, requested with a business justification.
• Proxies: privileged actions go through an audited proxy instead of direct credentials.
• Break-glass: an emergency bypass that is always available, always logged, alerts security when used, and is reviewed afterwards. It must exist, because outages happen when normal paths are broken. It must be loud, so it does not become the normal path.

Make denials diagnosable

Least privilege fails socially when denials are mysterious. Engineers then lobby for broad roles "to be safe". The book recommends making denial messages say who was denied what on which resource and how to request access. Kubernetes does this well ("User system:serviceaccount:ci:deploy-bot cannot patch resource deployments in API group apps in the namespace waybill"). That message is a complete specification of the missing permission.

The operator's discipline is to read the denial, grant exactly that, and record why. The tempting alternative, binding cluster-admin or running chmod 777 "temporarily", turns the next incident into a security incident.

Least privilege also protects against mistakes

Most damage from excessive access is not malicious. A script run against the wrong environment, a command typed on the wrong host, or an automation bug with admin credentials can all destroy as thoroughly as an attacker. BSRS stresses this overlap between security and reliability. Narrow, scoped access shrinks the blast radius of every kind of error.

Key terms

Least privilege

Every identity, human or machine, has only the access its current task requires, for only as long as needed.

Break-glass

An emergency path that bypasses normal controls. It is tightly audited and triggers alerts and review whenever used.

Multi-party authorisation

A sensitive action requires approval from a second person before it runs.

Blast radius

How much can be damaged if one identity or component is compromised or misused.

Read further

• Building Secure and Reliable Systems, Ch. 5, "Design for Least Privilege" (Free to read online)
  Classifying access by risk, small functional APIs, breakglass mechanisms, auditing, and the advanced controls (multi-party authorisation, temporary access, proxies). Also read the section on diagnosing access denials.
• Building Secure and Reliable Systems, Ch. 2, "Understanding Adversaries" (Free to read online)
  Attacker motivations and profiles, including insiders. Note why least privilege also limits honest mistakes.
• Container Security, Chapter "Container Security Threats" (Purchase)
  The threat model for containerised systems and the security principles at the end, especially least privilege and defence in depth.