On-call locked out of the bastion OPS-802

Restore SSH access without widening it, and keep installed software the same across a fleet.

Access is a chain

An SSH login passes through several independent checks, and any one can refuse:

1. Network: can the client reach port 22, or a firewall or VPN in between?
2. Server policy: AllowUsers, AllowGroups, DenyGroups, and PermitRootLogin in the effective configuration. sshd -T prints it after includes and Match blocks.
3. Account: does the account exist, and is it locked or expired, with a valid shell?
4. Key: is the public key in ~/.ssh/authorized_keys, and is the client offering the matching private key?
5. File safety: with StrictModes on (the default), sshd ignores key files that others could modify, including through a group- or world-writable home or .ssh.

Work the chain with evidence from both ends. ssh -v shows what the client offered and where it stopped. The server's auth log (journalctl -u ssh or -u sshd) usually states the reason plainly, for example "User sal from 10.0.4.7 not allowed because none of user's groups are listed in AllowGroups".

Groups change at login

Group membership is copied into a process when the session starts. Adding someone to a group changes the directory, but their open shells keep the old list. id shows the session's groups. This also matters for server policy. After a directory migration renames a group, every rule that names the old group, in sshd_config, in sudoers, or in file ownership, silently stops matching.

When access breaks after a migration, the fix is to update the rule to the new group, not to add individual users. Granting to roles keeps the next review short and the next revocation complete.

Shared accounts and privilege

The handbook argues for individual accounts with sudo over shared logins. Individual accounts give you accountability, because logs name a person, and clean revocation, because removing one person does not rotate everyone's password. Grant sudo for specific commands where possible, to groups rather than people, and log it.

Software is part of the host's state

Package managers keep a database of what is installed and from which repository. The questions to ask during "one host behaves differently":

apt-cache policy label-renderer     # installed, candidate, and which repo provides each
dpkg -l label-renderer
dnf list --showduplicates label-renderer
rpm -q --last | head                # recently changed packages

Drift creeps in through manual installs, uneven upgrade timing, a node pointing at a different mirror, or a pin applied on some hosts only. The remedy is to declare the version you want and apply it everywhere, which the Infrastructure as Code track's configuration management notes covers. Do that rather than hand-fixing the odd node.

Upgrade, pin, or roll back

When versions differ, decide deliberately. Roll back the odd node when the new version is the regression and you need service now. Pin (apt-mark hold, DNF versionlock) when a known-good version must stay until the incompatibility is fixed, and record why and until when. Upgrade the rest when the new version is correct and the odd node was simply early. In all cases, verify behaviour on each node, not just the version string.

Key terms

authorized_keys

Per-user file listing public keys allowed to log in. sshd ignores it if the file or its parent directories are writable by others.

AllowGroups

An sshd directive that restricts logins to members of the named groups. Anyone not in a listed group is refused even with a valid key.

Version pinning

Holding a package at a chosen version so routine upgrades do not change it (apt-mark hold, DNF versionlock).

Drift

Hosts that should be identical diverging over time, through manual changes or uneven upgrades.

Read further

• UNIX and Linux System Administration Handbook, 5th edition, Ch. 8, "User Management", and Ch. 3, "Access Control and Rootly Powers" (Purchase)
  How accounts and groups are stored, how sudo is configured, and the argument for least privilege. Note how the book recommends handling shared accounts.
• UNIX and Linux System Administration Handbook, 5th edition, Ch. 6, "Software Installation and Management" (Purchase)
  Package managers (APT and DNF/YUM), repositories and their trust, querying installed versions, and the section on keeping systems consistent.
• UNIX and Linux System Administration Handbook, 5th edition, Ch. 27, "Security" (Purchase)
  The SSH section, especially key-based authentication, the server configuration directives that restrict who may log in, and the guidance on root login.

Classify access by risk, grant narrow and auditable permissions, and make denials easy to diagnose.

Access is a design decision

Building Secure and Reliable Systems argues that least privilege has to be designed in. Retrofitting it to a system where everyone has root rarely works. The chapter's first step is to classify access by risk. Reading public metrics is low risk. Deploying a service, reading customer data, or changing access control is high risk. Each class gets proportionate controls:

The same thinking applies to machine identities. A deploy bot, a CI runner, and a monitoring agent each get a role scoped to what they do. Shared "admin" identities do not survive this classification.

Small interfaces beat big hammers

The book prefers small functional APIs over general access. If on-call engineers need to restart Waybill, give them a tool or API that restarts Waybill, not an interactive root shell on every host. A narrow interface:

• exposes only intended actions,
• can be audited meaningfully, since the logs say "restart waybill by sal" and not a list of shell keystrokes,
• can enforce policy (rate limits, approvals) in one place,
• and limits damage from mistakes and compromised accounts.

Kubernetes RBAC, sudo rules for specific commands, and deployment pipelines that hold the only production credentials all apply the idea.

Advanced controls

For high-risk actions the book describes several controls:

• Multi-party authorisation: a second person approves the action. It stops both insiders and single compromised accounts.
• Temporary access: grants that expire automatically, requested with a business justification.
• Proxies: privileged actions go through an audited proxy instead of direct credentials.
• Break-glass: an emergency bypass that is always available, always logged, alerts security when used, and is reviewed afterwards. It must exist, because outages happen when normal paths are broken. It must be loud, so it does not become the normal path.

Make denials diagnosable

Least privilege fails socially when denials are mysterious. Engineers then lobby for broad roles "to be safe". The book recommends making denial messages say who was denied what on which resource and how to request access. Kubernetes does this well ("User system:serviceaccount:ci:deploy-bot cannot patch resource deployments in API group apps in the namespace waybill"). That message is a complete specification of the missing permission.

The operator's discipline is to read the denial, grant exactly that, and record why. The tempting alternative, binding cluster-admin or running chmod 777 "temporarily", turns the next incident into a security incident.

Least privilege also protects against mistakes

Most damage from excessive access is not malicious. A script run against the wrong environment, a command typed on the wrong host, or an automation bug with admin credentials can all destroy as thoroughly as an attacker. BSRS stresses this overlap between security and reliability. Narrow, scoped access shrinks the blast radius of every kind of error.

Key terms

Least privilege

Every identity, human or machine, has only the access its current task requires, for only as long as needed.

Break-glass

An emergency path that bypasses normal controls. It is tightly audited and triggers alerts and review whenever used.

Multi-party authorisation

A sensitive action requires approval from a second person before it runs.

Blast radius

How much can be damaged if one identity or component is compromised or misused.

Read further

• Building Secure and Reliable Systems, Ch. 5, "Design for Least Privilege" (Free to read online)
  Classifying access by risk, small functional APIs, breakglass mechanisms, auditing, and the advanced controls (multi-party authorisation, temporary access, proxies). Also read the section on diagnosing access denials.
• Building Secure and Reliable Systems, Ch. 2, "Understanding Adversaries" (Free to read online)
  Attacker motivations and profiles, including insiders. Note why least privilege also limits honest mistakes.
• Container Security, Chapter "Container Security Threats" (Purchase)
  The threat model for containerised systems and the security principles at the end, especially least privilege and defence in depth.