Managing secrets
Where secrets live, how they reach a workload, how they leak, and how to rotate them before you need to.
A secret is a credential with a lifecycle
Passwords, API tokens, private keys, and connection strings share four needs that Container Security lists. They should be encrypted at rest and in transit, accessible only to the workload that needs them, auditable, and rotatable. Every storage and delivery choice is judged by those four. Most leaks happen because a secret was copied into a place that meets none of them: a Git commit, an image layer, a CI log, a chat message.
Where secrets end up by accident
| Place | How it happens | Prevention |
|---|---|---|
| Git history | git add ., test fixtures, .env files |
.gitignore, staged-diff review, pre-commit scanning, push protection |
| Image layers | COPY . ., ARG TOKEN, a file deleted in a later layer |
.dockerignore, build secrets mounts, multi-stage builds |
| IaC state and plans | Passing secrets as resource arguments | Encrypted, access-controlled backends. Fetch at run time where possible |
| CI logs | set -x, debug output, echoing env |
Masking, no debug in release jobs, least-privilege tokens |
| Process listings | Secrets as command-line arguments | Read from files or stdin |
| Crash dumps and debug endpoints | Environment variables | Files with tight permissions, scrubbed diagnostics |
Terraform: Up & Running's secrets chapter makes the IaC point sharply. Even when a variable is marked sensitive and hidden in plan output, its value is stored in state. Protect state as you would the secret itself.
Delivering secrets to workloads
The progression most organisations follow:
- Hard-coded in code or config: never acceptable.
- Environment variables from the platform's secret settings: simple and widely supported, with the leak paths above.
- Mounted files from a secret mechanism (Kubernetes Secrets, Docker secrets): tighter permissions, rotatable in place, and still needs encryption at rest.
- A secret store queried at run time with the workload's identity: central audit, fine-grained access, and dynamic secrets.
- No static secret at all: workload identity federation, where the platform issues the workload a short-lived token that the cloud trusts.
Each step reduces how many places a secret is copied and how long it stays valid.
Short-lived beats well-guarded
A static key that lives for years will eventually leak, through a laptop, a log, or a departing contractor's notes. Short-lived credentials issued on demand, for example CI jobs exchanging an OIDC token for cloud credentials scoped to one repository and environment, reduce what there is to steal and how long a stolen credential stays useful. Where static secrets remain, rotate them on a schedule so that rotation is routine when an emergency comes.
When a secret leaks
The order matches the “Releases, tags, and shared history” notes, and it holds for every kind of secret:
- Revoke and rotate immediately. Assume anything exposed is compromised.
- Investigate: audit logs for use of the old credential since exposure.
- Remove it from where it leaked (commit, log, image) and from caches and forks where possible.
- Fix the path that allowed it, such as missing ignore rules, missing masking, or a secret passed as an argument.
If the secret was caught before it left the machine, as in the staged-change lab, step 1 may not be necessary. Verify that it truly never left (no push, no CI run) before deciding.
Key terms
- Secret store
- A service that stores secrets encrypted, controls access by identity, audits reads, and supports rotation, such as Vault or a cloud secrets manager.
- Rotation
- Replacing a secret with a new value and revoking the old one, ideally routinely and automatically.
- Short-lived credential
- A credential that expires in minutes or hours, issued to a workload on demand, for example through OIDC federation from CI to a cloud.
- Secret sprawl
- Copies of secrets spread across repositories, wikis, CI settings, and laptops, where nobody can rotate or audit them.
Read further
- Terraform Up & Running, 3rd edition, Ch. 6, "Managing Secrets with Terraform" (Purchase)
The secret management basics, the comparison of tools (stores, KMS-encrypted files, environment variables), and the warning that secrets passed into resources end up in state and plan files. - Container Security, Chapter "Passing Secrets to Containers" (Purchase)
Secret properties (encrypted at rest and in transit, rotatable, accessible only to the right workload), and the trade-offs of environment variables versus mounted files. - Building Secure and Reliable Systems, Ch. 6, "Design for Understandability" (Free to read online)
The sections on identities, authentication, and access control as system invariants. They frame secrets as one part of a workload's identity.