The fix passed its tests and did not ship INC-2445

Every environment runs the same artifact, and every artifact can name the exact commit and pipeline that produced it.

The rule

Continuous Delivery lists it among the first pipeline practices: only build your binaries once. The commit stage produces an artifact, such as an image, a package, or a tarball, and every later stage, including production, uses that artifact. Rebuilding later, even "with the same Dockerfile", produces a different artifact. A base image tag may have moved, a dependency range may resolve to a new patch, a build flag may differ, or someone may check out a different commit. The tests you ran then vouch for something you are not shipping.

Its twin practice is deploy the same way to every environment. When the deploy to staging is the same script as the deploy to production, every staging deploy is also a test of the production deploy.

Provenance: make the artifact say where it came from

An artifact should answer "what are you?" without anyone consulting a spreadsheet:

ARG GIT_COMMIT
LABEL org.opencontainers.image.revision=$GIT_COMMIT \
      org.opencontainers.image.source=https://git.example/northstar/waybill

A /version endpoint or --version flag that prints the commit and build ID serves the same purpose at run time. The pipeline should record, for each release, the artifact digest, the source commit, and the pipeline run that tested it. With that record, "is production running what we tested?" is a single comparison.

Supply-chain frameworks formalise the idea. SLSA-style attestations are signed statements of how and from what an artifact was built. The Secure Operations track returns to them. The habit starts here, with labels and records.

How artifacts go stale

The labs reproduce the common ways the guarantee breaks:

• Rebuild at deploy: the deploy host builds its own image from its own checkout.
• Mutable tags: the pipeline pushes waybill:latest, and the deployer pulls latest later, after another build has overwritten it.
• Cached artifacts: a build step reuses an output directory from a previous run, so the "new" artifact contains old code.
• Deploy by branch: the deploy job checks out main at deploy time, not the commit that passed.

Each looks fine in isolation, with a green pipeline and a successful deploy. Comparing provenance on both sides catches all of them.

Configuration stays outside

If production differs from staging, the difference must be configuration, supplied at deploy time from version-controlled, reviewed sources (Ch. 2). This is the Twelve-Factor config rule from the Containers in Production track, viewed from the pipeline. One artifact plus per-environment config is testable. Per-environment artifacts are not.

Promotion, not reconstruction

A healthy pipeline moves references. The commit stage pushes waybill@sha256:3f1c… to the registry and records it. Acceptance deploys that digest, and production deploys that digest. Nothing downstream ever runs build. If a later stage finds a problem, the fix is a new commit and a new candidate, not a patch applied to the artifact in place.

Key terms

Release candidate

An artifact produced by the commit stage that may be released if it passes every later stage.

Provenance

The verifiable record of where an artifact came from, such as source commit, build job, builder, and inputs.

Artifact repository

The store (registry, package repository) that holds built artifacts by immutable identity for promotion.

Promotion

Moving the same artifact to the next environment, by reference, without rebuilding.

Read further

• Continuous Delivery, Ch. 5, "Anatomy of the Deployment Pipeline" (Purchase; summaries are free on the book's site)
  The deployment pipeline practices, especially "Only Build Your Binaries Once" and "Deploy the Same Way to Every Environment". Note the argument about what rebuilding does to the evidence from earlier stages.
• Continuous Delivery, Ch. 2, "Configuration Management" (Purchase; summaries are free on the book's site)
  Keeping everything needed to rebuild in version control, and managing application configuration separately from binaries.
• The DevOps Handbook, 2nd edition, Part III, "The First Way — The Technical Practices of Flow" (Purchase)
  The chapters on creating the foundations of the deployment pipeline and enabling low-risk releases, and how a single source of truth for artifacts supports flow.

Separate deployment from release, expose changes gradually, and make rollback a routine operation.

Deployment is not release

Two words are often used interchangeably, and separating them is the biggest step toward safe change:

• Deployment puts a new version of software into an environment.
• Release makes a capability available to users.

When they are the same event, every deploy is a big-bang exposure to all users. When they are separate, deploys become routine, frequent, and boring, and releases become controlled decisions.

Three techniques

Blue-green deployment. Run two identical production environments. Blue serves traffic while green receives the new version and is verified. Then the router switches traffic to green. Rollback is switching back. The cost is running two environments and handling state, such as databases and sessions, shared between them.

Canary releases. Deploy the new version to a small slice of capacity and route a small share of real traffic to it. Compare its behaviour with the rest. Widen in steps, for example 1%, 10%, 50%, then 100%, or roll back. The SRE Workbook's canarying chapter adds the discipline. Evaluate against a control population running the old version at the same time, because absolute numbers swing with time of day. Pick metrics that reflect user experience (error ratio, latency percentiles), and give each step long enough to see slow failures.

Feature toggles. Ship code paths turned off and enable them by configuration, for internal users first, then a percentage, then everyone. Toggles separate release from deployment entirely. They also carry a cost. Each live toggle doubles the paths to test, so remove them once the feature is fully launched.

Rollback is a deployment

Continuous Delivery treats rollback as deploying the previous known-good artifact with the same automated process. It is not a special emergency procedure. That works only if:

1. The previous artifact is still available by digest (the “Build once, and know what you built” notes).
2. Configuration for it is versioned and retrievable.
3. The database still works with the old version.
4. Rollback is rehearsed. A mechanism that runs only in emergencies is untested.

In Git terms, rolling back a change to shared configuration is a revert commit followed by a normal pipeline run (the “Undoing and recovering” notes), not a force-push.

Data changes need two versions at once

During a rolling deploy, canary, or rollback, old and new code run simultaneously against one database. Schema changes must therefore be backward compatible. The standard pattern is expand and contract:

1. Expand: add the new column or table. Old code ignores it.
2. Migrate: new code writes both shapes, and a backfill copies existing data.
3. Switch: reads move to the new shape once it is complete.
4. Contract: in a later release, when no running or rollback-target version needs the old shape, remove it.

It is slower than a one-step rename. It is also the difference between a routine rollback and an outage.

Decide in advance what "bad" looks like

Before widening a canary or switching traffic, write down the abort conditions, for example "error ratio above the control by 0.5 percentage points, or p99 latency above 800 ms for five minutes". The Site Reliability Engineering track connects these thresholds to SLOs and error budgets. Without pre-agreed criteria, rollouts continue on optimism.

Key terms

Blue-green deployment

Two identical production environments. Traffic switches from the old (blue) to the new (green) at once, and switches back to roll back.

Canary release

Sending a small share of traffic to the new version, comparing it with the old, and widening only if it behaves.

Feature toggle

A runtime switch that enables code paths independently of deployment.

Expand and contract

Changing a schema in backward-compatible steps (add, migrate, then remove) so old and new code can run at the same time.

Read further

• Continuous Delivery, Ch. 10, "Deploying and Releasing Applications" (Purchase; summaries are free on the book's site)
  Blue-green deployments, canary releasing, and the section on rolling back. Note the emphasis on rehearsing deployments and on rollback as a planned, tested operation.
• The Site Reliability Workbook, Ch. 16, "Canarying Releases" (Free to read online)
  What a canary is and is not, choosing canary population and duration, and evaluating canary metrics against a control group.
• Site Reliability Engineering, Ch. 8, "Release Engineering" (Free to read online (CC BY-NC-ND 4.0))
  Hermetic builds, the release process, and configuration management at Google. Compare it with the “Build once, and know what you built” notes.
• Continuous Delivery, Ch. 12, "Managing Data" (Purchase; summaries are free on the book's site)
  Database scripting and migrations, and decoupling application deployment from database migration.

The four key metrics, why speed and stability are not a trade-off, and how to choose the next improvement.

Measure outcomes, not activity

Accelerate opens its measurement chapter by dismissing familiar productivity metrics. Lines of code reward bloat, velocity is relative and easily inflated, and utilisation near 100% makes queues explode. Its authors instead chose four measures of delivery performance that are outcomes at the level of the system:

(Later DORA reports add a measure of operational reliability, which connects to the Site Reliability Engineering track's SLOs.)

Faster and safer together

The central finding is that these measures cluster. Teams with the shortest lead times and most frequent deployments also had the lowest change failure rates and fastest restoration. The pipeline practices in the other delivery notes explain why. Small changes are easier to review, test, and diagnose, and automated deployment makes rollback cheap. Batching changes to "reduce risk" produces bigger, harder-to-diagnose failures and slower recovery.

That argument helps when someone proposes slowing releases down after an incident. Ask what made the change risky (size, missing tests, no canary, no rollback) and address that instead.

The Three Ways

The DevOps Handbook organises the practices into three principles:

1. Flow: make work visible, reduce batch size, reduce handoffs, and remove constraints from commit to production. Deployment pipelines, trunk-based development, and automated testing are flow practices.
2. Feedback: see problems as they occur and swarm them. Telemetry, stop-the-line on red builds, canary analysis, and peer review are feedback practices.
3. Continual learning and experimentation: blameless post-incident reviews, time to improve daily work, and safe experiments.

A practical use is to classify the bottleneck you have. A three-week lead time spent in an approval queue is a flow problem. Production failures discovered by customers point to feedback. The same incident recurring points to learning.

Using the metrics well

1. Measure the system, not individuals. Lead time crosses teams and tools, so attributing it to people creates blame without insight.
2. Use them to find constraints. Break lead time into stages (coding, review wait, CI, approval wait, deploy) and look for the biggest wait.
3. Do not set them as targets. Goodhart's law applies. A deploy-count target produces trivial deploys, and a failure-rate target produces under-reported incidents.
4. Count change failures honestly. A red pipeline that stops a bad change is success. Count what reached users and needed remediation.
5. Pair every number with a story. "Lead time fell from 9 days to 2 because review waits dropped after we split the service" is a lesson. A number alone is not.

Choosing the next improvement

Finish by applying this to Northstar's pipeline. Write down its current four measures as honestly as you can estimate them. Identify which Way the worst one belongs to, and choose one delivery capability to address it. Accelerate's approach is to improve capabilities, measure the outcome, and repeat, rather than aiming for a maturity level.

Key terms

Lead time for changes

Time from code committed to code successfully running in production.

Deployment frequency

How often the organisation deploys to production (or releases to users).

Change failure rate

Share of production changes that result in degraded service and need remediation, such as rollback, hotfix, or patch.

Time to restore service

How long it takes to recover from an incident or defect that affects users.

Read further

• Accelerate, Ch. 2, "Measuring Performance" (Purchase; DORA publishes the capability research free)
  Why earlier measures (lines of code, velocity, utilisation) fail, how the four measures were chosen, and the clusters of performance found in the data.
• Accelerate, Ch. 1, "Accelerate" (Purchase; DORA publishes the capability research free)
  The argument for focusing on capabilities rather than maturity models.
• The DevOps Handbook, 2nd edition, Part I, "The Three Ways" (Purchase)
  The principles of flow, feedback, and continual learning, and the value stream view of delivery. Note how each one maps to a pipeline practice from the other delivery notes.

Declare early, assign roles, mitigate before you fully understand, and keep a living record of what is happening.

Incidents need structure

The SRE book's incident chapter opens with an unmanaged incident. A capable engineer tries to fix everything alone, colleagues freelance changes without telling anyone, communication with stakeholders lapses, and the outage drags on. Nobody did anything foolish. Nobody was coordinating. The fix is a lightweight structure adapted from emergency services' Incident Command System, and both books describe it.

Roles

Separate the responsibilities, even when one person holds several of them in a small incident:

• Incident commander (IC): owns the incident. Keeps the overall picture, assigns roles, sets priorities, and decides. The IC does not debug.
• Operations lead: works on the system and is the only role making production changes. Others propose, and operations executes and records.
• Communications lead: sends regular updates to stakeholders and users, so engineers are not interrupted by questions.
• Planning lead: handles longer-running needs, such as tracking follow-ups, arranging handoffs, and ordering food at hour six.

Roles scale recursively. In a large incident, sub-teams get their own leads reporting to the IC.

Declare early

The book's guidance is to declare an incident if any of these hold: you need to involve a second team, the outage is visible to customers, or the issue is unresolved after an hour of focused analysis. Declaring costs little. Standing down an incident that turned out minor is easy, and organising one late, after hours of uncoordinated work, is not.

Mitigate first

Users care that service is restored, not that the cause is understood. During an incident, prefer actions that stop the harm:

• Roll back a recent deploy or configuration change (the Continuous Delivery track makes this routine).
• Fail over, drain a bad zone, or shift traffic.
• Shed load or disable a feature with a toggle.

Rolling back a suspect change is often right even before it is proven to be the cause. Mitigation is reversible, and the investigation continues with less pressure. Preserve evidence as you go: keep the bad artifact, the logs, and the timeline.

The live document

A shared incident document, updated continuously, is the command post's memory:

• Status: impact, scope, current severity.
• Timeline: timestamped observations, decisions, and actions, in UTC.
• Hypotheses: what is being tested, by whom, and with what result.
• Actions: who is doing what, and what is pending.
• Communications log: what was told to whom, and when.

It makes handoffs possible, keeps late joiners from asking repeated questions, and becomes the backbone of the postmortem.

Troubleshooting under pressure

The troubleshooting loop from the SRE book's chapter 12 (report, triage, examine, diagnose, test, treat) and the path-splitting habit from the Networking for Operators track still apply, but stress makes people skip steps. Watch for the pitfalls the chapter names: fixating on symptoms that are irrelevant, misunderstanding how the system works, chasing coincidences, and favouring a recent change as the cause without testing it. Write each hypothesis in the document with the result that would disprove it.

Clean handoffs

Incidents outlast shifts. The IC role changes hands explicitly: "You are now IC. Do you accept?" "Yes." The live document is current, and in-flight actions are listed. A sustainable on-call rotation (chapter 11) is part of incident response too. Tired responders make errors, so plan for relief before you need it.

Key terms

Incident commander (IC)

Holds the overall picture, assigns roles and tasks, and decides. The IC does not do the hands-on debugging.

Operations lead

Works on the system to mitigate, and is the only role changing production during the incident.

Mitigation

Restoring service, for example by rollback, failover, or shedding load, before the root cause is fully understood.

Live incident document

A shared, continuously updated record of status, timeline, hypotheses, actions, and owners.

Read further

• Site Reliability Engineering, Ch. 14, "Managing Incidents" (Free to read online (CC BY-NC-ND 4.0))
  The unmanaged-incident case study, the elements of incident management (recursive separation of responsibilities, a recognised command post, a live incident state document, clear handoff), and when to declare an incident.
• The Site Reliability Workbook, Ch. 9, "Incident Response" (Free to read online)
  The Incident Command System roles adapted for software, the case studies, and the best practices summary.
• Site Reliability Engineering, Ch. 11, "Being On-Call", and Ch. 12, "Effective Troubleshooting" (Free to read online (CC BY-NC-ND 4.0))
  In Ch. 11, balanced on-call load and the feeling of safety. In Ch. 12, the report, triage, examine, diagnose, test, and treat loop, and the pitfalls that appear under pressure.

Write about what happened, not who to blame, and turn incidents into owned, verified improvements.

Why write it down

An incident is expensive. It cost user trust, engineering time, and sleep. The postmortem is how the organisation gets something back for that cost. The SRE book defines it as a written record of an incident, its impact, the actions taken to mitigate or resolve it, the contributing causes, and the follow-up actions to prevent recurrence.

When to write one

Agree on triggers in advance so the decision is not political. The book's list:

• User-visible downtime or degradation beyond a threshold.
• Any data loss.
• On-call intervention such as a rollback or traffic reroute.
• Resolution time above a threshold.
• A monitoring failure, where the incident was discovered by a human or a customer instead of an alert.

Anyone may also request a postmortem for an event that did not meet a trigger but was instructive.

Blameless

A blameless postmortem assumes everyone involved acted with good intentions and reasonable judgement given what they knew at the time. It asks what made the error possible and its effect large, not who erred. That is not leniency. People who fear blame hide details, and without details nobody can find the conditions to change. "An engineer pushed the wrong tag" is where analysis starts. Then ask why the tag was mutable, why nothing compared the deployed digest with the tested one, why there was no canary, and why rollback took forty minutes. Those answers produce action items.

Anatomy

The workbook's side-by-side comparison of a weak and a strong postmortem is the best checklist available. A strong one has:

1. Summary: two or three sentences on what happened and the impact.
2. Impact: who was affected, how, and for how long, quantified (requests failed, error budget consumed, depots affected).
3. Timeline: in UTC, from first trigger through detection, escalation, mitigation, and resolution. The live incident document provides it.
4. Contributing factors: plural. Incidents rarely have one root cause. Include what lengthened detection and recovery, not only what started it.
5. What went well, what went badly, where we got lucky. Luck is a risk to address.
6. Action items: each specific, owned, prioritised, and dated, and classified as prevent, detect, or mitigate.

Action items that change things

"Be more careful" is not an action item. Good ones change the system:

• Prevent: "Deploy job promotes the tested digest instead of rebuilding." (the Continuous Delivery track)
• Detect: "Page on absent(up{job="waybill"})." (the Observability and Monitoring track)
• Mitigate: "Canary at 5% with automatic rollback on error-ratio burn." (the Continuous Delivery track, the Site Reliability Engineering track)

Prioritise by how much risk each removes. Track items in the normal work system and verify completion, ideally with a drill or test that reproduces the original failure and shows it is now caught.

Culture

The book's advice on building postmortem culture is practical: review postmortems in a regular forum, praise good ones publicly, have senior leaders participate, share them widely (including a postmortem of the month), and make sure action items get engineering time. Across many incidents, aggregate the data, as chapter 16 describes, because patterns such as "a third of our incidents are config changes" are invisible one postmortem at a time.

Key terms

Blameless postmortem

A written review that assumes people acted reasonably with the information they had, and looks for system and process causes.

Contributing factor

A condition that made the incident possible, worse, or longer. Incidents rarely have a single root cause.

Action item

A specific change with an owner, a priority, and a due date, intended to prevent recurrence or reduce impact.

Postmortem trigger

A pre-agreed condition (user impact, data loss, on-call intervention, long resolution, monitoring failure) that requires a postmortem.

Read further

• Site Reliability Engineering, Ch. 15, "Postmortem Culture — Learning from Failure" (Free to read online (CC BY-NC-ND 4.0))
  Postmortem triggers, the meaning of "blameless", collaboration and knowledge sharing, and the practices for introducing a postmortem culture.
• The Site Reliability Workbook, Ch. 10, "Postmortem Culture — Learning from Failure" (Free to read online)
  The side-by-side bad and good postmortems, and the analysis of why one is better. Use it as a review checklist.
• Site Reliability Engineering, Ch. 16, "Tracking Outages" (Free to read online (CC BY-NC-ND 4.0))
  Aggregating outage data so that patterns across incidents become visible.