The fuel fee nobody can remove FIN-94

Every environment runs the same artifact, and every artifact can name the exact commit and pipeline that produced it.

The rule

Continuous Delivery lists it among the first pipeline practices: only build your binaries once. The commit stage produces an artifact, such as an image, a package, or a tarball, and every later stage, including production, uses that artifact. Rebuilding later, even "with the same Dockerfile", produces a different artifact. A base image tag may have moved, a dependency range may resolve to a new patch, a build flag may differ, or someone may check out a different commit. The tests you ran then vouch for something you are not shipping.

Its twin practice is deploy the same way to every environment. When the deploy to staging is the same script as the deploy to production, every staging deploy is also a test of the production deploy.

Provenance: make the artifact say where it came from

An artifact should answer "what are you?" without anyone consulting a spreadsheet:

ARG GIT_COMMIT
LABEL org.opencontainers.image.revision=$GIT_COMMIT \
      org.opencontainers.image.source=https://git.example/northstar/waybill

A /version endpoint or --version flag that prints the commit and build ID serves the same purpose at run time. The pipeline should record, for each release, the artifact digest, the source commit, and the pipeline run that tested it. With that record, "is production running what we tested?" is a single comparison.

Supply-chain frameworks formalise the idea. SLSA-style attestations are signed statements of how and from what an artifact was built. The Secure Operations track returns to them. The habit starts here, with labels and records.

How artifacts go stale

The labs reproduce the common ways the guarantee breaks:

• Rebuild at deploy: the deploy host builds its own image from its own checkout.
• Mutable tags: the pipeline pushes waybill:latest, and the deployer pulls latest later, after another build has overwritten it.
• Cached artifacts: a build step reuses an output directory from a previous run, so the "new" artifact contains old code.
• Deploy by branch: the deploy job checks out main at deploy time, not the commit that passed.

Each looks fine in isolation, with a green pipeline and a successful deploy. Comparing provenance on both sides catches all of them.

Configuration stays outside

If production differs from staging, the difference must be configuration, supplied at deploy time from version-controlled, reviewed sources (Ch. 2). This is the Twelve-Factor config rule from the Containers in Production track, viewed from the pipeline. One artifact plus per-environment config is testable. Per-environment artifacts are not.

Promotion, not reconstruction

A healthy pipeline moves references. The commit stage pushes waybill@sha256:3f1c… to the registry and records it. Acceptance deploys that digest, and production deploys that digest. Nothing downstream ever runs build. If a later stage finds a problem, the fix is a new commit and a new candidate, not a patch applied to the artifact in place.

Key terms

Release candidate

An artifact produced by the commit stage that may be released if it passes every later stage.

Provenance

The verifiable record of where an artifact came from, such as source commit, build job, builder, and inputs.

Artifact repository

The store (registry, package repository) that holds built artifacts by immutable identity for promotion.

Promotion

Moving the same artifact to the next environment, by reference, without rebuilding.

Read further

• Continuous Delivery, Ch. 5, "Anatomy of the Deployment Pipeline" (Purchase; summaries are free on the book's site)
  The deployment pipeline practices, especially "Only Build Your Binaries Once" and "Deploy the Same Way to Every Environment". Note the argument about what rebuilding does to the evidence from earlier stages.
• Continuous Delivery, Ch. 2, "Configuration Management" (Purchase; summaries are free on the book's site)
  Keeping everything needed to rebuild in version control, and managing application configuration separately from binaries.
• The DevOps Handbook, 2nd edition, Part III, "The First Way — The Technical Practices of Flow" (Purchase)
  The chapters on creating the foundations of the deployment pipeline and enabling low-risk releases, and how a single source of truth for artifacts supports flow.

Which tests go where, why flaky tests destroy a pipeline's value, and how to make green mean "fit to release".

Tests are a portfolio

Continuous Delivery borrows a map of testing with two axes. One asks whether a test is business-facing, expressed in terms users care about, or technology-facing, expressed in terms of code and systems. The other asks whether it supports the team while building or critiques the product once built:

A healthy strategy has something in every quadrant, automated where possible, and places each kind where its feedback is most useful.

Place tests by speed and confidence

The commit stage runs what is fast and specific: unit and component tests, static analysis, and a smoke test of the packaged artifact. It should answer in minutes. Later stages run what is slow and broad: acceptance tests against the deployed artifact, then performance and security checks. Because they run on the artifact from the commit stage, they also exercise its deployment.

When a stage becomes too slow, the answer is engineering, not omission. Parallelise, split suites, replace end-to-end tests with component tests where they test the same thing, and invest in test data and environments.

Flaky tests are a gate failure

A test that fails randomly does more harm than a missing test, because it teaches everyone that red means nothing. Then real failures are ignored, rerun, or marked advisory. The test-failure lab shows that end state. Handle flakiness explicitly:

1. Detect: track per-test pass rates across runs on unchanged code.
2. Quarantine: move the test out of the gating set, with an owner and a date. Keep running it and reporting it.
3. Fix or delete: most flakiness is a real bug in the test (timing, ordering, shared state) and occasionally in the product.
4. Never blanket-retry the whole suite. Retries convert "intermittent bug in production" into "green build".

Ownership predicts reliability

Accelerate reports that test automation correlates with delivery performance when the tests are reliable and developers create and maintain them, rather than a separate team writing them after the fact. Developers who own the tests fix them when they break and design code to be testable. That is a team design decision as much as a technical one.

What must be true to release

Write down the gate. For Waybill it might read: all commit-stage tests pass, acceptance tests pass on this digest in staging, no critical vulnerabilities in the image scan, and the database migration has been applied and verified in staging. Then make the pipeline enforce exactly that, with no advisory stages and no manual overrides without a recorded reason. If the team routinely overrides a gate, the gate or the tests need work, and the override is the evidence.

Key terms

Testing quadrants

A map of tests along two axes. Business-facing or technology-facing, and supporting the team or critiquing the product.

Acceptance test

An automated test that checks a deployed system meets a requirement from the user's point of view.

Flaky test

A test that passes and fails on the same code. It teaches people to ignore red.

Quality gate

A pipeline condition that must be true for a candidate to proceed.

Read further

• Continuous Delivery, Ch. 4, "Implementing a Testing Strategy" (Purchase; summaries are free on the book's site)
  The testing quadrants (business-facing and technology-facing, supporting development and critiquing the product), and the advice for introducing tests into a legacy system.
• Continuous Delivery, Ch. 8, "Automated Acceptance Testing" (Purchase; summaries are free on the book's site)
  Why acceptance tests run against a deployed artifact in a production-like environment, and how to keep them maintainable and reasonably fast.
• Accelerate, Ch. 4, "Technical Practices" (Purchase; DORA publishes the capability research free)
  The findings on test automation, specifically that reliable tests developers maintain themselves matter more than tests written by a separate team.

Treat the pipeline as production. Keep untrusted code away from secrets, and deploy only artifacts whose origin you can verify.

The pipeline is production

A deployment pipeline holds the keys to production. It has credentials to push images, change infrastructure, and deploy services. Building Secure and Reliable Systems therefore treats it as part of the production threat model. An attacker, or a mistake, anywhere along the chain can put unintended code in front of users:

developer → source repo → dependencies → build → artifact store → deploy → runtime

The chapter's best practices map onto that chain. Require code review for source. Automate builds and deploys so humans do not hold production credentials. Verify artifacts, not just people, so deploy decisions check facts about the artifact. Treat configuration as code, so it gets the same review and provenance.

Untrusted code meets secrets

The most common pipeline compromise needs no exotic attack. A workflow runs code from an untrusted contribution in a context where secrets are available. Fork pull requests are untrusted by definition. CI platforms normally withhold secrets from them, which is why the fork lab's deploy step fails. The dangerous "fix" is to make secrets available anyway, for example by using a privileged trigger (pull_request_target in GitHub Actions) and then checking out and running the PR's code.

The safe structure separates jobs by trust:

1. Untrusted jobs (any PR, including forks) build and test with no secrets and read-only tokens.
2. Trusted jobs (pushes to protected branches, tags, approved environments) deploy, with secrets scoped to their environment.
3. If a privileged job must react to a PR (for example to post a comment), it never executes the PR's code.

Provenance and choke points

The Continuous Delivery track introduced provenance to answer "is production running what we tested?". BSRS extends it into a security control:

• The build system produces signed provenance: this artifact digest was built from commit X of repository Y by builder Z.
• Verifiable or hermetic builds make that claim checkable. Inputs are pinned, and the build runs in an isolated, controlled environment.
• A deployment choke point, the single path to production, enforces policy on provenance, for example "only artifacts built by the official builder from reviewed commits on main".

With that in place, a stolen developer credential cannot deploy a locally built image, because it has no valid provenance. An insider cannot skip review either. The policy is checked against the artifact, not the person.

Scanning: necessary, not sufficient

Container Security's vulnerabilities chapter explains how scanners work. They identify packages in an image and match versions against vulnerability databases. That makes them good at catching known CVEs in OS and language packages, and blind to software they cannot identify, to unknown vulnerabilities, to misconfiguration, and to malicious code. Use them well:

• Scan in the pipeline and gate on severity policy, with exceptions recorded and owned.
• Rescan stored and running images regularly, because a new CVE makes an old image vulnerable without any change.
• Keep images small (the “Images and layers” notes). Fewer packages means fewer findings and less to exploit.

Pin your inputs

Floating dependencies (^1.2, latest, an unpinned base image tag) let upstream changes enter your artifact without review. Pin versions and digests, update them through pull requests with automated tooling, and you know exactly what each build contains. Reproducible builds and supply-chain security start from the same place.

Key terms

Supply chain

Everything between a developer's change and the running artifact, including source hosting, dependencies, build system, artifact store, and deployer.

Binary provenance

Signed, verifiable metadata stating how and from what inputs an artifact was built.

Untrusted input

Code or data from outside the trust boundary, such as a fork's pull request, which must never run with access to secrets.

Deployment choke point

A single enforced path to production where policy (provenance, review, tests) is checked.

Read further

• Building Secure and Reliable Systems, Ch. 14, "Deploying Code" (Free to read online)
  The supply-chain threat model, the best practices (code review, automation, verifying artifacts rather than people, configuration as code), binary provenance, provenance-based deployment policies, and verifiable builds.
• Container Security, Chapter "Software Vulnerabilities in Images" (Purchase)
  How image scanners work, what they can and cannot find, and where scanning belongs in the pipeline.
• Building Secure and Reliable Systems, Ch. 13, "Testing Code" (Free to read online)
  Fuzzing and static analysis as pipeline stages, and how they complement unit and integration tests.