Advisory tests, "for a week", three months ago OPS-861

What CI really requires, and how a pipeline turns every commit into a release candidate or a fast, clear rejection.

Integration is a behaviour, not a server

Continuous Delivery is explicit that continuous integration is a practice the team follows. Owning a CI server is not enough. Its prerequisites are modest: everything in version control, an automated build and test you can run from the command line, and an agreement. The agreement is the hard part:

• Merge small changes into the mainline at least once a day.
• Run the build and tests locally before pushing.
• Never push on a broken build, and fix or revert a break immediately.
• Do not go home with the build broken.

Accelerate gives this empirical weight. In its research, trunk-based development, with few active branches living less than a day and no code freezes, was among the capabilities that predicted higher delivery performance. Long-lived feature branches look safe, but they postpone integration until the moment when it is most expensive.

The deployment pipeline

A pipeline models the path from commit to production as stages. Each stage takes the same release candidate and increases confidence in it. Each stage may also reject it:

1. Commit stage (minutes): compile, run unit tests, run static analysis, and build the artifact. Failures here are cheap and fast to fix.
2. Automated acceptance stage: deploy the artifact to a production-like environment and test behaviour against requirements.
3. Further test stages as needed: performance, security, and exploratory or manual testing.
4. Release: deploy the proven artifact to production, ideally with one push-button or automatic step.

Two properties make this work. Fast feedback first: the stages most likely to fail and cheapest to run come earliest. Every stage is a gate: a red stage stops that candidate. A pipeline is valuable because a green result means something.

The cost of "advisory" stages

Teams under pressure sometimes mark a slow or flaky stage as advisory, with continue-on-error, allow_failure, or a manual "ignore". The pipeline turns green while one of its claims is false, and deploys proceed. That is the situation in the lab. The real problem is usually upstream. The tests are too slow, too flaky, or testing the wrong thing. Fix that directly by speeding them up, quarantining flaky tests with an owner and a deadline, or moving them to the right stage. Do not keep a gate that cannot close.

Stop the line

Borrowed from lean manufacturing, the rule is that a broken mainline is everyone's top priority. While it is red, new pushes stack untested changes on top of a known failure, and the next failure is hidden. Continuous Delivery recommends a time limit: if the break cannot be fixed within minutes, revert the offending commit and fix it calmly. Revert is not a defeat. It restores the pipeline's ability to tell the truth.

What "continuous delivery" means

The book's working definition is that the software is always in a releasable state, and releasing is a business decision rather than an engineering project. Continuous deployment goes one step further and releases every green candidate automatically. Either way, the pipeline is the mechanism that makes "releasable" a checked fact rather than a hope.

Key terms

Continuous integration

Everyone merges small changes into the mainline at least daily, and each merge is built and tested automatically. A broken build is fixed immediately.

Deployment pipeline

The automated path from a commit to production, in stages, where each stage gains confidence in one release candidate.

Commit stage

The first, fast stage, which compiles, unit-tests, analyses, and packages the artifact. It should take minutes.

Trunk-based development

Working in short-lived branches (or directly on trunk) that merge within a day or so, rather than long-lived feature branches.

Read further

• Continuous Delivery, Ch. 3, "Continuous Integration" (Purchase; summaries are free on the book's site)
  The prerequisites (version control, an automated build, team agreement) and the essential practices, especially "don't check in on a broken build" and "never go home on a broken build".
• Continuous Delivery, Ch. 5, "Anatomy of the Deployment Pipeline" (Purchase; summaries are free on the book's site)
  The stages from commit to release, the idea that each stage increases confidence in a release candidate, and the deployment pipeline practices listed in the chapter.
• Accelerate, Ch. 4, "Technical Practices" (Purchase; DORA publishes the capability research free)
  Which technical capabilities the research links to delivery performance, including continuous integration and trunk-based development, and how continuous delivery is defined and measured.

Which tests go where, why flaky tests destroy a pipeline's value, and how to make green mean "fit to release".

Tests are a portfolio

Continuous Delivery borrows a map of testing with two axes. One asks whether a test is business-facing, expressed in terms users care about, or technology-facing, expressed in terms of code and systems. The other asks whether it supports the team while building or critiques the product once built:

A healthy strategy has something in every quadrant, automated where possible, and places each kind where its feedback is most useful.

Place tests by speed and confidence

The commit stage runs what is fast and specific: unit and component tests, static analysis, and a smoke test of the packaged artifact. It should answer in minutes. Later stages run what is slow and broad: acceptance tests against the deployed artifact, then performance and security checks. Because they run on the artifact from the commit stage, they also exercise its deployment.

When a stage becomes too slow, the answer is engineering, not omission. Parallelise, split suites, replace end-to-end tests with component tests where they test the same thing, and invest in test data and environments.

Flaky tests are a gate failure

A test that fails randomly does more harm than a missing test, because it teaches everyone that red means nothing. Then real failures are ignored, rerun, or marked advisory. The test-failure lab shows that end state. Handle flakiness explicitly:

1. Detect: track per-test pass rates across runs on unchanged code.
2. Quarantine: move the test out of the gating set, with an owner and a date. Keep running it and reporting it.
3. Fix or delete: most flakiness is a real bug in the test (timing, ordering, shared state) and occasionally in the product.
4. Never blanket-retry the whole suite. Retries convert "intermittent bug in production" into "green build".

Ownership predicts reliability

Accelerate reports that test automation correlates with delivery performance when the tests are reliable and developers create and maintain them, rather than a separate team writing them after the fact. Developers who own the tests fix them when they break and design code to be testable. That is a team design decision as much as a technical one.

What must be true to release

Write down the gate. For Waybill it might read: all commit-stage tests pass, acceptance tests pass on this digest in staging, no critical vulnerabilities in the image scan, and the database migration has been applied and verified in staging. Then make the pipeline enforce exactly that, with no advisory stages and no manual overrides without a recorded reason. If the team routinely overrides a gate, the gate or the tests need work, and the override is the evidence.

Key terms

Testing quadrants

A map of tests along two axes. Business-facing or technology-facing, and supporting the team or critiquing the product.

Acceptance test

An automated test that checks a deployed system meets a requirement from the user's point of view.

Flaky test

A test that passes and fails on the same code. It teaches people to ignore red.

Quality gate

A pipeline condition that must be true for a candidate to proceed.

Read further

• Continuous Delivery, Ch. 4, "Implementing a Testing Strategy" (Purchase; summaries are free on the book's site)
  The testing quadrants (business-facing and technology-facing, supporting development and critiquing the product), and the advice for introducing tests into a legacy system.
• Continuous Delivery, Ch. 8, "Automated Acceptance Testing" (Purchase; summaries are free on the book's site)
  Why acceptance tests run against a deployed artifact in a production-like environment, and how to keep them maintainable and reasonably fast.
• Accelerate, Ch. 4, "Technical Practices" (Purchase; DORA publishes the capability research free)
  The findings on test automation, specifically that reliable tests developers maintain themselves matter more than tests written by a separate team.

Treat the pipeline as production. Keep untrusted code away from secrets, and deploy only artifacts whose origin you can verify.

The pipeline is production

A deployment pipeline holds the keys to production. It has credentials to push images, change infrastructure, and deploy services. Building Secure and Reliable Systems therefore treats it as part of the production threat model. An attacker, or a mistake, anywhere along the chain can put unintended code in front of users:

developer → source repo → dependencies → build → artifact store → deploy → runtime

The chapter's best practices map onto that chain. Require code review for source. Automate builds and deploys so humans do not hold production credentials. Verify artifacts, not just people, so deploy decisions check facts about the artifact. Treat configuration as code, so it gets the same review and provenance.

Untrusted code meets secrets

The most common pipeline compromise needs no exotic attack. A workflow runs code from an untrusted contribution in a context where secrets are available. Fork pull requests are untrusted by definition. CI platforms normally withhold secrets from them, which is why the fork lab's deploy step fails. The dangerous "fix" is to make secrets available anyway, for example by using a privileged trigger (pull_request_target in GitHub Actions) and then checking out and running the PR's code.

The safe structure separates jobs by trust:

1. Untrusted jobs (any PR, including forks) build and test with no secrets and read-only tokens.
2. Trusted jobs (pushes to protected branches, tags, approved environments) deploy, with secrets scoped to their environment.
3. If a privileged job must react to a PR (for example to post a comment), it never executes the PR's code.

Provenance and choke points

The Continuous Delivery track introduced provenance to answer "is production running what we tested?". BSRS extends it into a security control:

• The build system produces signed provenance: this artifact digest was built from commit X of repository Y by builder Z.
• Verifiable or hermetic builds make that claim checkable. Inputs are pinned, and the build runs in an isolated, controlled environment.
• A deployment choke point, the single path to production, enforces policy on provenance, for example "only artifacts built by the official builder from reviewed commits on main".

With that in place, a stolen developer credential cannot deploy a locally built image, because it has no valid provenance. An insider cannot skip review either. The policy is checked against the artifact, not the person.

Scanning: necessary, not sufficient

Container Security's vulnerabilities chapter explains how scanners work. They identify packages in an image and match versions against vulnerability databases. That makes them good at catching known CVEs in OS and language packages, and blind to software they cannot identify, to unknown vulnerabilities, to misconfiguration, and to malicious code. Use them well:

• Scan in the pipeline and gate on severity policy, with exceptions recorded and owned.
• Rescan stored and running images regularly, because a new CVE makes an old image vulnerable without any change.
• Keep images small (the “Images and layers” notes). Fewer packages means fewer findings and less to exploit.

Pin your inputs

Floating dependencies (^1.2, latest, an unpinned base image tag) let upstream changes enter your artifact without review. Pin versions and digests, update them through pull requests with automated tooling, and you know exactly what each build contains. Reproducible builds and supply-chain security start from the same place.

Key terms

Supply chain

Everything between a developer's change and the running artifact, including source hosting, dependencies, build system, artifact store, and deployer.

Binary provenance

Signed, verifiable metadata stating how and from what inputs an artifact was built.

Untrusted input

Code or data from outside the trust boundary, such as a fork's pull request, which must never run with access to secrets.

Deployment choke point

A single enforced path to production where policy (provenance, review, tests) is checked.

Read further

• Building Secure and Reliable Systems, Ch. 14, "Deploying Code" (Free to read online)
  The supply-chain threat model, the best practices (code review, automation, verifying artifacts rather than people, configuration as code), binary provenance, provenance-based deployment policies, and verifiable builds.
• Container Security, Chapter "Software Vulnerabilities in Images" (Purchase)
  How image scanners work, what they can and cannot find, and where scanning belongs in the pipeline.
• Building Secure and Reliable Systems, Ch. 13, "Testing Code" (Free to read online)
  Fuzzing and static analysis as pipeline stages, and how they complement unit and integration tests.