v2.5 is running, but which v2.5? INC-2374

Every environment runs the same artifact, and every artifact can name the exact commit and pipeline that produced it.

The rule

Continuous Delivery lists it among the first pipeline practices: only build your binaries once. The commit stage produces an artifact, such as an image, a package, or a tarball, and every later stage, including production, uses that artifact. Rebuilding later, even "with the same Dockerfile", produces a different artifact. A base image tag may have moved, a dependency range may resolve to a new patch, a build flag may differ, or someone may check out a different commit. The tests you ran then vouch for something you are not shipping.

Its twin practice is deploy the same way to every environment. When the deploy to staging is the same script as the deploy to production, every staging deploy is also a test of the production deploy.

Provenance: make the artifact say where it came from

An artifact should answer "what are you?" without anyone consulting a spreadsheet:

ARG GIT_COMMIT
LABEL org.opencontainers.image.revision=$GIT_COMMIT \
      org.opencontainers.image.source=https://git.example/northstar/waybill

A /version endpoint or --version flag that prints the commit and build ID serves the same purpose at run time. The pipeline should record, for each release, the artifact digest, the source commit, and the pipeline run that tested it. With that record, "is production running what we tested?" is a single comparison.

Supply-chain frameworks formalise the idea. SLSA-style attestations are signed statements of how and from what an artifact was built. The Secure Operations track returns to them. The habit starts here, with labels and records.

How artifacts go stale

The labs reproduce the common ways the guarantee breaks:

• Rebuild at deploy: the deploy host builds its own image from its own checkout.
• Mutable tags: the pipeline pushes waybill:latest, and the deployer pulls latest later, after another build has overwritten it.
• Cached artifacts: a build step reuses an output directory from a previous run, so the "new" artifact contains old code.
• Deploy by branch: the deploy job checks out main at deploy time, not the commit that passed.

Each looks fine in isolation, with a green pipeline and a successful deploy. Comparing provenance on both sides catches all of them.

Configuration stays outside

If production differs from staging, the difference must be configuration, supplied at deploy time from version-controlled, reviewed sources (Ch. 2). This is the Twelve-Factor config rule from the Containers in Production track, viewed from the pipeline. One artifact plus per-environment config is testable. Per-environment artifacts are not.

Promotion, not reconstruction

A healthy pipeline moves references. The commit stage pushes waybill@sha256:3f1c… to the registry and records it. Acceptance deploys that digest, and production deploys that digest. Nothing downstream ever runs build. If a later stage finds a problem, the fix is a new commit and a new candidate, not a patch applied to the artifact in place.

Key terms

Release candidate

An artifact produced by the commit stage that may be released if it passes every later stage.

Provenance

The verifiable record of where an artifact came from, such as source commit, build job, builder, and inputs.

Artifact repository

The store (registry, package repository) that holds built artifacts by immutable identity for promotion.

Promotion

Moving the same artifact to the next environment, by reference, without rebuilding.

Read further

• Continuous Delivery, Ch. 5, "Anatomy of the Deployment Pipeline" (Purchase; summaries are free on the book's site)
  The deployment pipeline practices, especially "Only Build Your Binaries Once" and "Deploy the Same Way to Every Environment". Note the argument about what rebuilding does to the evidence from earlier stages.
• Continuous Delivery, Ch. 2, "Configuration Management" (Purchase; summaries are free on the book's site)
  Keeping everything needed to rebuild in version control, and managing application configuration separately from binaries.
• The DevOps Handbook, 2nd edition, Part III, "The First Way — The Technical Practices of Flow" (Purchase)
  The chapters on creating the foundations of the deployment pipeline and enabling low-risk releases, and how a single source of truth for artifacts supports flow.

Name exactly what you built, protect shared branches, and handle the day a secret or a force-push goes wrong.

A release needs a name that cannot drift

A release is a claim: "this artifact was built from that source". Git's tool for naming the source is the annotated tag:

git tag -a v2.4.0 -m "Waybill 2.4.0: rollback command fix" <commit>
git push origin v2.4.0
git show v2.4.0

Annotated tags record who tagged, when, and why, and they can be signed (-s). Tags are not pushed by default, so push them explicitly. Treat a pushed release tag as immutable. If it is wrong, create v2.4.1 rather than moving v2.4.0 under people who have already fetched it.

A tag is only half of provenance. The other half is the artifact recording which commit it came from: an image label, a build manifest, or --version output. When the two disagree, trust what is running, investigate how they diverged, and fix the process so it cannot recur. The “Build once, and know what you built” notes build that link into the pipeline.

What a force-push does to everyone else

A normal push only moves a remote branch forward: the new commit must contain the old one. A force-push moves it anywhere, and the remote's commits that are not in your version drop off the branch. For collaborators:

• Their origin/main jumps to your rewritten history at their next fetch.
• Their local main still contains the old commits. A naive git pull merges both histories together.
• Tags pointing at the dropped commits still exist where they were fetched. A deleted tag stays in clones that already have it.

When a rewrite is truly needed, coordinate it: announce it, use git push --force-with-lease so you do not overwrite unseen work, and tell people how to reset. For shared release branches, prevent it entirely with branch protection on the server (no force-push, no deletion, required review).

Every clone is a backup

Because Git is distributed, a commit that "vanished" from the server usually still exists in a CI runner's checkout, in a colleague's clone or reflog, or in the source archive of a build. Recovery after a destructive push means collecting the commit from a surviving copy, verifying it (its hash should match the artifact's recorded commit), and pushing it back as a normal update. The capstone lab is exactly this.

When a secret reaches history

The order of operations matters more than the Git commands:

1. Revoke and rotate the credential. Anything pushed should be assumed copied, by forks, CI caches, and mirrors.
2. Remove it from the current code with a normal commit, and move the value to a secret store (the Secure Operations track).
3. Decide whether to rewrite history. If the secret is rotated, the remaining risk is reputational and the rewrite is optional. If it cannot be rotated quickly, a coordinated rewrite with git filter-repo plus hosting-provider cache purges reduces exposure.
4. Prevent recurrence: .gitignore, pre-commit secret scanning, and server-side push protection.

If the secret is caught before pushing, as in the staged-change lab, the fix is local and complete. Unstage it (git restore --staged) or amend the unpushed commit, then verify with git log -p.

Workflows are agreements

Pro Git's distributed workflows chapter describes several models, from one shared repository to integration managers merging contributions from forks. The model matters less than having one written down: which branches are protected, how changes get reviewed, who may tag releases, and what is never done to shared history. Most Git incidents are a gap in that agreement rather than a gap in anyone's command knowledge.

Key terms

Annotated tag

A full object with tagger, date, message, and optional signature, pointing at a commit. Use these for releases.

Force-push

Updating a remote ref to a commit that does not contain its current value, which discards the remote's commits from that branch.

--force-with-lease

A force-push that succeeds only if the remote ref still has the value you last saw, so it does not overwrite work you have not fetched.

Branch protection

Server-side rules that forbid force-pushes and deletion and require review or checks before merging.

Read further

• Pro Git, 2nd edition, 2.6 "Tagging" (Free to read online (CC BY-NC-SA 3.0))
  Lightweight versus annotated tags, tagging an earlier commit, and the fact that tags are not pushed unless you push them.
• Pro Git, 2nd edition, 3.5 "Remote Branches" (Free to read online (CC BY-NC-SA 3.0))
  Remote-tracking branches, fetch versus pull, and what happens to origin/main when someone else rewrites the remote.
• Pro Git, 2nd edition, 5.1 "Distributed Workflows" and 5.2 "Contributing to a Project" (Free to read online (CC BY-NC-SA 3.0))
  The centralised, integration-manager, and dictator-and-lieutenants workflows, and the commit guidelines. Think about which one your team actually uses.
• Pro Git, 2nd edition, 7.4 "Signing Your Work" (Free to read online (CC BY-NC-SA 3.0))
  Signed tags and commits, and how to verify them. Note what a signature proves and what it does not.