Undo one commit on a shared main INC-2358

Separate deployment from release, expose changes gradually, and make rollback a routine operation.

Deployment is not release

Two words are often used interchangeably, and separating them is the biggest step toward safe change:

• Deployment puts a new version of software into an environment.
• Release makes a capability available to users.

When they are the same event, every deploy is a big-bang exposure to all users. When they are separate, deploys become routine, frequent, and boring, and releases become controlled decisions.

Three techniques

Blue-green deployment. Run two identical production environments. Blue serves traffic while green receives the new version and is verified. Then the router switches traffic to green. Rollback is switching back. The cost is running two environments and handling state, such as databases and sessions, shared between them.

Canary releases. Deploy the new version to a small slice of capacity and route a small share of real traffic to it. Compare its behaviour with the rest. Widen in steps, for example 1%, 10%, 50%, then 100%, or roll back. The SRE Workbook's canarying chapter adds the discipline. Evaluate against a control population running the old version at the same time, because absolute numbers swing with time of day. Pick metrics that reflect user experience (error ratio, latency percentiles), and give each step long enough to see slow failures.

Feature toggles. Ship code paths turned off and enable them by configuration, for internal users first, then a percentage, then everyone. Toggles separate release from deployment entirely. They also carry a cost. Each live toggle doubles the paths to test, so remove them once the feature is fully launched.

Rollback is a deployment

Continuous Delivery treats rollback as deploying the previous known-good artifact with the same automated process. It is not a special emergency procedure. That works only if:

1. The previous artifact is still available by digest (the “Build once, and know what you built” notes).
2. Configuration for it is versioned and retrievable.
3. The database still works with the old version.
4. Rollback is rehearsed. A mechanism that runs only in emergencies is untested.

In Git terms, rolling back a change to shared configuration is a revert commit followed by a normal pipeline run (the “Undoing and recovering” notes), not a force-push.

Data changes need two versions at once

During a rolling deploy, canary, or rollback, old and new code run simultaneously against one database. Schema changes must therefore be backward compatible. The standard pattern is expand and contract:

1. Expand: add the new column or table. Old code ignores it.
2. Migrate: new code writes both shapes, and a backfill copies existing data.
3. Switch: reads move to the new shape once it is complete.
4. Contract: in a later release, when no running or rollback-target version needs the old shape, remove it.

It is slower than a one-step rename. It is also the difference between a routine rollback and an outage.

Decide in advance what "bad" looks like

Before widening a canary or switching traffic, write down the abort conditions, for example "error ratio above the control by 0.5 percentage points, or p99 latency above 800 ms for five minutes". The Site Reliability Engineering track connects these thresholds to SLOs and error budgets. Without pre-agreed criteria, rollouts continue on optimism.

Key terms

Blue-green deployment

Two identical production environments. Traffic switches from the old (blue) to the new (green) at once, and switches back to roll back.

Canary release

Sending a small share of traffic to the new version, comparing it with the old, and widening only if it behaves.

Feature toggle

A runtime switch that enables code paths independently of deployment.

Expand and contract

Changing a schema in backward-compatible steps (add, migrate, then remove) so old and new code can run at the same time.

Read further

• Continuous Delivery, Ch. 10, "Deploying and Releasing Applications" (Purchase; summaries are free on the book's site)
  Blue-green deployments, canary releasing, and the section on rolling back. Note the emphasis on rehearsing deployments and on rollback as a planned, tested operation.
• The Site Reliability Workbook, Ch. 16, "Canarying Releases" (Free to read online)
  What a canary is and is not, choosing canary population and duration, and evaluating canary metrics against a control group.
• Site Reliability Engineering, Ch. 8, "Release Engineering" (Free to read online (CC BY-NC-ND 4.0))
  Hermetic builds, the release process, and configuration management at Google. Compare it with the “Build once, and know what you built” notes.
• Continuous Delivery, Ch. 12, "Managing Data" (Purchase; summaries are free on the book's site)
  Database scripting and migrations, and decoupling application deployment from database migration.

Revert, reset, and restore each undo something different. The reflog means almost nothing committed is truly lost.

Four kinds of "undo"

"Undo it in Git" can mean four different operations. Choose by asking what must change and who already has the commits.

For anything already pushed to a shared branch, revert is the default answer. It records "we undid X, and why" as a normal commit, and it composes with the good commits that came after.

Reset, demystified

git reset <commit> always moves the current branch to <commit>. The flag decides how much else follows:

• --soft: only the branch moves. The index and working tree are untouched, so the undone commit's changes appear staged. This is useful for squashing the last few local commits.
• --mixed (default): the branch moves and the index is reset to match. Changes now appear as unstaged edits.
• --hard: branch, index, and working tree are reset. Uncommitted work is overwritten.

Only --hard destroys anything, and only uncommitted work. Committed work is still in the object database.

The reflog: Git's flight recorder

Every time HEAD or a branch moves (commit, reset, checkout, rebase, merge), Git appends an entry to the reflog:

git reflog            # where HEAD has been
git reflog show main  # where main has been

A commit that "disappeared" after a reset is still an object, now unreachable. Find its ID in the reflog, give it a name with git branch rescue <id>, and it is safe again. Unreachable objects are only pruned by garbage collection after a grace period (two weeks for loose objects and longer for reflog entries, by default). Recovery is usually possible even after a bad afternoon. git fsck --lost-found finds dangling commits when the reflog has no entry, for example in a fresh clone.

The reflog is local. It records your repository's ref movements, not the server's. A commit that only ever existed in someone else's clone must be recovered from their reflog.

Recovering calmly

A recovery routine that never makes things worse:

1. Stop and protect. Stash or commit any uncommitted work to a scratch branch. It has no other copy.
2. Find. git reflog, git log --all --oneline, and git fsck --lost-found.
3. Name it. git branch rescue/<what> <id>. Now nothing can garbage-collect it.
4. Inspect. git show rescue/<what> to confirm it is the approved commit.
5. Restore with a reviewed merge, cherry-pick, or fast-forward, not a force-push to a shared branch.

When a revert is not one command

git revert <commit> applies the inverse of one commit on top of today's branch. Two things make it harder than that:

• A later commit touched the same lines. Git stops with a conflict, because the inverse no longer applies cleanly. Resolve it to today's file with only the bad change undone. Taking either side wholesale silently undoes the later work or keeps the bug. Finish with git add and git revert --continue.
• The change came in through a merge. git revert -m 1 <merge> undoes everything the merged branch brought, which is rarely what you want when one commit inside it was bad. Revert that commit instead. Reverting a merge also has a lasting effect: Git considers the branch's commits already merged, so re-merging the branch later brings nothing back until you revert the revert.

Either way, run the check that failed before you push. A protected branch accepts the fix as a fast-forward and nothing else.

Rewriting history, with care

Interactive rebase and git filter-repo can rewrite many commits: squash, reorder, or strip a file from all history. They are the right tool for unpublished work and for real emergencies such as removing a leaked secret or a huge binary. On shared branches they need coordination, because every collaborator must re-clone or reset. The “Releases, tags, and shared history” notes cover how to do that and what to do first.

Key terms

Revert

A new commit that applies the inverse of an earlier commit. History is preserved and safe on shared branches.

Reset

Moves the current branch pointer (and optionally the index and working tree) to another commit.

Reflog

A local log of where HEAD and each branch pointed over time. It is the safety net for "lost" commits.

Unreachable object

A commit no ref points to. It survives until garbage collection, which by default waits weeks.

Read further

• Pro Git, 2nd edition, 2.4 "Undoing Things" (Free to read online (CC BY-NC-SA 3.0))
  commit --amend, unstaging, and unmodifying files, and the warnings about which of these lose work.
• Pro Git, 2nd edition, 7.7 "Reset Demystified" (Free to read online (CC BY-NC-SA 3.0))
  The three trees again, and the step-by-step diagrams of what reset --soft, --mixed, and --hard move. This section makes reset predictable.
• Pro Git, 2nd edition, 10.7 "Maintenance and Data Recovery" (Free to read online (CC BY-NC-SA 3.0))
  Recovering a lost commit with git reflog and git fsck --lost-found, and when Git's garbage collection actually removes unreachable objects.
• Pro Git, 2nd edition, 7.6 "Rewriting History" (Free to read online (CC BY-NC-SA 3.0))
  Interactive rebase and filter-branch (and the note pointing to git filter-repo). Read it to know the tools exist and why they are dangerous on shared branches.