The hotfix that is running nowhere in Git INC-2391

Assume something will go wrong. Know the intended state, keep the ability to roll back or revoke, and have a tested path back.

Assume you will need to recover

The BSRS chapter on recovery starts from a premise operators know well. Failures and compromises are certain. What varies is how long recovery takes and how much is lost. Systems can be designed to make recovery fast, safe, and possible at all. Most of the principles come down to three questions:

1. Do we know what the system should look like? (intended state)
2. Can we get back there? (rollback, restore, rebuild)
3. Can we take away what an attacker or a mistake left behind? (revocation)

Know your intended state

To recover, you need an authoritative description of "correct", ideally down to the bytes: which artifact digests run where, what configuration each service has, who has which access, and which data should exist. Infrastructure as code, versioned configuration, release records with provenance, and access-control definitions in code are all intended-state records. Without them, recovery becomes archaeology. The release-history incident in the Version Control with Git track is a small example. Once the tag was gone, only an independent release record could settle what had shipped.

Intended state also enables continuous validation: regularly comparing reality to the record and alerting on drift. Compromise and misconfiguration both show up as drift.

Rollback is not free

Reliability engineering likes rollback, and security engineering is wary of it. A rollback restores a known-good version, which may also be a known-vulnerable version. An attacker who can trigger a rollback can reintroduce a fixed bug. BSRS suggests keeping rollback available but bounded, with a minimum acceptable version or a deny list of known-bad releases enforced at the deployment choke point. The same reasoning applies to configuration and certificates.

Data adds its own constraint. Rolling back code is easy only if the data still fits the old code (the Continuous Delivery track's expand and contract). Rolling back data means restoring a backup, which loses everything written since.

Revocation must be designed in

When a credential, key, or certificate is compromised, you need to stop trusting it everywhere, quickly. That works only if:

• every consumer checks revocation (or credentials are short-lived enough to expire soon),
• the revocation mechanism scales and is fast,
• and there is a re-issue path that does not depend on the compromised thing.

Short-lived credentials (the “Managing secrets” notes) are partly a revocation strategy, because waiting out the lifetime is a revocation that always works.

Emergency access, again

Recovery often happens when normal systems are down: the identity provider is unreachable, CI is broken, or the network is partitioned. BSRS insists that emergency access paths exist and are tested, and that they are audited and alerting, which is the “Least privilege by design” notes’ break-glass. Improvised emergency access, such as sharing a root password in chat, causes the next incident.

Rehearse

Every recovery mechanism needs regular exercise: restoring a backup into a scratch environment, rolling back a release, rotating a credential, rebuilding a host from code. Measure time to recover and data lost, and compare them with what the business needs. A recovery plan that has never run is a guess. The lab capstones across this academy are rehearsals of this kind, done on purpose and checked by a validator.

Key terms

Intended state

An authoritative description of what a system should be, such as versions, configuration, and access, against which recovery is measured.

Revocation

Invalidating a credential, certificate, or artifact so it can no longer be used, quickly and verifiably.

Minimum acceptable version

A floor below which rollbacks are refused, so a known-vulnerable version cannot be redeployed.

Recovery rehearsal

Regularly exercising restore, rollback, and revocation procedures so they work when needed.

Read further

• Building Secure and Reliable Systems, Ch. 9, "Design for Recovery" (Free to read online)
  The design principles for recovery. Read closely the sections on rollbacks as a trade-off between security and reliability, explicit revocation mechanisms, knowing your intended state, and designing for testing and continuous validation. Note the guidance on emergency access.
• Building Secure and Reliable Systems, Ch. 8, "Design for Resilience" (Free to read online)
  Defence in depth, controlling degradation, and blast-radius controls (compartments and failure domains).
• Building Secure and Reliable Systems, Ch. 18, "Recovery and Aftermath" (Free to read online)
  Planning recovery after an incident, including the scope of recovery and the postmortem. It links the Secure Operations track to the Site Reliability Engineering track.

Declare early, assign roles, mitigate before you fully understand, and keep a living record of what is happening.

Incidents need structure

The SRE book's incident chapter opens with an unmanaged incident. A capable engineer tries to fix everything alone, colleagues freelance changes without telling anyone, communication with stakeholders lapses, and the outage drags on. Nobody did anything foolish. Nobody was coordinating. The fix is a lightweight structure adapted from emergency services' Incident Command System, and both books describe it.

Roles

Separate the responsibilities, even when one person holds several of them in a small incident:

• Incident commander (IC): owns the incident. Keeps the overall picture, assigns roles, sets priorities, and decides. The IC does not debug.
• Operations lead: works on the system and is the only role making production changes. Others propose, and operations executes and records.
• Communications lead: sends regular updates to stakeholders and users, so engineers are not interrupted by questions.
• Planning lead: handles longer-running needs, such as tracking follow-ups, arranging handoffs, and ordering food at hour six.

Roles scale recursively. In a large incident, sub-teams get their own leads reporting to the IC.

Declare early

The book's guidance is to declare an incident if any of these hold: you need to involve a second team, the outage is visible to customers, or the issue is unresolved after an hour of focused analysis. Declaring costs little. Standing down an incident that turned out minor is easy, and organising one late, after hours of uncoordinated work, is not.

Mitigate first

Users care that service is restored, not that the cause is understood. During an incident, prefer actions that stop the harm:

• Roll back a recent deploy or configuration change (the Continuous Delivery track makes this routine).
• Fail over, drain a bad zone, or shift traffic.
• Shed load or disable a feature with a toggle.

Rolling back a suspect change is often right even before it is proven to be the cause. Mitigation is reversible, and the investigation continues with less pressure. Preserve evidence as you go: keep the bad artifact, the logs, and the timeline.

The live document

A shared incident document, updated continuously, is the command post's memory:

• Status: impact, scope, current severity.
• Timeline: timestamped observations, decisions, and actions, in UTC.
• Hypotheses: what is being tested, by whom, and with what result.
• Actions: who is doing what, and what is pending.
• Communications log: what was told to whom, and when.

It makes handoffs possible, keeps late joiners from asking repeated questions, and becomes the backbone of the postmortem.

Troubleshooting under pressure

The troubleshooting loop from the SRE book's chapter 12 (report, triage, examine, diagnose, test, treat) and the path-splitting habit from the Networking for Operators track still apply, but stress makes people skip steps. Watch for the pitfalls the chapter names: fixating on symptoms that are irrelevant, misunderstanding how the system works, chasing coincidences, and favouring a recent change as the cause without testing it. Write each hypothesis in the document with the result that would disprove it.

Clean handoffs

Incidents outlast shifts. The IC role changes hands explicitly: "You are now IC. Do you accept?" "Yes." The live document is current, and in-flight actions are listed. A sustainable on-call rotation (chapter 11) is part of incident response too. Tired responders make errors, so plan for relief before you need it.

Key terms

Incident commander (IC)

Holds the overall picture, assigns roles and tasks, and decides. The IC does not do the hands-on debugging.

Operations lead

Works on the system to mitigate, and is the only role changing production during the incident.

Mitigation

Restoring service, for example by rollback, failover, or shedding load, before the root cause is fully understood.

Live incident document

A shared, continuously updated record of status, timeline, hypotheses, actions, and owners.

Read further

• Site Reliability Engineering, Ch. 14, "Managing Incidents" (Free to read online (CC BY-NC-ND 4.0))
  The unmanaged-incident case study, the elements of incident management (recursive separation of responsibilities, a recognised command post, a live incident state document, clear handoff), and when to declare an incident.
• The Site Reliability Workbook, Ch. 9, "Incident Response" (Free to read online)
  The Incident Command System roles adapted for software, the case studies, and the best practices summary.
• Site Reliability Engineering, Ch. 11, "Being On-Call", and Ch. 12, "Effective Troubleshooting" (Free to read online (CC BY-NC-ND 4.0))
  In Ch. 11, balanced on-call load and the feeling of safety. In Ch. 12, the report, triage, examine, diagnose, test, and treat loop, and the pitfalls that appear under pressure.

Write about what happened, not who to blame, and turn incidents into owned, verified improvements.

Why write it down

An incident is expensive. It cost user trust, engineering time, and sleep. The postmortem is how the organisation gets something back for that cost. The SRE book defines it as a written record of an incident, its impact, the actions taken to mitigate or resolve it, the contributing causes, and the follow-up actions to prevent recurrence.

When to write one

Agree on triggers in advance so the decision is not political. The book's list:

• User-visible downtime or degradation beyond a threshold.
• Any data loss.
• On-call intervention such as a rollback or traffic reroute.
• Resolution time above a threshold.
• A monitoring failure, where the incident was discovered by a human or a customer instead of an alert.

Anyone may also request a postmortem for an event that did not meet a trigger but was instructive.

Blameless

A blameless postmortem assumes everyone involved acted with good intentions and reasonable judgement given what they knew at the time. It asks what made the error possible and its effect large, not who erred. That is not leniency. People who fear blame hide details, and without details nobody can find the conditions to change. "An engineer pushed the wrong tag" is where analysis starts. Then ask why the tag was mutable, why nothing compared the deployed digest with the tested one, why there was no canary, and why rollback took forty minutes. Those answers produce action items.

Anatomy

The workbook's side-by-side comparison of a weak and a strong postmortem is the best checklist available. A strong one has:

1. Summary: two or three sentences on what happened and the impact.
2. Impact: who was affected, how, and for how long, quantified (requests failed, error budget consumed, depots affected).
3. Timeline: in UTC, from first trigger through detection, escalation, mitigation, and resolution. The live incident document provides it.
4. Contributing factors: plural. Incidents rarely have one root cause. Include what lengthened detection and recovery, not only what started it.
5. What went well, what went badly, where we got lucky. Luck is a risk to address.
6. Action items: each specific, owned, prioritised, and dated, and classified as prevent, detect, or mitigate.

Action items that change things

"Be more careful" is not an action item. Good ones change the system:

• Prevent: "Deploy job promotes the tested digest instead of rebuilding." (the Continuous Delivery track)
• Detect: "Page on absent(up{job="waybill"})." (the Observability and Monitoring track)
• Mitigate: "Canary at 5% with automatic rollback on error-ratio burn." (the Continuous Delivery track, the Site Reliability Engineering track)

Prioritise by how much risk each removes. Track items in the normal work system and verify completion, ideally with a drill or test that reproduces the original failure and shows it is now caught.

Culture

The book's advice on building postmortem culture is practical: review postmortems in a regular forum, praise good ones publicly, have senior leaders participate, share them widely (including a postmortem of the month), and make sure action items get engineering time. Across many incidents, aggregate the data, as chapter 16 describes, because patterns such as "a third of our incidents are config changes" are invisible one postmortem at a time.

Key terms

Blameless postmortem

A written review that assumes people acted reasonably with the information they had, and looks for system and process causes.

Contributing factor

A condition that made the incident possible, worse, or longer. Incidents rarely have a single root cause.

Action item

A specific change with an owner, a priority, and a due date, intended to prevent recurrence or reduce impact.

Postmortem trigger

A pre-agreed condition (user impact, data loss, on-call intervention, long resolution, monitoring failure) that requires a postmortem.

Read further

• Site Reliability Engineering, Ch. 15, "Postmortem Culture — Learning from Failure" (Free to read online (CC BY-NC-ND 4.0))
  Postmortem triggers, the meaning of "blameless", collaboration and knowledge sharing, and the practices for introducing a postmortem culture.
• The Site Reliability Workbook, Ch. 10, "Postmortem Culture — Learning from Failure" (Free to read online)
  The side-by-side bad and good postmortems, and the analysis of why one is better. Use it as a review checklist.
• Site Reliability Engineering, Ch. 16, "Tracking Outages" (Free to read online (CC BY-NC-ND 4.0))
  Aggregating outage data so that patterns across incidents become visible.

Name exactly what you built, protect shared branches, and handle the day a secret or a force-push goes wrong.

A release needs a name that cannot drift

A release is a claim: "this artifact was built from that source". Git's tool for naming the source is the annotated tag:

git tag -a v2.4.0 -m "Waybill 2.4.0: rollback command fix" <commit>
git push origin v2.4.0
git show v2.4.0

Annotated tags record who tagged, when, and why, and they can be signed (-s). Tags are not pushed by default, so push them explicitly. Treat a pushed release tag as immutable. If it is wrong, create v2.4.1 rather than moving v2.4.0 under people who have already fetched it.

A tag is only half of provenance. The other half is the artifact recording which commit it came from: an image label, a build manifest, or --version output. When the two disagree, trust what is running, investigate how they diverged, and fix the process so it cannot recur. The “Build once, and know what you built” notes build that link into the pipeline.

What a force-push does to everyone else

A normal push only moves a remote branch forward: the new commit must contain the old one. A force-push moves it anywhere, and the remote's commits that are not in your version drop off the branch. For collaborators:

• Their origin/main jumps to your rewritten history at their next fetch.
• Their local main still contains the old commits. A naive git pull merges both histories together.
• Tags pointing at the dropped commits still exist where they were fetched. A deleted tag stays in clones that already have it.

When a rewrite is truly needed, coordinate it: announce it, use git push --force-with-lease so you do not overwrite unseen work, and tell people how to reset. For shared release branches, prevent it entirely with branch protection on the server (no force-push, no deletion, required review).

Every clone is a backup

Because Git is distributed, a commit that "vanished" from the server usually still exists in a CI runner's checkout, in a colleague's clone or reflog, or in the source archive of a build. Recovery after a destructive push means collecting the commit from a surviving copy, verifying it (its hash should match the artifact's recorded commit), and pushing it back as a normal update. The capstone lab is exactly this.

When a secret reaches history

The order of operations matters more than the Git commands:

1. Revoke and rotate the credential. Anything pushed should be assumed copied, by forks, CI caches, and mirrors.
2. Remove it from the current code with a normal commit, and move the value to a secret store (the Secure Operations track).
3. Decide whether to rewrite history. If the secret is rotated, the remaining risk is reputational and the rewrite is optional. If it cannot be rotated quickly, a coordinated rewrite with git filter-repo plus hosting-provider cache purges reduces exposure.
4. Prevent recurrence: .gitignore, pre-commit secret scanning, and server-side push protection.

If the secret is caught before pushing, as in the staged-change lab, the fix is local and complete. Unstage it (git restore --staged) or amend the unpushed commit, then verify with git log -p.

Workflows are agreements

Pro Git's distributed workflows chapter describes several models, from one shared repository to integration managers merging contributions from forks. The model matters less than having one written down: which branches are protected, how changes get reviewed, who may tag releases, and what is never done to shared history. Most Git incidents are a gap in that agreement rather than a gap in anyone's command knowledge.

Key terms

Annotated tag

A full object with tagger, date, message, and optional signature, pointing at a commit. Use these for releases.

Force-push

Updating a remote ref to a commit that does not contain its current value, which discards the remote's commits from that branch.

--force-with-lease

A force-push that succeeds only if the remote ref still has the value you last saw, so it does not overwrite work you have not fetched.

Branch protection

Server-side rules that forbid force-pushes and deletion and require review or checks before merging.

Read further

• Pro Git, 2nd edition, 2.6 "Tagging" (Free to read online (CC BY-NC-SA 3.0))
  Lightweight versus annotated tags, tagging an earlier commit, and the fact that tags are not pushed unless you push them.
• Pro Git, 2nd edition, 3.5 "Remote Branches" (Free to read online (CC BY-NC-SA 3.0))
  Remote-tracking branches, fetch versus pull, and what happens to origin/main when someone else rewrites the remote.
• Pro Git, 2nd edition, 5.1 "Distributed Workflows" and 5.2 "Contributing to a Project" (Free to read online (CC BY-NC-SA 3.0))
  The centralised, integration-manager, and dictator-and-lieutenants workflows, and the commit guidelines. Think about which one your team actually uses.
• Pro Git, 2nd edition, 7.4 "Signing Your Work" (Free to read online (CC BY-NC-SA 3.0))
  Signed tags and commits, and how to verify them. Note what a signature proves and what it does not.