"Your site has a certificate error" INC-2440

Model a request as a chain of hops, test the middle first, and verify the fix from where the customer stands.

Draw the path before you test it

Every customer request crosses a chain of components, each of which can fail on its own:

scanner → DNS → load balancer → reverse proxy → Waybill :8080 → Ledger / database

Write the chain down at the start of an incident, with the address and port of each hop. Half of all proxy incidents become obvious at this point. The proxy's upstream says :8000, and the service's listener table says :8080.

Split the path

With a chain of six components, testing them in order from one end wastes time. Test in the middle instead. A request from the proxy host straight to the application's address and port answers "is the fault in front of or behind this point?". Then split the remaining half. This is the halving habit from the troubleshooting method concept note, and it turns a vague "the site is down" into a named hop within two or three requests.

Two rules make each test meaningful:

1. Test from the upstream side of each hop. "Can the proxy reach the app?" must be tested from the proxy host. Your laptop has different routes, DNS, and firewall rules.
2. Use the same name, port, and protocol the real hop uses. Check the proxy's configuration rather than assuming.

What proxies tell you

A reverse proxy is itself an excellent witness. Its error log records why an upstream request failed: connection refused, timeout, reset, or an invalid header. Its access log can record upstream address, status, and timing next to the client status. A client-side 502 paired with "connect() failed (111: Connection refused) while connecting to upstream" names both the hop and the failure shape.

What the proxy is running

A proxy reads its configuration at start and on reload, not every time you save a file. After a change, three things can differ: the file you edited, the configuration the proxy would load now (nginx -t and nginx -T print it, merged from every included file), and the configuration the running process loaded. Included files are read in name order, and a later definition can quietly replace an earlier one, so the file you fixed is not always the one that wins.

Apply a change with a test and a reload. A reload starts new workers with the new configuration and lets the old ones finish their requests. The listener stays open. A restart drops every connection the proxy is carrying, for every site behind it, not just the one you are fixing.

Health checks and load balancers

Load balancers send traffic only to backends whose health check passes, and they remove backends that fail. The SRE book's datacenter load-balancing chapter shows how much depends on that signal. Two failure modes recur:

• The check is too shallow. It requests / or a static page, so a backend with a dead database stays in rotation and fails every real request.
• The check is too deep. It calls every dependency, so one dependency outage removes all backends at once, and partial service becomes total outage.

A good readiness check answers "can this instance serve its main requests right now?" and is cheap enough to run every few seconds. The SRE book also describes a lame duck state: a backend that is shutting down stops accepting new work but finishes what it has. Graceful shutdown in containers and Kubernetes implements the same idea.

Balancing policy

Round robin sends equal request counts to each backend. When requests differ in cost or backends differ in capacity, equal counts produce unequal load. The SRE book discusses policies that use backend feedback, such as least-loaded and weighted round robin, and the subtle failures each one has. For operations work, know which policy your balancer uses and whether its view of backend health matches reality.

Close with the customer's view

After a change, verify from the outside, with the public name, through every hop, from a network like the customer's. Then confirm in the logs that the hop you changed is the one now succeeding. The incident timeline should include each test you ran, its result, and the time. That is the evidence that the halving method is meant to produce.

Key terms

Reverse proxy

A server that accepts client requests and forwards them to one or more upstream servers, for example nginx, HAProxy, or Envoy.

Upstream

The backend a proxy forwards to, defined by an address and port in its configuration.

Health check

A periodic probe a load balancer uses to decide whether to send traffic to a backend.

Split the path

Testing the middle hop first so that one result excludes half of the remaining components.

Read further

• Site Reliability Engineering, Ch. 19, "Load Balancing at the Frontend", and Ch. 20, "Load Balancing in the Datacenter" (Free to read online (CC BY-NC-ND 4.0))
  In Ch. 19, DNS-based balancing and its limits. In Ch. 20, health checking (the "lame duck" state), and why simple round robin misbehaves with uneven backends.
• High Performance Browser Networking, Ch. 1, "Primer on Latency and Bandwidth" (Free to read online)
  The components of latency (propagation, transmission, processing, queuing), and why adding hops adds latency that bandwidth cannot remove.
• UNIX and Linux System Administration Handbook, 5th edition, Ch. 19, "Web Hosting" (Purchase)
  The sections on load balancers and proxies, including reverse proxy configuration and health checks.

Declare early, assign roles, mitigate before you fully understand, and keep a living record of what is happening.

Incidents need structure

The SRE book's incident chapter opens with an unmanaged incident. A capable engineer tries to fix everything alone, colleagues freelance changes without telling anyone, communication with stakeholders lapses, and the outage drags on. Nobody did anything foolish. Nobody was coordinating. The fix is a lightweight structure adapted from emergency services' Incident Command System, and both books describe it.

Roles

Separate the responsibilities, even when one person holds several of them in a small incident:

• Incident commander (IC): owns the incident. Keeps the overall picture, assigns roles, sets priorities, and decides. The IC does not debug.
• Operations lead: works on the system and is the only role making production changes. Others propose, and operations executes and records.
• Communications lead: sends regular updates to stakeholders and users, so engineers are not interrupted by questions.
• Planning lead: handles longer-running needs, such as tracking follow-ups, arranging handoffs, and ordering food at hour six.

Roles scale recursively. In a large incident, sub-teams get their own leads reporting to the IC.

Declare early

The book's guidance is to declare an incident if any of these hold: you need to involve a second team, the outage is visible to customers, or the issue is unresolved after an hour of focused analysis. Declaring costs little. Standing down an incident that turned out minor is easy, and organising one late, after hours of uncoordinated work, is not.

Mitigate first

Users care that service is restored, not that the cause is understood. During an incident, prefer actions that stop the harm:

• Roll back a recent deploy or configuration change (the Continuous Delivery track makes this routine).
• Fail over, drain a bad zone, or shift traffic.
• Shed load or disable a feature with a toggle.

Rolling back a suspect change is often right even before it is proven to be the cause. Mitigation is reversible, and the investigation continues with less pressure. Preserve evidence as you go: keep the bad artifact, the logs, and the timeline.

The live document

A shared incident document, updated continuously, is the command post's memory:

• Status: impact, scope, current severity.
• Timeline: timestamped observations, decisions, and actions, in UTC.
• Hypotheses: what is being tested, by whom, and with what result.
• Actions: who is doing what, and what is pending.
• Communications log: what was told to whom, and when.

It makes handoffs possible, keeps late joiners from asking repeated questions, and becomes the backbone of the postmortem.

Troubleshooting under pressure

The troubleshooting loop from the SRE book's chapter 12 (report, triage, examine, diagnose, test, treat) and the path-splitting habit from the Networking for Operators track still apply, but stress makes people skip steps. Watch for the pitfalls the chapter names: fixating on symptoms that are irrelevant, misunderstanding how the system works, chasing coincidences, and favouring a recent change as the cause without testing it. Write each hypothesis in the document with the result that would disprove it.

Clean handoffs

Incidents outlast shifts. The IC role changes hands explicitly: "You are now IC. Do you accept?" "Yes." The live document is current, and in-flight actions are listed. A sustainable on-call rotation (chapter 11) is part of incident response too. Tired responders make errors, so plan for relief before you need it.

Key terms

Incident commander (IC)

Holds the overall picture, assigns roles and tasks, and decides. The IC does not do the hands-on debugging.

Operations lead

Works on the system to mitigate, and is the only role changing production during the incident.

Mitigation

Restoring service, for example by rollback, failover, or shedding load, before the root cause is fully understood.

Live incident document

A shared, continuously updated record of status, timeline, hypotheses, actions, and owners.

Read further

• Site Reliability Engineering, Ch. 14, "Managing Incidents" (Free to read online (CC BY-NC-ND 4.0))
  The unmanaged-incident case study, the elements of incident management (recursive separation of responsibilities, a recognised command post, a live incident state document, clear handoff), and when to declare an incident.
• The Site Reliability Workbook, Ch. 9, "Incident Response" (Free to read online)
  The Incident Command System roles adapted for software, the case studies, and the best practices summary.
• Site Reliability Engineering, Ch. 11, "Being On-Call", and Ch. 12, "Effective Troubleshooting" (Free to read online (CC BY-NC-ND 4.0))
  In Ch. 11, balanced on-call load and the feeling of safety. In Ch. 12, the report, triage, examine, diagnose, test, and treat loop, and the pitfalls that appear under pressure.