22 minutes of failed labels, nobody paged PM-2402

Every page should be urgent, actionable, and about user impact. Everything else goes to a ticket or a dashboard.

What a page is for

A page interrupts a person, at dinner or at 03:00, and asks for judgement. The SRE book holds pages to a strict standard. Every paging rule should be:

• Urgent: user impact is happening or imminent.
• Actionable: there is something a human should do now.
• About symptoms: users are, or will be, affected. "A server is unhappy" is not enough.
• Requiring intelligence: if the response is a script, automate the script instead.

Everything that fails these tests still has a place. Tickets cover work that matters but can wait, such as slow disk growth, certificate expiry in 21 days, or a degraded replica. Dashboards hold context for investigation, such as CPU, memory, and per-dependency latency.

Ratios, not counts

Alert on what users experience, normalised by traffic. An error count pages at peak because more traffic means more absolute errors, and misses failures at night. An error ratio does not:

- alert: WaybillRouteErrorRatioHigh
  expr: |
    sum by (route) (rate(http_requests_total{job="waybill",code=~"5.."}[1m]))
      /
    sum by (route) (rate(http_requests_total{job="waybill"}[1m]))
      > 0.05
  for: 20s
  labels: { severity: page }
  annotations:
    summary: '{{ $labels.route }} is failing {{ $value | humanizePercentage }} of requests'

sum by (route) keeps the dimension responders need. Without it, one broken route among ten healthy ones averages below the threshold.

Speed versus noise

Two knobs balance detection speed against false alarms:

• Rate window ([1m], [5m]): longer windows smooth noise and react more slowly.
• for duration: how long the condition must hold. It filters blips at the cost of delay.

Choose both from the impact. A total outage of checkout must page within a minute or two. A 2% error rate may be fine to tolerate for longer. The Site Reliability Engineering track replaces hand-tuned thresholds with burn-rate alerts derived from an SLO, which is the systematic answer to this trade-off.

When there is no data

An alert expression over series that no longer exist returns an empty result, and an empty result never fires. A broken exporter, a moved metrics port, or a renamed job silences every alert for that service, as in the metrics-went-dark lab. Every service with paging alerts needs:

up{job="waybill"} == 0 or absent(up{job="waybill"})

The first part catches a target that is discovered but failing to scrape. The second catches a target that has vanished from discovery entirely.

Keep alerts honest over time

Practical Monitoring recommends reviewing alerts regularly against what responders actually did. For each alert that fired, was it real, did someone act, and could the action be automated? Alerts acknowledged and ignored week after week should be deleted, retuned, or turned into tickets. Every page should link to a runbook that explains what it means and the first three things to check. If nobody can write that runbook, the alert is not ready to page.

Key terms

Page

An alert that interrupts a human immediately. Reserve it for urgent, actionable user impact.

Ticket

An alert that creates work for business hours. It suits slow-burning or non-urgent problems.

Error ratio

Failed requests divided by all requests over a window. It is independent of traffic volume, unlike an error count.

for duration

How long a condition must hold before the alert fires. It suppresses blips at the cost of detection delay.

Read further

• Site Reliability Engineering, Ch. 10, "Practical Alerting" (Free to read online (CC BY-NC-ND 4.0))
  Alerting on time-series data, rules and aggregation, alerting on rates and ratios rather than counts, and routing to pages versus tickets.
• Site Reliability Engineering, Ch. 6, "Monitoring Distributed Systems" (Free to read online (CC BY-NC-ND 4.0))
  Re-read "Tying These Principles Together" and "Monitoring for the Long Term". Note the questions every new paging rule must answer.
• Practical Monitoring, Ch. 3, "Alerts, On-Call, and Incident Management" (Purchase)
  What makes a good alert, stopping alert fatigue, and the advice to delete or tune alerts that do not lead to action.

100% is the wrong target. Choose a reliability objective from what users need, measure it, and spend the remainder deliberately.

Reliability is a product decision

The SRE book's chapter on risk starts with a provocation. 100% is the wrong reliability target for almost anything. Users reach a service through phones, Wi-Fi, ISPs, and browsers that are less reliable than the service itself, so they cannot tell 99.99% from 100%. Each extra "nine" costs disproportionately more, in redundancy, slower releases, and engineering time, while buying nothing users notice. The right target is the level at which users are happy, and that is a product decision informed by data.

Indicators, objectives, agreements

Chapter 4 defines the vocabulary:

• SLI (indicator): a measurement of service behaviour. In practice it is best expressed as a ratio of good events / valid events, so that 0% is total failure and 100% is perfect.
• SLO (objective): a target for an SLI over a time window, for example 99.5% of label requests good over a rolling 28 days.
• SLA (agreement): an external contract with consequences. Keep SLOs tighter than SLAs so you react before you owe credits.

The workbook adds a useful distinction. An SLI specification says what you want to measure ("label requests that succeed quickly, as users see them"). An SLI implementation says exactly how ("load balancer logs, non-5xx and < 500 ms, excluding health checks"). Write both. The implementation decides what counts.

Choosing SLIs from user journeys

Start from what users do, not from what is easy to measure. For Northstar:

Measure as close to the user as you practically can. Load balancer logs or client telemetry beat application metrics, which cannot see requests that never arrived. The Observability and Monitoring track's black-box probes are another source.

Choosing targets

The book warns against choosing a target by looking at current performance. That locks you into heroics to maintain an accidental level. Choose from user needs, start somewhat loose, and tighten with evidence. Keep the number of SLOs small, a few per service that cover the important journeys.

The error budget

If the SLO is 99.5%, the error budget is the remaining 0.5%: over 10 million requests in the window, 50,000 bad ones are acceptable. The budget changes conversations:

• Releases, experiments, migrations, and incidents all spend from the same budget.
• While budget remains, the team ships. Reliability is good enough by agreement.
• When the budget is exhausted, the error budget policy applies: freeze non-essential releases and prioritise reliability work until the service recovers.

The workbook stresses writing that policy down and getting agreement from product, development, and SRE before it is needed. Then the decision during a bad month is to follow the policy, not to argue.

Budgets are information, not punishment

Spending budget is fine. That is what it is for. Consistently ending windows with most of the budget unspent suggests the team could ship faster, or that the SLO is tighter than users need. Consistently exhausting it suggests the SLO, the system, or the release process needs attention. Review targets periodically, as the workbook recommends.

Key terms

SLI

Service level indicator, a measured ratio of good events to valid events, such as successful label requests faster than 500 ms.

SLO

Service level objective, a target for an SLI over a window, such as 99.5% over 28 days.

SLA

Service level agreement, a contract with consequences (credits, penalties) if a level is missed. It is usually looser than the SLO.

Error budget

1 − SLO, the unreliability the service may "spend" in the window on releases, experiments, and failures.

Read further

• Site Reliability Engineering, Ch. 3, "Embracing Risk" (Free to read online (CC BY-NC-ND 4.0))
  Measuring service risk, risk tolerance for consumer and infrastructure services, and the motivation for error budgets as a negotiation tool between development and SRE.
• Site Reliability Engineering, Ch. 4, "Service Level Objectives" (Free to read online (CC BY-NC-ND 4.0))
  The definitions of SLI, SLO, and SLA, choosing indicators by service type, and the advice on choosing targets (keep a safety margin, do not pick a target from current performance).
• The Site Reliability Workbook, Ch. 2, "Implementing SLOs" (Free to read online)
  The worked example of specifying SLIs, the SLI specification versus implementation distinction, the error budget policy document, and continuous improvement of targets.

Page on the rate at which the error budget is burning, using multiple windows, so you catch fast outages quickly and slow leaks before they cost the month.

From thresholds to budgets

The Observability and Monitoring track built alerts on hand-chosen thresholds: error ratio above 5% for 20 seconds. Once a service has an SLO, there is a better question to ask: how fast are we spending the error budget? An outage that burns 10% of the month's budget in an hour needs a human now. A slow leak that would use the budget over three weeks needs a ticket. A 30-second blip that costs 0.01% needs nothing.

Burn rate

Burn rate is the speed of budget consumption relative to the SLO. At burn rate 1, the budget lasts exactly the window. At burn rate 10, it lasts a tenth of the window. With a 99.9% SLO the budget is 0.1% errors, so an observed error ratio of 1.44% is a burn rate of 14.4. Over a 30-day window that would exhaust the budget in about two days, and every hour at that rate costs 2% of the month's budget.

The general relation is:

budget consumed = burn rate × (alert window / SLO window)

That lets you choose thresholds in terms of what matters, the share of budget lost, instead of raw percentages.

Measuring an alert

The workbook judges every alerting strategy on four attributes:

• Precision: are the alerts real?
• Recall: are real events alerted on?
• Detection time: how quickly does it fire?
• Reset time: how long does it keep firing after the problem is fixed?

It then works through six approaches in turn: target error rate over short windows, longer windows, alert durations, burn-rate alerts, multiple burn rates, and finally multiwindow, multi-burn-rate alerts. Each fixes a weakness of the one before.

The recommended shape

For a 99.9% SLO over 30 days, the workbook's starting configuration is:

An alert fires only when both windows exceed the burn rate. The long window ensures the burn is significant, and the short window ensures it is still happening, so the alert resets within minutes of recovery. In PromQL, using recorded error ratios, the first tier reads:

(
  job:slo_errors_per_request:ratio_rate1h{job="waybill"} > (14.4 * 0.001)
and
  job:slo_errors_per_request:ratio_rate5m{job="waybill"} > (14.4 * 0.001)
)

The constants come from the SLO, so changing the objective changes every alert consistently.

Practicalities

• Low traffic makes ratios jumpy. One failure in ten requests is a 10% error rate. The workbook suggests synthetic traffic, combining services, or minimum request counts.
• Latency SLOs work the same way, using the proportion of requests slower than the threshold as the "bad" ratio. That builds on the Observability and Monitoring track's histograms.
• Dependencies' SLOs are context, not pages, unless your users feel them.
• Keep a small number of SLO alerts per service. They replace most threshold alerts rather than adding to them.

Key terms

Burn rate

How fast the error budget is being consumed relative to the SLO. A burn rate of 1 uses exactly the whole budget over the window.

Precision

The fraction of alerts that correspond to significant events. Low precision means noise.

Recall

The fraction of significant events that produce an alert. Low recall means missed incidents.

Multiwindow alert

An alert that requires a high burn rate over both a long and a short window, so it fires on real burns and resets quickly once they stop.

Read further

• The Site Reliability Workbook, Ch. 5, "Alerting on SLOs" (Free to read online)
  The four alert attributes (precision, recall, detection time, reset time), the six approaches compared in turn, and the recommended multiwindow, multi-burn-rate configuration with its parameters.
• Site Reliability Engineering, Ch. 10, "Practical Alerting" (Free to read online (CC BY-NC-ND 4.0))
  Review the ideas on alerting from time-series data. Compare its rule style with the burn-rate rules from the workbook.