The average says 90 milliseconds CX-311

Counters, gauges, and histograms; why averages hide the users who suffer; and how labels multiply cost and power.

Three shapes of metric

Most metrics systems, Prometheus included, offer a few core types:

• Counter: only increases (resetting to zero on restart), for example http_requests_total. Its raw value is rarely interesting. Its rate is: rate(http_requests_total[1m]) is requests per second, and the function handles resets.
• Gauge: goes up and down, for example queue depth, memory in use, or open connections. Read the value directly.
• Histogram: counts observations into buckets, for example http_request_duration_seconds_bucket{le="0.25"} holds the count of requests that took at most 250 ms. Histograms are how you measure distributions such as latency.

Choose the type by the question. "How many requests per second?" needs a counter, "how deep is the queue now?" a gauge, and "how long do requests take?" a histogram.

Averages hide the users who suffer

Practical Monitoring's statistics primer and the SRE book's "worry about your tail" make the same point. The mean is a poor summary of latency. Latency distributions are skewed. Most requests are fast and a few are very slow, and the few are often a specific population, such as hazmat labels, large customers, or one depot. If 97% of label requests take 30 ms and 3% take 2 s, the mean is about 90 ms, and the dashboard looks green while one in thirty-three prints freezes.

Percentiles describe the distribution: p50 (the median), p90, p99. "p99 = 2 s" means one request in a hundred takes 2 s or longer. For user-facing services, alert and set objectives on high percentiles, and watch the median for typical experience.

Computing percentiles correctly

Percentiles cannot be averaged. The average of ten instances' p99 values is not the service's p99, and neither is the maximum. The fix is to aggregate before computing the percentile. Histogram buckets are counts, so they sum correctly across instances, routes, or time:

histogram_quantile(
  0.99,
  sum by (route, le) (rate(http_request_duration_seconds_bucket{job="waybill"}[1m]))
)

The sum by (route, le) keeps the bucket boundary (le) and the dimension you care about (route) and sums everything else, such as instances and pods. Recording this as a rule (route:http_request_duration_seconds:p99) makes dashboards and alerts cheap and consistent. That is the average-hides-slow-labels lab. Percentiles from histograms are estimates bounded by bucket boundaries, so choose buckets around your objectives (for example, include 0.5 s if 0.5 s is the target).

Resolution matters too

The SRE book also warns about measurement resolution. A one-minute average CPU can hide five-second saturation spikes, and a five-minute rate can hide a 90-second outage. Pick scrape intervals and rate windows to match the speed of the failures you need to see, and remember that longer windows smooth away exactly the short events users notice.

Labels and cardinality

Labels make metrics powerful. sum by (route), by (depot), and by (code) slice one metric many ways. Every unique combination of label values is a separate time series that costs memory, storage, and query time. Labels with a few dozen values (route, status class, region) are fine. Labels with unbounded values (customer ID, request ID, full URL with query string, email) can create millions of series and take down the metrics system.

Observability Engineering argues that this is a structural limit of metrics. They are pre-aggregated, so they cannot answer questions about dimensions you did not anticipate or cannot afford. High-cardinality questions such as "which customer?", "which build?", or "which request?" belong in structured events and traces, which the “Events, logs, and traces” notes cover. Use metrics for the known, bounded questions they answer cheaply.

Key terms

Counter

A value that only increases (and resets on restart), such as requests served. Use its rate, not its raw value.

Gauge

A value that goes up and down, such as queue length, memory in use, or temperature.

Histogram

Counts of observations in buckets (≤ 0.1 s, ≤ 0.25 s, …) from which percentiles can be estimated and correctly aggregated.

Cardinality

The number of unique label combinations of a metric. Each is a separate time series with storage and query cost.

Read further

• Practical Monitoring, Ch. 4, "Statistics Primer" (Purchase)
  Mean, median, percentiles, and standard deviation. Note why the book warns against averages for latency, and against averaging percentiles.
• Site Reliability Engineering, Ch. 6, "Monitoring Distributed Systems" (Free to read online (CC BY-NC-ND 4.0))
  Re-read "Worrying About Your Tail (or, Instrumentation and Performance)" and "Choosing an Appropriate Resolution for Measurements". Both are about how you aggregate.
• Observability Engineering, Chapter "Structured Events Are the Building Blocks of Observability" (Purchase; Honeycomb offers a sponsored download)
  The limitations of metrics as a building block, in particular pre-aggregation and the cost of high-cardinality dimensions.

Observability is the ability to ask new questions of production. Structured events and traces make that possible.

Known questions and new ones

Dashboards and alerts answer questions somebody anticipated: error ratio by route, p99 by route, queue depth. Many incidents are unknown unknowns. Slow requests share a printer firmware, a template version, and one build, a combination nobody would have pre-aggregated. Observability Engineering defines observability as the ability to understand any internal state of the system from its outputs, including states you never predicted, without shipping new code to ask.

That needs data with two properties metrics lack:

• High cardinality: fields with many unique values, such as user, depot, request ID, and build.
• High dimensionality: many fields per record, so you can slice by any combination.

Structured events

The building block is a structured event: one wide record per unit of work, usually one per request per service, carrying everything that might matter:

{
  "ts": "2026-10-02T14:31:07.412Z",
  "service": "waybill",
  "route": "/labels",
  "status": 500,
  "duration_ms": 2104,
  "depot": "7",
  "template": "hazmat-v3",
  "printer_fw": "4.2.0",
  "build": "2.4.1",
  "db_ms": 12,
  "render_ms": 2051,
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
}

From a stream of these you can compute any metric after the fact, and you can also ask "what do the slow ones have in common?". Free-text logs (label done in 2104ms) need fragile parsing and lose that power. Practical rules: log JSON or another structured format to stdout (the Containers in Production track's logs factor), emit one rich event per request rather than many fragments, and include identifiers that join data together, such as trace ID, build, and host or pod.

Traces

A trace stitches events across services. Each operation is a span, with a name, start, duration, parent span, and attributes. Spans that share a trace ID form a tree for one request. Services pass the trace context in request headers (the W3C traceparent standard), and instrumentation libraries such as OpenTelemetry create spans and propagate context automatically for common frameworks.

A trace view answers questions logs answer slowly: which service or dependency spent the time, which call was retried, and where the request fanned out. The book shows a trace as the same structured events with parent and child links. That is a useful way to think about it, because it means you can query spans like any other events.

Logs still matter

Logs remain essential evidence, especially for the host and platform layers: the journal (the Linux Administration track), kernel OOM messages, proxy error logs (the Networking for Operators track), and Kubernetes events (the Kubernetes Operations track). Make them structured where you control the format. Keep timestamps in UTC and synchronised, and keep retention long enough to cover a weekend incident discovered on Monday.

Below the application

When the question is about the host, Systems Performance's observability tools chapter maps the sources: counters (/proc, vmstat, iostat), profiling (sampling stacks with perf), and tracing (tracepoints, kprobes, and eBPF tools that record individual events with low overhead). Use the USE method to choose where to look, and these tools to see inside.

Putting it together

A practical observability stack has metrics for cheap, fast alerting on known symptoms, structured events and traces for exploring the unknown, and logs for platform evidence, all joined by shared identifiers. In an incident, the page comes from a metric, the trace shows where the time went, and the events show what the failing requests have in common.

Key terms

Structured event

A record of one unit of work (for example, one request) as key/value fields, not free text.

High cardinality

A field with many unique values (user ID, build ID, request ID). Expensive for metrics, essential for debugging.

Trace

The tree of spans recording one request's path through services, linked by a shared trace ID.

Span

One timed operation in a trace (an HTTP handler, a database query), with its parent, duration, and attributes.

Read further

• Observability Engineering, Chapter "What Is Observability?" (Purchase; Honeycomb offers a sponsored download)
  The book's definition of observability, why "unknown unknowns" defeat dashboards, and the role of high cardinality and high dimensionality.
• Observability Engineering, Chapters "Structured Events Are the Building Blocks of Observability" and "Stitching Events into Traces" (Purchase; Honeycomb offers a sponsored download)
  What makes an event useful (wide, structured, one per unit of work), and how trace and span IDs connect events across service boundaries.
• Systems Performance, 2nd edition, Ch. 4, "Observability Tools" (Purchase; the USE method summary is free on the author's site)
  The tool types (counters, profiling, tracing, monitoring) and the observability sources in Linux (/proc, /sys, tracepoints, and others). Skim the tool inventories.

Page on the rate at which the error budget is burning, using multiple windows, so you catch fast outages quickly and slow leaks before they cost the month.

From thresholds to budgets

The Observability and Monitoring track built alerts on hand-chosen thresholds: error ratio above 5% for 20 seconds. Once a service has an SLO, there is a better question to ask: how fast are we spending the error budget? An outage that burns 10% of the month's budget in an hour needs a human now. A slow leak that would use the budget over three weeks needs a ticket. A 30-second blip that costs 0.01% needs nothing.

Burn rate

Burn rate is the speed of budget consumption relative to the SLO. At burn rate 1, the budget lasts exactly the window. At burn rate 10, it lasts a tenth of the window. With a 99.9% SLO the budget is 0.1% errors, so an observed error ratio of 1.44% is a burn rate of 14.4. Over a 30-day window that would exhaust the budget in about two days, and every hour at that rate costs 2% of the month's budget.

The general relation is:

budget consumed = burn rate × (alert window / SLO window)

That lets you choose thresholds in terms of what matters, the share of budget lost, instead of raw percentages.

Measuring an alert

The workbook judges every alerting strategy on four attributes:

• Precision: are the alerts real?
• Recall: are real events alerted on?
• Detection time: how quickly does it fire?
• Reset time: how long does it keep firing after the problem is fixed?

It then works through six approaches in turn: target error rate over short windows, longer windows, alert durations, burn-rate alerts, multiple burn rates, and finally multiwindow, multi-burn-rate alerts. Each fixes a weakness of the one before.

The recommended shape

For a 99.9% SLO over 30 days, the workbook's starting configuration is:

An alert fires only when both windows exceed the burn rate. The long window ensures the burn is significant, and the short window ensures it is still happening, so the alert resets within minutes of recovery. In PromQL, using recorded error ratios, the first tier reads:

(
  job:slo_errors_per_request:ratio_rate1h{job="waybill"} > (14.4 * 0.001)
and
  job:slo_errors_per_request:ratio_rate5m{job="waybill"} > (14.4 * 0.001)
)

The constants come from the SLO, so changing the objective changes every alert consistently.

Practicalities

• Low traffic makes ratios jumpy. One failure in ten requests is a 10% error rate. The workbook suggests synthetic traffic, combining services, or minimum request counts.
• Latency SLOs work the same way, using the proportion of requests slower than the threshold as the "bad" ratio. That builds on the Observability and Monitoring track's histograms.
• Dependencies' SLOs are context, not pages, unless your users feel them.
• Keep a small number of SLO alerts per service. They replace most threshold alerts rather than adding to them.

Key terms

Burn rate

How fast the error budget is being consumed relative to the SLO. A burn rate of 1 uses exactly the whole budget over the window.

Precision

The fraction of alerts that correspond to significant events. Low precision means noise.

Recall

The fraction of significant events that produce an alert. Low recall means missed incidents.

Multiwindow alert

An alert that requires a high burn rate over both a long and a short window, so it fires on real burns and resets quickly once they stop.

Read further

• The Site Reliability Workbook, Ch. 5, "Alerting on SLOs" (Free to read online)
  The four alert attributes (precision, recall, detection time, reset time), the six approaches compared in turn, and the recommended multiwindow, multi-burn-rate configuration with its parameters.
• Site Reliability Engineering, Ch. 10, "Practical Alerting" (Free to read online (CC BY-NC-ND 4.0))
  Review the ideas on alerting from time-series data. Compare its rule style with the burn-rate rules from the workbook.