Which depots are down? OPS-2262

What it means for a service to listen, and how refusals and timeouts tell you different things.

A socket is an address, a port, and a program

A server program asks the kernel for a socket, binds it to a local address and port, and listens. Clients connect to that address and port. The kernel completes the TCP handshake and the program accepts the connection. Beej's guide shows the same sequence from inside the program: socket(), bind(), listen(), accept().

The bind address is the decision operators most often get wrong:

Development defaults usually bind to loopback for safety. Deployed unchanged, the service then passes every local health check and is unreachable from its proxy. That is the incident in the localhost-is-not-public lab.

Reading the listener table

ss -ltnp lists TCP listeners with their bind address, port, and owning process. Read it before you blame the network:

State  Local Address:Port  Process
LISTEN 127.0.0.1:8080      users:(("waybill",pid=4121,fd=7))

The first column of interest is the local address. 127.0.0.1 here already explains a remote "connection refused".

The handshake and its cost

TCP opens a connection with three segments: the client sends SYN, the server replies SYN-ACK, and the client sends ACK. That costs one round trip before the request can travel. High Performance Browser Networking builds a whole argument on this. With 80 ms of round-trip time, every new connection adds at least 80 ms before the first byte of the request, and TLS adds more. Connection reuse (keep-alive, pooling) is therefore a performance feature, and a proxy that opens a fresh upstream connection per request pays this cost every time.

Failures have shapes

The way a connection fails tells you where to look:

• Connection refused: the target host answered with a reset. The route works. Nothing listens on that address and port, or a firewall actively rejects.
• Timed out: silence. Packets were dropped by a firewall, a wrong route, or a host that is down. Look at the path, not the process.
• Connection reset mid-request: something closed the connection abruptly, such as a crashing process, a proxy timeout, or a middlebox.
• No route to host: the local routing decision or an ICMP reply says the destination is unreachable.

Treat the error text as data. "Refused in 2 ms" and "timed out after 75 s" exclude different halves of the system.

Verify from the user's side

A check run on the server against 127.0.0.1 is the narrowest possible test. After any change, repeat the request the way a customer makes it: the public name, through the proxy, from another host. The labs score you on that final state, not on the loopback check.

Key terms

Socket

An endpoint for communication, identified for TCP by address and port. A connection is a pair of them.

Bind address

The local address a listening socket accepts connections on. 127.0.0.1 means local only. 0.0.0.0 means every IPv4 address on the host.

Connection refused

The destination host answered with a reset because nothing listens on that address and port. The network path works.

Timeout

No answer at all. Something dropped the packets, such as a firewall, a wrong route, or a dead host.

Read further

• Beej's Guide to Network Programming, Sections "What is a socket?" and "System Calls or Bust" (`bind()`, `listen()`, `accept()`, `connect()`) (Free to read online)
  What a socket is from a program's side, and what bind() to a specific address means compared with INADDR_ANY. Read the example servers to see the call sequence.
• High Performance Browser Networking, Ch. 2, "Building Blocks of TCP" (Free to read online)
  The three-way handshake and its cost in round trips, and why new connections are slow at first (slow start). This is the latency view an operator needs.
• TCP/IP Illustrated, Volume 1, 2nd edition, Ch. 13, "TCP Connection Management" (Purchase)
  Connection establishment and teardown, the TCP state diagram, and how a host answers a connection to a port with no listener (a reset).

Model a request as a chain of hops, test the middle first, and verify the fix from where the customer stands.

Draw the path before you test it

Every customer request crosses a chain of components, each of which can fail on its own:

scanner → DNS → load balancer → reverse proxy → Waybill :8080 → Ledger / database

Write the chain down at the start of an incident, with the address and port of each hop. Half of all proxy incidents become obvious at this point. The proxy's upstream says :8000, and the service's listener table says :8080.

Split the path

With a chain of six components, testing them in order from one end wastes time. Test in the middle instead. A request from the proxy host straight to the application's address and port answers "is the fault in front of or behind this point?". Then split the remaining half. This is the halving habit from the troubleshooting method concept note, and it turns a vague "the site is down" into a named hop within two or three requests.

Two rules make each test meaningful:

1. Test from the upstream side of each hop. "Can the proxy reach the app?" must be tested from the proxy host. Your laptop has different routes, DNS, and firewall rules.
2. Use the same name, port, and protocol the real hop uses. Check the proxy's configuration rather than assuming.

What proxies tell you

A reverse proxy is itself an excellent witness. Its error log records why an upstream request failed: connection refused, timeout, reset, or an invalid header. Its access log can record upstream address, status, and timing next to the client status. A client-side 502 paired with "connect() failed (111: Connection refused) while connecting to upstream" names both the hop and the failure shape.

What the proxy is running

A proxy reads its configuration at start and on reload, not every time you save a file. After a change, three things can differ: the file you edited, the configuration the proxy would load now (nginx -t and nginx -T print it, merged from every included file), and the configuration the running process loaded. Included files are read in name order, and a later definition can quietly replace an earlier one, so the file you fixed is not always the one that wins.

Apply a change with a test and a reload. A reload starts new workers with the new configuration and lets the old ones finish their requests. The listener stays open. A restart drops every connection the proxy is carrying, for every site behind it, not just the one you are fixing.

Health checks and load balancers

Load balancers send traffic only to backends whose health check passes, and they remove backends that fail. The SRE book's datacenter load-balancing chapter shows how much depends on that signal. Two failure modes recur:

• The check is too shallow. It requests / or a static page, so a backend with a dead database stays in rotation and fails every real request.
• The check is too deep. It calls every dependency, so one dependency outage removes all backends at once, and partial service becomes total outage.

A good readiness check answers "can this instance serve its main requests right now?" and is cheap enough to run every few seconds. The SRE book also describes a lame duck state: a backend that is shutting down stops accepting new work but finishes what it has. Graceful shutdown in containers and Kubernetes implements the same idea.

Balancing policy

Round robin sends equal request counts to each backend. When requests differ in cost or backends differ in capacity, equal counts produce unequal load. The SRE book discusses policies that use backend feedback, such as least-loaded and weighted round robin, and the subtle failures each one has. For operations work, know which policy your balancer uses and whether its view of backend health matches reality.

Close with the customer's view

After a change, verify from the outside, with the public name, through every hop, from a network like the customer's. Then confirm in the logs that the hop you changed is the one now succeeding. The incident timeline should include each test you ran, its result, and the time. That is the evidence that the halving method is meant to produce.

Key terms

Reverse proxy

A server that accepts client requests and forwards them to one or more upstream servers, for example nginx, HAProxy, or Envoy.

Upstream

The backend a proxy forwards to, defined by an address and port in its configuration.

Health check

A periodic probe a load balancer uses to decide whether to send traffic to a backend.

Split the path

Testing the middle hop first so that one result excludes half of the remaining components.

Read further

• Site Reliability Engineering, Ch. 19, "Load Balancing at the Frontend", and Ch. 20, "Load Balancing in the Datacenter" (Free to read online (CC BY-NC-ND 4.0))
  In Ch. 19, DNS-based balancing and its limits. In Ch. 20, health checking (the "lame duck" state), and why simple round robin misbehaves with uneven backends.
• High Performance Browser Networking, Ch. 1, "Primer on Latency and Bandwidth" (Free to read online)
  The components of latency (propagation, transmission, processing, queuing), and why adding hops adds latency that bandwidth cannot remove.
• UNIX and Linux System Administration Handbook, 5th edition, Ch. 19, "Web Hosting" (Purchase)
  The sections on load balancers and proxies, including reverse proxy configuration and health checks.

Symptoms before causes, the four golden signals, and the anti-patterns that make monitoring noisy and blind at once.

Two questions monitoring must answer

The SRE book frames monitoring around two questions: what is broken, and why. The first is answered by symptoms, what users experience: errors, slowness, unavailability. The second is answered by causes, such as a full disk, an exhausted connection pool, or a failed dependency. Both matter, but they serve different purposes:

• Symptoms drive alerts. A user-facing problem exists, and someone must act.
• Causes drive dashboards and investigation. Here is where to look.

An alerting setup built on causes ("page if memory > 90%") fails twice. It pages for conditions that hurt nobody, and it stays silent for failures whose cause nobody predicted. The errors-without-a-page lab is the second failure exactly.

The four golden signals

For a user-facing service, the book recommends measuring at least:

• Latency: how long requests take. Measure successes and failures separately, because a fast error is still an error.
• Traffic: demand on the system, such as requests per second, by route.
• Errors: the rate of failed requests, explicit (5xx) and implicit (wrong content, too slow to count).
• Saturation: how "full" the service is, meaning the resource closest to its limit, often a leading indicator.

These describe the service from its users' side. The Linux Administration track's USE method describes each resource from the inside. You need both, the golden signals to notice and USE to explain.

White box and black box

White-box monitoring uses internals the service exposes: metrics, logs, and traces. It is detailed and often points at causes. Black-box monitoring tests behaviour from outside, as a user would. A probe requests the public URL, through DNS, TLS, and the load balancer. It sees what white-box cannot, such as an expired certificate, a broken route, or a proxy pointing at the wrong port, all from the Networking for Operators track. Use black-box probes for the few user journeys that matter most and white-box metrics for everything else.

Anti-patterns

Practical Monitoring opens with the mistakes most teams make and pairs each with a design pattern:

• Tool obsession: believing the next tool will fix monitoring. Decide what to measure first.
• Monitoring as a job: a separate team owns monitoring, so the service owners do not. Service owners should instrument and own their alerts.
• Checkbox monitoring: collecting easy metrics (CPU, ping) so that "we have monitoring", without knowing whether the service works. Start from the user's perspective.
• Using monitoring as a crutch: piling alerts on a fragile system instead of fixing it.
• Manual configuration: targets and alerts added by hand drift out of date. Configure monitoring automatically from the same source as deployment.

Monitor the monitoring

An alert over metrics that have stopped arriving evaluates to nothing, and nothing does not fire. If a deploy moves the metrics port, renames a job, or breaks the exporter, every alert on that service goes silent at the moment you most need it. Two safeguards are needed: an alert on scrape health (up{job="waybill"} == 0), and an alert on absence (absent(up{job="waybill"}), which covers the case where the target disappeared from discovery entirely). Then test them. Break the scrape on purpose in staging and confirm the page arrives.

Key terms

Symptom

What users experience, such as errors, slowness, or unavailability. "Is it broken?"

Cause

A reason for a symptom, such as a full disk, high memory, or a failed dependency. "Why is it broken?"

Golden signals

Latency, traffic, errors, and saturation. The SRE book's minimum set for a user-facing system.

Black-box monitoring

Testing externally visible behaviour as a user would, for example a probe that requests the public URL.

Read further

• Site Reliability Engineering, Ch. 6, "Monitoring Distributed Systems" (Free to read online (CC BY-NC-ND 4.0))
  Definitions (white-box, black-box, dashboards, alerts), why to monitor, symptoms versus causes, the four golden signals, worrying about your tail, and choosing an appropriate resolution for measurements.
• Practical Monitoring, Ch. 1, "Monitoring Anti-Patterns", and Ch. 2, "Monitoring Design Patterns" (Purchase)
  Each anti-pattern and its matching design pattern, especially composable monitoring, monitoring from the user's perspective, and continual improvement.