Waybill cannot find Ledger in the new stack INT-77

How containers find each other by name, why `localhost` means "me", and how Compose wires a stack together.

Every container has its own localhost

A container's network namespace contains its own interfaces, its own routing table, its own ports, and its own loopback. localhost inside a container means that container. Code written on a laptop, where the API and database both ran as local processes, often says localhost:7000. Moved into two containers, the same code fails with "connection refused", which is exactly the loopback-only bind from the Networking for Operators track seen from the other side.

How Docker wires containers together

For each container, Docker creates a virtual Ethernet pair. One end becomes the container's eth0, and the other is attached to a bridge on the host. Containers on the same user-defined network can reach each other directly, and Docker runs an embedded DNS server (shown as 127.0.0.11 in the container's resolv.conf) that resolves service and container names to their current addresses. The legacy default bridge does not provide that name resolution, which is one reason to always use project networks.

Compose creates a network per project and attaches every service to it, so in a Compose file:

services:
  api:
    environment:
      LEDGER_URL: http://ledger:7000
    networks: [backend]
  ledger:
    networks: [backend]
networks:
  backend:
    internal: true

ledger resolves from api, the address may change on every recreate, and the name stays the same. internal: true gives the network no route to the outside, which is useful for back-end tiers.

Exposing versus publishing

• Container-to-container traffic on a shared network needs nothing extra. Any port the process listens on is reachable by peers.
• Publishing (ports: ["8080:8080"]) maps a host port to a container port and makes the service reachable through the host's interfaces. Publish only what users or other hosts need, and bind it to a specific host address when you can (127.0.0.1:8080:8080).
• Host networking removes the network namespace entirely. It is fast and simple, and it gives up isolation.

The Container Security chapter adds a security reading: every unnecessary network path is an unnecessary attack path. Segment tiers onto separate networks so that a compromised front end cannot talk to the database directly.

Backing services are configuration

Twelve-Factor calls databases, queues, and other services attached resources. The code holds no knowledge of where they are. It reads a URL from configuration. That one rule makes containers portable. The same image runs in Compose with http://ledger:7000, in Kubernetes with a Service name, and in production behind a managed endpoint. It also makes the compose-DNS lab's fix a configuration change rather than a code change.

Debugging connectivity between containers

docker compose exec api getent hosts ledger          # does the name resolve?
docker compose exec api wget -qO- http://ledger:7000/health
docker network inspect <project>_backend             # who is attached?
docker compose exec ledger ss -ltn                   # is Ledger bound to 0.0.0.0 or only 127.0.0.1?

Work through name, path, listener, and bind address, the same order the Networking for Operators track taught, inside the container's own namespace.

Key terms

Network namespace

A private network stack (interfaces, routes, ports, and its own loopback) for a container.

User-defined network

A Docker network created for a project. Containers on it resolve each other by service or container name through embedded DNS.

Publishing a port

Mapping a host port to a container port (-p 8080:8080), which makes the service reachable from outside the host.

Internal network

A network with no route outside its members. Compose marks it with internal: true.

Read further

• Container Security, Chapter "Container Network Security" (Purchase)
  Container networking models, network namespaces and virtual Ethernet pairs, and the advice on limiting which containers can talk to which.
• The Twelve-Factor App, Factors IV "Backing services" and VII "Port binding" (Free online)
  Treating a database or another service as an attached resource located by configuration, and a service that exports itself by binding a port.
• UNIX and Linux System Administration Handbook, 5th edition, Ch. 25, "Containers" (Purchase)
  The Docker networking section, especially bridge networks and port publishing.

Trace a name from the application's lookup to an authoritative answer, and find where it goes wrong.

Two halves: the client library and the DNS system

A lookup begins inside the application, in the stub resolver provided by the C library or the language runtime. It does not just "ask DNS". It follows /etc/nsswitch.conf, which typically says to check files (that is, /etc/hosts) and then dns. For DNS it reads /etc/resolv.conf for nameservers, search domains, and options. Only then does a query go to a recursive resolver. The recursive resolver walks the hierarchy (root, then .example, then northstar.example) to an authoritative server and caches the answer for its TTL.

A failure can sit in either half, and the tools differ:

• getent hosts name follows the same path as most applications, including /etc/hosts and search domains.
• dig name sends a DNS query directly and bypasses nsswitch and /etc/hosts.
• dig @server name asks one specific server. dig +trace name walks the delegation yourself.

When dig works and the application does not, the difference is in the client half. When both fail, follow the server half.

Search domains and ndots

search corp.example lets people type short names. ndots:N says that names with fewer than N dots are first tried with each search domain appended. With Kubernetes-style ndots:5, a lookup for api.northstar.example first tries api.northstar.example.corp.example. That costs extra queries. If someone created a matching record in the search domain, the lookup returns the wrong address. A trailing dot (api.northstar.example.) marks a name absolute.

Caching and TTL

Every answer carries a TTL. Recursive resolvers, local caches such as systemd-resolved, and some applications hold answers that long, and some runtimes hold them longer. Two operational rules follow. Plan changes around TTL: lower it ahead of a migration, wait out the old TTL, switch, and then restore it. Negative answers are cached too: an NXDOMAIN returned during a misconfiguration persists for the zone's negative-caching time even after you fix the record.

Containers bring their own resolver

Container runtimes write a resolv.conf into each container. Docker on user-defined networks points it at an embedded DNS server that resolves other containers by service name. Kubernetes points it at the cluster DNS service with search domains for the namespace. Two classic failures follow. A container cannot reach the nameserver it was given. Or a program uses localhost to reach a peer that is actually another container, a name problem disguised as a network one. The Containers in Production track returns to the second.

A resolution checklist

1. What exactly does the application report, and with which name?
2. getent hosts <name> on the failing system: same failure?
3. cat /etc/resolv.conf: which servers, search domains, and options?
4. dig @<that server> <name>: does the configured server answer?
5. dig +trace <name> or the authoritative server: is the record itself right?
6. If the record changed recently, which TTL is still running?

Key terms

Stub resolver

The resolver library inside each program. It reads /etc/nsswitch.conf, /etc/hosts, and /etc/resolv.conf, and asks a recursive server.

Recursive resolver

A server that answers clients by querying the hierarchy and caching results.

Authoritative server

A server that holds the zone data for a domain and is the source of truth for its records.

TTL

Time to live, the number of seconds a record may be cached. Old answers persist for up to the TTL after a change.

Read further

• TCP/IP Illustrated, Volume 1, 2nd edition, Ch. 11, "Name Resolution and the Domain Name System (DNS)" (Purchase)
  The namespace and delegation, recursive and iterative queries, caching and TTLs, and the main record types (A, AAAA, CNAME, NS, SOA). Note the role of the resolver library on the client.
• UNIX and Linux System Administration Handbook, 5th edition, Ch. 16, "DNS — The Domain Name System" (Purchase)
  How DNS works for an administrator, /etc/resolv.conf and nsswitch.conf, and the debugging section on dig. Skim the server configuration chapters.

Integration points fail, and the system must not fail with them. Timeouts, circuit breakers, bulkheads, back pressure, and load shedding.

Integration points are where systems die

Release It! starts from production experience: the most dangerous parts of a system are its integration points, every call to another service, database, or API. Remote calls can fail fast, fail slowly, hang, return garbage, or succeed and take forever. The book's stability antipatterns describe how one bad integration point spreads:

• Blocked threads: callers wait on a slow dependency until every worker is stuck.
• Chain reactions: one instance fails, its load shifts to its peers, and they fail in turn.
• Cascading failures: a failure in one layer propagates to the layers calling it.
• Slow responses: worse than fast failures, because they tie up resources on both sides.
• Unbounded result sets: a query that returned 100 rows in testing returns ten million in production.

The common thread is a small problem in one place consuming a shared resource (threads, connections, memory, capacity) that everything else needs.

Patterns that contain failure

Each stability pattern counters one or more antipatterns:

Timeouts. Every remote call needs one, sized to the caller's own deadline. Without timeouts, every other pattern fails, because a hung call holds its thread forever. Propagate deadlines downstream (the SRE book recommends it) so work abandoned by the client is not still being done three services deep.

Retries, carefully. Retrying a transient failure is fine. Retrying immediately, many times, at every layer multiplies load during exactly the moment the system is weakest. Use exponential back-off with jitter, a small maximum, a retry budget (for example, at most 10% extra requests), and retries at one layer only.

Circuit breaker. Track failures to a dependency. When they cross a threshold, open the circuit and fail calls immediately. After a cool-off, let a trial request through (half-open). If it succeeds, close the circuit. The caller saves its resources, and the dependency gets breathing room. Combine with a fallback, such as cached data or a degraded feature, where one exists.

Bulkheads. Partition resources so a failure stays local: separate pools for Ledger calls and for label rendering, separate instances for critical and batch traffic. A flood in one compartment does not sink the ship.

Fail fast. If a request cannot succeed, for example because a dependency's circuit is open or a required resource is missing, reject it at once instead of doing partial work.

Steady state. Anything that accumulates (logs, temp files, sessions, caches, queues) needs an automatic bound or purge. Otherwise the system eventually fails from its own debris, as in the Linux Administration track's notes on disk-full and inode incidents.

Overload and back pressure

The SRE book's chapters on overload and cascading failures add the capacity view. As load approaches capacity, latency climbs, requests time out, clients retry, and the effective load rises further, a positive feedback loop that can take down a healthy fleet. The defences:

• Load shedding: when overloaded, reject excess requests early and cheaply, so the requests you accept still finish within deadline. Throughput of successful requests (goodput) stays high.
• Back pressure: bounded queues that push "slow down" upstream instead of buffering indefinitely.
• Criticality: tag requests by importance (label printing above analytics) and shed the least critical first.
• Client-side throttling: clients that see many rejections reduce their own request rate.
• Graceful degradation: serve a reduced but useful response, such as tracking without the map, rather than none.

Health checks are part of stability

The Containers in Production track and the Kubernetes Operations track warned against health checks that fail on dependency problems. In stability terms, such checks cause chain reactions. Every instance is marked unhealthy, restarted, or removed at once, and a partial outage becomes total. A good design checks what the instance itself controls, reports dependency trouble through readiness or degraded responses, and lets circuit breakers handle the dependency.

Reviewing a service

For each dependency of a service, ask: Is there a timeout, and is it shorter than our own deadline? What is the retry policy? Is there a circuit breaker and a fallback? Is the connection pool isolated? What happens to our users when this dependency is slow rather than down? Writing the answers down for Waybill, covering Ledger, the label renderer, and the database, is a worthwhile exercise. It is also the most effective reliability review most teams never do.

Key terms

Timeout

A bound on how long to wait for a remote call. Without one, a slow dependency holds your resources indefinitely.

Circuit breaker

Stops calling a failing dependency after repeated errors, fails fast for a while, then probes to see if it has recovered.

Bulkhead

Partitioning resources (thread pools, connection pools, instances) so a failure in one part cannot exhaust all of them.

Load shedding

Deliberately rejecting some work when overloaded, so the rest can succeed.

Read further

• Release It!, 2nd edition, Ch. 4, "Stability Antipatterns", and Ch. 5, "Stability Patterns" (Purchase)
  Integration points, chain reactions, cascading failures, blocked threads, slow responses, and unbounded result sets. Then timeouts, circuit breaker, bulkheads, steady state, fail fast, shed load, and create back pressure. For each pattern, note which antipattern it counters.
• Site Reliability Engineering, Ch. 22, "Addressing Cascading Failures" (Free to read online (CC BY-NC-ND 4.0))
  Causes of cascading failure (server overload, resource exhaustion, retries), preventing overload, and the guidance on retries and deadlines.
• Site Reliability Engineering, Ch. 21, "Handling Overload" (Free to read online (CC BY-NC-ND 4.0))
  Per-customer limits, client-side throttling, criticality, and why handling overload gracefully beats trying never to be overloaded.