The network is up and nothing resolves

Lab machine

A private Linux machine with the problem already set up. Sessions last up to 60 minutes.

Depot-5 on-call opened INC-2330 at 07:20SEV-3

Since this morning's scanner container restart, depot-5 cannot resolve the API name. The on-call wants to hardcode the IP.

depot-5 is the airport depot; its scanners upload customs data that has to arrive before each flight closes.

"Network's up, API's up. Should I just hardcode the IP in scanner.conf?" (depot-5 on-call)

The IP works because the network is fine. The name fails because something between the name and the IP is not. A hardcoded IP would work until the API moves, and then fail with nobody remembering why.

Your task

Find which resolver the container uses and what it answers, then repair the container's resolver configuration so the API name resolves to its LAN address. Leave the API name in scanner.conf, and do not pin it in /etc/hosts.

On the machine

The container's depot/etc/nsswitch.conf, depot/etc/resolv.conf, depot/etc/hosts, depot/etc/scanner.conf
depot/bin/getent hosts NAME and depot/bin/curl -v URL use the container's resolver
dig -p <port> @<server> against each DNS server (port in TICKET.md)
The DHCP lease under depot/, and CHG-4471 and the image notes under docs/

Timeline

Last weekCHG-4471 retires the old depot resolver.

06:55depot-5 scanner container restarts after an image update.

07:02"Could not resolve host: api.northstar.example". Scan events queue locally.

07:20INC-2330: "Should I just hardcode the IP?"

Done when

The name resolves to the API LAN address (10.0.4.12 in production, 127.0.4.12 in the lab).
HTTP by name reaches the service.

How it is checked:The checker confirms the DNS servers were not modified, `scanner.conf` still uses the name, `hosts` does not pin it, and that the name resolves and HTTP by name works from the container.

Passed. Here is how others fixed it

This lab has 2 versions. Reset for a fresh machine, and you may get a different one.

Break it for a friend. Pick the version they get, and the link is copied for you to send: version 1 · version 2

Other fixes that also pass:

Point resolv.conf at the internal resolver named in the DHCP lease, then fix the image so a rebuild keeps it.
Use `search` and `nameserver` lines from CHG-4471 together, so short names also resolve.

Fixes that look right but are not:

Changing application code or restarting Waybill first.
Pinning the name in /etc/hosts or hardcoding the IP, which hides the fault until the API moves.
Pointing the container at a public resolver that cannot see internal names.

Why it broke. IP connectivity and DNS are independent stages; a direct IP success narrows the fault but does not prove name resolution.

How Mara worked it

depot/bin/curl -v http://api.northstar.example:<port>/healthz and the same by IP
By IP it works, by name it does not. The fault is in name resolution, not the network.
cat depot/etc/nsswitch.conf depot/etc/resolv.conf depot/etc/hosts
Which sources the container asks for names (nsswitch.conf), which DNS servers (resolv.conf), and whether anything pins the name.
dig -p <port> @<each nameserver> api.northstar.example
Ask each server directly. If one returns the A record, DNS works and the fault is in how the container asks.
Read the DHCP lease, CHG-4471, and the image notes under docs/
What changed in this container, a retired resolver baked into the image or a new base image.
Fix the container's resolver config, then depot/bin/getent hosts and depot/bin/curl by name
Name resolution and HTTP by name, from inside the container.

Hints

Hint 1

By IP the API answers. The network is fine; the problem is turning the name into that IP.

Hint 2

Ask each DNS server directly with dig. If one gives the answer, DNS works, and the question is why the container does not get it.

Hint 3

Two files decide how the container looks up a host name: which sources to ask (nsswitch.conf), and which DNS servers (resolv.conf).

Hint 4

Fix the container's configuration, not the DNS servers, and keep the scanner addressing the API by name.

Show the solution

Confirm the layer first: by IP the API answers, by name it does not, and `dig -p <port> @127.0.10.53` returns the A record, so DNS itself works. Then find why the container's own lookups fail. Either `depot/etc/resolv.conf` still lists the retired 127.0.0.53, which answers SERVFAIL (point it at 127.0.10.53 from the DHCP lease and CHG-4471), or `depot/etc/nsswitch.conf` has `hosts: files` without `dns`, so the container never asks DNS at all (make it `hosts: files dns`). Test with `depot/bin/getent hosts` and `depot/bin/curl` by name. In production, fix the image so the next rebuild does not bring the fault back.

Networking for Operators

DNS and name resolution

Trace a name from the application's lookup to an authoritative answer, and find where it goes wrong.

Two halves: the client library and the DNS system

A lookup begins inside the application, in the stub resolver provided by the C library or the language runtime. It does not just "ask DNS". It follows /etc/nsswitch.conf, which typically says to check files (that is, /etc/hosts) and then dns. For DNS it reads /etc/resolv.conf for nameservers, search domains, and options. Only then does a query go to a recursive resolver. The recursive resolver walks the hierarchy (root, then .example, then northstar.example) to an authoritative server and caches the answer for its TTL.

A failure can sit in either half, and the tools differ:

getent hosts name follows the same path as most applications, including /etc/hosts and search domains.
dig name sends a DNS query directly and bypasses nsswitch and /etc/hosts.
dig @server name asks one specific server. dig +trace name walks the delegation yourself.

When dig works and the application does not, the difference is in the client half. When both fail, follow the server half.

Search domains and ndots

search corp.example lets people type short names. ndots:N says that names with fewer than N dots are first tried with each search domain appended. With Kubernetes-style ndots:5, a lookup for api.northstar.example first tries api.northstar.example.corp.example. That costs extra queries. If someone created a matching record in the search domain, the lookup returns the wrong address. A trailing dot (api.northstar.example.) marks a name absolute.

Caching and TTL

Every answer carries a TTL. Recursive resolvers, local caches such as systemd-resolved, and some applications hold answers that long, and some runtimes hold them longer. Two operational rules follow. Plan changes around TTL: lower it ahead of a migration, wait out the old TTL, switch, and then restore it. Negative answers are cached too: an NXDOMAIN returned during a misconfiguration persists for the zone's negative-caching time even after you fix the record.

Containers bring their own resolver

Container runtimes write a resolv.conf into each container. Docker on user-defined networks points it at an embedded DNS server that resolves other containers by service name. Kubernetes points it at the cluster DNS service with search domains for the namespace. Two classic failures follow. A container cannot reach the nameserver it was given. Or a program uses localhost to reach a peer that is actually another container, a name problem disguised as a network one. The Containers in Production track returns to the second.

A resolution checklist

What exactly does the application report, and with which name?
getent hosts <name> on the failing system: same failure?
cat /etc/resolv.conf: which servers, search domains, and options?
dig @<that server> <name>: does the configured server answer?
dig +trace <name> or the authoritative server: is the record itself right?
If the record changed recently, which TTL is still running?

Key terms

Stub resolver: The resolver library inside each program. It reads /etc/nsswitch.conf, /etc/hosts, and /etc/resolv.conf, and asks a recursive server.
Recursive resolver: A server that answers clients by querying the hierarchy and caching results.
Authoritative server: A server that holds the zone data for a domain and is the source of truth for its records.
TTL: Time to live, the number of seconds a record may be cached. Old answers persist for up to the TTL after a change.

Read further

TCP/IP Illustrated, Volume 1, 2nd edition, Ch. 11, "Name Resolution and the Domain Name System (DNS)" (Purchase)
The namespace and delegation, recursive and iterative queries, caching and TTLs, and the main record types (A, AAAA, CNAME, NS, SOA). Note the role of the resolver library on the client.
UNIX and Linux System Administration Handbook, 5th edition, Ch. 16, "DNS — The Domain Name System" (Purchase)
How DNS works for an administrator, /etc/resolv.conf and nsswitch.conf, and the debugging section on dig. Skim the server configuration chapters.

The network is up and nothing resolves INC-2330