Healthy on localhost, 502 everywhere else CHG-5120

Open2 versionsNetworking · Easy · Fix · about 20 min ·Linux

Lab machine

A private Linux machine with the problem already set up. Sessions last up to 60 minutes.
Sasha Lind opened CHG-5120 at 10:40SEV-3

Waybill moved to a new host this morning and the change record says the health check passed. Every scanner request through the depot proxy gets 502.

The new host, depot-app-1, has three addresses: loopback, the depot LAN that Gatehouse uses, and a public uplink that faces the internet.

"Health check passed on the box. I tested it before closing the change, I promise." (Sasha)

She did, from the host itself. That proves the service answers on an address this machine can reach, not on the one Gatehouse dials. The security review adds a condition:

"depot-app-1 has a public uplink (eth1). Waybill must not be reachable on it, only through Gatehouse." (Priya)

Your task

Make Waybill listen where Gatehouse can reach it, on the depot LAN address, and nowhere public. Change the service's configuration (not the proxy), restart it through the service manager, and test the path the scanners use.

On the machine

  • etc/network/interfaces: which loopback address stands in for which NIC
  • ss -ltn: what Waybill really listens on
  • etc/waybill/waybill.env
  • The customer path through Gatehouse, in TICKET.md

Timeline

09:30CHG-5120: Waybill moves from depot-app-0 to depot-app-1.
09:52Sasha runs curl 127.0.0.1/healthz on the new host: 200. Change closed.
10:05Scanners: 502 Bad Gateway through Gatehouse.
10:40CHG-5120 reopened.

Done when

  1. Scanners get 200 through Gatehouse, from the managed Waybill process.
  2. Nothing answers on the public uplink.

Hints

Hint 1

`ss -ltn`: which addresses does Waybill listen on?

Hint 2

A successful request to 127.0.0.1 proves only that this host can reach it.

Hint 3

The proxy dials the depot LAN address of this host (10.0.4.12, or 127.0.4.12 in the lab). Check etc/network/interfaces.

Hint 4

Bind the service to this host's private interface, and not to the public one.

Show the solution

`ss -ltn` shows where Waybill listens: only on 127.0.0.1, or on 127.0.0.1 and 127.0.4.21, the LAN address of depot-app-2 that came along with the copied config. Gatehouse dials this host's LAN address, 127.0.4.12 (10.0.4.12 in production). Set `WAYBILL_BIND` in `etc/waybill/waybill.env` to `127.0.4.12` (keeping 127.0.0.1 alongside it for local checks is fine), restart the service, then confirm with `ss -ltn`, a request through Gatehouse, and a refused connection on the public uplink.