A VPN update stole the API route NET-944

OpenNetworking · Hard · Investigate · about 35 min ·Linux + Docker

Lab machine

A private Linux machine with Docker Engine. Starting takes about 30 seconds. Sessions last up to 60 minutes.
Teo Marin opened NET-944 at 08:25SEV-3

Since this morning's partner VPN update, depot-7 cannot reach the Waybill API. DNS works, the API is up, and the partner link works.

The partner carrier's VPN pushes a list of routes to depot-7. This morning's update added one more.

"It is not DNS, I checked. The API answers for every other depot. And please do not just turn the VPN off, rail freight goes over it." (Teo)

Your task

Find the route that sends API traffic into the tunnel, remove only that one from the pushed route list, and re-apply the VPN routes. Partner traffic must keep using the tunnel.

On the machine

  • docker exec depot7 ip route, ip route get <address>
  • vpn/routes.conf (mounted at /etc/vpn in depot7) and the vpn-up script
  • Containers waybill-api and partner-gw

Timeline

07:30Partner carrier pushes a VPN routes update (PNR-77).
07:31depot-7 re-applies its VPN routes.
07:40depot-7 scanners time out talking to api.northstar.example.
08:25NET-944 opened.

Done when

  1. From depot7 the API answers over the LAN, even after vpn-up re-applies its routes.
  2. Partner traffic still goes through the tunnel.

Hints

Hint 1

`ip route get <address>` shows the exact route the kernel uses, not just the table.

Hint 2

The most specific prefix wins: a /24, a /28, or a /32 beats a /16 no matter which was added first.

Hint 3

Fixing only the live route lasts until the VPN client reconnects.

Hint 4

Find the pushed line in vpn/routes.conf that covers the API's address, remove it, then run `vpn-up` in depot7.

Show the solution

`docker exec depot7 ip route get 10.20.14.9` shows the API going via 10.99.0.20, the tunnel peer. The VPN update pushed a prefix inside 10.20.14.0/24 (a /24, a /28, or the API's own /32), which is more specific than the corporate 10.20.0.0/16. Remove that line from vpn/routes.conf and run `docker exec depot7 vpn-up`; the API answers over the corporate network again while 10.30.0.0/16 still uses the tunnel.