Valid certificate, broken chain

Lab machine

A private Linux machine with the problem already set up. Sessions last up to 60 minutes.

Ana Costa opened INC-2519 at 07:12SEV-2

depot-3's gateway certificate was renewed yesterday. This morning every cold-storage scanner rejects it, yet the gateway opens fine in a browser on Ana's laptop.

Pallets in the cold store must be scanned within the hour or checked again by hand. The scanners talk to depot-3's gateway over TLS and trust exactly one certificate: the Northstar root.

"Opens fine in my browser, padlock and everything. So it's the scanners?" (Ana, 07:05)

The renewed certificate is not expired, and its name matches the gateway.

Your task

Make Gatehouse send a chain that a client trusting only the Northstar root can verify. Fix it on the server, not on the scanners.

On the machine

bin/scanner-test runs what a scanner runs
openssl s_client -connect 127.0.0.3:8443 -servername depot3.northstar.example -showcerts
etc/nginx/nginx.conf, etc/ssl/, and the files from the PKI portal in home/sasha/renewal/
trust/northstar-root.crt, the only certificate the scanners trust

Timeline

2026-10-03Sasha renews the depot-3 gateway certificate (CHG-4460) and installs it.

06:30Cold-storage scanners: "unable to get local issuer certificate". Pallets queue offline.

07:05"Opens fine in my browser, padlock and everything." (Ana)

07:12INC-2519 lands with you.

Done when

A scanner that trusts only the Northstar root verifies depot3.northstar.example.
Gatehouse sends the depot-3 certificate followed by the 2026 issuing CA, and nothing else.
The scanners' trust store is unchanged; it holds only the Northstar root.
The fix survives a restart and a reboot of depot-3.

How it is checked:The checker requires trust/ to hold only the original Northstar root. It then handshakes like a scanner, trusting only that root and checking the host name, and compares the certificates Gatehouse sends with the depot-3 certificate and the 2026 issuing CA. It repeats this after restarting Gatehouse and after rebooting the simulated host.

Passed. Here is how others fixed it

This lab has 2 versions. Reset for a fresh machine, and you may get a different one.

Break it for a friend. Pick the version they get, and the link is copied for you to send: version 1 · version 2

Other fixes that also pass:

Append the 2026 issuing CA to the existing depot-3 certificate file itself, after the depot-3 certificate, and reload.

Fixes that look right but are not:

Adding the intermediate to the client's trust store. It fixes one device and leaves 300 broken.
Using last year's issuing CA because it is already on the host.
Putting the CA first. nginx then pairs the wrong certificate with the key and refuses the reload.
Including the root in the chain. Clients must already have it, so it only adds bytes.

Why it broke. The scanners trust the Northstar root and nothing else, while the depot-3 certificate was issued by "Northstar Issuing CA 2026". Without that intermediate in the handshake there is no path from the certificate to the root. Browsers often fill the gap from a cache or by downloading the issuer, which is why the gateway looked fine on a laptop. The old 2024 issuing CA on the host has a different name and key, so it cannot complete the chain either.

How Mara worked it

bin/scanner-test
Reproduce exactly what the scanners see, with their trust store.
openssl s_client -connect 127.0.0.3:8443 -servername depot3.northstar.example -showcerts < /dev/null
Count the certificates the server sends and read each Issuer and Subject.
openssl x509 -noout -subject -issuer -in <file> on every CA certificate you have
The chain only works if each certificate's Issuer is the next one's Subject.
cat depot3.crt northstar-issuing-ca-2026.crt > fullchain.pem, point ssl_certificate at it
The server's own certificate first, then its issuer. Never the root.
bin/hostctl reload gatehouse, then bin/scanner-test again
Verify like a scanner, then restart and reboot to be sure it sticks.

Hints

Hint 1

Run `bin/scanner-test`, then look at how many certificates `openssl s_client -showcerts` prints.

Hint 2

Compare the Issuer of the depot-3 certificate with the Subject of each CA certificate you have.

Hint 3

nginx's ssl_certificate file may hold several certificates. Order matters: the server's own first.

Hint 4

The issuing CA in etc/ssl/northstar/ is last year's. Check its Subject.

Show the solution

Concatenate the depot-3 certificate and `home/sasha/renewal/northstar-issuing-ca-2026.crt`, in that order, into one file (for example `etc/ssl/depot3/fullchain.pem`), point `ssl_certificate` at it, check the configuration, and reload Gatehouse with `bin/hostctl reload gatehouse`.

Networking for Operators

TLS and HTTP

What a TLS handshake verifies, why a correct certificate can fail on one client, and what HTTP status codes say about where a failure is.

What TLS proves

TLS gives a client three guarantees: the bytes are encrypted, they are not modified in transit, and the server holds the private key for a certificate that a trusted authority issued for the name the client asked for. To establish the last one, the client checks:

Chain: the certificate links through intermediates to a root in the client's trust store.
Name: the requested hostname matches one of the certificate's subject alternative names.
Validity: the current time is between notBefore and notAfter, according to the client's clock.
Optionally, revocation, through OCSP or CRLs.

Each check fails with its own message, and the message names the part of the system to look at. "Unable to get local issuer certificate" means a missing intermediate or a trust store problem. "Hostname mismatch" means you connected by a different name than the certificate covers. "Expired" or "not yet valid" mean the dates disagree with someone's clock.

The clock is part of the trust decision

Because validity is judged by the client's clock, a client whose clock is wrong by days sees a perfectly good certificate as expired or not yet valid. The giveaway is scope. One site or one class of device fails, everyone else succeeds, and the certificate's dates are fine when checked from a correct machine. Look at timedatectl or the device's NTP configuration, not at the certificate. The same skew breaks other time-based security too: Kerberos tickets, signed URLs, and one-time codes.

Looking at a handshake

openssl s_client -connect 10.0.4.12:443 -servername tracking.northstar.example -showcerts </dev/null
openssl s_client -connect ... </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer -dates -ext subjectAltName
curl -v --resolve tracking.northstar.example:443:10.0.4.12 https://tracking.northstar.example/health

-servername sends SNI, without which a server hosting several names may return a default certificate and mislead you. curl --resolve pins the address while keeping the right name, which is the clean way to test one backend.

Latency is a protocol property

HPBN's central lesson for operators is that round trips are expensive. TCP's handshake costs one round trip, and a full TLS 1.2 handshake adds two more (TLS 1.3 adds one, or none on resumption). A proxy that does not reuse upstream connections, or clients that never resume sessions, pay this on every request. When latency is high but servers are idle, count handshakes before tuning code.

Reading HTTP status codes as locations

Status codes are produced by whichever component answered:

2xx/3xx: the request reached something that handled it.
4xx: the server judged the request wrong, for example 404, 401/403, or 429 (rate limiting).
500: the application itself failed.
502 Bad Gateway: a proxy got no valid response from its upstream (connection refused, reset, or garbage).
503 Service Unavailable: overload, maintenance, or a proxy with no healthy backends.
504 Gateway Timeout: a proxy waited for its upstream and gave up.

A 502 or 504 tells you the request crossed the proxy successfully. The fault is behind it. That one fact halves the search, and the notes on proxies build on it.

Key terms

Certificate chain: The server certificate plus intermediates, leading to a root the client already trusts.
SNI: Server Name Indication. The client sends the hostname in the handshake so one IP address can serve several certificates.
Validity period: The notBefore and notAfter dates. Clients compare them with their own clock.
502 / 504: Status codes generated by a proxy or gateway: 502 means the upstream gave an invalid response or none, and 504 means the upstream did not answer in time.

Read further

High Performance Browser Networking, Ch. 4, "Transport Layer Security (TLS)" (Free to read online)
The handshake and its round trips, the chain of trust and certificate authorities, SNI, session resumption, and OCSP and revocation. Note exactly what the client checks.
High Performance Browser Networking, Ch. 9, "Brief History of HTTP", and Ch. 11, "HTTP/1.X" (Free to read online)
The request/response model, keep-alive, and why one connection per request is costly. Skim the browser optimisation advice.
TCP/IP Illustrated, Volume 1, 2nd edition, Ch. 18, "Security — EAP, IPsec, TLS, DNSSEC, and DNS Security" (Purchase)
The TLS section, for certificate structure and the handshake at the protocol level.

Valid certificate, broken chain INC-2519