The certificate expired 75 days early

Lab machine

A private Linux machine with the problem already set up. Sessions last up to 60 minutes.

Sasha Lind opened OPS-833 at 06:40SEV-3

depot-7's scanners stopped printing at 06:10: certificate expired. Every other depot accepts the same certificate. Platform wants to renew it early.

The label service certificate is valid for weeks yet. From depot-7, it looks expired.

"Easiest fix is to renew it early, right? Takes five minutes." (Sasha)

A certificate has a start and an end date. A client decides whether it is valid by comparing those dates with its own clock. If the client's clock is wrong, a perfectly good certificate looks expired or not yet valid.

Your task

Prove whether the certificate or the clock is wrong, then fix the cause: give the depot a working time source so its clock corrects itself and the scanners recover. Renewing the certificate or turning off verification does not count.

On the machine

/var/log/depot/scanner.log
depot-timesync status
depot-run date, depot-run openssl s_client … (commands on the depot's clock)
/etc/depot/timesync.conf

Timeline

Sepntp1.depot7 is decommissioned with the old rail-yard switch.

Overnightdepot-7's clock drifts further ahead with nothing to correct it.

06:10Scanners: "certificate has expired". Printing stops at the rail yard.

06:40OPS-833: "Shall I just renew the cert?"

Done when

The depot clock is synchronised from a working time source.
The scanners reach the label service with normal certificate verification, and the certificate was not replaced.

How it is checked:The checker confirms the certificate was not replaced and verification is on, measures the depot clock's skew, and makes a verified request from the depot clock to the label service.

Passed. Here is how others fixed it

This lab has 2 versions. Reset for a fresh machine, and you may get a different one.

Break it for a friend. Pick the version they get, and the link is copied for you to send: version 1 · version 2

Other fixes that also pass:

Set the depot's time source to any healthy internal NTP server listed in the network docs, then confirm with depot-timesync.
Keep a step limit but raise it above the current offset, so the first correction goes through.

Fixes that look right but are not:

Renewing an unexpired certificate while leaving the clock wrong.

Why it broke. TLS validation depends on the client clock as well as certificate contents. A clock is only as good as its synchronisation: a dead server, or a safety limit that refuses a large correction, both leave it free-running, and nothing complains until something checks a date.

How Mara worked it

tail /var/log/depot/scanner.log
The scanners say expired. Every other depot accepts the same certificate.
depot-run date against date
The depot's clock is 75 days ahead. To programs there, the certificate has expired.
depot-timesync
Whether its server answers, and whether the daemon refused to step the clock.
Fix /etc/depot/timesync.conf: the retired server, or the step limit
Fix the cause: the clock, not the certificate.
depot-run date and depot-run openssl s_client …
The clock is right and verification passes with the same certificate.

Hints

Hint 1

Read the scanner log's timestamps. What date does the depot think it is?

Hint 2

`openssl s_client -connect labels.northstar.example:8443 </dev/null 2>/dev/null | openssl x509 -noout -dates` shows the certificate's validity.

Hint 3

`depot-timesync` shows which server the depot asks, whether it answers, and whether the daemon refused to correct the clock.

Hint 4

Compare /etc/depot/timesync.conf with /etc/depot/README: the right time source, and any setting that keeps the clock from being corrected.

Show the solution

The certificate is valid; the depot's clock is 75 days fast, so to depot programs it looks expired. `depot-timesync` says why it stays wrong. Either the only configured server, ntp1.depot7, gets no response because it was retired: point /etc/depot/timesync.conf at time.northstar.example. Or the server is right and answers, but `maxstep 1000` refuses a 75-day correction: remove that line (or raise it past the offset). Within seconds the depot clock synchronises and the scanner's requests succeed. Renewing the certificate would not have helped.

Networking for Operators

TLS and HTTP

What a TLS handshake verifies, why a correct certificate can fail on one client, and what HTTP status codes say about where a failure is.

What TLS proves

TLS gives a client three guarantees: the bytes are encrypted, they are not modified in transit, and the server holds the private key for a certificate that a trusted authority issued for the name the client asked for. To establish the last one, the client checks:

Chain: the certificate links through intermediates to a root in the client's trust store.
Name: the requested hostname matches one of the certificate's subject alternative names.
Validity: the current time is between notBefore and notAfter, according to the client's clock.
Optionally, revocation, through OCSP or CRLs.

Each check fails with its own message, and the message names the part of the system to look at. "Unable to get local issuer certificate" means a missing intermediate or a trust store problem. "Hostname mismatch" means you connected by a different name than the certificate covers. "Expired" or "not yet valid" mean the dates disagree with someone's clock.

The clock is part of the trust decision

Because validity is judged by the client's clock, a client whose clock is wrong by days sees a perfectly good certificate as expired or not yet valid. The giveaway is scope. One site or one class of device fails, everyone else succeeds, and the certificate's dates are fine when checked from a correct machine. Look at timedatectl or the device's NTP configuration, not at the certificate. The same skew breaks other time-based security too: Kerberos tickets, signed URLs, and one-time codes.

Looking at a handshake

openssl s_client -connect 10.0.4.12:443 -servername tracking.northstar.example -showcerts </dev/null
openssl s_client -connect ... </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer -dates -ext subjectAltName
curl -v --resolve tracking.northstar.example:443:10.0.4.12 https://tracking.northstar.example/health

-servername sends SNI, without which a server hosting several names may return a default certificate and mislead you. curl --resolve pins the address while keeping the right name, which is the clean way to test one backend.

Latency is a protocol property

HPBN's central lesson for operators is that round trips are expensive. TCP's handshake costs one round trip, and a full TLS 1.2 handshake adds two more (TLS 1.3 adds one, or none on resumption). A proxy that does not reuse upstream connections, or clients that never resume sessions, pay this on every request. When latency is high but servers are idle, count handshakes before tuning code.

Reading HTTP status codes as locations

Status codes are produced by whichever component answered:

2xx/3xx: the request reached something that handled it.
4xx: the server judged the request wrong, for example 404, 401/403, or 429 (rate limiting).
500: the application itself failed.
502 Bad Gateway: a proxy got no valid response from its upstream (connection refused, reset, or garbage).
503 Service Unavailable: overload, maintenance, or a proxy with no healthy backends.
504 Gateway Timeout: a proxy waited for its upstream and gave up.

A 502 or 504 tells you the request crossed the proxy successfully. The fault is behind it. That one fact halves the search, and the notes on proxies build on it.

Key terms

Certificate chain: The server certificate plus intermediates, leading to a root the client already trusts.
SNI: Server Name Indication. The client sends the hostname in the handshake so one IP address can serve several certificates.
Validity period: The notBefore and notAfter dates. Clients compare them with their own clock.
502 / 504: Status codes generated by a proxy or gateway: 502 means the upstream gave an invalid response or none, and 504 means the upstream did not answer in time.

Read further

High Performance Browser Networking, Ch. 4, "Transport Layer Security (TLS)" (Free to read online)
The handshake and its round trips, the chain of trust and certificate authorities, SNI, session resumption, and OCSP and revocation. Note exactly what the client checks.
High Performance Browser Networking, Ch. 9, "Brief History of HTTP", and Ch. 11, "HTTP/1.X" (Free to read online)
The request/response model, keep-alive, and why one connection per request is costly. Skim the browser optimisation advice.
TCP/IP Illustrated, Volume 1, 2nd edition, Ch. 18, "Security — EAP, IPsec, TLS, DNSSEC, and DNS Security" (Purchase)
The TLS section, for certificate structure and the handshake at the protocol level.

The certificate expired 75 days early OPS-833