Same job, two render nodes, one fails OPS-814

Open2 versionsLinux · Medium · Investigate · about 25 min ·Linux, root

Lab machine

A private Linux machine with the problem already set up. Sessions last up to 60 minutes.
Dmitri Vos opened OPS-814 at 07:50SEV-3

render-a printed last night's labels. render-b failed on the same job with an option it no longer understands.

The two render nodes are supposed to be identical. They are configured by the same automation, which says to install the latest label-renderer. render-b upgraded on its own; render-a has not yet.

"2.5.0 dropped --legacy-dpi and we still need it for the old Zebra printers. Approved is 2.4.x until the printers are replaced in Q1." (Dmitri)

Fixing render-b by hand is half the job. The same automation will upgrade render-a tonight.

Your task

Put render-b back on an approved version so render-nightly passes, stop unattended upgrades from moving it again, and change the automation so every render node is pinned to the approved version.

On the machine

  • dpkg -l label-renderer, apt-cache policy label-renderer, apt-mark showhold
  • fleet/inventory.txt, fleet/approved-versions.txt
  • automation/render-node.yml
  • /var/log/dpkg.log and /var/log/depot/

Timeline

TueUnattended upgrades install label-renderer 2.5.0 on render-b.
02:00Nightly label run fails on render-b: "unrecognized option --legacy-dpi".
07:50OPS-814 opened. render-a carried the whole load overnight.

Done when

  1. render-b runs an approved 2.4.x version, held or pinned, and the automation declares that version.
  2. render-nightly passes.

Hints

Hint 1

`apt-cache policy label-renderer` lists every version the repository offers, and marks a held package.

Hint 2

Installing an older version needs `--allow-downgrades`. If apt refuses because the package is held, find out who held it and why before you unhold it.

Hint 3

`sudo apt-mark hold label-renderer` keeps the node on the version you install.

Hint 4

`state: latest` in automation is what reintroduces drift. Declare `label-renderer=2.4.3`.

Show the solution

Install the approved release with `sudo apt-get install --allow-downgrades label-renderer=2.4.3`. If apt refuses because 2.5.0 is held (`apt-mark showhold`; someone held it to stop it flapping), `sudo apt-mark unhold label-renderer` first. Then hold the approved version with `sudo apt-mark hold label-renderer`, and rerun `render-nightly`. In `automation/render-node.yml`, replace `state: latest` with `name: label-renderer=2.4.3`, `state: present`, so every render node converges on the approved version.