The port change that did not stick

Lab machine

A private Linux machine with the problem already set up. Sessions last up to 60 minutes.

Tomas Berg opened INC-2512 at 03:58SEV-3

The depot-4 printers moved to port 9105 overnight. Tomas changed the spooler's port in its unit and restarted it twice. It still listens on 9100, and the 06:00 trucks need their labels.

Every parcel leaving depot-4 needs a printed label. The new printer firmware sends jobs to port 9105, so until the spooler moves, nothing prints.

"Changed it in the unit, restarted twice. Status says active. Printers still say connection refused on 9105. I'm off at 04:00, sorry." (Tomas, 03:40)

The host runs a small service manager (bin/hostctl, behaves like systemctl) with a journal (bin/hostlog, like journalctl). Tomas did change the unit file. Find out why the running spooler never saw it.

Your task

Move the label spooler to 127.0.0.4:9105 so that the running process, a restart, and a reboot all agree, and leave the configuration in a state the next person can read.

On the machine

bin/hostctl status label-spooler, bin/hostlog -u label-spooler -b
etc/units/label-spooler.service and etc/default/label-spooler
home/tomas/.bash_history
curl -s http://127.0.0.4:9100/healthz reports the address the spooler listens on

Timeline

02:30New label printer firmware on depot-4 switches print jobs to port 9105.

03:10Tomas edits label-spooler.service and restarts it twice.

03:40"Status says active. Printers still say connection refused on 9105."

05:15INC-2512 lands with you. The first trucks leave at 06:00.

Done when

label-spooler answers on 127.0.0.4:9105 under the service manager, and 9100 is closed.
It comes back on 9105 after a reboot.
The service manager has loaded the unit file that is on disk.
The port is not set to different values in different places.

How it is checked:The checker rejects a unit that still needs a daemon-reload and a port given different values in different places (Environment=, the environment file, or ExecStart). It then requires the managed spooler on 127.0.0.4:9105 with 9100 closed, reboots the simulated host, restarts the service, and checks again after each step.

Passed. Here is how others fixed it

This lab has 2 versions. Reset for a fresh machine, and you may get a different one.

Break it for a friend. Pick the version they get, and the link is copied for you to send: version 1 · version 2

Other fixes that also pass:

Remove the EnvironmentFile= line and keep SPOOL_PORT=9105 in the unit, after moving SPOOL_QUEUE into the unit too. Then daemon-reload and restart.

Fixes that look right but are not:

Restarting again and again without `daemon-reload`.
Starting `spooler.py` by hand with SPOOL_PORT=9105; it disappears at the next reboot.
Setting 9105 in the unit and 9100 in the settings file, which leaves the next person guessing.

Why it broke. Tomas's edit never reached the process for two reasons. The service manager keeps the unit definition it loaded at boot until `daemon-reload`, so his restarts reused the old one. And even after a reload, EnvironmentFile= is applied after Environment=, so the 9100 in the settings file wins. "Active" only says a process is running, not that it runs with the settings you meant.

How Mara worked it

curl -s http://127.0.0.4:9100/healthz and bin/hostctl status label-spooler
Confirm what is actually running and on which port before touching anything. Status warns that the unit file changed on disk.
bin/hostctl daemon-reload, then bin/hostctl restart label-spooler
Tomas skipped the reload, so every restart reused the old definition.
bin/hostlog -u label-spooler -b | tail -3
The spooler logs the address it listens on. Still 9100 means another setting wins.
bin/hostctl cat label-spooler or read the unit and every file it points at
Find each place a port can come from, and which one takes precedence.
Set 9105 in the one place that wins, remove or align the others, reload, and restart
One source of truth, so the next person does not repeat this at 03:00.
bin/hostctl reboot, then curl -s http://127.0.0.4:9105/healthz
Prove it survives what the printers will see next week.

Hints

Hint 1

`bin/hostctl status` warns when the unit file on disk is newer than what is loaded.

Hint 2

After a daemon-reload, check which port the spooler actually got. The journal prints it.

Hint 3

Read the whole unit. One setting can be given in more than one place.

Hint 4

In systemd, a value from EnvironmentFile= overrides the same name in Environment=, and a program may prefer its command-line flags over both.

Show the solution

Set `SPOOL_PORT=9105` in `etc/default/label-spooler` (the file meant for local changes) and remove SPOOL_PORT from the unit's Environment= line or set it to 9105 there too. Run `bin/hostctl daemon-reload`, then `bin/hostctl restart label-spooler`, and confirm with `curl http://127.0.0.4:9105/healthz`.

Linux Administration

Boot and service managers

How a host gets from power-on to running services, and why a service started by hand differs from a managed one.

From power-on to PID 1

A host boots in stages. Firmware finds a boot loader. The boot loader loads a kernel and an initial RAM filesystem. The kernel initialises hardware, mounts the root filesystem, and starts exactly one user-space program, init, as PID 1. Every other process on the system descends from it. On almost every current distribution init is systemd. Its job is to bring the system to a goal state, called a target, by starting units in dependency order, and to supervise them afterwards.

You rarely debug firmware or the kernel in day-to-day operations. You debug the last stage constantly. When a host "comes back from a reboot without the app", the question is almost always about how the service manager was told to run it.

A unit is a contract

A service unit describes how to run a program. The parts that matter most in incidents:

[Unit]
Description=Waybill API
After=network-online.target
Wants=network-online.target

[Service]
User=waybill
WorkingDirectory=/srv/waybill
EnvironmentFile=/etc/waybill/env
ExecStart=/srv/waybill/bin/server --config /etc/waybill/config.yaml
Restart=on-failure

[Install]
WantedBy=multi-user.target

Each line replaces something a human shell would have provided implicitly. User= replaces "whoever is logged in". WorkingDirectory= replaces "wherever I cd'd". EnvironmentFile= replaces "what my profile exported", and ExecStart= needs an absolute path because there is no interactive PATH. [Install] says which target pulls the unit in at boot, but only once you enable it.

Why "works by hand" fails under the manager

Starting a program from your shell gives it your identity, your directory, your environment, and your terminal. Starting it from the manager gives it whatever the unit says, and nothing else. List the differences before you change anything:

Identity. Can user waybill read the config and write the data directory?
Directory. Does the program open relative paths?
Environment. Which variables did your shell have that the unit lacks?
Ordering. Does it need the network or a mount that is not ready yet?
Persistence. Is it enabled, or only started?

systemctl show waybill -p User -p WorkingDirectory -p Environment prints what the manager will actually use. systemctl cat waybill shows the unit together with any drop-ins.

Change units the maintainable way

Vendor units live in /usr/lib/systemd/system and belong to the package. Local units and overrides go in /etc/systemd/system. systemctl edit waybill creates a drop-in containing only your change, which survives package upgrades and documents intent. After any edit, systemctl daemon-reload makes systemd re-read the definitions. Then restart the unit and verify its effect, not just its status. A unit can be "active (running)" while the program inside it is listening on the wrong address.

Restart policy is not a fix

Restart=on-failure keeps a service available through transient crashes. It also hides a crash loop until the start limit is hit and systemd gives up. If a service is restarting, read why in journalctl -u waybill -b. Treat a high restart count as an open incident, not a solved one.

When an edit does not take effect

Two things stand between a changed unit file and a changed process. First, systemd runs the definition it loaded, not the file on disk. Until systemctl daemon-reload, a restart reuses the old definition, and systemctl status warns that the unit file changed on disk. Second, a setting can come from several places. Environment= lines are applied first and EnvironmentFile= files after them, so a variable set in both takes the file's value. Drop-ins in /etc/systemd/system/<unit>.d/ override the main file. systemctl cat <unit> shows every piece in order, and systemctl show -p Environment <unit> shows what was loaded. After the change, check the process itself: the port it listens on, or the address it logs at startup. "active (running)" only means a process exists.

Key terms

init: PID 1, the first user-space process. On most current distributions it is systemd, which starts and supervises everything else.
Unit: A systemd object (service, socket, timer, mount, and others) described by a unit file.
Enable versus start: start runs the unit now. enable links it into a target so it starts at boot. You usually want both.
Restart policy: Whether the manager restarts a service after it exits, for example Restart=on-failure.

Read further

How Linux Works, 3rd edition, Ch. 5, "How the Linux Kernel Boots", and Ch. 6, "How User Space Starts" (Purchase)
In Ch. 5, only the overview of the boot sequence. In Ch. 6, read the systemd sections in full. Note unit types, dependencies, and the difference between a unit file and its runtime state.
UNIX and Linux System Administration Handbook, 5th edition, Ch. 2, "Booting and System Management Daemons" (Purchase)
The systemd sections, especially systemctl subcommands, unit file locations and precedence, and the warning about editing vendor unit files instead of using overrides.

The port change that did not stick INC-2512