The timeout change never took effect

Lab machine

A private Linux machine with the problem already set up. Sessions last up to 60 minutes.

Teo Marin opened OPS-2320 at 16:00SEV-4

CHG-5520 raised the request timeout from 5 to 30 seconds. Every waybill.conf says 30s. The running services still time out after 5.

A service reads its config when it starts. The playbook has a handler that restarts Waybill when the config changes. Somewhere between the change and the restart, the link is cut.

"We quietened that playbook during the 2.3 migration because it was too noisy. I did not realise it would stop the restarts too." (Teo)

Fixing the playbook is half of it. The files already say 30s, so the next run will see nothing to change.

Your task

Make the template task report changes honestly so the handler restarts Waybill when, and only when, the config changes. Then get every running Waybill onto the current config once.

On the machine

site.yml (tasks and handlers)
bin/svc status depot-N shows what each running process loaded
group_vars/depots.yml

Timeline

AugCHG-5402 adds changed_when: false to the template task "to quiet the nightly report".

MonCHG-5520: request_timeout 30s. The playbook runs; files change; nothing restarts.

16:00Carriers still time out after 5 s. OPS-2320.

Done when

Every running Waybill uses the request_timeout from group_vars.
When the config changes, one playbook run restarts Waybill on every host.
A run without config changes restarts nothing, and the template task reports changes honestly.

How it is checked:The checker reads each running service's loaded timeout, changes the timeout to a new value and runs your playbook (every host must restart and load it), runs again (nothing may restart), then restores your settings.

Passed. Here is how others fixed it

This lab has 2 versions. Reset for a fresh machine, and you may get a different one.

Break it for a friend. Pick the version they get, and the link is copied for you to send: version 1 · version 2

Other fixes that also pass:

Run ansible depots -m command -a '{{ playbook_dir }}/bin/svc restart {{ inventory_hostname }}' once (or a restart play with a tag) instead of restarting each host by hand.
Add a task that compares the running config with the file and notifies the handler when they differ; more work, but it catches drift automatically.

Fixes that look right but are not:

Restarting the services by hand and leaving the playbook as it is, so the next change is lost the same way.
Making the handler run on every play (for example with a command task that always restarts), which restarts the fleet every night.
Setting the flag with --extra-vars on one run, which leaves the nightly run without it.

Why it broke. Handlers are triggered by change notifications, and they obey their own conditions. Suppressing changed on the template task, or guarding the handler with a flag that stayed off, cuts the link between a config change and the restart. The config moves and the process never notices. After fixing the playbook, the drift that already happened still needs a one-off restart, because the playbook now sees nothing to change.

How Mara worked it

bin/svc status depot-1
The file says 30s, the running process loaded 5s. Waybill was never restarted.
ansible-playbook site.yml and read the handler lines
Was the handler notified? Was it skipped?
Read the template task and the handler in site.yml
changed_when: false means no notification; a when: on the handler means it is skipped.
Remove what cuts the link
Real changes notify the restart, and the restart runs.
bin/svc restart depot-1 (and 2, 3) once, then ansible-playbook site.yml twice
Last week's change needs one restart now; after that, runs restart only when something changes.

Hints

Hint 1

bin/svc status depot-1 shows what the running process loaded.

Hint 2

A handler runs only when a task that notifies it reports changed, and only if its own conditions allow. Run the playbook and read what it says about handlers.

Hint 3

Look at both ends of the link: the template task (does it report changes?) and the handler (does it have a `when:`?).

Hint 4

The files already say 30s, so the next run will not notify anything. Restart each host once with bin/svc restart, or run the playbook with a one-off restart.

Show the solution

`bin/svc status` shows each process still running with 5s. Read both ends of the notify link in `site.yml`: either the template task has `changed_when: false`, so it never notifies, or the handler has a `when:` on a flag nobody turned back on, so it is skipped. Remove the one that cuts the link. The files already changed last week, so restart each host once (`bin/svc restart depot-1`, `depot-2`, `depot-3`) and confirm with `bin/svc status`.

Infrastructure as Code

Configuration management with Ansible

Converge existing hosts to a declared configuration, idempotently, and use check mode to see drift before you fix it.

Configuring what already exists

Provisioning tools create infrastructure. Configuration management keeps the software and settings on existing hosts in a declared state: packages, files, users, and services. Ansible's model, which Ansible for DevOps builds up chapter by chapter:

An inventory lists hosts and groups ([renderers], [depot_proxies]) with variables.
A playbook maps groups to tasks.
Each task calls a module with arguments describing the desired state: package: name=label-renderer state=present.
Handlers react to changes, for example "restart renderer if its config changed".
Roles package related tasks, templates, handlers, and defaults for reuse.

Ansible connects over SSH and needs no agent. That makes it easy to start with and easy to run from a pipeline.

Idempotence is a property of tasks

Ansible is only as idempotent as its tasks. State-describing modules check before they act. package installs only if absent, template writes only if content differs, and service: state=started starts only if stopped. Each reports ok or changed. Raw command and shell tasks run every time and report changed every time, unless you add creates:, removes:, or changed_when: to describe their effect.

A healthy playbook run on a converged host reports zero changed. As with Terraform's empty second plan, that is a test. A playbook that always reports changes hides the one change that matters among the noise.

- name: Renderer configuration
  template:
    src: renderer.conf.j2
    dest: /etc/renderer/renderer.conf
    mode: "0640"
  notify: restart renderer

handlers:
  - name: restart renderer
    service:
      name: renderer
      state: restarted

The service restarts only when the file actually changed, and only once per run.

Seeing drift before fixing it

ansible-playbook site.yml --check --diff --limit renderers reports, per host, what would change, without changing it. Against a fleet that should be identical, check mode works as a drift report. The odd node shows a pending package change or config diff. This matches the Linux Administration track's notes on drift. The difference is that the remedy is now a reviewed line of code applied to every host, not a hand fix on one.

Pin what you depend on

state: latest means "whatever the repository has right now". Hosts configured on different days end up with different versions, which is drift created by the tool itself. Pin versions in variables, change them through pull requests, and let the playbook converge everyone to the same release. Unpinned latest is reasonable only for packages where any version is acceptable.

Roll out in batches

A configuration change applied to every host at once has the same blast radius as a big-bang deploy. serial applies a play to hosts in batches (one host, then 25%, and so on), and max_fail_percentage stops the run when a batch fails. Add a verification task at the end of each batch, such as an HTTP check of the service, and you have a canary for configuration. It is the same idea as the “Releasing safely” notes, applied to hosts.

Where Ansible fits

The administrator's handbook and Brikman agree that tools overlap and teams combine them. A common split: Terraform or OpenTofu provisions networks and machines, images are built immutably where possible, and Ansible handles what still has to be configured in place. Whatever the split, the principles stay the same: declare the state, review the change, preview it, apply it gradually, and verify it.

Key terms

Inventory: The list of managed hosts and groups, static or generated, with per-host and per-group variables.
Module: A unit of work Ansible runs on a host. Good modules check current state and change only what differs.
Handler: A task run only when notified by a changed task, for example restarting a service after its config template changes.
Check mode: --check runs a playbook without changing anything and reports what would change. With --diff it also shows the differences.

Read further

Ansible for DevOps, Chapters "Ansible Playbooks" and "Inventories" (Purchase)
Playbook structure, handlers, variables, and how inventories group hosts. Note the difference between modules that declare state (package, template, service) and raw command or shell tasks.
Ansible for DevOps, Chapter "Playbook Organization — Roles, Includes, and Imports" (Purchase)
How roles package tasks, handlers, templates, and defaults for reuse, and when to use static imports versus dynamic includes.
UNIX and Linux System Administration Handbook, 5th edition, Ch. 23, "Configuration Management" (Purchase)
The Ansible section and the comparison with Salt and other tools. Note the handbook's points on idempotence and on choosing push versus pull.

The timeout change never took effect OPS-2320