The job that only works by hand OPS-2246

Tell processes apart, read what they inherited, and use exit status as evidence.

A process is a running program plus its context

When the shell starts a program, the kernel creates a process. It has a numeric PID, a parent PID, a user and group identity, a working directory, a set of open files (including the three standard streams), and an environment. Two copies of the same program are two processes with different PIDs and possibly very different contexts.

That context is what you investigate. ps -o pid,ppid,user,etime,args -p <pid> shows identity and age. ls -l /proc/<pid>/cwd shows the working directory, and /proc/<pid>/environ shows the environment it started with. ps -ef --forest draws the parent-child tree, which shows whether a process was started by a service manager, by cron, or by someone's shell.

Never act on a name fragment

pgrep waybill or ps aux | grep waybill is a search, not an identification. It can match a deploy helper, an editor with the file open, or the grep itself. Before signalling or reporting a process, confirm it by at least two independent facts. Useful ones are the full command line, the owner, the parent, and the listening socket. ss -ltnp maps listening ports to PIDs, which is usually the decisive fact for a service. The Linux Administration track builds signals on this. Sending a signal to the wrong PID is the classic way to turn a slow incident into an outage.

Inheritance flows one way

A child process receives a copy of its parent's environment at start. Afterwards they are independent. Three operational consequences follow:

1. Your shell is not the service's shell. Variables you exported, a PATH your profile set, or a directory you cd'd into do not exist for a cron job or a service. A command that "works by hand" but fails when scheduled is almost always missing context its author's shell provided.
2. Children cannot change parents. A script that cds or exports affects only itself. source script.sh runs it in the current shell when that is what you want.
3. Startup files matter. Login shells, interactive shells, and non-interactive shells read different files. Something set in .bashrc may be absent in a script.

env prints the environment, and env -i PATH=/usr/bin:/bin ./job runs a command with an almost empty one. That is a quick way to reproduce how a scheduled job sees the world.

PATH chooses the program

When you type a bare name, the shell checks aliases, functions, and built-ins, then searches each directory in PATH from left to right and runs the first match. type -a name lists every candidate. A stale copy earlier in PATH, or a missing directory in a service's minimal PATH, explains many "wrong version" and "command not found" reports. Scripts that run unattended should call programs by absolute path or set PATH explicitly.

Exit status is the first piece of evidence

Every process ends with a status. Zero means success, and the meaning of anything else is defined by the program. $? holds the last status. && runs the next command only on success and || only on failure. In scripts, set -e stops at the first failure and set -o pipefail makes a pipeline fail if any stage fails, not only the last. Read the EXIT STATUS section of a tool's manual before relying on its codes. grep's "1 means no match" has fooled many alerting scripts.

Key terms

PID

Process ID, a number the kernel assigns. It is the only unambiguous handle on a running process.

Parent process

The process that started this one. Children inherit its environment, working directory, and open files.

Environment variable

A NAME=value pair copied into each child process at start. Changes in a child never flow back to the parent.

Exit status

A number from 0 to 255 a process returns. 0 means success. Anything else is a program-defined failure.

Read further

• The Linux Command Line, Chapter "Processes" (Free PDF (CC BY-NC-ND))
  How ps, top, and job control show processes, and what the process states mean. Read the section on signals but leave the details for the Linux Administration track.
• The Linux Command Line, Chapter "The Environment" (Free PDF (CC BY-NC-ND))
  The difference between shell variables and environment variables, what export does, and which startup files a login shell and a non-login shell read.
• How Linux Works, 3rd edition, Ch. 8, "A Closer Look at Processes and Resource Utilization" (Purchase)
  The first sections on tracking processes with ps and lsof. Note how open files and the working directory belong to a process.

Reconstruct events across restarts and rotations, and make scheduled work run the way it runs by hand.

Two log systems, one question

Most distributions run the systemd journal, often alongside a traditional syslog daemon that writes text files under /var/log. The journal stores structured entries. Every line carries fields such as _SYSTEMD_UNIT, _PID, PRIORITY, and _BOOT_ID. That makes the questions you ask during an incident direct to express:

journalctl -u docklight -b -1 --since "02:00" -p warning
journalctl -u docklight -o short-precise --no-pager | tail -50
journalctl --list-boots

One setting surprises people. If /var/log/journal does not exist (or Storage= is volatile), the journal lives in memory and previous boots are lost. On any host you may need to investigate after a crash, make storage persistent.

Time is evidence

Every log line is only as good as its timestamp. Check three things before building a timeline. Is the clock synchronised (timedatectl)? Which timezone do the logs use? Do different hosts agree? A five-minute skew between a proxy and an application turns "the proxy erred first" into a false story. Prefer UTC for servers and in incident notes.

Rotation is a cooperation

Log files are rotated so they do not grow forever: renamed, compressed, and eventually deleted. A process writes to an open file descriptor, not to a name. After app.log is renamed to app.log.1, the process keeps writing to app.log.1 until it reopens. logrotate offers two answers:

• create + postrotate: rename, create a fresh file, then signal the program (often SIGHUP) to reopen. This is clean and needs the program's cooperation.
• copytruncate: copy, then empty the original in place. It works with programs that never reopen, and it can lose the lines written between copy and truncate.

When a log "looks clean", ask whether you are reading the file the process is actually writing. ls -l /proc/<pid>/fd | grep log tells you. Rotation without a reopen is also the usual source of the “Storage, filesystems, and inodes” notes’ deleted-but-open disk usage.

Scheduled jobs run in a different world

A cron job is started by the cron daemon, not by you. Its environment is minimal: a short PATH, often /usr/bin:/bin, few variables, the job owner's home as the working directory, and no terminal. Output is mailed if a mailer exists and otherwise lost. Most "works by hand, fails at 06:00" incidents reduce to one of these:

1. A command not on cron's PATH.
2. A variable your profile set, absent in cron.
3. A relative path.
4. A % in the crontab line, which cron turns into a newline.
5. Output discarded, so nobody saw the error.

The fix pattern: put the logic in a script with absolute paths and explicit variables, call it from the crontab, and append its output and errors to a log you can read.

Missed runs and doubled runs

Classic cron runs a job when the clock matches while the daemon is running. A host that is down at 06:00 skips the 06:00 run, and nothing catches up. systemd timers with Persistent=true record the last run and fire once at boot if one was missed. anacron serves the same role on some systems. The SRE book's cron chapter frames the trade-off well. Some jobs are worse if skipped, such as a daily invoice run, and some are worse if doubled, such as a charge. Decide which kind you have, and design the job to be idempotent so that a late or repeated run is harmless.

Key terms

Journal

systemd's structured log store. Entries carry fields such as unit, PID, priority, and boot ID.

Log rotation

Periodically renaming, compressing, and deleting log files so they do not grow without bound.

copytruncate

A logrotate mode that copies the file and then empties it in place, for programs that cannot reopen logs. It can lose lines written between the copy and the truncate.

Persistent timer

A systemd timer with Persistent=true. It runs a missed activation at the next boot.

Read further

• UNIX and Linux System Administration Handbook, 5th edition, Ch. 10, "Logging" (Purchase)
  The systemd journal and journalctl, how syslog and the journal coexist, and the section on log rotation with logrotate. Note the options that control what happens to the file a process has open.
• How Linux Works, 3rd edition, Ch. 7, "System Configuration — Logging, System Time, Batch Jobs, and Users" (Purchase)
  The sections on system logging, time and clock synchronisation, and scheduling recurring tasks with cron and systemd timer units.
• Site Reliability Engineering, Ch. 24, "Distributed Periodic Scheduling with Cron" (Free to read online (CC BY-NC-ND 4.0))
  Read the opening sections on cron's reliability model, especially skipped versus doubled launches and idempotency. Skim the distributed design that follows.