Fix a script that breaks on spaces OPS-2251

Turn logs into answers with streams, filters, and quoting that survives real file names.

Every process gets three streams

A process starts with three open file descriptors: 0 for input, 1 for normal output, and 2 for errors. By default all three are attached to your terminal, which is why errors and results appear interleaved. Redirection reattaches them before the program starts. The program does not know or care.

• > file points stdout at a file and truncates it first.
• >> file points stdout at a file and appends.
• 2> file does the same for stderr.
• 2>&1 makes fd 2 a copy of whatever fd 1 points at right now.

The last rule explains the classic ordering bug. cmd > out 2>&1 first points stdout at the file, then copies that to stderr, so both go to the file. Reversed, stderr copies the terminal and keeps it. When you keep evidence, think about the overwrite. A > meant for a new report that lands on an existing one destroys exactly what someone needed in the morning.

Pipelines are small tools in a row

a | b | c starts all three programs at once and connects each one's stdout to the next one's stdin. Most text tools read lines, change them, and write lines, so they combine freely:

Build a pipeline one stage at a time and look at the output after each stage. The question "how many distinct jobs failed?" is answered by grep FAIL log | cut -d' ' -f2 | sort -u | wc -l. Each stage removes one kind of noise, and you can verify each step. Remember that uniq collapses only adjacent duplicates, so it almost always follows sort.

tee file copies a stream to a file while passing it on. It is useful when you want to see and keep output at the same time.

The shell rewrites your words

Before running anything, the shell performs expansions in a fixed order: braces, tilde, parameters and variables, command substitution, arithmetic, word splitting, and then pathname expansion (wildcards). Two of these cause most script bugs:

• Word splitting. An unquoted $f holding West Dock 2.csv becomes three arguments.
• Pathname expansion. An unquoted * in a variable's value can turn into a list of files.

Double quotes stop both but still allow $var and $(cmd). Single quotes stop everything. The rule that prevents almost every problem is to quote every expansion, as in "$file" and "$(date +%F)", unless you can explain why you want splitting.

Regular expressions match lines, not words

grep searches with regular expressions. The essentials are ^ (line start), $ (line end), . (any character), [abc] (one of), * (zero or more of the previous item), and with -E, +, ?, |, and groups. Most mistakes come from patterns that match too much. grep waybill also matches waybill-archive, and grep ACTIVE matches INACTIVE. Anchor patterns and add context, as in grep -E '^status: active$'. Use -i for case, -v to invert, -c to count, -l to list files, and -r to search a tree.

Confirm a pattern against the file before you build anything on it. A pipeline whose first stage selects the wrong lines can still look entirely plausible.

Key terms

Standard streams

File descriptors 0 (stdin), 1 (stdout), and 2 (stderr), which every process inherits.

Pipeline

Commands joined with |. Each one's stdout becomes the next one's stdin, and they run concurrently.

Word splitting

The shell's step that breaks unquoted expansion results on whitespace into separate arguments.

Anchor

A regular expression position marker. ^ is line start and $ is line end.

Read further

• The Linux Command Line, Chapter "Redirection" (Free PDF (CC BY-NC-ND))
  The three standard streams, > versus >>, 2> and 2>&1, and why order matters in > file 2>&1. Also note what tee adds to a pipeline.
• The Linux Command Line, Chapter "Seeing the World as the Shell Sees It" (Free PDF (CC BY-NC-ND))
  The order of expansions, and the difference between double quotes, single quotes, and escaping. This chapter explains why a file named West Dock 2.csv breaks a script.
• The Linux Command Line, Chapters "Regular Expressions" and "Text Processing" (Free PDF (CC BY-NC-ND))
  Anchors, character classes, and the difference between basic and extended syntax. In text processing, focus on sort, uniq, cut, and comm.