Pipelines, redirection, and text
Turn logs into answers with streams, filters, and quoting that survives real file names.
Every process gets three streams
A process starts with three open file descriptors: 0 for input, 1 for normal output, and 2 for errors. By default all three are attached to your terminal, which is why errors and results appear interleaved. Redirection reattaches them before the program starts. The program does not know or care.
> filepoints stdout at a file and truncates it first.>> filepoints stdout at a file and appends.2> filedoes the same for stderr.2>&1makes fd 2 a copy of whatever fd 1 points at right now.
The last rule explains the classic ordering bug. cmd > out 2>&1 first points stdout at the file, then copies that to stderr, so both go to the file. Reversed, stderr copies the terminal and keeps it. When you keep evidence, think about the overwrite. A > meant for a new report that lands on an existing one destroys exactly what someone needed in the morning.
Pipelines are small tools in a row
a | b | c starts all three programs at once and connects each one's stdout to the next one's stdin. Most text tools read lines, change them, and write lines, so they combine freely:
| Question | Tool |
|---|---|
| Which lines mention it? | grep |
| Which field do I care about? | cut -d' ' -f2, awk '{print $2}' |
| In what order? | sort, sort -n, sort -k2 |
| How many distinct? | sort -u, or sort then uniq -c |
| How many in total? | wc -l |
Build a pipeline one stage at a time and look at the output after each stage. The question "how many distinct jobs failed?" is answered by grep FAIL log | cut -d' ' -f2 | sort -u | wc -l. Each stage removes one kind of noise, and you can verify each step. Remember that uniq collapses only adjacent duplicates, so it almost always follows sort.
tee file copies a stream to a file while passing it on. It is useful when you want to see and keep output at the same time.
The shell rewrites your words
Before running anything, the shell performs expansions in a fixed order: braces, tilde, parameters and variables, command substitution, arithmetic, word splitting, and then pathname expansion (wildcards). Two of these cause most script bugs:
- Word splitting. An unquoted
$fholdingWest Dock 2.csvbecomes three arguments. - Pathname expansion. An unquoted
*in a variable's value can turn into a list of files.
Double quotes stop both but still allow $var and $(cmd). Single quotes stop everything. The rule that prevents almost every problem is to quote every expansion, as in "$file" and "$(date +%F)", unless you can explain why you want splitting.
Regular expressions match lines, not words
grep searches with regular expressions. The essentials are ^ (line start), $ (line end), . (any character), [abc] (one of), * (zero or more of the previous item), and with -E, +, ?, |, and groups. Most mistakes come from patterns that match too much. grep waybill also matches waybill-archive, and grep ACTIVE matches INACTIVE. Anchor patterns and add context, as in grep -E '^status: active$'. Use -i for case, -v to invert, -c to count, -l to list files, and -r to search a tree.
Confirm a pattern against the file before you build anything on it. A pipeline whose first stage selects the wrong lines can still look entirely plausible.
Key terms
- Standard streams
- File descriptors 0 (stdin), 1 (stdout), and 2 (stderr), which every process inherits.
- Pipeline
- Commands joined with
|. Each one's stdout becomes the next one's stdin, and they run concurrently. - Word splitting
- The shell's step that breaks unquoted expansion results on whitespace into separate arguments.
- Anchor
- A regular expression position marker.
^is line start and$is line end.
Read further
- The Linux Command Line, Chapter "Redirection" (Free PDF (CC BY-NC-ND))
The three standard streams,>versus>>,2>and2>&1, and why order matters in> file 2>&1. Also note whatteeadds to a pipeline. - The Linux Command Line, Chapter "Seeing the World as the Shell Sees It" (Free PDF (CC BY-NC-ND))
The order of expansions, and the difference between double quotes, single quotes, and escaping. This chapter explains why a file namedWest Dock 2.csvbreaks a script. - The Linux Command Line, Chapters "Regular Expressions" and "Text Processing" (Free PDF (CC BY-NC-ND))
Anchors, character classes, and the difference between basic and extended syntax. In text processing, focus onsort,uniq,cut, andcomm.