Formatting the Output of Commands

Software developers are often required to work with large datasets or files, such as CSVs or logs.

Rather than loading the entire data into memory, developers can filter out, narrow down, and transform output to only focus on relevant lines or sections, such as specific logs, function calls, errors, and so on.

Filtering output based on patterns

The grep command — which stands for global regular expression print — is used to filter the lines of any given input based on patterns:

$ grep [-i] [-v] pattern [file ...]

Where:

• -i is an optional flag used to make grep case insensitive.
• -v is an optional flag used to invert the search.
• pattern is a string literal or a basic regular expression to find.
• file ... is a list of file paths to read the content from.

Note: The grep command can also read input from a piped command:

$ command | grep [options] pattern

Example

Let's consider this file named logs.txt, that contains a list of server logs:

$ cat logs.txt
2024-05-01 | ERROR 401 | Login failure
2024-05-06 | INFO 200 | Login success
2024-06-22 | INFO 201 | Account created
2024-06-01 | ERROR 404 | Page not found
2024-06-01 | WARN 429 | Request limit reached
2024-06-03 | ERROR 404 | Page not found

This command will output the lines that contain the string "error", regardless of case:

$ grep -i "error" logs.txt
2024-05-01 | ERROR 401 | Login failure
2024-06-01 | ERROR 404 | Page not found
2024-06-03 | ERROR 404 | Page not found

This command will output the lines that don't contain the string "ERROR" or "WARN":

$ grep -v "ERROR" -v "WARN" logs.txt
2024-05-06 | INFO 200 | Login success
2024-06-22 | INFO 201 | Account created

Filtering output using regular expressions

A regular expression, often abbreviated regexp, is a sequence of characters that defines a search pattern, allowing for flexible and precise matching of text within a given dataset.

Basic regular expressions (BREs) include a minimal set of special characters:

• .: matches any single character except a newline.
• []: matches any one of the characters in brackets.
• ^: matches the start of a line.
• $: matches the end of a line.
• *: matches zero or more occurrences of the preceding character.

Extended regular expressions (EREs) include an additional set of special characters:

• +: matches 1 or more occurrences of the preceding character or group.
• ?: matches 0 or 1 occurrence of the preceding character or group.
• {n}: matches exactly n occurrences of the preceding character or group.
• {n,}: matches n or more occurrences of the preceding character or group.
• {n,m}: matches between n and m occurrences of the preceding character or group.
• |: matches either the pattern on the left or right.
• [^]: matches any character not inside the brackets.
• (): groups expressions.
• \: escapes special characters.

To use EREs with grep, you can use the -E flag:

$ grep -E pattern [file ...]

Example

Let's consider this directory:

$ ls -1p
index.js
node_modules/
package-lock.json
package.json
src/

This command will output the filenames that start with the string 'package':

$ ls | grep '^package'
package-lock.json
package.json

This command will output the filenames that end with the string '.js' or '.json':

$ ls | grep -E '\.js|\.json$'
index.js
package-lock.json
package.json

This command will output the filenames that don't contain an underscore '_' or a hyphen '-':

$ ls | grep -v -E '_|-'
index.js
package.json
src

Translating individual characters

The tr command — which stands for translate — is used to translate, delete, or compress the characters of any given input:

$ command | tr string1 string2
$ command | tr [-d | -s] string

Where:

• The individual characters of string1 are replaced by the characters in string2.
• The -d flag is used to remove characters.
• The -s flag is used to replace consecutive occurrences of a character with a single occurrence.

Note: The tr command provides a list of character classes that allow you to match pre-defined groups of characters, such as [:alnum:] for alphanumeric characters, [:lower:] for lower-case alphabetic characters, [:blank:] for whitespace characters, and so on.

Example

Let's consider this file named data.csv, that contains CSV-formatted data with additional spaces, tabs, and newlines:

$ cat data.csv
first_name,last_name,email,phone,birthdate
John , Doe  , john.doe@example.com, (555) 123-4567    , 1988-02-23

Jane,   Smith   , jane.smith@example.com, (555) 987-6543 , 1991-06-63
Bob , Brown, bob.brown@example.com ,(555) 555-5555 , 1972-09-11


Alice , Johnson , alice.j@example.net,    (555) 321-4321 , 1985-12-12

  Charlie,Williams,   charlie.williams@example.net  , (555) 654-3210, 1995-07-27

This command will delete all the whitespace characters, including spaces and tabs, and squeeze all the extra newline characters:

$ cat data.csv | tr -d '[:blank:]' | tr -s '\n'
first_name,last_name,email,phone,birthdate
John,Doe,john.doe@example.com,(555)123-4567,1988-02-23
Jane,Smith,jane.smith@example.com,(555)987-6543,1991-06-63
Bob,Brown,bob.brown@example.com,(555)555-5555,1972-09-11
Alice,Johnson,alice.j@example.net,(555)321-4321,1985-12-12
Charlie,Williams,charlie.williams@example.net,(555)654-3210,1995-07-27

Splitting lines into columns

To split the lines of an input into columns based on a delimiting character and process them in specific ways, you can use the awk command:

$ awk [-F delimiter] '/pattern/ {action; ...}' file

Where:

• -F is an optional flag used to specify a delimiter used to split each line into columns. Defaults to the space character ' '.
• /pattern/ is an optional pattern used to match lines (similar to grep).
• {action} is a list of actions to perform on the columns, separated by a semicolon character ';'.

Printing columns

To output specific columns with awk, you can use the print action:

'{print $column, ...}'

Where $column, ... is a list of column numbers, where $0 represents the entire line, $1 the first column, $2 the second column, etc.

Example

Let's consider this file named data.csv:

$ cat data.csv
first_name,last_name,email,phone,birthdate
John,Doe,john.doe@example.com,(555)123-4567,1988-02-23
Jane,Smith,jane.smith@example.com,(555)987-6543,1991-06-63
Bob,Brown,bob.brown@example.com,(555)555-5555,1972-09-11
Alice,Johnson,alice.j@example.net,(555)321-4321,1985-12-12
Charlie,Williams,charlie.williams@example.net,(555)654-3210,1995-07-27

This command will split each line into individual columns using the ',' character as delimiter and only output the first two columns of each line that start with the 'J' character:

$ awk -F ',' '/^J/ {print $1, $2}' data.csv
first_name last_name
John Doe
Jane Smith

Formatting columns

To output the columns in a specified format using a template string, you can use the printf action, which is similar to the Unix printf command:

'{printf "template", $column, ...}'

Where "template" is a template string in which specifiers will be replaced by the value of the specified $columns.

Note: You can learn more about template strings by reading the printf manual page.

Example

Let's consider this file named employees.csv:

$ cat employees.csv
ID,Name,Department,Salary
1,John Doe,Engineering,50000
2,Jane Smith,Marketing,55000
3,Bob Brown,Sales,45000
4,Alice Johnson,HR,47000

This command will output each line in the form of table, where columns are separated by a '|' character:

$ awk -F ',' 'NR > 1 {printf "| %-15s | %-15s | $ %d |\n", $2, $3, $4}' employees.csv
| John Doe        | Engineering     | $ 50000 |
| Jane Smith      | Marketing       | $ 55000 |
| Bob Brown       | Sales           | $ 45000 |
| Alice Johnson   | HR              | $ 47000 |

Where:

• NR > 1 is used to skip the first line of the file.
• %-15s is used to output a string with a right whitespace padding up to 15 characters.
• %d is used to output an integer.

Substituting patterns

To substitute a pattern with a string within a column, you can use the sub action and combine it with the print action:

'{sub(/pattern/, "replacement", $column); print $column, ...}'

Where:

• /pattern/ is a regular expression to match.
• "replacement" is a string used as replacement.

Example

Let's consider this file named ssh.logs, that contains a list of connection logs:

$ cat ssh.logs
Dec 18 10:21:34: Accepted password for john from 192.168.0.1 port 22
Dec 18 10:22:15: Accepted password for alice from 126.12.1.10 port 22
Dec 18 10:23:00: Accepted password for bob from 89.3.123.8 port 22

This command will anonymize the IP addresses by replacing them with the '*.*.*.*' string:

$ awk '{sub(/([0-9]{1,3}\.){3}[0-9]{1,3}/, "*.*.*.*", $0); print $0}' ssh.logs
Dec 18 10:21:34: Accepted password for john from *.*.*.* port 22
Dec 18 10:22:15: Accepted password for alice from *.*.*.* port 22
Dec 18 10:23:00: Accepted password for bob from *.*.*.* port 22

Summary

Here's a summary of what you've learned in this lesson:

• The grep command is used to filter lines based on strings or patterns.
• The tr command is used to translate individual characters.
• The awk command is used to process lines as columns.