Redirecting Input & Output Streams

By default, all the commands executed in a shell read their input from and write their output to the terminal.

However, it is sometimes useful to be able to change this behavior so that commands can read from and write to files instead of the terminal.

This can be achieved using a feature called input/output redirections.

A redirection is the action of capturing the output from a file, a program, or a script and sending it as input to another file, program, or script.

These redirections are mostly used for reading the content of a file, saving the output of a command into a file, or even chaining multiple commands together to create a data processing pipeline.

Standard streams

A stream is a communication channel that transfers data between a program and its environment, like for example the shell and its terminal.

In Unix-like operating systems, there are three types of streams called standard streams:

• The standard input, often abbreviated stdin, that programs can read data from.
• The standard output, often abbreviated stdout, that programs can write data to when a successful operation was performed.
• The standard error, often abbreviated stderr, that programs can write data to when something goes wrong during their execution.

These streams are by default attached to the terminal the shell runs in so that programs can use them, like for example the echo command that writes back its supplied arguments on the standard output.

File descriptors

A file descriptor is a unique resource identifier (i.e., an integer) assigned to a data stream by the operating system.

Whenever launching a new terminal window, the operating system automatically assigns the same file descriptor to each standard stream, which are:

• 0 for stdin.
• 1 for stdout.
• 2 for stderr.

Displaying the content of files

To display the content of one or more files to the standard output, you can use the cat command — which stands for concatenate and print — as follows:

$ cat file ...

Where file ... is a list of paths to the files you want to display.

Example

This command will write the content of the hello.txt file to the standard output:

$ cat hello.txt
Hello world
Hallo Welt
Bonjour monde
Hola mundo

Redirecting output streams

Redirecting an output stream essentially consists in redirecting the standard output and/or the standard error of a command, so that it writes into a file instead of the terminal window, using the following syntax:

$ command fd> file

Where:

• fd is a file descriptor.
• > is the output redirection operator.
• file is the path to the file the executed command will write to.

Note: If the destination file doesn't exist, the redirection operator will automatically create it for you.

Redirecting stdout into a file

To redirect the standard output into a file, you can use the following syntax:

$ command 1> file

Note: In case of the standard output, 1> is equivalent to >.

Example

This command will write the string "Hello World" to the standard output:

$ echo "Hello World"
Hello World

This command will write the same string into a file named hello.txt by redirecting the file descriptor 1 (i.e. stdout) into that file instead of the terminal window:

$ echo "Hello World" > hello.txt

This command will display the content of the newly created hello.txt file:

$ cat hello.txt
Hello World

Redirecting stderr into a file

To redirect the standard error into a file, you can use the following syntax:

$ command 2> file

Note that you can combine multiple redirections by specifying them one after the other.

$ command > file 2> file

Example

This command will write an error to the standard error (i.e. stderr):

$ ls xxx
ls: xxx: No such file or directory

This command will write the same error into a file named error.txt by redirecting the file descriptor 2 (i.e., stderr) into that file instead of the terminal window:

$ ls xxx 2> error.txt

This command will display the content of the newly created error.txt file:

$ cat error.txt
ls: xxx: No such file or directory

Overwriting and appending to a file

By default, the output redirection operator > will completely overwrite the content of the specified destination file.

To prevent this from happening, you can use a double right chevron >> that will tell the shell to append the output of the executed command to the file's content instead of replacing it:

$ command fd>> file

Example

This command will write the string "Good morning" to the hello.txt file:

$ echo "Good morning" > hello.txt
$ cat hello.txt
Good morning

This command will completely overwrite the content of the hello.txt file with the string "Guten Tag":

$ echo "Guten Tag" > hello.txt
$ cat hello.txt
Guten Tag

This command will append the string "Buenos dias" at the end of the hello.txt file:

$ echo "Buenos dias" >> hello.txt
$ cat hello.txt
Guten Tag
Buenos dias

Silencing a file descriptor

It will sometimes happen that a stream creates too much noise in the terminal or in a file by generating an excessive amount of data.

To silence this stream, you can redirect its output in a special file called /dev/null, that acts as a kind of black hole that simply discards all the data it receives:

$ command fd> /dev/null

Example

This command will silence the standard error, thus preventing it from outputting any errors the ls command might encounter:

$ ls xxx 2> /dev/null

Redirecting streams into streams

When redirecting an output stream into a file, the shell will first try to create the file if it doesn't exist, and then open this file for writing.

When doing so, the operating system will attribute this file to a file descriptor and send it back to the shell, so it knows where to redirect the stream of data sent through the standard output or the standard error.

Consequently, redirecting the output of a command into a file essentially consists in redirecting a stream into a stream.

That being said, it is possible to manually redirect a stream into a stream using the following syntax:

$ command M>&N

Where:

• M is a file descriptor.
• >& is the output stream redirection operator.
• N is a file descriptor.

Note: You can use the following syntax to redirect both stdout and stderr at once into the same file:

$ command &> file

Example

This command will redirect the standard output into a file named logs.txt and redirect the standard error into the standard output:

$ ls . xxx >> logs.txt 2>&1

Which will result in both streams being redirected to the same file:

$ cat logs.txt
ls: xxx: No such file or directory
/home/razvan:
Documents
Downloads
Library
Pictures
Projects
Public

Feeding data to a program through stdin

By default, many commands like cat will read data from the standard input when not supplied with a filepath argument.

These commands will read data until they encounter an end-of-file (EOF), which is a special sequence that indicates the end of a file or a data set.

This behavior can be illustrated by executing the cat command without arguments, which will result in the program awaiting for the user to type and send data through the standard input:

$ cat
▮

When typing a string and pressing ENTER:

$ cat
Hello World▮

The cat command will read this data from the standard input and write it back to the standard output, which is normal since all three standard streams are, by default, attached to the terminal:

$ cat
Hello World
Hello World
▮

Finally, to send an end-of-file sequence indicating that the data set is finished, you can use the CTRL + D key combination, which will cause the process to terminate and the command prompt to reappear, allowing you to type other commands:

$ cat
Hello World
Hello World
$

Redirecting input streams

Redirecting an input stream consists in redirecting the standard input of a command, so that it reads from a file instead of the terminal, using the following syntax:

$ command < file

Where:

• < is the input redirection operator.
• file is the path to the file the command should read from.

Example

Let's consider this file named lorem.txt:

$ cat lorem.txt
Lorem ipsum dolor sit amet.
Etiam eu pulvinar magna.
Phasellus ultricies neque libero.

This command will read its input from the lorem.txt file instead of the standard input and write it back to the terminal on the standard output:

$ cat < lorem.txt
Lorem ipsum dolor sit amet.
Etiam eu pulvinar magna.
Phasellus ultricies neque libero.

Input redirection vs filepath argument

Although the produced output is identical using either of these methods, there is a slight difference between passing a file as an argument and through an input redirection.

When passing a file as an argument, you are putting the specified command in charge of opening the file, reading its content, and writing it to the terminal.

On the other hand, when passing the content of the file through an input redirection, you are putting the shell in charge of opening the file and sending its content to the specified command to be then written back to the terminal.

Chaining commands with pipes

A pipe | is a special redirection operator that enables to feed the standard output of a command into the standard input of another command to form a processing pipeline using the following syntax:

$ command | command [| command]

Note: Pipes are mostly used in combination with filter commands, such as sort, uniq, head, or tail to process data in various ways.

Sorting the output of a command

To sort the output of a command in alphabetical order, you can use the sort command as follows:

$ command | sort

Example

Let's consider this file named cities.txt:

$ cat cities.txt
Athens
Bucharest
Kiev
Madrid
Paris

This cat command will write the content of the cities.txt file to the standard output, which will then be piped into (or forwarded to) the standard input of the sort command, and finally printed in reverse alphabetical order using the -r flag (short for reverse):

$ cat cities.txt | sort -r
Paris
Madrid
Kiev
Bucharest
Athens

Deduplicating the output of a command

To filter out repeated adjacent lines, you can use the uniq command as follows:

$ command | uniq

Example

Let's consider this file named students.txt:

$ cat students.txt
Rose
Anna
Jack
Anna
Rose

This cat command will write the content of the students.txt file to the standard output, which will then be piped into the standard input of the sort command, whose output will be piped into the standard input of the uniq command, and finally printed without duplicated lines:

$ cat students.txt | sort | uniq
Anna
Jack
Rose

Displaying the first and last lines of a file

To respectively display the first and last lines of a file, you can use the head and tail command as follows:

$ command | head
$ command | tail

Example

Let's consider this file named lorem.txt:

$ cat lorem.txt
Lorem ipsum dolor sit amet
Consectetur adipiscing elit
Praesent molestie eros quis diam hendrerit
Id commodo augue porttitor
Donec quis nunc non ante gravida imperdiet ut in mi

This cat command will write the content of the lorem.txt file to the standard output, which will then be piped into the standard input of the head command, which will finally only output the first 2 lines of that file:

$ cat lorem.txt | head -n 2
Lorem ipsum dolor sit amet
Consectetur adipiscing elit

This cat command will write the content of the lorem.txt file to the standard output, which will then be piped into the standard input of the tail command, which will finally only output the last 2 lines of that file:

$ cat lorem.txt | tail -n 2
Id commodo augue porttitor
Donec quis nunc non ante gravida imperdiet ut in mi

Summary

Here's a summary of what you've learned in this lesson:

• There are three standard streams: stdin (0), stdout (1), and stderr (2).
• The 1> (or >) operator redirects the standard output into a file.
• The 2> operator redirects the standard error into a file.
• The M>> operator redirects an output stream M into a file and appends the data.
• The M>&N operator redirects a stream M into another stream N.
• The < operator redirects the standard input from a file.
• The | operator redirect the standard output of a command into the standard input of another command.
• The cat command reads data from files or the standard input and writes the data to the standard output.
• The sort command sorts or merges lines of text.
• The uniq command reports or filters out repeated adjacent lines of text.
• The head command displays the first N lines of text.
• The tail command displays the last N lines of text.