Chapter 1: Sed Syntax and Basic Commands

For all sed examples, we'll be using the following employee.txt file. Please create this text file to try out the commands given in this book.

$ vi employee.txt
101,John Doe,CEO
102,Jason Smith,IT Manager
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer
105,Jane Miller,Sales Manager

The above employee database contains the following fields for every record:

  • Employee Id
  • Employee Name
  • Title

Sed stands for Stream Editor. It is very powerful tool to manipulate, filter, and transform text. Sed can take input from a file, or from a pipe. You might even have several sed one line commands in your bash startup file that you use for various scenarios without exactly understanding the sed scripts.

For beginners, sed script might look cryptic. Once you understand the sed commands in detail, you'll be able to solve a lot of complex text manipulation problems by writing a quick sed script.

In this book, I've explained all sed commands and provided easy-to-understand examples.

1. Sed Command Syntax‌

The purpose of this section is to get you familiarized with sed syntax and command structure. This is not meant to explain the individual sed commands, which are covered in detail later.

Basic sed syntax:

sed [options] {sed-commands} {input-file}

Sed reads one line at a time from the {input-file} and executes the {sed-commands} on that particular line.

It reads the 1st line from the {input-file} and executes the {sed- commands} on the 1st line. Then it reads the 2nd line from the
{input-file} and executes the {sed-commands} on the 2nd line. Sed repeats this process until it reaches the end of the {input-file}.

There are also a few optional command line options that can be passed to sed as indicated by [options].

The following example basic sed syntax. This simple sed example prints all the lines from the /etc/passwd file.

sed -n 'p' /etc/passwd

The main focus here is on the {sed-commands}, which can be either a single sed command or multiple sed commands. You can also combine multiple sed-commands in a file and call the sed script file using the -f option as shown below.

Basic sed syntax for use with sed-command file:

sed [options] -f {sed-commands-in-a-file} {input-file}

The following example demonstrates the basic syntax. This example prints lines beginning with root and nobody from the /etc/passwd file.

$ vi test-script.sed
/^root/ p
/^nobody/ p

$ sed -n -f test-script.sed /etc/passwd

While executing multiple sed commands, you can also directly specify them in the command line using -e as shown below.

Basic sed syntax using -e:

sed [options] -e {sed-command-1} -e {sed-command-2}{input-file}

The following example demonstrates the basic syntax. This prints lines beginning with root and nobody from /etc/passwd file:

sed -n -e '/^root/ p' -e '/^nobody/ p' /etc/passwd

If you are executing a lot of commands in a single line using several -e arguments, you can split them into multiple lines using a back slash as shown below.

sed -n \
-e '/^root/ p' \
-e '/^nobody/ p' \
/etc/passwd

You can also execute multiple sed commands in the command line by grouping them together using { }:

Basic sed syntax using { }:

sed [options] '{
sed-command-1
sed-command-2
}' input-file

The following example demonstrates this version of the basic syntax. This also prints lines beginning with root and nobody from
/etc/passwd file.

sed -n '{
/^root/ p
/^nobody/ p
}' /etc/passwd

Note: Sed never modifies the original file. It always prints the output to stdout. If you want to save the changes, you should redirect the output to a file by explicitly specifying > filename.txt.

2. Sed Scripting Flow‌

Sed scripting follows the easily remembered sequence Read, Execute, Print, Repeat. Use the simple REPR acronym to remember sed execution flow.

We look at the steps in this sequence. Sed will:

  • Read a line into the pattern space (an internal temporary sed buffer, where it places the line it reads from the input file).
  • Execute the sed command on the line in the sed pattern space. If there are more than one sed commands available, either via a sed script, -e options, or { }, it executes all the sed commands one by one in sequence on the line that is currently in the pattern space.
  • Print the line from the pattern space. After printing this line, the sed pattern space will be empty.
  • Repeat this again until the end of the input file is reached.

Read a line, Execute the sed command, Print the line, Repeat this again

3. Print Pattern Space (p command)

Using the sed p command, you can print the current pattern space.

You may wonder why you would need the p command, since by default sed prints the pattern buffer after executing its commands.

There are reasons, as you will see; the command allows you to specifically control what is printed to stdout. Usually when p is used you will use the -n option to suppress the the default printing that happens as part of the standard sed flow. Otherwise, when execute p (print) as one of the commands, the line will be printed twice.

The following example prints every line of employee.txt twice:

$ sed 'p' employee.txt
101,John Doe,CEO
101,John Doe,CEO
102,Jason Smith,IT Manager
102,Jason Smith,IT Manager
103,Raj Reddy,Sysadmin
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer
104,Anand Ram,Developer
105,Jane Miller,Sales Manager
105,Jane Miller,Sales Manager

Print each line once (functionally the same as 'cat employee.txt'):

$ sed -n 'p' employee.txt
101,John Doe,CEO
102,Jason Smith,IT Manager
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer
105,Jane Miller,Sales Manager

Specifying an Address Range

If you don't specify an address range before the sed command, by default it matches all the lines. The following are some examples of specifying an address range before the sed command.

Print only the 2nd line:

$ sed -n '2 p' employee.txt
102,Jason Smith,IT Manager

Print from line 1 through line 4:

$ sed -n '1,4 p' employee.txt
101,John Doe,CEO
102,Jason Smith,IT Manager
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer

Print from line 2 through the last line ($ represents the last line):

$ sed -n '2,$ p' employee.txt
102,Jason Smith,IT Manager
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer
105,Jane Miller,Sales Manager

Modify Address Range

You can modify address range using comma, plus, and tilde.

In the examples above, we saw the use of the comma (,) as part of an address range specification. Its meaning is clear: n,m indicates n through m.

The plus (+) may be used in conjunction with the comma, to specify a number of lines instead of an absolute line number. For example, n,
+m means the m lines starting with n.

The tilde (~) may also be used in an address range. Its special meaning is to skip lines between commands. For example, address range n~m indicates that sed should start at the nth line and pick up every mth line from there.

  • 1~2 matches 1,3,5,7, etc.
  • 2~2 matches 2,4,6,8, etc.
  • 1~3 matches 1,4,7,10, etc.
  • 2~3 matches 2,5,8,11, etc.

Print only odd numbered lines:

$ sed -n '1~2 p' employee.txt
101,John Doe,CEO
103,Raj Reddy,Sysadmin
105,Jane Miller,Sales Manager

Pattern Matching

Just as you can specify a numbered address (or address range), you can also specify a pattern (or pattern range) to match, as shown in the next few examples.

Print lines matching the pattern “Jane”:

$ sed -n '/Jane/ p' employee.txt
105,Jane Miller,Sales Manager

Print lines starting from the 1st match of "Jason" until the 4th line:

$ sed -n '/Jason/,4 p' employee.txt
102,Jason Smith,IT Manager
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer

Note: If there were no matches for "Jason" in the 1st 4 lines, this command would print the lines that match "Jason" after the 4th line.

Print lines starting from the 1st match of "Raj" until the last line:

$ sed -n '/Raj/,$ p' employee.txt
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer
105,Jane Miller,Sales Manager

Print lines starting from the line matching "Raj" until the line matching "Jane":

$ sed -n '/Raj/,/Jane/ p' employee.txt
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer
105,Jane Miller,Sales Manager

Print the line matching "Jason" and 2 lines immediately after that:

$ sed -n '/Jason/,+2 p' employee.txt
102,Jason Smith,IT Manager
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer

4. Delete Lines (d command)

Using the sed d command, you can delete lines. Please note that the lines are only deleted from the output stream. Just like any other sed command, the d command doesn't modify the content of the original input file.

By default if you don't specify any address range before the sed command, it matches all the lines. So, the following example will not print anything, as it matches all the lines in the employee.txt and deletes them.

sed 'd' employee.txt

It's more useful to specify an address range to be deleted. The following are some examples:

Delete only the 2nd line:

$ sed '2 d' employee.txt
101,John Doe,CEO
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer
105,Jane Miller,Sales Manager

Delete from line 1 through 4:

$ sed '1,4 d' employee.txt
105,Jane Miller,Sales Manager

Delete from line 2 through the last line:

$ sed '2,$ d' employee.txt
101,John Doe,CEO

Delete only odd number of lines:

$ sed '1~2 d' employee.txt
102,Jason Smith,IT Manager
104,Anand Ram,Developer

Delete lines matching the pattern "Manager":

$ sed '/Manager/ d' employee.txt
101,John Doe,CEO
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer

Delete lines starting from the 1st match of "Jason" until the 4th line:

$ sed '/Jason/,4 d' employee.txt
101,John Doe,CEO
105,Jane Miller,Sales Manager

If there are no matches for "Jason" in the 1st 4 lines, this command deletes only the lines that match "Jason" after the 4th line.

Delete lines starting from the 1st match of "Raj" until the last line:

$ sed '/Raj/,$ d' employee.txt
101,John Doe,CEO
102,Jason Smith,IT Manager

Delete lines starting from the line matching "Raj" until the line matching "Jane":

$ sed '/Raj/,/Jane/ d' employee.txt
101,John Doe,CEO
102,Jason Smith,IT Manager

Delete lines starting from the line matching "Jason" and 2 lines immediately after that:

$ sed '/Jason/,+2 d' employee.txt
101,John Doe,CEO
105,Jane Miller,Sales Manager

Useful Delete Examples

The following examples are very helpful in actual day-to-day operations.

Delete all the empty lines from a file:

sed '/^$/ d' employee.txt

Delete all comment lines (assuming the comment starts with #):

sed '/^#/ d' employee.txt

Note: When you have multiple sed commands, the moment sed encounters the 'd' command, the whole line matching the pattern will be deleted, and no further commands will be executed on the deleted line.

5. Write Pattern Space to File (w command)

Using the sed w command, you can write the current pattern space to a file. By default as per the sed standard flow, the pattern space will be printed to stdout, so if you want output to file but not screen you should also use the sed option -n.
The following are some examples.

Write the content of employee.txt file to file output.txt (and display on screen):

$ sed 'w output.txt' employee.txt
101,John Doe,CEO
102,Jason Smith,IT Manager
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer
105,Jane Miller,Sales Manager

$ cat output.txt
101,John Doe,CEO
102,Jason Smith,IT Manager
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer
105,Jane Miller,Sales Manager

Write the content of employee.txt file to output.txt file but not to screen:

$ sed -n 'w output.txt' employee.txt

cat output.txt
101,John Doe,CEO
102,Jason Smith,IT Manager
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer
105,Jane Miller,Sales Manager

Write only the 2nd line:

$ sed -n '2 w output.txt' employee.txt

$ cat output.txt
102,Jason Smith,IT Manager

Write lines 1 through 4:

$ sed -n '1,4 w output.txt' employee.txt

$ cat output.txt
101,John Doe,CEO
102,Jason Smith,IT Manager
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer

Write from line 2 through the last line:

$ sed -n '2,$ w output.txt' employee.txt

$ cat output.txt
102,Jason Smith,IT Manager
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer
105,Jane Miller,Sales Manager

Write only odd numbered lines:

sed -n '1~2 w output.txt' employee.txt

$ cat output.txt
101,John Doe,CEO
103,Raj Reddy,Sysadmin
105,Jane Miller,Sales Manager

Write lines matching the pattern "Jane":

$ sed -n '/Jane/ w output.txt' employee.txt

$ cat output.txt
105,Jane Miller,Sales Manager

Write lines starting from the 1st match of "Jason" until the 4th line:

$ sed -n '/Jason/,4 w output.txt' employee.txt

$ cat output.txt
102,Jason Smith,IT Manager
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer

If there are no matches for "Jason" in the 1st 4 lines, this command writes only the lines that match "Jason" after the 4th line.

Write lines starting from the 1st match of "Raj" until the last line:

$ sed -n '/Raj/,$ w output.txt' employee.txt

$ cat output.txt
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer
105,Jane Miller,Sales Manager

Write lines starting from the line matching "Raj" until the line matching "Jane":

$ sed -n '/Raj/,/Jane/ w output.txt' employee.txt

$ cat output.txt
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer
105,Jane Miller,Sales Manager

Write the line matching "Jason" and the next 2 lines immediately after that:

$ sed -n '/Jason/,+2 w output.txt' employee.txt

$ cat output.txt
102,Jason Smith,IT Manager
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer

Note: You might not use the w command frequently. Most people use UNIX output redirection, instead, to store the output of sed to a file. For example:

sed 'p' employee.txt > output.txt