Chapter 1: Sed Syntax and Basic Commands
For all sed examples, we'll be using the following employee.txt file. Please create this text file to try out the commands given in this book.
The above employee database contains the following fields for every record:
$ vi employee.txt 101,John Doe,CEO 102,Jason Smith,IT Manager 103,Raj Reddy,Sysadmin 104,Anand Ram,Developer 105,Jane Miller,Sales Manager
- Employee Id
- Employee Name
- Title
1. Sed Command Syntax
The purpose of this section is to get you familiarized with sed syntax and command structure. This is not meant to explain the individual sed commands, which are covered in detail later. Basic sed syntax:
sed [options] {sed-commands} {input-file}
Sed reads one line at a time from the {input-file} and executes the {sed-commands} on that particular line.
It reads the 1st line from the {input-file} and executes the {sed- commands} on the 1st line. Then it reads the 2nd line from the
{input-file} and executes the {sed-commands} on the 2nd line. Sed repeats this process until it reaches the end of the {input-file}.
There are also a few optional command line options that can be passed to sed as indicated by [options].
The following example basic sed syntax. This simple sed example prints all the lines from the /etc/passwd file.
sed -n 'p' /etc/passwdThe main focus here is on the {sed-commands}, which can be either a single sed command or multiple sed commands. You can also combine multiple sed-commands in a file and call the sed script file using the -f option as shown below. Basic sed syntax for use with sed-command file:
sed [options] -f {sed-commands-in-a-file} {input-file}
The following example demonstrates the basic syntax. This example prints lines beginning with root and nobody from the /etc/passwd file.
$ vi test-script.sed /^root/ p /^nobody/ p $ sed -n -f test-script.sed /etc/passwdWhile executing multiple sed commands, you can also directly specify them in the command line using -e as shown below. Basic sed syntax using -e:
sed [options] -e {sed-command-1} -e {sed-command-2}{input-file}
The following example demonstrates the basic syntax. This prints lines beginning with root and nobody from /etc/passwd file:
sed -n -e '/^root/ p' -e '/^nobody/ p' /etc/passwdIf you are executing a lot of commands in a single line using several -e arguments, you can split them into multiple lines using a back slash as shown below.
sed -n \ -e '/^root/ p' \ -e '/^nobody/ p' \ /etc/passwdYou can also execute multiple sed commands in the command line by grouping them together using { }: Basic sed syntax using { }:
sed [options] '{
sed-command-1
sed-command-2
}' input-file
The following example demonstrates this version of the basic syntax. This also prints lines beginning with root and nobody from
/etc/passwd file.
sed -n '{
/^root/ p
/^nobody/ p
}' /etc/passwd
Note: Sed never modifies the original file. It always prints the output to stdout. If you want to save the changes, you should redirect the output to a file by explicitly specifying > filename.txt.
2. Sed Scripting Flow
Sed scripting follows the easily remembered sequence Read, Execute, Print, Repeat. Use the simple REPR acronym to remember sed execution flow. We look at the steps in this sequence. Sed will:- Read a line into the pattern space (an internal temporary sed buffer, where it places the line it reads from the input file).
- Execute the sed command on the line in the sed pattern space. If there are more than one sed commands available, either via a sed script, -e options, or { }, it executes all the sed commands one by one in sequence on the line that is currently in the pattern space.
- Print the line from the pattern space. After printing this line, the sed pattern space will be empty.
- Repeat this again until the end of the input file is reached.
3. Print Pattern Space (p command)
Using the sed p command, you can print the current pattern space. You may wonder why you would need the p command, since by default sed prints the pattern buffer after executing its commands. There are reasons, as you will see; the command allows you to specifically control what is printed to stdout. Usually when p is used you will use the -n option to suppress the the default printing that happens as part of the standard sed flow. Otherwise, when execute p (print) as one of the commands, the line will be printed twice. The following example prints every line of employee.txt twice:$ sed 'p' employee.txt 101,John Doe,CEO 101,John Doe,CEO 102,Jason Smith,IT Manager 102,Jason Smith,IT Manager 103,Raj Reddy,Sysadmin 103,Raj Reddy,Sysadmin 104,Anand Ram,Developer 104,Anand Ram,Developer 105,Jane Miller,Sales Manager 105,Jane Miller,Sales ManagerPrint each line once (functionally the same as 'cat employee.txt'):
$ sed -n 'p' employee.txt 101,John Doe,CEO 102,Jason Smith,IT Manager 103,Raj Reddy,Sysadmin 104,Anand Ram,Developer 105,Jane Miller,Sales Manager
Specifying an Address Range
If you don't specify an address range before the sed command, by default it matches all the lines. The following are some examples of specifying an address range before the sed command. Print only the 2nd line:$ sed -n '2 p' employee.txt 102,Jason Smith,IT ManagerPrint from line 1 through line 4:
$ sed -n '1,4 p' employee.txt 101,John Doe,CEO 102,Jason Smith,IT Manager 103,Raj Reddy,Sysadmin 104,Anand Ram,DeveloperPrint from line 2 through the last line ($ represents the last line):
$ sed -n '2,$ p' employee.txt 102,Jason Smith,IT Manager 103,Raj Reddy,Sysadmin 104,Anand Ram,Developer 105,Jane Miller,Sales Manager
Modify Address Range
You can modify address range using comma, plus, and tilde. In the examples above, we saw the use of the comma (,) as part of an address range specification. Its meaning is clear: n,m indicates n through m. The plus (+) may be used in conjunction with the comma, to specify a number of lines instead of an absolute line number. For example, n, +m means the m lines starting with n. The tilde (~) may also be used in an address range. Its special meaning is to skip lines between commands. For example, address range n~m indicates that sed should start at the nth line and pick up every mth line from there.- 1~2 matches 1,3,5,7, etc.
- 2~2 matches 2,4,6,8, etc.
- 1~3 matches 1,4,7,10, etc.
- 2~3 matches 2,5,8,11, etc.
$ sed -n '1~2 p' employee.txt 101,John Doe,CEO 103,Raj Reddy,Sysadmin 105,Jane Miller,Sales Manager
Pattern Matching
Just as you can specify a numbered address (or address range), you can also specify a pattern (or pattern range) to match, as shown in the next few examples. Print lines matching the pattern “Jane”:$ sed -n '/Jane/ p' employee.txt 105,Jane Miller,Sales ManagerPrint lines starting from the 1st match of "Jason" until the 4th line:
$ sed -n '/Jason/,4 p' employee.txt 102,Jason Smith,IT Manager 103,Raj Reddy,Sysadmin 104,Anand Ram,DeveloperNote: If there were no matches for "Jason" in the 1st 4 lines, this command would print the lines that match "Jason" after the 4th line. Print lines starting from the 1st match of "Raj" until the last line:
$ sed -n '/Raj/,$ p' employee.txt 103,Raj Reddy,Sysadmin 104,Anand Ram,Developer 105,Jane Miller,Sales ManagerPrint lines starting from the line matching "Raj" until the line matching "Jane":
$ sed -n '/Raj/,/Jane/ p' employee.txt 103,Raj Reddy,Sysadmin 104,Anand Ram,Developer 105,Jane Miller,Sales ManagerPrint the line matching "Jason" and 2 lines immediately after that:
$ sed -n '/Jason/,+2 p' employee.txt 102,Jason Smith,IT Manager 103,Raj Reddy,Sysadmin 104,Anand Ram,Developer
4. Delete Lines (d command)
Using the sed d command, you can delete lines. Please note that the lines are only deleted from the output stream. Just like any other sed command, the d command doesn't modify the content of the original input file. By default if you don't specify any address range before the sed command, it matches all the lines. So, the following example will not print anything, as it matches all the lines in the employee.txt and deletes them.sed 'd' employee.txtIt's more useful to specify an address range to be deleted. The following are some examples: Delete only the 2nd line:
$ sed '2 d' employee.txt 101,John Doe,CEO 103,Raj Reddy,Sysadmin 104,Anand Ram,Developer 105,Jane Miller,Sales ManagerDelete from line 1 through 4:
$ sed '1,4 d' employee.txt 105,Jane Miller,Sales ManagerDelete from line 2 through the last line:
$ sed '2,$ d' employee.txt 101,John Doe,CEODelete only odd number of lines:
$ sed '1~2 d' employee.txt 102,Jason Smith,IT Manager 104,Anand Ram,DeveloperDelete lines matching the pattern "Manager":
$ sed '/Manager/ d' employee.txt 101,John Doe,CEO 103,Raj Reddy,Sysadmin 104,Anand Ram,DeveloperDelete lines starting from the 1st match of "Jason" until the 4th line:
$ sed '/Jason/,4 d' employee.txt 101,John Doe,CEO 105,Jane Miller,Sales ManagerIf there are no matches for "Jason" in the 1st 4 lines, this command deletes only the lines that match "Jason" after the 4th line. Delete lines starting from the 1st match of "Raj" until the last line:
$ sed '/Raj/,$ d' employee.txt 101,John Doe,CEO 102,Jason Smith,IT ManagerDelete lines starting from the line matching "Raj" until the line matching "Jane":
$ sed '/Raj/,/Jane/ d' employee.txt 101,John Doe,CEO 102,Jason Smith,IT ManagerDelete lines starting from the line matching "Jason" and 2 lines immediately after that:
$ sed '/Jason/,+2 d' employee.txt 101,John Doe,CEO 105,Jane Miller,Sales Manager
Useful Delete Examples
The following examples are very helpful in actual day-to-day operations. Delete all the empty lines from a file:sed '/^$/ d' employee.txtDelete all comment lines (assuming the comment starts with #):
sed '/^#/ d' employee.txtNote: When you have multiple sed commands, the moment sed encounters the 'd' command, the whole line matching the pattern will be deleted, and no further commands will be executed on the deleted line.
5. Write Pattern Space to File (w command)
Using the sed w command, you can write the current pattern space to a file. By default as per the sed standard flow, the pattern space will be printed to stdout, so if you want output to file but not screen you should also use the sed option -n. The following are some examples. Write the content of employee.txt file to file output.txt (and display on screen):$ sed 'w output.txt' employee.txt 101,John Doe,CEO 102,Jason Smith,IT Manager 103,Raj Reddy,Sysadmin 104,Anand Ram,Developer 105,Jane Miller,Sales Manager $ cat output.txt 101,John Doe,CEO 102,Jason Smith,IT Manager 103,Raj Reddy,Sysadmin 104,Anand Ram,Developer 105,Jane Miller,Sales ManagerWrite the content of employee.txt file to output.txt file but not to screen:
$ sed -n 'w output.txt' employee.txt cat output.txt 101,John Doe,CEO 102,Jason Smith,IT Manager 103,Raj Reddy,Sysadmin 104,Anand Ram,Developer 105,Jane Miller,Sales ManagerWrite only the 2nd line:
$ sed -n '2 w output.txt' employee.txt $ cat output.txt 102,Jason Smith,IT ManagerWrite lines 1 through 4:
$ sed -n '1,4 w output.txt' employee.txt $ cat output.txt 101,John Doe,CEO 102,Jason Smith,IT Manager 103,Raj Reddy,Sysadmin 104,Anand Ram,DeveloperWrite from line 2 through the last line:
$ sed -n '2,$ w output.txt' employee.txt $ cat output.txt 102,Jason Smith,IT Manager 103,Raj Reddy,Sysadmin 104,Anand Ram,Developer 105,Jane Miller,Sales ManagerWrite only odd numbered lines:
sed -n '1~2 w output.txt' employee.txt $ cat output.txt 101,John Doe,CEO 103,Raj Reddy,Sysadmin 105,Jane Miller,Sales ManagerWrite lines matching the pattern "Jane":
$ sed -n '/Jane/ w output.txt' employee.txt $ cat output.txt 105,Jane Miller,Sales ManagerWrite lines starting from the 1st match of "Jason" until the 4th line:
$ sed -n '/Jason/,4 w output.txt' employee.txt $ cat output.txt 102,Jason Smith,IT Manager 103,Raj Reddy,Sysadmin 104,Anand Ram,DeveloperIf there are no matches for "Jason" in the 1st 4 lines, this command writes only the lines that match "Jason" after the 4th line. Write lines starting from the 1st match of "Raj" until the last line:
$ sed -n '/Raj/,$ w output.txt' employee.txt $ cat output.txt 103,Raj Reddy,Sysadmin 104,Anand Ram,Developer 105,Jane Miller,Sales ManagerWrite lines starting from the line matching "Raj" until the line matching "Jane":
$ sed -n '/Raj/,/Jane/ w output.txt' employee.txt $ cat output.txt 103,Raj Reddy,Sysadmin 104,Anand Ram,Developer 105,Jane Miller,Sales ManagerWrite the line matching "Jason" and the next 2 lines immediately after that:
$ sed -n '/Jason/,+2 w output.txt' employee.txt $ cat output.txt 102,Jason Smith,IT Manager 103,Raj Reddy,Sysadmin 104,Anand Ram,DeveloperNote: You might not use the w command frequently. Most people use UNIX output redirection, instead, to store the output of sed to a file. For example:
sed 'p' employee.txt > output.txt