Главная › Sed and Awk 101 Hacks › Chapter 2. Sed Substitute Command

Chapter 2. Sed Substitute Command

The most powerful command in the stream editor is substitute. It has such power and so many options that we give it a whole chapter.

6. Sed Substitute Command Syntax

sed '[address-range|pattern-range] s/originalstring/replacement-string/[substitute-flags]' inputfile
In the above sed substitute command syntax:
  • address-range or pattern-range is optional. If you don't specify one, sed will execute the substitute command on all lines.
  • s – tells Sed to execute the substitute command.
  • original-string – this is the string to be searched for in the input file. The original-string can also be a regular expression.
  • replacement-string – Sed will replace original-string with this string.
  • substitute-flags are optional. More on this in the next section.
Remember that the original file is not changed; the substitution takes place in the pattern space buffer which is then printed to stdout. The following are couple of simple sed substitute examples (changes shown in bold). Replace all occurrences of Manager with Director: 
$ sed 's/Manager/Director/' employee.txt
101,John Doe,CEO
102,Jason Smith,IT Director
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer
105,Jane Miller,Sales Director
Replace Manager with Director only on lines that contain the keyword 'Sales': 
sed '/Sales/s/Manager/Director/' employee.txt
101,John Doe,CEO
102,Jason Smith,IT Manager
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer
105,Jane Miller,Sales Director
Note that the use of the address range caused just one change rather than the two shown in the previous example.

7. Global Flag (g flag)

Sed substitute flag g stands for global. By default sed substitute command will replace only the 1st occurrence of the {original-string} on each line. If you want to change all the occurrences of the {original-string} in the line to the {replacement-string}, you should use the global flag g. Replace the 1st occurrence of lower case a with upper case A: 
$ sed 's/a/A/' employee.txt
101,John Doe,CEO
102,JAson Smith,IT Manager
103,RAj Reddy,Sysadmin
104,AnAnd Ram,Developer
105,JAne Miller,Sales Manager
Replace all occurrences of lower case a with upper case A: 
$ sed 's/a/A/g' employee.txt
101,John Doe,CEO
102,JAson Smith,IT MAnAger
103,RAj Reddy,SysAdmin
104,AnAnd RAm,Developer
105,JAne Miller,SAles MAnAger
Note: these examples were applied to the entire file because no address range was specified.

8. Number Flag (1,2,3.. flag)

Use the number flag to specify a specific occurrence of the original-string. Only the n-th instance of original-string will trigger the substitution. Counting starts over on each line, and n can be anything from 1 to 512. For example, /11 will replace only the 11th occurrence of the original-string in a line. The following are few examples. Replace the 2nd occurrence of lower case a to upper case A: 
$ sed 's/a/A/2' employee.txt
101,John Doe,CEO
102,Jason Smith,IT MAnager
103,Raj Reddy,SysAdmin
104,Anand RAm,Developer
105,Jane Miller,SAles Manager
For this example, create the following file with three lines: 
$ vi substitute-locate.txt
locate command is used to locate files
locate command uses database to locate files
locate command can also use regex for searching
In the file you just created, change only the 2nd occurrence of locate to find: 
$ sed 's/locate/find/2' substitute-locate.txt
locate command is used to find files
locate command uses database to find files
locate command can also use regex for searching
Note: On line 3 in the above example, there is only one "locate" in the original substitute-locate.txt file. So, nothing is changed on line 3.

9. Print Flag (p flag)‌

The sed substitute flag p stands for print. When the substitution is successful, it prints the changed line. Like most print commands in sed, it is most useful when combined with the -n option to suppress default printing of all lines. Print only the line that was changed by the substitute command: 
$ sed -n 's/John/Johnny/p' employee.txt 101,Johnny Doe,CEO
In our number flag example, we used /2 to change the 2nd occurrence of "locate" to "find". On line 3 of locate.txt there is no 2nd occurrence and substitution never happened on that line. Adding the p flag to the command we used before will print the two lines that did change. Change the 2nd instance of “locate” to “find” and print the result: 
$ sed -n 's/locate/find/2p' substitute-locate.txt
locate command is used to find files
locate command uses database to find files

10. Write Flag (w flag)

The sed substitute flag w stands for write. When the substitution is successful, it writes the changed line to a file. Most people use the p flag instead, and redirect the output to a file. We include this command for completeness. Write only the line that was changed by the substitute command to output.txt: 
$ sed -n 's/John/Johnny/w output.txt' employee.txt
$ cat output.txt 
101,Johnny Doe,CEO
Just as we showed for the p command, adding w to our example with the substitute-locate.txt file will send the two lines that were changed to the output file. Change the 2nd instance of “locate” to “find”,write the result to a file, print all lines: 
$ sed 's/locate/find/2w output.txt' substitute-locate.txt
locate command is used to find files
locate command uses database to find files
locate command can also use regex for searching

$ cat output.txt
locate command is used to find files
locate command uses database to find files

11. Ignore Case Flag (i flag)‌

The sed substitute flag i stands for ignore case. You can use the i flag to match the original-string in a case-insensitive manner. This is available only in GNU Sed. In the following example Sed will not replace "John" with "Johnny", as the original-string was given in lower case "john". Replace “john” with Johnny: 
$ sed 's/john/Johnny/' employee.txt
101,John Doe,CEO
102,Jason Smith,IT Manager
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer
105,Jane Miller,Sales Manager
Replace “john” or “John” with Johnny: 
$ sed 's/john/Johnny/i' employee.txt
101,Johnny Doe,CEO
102,Jason Smith,IT Manager
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer
105,Jane Miller,Sales Manager

12. Execute Flag (e flag)‌

The sed substitute flag e stands for execute. Using the sed e flag, you can execute whatever is available in the pattern space as a shell command, and the output will be returned to the pattern space. This is available only in the GNU sed. The following are few examples. For these examples create the following files.txt that contains a list of filenames with their full path: 
$ cat files.txt
/etc/passwd
/etc/group
Add the text "ls -l " in front of every line in the files.txt and print the output: 
$ sed 's/^/ls -l /' files.txt
ls -l /etc/passwd
ls -l /etc/group
Add the text "ls -l " in front of every line in the files.txt and execute the output: 
$ sed 's/^/ls -l /e' files.txt
-rw-r--r-- 1 root root 1547 Oct 27 08:11 /etc/passwd
-rw-r--r-- 1 root root 651 Oct 27 08:11 /etc/group

13. Combine Sed Substitution Flags‌

You can combine one or more substitute flags as required. The following example will replace all occurrences of "Manager" or "manager" to "Director". This will also print only the line that was changed by the substitute command to the screen, and write the same information to the output.txt file. Combine g,i,p and w flags: 
$ sed -n 's/Manager/Director/gipw output.txt'
employee.txt
102,Jason Smith,IT Director
105,Jane Miller,Sales Director

$ cat output.txt
102,Jason Smith,IT Director
105,Jane Miller,Sales Director

14. Sed Substitution Delimiter‌

In all the above sed examples, we used the default sed delimiter /. i.e. s/original-string/replacement-string/ When there is a slash / in the original-string or the replacement-string, we need to escape it using \. For this example create a path.txt file which contains a directory path as shown below. 
$ vi path.txt reading /usr/local/bin directory
Now, let us change /usr/local/bin to /usr/bin using the sed substitute command. In this sed substitution example, the delimiter path delimiter ‘/’ was escaped using back slash '\' in the original-string and the replacement-string. 
$ sed 's/\/usr\/local\/bin/\/usr\/bin/' path.txt reading /usr/bin directory
Ugly isn't it? When you are trying to replace a long path name, it might be very confusing to use all those escape characters '\'. Fortunately, you can use any character as substitution delimiter. For example, | or ^ or @ or !. All of the following are valid and easy to read. I have not shown the output of the commands since they all produce exactly the same result. I prefer to use @ (or !) symbol when replacing a directory path but it is your personal choice. 
sed 's|/usr/local/bin|/usr/bin|' path.txt
sed 's^/usr/local/bin^/usr/bin^' path.txt
sed 's@/usr/local/bin@/usr/bin@' path.txt
sed 's!/usr/local/bin!/usr/bin!' path.txt

15. Multiple Substitute Commands Affecting the Same Line‌

As we discussed earlier, the sed execution flow is Read, Execute, Print, Repeat. The Execute portion, as we mentioned, may consist of multiple sed commands, which sed will execute one-by-one. For example, if you have two sed commands, sed will execute command-1 on the pattern space, then execute command-2 on the pattern space. If command-1 changed something in the pattern space, command-2 will be executed on the newly changed pattern space (and not the original line that was Read). The following example demonstrates the execution of two sed substitute commands on the pattern space. Change Developer to IT Manager, then change Manager to Director: 
$ sed '{
s/Developer/IT Manager/
s/Manager/Director/
}' employee.txt
101,John Doe,CEO
102,Jason Smith,IT Director
103,Raj Reddy,Sysadmin
104,Anand Ram,IT Director
105,Jane Miller,Sales Director
Let us analyze the sed execution flow for line 4 in the example. 1. Read: At this stage, Sed reads the line and puts it in the pattern space. So, the following is the content of the pattern space. 
104,Anand Ram,Developer
2. Execute: Sed executes the 1st sed command on the pattern space, which is s/Developer/IT Manager/. So, after this command, the following is the content of the pattern space. 
104,Anand Ram,IT Manager
Now, sed executes the 2nd sed command on the pattern space, which is s/Manager/Director/. After this command, the following is the content of the pattern space. 
104,Anand Ram,IT Director
Remember: Sed executes the 2nd command on the content of the current pattern space after execution of the first command. 3. Print: It prints the content of the current pattern space, which is the following. 
104,Anand Ram,IT Director
4. Repeat: It moves on to the next line and repeats from step#1.

16. Power of & - Get Matched Pattern

When & is used in the replacement-string, it replaces it with whatever text matched the original-string or the regular-expression. This is very powerful and useful. The following are few examples. Enclose the employee id (the 1st three numbers) between [ and ], i.e. 101 becomes [101], 102 becomes [102], etc. 
$ sed 's/^[0-9][0-9][0-9]/[&]/g' employee.txt
[101],John Doe,CEO
[102],Jason Smith,IT Manager
[103],Raj Reddy,Sysadmin
[104],Anand Ram,Developer
[105],Jane Miller,Sales Manager
Enclose the whole input line between < and > 
$ sed 's/^.*/<&>/' employee.txt
<101,John Doe,CEO>
<102,Jason Smith,IT Manager>
<103,Raj Reddy,Sysadmin>
<104,Anand Ram,Developer>
<105,Jane Miller,Sales Manager>

17. Substitution Grouping (Single Group)

Grouping can be used in sed just like in a normal regular expression. A group is opened with “\(” and closed with “\)”. Grouping can be used in combination with back-referencing. A back-reference is the re-use of a part of a regular expression selected by grouping. Back-references in sed can be used in both a regular expression and in the replacement part of the substitute command.

Single grouping:

$ sed 's/\([^,]*\).*/\1/g' employee.txt
101
102
103
104
105
In the above example:
  • Regular expression \([^,]*\) matches the string up to the 1st comma.
  • \1 in the replacement-string replaces the first matched group.
  • g is the global substitute flag.
This sed example displays only the first field from the /etc/passwd file, i.e. it displays only the username: 
sed 's/\([^:]*\).*/\1/' /etc/passwd
The following example encloses the 1st letter in every word inside (), if the 1st character is upper case. 
$ echo "The Geek Stuff" | sed 's/\(\b[A-Z]\)/\(\1\)/g'
(T)he (G)eek (S)tuff
For the next example create a numbers.txt sample file as shown below. 
$ vi numbers.txt
1
12
123
1234
12345
123456
Commify numbers, i.e. insert commas to make them more readable: 
$ sed 's/\(^\|[^0-9.]\)\([0-9]\+\)\([0-9]\
{3\}\)/\1\2,\3/g' numbers.txt
1
12
123
1,234
12,345
123,456
The above command should be executed in a single line as shown below. 
sed 's/\(^\|[^0-9.]\)\([0-9]\+\)\([0-9]\{3\}\)/\1\2,\3/g' numbers.txt

18. Substitution Grouping (Multiple Group)‌

In multi grouping, you can have multiple groups enclosed in multiple “\(” and “\)”. When you have multiple groups in the substitute regular expression, you can use \n to specify the nth group in the sed replacement string. An example is shown below. Get only the 1st column (employee id) and the 3rd column (title): 
$ sed 's/\([^,]*\),\([^,]*\),\([^,]*\).*/\1,\3/g'
employee.txt
101,CEO
102,IT Manager
103,Sysadmin
104,Developer
105,Sales Manager
The above command should be executed in a single line as shown below. 
sed 's/\([^,]*\),\([^,]*\),\([^,]*\).*/\1,\3/g' employee.txt
In the above example, you can see three groups mentioned in the original-string (reg-ex). These three groups are separated by commas.
  • ([^,]*\) is group 1 that matches the employee id
  • , is the field separator after group 1
  • ([^,]*\) is group 2 that matches the employee name
  • , is the field separator after group 2
  • ([^,]*\) is group 3 that matches the employee title
  • , is the field separator after group 3 The replacement-string section of the above example indicates how these groups should be used.
  • \1 is to print group 1 (employee id)
  • , is to print a comma after printing group 1
  • \3 is to print group 1 (title)
Note: Sed can hold a maximum of 9 groups referenced using \1 through \9 Swap field 1 (employee id) with field 2 (employee name); print the employee.txt file: 
$ sed 's/\([^,]*\),\([^,]*\),\(.*\).*/\2,\1,\3/g'
employee.txt
John Doe,101,CEO
Jason Smith,102,IT Manager
Raj Reddy,103,Sysadmin
Anand Ram,104,Developer
Jane Miller,105,Sales Manager
The above command should be executed in a single line as shown below. 
sed 's/\([^,]*\),\([^,]*\),\(.*\).*/\2,\1,\3/g' employee.txt

19. Gnu Sed Only Replacement String Flags

These flags are available only in GNU version of sed. They can be used in the replacement-string part of the sed substitute command.

\l replacement string flag

When you specify \l in the replacement-string part, it treats the character that immediately follows \l as lower case. You already know the following simple example will change John to JOHNNY. 
sed 's/John/JOHNNY/' employee.txt
The following example contains \l before H in the replacement-string (i.e. JO\lHNNY). This will change only the character h in JOHNNY to lower case. Change John to JOhNNY: 
$ sed -n 's/John/JO\lHNNY/p' employee.txt
101,JOhNNY Doe,CEO

\L replacement string flag

When you specify \L in the replacement-string part, it treats the rest of the characters as lower case. The following example contains \L before H in the replacement-string (i.e. JO\lHNNY). This will change the rest of the characters from h to lower case. Change Johnny to JOhnny: 
$ sed -n 's/John/JO\LHNNY/p' employee.txt
101,JOhnny Doe,CEO

\u replacement string flag

Just like \l, but for upper case. When you specify \l in the replacement-string part, it treats the character that immediately follows \u as upper case. The following example contains \u before h in the replacement-string (i.e. jo\uhnny). This will change only the character h in johnny to upper case. Change John to joHnny: 
$ sed -n 's/John/jo\uhnny/p' employee.txt
101,joHnny Doe,CEO

\U replacement string flag

When you specify \U in the replacement-string part, it treats the rest of the characters as upper case. The following example contains \U before h in the replacement-string (i.e. jo\Uhnny). This will change the rest of the characters from h in johnny to upper case. Change John to joHNNY: 
$ sed -n 's/John/jo\Uhnny/p' employee.txt
101,joHNNY Doe,CEO

\E replacement string flag

This should be used in conjunction with either \L or \U. This stops the conversion initiated by either \L or \U. The following example prints the whole replacement string "Johnny Boy" in upper case, as we have \U at the beginning of the replacement-string. Change John to JOHNNY BOY: 
$ sed -n 's/John/\UJohnny Boy/p' employee.txt
101,JOHNNY BOY Doe,CEO
Change John to JOHNNY Boy: 
$ sed -n 's/John/\UJohnny\E Boy/p' employee.txt
101,JOHNNY Boy Doe,CEO
The above example prints only "Johnny" in the upper case, as we have \E immediately after "Johnny" in the replacement-string.

Replacement String Flag Usages

The above static examples are shown only to understand how these switches works. However, the flags don't have much value when used with static values, as you can just type the static values in the exact case needed. The flags are quite useful when combined with grouping. In the previous example we learned how to swap field 1 with field 3 using grouping. You can convert a whole grouping to either upper or lower case using these switches. Employee name in all upper case, and title in all lower case: 
$ sed 's/\([^,]*\),\([^,]*\),\(.*\).*/\U\2\E,\1,\L\3/g'
employee.txt
JOHN DOE,101,ceo
JASON SMITH,102,it manager
RAJ REDDY,103,sysadmin
ANAND RAM,104,developer
JANE MILLER,105,sales manager
The above command should be executed in a single line as shown below. 
sed 's/\([^,]*\),\([^,]*\),\(.*\).*/\U\2\E,\1,\L\3/g' employee.txt
In the above example, in the replacement-string, we have the following:
  • \U\2\E - This indicates that this field, which is the 2nd group (employee name), should be converted to upper case. \U start the upper case conversion, and \E stops it.
  • \L\3 - This indicates that this field, which is 3rd group (title), should be converted to lower case. \L starts the lower case conversion for rest of the characters.