Removing duplicate characters from a string on Linux with awk

The awk command can make it easy to remove duplicate characters from a string even when those characters aren’t sequential, especially when the process is turned into a script.

First, the awk command that we’ll be using starts by running through each letter in the string. In a more common command, you might see awk doing something like this:

$ echo one:two:three | awk ‘BEGIN {FS =":"} ; { print $2 }’
two

The FS portion of that command specifies the field separator—the character that is used to separate the fields in the string so that they can be processed separately.

What our script does, however, is use a field separator of “” (i.e., no character). This tells awk that there are no field separators. In other words, every character is treated as if it is itself a field. Here’s are a couple examples:

$ echo one:two:three | awk ‘BEGIN { FS ="" } ; { print $2 }’
n
$ echo one:two:three | awk ‘BEGIN { FS ="" } ; { print $4 }’
:

Note that the commands above end up displaying the second and fourth characters in the string, not the second and fourth “fields” and that no distinction is made between blanks, letters and various punctuation characters.

A bash script that uses awk to remove duplicate characters might look like this:

#!bin/bash

echo -n “Enter string: “
read string

awk -v FS="" ‘{
  for(i=1;i<=NF;i++)
    str=(++a[$i]==1?str $i:str)
}
END {print str}’ <<< $string

That script prompts for a string and then uses awk to run through it one character at a time. It adds each successive character to the string (str) only if that character isn’t already included. The characters are otherwise left in their original positions, with no sorting or further processing. Here’s an example of running it:

$ ./rmdups
Enter string: Let’s go fly a kite!
Let’s goflyaki!

Notice that each character appears only once in the “Let’s goflyaki!” results. The final result of the process is displayed in the print statement in the END portion of the awk command.

If you want to see how the script works by viewing the string of characters growing as characters are added, you can use this version of the script instead:

#!/bin/bash

echo -n “Enter string: “
read characters

awk -v FS="" ‘{
  for(i=1;i<=NF;i++)
  {
    str=(++a[$i]==1?str $i:str)
    print str	                 # <== watch it grow
   }
}

END {print str}’ <<< $string

Running the script with the extra print command, you would see output like this:

$ ./rmdups2
Enter string: Let’s go fly a kite!
L
Le
Let
Let’
Let’s
Let’s
Let’s g
Let’s go
Let’s go
Let’s gof
Let’s gofl
Let’s gofly
Let’s gofly
Let’s goflya
Let’s goflya
Let’s goflyak
Let’s goflyaki
Let’s goflyaki
Let’s goflyaki
Let’s goflyaki!
Let’s goflyaki!

Notice that the string grows only when the current character is not already included in the string.

You could also implement the script simply as an awk script like this:

awk -v FS="" ‘{
  for(i=1;i<=NF;i++)
    str=(++a[$i]==1?str $i:str)
}
END {print str}’

You could then run the awk script like this:

$ echo “Let’s go fly a kite!” | rmdups.awk
Let’s goflyaki!

Wrap-Up

Whenever processing duplicated characters more than once would be a serious waste of processing power, an awk command like that shown in this post can remove them quite easily.