Linux Command - uniq

The linux command uniq will filter duplicate adjacent lines. Lines are only matches as dulicates if they are adjacent. This command is commonly combined with the sort command.

Removes adjacent duplicates leaving only one of each:

uniq test1.txt

Sort first and then remove all duplicates file wide:

sort test1.txt | uniq

There are more options but you probably won’t even need what I’ve listed here:

-c	show duplicate count for each line
-d	only show duplicate lines
-D	only show duplicate lines but show all of them
-i	ingore case
-u	only print uniq lines, anything duplicated won’t be printed at all

NOTE - Pay attention to what ‘-u’ does. It usually isn’t want you need.