Unix / Linux: Remove duplicate lines from a text file using awk or perl

You can use the following command to remove duplicate lines from a text file using awk:

awk '!seen[$0]++' file.txt > file_without_duplicates.txt

Alternatively, you can use the following command to achieve the same result using perl:

perl -ne 'print unless $seen{$_}++' file.txt > file_without_duplicates.txt

These commands will read the contents of file.txt line by line and store each line in a hash table. If a line has already been seen, it will not be printed to the output file file_without_duplicates.txt.