How to remove duplicate subdomains from result of different tools and combine it together?

hello here
how to remove duplicate subdomains from result of different tools and combine it together?

for example
1. I ran amass and i save output in a txt file sub1.txt
2. I ran subfinder and i save as sub2.txt
3. So i want to combine sub1.txt together with sub2.txt and save as unique.txt
how will i remove duplicate subs(i.e subs appearing in both sub1 and sub2) and append to unique.txt

1 Like

Hello @Venus, welcome to @webs3c and thank you for your participation!

Regarding your questions:

So i want to combine sub1.txt together with sub2.txt and save as unique.txt

cat sub1.txt > not-unique.txt && cat sub2.txt >> not-unique.txt

how will i remove duplicate subs(i.e subs appearing in both sub1 and sub2

sort not-unique.txt | uniq > unique.txt

2 Likes

Hey @Venus if you want to use just built-in tools in every Linux machine I would do:

cat amass_output.txt subfinder_output.txt | sort -u > final_result.txt

But you can also use anew TomNomNom’s tool to do this:

cat amass_output.txt subfinder_output.txt | anew -q > final_result.txt

2 Likes

thanks bro very helpful