How to remove duplicate subdomains from result of different tools and combine it together?

General
1

hello here
how to remove duplicate subdomains from result of different tools and combine it together?

for example
1. I ran amass and i save output in a txt file sub1.txt
2. I ran subfinder and i save as sub2.txt
3. So i want to combine sub1.txt together with sub2.txt and save as unique.txt
how will i remove duplicate subs(i.e subs appearing in both sub1 and sub2) and append to unique.txt

1 Like
2

Hello @Venus, welcome to @webs3c and thank you for your participation!

Regarding your questions:

So i want to combine sub1.txt together with sub2.txt and save as unique.txt

cat sub1.txt > not-unique.txt && cat sub2.txt >> not-unique.txt

how will i remove duplicate subs(i.e subs appearing in both sub1 and sub2

sort not-unique.txt | uniq > unique.txt

3 Likes
3

Hey @Venus if you want to use just built-in tools in every Linux machine I would do:

cat amass_output.txt subfinder_output.txt | sort -u > final_result.txt

But you can also use anew TomNomNom’s tool to do this:

cat amass_output.txt subfinder_output.txt | anew -q > final_result.txt

2 Likes
4

thanks bro very helpful

7

anew

1 Like