Zone Transfers on The Alexa Top 1 Million
At work as part of every assessment we do a some reconnaissance which includes attempting a DNS Zone Transfer (axfr) and conducting a subdomain brute force on the target domain/s. The subdomain brute force is only as good as your wordlist, the Zone Transfer is a matter of luck.
Alexa release a list of the top 1 million sites which is updated on a daily basis. To create a better subdomain wordlist to conduct subdomain brute forcing I attempted a DNS Zone Transfer against the first 2000 sites in the Alexa Top 1 Million list. With every successful Zone Transfer the DNS A records were stored in a CSV file.
This was all done using Carlos Perez's dnsrecon DNS enumeration tool. Dnsrecon was ever so slightly modified to only save A records, apart from that I just used a bash script to iterate over the Top 1 Million list and ran dnsrecon's axfr option for each site with CSV output enabled.
A nice side effect to creating the subdomain wordlist is knowing how many DNS Name Servers have Zone Transfers enabled and which sites. Out of the top 2000 sites, 98 had at least one Name Server with Zone Transfer enabled (4.9%). This included sites we all know and/or use such as Pingdom, Mega Upload, Spotify, Gravatar, American Express and 93 other sites in the top 2000. Some of these sites may have Zone Transfers enabled on purpose, the majority probably don't know it is enabled. The full list of domains with Zone Transfers enabled and their Alexa Ranking can be found here - http://ethicalhack3r.co.uk/files/misc/axfr_domains.txt
Top 10 Alexa domains with Zone Transfers enabled:
In total there were 55,450 A records gathered from the 98 sites. After sorting the list of subdomains by the number of sites each subdomain was found on, removing any duplicates (some sites listed more than one of the same subdomain) and removing subdomains that were only found on one site, the final subdomain list consists of 859 lines. The final list including the number of instances each subdomain was seen across the 98 sites can be found here - http://ethicalhack3r.co.uk/files/misc/subdomain_count.txt
The top 10 subdomains were:
The ns2 subdomain is apparently more popular than the ns1 subdomain which is unexpected. The localhost subdomain seemed to always point to the localhost (127.0.0.1). The mail subdomain was the most popular subdomain overall.
And finally, the subdomain wordlist itself sorted by popularity can be found here - http://ethicalhack3r.co.uk/files/fuzzing/subdomains.txt (859 lines). I would recommend combining this list with the list you're already using for the best results.
And this is the code used to sort the dnsrecon CSV output files:
The next step if anyone has the time and resources is to conduct the test against the full top 1 million list. The top 2000 took about 12 hours or so.