App to return all URLs from root domain

I need an app or script which can process a list of root domains, e.g. [login to view URL] and return all web page URLs for that site. It MUST be able to process a large list of 100,000 root domains and return ALL URLs in each site for all of them. Please note that I do NOT want to download the websites, I just want a list of all URLs in the sites. Since it would be going through large lists of root domains and return all URLs for each domain, it must be fast, please consider the time it takes to list URLs of 100,000 root domains because that is the rough size of the lists I have.