Looking to have an application developed that will spider websites to find external links.
The application should follow all internal links on a site and export all external links to a text file one per line. No duplicate URL's should be added to this external file.
The program should support loading a list of multiple sites to be spidered and running through all of them without additional input.
The app should show a running count for the amount of URL's found.
The program should allow for the use of multiple threads for speed.
I prefer a windows application but am open to exploring web based solutions as well.