I am looking for a server-side web scraper. The script will run on our Linux dedicated server and download all the flash videos from the protected members area of a website. We want to get the flash videos played in the flash player, it has a .flv file extension.
That represents about 6000 videos and will be more than 600GB of data. We need to be able to set limits within the scripts, to limit the number of concurrent downloads for example and make sure we are not overloading their server and neither our server. I am looking for suggestions from you in that regard.
Please let me know if you are interested and I will send you more details.