Scripted Scraping for 150 different sites.
I need about 150 sites setup for automated twice a day scraping. Many of these sites have rss feeds or basic html, but alot do not. There are a list of about 20 total fields of data that I need scraped from these sites, with about 10 of these as CRITICAL components. You will be setting up the scraper to gather this data for an existing site/database so it is important that you/your team can communicate well w/ the current site developers.
Another important factor here is that these are to be queried twice a day using proxies so our IPs do not get banned. Duplicate information filtering also required.
Please pm me for the list of sites and fields you need to gather. You can go through all of the sites and then tell me the cost.
Looking to start within 24 hours.