I need a program written that will crawl specific websites ( about 200) and extract data from them, then populate my data base with that info. This process needs to be repeated daily and any new or changed information would then have to be updated. We are a construction information service. These websites are usually city and county websites that post contract/projects that are out for bid.
We want to collect the information from all cities and counties in our state and aggregate that information in one locattion to the benefit of our subscribers. Here is a sample site...[url removed, login to view] is our site [url removed, login to view]