Generic crawler for download webpage (like search engine), and extract the required data

Part 1: Generic crawling. Need a generic crawler who can crawl webpage like search engine (scan tons of website and and save it into html file. You can say the project is to build all purpose generic crawler. (Also deal with the JS and pagination). We need the source code and the program, not the crawl result. #search engine experience is preferred Part 2: Data extraction. It will be tons of website from part 1, we want to extract the required data not according to the web structure (different website will have different structure), but using natural language programming. Below is sample data attribute : 1. Geographic preference / country 2. Industry type 3. Investment round 4. Investment amount If you can only do the part 1, you can also apply. But generally the person who can do the part 2 can also do the part 1. You can find the attached job desc for your reference.