Our requirement is a full featured dynamic distributed crawler, which need to crawl all the shopping sites in India. The crawler need to extract all the information about the products it’s price, offers, seller and etc. from all the shopping sites and need to store all this collected data in a fully structure and perfectly indexed database for easy extraction of data from the DataBase. The dynamic pages of those shopping sites should be crawled repeatedly within in minimum span of time. The crawler should be dynamic and distributed in nature. The DataBase should be well structured, well organized and perfectly indexed. As on above we need a full functioning stable crawler, database system, proper indexing mechanism.
Note: With your bids share your proposed design model for the above requirements.