We are currently seeking professionals who can build a very extensive clone of the Google bot and the Yahoo Slup bot.
The crawler needs to be VERY extensive, and we will be running it 24 hours per day 7 days per week on 4 different dedicated high end servers, to create our very own search index. We do not have a certain requirement as to how many sites the crawler must crawl per day, but we eventually want a database that is very comparable to Yahoo’s and Google’s.
By or own research Google currently has over billions and billions of results in their database, placed over 10 million dollars worth of high end redundant servers. Although we do not have the same type of high end servers we currently have our servers setup like this.
We have 4 servers with the following configuration….
Quad Intel Xeon 2.8 GHz HT
4 X 250 Gigabytes Hard Drives
Raid 5 Array
Dedicated 100mbps Bandwidth ( Per Server )
These 4 servers will hold the database for our very own search engine. With over 4,000 Gigabytes of storage and extremely high end processors and dedicated bandwidth we anticipate that our servers can handle extremely large loads.
Your bid is to build ONE completely high end custom crawler, which we will run on 4 servers at once to build the database.
We will need complete training on how to use the crawler, and full tech support for 5 months if needed.
If you have a demo crawler please give us the URL to where we can see, and or test the crawler.
YOU MUST PROVIDE IN YOUR BID THE AMOUNT OF SITES THAT THE CRAWLER CAN CRAWL PER MONTH, PER EACH SERVER THAT IT IS RUNNING ON.
WE ARE NOT LOOKING FOR A CHEAP CRAWLER, AS YOU CAN TELL BY OUR BUDGET, WE ARE SERIOUS ABOUT A HIGH END CRAWLER