This will be several projects. We would like to hire a team or someone that can work with a team to do different parts of this project. We currently have code for this project in PHP, but would like to revamp or redo the entire application. This will be a ongoing project. These are the different parts of the project in a general overview:
1. We need an application that will be able to crawl different websites. We will need to login in to the website, scrape data from the website, and depending on the data that we gather, submit certain data to the website.
- This can be be in any language, but we like this to be fast. Currently, from the time it finishes parsing a page to submit the data, it takes 3 seconds. We would like to lower this time to about 1 second. We would like someone who can optimize the application and code and identify bottle necks.
- We might want multiple crawler instances to be running at once so would like someone familiar with threading and process forking.
- This will need to be maintainable, so might want this to be in OOP. We will want to to be able to add different sites to scrape to our application easily. Might want to have like a plugin system to easily plug in new sites.
- We want someone that can start a project from beginning and also be able to work with existing code and integrate their implemenation elegantly.
2. We need an admin area, preferably web based. The admin area will manage our application and should allow us to run our application and see output from our application.
3. This application will be database driven. Data for the sites we are parsing will be stored in the database. Will need someone or team that will have experience in good database design. The database will need to be easily maintainable and extensible.
4. This application will be run across different servers for different clients. This data will be stored in the database to be accessed by the application.
Desire a team/individual with experience in:
- Many different languages
- Database Design
- OOP and Design Patterns
- Process forking/threading
- Performance optimization
I will need to have access and the final rights and ownership to the source code.
Please reply with:
1. What technologies you will do this in
2. If you think you can develop the scraper/crawler to parse a page and submit to 2 pages in 1 second or less
25 freelancer bu iş için ortalamada 1164$ teklif veriyor
Quality is a great issue. Please have a look at our profile, reviews and portfolio ( [url removed, login to view] ) before selecting the winning bidder. Thanks, Tonmay
Hi I can develop the scrappers in perl, PHP and .net technologies. I have done many bots and scrappers using these technologies. Can send some sample apps also
I am Ranjana from Sisoft Info Solutions, Delhi NCR, India ([url removed, login to view]). We are a team of highly skilled professionals including project managers, domain experts, designers, analyst programmers and quality assuranc Daha fazlası
Hi, I can do this with .Net C# programming language, "Html Agility Pack" HTML parser and ASP.Net MySql/MSSql for a admin area. Regards... Dmitry
Saw you project and it looks like we can do this for you probably in less time. We are from the States so communication should not be an issue and we can PM you the tech proposal right away.