We are launching a new aggregation service for specialty financial data. The service will work just like Yodlee but serve different data related to financial transactions. All sites are secured with credentials, which we will store, and some have challenge questions.
We need a web scraping engine developed that can operate in real time and in batch. The engine must be able to scale to an enterprise level and be multi-threaded. We will store the user credentials so the engine must be able to cycle through different credentials to gather the data for that user and populate a database. Some of the sites are not straight-forward and use Ajax and other technologies that make scraping more difficult. For this project we will want five sites scraped initially but will add many more if the engine works properly.
In short, we are looking for web scraping experts that can develop not only the initial product but also help maintain it and add sites. We can provide some initial site credentials as samples if you like.
We have developed a proof of concept using the technologies listed below but we are open to whatever works.
[url removed, login to view] MVC
[url removed, login to view] Web API
HTML Agility Pack
26 freelancer bu iş için ortalamada 569$ teklif veriyor
I specialize in web scraping jobs like this. You can see on my profile that I have completed many similar jobs. I can have this project done properly and in a timely manner.
Hi sir, I am scraping expert, I have did too many similar projects, please check my feedback then you will know. Can you tell me more details? then I will provide demo data for you. Thanks, Kimi
I have 6 years of experience with data extracting and manipulating. The output for scraping process can be various from XML, CSV or database. I use Perl to do all of my scraping with LWP and HTMLUnit library.