I have had a programmer build me a web scraper that will pull data from one site and insert the data into mysql database. This includes text and pictures. It is build and working the only issue I have is...
I need the scraper to perform much faster work.I am pulling 4 million pieces of data and at this rate it will take me a year to complete this project. Also I must monitor the scraper as it stops and I must refresh it to start.
I already have the scraper programmed and inserting data...I need...a programmer to optimize the scraper and make it pull the data in a much quicker pace. I need to be done in a week or two maximum time.
Keep in mind that we are talking about text and pictures. So if you do not have experience and need to find a solution for this do not apply....If you understand what exactly what I am saying in terms of the 4 million listings needed in two weeks and that I ALREADY have the scraper build and you are expert in php curl and mysql then apply.
My budget is 30 to 60 dollars for this project as all the hard work is done regarding the design and implement of the scraper.I just need an expert to modify the steps it takes to move much faster as I said it is pulling 1k to 2 k listings a day if monitored. I need it to pull 4 million in two weeks. So for someone experienced this should be easy and take minimal time.
Also cannot send everyone a copy of the script to [url removed, login to view] you are experienced it will be as if some one asks a mechanic to change his [url removed, login to view] mechanic will say yes sure 2 hours. A non mechanic will say well let me look into it. I need someone that has experience as the project was kind of complex. Building a spider to pull data is one thing but knowing how to insert into mysql in a specified way is quite another. All that is done....just make it work faster.
At the present the script will stop and needs to be restarted I need it to run without need of monitoring and ten times the pulling speed so in essence this is a optimization job for already existing script.
If you can do this and have experience bid if you have 20 jobs already and wont have the time to finish please do not bid. If you have a idea of how you might be able to do this do not [url removed, login to view] you know exactly how to fix this then by all means bid.
Also when bidding send me a message explaining how you plan on doing the job as well as how long it will take to complete and at what rate you can get the scraper to pull data(keep in mind this includes text and pictures which the pictures are taking the most time).
Thank you for your consideration