I have a simple python program that can parse and extract data from one website and store the data into mongoDB and send an email to notify.
The source code is quite simple (core functions are just about 150 lines of code) and is based on python requests.
It only extracts certain data from fixed pages/links and will not follow, so it is a not a web-crawler. I have attached two of the core functions, as you can see, they are very simple.
And it is single threaded. It works quite well and stable but is a bit slow. As I need to monitor more pages, so I am planning to move it to Scrapy to speed up the program.
You don’t need to change the functions of web parsing, database or sending email. The focus is just to use the existing functions in Scrapy to improve the speed and performance.
If you need more information, please feel free to contact me. Thank you for your attention.
Dear Sir or Madam, I am a professional ETL Developer maintaining scrapers for several real estates websites for Poland-Based startup mining 45.000 records daily, uploading and updating our databases with deduplication Daha Fazla
Bu iş için 24 freelancer ortalamada £148 teklif veriyor
Hi If you care about performance speed. I can implement a "multi-threading" desktop application in C# which do the exact as python. Can work on a demo if you like. Thanks
I'm one of the best Scrapy experts here that's why I'm the right candidate for your project. I can provide you a Scrapy spider that will use your scraping logic as a base. I can complete it in less than a day for ju Daha Fazla
i am scrapy expert............................................................................................................................
Hello How are you My name is Xu i have full time and I can start to work immediately Please contact me and do let us discuss about your project Thanks for your posting
Hello I have a lot of experience in creating various web scrapers on various platforms. I carefully read the description of the project and view the attached file. This is a fairly simple task, so I can do it quickl Daha Fazla
Hi, Hope you are doing good today! I am a software developer and have experience with Threading and Data Structure and algorithms. I can help you to make your script faster. Please share your code details s Daha Fazla