1) store data scraped into either dynamodb
2) listen to notifications to fire news scrape
3) being able to run cronjobs
4)being able to unify all time to one timezone
5) being able to use a proxy to prevent ip blocking ( will purchase and provide)
6) 3 lambdas: *one for news,
*one for stats,table,fixtures,googlesheet
*another for team fixtures
*The required are 8 apis end points,
*6 simple html page scripts from scratch and one to be duplicated with different url
*one rss feed listener, and one to be duplicated with different url
*two apis response scraping scripts
the project is divided into these steps:
1)scripts: there are 5 rss feeds created, there is one should be created, and another one duplicated with different url. All these rss feeds should run in one lambda every 5 minutes and if there are news urls open the html content and scrape it.
2) 4 simple html pages scraping to run if there were new news posted in the rss feeds
3) db table is already done only missing isLocal coloumn
4) 2 apis, one with paging
*Standing:script,db is created ( should run every 10 minutes from 1 pm to 12 am)
1)1 api to create
2) add a column to detect if the team position increased(1) or decreased(-1) ,or remained the same(0) from te previous scrape insert
*Fixtures: scraping api is done, db is done
need to create one api end points
Stats html page, googlesheet scrape and team fixtures will be done from scratch
start job offer with RD07
check pdf file for more details
preferably to use python for scraping as most scripts are done with python
Bu iş için 23 freelancer ortalamada $558 teklif veriyor
https://www.freelancer.com/projects/php/Extracting-Data-Software/ https://www.freelancer.com/projects/php/PCB-Quotation-Tool/ done similar works lets chat
Hi. How are you? I have heavy experiences in aws lamda, aws elastic beanstalk, aws securitygroup ,ec2 etc, aws rds db.... Let's discuss detail in private chat. Thank you. From smolina A. :)
Hello! I am a python developer. I looked at your project and it seems interesting. I have all necessary skills required for this project. Ping me to discuss in detail.
Hello. I have good skills in Amazon Web Services, Python, Web Scraping. I have checked your project description carefully and i can do it. I hope to work with you. Thank you.
Hi, I can do what you describe, scraping, storing the data, etc. I am currently a bit overwhelmed, hence the 2 week period. I am one of the best python developers, as well as web scrapers out there. Cheers!