Research off the shelf solutions for a back end batch system (scraping application)
- Durum: Closed
- Ödül: $170
- Alınan Girdiler: 2
- Kazanan: TheGuyver2040
The task is to research and suggest possible off the shelf solutions for a back end batch system (scraping application) that fulfill the following requirements:
Get competitors product information for branded shoes on a daily basis. Information like Product name, Product colorname, other product attributes example materials, list of competitors, Competitor prices (Black and discounted prices)).
Proposed Work Flow:
The system fetches a product list from our system with EAN codes (for identification). The system then uses the EAN in google shopping to fetch product information and prices per competitor.
Scraping engine to handle over 15.000 requests per day. The normal request per day will be 50-500. The information should be structured on model level and SKU / EAN level (ie we send over/fetch one size/EAN to scrape).
There will be a product feed into the system from our product system. Alternativly we want to be able to fetch the data through a feed / API.
The system should have an easy way to export / import information preferably using some API / feed.
There should be an API available. To fetch the scraped products. And possibly what products of interest to scrape.
Server / Hosting:
AWS on a Kubernetes server, if hosted on our servers. Otherwise software as a service is fine.
Performance is hard to estimate but roughly 50-500 daily requests but up to 15.000 when going through our total range of products.
A summary of the system you have found with a short explanation of its functionalities and how it can be used to fulfil our requirements
Tavsiye Edilen Beceriler
“It was great! ”