MiniPy - Python web scraping

Write a standalone Python program to Automate the process of scraping some data from different web URL sources to aggregate the content as it's updated. The program will launch a spider(s) each with a specific name, URL, table, class title, href, description, and category. Once data is aggregated into a list - we want to make text edits. Next, the output "edited" data format will be parsed into json files and google sheets as a backup. Then present the data by category - in a new list area it in a flask web page that will be published online. Url test sites will be provided. Coder will only use PyQt5 or Tkinter for the GUI. We will use Anaconda (python) 64/bit and Sublime Text 3 or Atom so must also work with this.

The program is specifically for when the content structure changes. Must be able to launch a new spider with the category name and save it as a mini python script with the "category" name. The, we must be able to change the URL and data to scrape another source. Must be able to run multiple scripts (mini-programs) to scrape.

** Budget $100 Max

** Timeline to complete - deadline 7/days Max

** Auto responses will be deleted - Use the word "MiniPy" when you bid.



1. scrape every _ five days (setting for days) 5, 10, 15, 20 etc

2. collect data

a. Use 5 random headers, and 5 random user agents

b. rest and close every time once data is collected.

3. parse or format data into:

a. json data to be used

b. google sheet (backup)

4. Eliminate duplicates for all spider data collected

5. Edit - Add replacement words in data for all spiders

6. Output the data in two formats

Edited data:

a. format #1 output to json = formatted data with replacement words to a json web file

b. format #2 output data format data with replacement words to a Google sheet by category.

7. Web List

a. by category

b. is dynamic content

scheduling python script

[login to view URL]


[login to view URL]

Interested Coders/Programmers:

a. Only provide examples of your WordPress work

b. We will supply a screen shot/images and files to coders we are interested in working with.

c. Budget Requirements: Not more than ($100 max) will be accepted to complete this project

d. You must have more than 10 successful projects to be considered for this project, along with favorable ratings.

e. Project timeline - 7 days max coders deadline

*** Bids over the budget will be automatically deleted.

Beceriler: Django, Javascript, Python, Yazılım Mimarisi, Web Scraping

Daha fazlasını gör: python web scraping, java python web scraping, perl python web scraping, web scraping python beautifulsoup, web scraping with python: collecting data from the modern web, python 3 web scraping, web scraping with python pdf, beautifulsoup tutorial python 3, web scraping with python amazon, web scraping with python o'reilly pdf, web scraping python tutorial, downloading files python web scraping, python web scraping scripts, python web scraping framework, python web scraping css, python web scraping script, python web scraping ajax content, python web scraping javascript, python web scraping form, forms python web scraping

İşveren Hakkında:
( 76 değerlendirme ) West Fargo, United States

Proje NO: #15537037



I have been working as a software developer for more than two years on python scripting and having good experience in web scraping using scrapy framework and selenium and having good experience in flask framework. Mini Daha Fazla

%selectedBids___i_period_sub_7% gün içinde 100%project_currencyDetails_sign_sub_9% %project_currencyDetails_code_sub_10%
(7 Değerlendirme)

Bu iş için 11 freelancer ortalamada $149 teklif veriyor


Hi sir, I am scraping expert, I have did more than 350+ scraping project, please check my feedback then you will know. Can we discuss more details about this project? then I will provide example data/scr Daha Fazla

in %bids___i_period_sub_35% gün içinde130%project_currencyDetails_sign_sub_37% %project_currencyDetails_code_sub_38%
(273 Değerlendirme)

Dear Sir, I specialize in web development, web scraping and data mining projects. Please open chat with me and lets discuss in more details about the project. Many thanks!!! Relevant Skills and Experience PHP-cURL, Daha Fazla

in %bids___i_period_sub_35% gün içinde100%project_currencyDetails_sign_sub_37% %project_currencyDetails_code_sub_38%
(67 Değerlendirme)

please ignore the bid amount we will discuss the price later in the chat after we have discussed the project thoroughly Relevant Skills and Experience: . Proposed Milestones: 190 - . Hi pink3d! please inbox me to discu Daha Fazla

in %bids___i_period_sub_35% gün içinde190%project_currencyDetails_sign_sub_37% %project_currencyDetails_code_sub_38%
(31 Değerlendirme)

Hello. Hope you are doing well. I am a professional bot developer and have 10+ years experience. [login to view URL] [login to view URL] [login to view URL] Waiting for your positive response to make a grea Daha Fazla

in %bids___i_period_sub_35% gün içinde100%project_currencyDetails_sign_sub_37% %project_currencyDetails_code_sub_38%
(39 Değerlendirme)

A proposal has not yet been provided

in %bids___i_period_sub_35% gün içinde277%project_currencyDetails_sign_sub_37% %project_currencyDetails_code_sub_38%
(12 Değerlendirme)

A proposal has not yet been provided

in %bids___i_period_sub_35% gün içinde222%project_currencyDetails_sign_sub_37% %project_currencyDetails_code_sub_38%
(4 Değerlendirme)

Dear,Sir I have experienced in developing MinPy,Python and JavaScript,Web Scraping for 5 years. I will work very hard and best for you. Best Regards

in %bids___i_period_sub_35% gün içinde100%project_currencyDetails_sign_sub_37% %project_currencyDetails_code_sub_38%
(8 Değerlendirme)

Greetings! from your project description I understood that you are looking for python experts to develop a script for you to scrape from [login to view URL] reply so that we can discuss further. Relevant Skills and Experience Daha Fazla

in %bids___i_period_sub_35% gün içinde155%project_currencyDetails_sign_sub_37% %project_currencyDetails_code_sub_38%
(0 Değerlendirme)

i'm expert in Automation + data scraping field and kindly contact me for this project. thanks Relevant Skills and Experience i have fiver year experience you can check review about scraping on below link: https://www. Daha Fazla

in %bids___i_period_sub_35% gün içinde111%project_currencyDetails_sign_sub_37% %project_currencyDetails_code_sub_38%
(1 Yorum)

Hi, I have gone through your project description and I'm interested to work with you. I'm very passionate about my work and give good quality of work to client. Relevant Skills and Experience I have more than 5+ year Daha Fazla

in %bids___i_period_sub_35% gün içinde155%project_currencyDetails_sign_sub_37% %project_currencyDetails_code_sub_38%
(0 Değerlendirme)