write 2 python web crawlers using scrapy framwork to read wikipedia data

$30-250 SGD

Tamamlandı

İlan edilme:

yaklaşık 8 yıl önce

$30-250 SGD

Teslimde ödenir

Implement two web crawlers in python using the scrapy (1.0.5) framework 1) Get the full list of countries and territories from here [login to view URL] write a. Country / territory name b. Wikipedia URL c. Status (Membership) d. Dispute status e. Further information f. Polling date into a mariadb based database db schema: id(auto-incement), createdate(timestamp), all other fields are type text 2) Get the list of URLs in 1b) and crawl each one of the countries websites to extract information of each one of them: a. Abstract b. VCard data: i. Name ii. URL for flag iii. URL for emblem iv. Motto v. Anthem vi. URL to location on globe vii. URL to map viii. Capital(s) 1. Name 2. URL ix. Official language(a) 1. Name 2. URL x. Religion(s) 1. Name 2. URL xi. Demonym(s) 1. Name 2. URL xii. Government 1. Name 2. URL xiii. Establishment(s) 1. Name 2. Date xiv. Area 1. Total km2 2. Water km2 xv. Population 1. Total estimate 2. Date of counting / estimate xvi. GDP 1. Total 2. Per capita xvii. HDI index 1. Total 2. Rank xviii. Currency 1. Name 2. 3 letter code xix. TimeZone(s) 1. Name 2. Deviation from GMT 3. URL to timezone xx. Driving on left or right? xxi. Calling code(s) xxii. ISO code xxiii. Internet TLD(s) c. Date of polling Write the data above into a mariadb based database db schema: id(auto-incement), createdate(timestamp), all other fields are type text In case of multiple entries (e.g. languages) write a comma separated list in the db text field. Make sure the original text is comma free. Requirements: - Running on Ubuntu 14.04 lts (x64) standard installation (scrapy 1.0.5 installation [login to view URL]) - Mariadb 5.5 - Needs to run failsafe with correct results for all countries – especially for countries with several entries for capitol, timezones, languages etc Copyright: - All code belongs to employer Delivery: - 2 web crawlers with pipeline into MariaDB - Code with comments / documentation to be maintained - After uploading the full code, I will run it on my system and proof-read code before payment - No milestones, payment in full after successful test

Proje No: 10207457

Proje hakkında

13 teklif

Uzaktan proje

Son aktiviteden bu yana geçen zaman 8 yıl önce

Biraz para mı kazanmak istiyorsunuz?

E-posta adresi

Freelancer'da teklif vermenin faydaları

Bütçenizi ve zaman çerçevenizi belirleyin

Çalışmanız için ödeme alın

Teklifinizin ana hatlarını belirleyin

Kaydolmak ve işlere teklif vermek ücretsizdir

Seçilen:

@avishai445

Hi, I can do this job for you. Message me if you want me to get started on it ill do it right away .

$83 SGD 4 gün içinde

4,9

(4 değerlendirme)

2,7

13 freelancer bu proje için ortalama $284 SGD teklif veriyor

@SuiGenSolutions

Hello Sir, We've done a number of web scraping projects for our clients. We have scraped many directory websites including yellowpages, yelp and e-commerce websites including amazon, walmart etc and many more. We can deliver the data very quickly. We use proxies with IP rotation to avoid being detected as bots. We use python with wget, scrapy, urllib and other tools to fetch webpages and parsers like HtmlXPathSelector, regular expressions etc to extract information from the html. We have the right skill set to do this job effectively and within time and would like to discuss more about this opportunity. Looking forward to hear from you. Thanks, Shiv Agrawal SuiGen Solutions

$421 SGD 3 gün içinde

4,8

(88 değerlendirme)

6,6

@DanielVizcaya91

Hi there. I would be glad to help you out with this project. I am a professional data scraper, with experience creating easy to use scripts to extract data from the web. I can guarantee you an excellent job and deliver asap. However, I do not work without milestone creation, I only ask for them to be created but no payment in advance. Once the job is completed you can release them. Thanks, Daniel

$336 SGD 3 gün içinde

4,7

(94 değerlendirme)

6,9

@mananraja

Hi, I have read the description & would like to discuss.. I have good web scraping experience & reviews. & can develop web scraping scripts in Python & C# Hope we can discuss details..

$200 SGD 2 gün içinde

5,0

(137 değerlendirme)

6,2

@Verz1Lka

Hello! I'm web scraping expert and i can done your project in 3 days. I use python language and scrapy framework. My scripts works on windows, mac or linux, but linux is preferably. I can schedule scripts on server if it is required. I have more 200 finished projects (google scraping, facebook scraping, yellow pages, linkedinIn, amazon, webshops and other sites with lists of any items). I can export data into json, xml, csv (excel), or any database (mysql, mongodb, mssql, etc). Message me, if you have any questions!

$299 SGD 5 gün içinde

4,8

(110 değerlendirme)

6,5

@dabing1205

hi, I am an expert with python/scrapy, and have many scrapy project done here. Your project looks OK for me at first glance, please contact me to discuss more detailed requirement, Thanks

$222 SGD 5 gün içinde

5,0

(25 değerlendirme)

5,2

@lufte

Hi, I'm a frequent user of Scrapy and I've already written solutions that integrate with MySQL. I have a few questions regarding your project: 1- Are you looking to run the script periodically? If so, it should be aware that the data may already exist to avoid duplication or unique contraint violations. 2- If you plan to run it periodically, will you do it manually or do you need an additional solution to schedule the spiders? 3- The installation of the required software is part of the project or you will be doing that yourself? Thanks.

$200 SGD 10 gün içinde

5,0

(19 değerlendirme)

4,6

@WestEastDevs

Hey there ! We're 2 developers with vast and wide knowledge in Python and scripting specializing in Web scraping. We'll gladly do your project as it seems like something we can pull-off with a script. You can look in our profile for previous projects we've done regarding Web Scraping. Contact us for further details.

$277 SGD 3 gün içinde

5,0

(7 değerlendirme)

4,0

@marketiplace

I got 7+years work experience in Data Collection,Bulk Email Campaign,Excel VBA and Internet Research in IT companies here.I can do create crawler and scrap datas from Directory and yellowpages using C++,Python and Perl coding as per your requirements in excel with multiple ip rotations.I have dealt with US,UK and Australia companies President,Directors and Managers for web design and development projects successfully and I have Good Communication with writing skills.I am well versed in Internet,MS Office Applications and Phone Etiquette manners with latest Technologies.I can accept your payment terms.

$188 SGD 2 gün içinde

3,9

(6 değerlendirme)

4,4

@danielunderwood

I am a seasoned python programmer with past experience using scrapy. I can write this software for you and even test it on Ubuntu 14.04 LTS before you receive the code. I frequently comment my code and try to write code that is simple and easy to understand. I require no milestone until you are satisfied with the work. Please let me know if you have any questions. Regards, Daniel

$277 SGD 15 gün içinde