Web Scraping to Database

I need a PHP or Python script which will be run periodically to scrape the data I describe in this project and store it locally in a MySQL database running in a typical LAMP environment on a Debian 11 server. The script should start by emptying the table to be used (TRUNCATE), the populate the table only with the results of the new data scraping described in this project.

The source data is on a public county GIS mapping server and is strictly limited to the owners of property in the subdivision for which this script is being written.

Currently, we do the very tedious work of manually updating our data. To get to the data I need manually, I use these steps:

1: Open web page: [login to view URL]

2: Use the Search Field and enter "Upper Little Warm Spring Creek"

3: On the search results page, click "Select All", then click "Map Selection"

4: The resulting page is shown in the attached screenshot and the purple circles are where the data is--as explained below. PLEASE NOTE that the number of results may change from time to time, so we cannot simply use the URL that results from the search.

5: Use the "Copy to Clipboard" button on the web page to obtain the primary data.

6: Use the "Property Detail" link next to each of the property parcel entries to obtain further data to be added to the primary data.

I need this to be fully automated so that simply accessing the script will scrape and update all data in my database.


After getting to the correct data set, the results of the "Copy to Clipboard" button (circled in purple on attached screenshot) will be 23 tab delimited values for each record, from which I need the first 14 columns which need to be populated into a MySQL database table using the following column names:

Parcel, Parent, Account, TaxID, Plat, Lot, Owner, Mail_Street, Mail_CSZ, St_Addr, Deed, Location, Tax_Class, Acerage

Once the main data is imported, then every record's "Property Details" link needs to be accessed to get more data for the records already collected (2 "Property Details" links circled in purple on attached screenshot). From each "Property Details" record, I need the following additional columns for each of the records created from the "Copy to Clipboard" data:

MV_Year, MV_Total, MV_Land, MV_Improve, AV_Year, AV_Total

These values are found by searching the result from each Property Details link for the FIRST line that contains "Market Value" (MV) and the FIRST line that contains "Assessed Value" (AV) extracting the data from lines similar to this example:

2021 Market Value: $ 450,273 ($ 83,886 Land + $ 366,387 Improvements)

2021 Assessed Value: $ 42,776

results in these values:

MV_Year = 2021

MV_Total = 450273

MV_Land = 83886

MV_Improve= 366387

AV_Year = 2021

AV_Total = 42776

As a result of both steps each line of the database table should include these columns:

Parcel, Parent, Account, TaxID, Plat, Lot, Owner, Mail_Street, Mail_CSZ, St_Addr, Deed, Location, Tax_Class, Acerage, MV_Year, MV_Total, MV_Land, MV_Improve, AV_Year, AV_Total

I am available to explain further as needed and to participate in testing.

Please do NOT ask me for access to my server as this script needs to run in most any current Linux-based LAMP environment

Desired deliverables:

1) MySQL "CREATE TABLE" script to set up the database table I describe in this project.

2) PHP or Python script to accomplish the tasks described in this project.

Great performance on this project will likely lead to further projects from me as well.

Beceriler: Python, Web Scraping, MySQL, PHP

Daha fazlasını gör: web site fashion accessories store, access database hardware store, web forms database php, upload data web page database, web template database entry sql server, free php web templates database sql hosting nuke, web programmer database, web designing database electronics flash photoshop, web driven database quote, convert static web site database, sales database computer store, web interface database application, web design database photoshop, web albums database iis, scraping web site database listings, web scraping database, web scraping import magento store, web scraping google play store reviews

İşveren Hakkında:
( 35 değerlendirme ) Dubois, United States

Proje NO: #32283207


(1179 Değerlendirme)

Bu iş için 15 freelancer ortalamada $242 teklif veriyor


Hello ! I am professional Python coder. I am familiar with Python and I have a lot of work experiences in Web Scraping, Python, PHP and MySQL. I can start right away. I want to discuss for this project in detail. Ple Daha Fazla

$150 USD in 5 gün içinde
(32 Değerlendirme)

Hi , I have read your requirements for your project " Web Scraping to Database " and understood them to a great deal but it would be better if you initiate chat and we discuss details. We have full skills to develop y Daha Fazla

$250 USD in 72 gün içinde
(19 Değerlendirme)
(36 Değerlendirme)

Hi there scraping expert here I've scraped 45000+emails in here(If you want to see, I can show you) I'm quite familiar with Selenium and beautifulsoup in python I've already scraped many shopping sites before if you c Daha Fazla

$200 USD in 3 gün içinde
(13 Değerlendirme)

Hello there, I can use Selenium to help you scrape the data you need :) It will be ready within 24 hours hopefully and you can use it to replace the manual work you are doing. I can also make it in PHP depends on you Daha Fazla

$150 USD in 2 gün içinde
(7 Değerlendirme)

Hello. How are you? I am excellent web scraper and I have nice experience in web scraping by using python and node.js, etc. So I am sure I can give you perfect results. I hope to discuss via private chat. I can start w Daha Fazla

$140 USD in 3 gün içinde
(8 Değerlendirme)

Hello I am a senior developer and I have full experiences in this field. Web scrapping is my major skill and I am very interested in your job. There is no problem about the create database table. There are only one pro Daha Fazla

$1200 USD in 7 gün içinde
(3 Değerlendirme)

⭐⭐⭐⭐⭐ 5-Stars Rating for all of my Projects. Hi Dear, I hope you are doing well and safe from COVID-19. I am working for 14 years into Full-Stack developer. I already worked on similar projects to this one and I can fi Daha Fazla

$140 USD in 7 gün içinde
(1 Yorum)

How are you?. I have read your description carefully and very interesting in your project. I'm a web scraping developer with a decade of experience developing software across a wide variety of platforms. My focus now i Daha Fazla

$140 USD in 7 gün içinde
(3 Değerlendirme)

Hi, I'm Goli, I read your description and saw the attachment. And I can complete your project with a PHP script that automatically saves data in the database. Please request a chat to negotiate the price and time fra Daha Fazla

$245 USD in 5 gün içinde
(3 Değerlendirme)

Dear Client Thank you for your project. I've just checked your job description carefully. I'm senior developer with 9+ years of Python. By using Python, I developed AI engine, BOT, Web Scraping Tools, Web Searching Too Daha Fazla

$250 USD in 7 gün içinde
(1 Yorum)

***Scraping Expert*** Hi, pleased to apply for this project. I read your job description and you are seeking a scraping expert. I have vast experience in web scraping with python(Scrapy), C#, JavaScript(Node.js) for mo Daha Fazla

$140 USD in 7 gün içinde
(2 Değerlendirme)

Hello! I'm glad you reached out! We have experience✅, professional staff✅ and we are highly motivated! ✅ What else do you need? ✅ We have a lot of experience with: MySQL, Web Scraping, PHP and Python ✅ If you act Daha Fazla

$223 USD in 26 gün içinde
(0 Değerlendirme)

Hey Mate! I am a web scraping and automation expert. Very familiar in PYTHON and Node.js. It is an enough experiences for me like Web Scraping, PHP, MySQL and Python. I can start right away. I am coding from home. I Daha Fazla

$100 USD in 3 gün içinde
(0 Değerlendirme)