I'm a private individual looking to automate the collection of my personal wearable sensor data. This data is logged automatically by the wearable device to the company's website, and then they have a dashboard, behind a login, where the data is available as JSON format.
This project will consist of:
1. Web scraping the data after logging in
2. Storing the data in a database long term
3. retrieving the data by date range, reformatted, on demand, via all three output methods below:
1. To google sheets
2. To CSV Text
3. In JSON
I would like advice on how to automate this on a daily scheduled basis with existing hosted providers, such as Integromat, Zapier, Apify, Google app script, Google sheets or similar, that provide free access for private hobbyists.
I do not have a server to host any code on, so if would like to demonstrate a solution that is not integrated in the previously mentioned type of automation platform (apify, zapier, etc) (i.e. requires running scripts locally), please make sure to show how to run it scheduled in a free (or less than USD 5$/month) commercial online service that has point-and-click web user interface for scheduling. (if proposing use of a VPS, must not use Windows)
You must demonstrate that the web login and scraping actions will function (not be blocked by the host as robot or scraper) while it is running on the platform that you suggest.
Provide all source code with an MIT style software license ( [login to view URL] ) , or we can do the Intellectual Property Agreement
Please indicate in your proposal the following things:
* Where you would propose to host such a tool (multiple ideas ok)
* What language(s) you suggest that it be written in (multiple ideas ok)
* What kind of database
* How you would prefer to interface with Google Sheets (custom macro, Google Apps Script, using a third party service that already provides API authentication with Google Oauth2 like Apify, etc.)
* Have you written a regularly executing web scraping LOGIN script that ran on a commercial hosting provider that didn’t get blocked by the hosts as a robot?
Details THESE ARE PROVIDED IN A FILE AS WELL FOR READING CLARITY:
These importing and processing tasks should happen automatically, once daily.
1. Web scraping:
1. Login to: https: //[login to view URL] using given credentials
2. Request the given date range of data
3. Receive the JSON data result
1. Data storage:
1. De-duplicate any data that is already present in the database
2. Store in online database
1. Retrieve data on demand:
1. Data output formats:
1. Direct populate a google sheet/cells
1. Sheet that function is called from, or via some other web API site such as Apify, zapier, integromat etc that can access google sheets.
1. CSV text
1. User Interface
1. Adjusting settings at this phase of the project should be limited to simple OPTIONAL variable inputs loaded at runtime
1. Possible examples:
1. passing arguments to a function in a function call: functionname(20190321,12,5)
2. Reading from a preferences file
3. Reading from a set of google sheet cells
1. Input variables. Variables should be optional, and if not provided, default to the value provided here:
1. Starting and ending dates for retrieved data
1. Format: (same as imported date/time format)
1. YYYY-MM-dd HH:mm:ss
2. Example: 2019-03-17 00:17:32
1. Default to the current date in the US/Los Angeles time zone. (Not the current date GMT)
1. Error handling:
1. Return numbered english language error messages
1. Unparsable date/time range
2. Other errors as needed
3. Use your best judgement on the format of the machine readable error messages.
1. Log errors to file or database
Hello sir, I am a Python developer with huge experience in Web Scraping. For the presented project, first I took a look at the HTTP response from healbe. It looks easy to automate a login and retrieve data, I've don Daha Fazla
Bu iş için 20 freelancer ortalamada $470 teklif veriyor
hi there , i have scrap with script which site have login let me know more detail like url credential i didn't find it . waiting for response regards saleem
I am confident I am the right candidate for this project as I have done many similar projects in the past. With years of experience in this field, I believe this project will be very easy for me.