I need a Python script written that will connect to a remote site (with a username and password) and check to see if new data files are present. If new data files are found, they need to be downloaded to the local file system (with something like wget). I would like the script to read the remote site, username and password from a config file.
The files will be in tbz format, so once downloaded they need to be uncompressed and particular files extracted from the tarball. The files are large, so I don't want every file in the tar archive extracted.
Once extraction from the data files is complete, an already existing Python script needs to be invoked. That Python script will ingest the data files to a MySQL database.
The script needs to do error checking to ensure that file downloads complete and that there are no errors with the uncompress and tar processes. Ideally it would also ensure that the existing Python script it invokes returns cleanly.