Scraping a web site archives

There is a web site with newspapers archives. I need for this web site : 2. To get the list of newspapers titles 1. To get all the newspapers content in txt The script (it can be two scripts): in a first pass: - need to get meta-data for each newspapers (title, location, notes) and save it as a csv file (tabulation separated) in a second pass: - get all the content of each newspaper in txt combined with a csv file (tabulation séparated) with title, date, url, name of the file. - need to be able to be relaunched (if there is a crash or new newspapers) and not download again the previous newspapers - have a check control (in case the connection is down and there is a trouble on the target web site) - have warnings - not overload the target web site If needed i can provide an existing perl script that do similar job and can be adapted. The script use several regex with while loops. This is a test for one web site. If the project is achieved properly there is larger job with several web site.