I'm looking for a spider, written in Java PHP or Perl (in that order of preference), that can take a selected friend in Myspace and spider their friends, and continue going to a specified depth level of friends out (5 users out, 10 users out, etc) collecting specific information along the way.
The spider will recognize when it's already stored a user, so it won't circularly add the same user over and over. However if the user that it has stored is an outlier friend (That hasn't been spidered yet for thier friends) it will update the friends field in the database.
Will spider/parse the pages of the username and his/her friends for:
1) Username of the user in question
2) http:// link to the user's myspace page.
3) An http:// link to the icon of the user.
4) ALL of the friends of that user, not just the first page of friends in a multipage list.
The spider will output the results to a Mysql database. The database will have fields:
UserName - The username of the person
LinkToPage - http:// link to the users page
Icon - http:// link to the users's icon.
Private - A flag to indicate if the user has a private page.
Friends - The friends of the person in a comma delimited field
Outlier - A flag to indicate if this user hasn't been spidered for thier friends yet because they are an outlier friend and not within the depth of the search.
Outlier friends (friends that are on the fringe of the depth of the search) will be created as new users in the database with all fields filled out except for friends. Friends field will only have the username of the user who was inside the depth of the search.
Will pay only through PayPal.
I apologize, if it's possible, I'd like to have the location of the user in an additional field of the Mysql DB.