AWS HDFS Replica Policy

- Build HDFS 1 master and 5 slaves

- Use articles dataset (ex. News huffpost) to upload corresponding files to HDFS

- Use google trends to get popularity score per each news category

- Using the popularity score to simulate read access for each article on HDFS for generating log files

Now we have for example a football article with a popularity score/ read access x, based on this score (how many times it got accessed) we categorize each article as HOT, WARM, COLD.

With the default hadoop HDFS replica policy 3x, I need to measure system performance and storage.

Then I need to modify existing files of the system to be like this HOT files get replicated 3x, WARM 2x and COLD 1x and if a new file is inserted to the system it should be added as HOT 3x. So we measure the performance and storage here.

Then I need a machine learning model trained on the articles dataset to be able to detect if a new file is added to the system shall It be considered as HOT, WARM or COLD. And we measure the system performance and storage again.

Hadoop Map Reduce Amazon Web Services

Proje NO: #36506143

Proje hakkında