RunningMapReduceExampleTFIDF hadoopclusternet This document describes how to run the TFIDF MapReduce example against ascii books. This project is for those who wants to experiment hadoop as a skunkworks in a small cluster (110 nodes) Google Pro
RunningMapReduceExampleTFIDF - hadoop-clusternet - This document describes how to run the TF-IDF MapReduce example against ascii books. - This project is for those who wants to experiment hadoop as a skunkworks in a small cluster (1-10 nodes) - Google Project Hosting
Introduction
The first application written with Clusternet was an example to produce weights for terms in a corpus of ascii books. The example is developed using 3 steps to transform the results in separate MapReduce Jobs. This example can actually be run in any Hadoop cluster.
TF-IDF Algorithm
The term frequency, inverted document frequency is a well-known information retrieval algorithm for giving weights for terms of documents. It is important to note that the algorithm must not consider stop words, and for this reason, we will be using filtering the terms in the input for the algorithm with the Google Stop words. The algorithm is described as follows:
weight(t,d) = tf * idf tf = # terms t in (d) / total # of terms in (d) = tf