Hadoop MapReduce Program Samples

Hadoop is an opensource implementation of MapReduce framework which is used for processing BigData by using parallel computation on distributed file system across cluster of commodity systems.

What MapReduce does is to split large data sets and send each process to seperate nodes in cluster which uses local computation and storage and combines the result to an output.

Since it uses local computation and storage (hdfs) of each node it is more reliable and fast.

Hadoop single and multi node installations can be followed from the below links respectively,

Single Node Setup

Multi Node Setup

Common commands..

List files -> hadoop fs -ls

Open file -> hadoop fs -cat filename

Copy file -> hadoop fs -copyFromLocal input.txt

Copy file -> hadoop fs -copyFromLocal input.txt /newfolder

Test your installation,

hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5

This will return the value of pi.

------------------------------------------------------------------------------------------------

Hadoop Inbuilt example for wordcount,

vim input.txt

 linuxcrafts is cool
 linuxcrafts is best
 All about linux

hadoop fs -copyFromLocal input.txt

hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount input.txt /output

hadoop fs -ls /output

hadoop fs -cat /output/part-r-00000

 All     1
 about   1
 best    1
 cool    1
 is      2
 linux   1
 linuxcrafts  2

hadoop fs -copyToLocal /output/part-r-00000 output.txt

cat output.txt

 All     1
 about   1
 best    1
 cool    1
 is      2
 linux   1
 linuxcrafts  2

------------------------------------------------------------------------------------------------

Above example using our own java file,

wget https://sites.google.com/site/linuxcrafts/WordCount.zip

mkdir class

javac -cp $CLASSPATH -d class/ WordCount.java

jar cvf WordCount.jar -C class/ .

hadoop jar WordCount.jar org.apache.linuxcrafts.WordCount input.txt /output1

hadoop fs -cat /output1/part-r-00000

-------------------------------------------------------------------------------------------------

Another example to get output file with pipe seperated values from all dictionaries,

wget https://sites.google.com/site/linuxcrafts/Dictionary.zip

cat French.txt >> fulldictionary.txt

cat Italian.txt >> fulldictionary.txt

cat Spanish.txt >> fulldictionary.txt

mkdir classes

hadoop fs -copyFromLocal fulldictionary.txt dictionary.txt

javac -cp $CLASSPATH -d classes/ Dictionary.java

jar cvf Dictionary.jar -C classes/ .

hadoop jar Dictionary.jar linuxcrafts.Dictionary

hadoop fs -cat out-put/part-r-00000

-------------------------------------------------------------------------------------------------

Single node,

time hadoop jar Dictionary.jar linuxcrafts.Dictionary

real    2m16.435s
user    0m14.941s
sys     0m13.285s

Multi node,

time hadoop jar Dictionary.jar linuxcrafts.Dictionary

real    0m46.264s
user    0m16.456s
sys     0m12.192s

Thanks for reading,

Cheers /-