Hadoop MapReduce Program Samples

Hadoop is an opensource implementation of MapReduce framework which is used for processing BigData by using parallel computation on distributed file system across cluster of commodity systems.

What MapReduce does is to split large data sets and send each process to seperate nodes in cluster which uses local computation and storage and combines the result to an output.

Since it uses local computation and storage (hdfs) of each node it is more reliable and fast.

Hadoop single and multi node installations can be followed from the below links respectively,

Single Node Setup

Multi Node Setup

Common commands..

List files -> hadoop fs -ls

Open file -> hadoop fs -cat filename

Copy file -> hadoop fs -copyFromLocal input.txt

Copy file -> hadoop fs -copyFromLocal input.txt /newfolder

Test your installation,

hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5

This will return the value of pi.

Hadoop Multi Node Setup

Hadoop is an opensource implementation of MapReduce framework which is used for processing BigData by using parallel computation on distributed file system across cluster of commodity systems.
What MapReduce does is to split large data sets and send each process to seperate nodes in cluster which uses local computation and storage and combines the result to an output.
Since it uses local computation and storage (hdfs) of each node it is more reliable and fast.

Here I will explain how to install hadoop on multi node clusters.

h1.hadoop-test.com (master/slave)
h2.hadoop-test.com (slave)

Follow the below steps in all nodes,

Hadoop framework is built in java.  So we need to install java (my case ) in /usr/lib/java/

My java version,

java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Oracle JRockit(R) (build R28.1.5-20-146757-1.6.0_29-20111004-1747-linux-ia32, compiled mode)

Add user and group for Hadoop,

addgroup hadoop
adduser --ingroup hadoop hduser

Hadoop Single Node Setup

Hadoop is an opensource implementation of MapReduce framework which is used for processing BigData by using parallel computation on distributed file system across cluster of commodity systems.
What MapReduce does is to split large data sets and send each process to seperate nodes in cluster which uses local computation and storage and combines the result to an output.
Since it uses local computation and storage (hdfs) of each node it is more reliable and fast.

Here I will explain how to install hadoop in a single node.

Hadoop framework is built in java.  So we need to install java (my case ) in /usr/lib/java/

My java version,

java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Oracle JRockit(R) (build R28.1.5-20-146757-1.6.0_29-20111004-1747-linux-ia32, compiled mode)

Add user and group for Hadoop,

addgroup hadoop
adduser --ingroup hadoop hduser

For hadoop to work, the user should need passwordless login, For that ..

Install Perl Module from Local Repository

Perl Module installation is Fun, but fun ends when we deal with hundreds of perl modules. One of the best things about Perl is that they are available on CPAN. But we can't trust that type of installation since it installs only new releases and you never know that version of module will bring a bug that breaks your application.

So I used to keep my version of perl modules and installs it one by one, but it was really time consuming. Then I found a tool called Pinto which allows us to create our own local perl repository.

Here I will explain how to install Pinto as a standalone application and create repository from it.


For installing Pinto issue the below command,

curl -L http://getpinto.stratopan.com | bash

source ~/opt/local/pinto/etc/bashrc


Create a Repository named 'repo' in the /opt folder,

pinto -r /opt/repo init


List the Modules inside the repository by the following command,

pinto -r /opt/repo list

PHP-FPM and Nginx configuration on linux

Here I will explain how to install php 5.5 under debian based system and configure apache and nginx webservers to get php support.

mkdir /opt/source/

cd /opt/source/

Download the latest php package from the below link,

wget http://ar2.php.net/distributions/php-5.5.7.tar.gz

Now install the dependencies,

apt-get install libexpat1-dev libssl-dev libjpeg62-dev libpng12-dev libt1-dev libfreetype6-dev fontconfig libxpm-dev libxml2-dev libfontconfig1-dev

Download and install curl package from this link,

wget http://curl.haxx.se/download/curl-7.34.0.tar.gz

tar -zxvf curl-7.34.0.tar.gz
cd curl-7.34.0

Nginx installation on linux

Nginx is a high-performance, open source web server and reverse proxy, extremly popular for handling and serving static contents.

Here I will explain how to install nginx webserver under debian based system.

First install the compliers and dependencies,

apt-get install gcc g++ make zlib1g-dev zlib-dev

mkdir /opt/source

cd /opt/source

Now download the latest nginx software,

wget http://nginx.org/download/nginx-1.4.4.tar.gz

For 3rd party headers more module support,

wget --no-check-certificate https://codeload.github.com/agentzh/headers-more-nginx-module/zip/master

mv master master.zip

unzip master.zip

MySQL 5.6 source compiled installation on linux

Here I will explain how to install MySQL 5.6 server under Debian based system.

First install proper compilers and dependencies.

apt-get install gcc g++ make bison libncurses5-dev

Download cmake,
wget http://www.cmake.org/files/v2.8/cmake-2.8.12.1.tar.gz

tar -zxvf cmake-2.8.12.1.tar.gz
cd cmake-2.8.12.1

./configure
make
make install

Now, download latest MySQL 5.6 package,

wget http://cdn.mysql.com/Downloads/MySQL-5.6/mysql-5.6.15.tar.gz

create users and group,
groupadd mysql
useradd -g mysql -d /var/mysql -s /bin/bash -m mysql

mkdir /var/mysql/src/

mv mysql-5.6.15.tar.gz /var/mysql/src/