Hadoop Multi Node Setup

Hadoop is an opensource implementation of MapReduce framework which is used for processing BigData by using parallel computation on distributed file system across cluster of commodity systems.
What MapReduce does is to split large data sets and send each process to seperate nodes in cluster which uses local computation and storage and combines the result to an output.
Since it uses local computation and storage (hdfs) of each node it is more reliable and fast.

Here I will explain how to install hadoop on multi node clusters.

h1.hadoop-test.com (master/slave)
h2.hadoop-test.com (slave)

Follow the below steps in all nodes,

Hadoop framework is built in java.  So we need to install java (my case ) in /usr/lib/java/

My java version,

java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Oracle JRockit(R) (build R28.1.5-20-146757-1.6.0_29-20111004-1747-linux-ia32, compiled mode)

Add user and group for Hadoop,

addgroup hadoop
adduser --ingroup hadoop hduser


Disable ipv6,

vim /etc/sysctl.conf

Add the following to the last of this page,

 # disable ipv6
 net.ipv6.conf.all.disable_ipv6 = 1
 net.ipv6.conf.default.disable_ipv6 = 1
 net.ipv6.conf.lo.disable_ipv6 = 1

:wq

sysctl -p

The below file should return 1,

cat /proc/sys/net/ipv6/conf/all/disable_ipv6
1

or (Another method)

Disabling ipv6 only for hadoop (Described below)

vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh

export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true

Now we can start installing hadoop. Download the latest hadoop package from the link below,

wget http://apache.mirrors.tds.net/hadoop/core/stable/hadoop-2.2.0.tar.gz

tar -zxvf hadoop-2.2.0.tar.gz

mv hadoop-2.2.0 /usr/local/hadoop

chown hduser:hadoop /usr/local/hadoop/ -R


Now set the environment variables,

su - hduser

vim /home/hduser/.bashrc

#Java Specific#
export JAVA_HOME=/usr/lib/java/
export PATH=$PATH:$JAVA_HOME/bin/

#Hadoop Specific#
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_INSTALL=/usr/local/hadoop
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export YARN_HOME=$HADOOP_INSTALL

#ClassPath For java Compling#
export CLASSPATH=/usr/local/hadoop/share/hadoop/common/hadoop-common-2.2.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.2.0.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-annotations-2.2.0.jar

############

Change the hostname of each node respectively,

1. h1.hadoop-test.com (master/slave)
2. h2.hadoop-test.com (slave)

Edit the hosts files of each node and map both ips (not 127.0.0.1) to hostnames.

The time on each node should be same.

For hadoop to work, the user should need passwordless login, For that ..

On h1.hadoop-test.com

su - hduser

Now login as hduser via ssh,

ssh hduser@h1.hadoop-test.com

ssh-keygen -t rsa -P ""

cd /home/hduser/.ssh/

cat /home/hduser/.ssh/id_rsa.pub >> /home/hduser/.ssh/authorized_keys

quit;

Now login to that user via ssh again, It should enter without password.


On h2.hadoop-test.com (do the same)

su - hduser
ssh hduser@h2.hadoop-test.com
ssh-keygen -t rsa -P ""
cd /home/hduser/.ssh/
cat /home/hduser/.ssh/id_rsa.pub >> /home/hduser/.ssh/authorized_keys
quit;
Now login to that user via ssh again, It should enter without password.



Now copy public key of both to authorized_keys of other host.

Now try SSH,

On h1.hadoop-test.com

su - hduser

ssh hduser@h2.hadoop-test.com (passwordless login)



On h2.hadoop-test.com

su - hduser

ssh hduser@h1.hadoop-test.com (passwordless login)


Configure both clusters like below ..

cd /usr/local/hadoop/etc/hadoop/

vim hadoop-env.sh

 export JAVA_HOME=/usr/lib/java/

 #For disabling ipv4 for hadoop#
 export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true


vim core-site.xml

 <configuration>
 <property>
  <name>fs.default.name</name>
  <value>hdfs://h1.hadoop-test.com:8080</value>
 </property>
 </configuration>



cp mapred-site.xml.template mapred-site.xml
chown hduser:hadoop mapred-site.xml

vim mapred-site.xml

 <configuration>
 <property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
 </property>
 </configuration>


vim hdfs-site.xml

 <configuration>
 <property>
  <name>dfs.replication</name>
  <value>2</value>
 </property>
 <property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/home/hduser/app/namenode</value>
 </property>
 <property>
   <name>dfs.datanode.data.dir</name>
   <value>file:/home/hduser/app/datanode</value>
 </property>
 </configuration>


vim yarn-site.xml

 <configuration>
 <property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
 </property>
 <property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
 </property>
 <property>
   <name>yarn.resourcemanager.hostname</name>
   <value>h1.hadoop-test.com</value>
   <description>The hostname of the Resource Manager</description>
 </property>
 </configuration>

cd ~

Create name and data node folders,

mkdir -p app/namenode
mkdir -p app/datanode

Now in master only,

vim /usr/local/hadoop/etc/hadoop/slaves

 h1.hadoop-test.com
 h2.hadoop-test.com

Format with hadoop file system,

hdfs namenode -format

Start all HDFS services,

start-dfs.sh


Start all YARN services,

start-yarn.sh

Required services in slave will start automatically.

Test Installation (master)
jps

22142 DataNode
30102 ResourceManager
22316 SecondaryNameNode
30770 Jps
30202 NodeManager
22039 NameNode

Test Installation (slave)
jps

10111 DataNode
3003 NodeManager
4289 Jps

Test the working of hadoop (In master),

su - hduser

hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5

Common commands..

List files -> hadoop fs -ls

Open file -> hadoop fs -cat filename

Copy file -> hadoop fs -copyFromLocal input output


Go through this link to remove the below warning (optional)

http://www.ercoppa.org/Linux-Compile-Hadoop-220-fix-Unable-to-load-native-hadoop-library.htm

WARNING  -  util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable


Web Interface,

http://h1.hadoop-test.com:50070/dfshealth.jsp


For Single Node hadoop setup, follow the below link.

Single Node Setup

Example mapreduce programs,

Program Samples

Thanks for reading,

Cheers /-