Hadoop Single Node Setup

Hadoop is an opensource implementation of MapReduce framework which is used for processing BigData by using parallel computation on distributed file system across cluster of commodity systems.
What MapReduce does is to split large data sets and send each process to seperate nodes in cluster which uses local computation and storage and combines the result to an output.
Since it uses local computation and storage (hdfs) of each node it is more reliable and fast.

Here I will explain how to install hadoop in a single node.

Hadoop framework is built in java.  So we need to install java (my case ) in /usr/lib/java/

My java version,

java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Oracle JRockit(R) (build R28.1.5-20-146757-1.6.0_29-20111004-1747-linux-ia32, compiled mode)

Add user and group for Hadoop,

addgroup hadoop
adduser --ingroup hadoop hduser

For hadoop to work, the user should need passwordless login, For that ..



su - hduser

Now login as hduser via ssh,

ssh hduser@localhost

ssh-keygen -t rsa -P ""

cd /home/hduser/.ssh/

cat /home/hduser/.ssh/id_rsa.pub >> /home/hduser/.ssh/authorized_keys

quit;

Now login to that user via ssh again, It should enter without password.


Now disable ipv6,

As root,

Add the following to the last of the file,

vim /etc/sysctl.conf

 # disable ipv6
 net.ipv6.conf.all.disable_ipv6 = 1
 net.ipv6.conf.default.disable_ipv6 = 1
 net.ipv6.conf.lo.disable_ipv6 = 1

:wq

sysctl -p

The below file should return 1,

cat /proc/sys/net/ipv6/conf/all/disable_ipv6
1

or (Another method)

Disabling ipv6 only for hadoop (Described below)

vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh

export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true

Now we can start installing hadoop. Download the latest hadoop package from the link below,

wget http://apache.mirrors.tds.net/hadoop/core/stable/hadoop-2.2.0.tar.gz

tar -zxvf hadoop-2.2.0.tar.gz

mv hadoop-2.2.0 /usr/local/hadoop

chown hduser:hadoop /usr/local/hadoop/ -R


Now set the environment variables,

su - hduser

vim /home/hduser/.bashrc

#Java Specific#
export JAVA_HOME=/usr/lib/java/
export PATH=$PATH:$JAVA_HOME/bin/

#Hadoop Specific#
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_INSTALL=/usr/local/hadoop
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export YARN_HOME=$HADOOP_INSTALL

#ClassPath For java Compling#
export CLASSPATH=/usr/local/hadoop/share/hadoop/common/hadoop-common-2.2.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.2.0.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-annotations-2.2.0.jar


Test Java,

java -version

java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Oracle JRockit(R) (build R28.1.5-20-146757-1.6.0_29-20111004-1747-linux-ia32, compiled mode)

Configure hadoop to your requirement,

cd /usr/local/hadoop/etc/hadoop/

vim hadoop-env.sh

 export JAVA_HOME=/usr/lib/java/

 #For disabling ipv4 for hadoop#
 export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true


vim core-site.xml

 <configuration>
 <property>
  <name>fs.default.name</name>
  <value>hdfs://localhost:8080</value>
 </property>
 </configuration>



cp mapred-site.xml.template mapred-site.xml
chown hduser:hadoop mapred-site.xml

vim mapred-site.xml

 <configuration>
 <property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
 </property>
 </configuration>


vim hdfs-site.xml

 <configuration>
 <property>
  <name>dfs.replication</name>
  <value>1</value>
 </property>
 <property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/home/hduser/app/namenode</value>
 </property>
 <property>
   <name>dfs.datanode.data.dir</name>
   <value>file:/home/hduser/app/datanode</value>
 </property>
 </configuration>


vim yarn-site.xml

 <configuration>
 <property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
 </property>
 <property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
 </property>
 </configuration>

cd ~

Create name and data node folders,

mkdir -p app/namenode
mkdir -p app/datanode


Format the node with hadoop file system,

hdfs namenode -format

Start all HDFS services,

start-dfs.sh


Start all YARN services,

start-yarn.sh

Test Installation,

jps

22142 DataNode
30102 ResourceManager
22316 SecondaryNameNode
30770 Jps
30202 NodeManager
22039 NameNode


Test the working of hadoop,

su - hduser

hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5

Common commands..

List files -> hadoop fs -ls

Open file -> hadoop fs -cat filename

Copy file -> hadoop fs -copyFromLocal input output


Go through this link to remove the below warning (optional)

http://www.ercoppa.org/Linux-Compile-Hadoop-220-fix-Unable-to-load-native-hadoop-library.htm

WARNING  -  util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable


Web Interface,

http://<ip>:50070/dfshealth.jsp


For Multi Node hadoop setup, follow the below link.

Multi Node Setup

Example mapreduce programs,

Program Samples

Thanks for reading,

Cheers /-