How to use Simple Storage Service with Amazon EC2

Simple Storage Service is a pay as you go service from Amazon. There is no size limit for this service and we will be billed according to our usage.
There are many advantages for S3 over EBS since we don't need to associate the volume with any instance for access. S3 can be accessed through many services and APIs.

For creating S3,

Select the S3 option under AWS console. And create Buckets.

Now setup Access/Secret key pair for accessing the S3 buckets through APIs. For that,

Select the IAM (Identity and Access Management) option. Here we can create Users and add permissions to each Buckets.

 Note: The main AWS account Access/Secret key pair can be found and created from the security_credential option in the AWS console.
     
At IAM section we can create Users and access/secret keys and add permissions under user policies. This user policies allows us to restrict that user privileges to certain services only.

Logical Volume Management in linux

LVM is a userspace toolset that provide logical volume management on linux. With this tool we can create logical volumes, expand, delete, take snapshots of existing logical volumes etc.

For LVM to work properly, your kernel should support device-mapper libraries.

To install the latest LVM software, download the package from the link below,

wget https://git.fedorahosted.org/cgit/lvm2.git/snapshot/lvm2-2_02_105.tar.gz

tar -zxvf lvm2-2_02_105.tar.gz
cd lvm2-2_02_105

./configure
make && make install

If you are on ubuntu, you can simply install it by issuing the below command.

apt-get install lvm2

First we can create physical volumes that will be used to create the lvm

pvcreate /dev/sda8 /dev/sda9

Try Amazon EC2 for Free

Amazon Webservices is one of the top cloud computing service providers around the globe. They have introduced a Free Tier plan which helps to experience the Amazon cloud services at free of cost for one year. Here I will provide a detailed info about the Free Trier plan and pricing after the Free Tier duration.

Through Free Tier Plan we will be getting an AWS EC2 Micro instance named t1.micro !


[t1.micro] specification under Free Tier is listed below.

Duration - 365 Days

Linux Operating System - Ubuntu and Debian ( 64/32 bit)  AMI images.

Time -750 hours/month

Hardware- 613 MB with 32-bit and 64-bit platform support

15 GB data processing / month through Elastic Load Balancer.

30GB of EBS Storage, 2 million I/Os, and 1GB of snapshot storage

15 GB of bandwidth out.

1 GB of Regional Data Transfer.

Free One Elastic IP address associated with a running instance

ModSecurity Installation on Linux

ModSecurity is an open source web application layer firewall which primarily
supported Apache and now extends its security features to IIS and Nginx. Modsecurity
acts as a filter for every request supplied to the webserver which accepts/rejects
the request according to the rule sets.

Here I will explain how to install Modsecurity for Apache and configure OWASP core
rule sets on it.

Apache webserver installation can be followed from the link below,

Apache-Installation

Now download the latest Apache Modsecurity package,

wget https://www.modsecurity.org/tarball/2.7.7/modsecurity-apache_2.7.7.tar.gz

Install the missing dependencies if any,

apt-get install libxml2 libxml2-dev libxml2-utils pcre curl

tar -zxvf modsecurity-apache_2.7.7.tar.gz
cd modsecurity-apache_2.7.7

Hadoop MapReduce Program Samples

Hadoop is an opensource implementation of MapReduce framework which is used for processing BigData by using parallel computation on distributed file system across cluster of commodity systems.

What MapReduce does is to split large data sets and send each process to seperate nodes in cluster which uses local computation and storage and combines the result to an output.

Since it uses local computation and storage (hdfs) of each node it is more reliable and fast.

Hadoop single and multi node installations can be followed from the below links respectively,

Single Node Setup

Multi Node Setup

Common commands..

List files -> hadoop fs -ls

Open file -> hadoop fs -cat filename

Copy file -> hadoop fs -copyFromLocal input.txt

Copy file -> hadoop fs -copyFromLocal input.txt /newfolder

Test your installation,

hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5

This will return the value of pi.

Hadoop Multi Node Setup

Hadoop is an opensource implementation of MapReduce framework which is used for processing BigData by using parallel computation on distributed file system across cluster of commodity systems.
What MapReduce does is to split large data sets and send each process to seperate nodes in cluster which uses local computation and storage and combines the result to an output.
Since it uses local computation and storage (hdfs) of each node it is more reliable and fast.

Here I will explain how to install hadoop on multi node clusters.

h1.hadoop-test.com (master/slave)
h2.hadoop-test.com (slave)

Follow the below steps in all nodes,

Hadoop framework is built in java.  So we need to install java (my case ) in /usr/lib/java/

My java version,

java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Oracle JRockit(R) (build R28.1.5-20-146757-1.6.0_29-20111004-1747-linux-ia32, compiled mode)

Add user and group for Hadoop,

addgroup hadoop
adduser --ingroup hadoop hduser

Hadoop Single Node Setup

Hadoop is an opensource implementation of MapReduce framework which is used for processing BigData by using parallel computation on distributed file system across cluster of commodity systems.
What MapReduce does is to split large data sets and send each process to seperate nodes in cluster which uses local computation and storage and combines the result to an output.
Since it uses local computation and storage (hdfs) of each node it is more reliable and fast.

Here I will explain how to install hadoop in a single node.

Hadoop framework is built in java.  So we need to install java (my case ) in /usr/lib/java/

My java version,

java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Oracle JRockit(R) (build R28.1.5-20-146757-1.6.0_29-20111004-1747-linux-ia32, compiled mode)

Add user and group for Hadoop,

addgroup hadoop
adduser --ingroup hadoop hduser

For hadoop to work, the user should need passwordless login, For that ..

Install Perl Module from Local Repository

Perl Module installation is Fun, but fun ends when we deal with hundreds of perl modules. One of the best things about Perl is that they are available on CPAN. But we can't trust that type of installation since it installs only new releases and you never know that version of module will bring a bug that breaks your application.

So I used to keep my version of perl modules and installs it one by one, but it was really time consuming. Then I found a tool called Pinto which allows us to create our own local perl repository.

Here I will explain how to install Pinto as a standalone application and create repository from it.


For installing Pinto issue the below command,

curl -L http://getpinto.stratopan.com | bash

source ~/opt/local/pinto/etc/bashrc


Create a Repository named 'repo' in the /opt folder,

pinto -r /opt/repo init


List the Modules inside the repository by the following command,

pinto -r /opt/repo list

PHP-FPM and Nginx configuration on linux

Here I will explain how to install php 5.5 under debian based system and configure apache and nginx webservers to get php support.

mkdir /opt/source/

cd /opt/source/

Download the latest php package from the below link,

wget http://ar2.php.net/distributions/php-5.5.7.tar.gz

Now install the dependencies,

apt-get install libexpat1-dev libssl-dev libjpeg62-dev libpng12-dev libt1-dev libfreetype6-dev fontconfig libxpm-dev libxml2-dev libfontconfig1-dev

Download and install curl package from this link,

wget http://curl.haxx.se/download/curl-7.34.0.tar.gz

tar -zxvf curl-7.34.0.tar.gz
cd curl-7.34.0

Nginx installation on linux

Nginx is a high-performance, open source web server and reverse proxy, extremly popular for handling and serving static contents.

Here I will explain how to install nginx webserver under debian based system.

First install the compliers and dependencies,

apt-get install gcc g++ make zlib1g-dev zlib-dev

mkdir /opt/source

cd /opt/source

Now download the latest nginx software,

wget http://nginx.org/download/nginx-1.4.4.tar.gz

For 3rd party headers more module support,

wget --no-check-certificate https://codeload.github.com/agentzh/headers-more-nginx-module/zip/master

mv master master.zip

unzip master.zip

MySQL 5.6 source compiled installation on linux

Here I will explain how to install MySQL 5.6 server under Debian based system.

First install proper compilers and dependencies.

apt-get install gcc g++ make bison libncurses5-dev

Download cmake,
wget http://www.cmake.org/files/v2.8/cmake-2.8.12.1.tar.gz

tar -zxvf cmake-2.8.12.1.tar.gz
cd cmake-2.8.12.1

./configure
make
make install

Now, download latest MySQL 5.6 package,

wget http://cdn.mysql.com/Downloads/MySQL-5.6/mysql-5.6.15.tar.gz

create users and group,
groupadd mysql
useradd -g mysql -d /var/mysql -s /bin/bash -m mysql

mkdir /var/mysql/src/

mv mysql-5.6.15.tar.gz /var/mysql/src/