How to use Simple Storage Service with Amazon EC2

Simple Storage Service is a pay as you go service from Amazon. There is no size limit for this service and we will be billed according to our usage.
There are many advantages for S3 over EBS since we don't need to associate the volume with any instance for access. S3 can be accessed through many services and APIs.

For creating S3,

Select the S3 option under AWS console. And create Buckets.

Now setup Access/Secret key pair for accessing the S3 buckets through APIs. For that,

Select the IAM (Identity and Access Management) option. Here we can create Users and add permissions to each Buckets.

 Note: The main AWS account Access/Secret key pair can be found and created from the security_credential option in the AWS console.
     
At IAM section we can create Users and access/secret keys and add permissions under user policies. This user policies allows us to restrict that user privileges to certain services only.

Sample user policy :- Access only inside 'sample-bucket' S3 Bucket.
 {
   "Version": "2012-10-17",
   "Statement": [
     {
       "Sid": "Stmt1395124947000",
       "Effect": "Allow",
       "Action": [
         "s3:*"
       ],
       "Resource": [
         "arn:aws:s3:::sample-bucket",
         "arn:aws:s3:::sample-bucket/*"
       ]
     }
   ]
 }

Now we can try accessing the S3 buckets from Ec2 instances. For Accessing S3 either we could use a command line tool or we can mount the S3 as a file system.

For S3 command line,

apt-get install s3cmd

s3cmd --configure (proper permissions should be needed)

New settings:
  Access Key: <access key>
  Secret Key: <secret key>
  Encryption password: <password>
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name:
  HTTP Proxy server port: 0

/root/.s3cfg (This file will have all the s3 related info)
 
 Common commands samples:-
    s3cmd mb s3://sample-bucket  (create bucket)
    s3cmd put file.txt s3://sample-bucket/ (copy file to s3)
    s3cmd put -r backup s3://sample-bucket/  (copy folder to s3)
    s3cmd ls s3://sample-bucket/ (list)
    s3cmd get s3://sample-bucket/file.txt (copy file to ec2)
    s3cmd del s3://sample-bucket/file.txt (delete file)
    s3cmd del s3://sample-bucket/backup (delete folder)
    s3cmd rb s3://sample-bucket(remove bucket)
    s3cmd du -H s3://sample-bucket/ (current size of bucket)

   
For mounting S3 Bucket as a filesystem under EC2.

apt-get install libssl-dev

Install the below package for fuse support, and load the module to the kernel.

tar -zxvf fuse-2.8.5.tar.gz
cd fuse-2.8.5
./configure
make && make install
modprobe fuse

ldconfig

Install s3fs,

tar -zxvf s3fs-1.74.tar.gz
cd s3fs-1.74
./configure
make && make install

Now, create the below file and add the access/secret keys as shown.

vim /etc/passwd-s3fs
accesskey:secretkey

chmod 640 /etc/passwd-s3fs

Try mounting the bucket by,

mkdir /backup

s3fs sample-bucket /backup

Now the sample-bucket is mounted under /backup folder and can be accessible as a file system.

Thanks for reading,

Cheers /-