Steps For Single Node Cluster With Hadoop on AWS

Steps For Single Node Cluster With Hadoop on AWS, in this Tutorial one, you can learn Steps For Single Node Cluster With Hadoop on AWS. Are you the one who is looking for the best platform which provides information about Steps For Single Node Cluster With Hadoop on AWS? Or the one who is looking forward to taking the advanced Certification Course from India’s Leading AWS Training institute? Then you’ve landed on the Right Path.

The Below mentioned Tutorial will help to Understand the detailed information about Steps To Launch An Amazon EC2 Instance, so Just Follow All the Tutorials of India’s Leading Best AWS Training institute and Be a Pro AWS Developer.

To set up a single node cluster with Hadoop in EC2 instance on AWS, we have to first login to EC2 instance in AWS Management Console by using Login Id and Password

After login to EC2 click on Instances option available at the left side of the dashboard, you will see an instance state in stopped mode

Start instance by right-clicking on stopped instance > instance state > start > pop-up will appear asking for to start to click YES

After clicking Yes instance state will start and in running state

Open PuTTY tool where you saved and run it, PuTTY Configuration window will open as given below:

• Copy Public DNS address in the Host Name field
• Expand SSH > click Auth > Browse for .ppk file > click open
• The pop-up will appear to ask permission click, yes, then you will get in Ubuntu Linux instance window in EC2

Type ubuntu > Enter

Now start to set up a single node cluster by downloading Jdk 1.8.0 and Hadoop 2.9.0 version

Log in as the root user to install base packages

sudo su

sudo apt-get update

Type exit from root user in the terminal and Enter 

Download jdk 1.8.0 using wget command

wget -c –header “Cookie: oraclelicense=accept-secure backup-cookie” http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.tar.gz

Untar downloaded file(tar.gz) format by using

tar -xvf  jdk-8u131-linux-x64.tar.gz

Download the latest stable Hadoop using wget from one of the Apache mirrors.

wget https://archive.apache.org/dist/hadoop/core/hadoop-2.9.0/hadoop-2.9.0.tar.gz

After download untar Hadoop-2.9.0-tar.gz

tar -xvf hadoop-2.9.0.tar.gz

Create a dir for Hadoop and jdk using below command :

$sudo mkdir /usr/lib/hadoop

$sudo mkdir /usr/lib/jvm

Move extracted hadoop and jdk folder to newly created dir :

$mv hadoop-2.9.0 /usr/lib/hadoop

$mv jdk-1.8.0_131 /usr/lib/jvm

Set jdk and hadoop path in .bashrc file, which is present at /home/ubuntu dir :

$ls –al

$vi .bashrc

After opening .bashrc file press ESC button

To edit .bashrc file press i to INSERT

After configure .bashrc file press : (colon) and wq! To save,quit and overwrite

Execute .bashrc using below command:

$. .bashrc

Check java and hadoop version :

$java -version

$hadoop version

Go to hadoop sbin dir :

$cd $HADOOP_HOME/sbin

$ls –lrt

Setting a hadoop environment for passwordless ssh access. Passwordless SSH Configuration is a mandatory installation requirement for a single node cluster. However, it is more useful in a multi-node cluster environment.

Enter command $ssh-keygen -t rsa

$ ls –alr

$ls –lrt

cat .ssh/id_rsa.pub >> .ssh/authorized_keys

ssh-copy-id -i .ssh/id_rsa.pub ubuntu@localhost

1. check password less ssh access to localhost

$ ssh localhost

Exit from inner localhost shell by using below command:

 $exit

Set the Hadoop config files for a single node cluster. We need to set the below files in order to Hadoop a single node cluster to function properly.

♦ core-site.xml

hadoop-env.sh

♦ yarn-site.xml

hdfs-site.xml

♦ mapred-site.xml

Create a directory where the Hadoop will store its data. We will set this directory path in hdfs-site.

$mkdir  hadoopdata

To Configuration in core-site.xml

The command is $ vi core-site.xml

vi hadoop-env.sh

$ vi yarn-site.xml

vi hdfs-site.xml

NOTE: Make mapred-site.xml file from mapred-site.xml.template.

$ mapred-site.xml

► Formatting the HDFS file system via NameNode (after installing Hadoop, for the first time we have to format the HDFS file system to make it work)

Enter the command as given below:
$hadoop namenode –format

Go to sbin directory of hadoop :
$cd $HADOOP_HOME/sbin

$ls –lrt

►Start One service at a time :

$hadoop-daemon.sh start namenode

Start service from HDFS components :$start-dfs.sh

Start all service:

$start-all.sh

►Check for Hadoop processes /daemons running on Hadoop with Java Virtual Machine Process Status Tool.

$jps

Port number of Hadoop-daemons given below:

NameNode – localhost: 50070

DataNode – localhost: 50075

SecondaryNameNode – localhost: 50090

ResourceManager –  localhost: 8088

# Single Node Cluster with Hadoop in EC2 Instance on AWS setup here successfully, now you can Copy file from LFS to HDFS and HDFS to LFS by using various Hadoop commands like:

► Following commands are used for copy files from local file system to Hadoop distributed file system:

$hadoop fs -copyFromLocal LFS HDFS   Or

$hadoop fs -put LFS HDFS

mkdir dir_name

► Following commands are used for Hadoop distributed file system (HDFS):

$hadoop fs -mkdir /dir_name

hadoop fs -cp src dst

$ hadoop fs -mv src dst

# Last but not least, always asks for help!