Zookeeper <\/strong><\/p>\nThis is basically a centralized system that maintains<\/p>\n
1.Configuration information<\/p>\n
2.Naming information<\/p>\n
3.Synchronization information<\/p>\n
Besides these, Zookeeper is also responsible for group services and is
\nutilized by HBASE. It also comes to use for MapReduce programs.Solr\/Lucene \u2013 This is nothing but a search engine. Its libraries are\u00a0developed by Apache and required over 10 years to be developed in its
\npresent robust form.<\/p>\n
Programming Languages<\/strong>
\nThere are basically two programming\u00a0languages that are identified as original Hadoop programming\u00a0languages,<\/p>\n1.Hive
\n2.PIG<\/p>\n
Besides these, there are a few other programming languages that can be
\nused for writing programs, namely C, JAQL and Java. We can also
\nmake direct usage of SQL for interaction with the database, although
\nthat requires the use of standard JDBC or ODBC drivers.<\/p>\n
<\/p>\n
Define HDFS and YARN, and talk about their respective components.<\/h3>\n
<\/p>\n
Hadoop Distributed File System (HDFS)<\/strong>
\nHDFS is a distributed file system that provides access to data across\u00a0Hadoop clusters. A cluster is a group of computers that work together.\u00a0Like other Hadoop-related technologies, HDFS is a key tool that\u00a0manages and supports the analysis of very large volumes petabytes and\u00a0zettabytes of data.<\/p>\nHDFS Components<\/strong><\/p>\nThe main components of HDFS are,
\n1.Namenode
\n2.Secondary Namenode
\n3.File system
\n4.Metadata
\n5.Datanode<\/p>\n
HDFS Command Line<\/strong>
\nThe following are a few basic command lines of HDFS.<\/strong><\/p>\nTo copy the file prwatech.txt from the local disk to the user's directory,<\/p>\n
type the command line:<\/strong>
\n1.hdfsdfs \u2013put prwatech.txt prwatech.txt
\nThis will copy the file to \/user\/username\/prwatech.txt<\/p>\nTo get a directory listing of the user's home directory, type the command\u00a0line:<\/strong><\/p>\n$hdfsdfs \u2013ls<\/p>\n
To create a directory called testing under the user's home directory, type\u00a0the command line:<\/strong><\/p>\n$hdfsdfs \u2013mkdir<\/p>\n
To delete the directory testing and all of its components, type the<\/strong>
\ncommand line:<\/strong><\/p>\nhdfsdfs -rm -r<\/p>\n
<\/p>\n
What is YARN?<\/h3>\n
YARN is the acronym for Yet Another Resource Negotiator. YARN is a\u00a0resource manager created by separating the processing engine and the\u00a0management function of MapReduce.\u00a0It monitors and manages workloads, maintains a multi-tenant\u00a0environment manages the high availability features of Hadoop, and\u00a0implements security controls.\u00a0Before 2012, users could write MapReduce programs using scripting\u00a0languages such as Java, Python, and Ruby. They could also use Pig, a\u00a0language used to transform data. No matter what language was used, its\u00a0implementation depended on the MapReduce processing model.<\/p>\n
In May 2012, during the release of Hadoop version 2.0, YARN was\u00a0introduced. You are no longer limited to working with the MapReduce\u00a0framework anymore as YARN supports multiple processing models in\u00a0addition to MapReduce, such as Spark.\u00a0Other features of YARN\u00a0 include significant performance improvement\u00a0and a flexible execution engine.<\/p>\n
Let us first understand the important three Elements of YARN
\nArchitecture.<\/p>\n
The three important elements of the YARN architecture are:
\n1.Resource Manager
\n2.Application Master
\n3.Node Managers<\/p>\n
Resource Manager,<\/strong>
\nThe ResourceManager, or RM, which is usually one per cluster, is the\u00a0master server. Resource Manager knows the location of the DataNode\u00a0and how many resources they have. This information is referred to as\u00a0Rack Awareness. The RM runs several services, \u009dthe most important of which is the Resource Scheduler decides how to assign resources.<\/p>\nApplication Master,<\/strong>
\nThe Application Master is a framework-specific process that negotiates resources for a single application, that is, a single job or a\u00a0directed acyclic graph of jobs, which runs in the first container\u00a0allocated for the purpose.
\nEach Application Master requests resources from the Resource\u00a0Manager and then works with containers provided by Node\u00a0Managers.<\/p>\n <\/p>\n
What is the purpose of the JPS command in Hadoop?<\/h3>\n
JPS (Java virtual machine process tool) is a command used to check all\u00a0the Hadoop daemons are running or not on the machine-like Namenode,\u00a0Secondary Namenode, Datanode, Resource Manager, Node Manager.<\/p>\n
<\/p>\n
Why do we need Hadoop for Big Data Analytics?<\/h3>\n
The primary function of Hadoop is to facilitate quickly doing\u00a0analytics on huge sets of unstructured data. In other words, Hadoop is all\u00a0about handling " big data." So the first question to ask is whether that.\u00a0the kind of data you are working with. Secondly, does your data require\u00a0real-time, or close to real-time analysis? Where Hadoop excels is in\u00a0allowing large datasets to be processed quickly. Another consideration is the rate at which your data storage\u00a0requirements are growing. A big advantage of Hadoop is that it is extremely scalable. You can add new storage capacity simply by adding server nodes in your Hadoop cluster. In theory, a Hadoop cluster can be almost infinitely expanded as needed using low-cost commodity server\u00a0and storage hardware.<\/p>\n
If your business faces the combination of huge amounts of data, along\u00a0with a much less than huge storage budget, Hadoop may well be the best\u00a0solution for you.<\/p>\n
<\/p>\n
Explain the different features of Hadoop.<\/h3>\n
Scalable,<\/strong><\/p>\nHadoop is a highly scalable storage platform because it can store and distribute very large data sets across hundreds of inexpensive servers that operate in parallel. Unlike a traditional relational database system (RDBMS) that can\u2019t scale to process large amounts of data, Hadoop enables businesses to run applications on thousands of nodes\u00a0involving many thousands of terabytes of data.<\/p>\n
Varied Data Sources,<\/strong><\/p>\nHadoop accepts a variety of data. Data can come from a range of\u00a0sources like email conversation, social media, etc. and can be of the structured or unstructured form. Hadoop can derive value from diverse data. Hadoop can accept data in a text file, XML file, images, CSV files, etc.<\/p>\n
Cost-effective,<\/strong><\/p>\nHadoop is an economical solution as it uses a cluster of commodity hardware to store data. Commodity hardware is cheap machines hence the cost of adding nodes to the framework is not much high. In Hadoop 3.0 we have only 50% of storage overhead as opposed to 200% in Hadoop2.x. This requires less machine to store data as the redundant data decreased significantly.<\/p>\n
Flexible,<\/strong><\/p>\nHadoop enables businesses to easily access new data sources and tap into different types of data (both structured and unstructured) to generate value from that data. This means businesses can use Hadoop to derive valuable business insights from data sources such as social media, email conversations.\u00a0 Hadoop can be used for a wide variety of purposes, such as log processing, recommendation systems, data warehousing, market campaign analysis, and fraud detection.<\/p>\n
Fast :<\/strong><\/p>\nHadoop\u2019s unique storage method is based on a distributed file system that basically \u2018maps\u2019 data wherever it is located on a cluster. The tools for data processing are often on the same servers where the data is located, resulting in the much faster data processing. If you\u2019re dealing with large volumes of unstructured data, Hadoop is able to efficiently process terabytes of data in just minutes, and petabytes in hours.<\/p>\n
Performance :<\/strong><\/p>\nHadoop with its distributed processing and distributed storage architecture processes huge amounts of data with high speed. Hadoop even defeated supercomputer the fastest machine in 2008. It divides the input data file into a number of blocks and stores data in these blocks over several nodes. It also divides the task that the user submits into various sub-tasks which assign to these worker nodes containing required data and these sub-task run in parallel thereby improving the performance.<\/p>\n
Fault-Tolerant:<\/strong><\/p>\nIn Hadoop 3.0 fault tolerance is provided by erasure coding. For example, 6 data blocks produce 3 parity blocks by using erasure coding technique, so HDFS stores a total of these 9 blocks. In the event of failure of any node the data block affected can be recovered by using these parity blocks and the remaining data blocks.<\/p>\n
Highly Available :<\/strong><\/p>\nIn Hadoop 2.x, HDFS architecture has a single active NameNode and a single Standby NameNode, so if a NameNode goes down then we have standby NameNode to count on. But Hadoop 3.0 supports multiple standby NameNode making the system even more highly available as it can continue functioning in case of two or more NameNodes crashes.<\/p>\n
Resilient to failure,<\/strong><\/p>\nA key advantage of using Hadoop is its fault tolerance. When data is sent to an individual node, that data is also replicated to other nodes in the cluster, which means that in the event of failure, there is another copy available for use.<\/p>\n
\u00a0Low Network Traffic,<\/strong><\/p>\nIn Hadoop, each job submitted by the user is split into a number of independent sub-tasks and these sub-tasks are assigned to the data nodes thereby moving a small amount of code to data rather than moving huge data to code which leads to low network traffic.<\/p>\n
High Throughput:<\/strong><\/p>\nThroughput means job done per unit time. Hadoop stores data in a distributed fashion which allows using distributed processing with ease. A given job gets divided into small jobs that work on chunks of data in parallel thereby giving high throughput.<\/p>\n
Ease of use,<\/strong><\/p>\nThe Hadoop framework takes care of parallel processing, MapReduce programmers do not need to care for achieving distributed processing, it is done at the backend automatically.<\/p>\n
Compatibility :<\/strong><\/p>\nMost of the emerging technology of Big Data is compatible with Hadoop like Spark, Flink, etc. They have got processing engines that work over Hadoop as a backend i.e. We use Hadoop as data storage platforms for them.<\/p>\n
Multiple Languages Supported,<\/strong><\/p>\nDevelopers can code using many languages on Hadoop like C, C++, Perl, Python, Ruby, and Groovy.<\/p>\n
<\/p>\n
What are the Edge Nodes in Hadoop?<\/h3>\n
Edge nodes are the interface between the Hadoop cluster and the outside network. For this reason, they\u2019re sometimes referred to as gateway nodes. Most commonly, edge nodes are used to run client applications and cluster administration tools. They\u2019re also often used as staging areas for data being transferred into the Hadoop cluster. As such, Oozie, Pig, Sqoop, and management tools such as Hue and Ambari run well there. The figure shows the processes you can run on the Edge nodes.<\/p>\n
Edge nodes are often overlooked in Hadoop hardware architecture discussions. This situation is unfortunate because edge nodes serve an important purpose in a Hadoop cluster, and they have hardware requirements that are different from master nodes and slave nodes. In general, it\u2019s a good idea to minimize deployments of administration\u00a0tools on master nodes and slave nodes to ensure that critical Hadoop\u00a0services like the NameNode have as little competition for resources as\u00a0possible.<\/p>\n
The figure shows two edge nodes, but for many Hadoop clusters, a single\u00a0edge node would suffice. Additional edge nodes are most commonly\u00a0needed when the volume of data being transferred in or out of the cluster\u00a0is too much for a single server to handle.<\/p>\n
<\/p>\n
What are the five V\u2019s of Big Data?<\/h3>\n
In recent years, Big Data as defined by the \u201c3Vs\u201d but now there is\u00a0\u201c5Vs\u201d of Big Data which are also termed as the characteristics of Big\u00a0Data as follows:
\nVolume<\/strong>:<\/p>\n1. The name \u2018Big Data\u2019 itself is related to a size that is enormous.
\n2. Volume is a huge amount of data.
\n3. To determine the value of data, size of data plays a very crucial role.
\n4. If the volume of data is very large then it is actually considered as a \u2018Big Data\u2019. This means whether a particular data can actually be considered as a Big Data or not, is dependent upon the volume of data.
\n5. Hence while dealing with Big Data it is necessary to consider a characteristic \u2018Volume\u2019.<\/p>\n
Example: In the year 2016, the estimated global mobile traffic was 6.2 Exabytes(6.2 billion GB) per month. Also, by the year 2020, we will have almost 40000 ExaBytes of data.<\/p>\n
Velocity<\/strong>:<\/p>\n1. Velocity refers to the high speed of accumulation of data.
\n2. In Big Data velocity data flows in from sources like machines, networks, social media, mobile phones, etc.
\n3. There is a massive and continuous flow of data. This determines the potential of data that how fast the data is generated and processed to meet the demands.
\n4. Sampling data can help in dealing with the issue like \u2018velocity\u2019.Example: There are more than 3.5 billion searches per day are made on Google. Also, Facebook users are increasing by 22%(Approx.) year by year.<\/p>\n
Variety<\/strong>:<\/p>\n1. It refers to the nature of data that is structured, semi-structured and unstructured data.
\n2. It also refers to heterogeneous sources.
\n3. Variety is basically the arrival of data from new sources that are both inside and outside of an enterprise. It can be structured, semi-structured and unstructured.
\n4.Structured data: This data is basically organized data. It generally refers to data that has defined the length and format of data.
\n5.Semi-Structured Data: This data is basically a semi-organized data. It is generally a form of data that does not conform to the formal structure of data. Log files are examples of this type of data.
\n6.Unstructured data: This data basically refers to unorganized data. It generally refers to data that doesn\u2019t fit neatly into the traditional row and column structure of the relational database. Texts, pictures, videos, etc. are examples of unstructured data that can\u2019t be stored in the form of rows and columns.<\/p>\n
Veracity<\/strong>:<\/p>\n1. It refers to inconsistencies and uncertainty in data, that is data that is available can sometimes get messy and quality and accuracy are difficult to control.
\n2. Big Data is also variable because of the multitude of data dimensions resulting from multiple disparate data types and sources.
\n3.Example: Data in bulk could create confusion whereas less amount of data could convey half or Incomplete Information.<\/p>\n
Value:<\/strong><\/p>\n1. After having the 4 V\u2019s into account there comes one more V which stands for Value. The bulk of Data having no Value is of no good to the company, unless you turn it into something useful.
\n2. Data in itself is of no use or importance but it needs to be converted into something valuable to extract Information. Hence, you can state that Value! is the most important V of all the 5Vs.<\/p>\n
<\/p>\n
Define respective components of HDFS and YARN<\/h3>\n
Hadoop Distributed File System (HDFS)\u00a0HDFS is a distributed file system that provides access to data across\u00a0Hadoop clusters. A cluster is a group of computers that work together. Like other Hadoop-related technologies, HDFS is a key tool that\u00a0manages and supports the analysis of very large volumes of petabytes and\u00a0zettabytes of data.<\/p>\n
<\/p>\n
HDFS Components<\/h3>\n
The main components of HDFS are,<\/strong><\/p>\n1.Namenode
\n2.Secondary Namenode
\n3.File system
\n4.Metadata
\n5.Datanode
\n6.HDFS Command Line<\/p>\n
<\/p>\n