Hadoop MapReduce interview questions and answers
Hadoop MapReduce interview questions and answers, are you looking for the best Interview Questions on Hadoop MapReduce? Or hunting for the best platform which provides a list of Top Rated Hadoop MapReduce interview questions and answers for experienced? Then stop hunting and follow Best Big Data Training Institute for the List of Top-Rated Hadoop MapReduce interview questions and answers for experienced for which are useful for both Fresher’s and experienced.
Are you the one who is a hunger to become Pro certified Hadoop Developer then ask your Industry Certified Experienced Hadoop Trainer for more detailed information? Don’t just dream to become Pro-Developer Achieve it learning the Hadoop Course under world-class Trainer like a pro. Follow the below mentioned Hadoop MapReduce interview questions and answers to crack any type of interview that you face.
What is Hadoop Map Reduce?
Ans: MapReduce is the heart of Hadoop. It is the programming paradigm that allows for massive scalability across hundreds or thousands of servers in a Hadoop cluster. It’s a processing layer of Hadoop.
Map Reduce is a programming model designed for processing large volumes of data in parallel by dividing the task into the set of chunks. We need to write the business logic, and then rest work will be taken care of by the framework. The problem is divided into a large number of the smaller part each of which is processed independently to produce individual outputs. These individual outputs are produced final outputs.
There are two processes one is Mapper and another is the Reducer.
What is Mapper?
Ans: Mapper is used to processing the input data, Input data is in the form of file or directory which resides in HDFS. The client needs to write the map-reduce program and need to submit the input data/file. The input file is passed to the mapper line by line. It will process the data & produce the output which is called intermediate output. The output of the map is stored on the local disk from where it is shuffled to reduce nodes. The number of maps is usually driven by the total volume of the inputs that is the total number of blocks of the input files.
What is Reducer?
Ans: Reducer: takes an intermediate key/value pairs produced by Map. The reducer has 3 primary phases which are shuffle, sort and reduce.
1.Shuffle: Input to the Reducer is the sorted output of mappers. In this stage, the framework fetches all output of mappers.
2.Sort: The framework groups reducer input by keys.
Reducer is the second stage of processing when the client needs to write the business logic. The output of reducer is the final output that is written to HDFS.
What is the need of Map Reduce?
Ans: MapReduce is a Fault-Tolerant programming model present as the heart of the Hadoop ecosystem. Because of all the above features, Map Reduce has become the favorites of the industry. This is also the reason that it is present in lots of Big Data Frameworks.
Hadoop HDFS commands with examples and Usage
What are the main components of Map Reduce Job?
Ans:
Main Driver Class: it providing job configuration parameters
Mapper Class: it must extend org.apache.hadoop.mapreduce.Mapper class and performs execution of map () method
Reducer Class: must extend org.apache.hadoop.mapreduce.Reducer class
What is Shuffling and Sorting in Map Reduce?
Ans: Shuffling and sorting are two major processes operating simultaneously during the working of mapper and reducer. The process of transferring data from a Mapper to a reducer is Shuffling. It is a mandatory operation for reducers to proceed with their jobs further as the shuffling process serves as input for the reduce tasks/works. In Map Reduce, the output key-value pairs between the map and reduce phases (after the mapper) are automatically sorted before moving towards Reducer. This feature is helpful in programs where you need to sort at some stages. It also saves the programmer’s overall time.
What are Partitioner and its usage?
Ans: Partitioner is yet another important phase that controls the partitioning of the intermediate map-reduce output keys using a hash function. The process of partitioning determines in which reducer, a key-value pair (of the map output) is sent. The number of partitions is equal to the total number of reducing jobs for the process.
Hash Partitioner is the default class available in Hadoop, which implements the following function.int getPartition(K key, V value, intnumReduceTasks)
The function returns the partition number using the numReduceTasks is the number of fixed reducers.
Which main configuration parameters are specified in Map Reduce?
Ans: The Map-Reduce programmers need to specify the following configuration parameters to perform the map and reduce jobs:
1.The input location of the job in HDFS
2.The output location of the job in HDFS.
3.The input and output format.
4.The classes containing a map and reduce functions, respectively.
5.The .jar file for mapper, reducer and driver classes
Which are Job control options specified by Map Reduce?
Ans: Since this framework supports chained operations wherein an input of one map job serves as the output for others, there is a need for job controls to govern these complex operations.
The various job control options are:
1.Job.submit (): to submit the job to the cluster and immediately return
2.Job.waitforCompletion (Boolean): to submit the job to the cluster & wait for its completion
What is Input Format in Hadoop?
Another important feature in Map Reduce programming, Input Format defines the input specifications for a job. It performs the following functions:
1. Validates the input-specification of the job.
2. Split the input file(s) into logical instances called Input Split. Each of these split files is then assigned to individual Mapper.
3. Provides an implementation of Record Reader to extract input records from the above instances for further Mapper processing
What is Job Tracker?
Job Tracker is a Hadoop service used for the processing of Map Reduce jobs in the Hadoop cluster. It submits & tracks the jobs to specific nodes having data. Only one Job Tracker runs on a single Hadoop cluster on its own JVM process. If Job Tracker goes down, all the jobs halt.
What is SequenceFileInputFormat?
A compressed binary output file format to read in sequence files & extends the FileInputFormat.It passes data between output-input (between the output of one Map Reduce job to input of another Map Reduce job) phases of Map Reduce jobs.
How to set mappers and reducers for Hadoop jobs?
Users can be configureJobConf variable to set number of mappers and reducers.
1.job.setNumMaptasks()
2.job.setNumreduceTasks()
Explain JobConf in Map Reduce.
It is a primary interface to define a map-reduce job in the Hadoop for job execution. JobConf specifies mapper, Combiner, partitioner, Reducer, InputFormat, OutputFormat implementations and other advanced job faetsliek Comparators.
what is a Map-Reduce Combiner?
Also known as semi-reducer, Combiner is an optional class to combine the map out records using the same key. The main function of a combiner is to accept the inputs from Map Class and pass those key-value pairs to Reducer class
what is Record Reader in a Map-Reduce?
Record Reader is used to reading key/value pairs form the Input Split by converting the byte-oriented view and presenting a record-oriented view to Mapper.
Define Writable data types in Map Reduce.
Hadoop reads and writes the data in a serialized form in the writable interface. The Writable interface has several classes like Text (storing String data), IntWritable, LongWriatble, FloatWritable, and BooleanWritable. users can define their personal Writable classes as well.
What is OutputCommitter?
OutPutCommitter describes the commit of the MapReduce task. FileOutputCommitter is the default available class available for OutputCommitter in the MapReduce. It performs the following operations:
1. Create a temporary output directory for the job during the initialization.
2. Then, it cleans the job as it removes the temporary output directory after job completion.
3. Sets up the task temporary output.
4. Identifies whether a task needs to commit. The commit is applied if required.
5. job setup, JobCleanup, and TaskCleanup are important tasks during output commit.
What are the parameters of mappers and reducers?
The four parameters for mappers are:
1.LongWritable (input)
2.text (input)
3.text (intermediate output)
4.IntWritable (intermediate output)
The four parameters for reducers are:
1.Text (intermediate output)
2.IntWritable (intermediate output)
3.Text (final output)
4.IntWritable (final output)
MapReduce interview questions and answers for Fresher
What is partitioning?
Partitioning is a process to identify the reducer instance which would be used to supply the mappers output. Before the mapper emits the data (Key Value) pair to the reducer, the mapper identifies the reducer as a recipient of the mapper output. All the key, no matter which mapper has generated this, must lie with the same reducer.
How to set which framework would be used to run the MapReduce program?
MapReduce.framework.name. it can be
1. Local
2. classic
3. Yarn
Can the MapReduce program be written in any language other than Java?
Yes, MapReduce can be written in many programming languages like Java, R, C++, Scripting Languages (Python, PHP). Any language able to read from stadin and write to stdout and parse tab and newline characters should work. Hadoop streaming (A Hadoop Utility) allows you to create and run Map/Reduce jobs with any executable or scripts as the mapper or the reducer.
What are MapReduce and list its features?
Ans. MapReduce is a programming model used for processing and generating large datasets on the clusters with parallel and distributed algorithms.
The syntax for running the MapReduce program is
1 hadoop_jar_file.jar /input_path /output_path.
What are the features of MapReduce?
Ans:-
1.Automatic parallelization and distribution.
2. Built-in fault-tolerance and redundancy are available.
3.MapReduce Programming model is language independent
4. Distributed programming complexity is hidden
5.Enable data local processing
6.Manages all the inter-process communication
What does the MapReduce framework consist of?
Ans. MapReduce framework is used to write applications for processing large data in parallel on large clusters of commodity hardware.
It consists of:
ResourceManager (RM)
1.Global resource scheduler
2.One master RM
NodeManager (NM)
1.One slave NM per cluster-node.
Container
2.ResourceManager creates Containers upon request by ApplicationMaster
3. The application runs in one or more containers
ApplicationMaster (AM)
1.One AM per application
2.Runs in Container
What are the two main components of ResourceManager?
Ans.
1.Scheduler
It allocates the resources (containers) to various running applications based on resource availability & configured shared policy.
2.ApplicationManager
It is mainly responsible for managing a collection of submitted applications
What is a Hadoop counter?
Ans. Hadoop Counters measures the progress or tracks the number of operations that occur within a MapReduce job. Counters are useful for collecting statistics about MapReduce job for the application-level or quality control.
What are the main configuration parameters for a MapReduce application?
Ans. The job configuration requires the following:
1.Job’s input and output path in the distributed file system
2.The input format of data
3. The output format of data
4.Class containing the map function and reduce function
5. JAR file containing the reducer, driver, and mapper classes
What are the steps involved to submit a Hadoop job?
Ans. Steps involved in Hadoop job submission:
1. Hadoop job client submits the job jar/executable and configuration to the ResourceManager.
2.ResourceManager then distributes the software/configuration to the slaves.
3.ResourceManager then scheduling tasks and monitoring them.
4. Finally, job status and diagnostic information are provided to the client.
How does the MapReduce framework view its input internally?
Ans. It views the input data set as a set of pairs and processes the map tasks in a completely parallel manner.
What are the basic parameters of Mapper?
Ans. The basic parameters of Mapper are listed below:
1. LongWritable and Text
2. Text and IntWritable
What are Writables and explain its importance in Hadoop?
Ans.
1. Writables are interfaces in Hadoop. They act as a wrapper class to almost all the primitive data types of Java.
2.A serializable object which executes a simple and efficient serialization protocol, based on DataInput and DataOutput.
3. Writables are for creating serialize data types in Hadoop.
Why comparison of types is important for MapReduce?
Ans.
1.It is important for MapReduce as in the sorting phase the keys are compare with one another.
2.For a Comparison of types, the WritableComparable interface is implemented.
What is “speculative execution” in Hadoop?
Ans. In Apache Hadoop, if nodes do not fix or diagnose the slow-running tasks, the master node can redundantly perform another instance of the same task on another node as a backup (the backup task is called a Speculative task). This process is called Speculative Execution in Hadoop.
What are the methods used for restarting the NameNode in Hadoop?
Ans. The methods used for restarting the NameNodes are the following:
1.You can use /sbin/hadoop-daemon.sh stop namenode command for stopping the NameNode individually and then start the NameNode using /sbin/hadoop-daemon.sh start namenode.
2.Use /sbin/stop-all.sh and then use /sbin/start-all.sh command for stopping all the demons first and then start all the daemons.
These script files are stored in the sbin directory inside the Hadoop directory store.
What is the difference between an “HDFS Block” and “MapReduce Input Split”?
Ans.
1.HDFS Block is the physical division of the disk which has the minimum amount of data that can be read/write, while MapReduceInputSplit is the logical division of data created by the InputFormat specified in the MapReduce job configuration.
2.HDFS divides data into blocks, whereas MapReduce divides data into input split and empower them to mapper function.
What are the different modes in which Hadoop can run?
Ans.
1.Standalone Mode (local mode) – This is the default mode where Hadoop is configure to run. In this mode, all the components of Hadoop such as DataNode, NameNode, etc., run as a single Java process and useful for debugging.
2.Pseudo Distributed Mode (Single-Node Cluster) – Hadoop runs on a single node in a pseudo-distributed mode. Each Hadoop daemon works in a separate Java process in Pseudo-Distributed Mode, while in Local mode, each Hadoop daemon operates as a single Java process.
MapReduce interview questions and answers for Experienced
Why aggregation cannot be perform in Mapperside?
Ans.
1. We cannot perform Aggregation in mapping because it requires sorting of data, which occurs only at the Reducer side.
2. For aggregation, we need the output from all the mapper functions, which is not possible during the map phase as map tasks will be running in different nodes, where data blocks are present.
What is the importance of “RecordReader” in Hadoop?
Ans.
1.RecordReader in Hadoop uses the data from the InputSplit as input and converts it into Key-value pairs for Mapper.
2. The MapReduce framework represents the RecordReader instance through InputFormat.
What is the purpose of Distributed Cache in a MapReduce Framework?
Ans.
1. The Purpose of Distributed Cache in the MapReduce framework is to cache files when needed by the applications. It caches read-only text files, jar files, archives, etc.
2. When you have cached a file for a job, the Hadoop framework will make it available to each and every data node where map/reduces tasks are operating.
How do reducers communicate with each other in Hadoop?
Ans. Reducers always run in isolation and the Hadoop Mapreduce programming paradigm never allows them to communicate with each other.
What is Identity Mapper?
Ans. Identity Mapper is a default Mapper class that automatically works when no Mapper is specified in the MapReduce driver class.
1. It implements mapping inputs directly into the output.
What are the phases of MapReduce Reducer?
Ans. The MapReduce reducer has three phases:
1.Shuffle phase – In this phase, the sorted output from a mapper is an input to the Reducer. This framework will fetch the relevant partition of the output of all the mappers by using HTTP.
2. Sort phase – In this phase, the input from various mappers are sort on related keys. This framework groups reducer inputs by keys. Shuffle and sort phases occur concurrently.
3. Reduce phase – In this phase, reduce task aggregates the key-value pairs after shuffling and sorting phases. The OutputCollector.collect() method, writes the output of the reduce task to the Filesystem.
What is the purpose of MapReducePartitioner in Hadoop?
Ans. The MapReducePartitioner manages the partitioning of the key of the intermediate mapper output. It makes sure that all the values of a single key pass to the same reducers by allowing the even distribution over the reducers.
How will you write a custom partitioner for a Hadoop MapReduce job?
Ans:
1.Build a new class that extends Partitioner Class
2. Override the get partition method in the wrapper.
3. Add the custom partitioner to the job as a config file or by using the method set Partitioner.
What is Combiner?
Ans.
A Combiner is a semi-reducer that executes the local reduce task. It receives inputs from the Map class and passes the output key-value pairs to the reducer class.
We, Prwatech India’s Leading Big Data Training Institute listed some of the Best Top Rated interview questions on Hadoop MapReduce interview questions and answers in which most of the Interviewers are asking Candidates nowadays. So follow the Below Mentioned Best interview questions on Hadoop MapReduce and Crack any Kind of Interview Easily.
