This topic contains 0 replies, has 1 voice, and was last updated by  srikanth 2 years, 6 months ago.

Viewing 1 post (of 1 total)
  • Author
    Posts
  • #1989

    srikanth
    Participant

    A custom shuffle implementation requires a org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.AuxiliaryService implementation class running in the NodeManagers and a org.apache.hadoop.mapred.ShuffleConsumerPlugin implementation class running in the Reducer tasks.

    The default implementations provided by Hadoop can be used as references:

    org.apache.hadoop.mapred.ShuffleHandler
    org.apache.hadoop.mapreduce.task.reduce.Shuffle
    A custom sort implementation requires a org.apache.hadoop.mapred.MapOutputCollector implementation class running in the Mapper tasks and (optionally, depending on the sort implementation) a org.apache.hadoop.mapred.ShuffleConsumerPlugin implementation class running in the Reducer tasks.

    The default implementations provided by Hadoop can be used as references:

    org.apache.hadoop.mapred.MapTask$MapOutputBuffer
    org.apache.hadoop.mapreduce.task.reduce.Shuffle

Viewing 1 post (of 1 total)

The forum ‘BIG DATA & HADOOP INTERNSHIP BATCH – 2015’ is closed to new topics and replies.