Forum

This topic contains 0 replies, has 1 voice, and was last updated by  srikanth 1 year, 6 months ago.

Viewing 1 post (of 1 total)
  • Author
    Posts
  • #1989 Reply

    srikanth
    Participant

    A custom shuffle implementation requires a org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.AuxiliaryService implementation class running in the NodeManagers and a org.apache.hadoop.mapred.ShuffleConsumerPlugin implementation class running in the Reducer tasks.

    The default implementations provided by Hadoop can be used as references:

    org.apache.hadoop.mapred.ShuffleHandler
    org.apache.hadoop.mapreduce.task.reduce.Shuffle
    A custom sort implementation requires a org.apache.hadoop.mapred.MapOutputCollector implementation class running in the Mapper tasks and (optionally, depending on the sort implementation) a org.apache.hadoop.mapred.ShuffleConsumerPlugin implementation class running in the Reducer tasks.

    The default implementations provided by Hadoop can be used as references:

    org.apache.hadoop.mapred.MapTask$MapOutputBuffer
    org.apache.hadoop.mapreduce.task.reduce.Shuffle

Viewing 1 post (of 1 total)
Reply To: write a customize shuffle algorithm
Your information:




cf22

Your Name (required)

Your Email (required)

Subject

Phone No

Your Message

Cart

  • No products in the cart.