shuffle phase as the bottleneck in hadoop terasort
TRANSCRIPT
Profiling the network performance of data transfers in Hadoop jobsTeam : Pramod Biligiri & Sayed Asad Ali
AbstractWe have attempted to reproduce existing research which shows that the Shuffle phase of Hadoop is network intensive and can constitute a bottleneck for many Hadoop jobs. We ran the Terasort and Ranked Inverted Index jobs on an Amazon Elastic MapReduce cluster. Our experiments show that the Shuffle phase can form a significant fraction (upto nearly 30%) of the time consumed in these jobs.
We do not have decisive results showing that the network is saturated during this phase. This is due to a) lack of precise documentation on the network capacity of EMR, and b) inconsistent results between our network benchmark tests and the results from the Hadoop jobs. See Section 6 for a detailed discussion of both these factors.
1. IntroductionData intensive computing on large scale, commodity clusters is becoming commonplace. Hadoop[9] is a popular framework used in such computing environments. While performance analysis is traditionally focused on the algorithm, the processing unit, memory and disk, the rise of cluster computing adds the communication patterns of the algorithm and the underlying network capacity as factors to consider while evaluating performance.
In this project we profile the data transfers that happen between the different stages of a Hadoop job, with an aim to understand the utilization of network resources during the process. We hope to reproduce some well known results which show that network utilization is a bottleneck in MapReduce. We intend to focus on the shuffle phase of the MapReduce pipeline, and the manytomany pattern of data movement therein.
1.1. HadoopHadoop is a framework for distributed processing of large data sets across clusters of computers using simple programming models based on Google’s MapReduce [7]. Hadoop is open source and implemented in Java.
Hadoop can be characterized by the following distinct features:● Designed for commodity hardware● Fault tolerant● Horizontally scalable● Push computation to data
1.2. MapReduceMapReduce is a programming model for processing large data sets with a parallel, distributed algorithm on a cluster. In this model, a program consists of two phases: Map and Reduce. In the Map phase, each input record is processed to generate a (key, value) pair. In the Reduce phase,
values associated with the same key are grouped together and an operation is applied on them to obtain the final results.
The following figure illustrates the flow of a MapReduce job:
1.3. ShuffleThe Shuffle is a phase where each reducer fetches its part of the sorted map outputs from all the mapper nodes. This phase results in a nton communication among a set of n nodes.
2. Related Work:We have studied a few papers which cite that the shuffle phase is an expensive operation. As stated by the Orchestra paper[1]: “On average, the shuffle phase accounts for 33% of the running time in these jobs. In addition,in 26% of the jobs with reduce tasks, shuffles account for more than 50% of the running time, and in 16% of jobs, they account for more than 70% of the running time. This confirms widely reported results that the network is a bottleneck in MapReduce”
More information from the Hedera paper[2] corrobrates this:“A data shuffle is an expensive but necessary operation for many MapReduce/Hadoop operations in which every host transfers a large amount of data to every other host participating in the shuffle. In this experiment, each host sequentially transfers 500MB to every other host using TCP (a 120GB shuffle).”
Furthermore, the VL2 paper[3] also establishes this observation:“we consider an alltoall data shuffle stress test: all servers simultaneously initiate TCP transfers to all other servers. This data shuffle pattern arises in large scale sorts, merges and join operations in the data center. We chose this test because, in our interactions with application developers, we learned that many use such operations with caution, because the operations are highly expensive in today’s data center network. However, data shuffles are required, and, if data shuffles can be efficiently supported, it could have large impact on the overall algorithmic and data storage strategy.”
3. Choice of Benchmarks:3.1. Terasort3.1.1. Why Terasort?Terasort [4] is a popular benchmark for Hadoop and is also shipped with most Hadoop distributions. This benchmark program sorts 1 terabyte of data. Each data item is 100 bytes in size. The first 10 bytes of a data item constitute its sort key.Each key is represented as:<key 10 bytes><rowid 10 bytes><filler 78 bytes>\r\n
key : random characters from ASCII 32126rowid : an integerfiller : random characters from the set AZ
The Terasort workload utilizes all aspects of the cluster cpu, network, disk and memory and also has a large amount of data to shuffle (240 GB). Moreover, this is representative of real world workloads, as mentioned in the VL2 paper[3]:“we consider an alltoall data shuffle stress test: all servers simultaneously initiate TCP transfers to all other servers. This data shuffle pattern arises in large scale sorts, merges and join operations in the data center. We chose this test because, in our interactions with application developers, we learned that many use such operations with caution, because the operations are highly expensive in today’s data center network. However, data shuffles are required, and, if data shuffles can be efficiently supported, it could have large impact on the overall algorithmic and data storage strategy.”
3.1.2. How it works?The Map phase of Terasort partitions input keys into different buckets and then leverages Hadoop’s default sorting of Map output. Finally, the reducer only collects outputs from different maps and does not perform a computationintensive task. Due to its simple application logic and usage of Hadoop’s default sorting mechanism, Terasort is considered a good benchmarking application.
3.2. Ranked Inverted Index3.2.1. Why Ranked Inverted Index?This benchmark was chosen as it is mentioned in the Tarazu[4] paper as a Shuffle heavy workload. Also, a ranked inverted index is used often in text processing and information retrieval tasks and is therefore a commonly executed job. For a given text corpus, for each word it generates a list of documents containing the word in decreasing order of frequencyword > (count1 | file1), (count2 | file2), ...count1 > count2 > …
4. Experimental Setup:4.1. ConfigurationWe utilised three configurations as a testbed for our experiments. Two of these were configured on Amazon’s Elastic MapReduce (EMR) clusters and we used a cluster at SDSC as a learning testbed. However, Config 1 on EMR is the one we chose for a majority of our tests and our results are based on that.
Both the EMR configurations have 1 NameNode, 10 DataNode/Tasktrackers.
Instance type
Memory CPU ECU Disk Network performance
Config 1 m1.large 7.5 GB 64bit 4 2 x 420 GB Moderate
Config 2 m1.xlarge 15 GB 64bit 8 4 x 420 GB High
SDSC custom 8 GB 64bit/ Intel Xeon CPU 5140 @2.33 GHz, 4 cores
2 x 1.5 TB 1 Gb/s
4.2. Network TestSource 1 : with AppNeta pathtest[8]average : 753 Mb/s
Source 2 : “The available bandwidth is still 1 Gb/s, confirming anecdotal evidence that EC2 has full bisection bandwidth."[5]Source 3 : “The median TCP/UDP throughput of medium instances are both close to 760 Mb/s." [6]
5. Results:5.1. Terasort5.1.1. Comparison of running Terasort on different Configurations
Total job Time (min)
Map Time (min)
Reduce Time (min)
Shuffle Average Time
Shuffle Time %
Config 1 205 84 205 60 29.3
SDSC 166 60 90 36 21.7
Config 2 86 40 75 22 25.5
5.1.2. CDF of Transferred Data
The CDF shows that network traffic happens in two distinct phases. First is the Map phase during which there is steady traffic, although not at high rates. Approximately half the amount of the total volume of 240GB is transferred during this time.
Following the Map, the traffic reduces as the map outputs are sorted locally. Then the job enters the Shuffle phase, where the data is transferred to all the reducers. During this phase, the network traffic saturates the links, as we show in subsequent sections. This phase transfers the remaining half of the 240GB of data.
5.1.3. Network activityThe following figure shows the network transfer rates over the lifetime of the job. It shows that during the Shuffle phase the network traffic reaches 700 Mbps, which was the peak transfer rate as measured by one of our tests.
5.1.4. Disk activityThe figure shows that the map phase has a good mix of Read and Write. However, once the map is done which is around the 5100sec mark, a marked reduction in the data read is observed and this phase shows a increase in writing activity of the cluster. This pattern continues till the end of the shuffle phase around 6900s, where shuffle starts and the pattern shifts to read/write but a marked reduction in read activity. The observed peak value is around 60 to 80 MB/s which is well below the threshold value of 100 MB/s according to the dd performance metric which was executed on the Amazon EC2 machines.
5.1.5. Memory ActivityThe captured logs indicate that throughout the timeline of the job, the memory shows a somewhat consistent utilisation of nearly 4.5 GB on all boxes and never overshoots this mark. As nearly 7.5 GB of memory is available on each box, this proves that the memory was never stressed to capacity, therefore hinting that the bottleneck lies elsewhere.
5.1.6. CPU UtilisationThe graphs below show the CPU utilisation on the EC2 boxes over the lifetime of the project. As indicated by the figure, the CPU may have been stressed to capacity with 100% utilisation but the pattern seen in the initial portion of the graph is very erratic and shows a high oscillation between full and partial CPU utilisation. However, during the shuffle phase, it is quite evident that the CPU utilisation drops below the 50% threshold, thus the CPU utilisation cannot be the limiting factor in the case of Shuffle phase of a job.
5.2. Ranked Inverted Index (RII)5.1.1. Data for a RII run on Config 1
Total job Time (min)
Map Time (min)
Reduce Time (min)
Shuffle Average Time
Shuffle Time %
Config 1 12 5.5 11.5 3.5 27.14
5.1.2. CDF of data
The CDF shows that network traffic happens in three distinct phases. First is the Map phase during which there is steady traffic, although not at high rates. Once, shuffle is activated then the network traffic picks up, this is where 13 GB of data gets transferred across the network in a short duration and we see the saturation point of the network. The burst of traffic which happens after this is the replication of the results to 3 nodes.
5.1.3. Network ActivityThe following figure shows the network transfer rates over the lifetime of the job. It shows that during the Shuffle phase the network traffic reaches 1.5 Gbps, which is something we have not been able to explain as the maximum expected rate should have been in the range of 700 800 Mbps.
5.1.4. Disk ActivityThe figure shows that the map phase has a good mix of Read and Write. However, once the map is done which is around the 350sec mark, a marked reduction in the data read is observed and this phase shows an increase in writing activity of the cluster. This pattern continues till the end of the shuffle phase around 550s, where shuffle starts and the pattern shifts to read/write but a marked reduction in read activity. The sporadic maximum read value is around 140 to 150 MB/s which does not stress the cluster as the consistent read rate is well below that.
5.1.5. Memory ActivityThe captured logs indicate that throughout the timeline of the job, the memory shows a somewhat consistent utilisation of nearly 4.5 GB on all boxes and never overshoots this mark. As nearly 7.5 GB of memory is available on each box, this proves that the memory was never stressed to capacity, therefore hinting that the bottleneck lies elsewhere.
5.1.6. CPU Utilisationhe graphs below show the CPU utilisation on the EC2 boxes over the lifetime of the project. As indicated by the figure, the CPU may have been stressed to capacity with 100% utilisation during the map phase but the pattern seen in the initial portion of the graph is very erratic and shows a high oscillation between full and partial CPU utilisation. However, during the shuffle phase, it is quite evident that the CPU utilisation drops below the 50% threshold, thus the CPU utilisation cannot be the limiting factor in the case of shuffle phase of a job.
6. Unresolved Issues6.1 Maximum network bandwidth of EMRAmazon does not provide maximum network bandwidth rates for EMR. The network performance is only described in qualitative terms (Low, Moderate and High). We measured the network bandwidth between 2 nodes using the pathtest application from AppNeta [8]. We found a peak transfer rate of 753Mb/s.
We surveyed existing literature for the same, and found two conflicting versions:1. “The available bandwidth is still 1 Gb/s, confirming anecdotal evidence that EC2 has full bisection bandwidth" Opening Up Black Box Networks with CloudTalk, by Costin Raiciu et al
2. “The median TCP/UDP throughput of medium instances are both close to 760 Mb/s" The Impact of Virtualization on Network Performance of Amazon EC2 Data Center, by Guohui Wang et al
Further, the Terasort job maxed out at 800 Mb/s during our runs, whereas the Ranked Inverted Index crossed 1Gb/s. We are not able to reconcile these results, and believe it needs further investigation.
6.2 Sorting phase on Ranked Inverted IndexThe sorting phase of the Ranked Inverted Index lasts for a very short duration. In fact it is not noticeable on the network CDF graph. But from the Disk activity graph it can be seen that there is write heavy activity between the 300400 seconds mark, which correspond to a period of low network activity. It should be investigated why the network transfer does not flatline during this period for this job, whereas it does in the case of Terasort.
7. SummaryWe have shown that both for the Terasort and Ranked Inverted Index jobs, the shuffle phase of MapReduce can constitute a large fraction of the overall job runtime (nearly 30%). We infer that for this phase, the network is potentially a bottleneck, as there is low activity on CPU, disk and memory. A better understanding of EMR’s network performance can lead to more a conclusive result in this regard.
8. Future WorkApart from the issues raised in Section 6, we see other avenues of investigation for this project.The experiments should be run on different kinds of hardware, and different values for certain Hadoop parameters. The important Hadoop parameters to consider are: io.sort.mb, io.sort.factor, and fs.inmemory.size.mb.
Jobs should be investigated with and without the presence of Combiners, which help to reduce the amount of data shuffled. The number of map and reduce tasks can be varied to see if that has an impact on the results.Also, the locality of the tasks is an important factor to be considered while evaluating Hadoop
jobs eg a racklocal or machine local setup may perform better.There is scope for extensive work to determine the topology and bandwidth expectations of Amazon’s EMR clusters.
9. References1. Mosharaf Chowdhury, Matei Zaharia, Justin Ma, Michael I. Jordan, Ion Stoica. Managing Data Transfers in Computer Clusters with Orchestra in SIGCOMM ’112. M. AlFares, S. Radhakrishnan, B. Raghavan, N. Huang, and A. Vahdat. Hedera: Dynamic flow scheduling for data center networks. In NSDI, 20103. A. Greenberg, J. R. Hamilton, N. Jain, S. Kandula, C. Kim, P. Lahiri,D. A. Maltz, P. Patel, and S. Sengupta. VL2: A scalable and flexible data center network. In SIGCOMM, 20094. Tarazu: Optimizing MapReduce on Heterogeneous ClustersFaraz Ahmad, Srimat T. Chakradhar, Anand Raghunathan, T.N. Vijaykumar5. Opening Up Black Box Networks with CloudTalk, by Costin Raiciu et al 6. The Impact of Virtualization on Network Performance of Amazon EC2 Data Center, by Guohui Wang et al7. MapReduce: Simplified Data Processing on Large Clusters, by Jeffrey Dean and Sanjay Ghemawat8. AppNeta pathtest http://www.appneta.com/resources/pathtestdownload.html9. Apache Hadoop http://hadoop.apache.org/