Optimized Broadcast for Deep Learning Workloads on Dense-GPU InfiniBand Clusters: MPI or NCCL?

I Introduction and Motivation

Deep Learning (DL) is seeing a phenomenal growth spike where Computer Science (CS) and Machine Learning (ML) communities alike are rapidly adopting Deep Neural Networks (DNNs) to solve a huge array of classical problems. In fact, DNNs in specific, and DL, in general, has become one of the most disruptive technologies of its time. From Image Recognition to Speech Processing, various ML areas are being completely taken over by DNN-based approaches. With Neural Machine Translation (NMT) based approaches, several years of classical ML research has become obsolete in a matter of few years . Historically, machine learning experts used to create and tune features that captured the ML task (e.g. image recognition) by hand. However, DNNs have drastically changed this where the deep convolutional network automatically extracts these features in a layered fashion. At the core of DNN adoption are fast parallel hardware architectures like NVIDIA GPUs and more recently, the Tensor Processing Units (TPUs) by Google. In this context, the need for faster DNN training has pushed the current hardware, system software, and communication middleware to their limits.

DNN training is a compute intensive problem and many in the community have stressed the need for better exploiting HPC resources to deal with it . As DNN training becomes too large to be performed on a single GPU (or a single compute node), several DL toolkits and frameworks like Caffe , TensorFlow , and Microsoft Cognitive Toolkit (CNTK) have resorted to parallel/distributed training on multiple GPUs and compute nodes. To this end, there are new opportunities and challenges for the High-Performance Computing (HPC) community to rethink and enhance communication middleware like Message Passing Interface (MPI) and enable low-latency and high-bandwidth communication. The key idea in DNN training is that training parameters (or weights) need to be exchanged among parallel trainers (or optimizers) before the start of a training iteration. This can be achieved by using an MPI_Bcast on the training parameters. The details of DNN training are beyond the scope of this paper and can be reviewed from .

In this paper, we investigate efficient broadcast mechanisms used today to realize data parallel training in current DL frameworks and address the following research challenges:

What are the problems in adoption of MPI Collectives like Bcast for large scale DNN training?

Why is the DL community developing in-house and special-purpose communication runtimes like NCCL?

How can CUDA-Aware MPI runtimes be enhanced to better tackle large message communication requirements brought forward by DL applications?

Can general-purpose MPI runtimes like MVAPICH2 be optimized to better deal with diverse communication requirements for DNNs like LeNet, AlexNet, ResNet, and VGG?

How can we design an efficient and scalable MPI_Bcast implementation that can meet or surpass the performance of a special-purpose library like NCCL?

To address these challenges, we first highlight the design complexity of CUDA-Aware MPI runtimes like MVAPICH2-GDR and how it enables end applications to deliver the best performance across all message ranges for the candidate MPI_Bcast collective operation. To provide a holistic view, we compare the performance of proposed enhanced designs for MPI_Bcast, GPU-optimized NCCL Broadcast, and MPI+NCCL integrated MPI_Bcast using micro-benchmarks and real-world DL frameworks. Along with the broad goal of developing an efficient MPI_Bcast design for DL workloads, we make the following key contributions in this paper:

Discuss the emerging trends in DL community and how the HPC community can address these by advocating the adoption of MPI runtimes like MVAPICH2 for new application areas like parallel DNN training

Propose and design new MPI_Bcast algorithms that provide efficient GPU-based communication across all message sizes (DL models) and multi-GPU nodes of a dense GPU cluster

Provide performance insights for NCCL broadcast and its limitations for certain DL model sizes and system architectures along with solutions at the MPI runtime level that can overcome these limitations

Model and evaluate the performance of several algorithms and design techniques to realize an efficient MPI_Bcast that can provide better performance than NCCL for both intranode and internode communication

Illustrate application level benefits for the proposed designs using Microsoft CNTK and the popular VGG network.

With the proposed designs integrated into the MVAPICH2-GDR runtime, we report up to 14X and 16.6X lower latency (compared to NCCL-based solutions) for intra- and inter-node broadcast of GPU buffers. For large messages, our proposed designs achieve comparable performance to NCCL-based solutions. At the application level, proposed designs provide up to 7% improvement over NCCL-based solutions for data parallel training of the VGG network on 128 GPUs using Microsoft CNTK. To the best of our knowledge, this is the first study that presents pure MPI-based designs that provide comparable or better performance than NCCL-based solutions for Deep Learning workloads.

II The Design Space for GPU-based Collective Communication

We first discuss the emerging trend of dense multi-GPU systems like NVIDIA DGX-1 . Next, we describe the NCCL library and its importance for DL workloads. Finally, we explain the concept of CUDA-Aware MPI along with the limitations of NCCL integration in MVAPICH2-GDR.

Since GPU-enabled systems show promising capabilities to significantly accelerate training process for deep learning , higher number of GPUs is desired for the recent HPC systems. Thus, many dense multi-GPU systems have been introduced. Cray CS-Storm cluster supercomputer is equipped with InfiniBand interconnects and up to eight NVIDIA GPUs to enable high-performance computation and communication for traditional HPC applications as well as DL workloads. NVIDIA’s new DGX-1V with eight NVIDIA Tesla V100 GPUs and 300GB/s NVIDIA NVLINK2 interconnect provides 96X faster training than Intel Xeon CPU-only server for ResNet50 training . The next generation HPC systems, SUMMIT , will be equipped with multiple NVIDIA Tesla V100 GPUs, IBM Power9 processors, and InfiniBand EDR HCAs.

II-B NVIDIA NCCL for GPU-based Collective Communication

NCCL (pronounced Nickel) is a recent GPU-based collective communication library geared towards Deep Learning (DL) workloads. NVIDIA NCCL 1.3 is the latest open-source version available from the GitHub site . NCCL 2.0 announced at NVIDIA GTC will be released later this year. In this paper, we have used NCCL 1.3 for all the experiments and design discussions.

NCCL’s API closely resembles the MPI interface and provides communication primitives for broadcast, all-gather, reduce, reduce-scatter, and all-reduce. Precisely, NCCL’s goal is to provide fast communication of messages between GPUs in dense multi-GPU machines like the DGX-1. In this sense, NCCL is a special-purpose GPU-optimized collective communication library. However, the current publicly released version is limited to only a single shared-memory multi-GPU node where GPUs are attached to a PCIe level interconnect like a PLX switch. This offers the ability to optimize data exchange at very fine warp-level granularity as well as the exploitation of Peer-Access and CUDA kernels for copying the data instead of cudaMemcpy. Despite being similar to MPI, NCCL’s design goals and the target platforms are different. MPI is geared towards efficient communication across thousands of nodes in a cluster while NCCL is optimized for dense multi-GPU systems.

II-C CUDA-Aware MPI

In recent years, accelerators like NVIDIA GPUs are widely adapted by the modern HPC systems . As a result, extensions have been proposed for MPI runtimes to support efficient communication between GPUs. Initially, without the capability of direct access of GPU memory, MPI applications required explicit copying of GPU data to a staging buffer on the host memory in order to push the data to the network. Similarly, a data movement from CPU to GPU was needed after receiving the data on the host through an MPI_Recv operation. This significantly impacts performance as well as productivity. Thus, several MPI libraries including OpenMPI and MVAPICH2-GDR provide CUDA-Aware MPI primitives to transparently perform such copy operations. These CUDA-Aware MPI libraries significantly improve performance and productivity for the CUDA-enabled applications.

The internals of a CUDA-Aware MPI runtime are designed to have many optimized GPU-based point-to-point communication schemes such as staging, pipelining, CUDA IPC, and GPUDirect RDMA (GDR) to provide the best performance across various scenarios like intra-node, intra-socket, inter-node, and several other communication paths. This enables efficient designs for collectives like the MPI_Bcast for direct exchange of GPU-resident data.

II-D Limitations of NCCL-integrated MPI Designs

CUDA-Aware MPI runtimes like MVAPICH2-GDR are flexible enough to integrate third-party libraries like NCCL. In this context, we designed and evaluated NCCL-based MPI_Bcast designs in our earlier work . The hierarchical nature of collective communication in MVAPICH2 allowed us to exploit NCCL for intranode communication along with efficient and tuned designs for internode communication. However, there are certain limitations of integrating NCCL and its likes in MPI runtimes. These include dealing with CUDA-specific features like stream creation and management, and NCCL communicator creation and management in addition to the MPI communicators. In addition, for systems that do not have peer-access for GPUs, multiple NCCL communicators might need to be created to extract the best performance. In general, NCCL integration with MPI runtimes might lead to very complicated designs. Thus, the proposed work is a step towards achieving similar or better performance without utilizing NCCL.

III Performance Modeling of Existing Broadcast Algorithms

Broadcast algorithms have been well-studied in the literature over the decades . However, the emergence of accelerators such as GPUs has significantly changed this field of research. There is a need to re-evaluate existing algorithms as well as explore newer design techniques that take advantage of GPU-specific hardware features and mechanisms for realizing an efficient implementation of these algorithms. In this section, we model and analyze a few well-known broadcast algorithms to study their performance characteristics. Table I provides the notations used for the performance model throughout this paper.

We first analyze the most common broadcast algorithms in the literature that can be applied for the GPU-based broadcast. Note that the specialized broadcast schemes that require hardware assisted features like the InfiniBand hardware multicast are beyond the scope of this paper.

Direct Algorithm: A direct algorithm to broadcast the data from the root to all other processes is to simply use a serialized loop of point-to-point communication calls. In MPI, such an algorithm will essentially be a loop of MPI_Send and MPI_Recv calls. The approach can be modeled as:

However, this is not used in practice because of its poor scalability w.r.t to the number of nodes ‘n’.

The Chain algorithm is essentially an optimization of the direct algorithm. Instead of having a single sender, i.e., the root process, each successive recipient of the data becomes a sender for the next process in the chain. For rooted collectives like MPI_Bcast, the chain is a logical ring of communicating processes without a wrap-around between the last and first process in the chain. For non-rooted collectives, the wrap-around for the last and first process essentially makes it a ring.

Eq. 2 models the cost for the Chain algorithm.

Knomial/Binomial Tree Algorithm: As the scale of the HPC clusters becomes larger and larger, scalability with respect to the number of nodes becomes a critical performance issue. Thus, the tree-based approaches have been proposed to further improve the scalability of broadcast. In the Knomial tree-based approach, the set of communicating processes is considered as a logical tree where the root of the tree is the root of the Broadcast operation. In each communication step, root forwards data to one of its child nodes unless the data has already been sent to it. In each step, the non-leaf nodes, which have received data from its parent, follow the same procedure to forward data to their children nodes. In the Knomial tree, the root has at most $\lceil\log_{k}n\rceil$ children to maximize the overlap of communications. When K equals to two, it becomes the well-known binomial tree algorithm. The communication cost of Knomial tree-based approach can be represented as follows.

As can be seen in Eq. 3, the communication can be reduced to the scale of $O(\log_{k}n)$, which is significantly lower than the approaches mentioned previously, which are $O(n)$. As the tree-based algorithms improve the scalability of the broadcast operation, these are widely used in MPI runtimes and many other collective operations are built on top of it.

Scatter-Allgather Algorithm: The Scatter-Allgather algorithm is slightly non-intuitive yet very powerful when scalability with respect to the message size ‘M’ is of higher importance than scalability with respect to the number of nodes ‘n’. The scatter-allgather schemes to realize a broadcast have been proposed to improve the performance for large-message broadcast operations. The idea is to use a Scatter operation, followed by an Allgather operation to achieve a bandwidth-optimal broadcast. Typically, a binomial-tree based Scatter followed by a ring-based Allgather operation is used to realize the broadcast operation. The cost of this algorithm can be represented by Eq. III-A, which is described in .

However, efficient implementation of this algorithm may not be widely available as conventional HPC applications lie at the lower end of the spectrum with respect to the message size ‘M’.

III-B Discussion

Depending on the number of nodes and message size, these algorithms and corresponding implementations in MPI runtimes are judiciously applied to obtain optimal performance for a specific message range and node count. Despite the rich space of these algorithms, not all MPI implementations provide efficient designs for these algorithms especially when it comes to CUDA-Aware MPI runtimes.

IV Proposed Design Schemes for CUDA-Aware MPI_Bcast

We first discuss some advanced schemes and techniques that build on top of existing algorithms discussed in Section III. Next, we present the design of the proposed chain algorithm and how we enhance the optimization framework in MVAPICH2-GDR to exploit this new design. Finally, we provide an overview of the various designs and algorithms in MVAPICH2-GDR that enable us to achieve the best performance for the entire message range.

The cost functions presented in Section III-A always considered the transfer for the entire message in each communication step, except for the Scatter-Allgather algorithm. As the interconnect bandwidth has improved over the years, algorithms have been proposed to take better advantage of it by chunking and/or pipelining these transfers. To better utilize the network resources, i.e., to saturate the available bandwidth, pipelining schemes that divide the message into multiple smaller chunks need to be explored. Issuing multiple non-blocking point-to-point communication calls, i.e., MPI_Isend, MPI_Irecv, to allow overlap of these communications is one strategy to implement a pipelined broadcast that achieves better bandwidth utilization. Classically, the chain (or ring) algorithm has been considered an inefficient algorithm by MPI implementers. However, with the advent of Deep Learning applications, very large message transfers and a relatively smaller number of nodes (GPUs) are becoming a new use-case for MPI runtimes. Thus, the conventional intuition around the broadcast algorithms needs to be revisited.

IV-B Proposed Pipelined Chain Design

We propose a CUDA-Aware pipelined chain design for MPI_Bcast in MVAPICH2-GDR. To realize the pipelined chain design, the root process chunks the data and starts pushing the chunks to its right neighbor in the logical chain of processes. All non-root processes except the last process in the chain, receive several chunks from their left neighbor and forward it to their right neighbor. The last process in the chain receives the chunks from its left neighbor only and doesn’t need to forward them any further.

The cost model for the pipelined chain can be extended based on the earlier discussed chain algorithm from Section III.

Pipelining schemes theoretically yield lower communication costs; however, it is always non-trivial to select the proper chunk size that yields optimal performance for different message sizes and various architectures. For our implementation, we experimentally determine the optimal chunk size and allow the collective tuning infrastructure in the MVAPICH2-GDR runtime to select the correct chunk-size for best performance across a wide range of message sizes and process counts.

IV-C GPU-specific Optimizations for CUDA-Aware MPI_Bcast

Besides the traditional broadcast schemes mentioned in previous sections, many optimized schemes have been proposed specifically for GPU-based broadcast operation.

Host-staging Scheme: The performance issues with GDR reads for GPU-based point-to-point communications has been reported in . Thus, the direct broadcast of GPU buffers can suffer performance degradation for certain message ranges. A host-staging scheme is used to avoid these bottlenecks. Essentially, root first moves the data from GPU memory to the host memory. Then, a host-based broadcast is performed. On the receiver side, data can be either directly written to the GPU memory by using NVIDIA GDR feature or staged through the host.

A K-nomial tree design with host-staging can be modeled as follows.

Clearly, we can see that this design can only perform better when the first part of the equation, i.e., the time for copying the data to the host, doesn’t become the dominant part of the equation. For large and very-large messages, the staging can become very expensive. Hence, for the pipelined chain design, we do not perform host-staging and use direct mechanisms like CUDA IPC for intranode transfers when GPUs have peer-access and CUDA GDR for internode transfers when appropriate.

V Performance Analysis

We first provide the details of the evaluation platform we have used to perform all the experiments. Next, we provide a comprehensive performance comparison of several MPI_Bcast schemes in MVAPICH2-GDR, NCCL-integrated MVAPICH2, and NCCL using a micro-benchmark as well as a real DNN training framework called Microsoft Cognitive Toolkit (CNTK).

We used a Cray CS-Storm based GPU cluster called KESCH for our experiments. The cluster is located at the Swiss National Supercomputing Center. KESCH is a dense multi-GPU 12-node cluster. Each node has eight NVIDIA K-80 GK210GL GPUs. Thus, a total 16 CUDA devices per node are available. With a total of 192 GPUs for the 12 nodes and just 24 conventional CPUs, KESCH is a dense cluster. Each node has two InfiniBand FDR HCAs. Thus, KESCH is a multi-rail IB cluster that provides an opportunity for advanced designs in MVAPICH2-GDR to exploit both the HCAs.

V-B Intranode Performance Comparison (Micro-benchmark)

The goal of this section is to provide a comparison of performance for NCCL and MVAPICH2-GDR. We used a single node of the KESCH cluster and compare the performance of NCCL and MVAPICH2-GDR for 2, 4, 8, and 16 GPUs. For brevity, we compared the proposed tuned version of MVAPICH2-GDR (labeled as MV2-GDR-Opt) and NCCL (labeled as NCCL). We note that we exploit the ncclBroadcast (NCCL) to realize the intranode broadcast in the NCCL-integrated MPI_Bcast design. Thus, we only compare NCCL and MVAPICH2 in this section and omit the numbers for NCCL-integrated MPI_Bcast. However, we show NCCL-integrated MPI_Bcast numbers for the inter-node comparison in the next section (Section V-C).

Figure 1 shows the intra-node (up to 16 GPUs) performance using the CUDA-Aware osu_bcast benchmark. For small to medium messages, i.e., up to 8K bytes, we observed up to 14X, 10.6X, 9.4X, and 13X lower latency in MPI_Bcast of MVAPICH2-GDR compared to NCCL Broadcast for 2, 4, 8, and 16 GPUs, respectively. This is because of the advanced point-to-point designs available in MVAPICH2-GDR that allow efficient workarounds for various bottlenecks like the GDR read bottleneck across sockets . In addition, MV2-GDR-Opt exploits pipelined CUDA IPC designs for large messages to better utilize the available bandwidth. These optimized schemes cannot be done for special-purpose libraries like NCCL and hence we see this degradation for small and medium message ranges. For large and very large message range, we see that NCCL provides scalable performance. At the same time, our proposed pipelined chain designs in MVAPICH2-GDR allow us to achieve similar or better performance essentially alleviating the need to resort to NCCL augmented broadcast designs proposed in .

V-C Internode Performance Comparison (Micro-benchmark)

As mentioned earlier, NCCL 1.x series only works for a single node so we cannot directly compare the performance of NCCL for the internode case. However, our earlier work on NCCL-based broadcast allows us to provide a comprehensive internode performance comparison as well.

We compared two cases: the proposed tuned version of MVAPICH2-GDR (labeled MV2-GDR-Opt) and the NCCL-integrated MPI_Bcast (labeled NCCL-MV2-GDR). Several internode performance enhancements like the SGL-based designs that take advantage of IB features like Scatter-Gather lists help us provide excellent small message internode performance. As shown in Figure 2, MV2-GDR-Opt provides up to 16.4X and 16.6X improvement, respectively, on 64 and 128 GPUs, for the small and medium message range. For large and very large messages, the proposed designs are able to achieve comparable performance to the NCCL-based designs. Thus, it is clear that MPI_Bcast can provide the desired performance for the various type of workloads including DL workloads. To better illustrate the usefulness of these designs, we evaluate the performance of data parallel DNN training using Microsoft Cognitive Toolkit (CNTK). The details follow in the next subsection.

V-D Application Level Performance Comparison using CNTK

We now provide a comprehensive view of the benefits that can be observed for a real DNN training framework. There are several frameworks that can be utilized but we used a CUDA-Aware adaptation of the CNTK framework called CA-CNTK . CA-CNTK uses CUDA-Aware MPI_Bcast for the exchange of training parameters (or weights) throughout the training process. Figure 3 shows the performance comparison of training time using the VGG model for NCCL-integrated MVAPICH2 (labeled NCCL-MV2-GDR) and the proposed optimized version of MVAPICH2-GDR (labeled MV2-GDR-Opt). As can be seen from Figure 3, MV2-GDR-Opt yields 7% reduction of training time on 32 GPUs, and matches or beats the performance of NCCL-MV2-GDR for all other cases. This is because the broadcast operation used in VGG training uses a mix of message sizes including some small and medium and mostly large messages.

This result illustrates that an optimized MPI runtime can offer better performance than NCCL/NCCL-based approaches even when the main requirement of an application (the VGG model in this case) is large-message communication. VGG model due to its large number of parameters forces large-message communication. Thus, the performance benefits observed for MV2-GDR-Opt on micro-benchmarks in the small/medium message range do not exhibit significant benefits at the application level. However, it is pertinent to note that CNTK divides the communication based on the process count so the message-sizes can vary considerably and thus the proposed designs provide some performance benefit (7%) for VGG. We expect the benefits to increase for other models like GoogLeNet that have lesser number of parameters and thus a small/medium message communication requirement.

VI Related Work

An efficient broadcast design was presented by Liu et al. in that takes advantage of InfiniBand features like Hardware Multicast. The motivation of this work is similar to our proposed work, i.e. to optimize and redesign MPI_Bcast by exploiting new design schemes for newer architectures. Kandalla et. al presented designs to optimize MPI broadcast and reductions for Intel Many Integrated Cores (MIC) in .

Chu et al. proposed an IB-Multicast based Broadcast for streaming and DL applications. Zhou et. al proposed an optimized broadcast for large message sizes in . A slightly orthogonal effort to optimize intra-node communication by exploiting shared-memory programming and MPI one-sided features was presented in . Awan et al. described the NCCL-integrated MPI_Bcast designed for DL workloads. The main difference between NCCL-integrated MPI_Bcast and our proposed design is that it alleviates the need to use NCCL or any other external library to accelerate the intranode communication. Instead, we propose that efficient implementation of MPI_Bcast in the MVAPICH2 runtime can meet or exceed the performance offered by NCCL and its likes.

VII Conclusion and Future Work

As we enter the era of next-generation AI systems powered by Deep Learning and Deep Neural Networks, we see an increased exploration around communication runtimes. Industry developments like NVIDIA NCCL is a good example of such investigations. However, we propose that communication runtimes like MVAPICH2 and other MPI implementations provide an excellent and rigorous foundation that can be exploited to support communication with special requirements. As such, there are no fundamental limitations that should sway the researchers to not explore new designs in MPI runtimes to support DL workloads. In this paper, we present this case using an in-depth investigation of MPI_Bcast as a candidate collective that is important for parallel DNN training. Using micro-benchmark and a real DNN training framework, we highlighted that MPI_Bcast designed and tuned efficiently can provide up to 16.6X speedup at the micro-benchmark level and up to 7% improvement for training the VGG network on 128 GPUs using Microsoft CNTK. We plan to make these solutions available in the next MVAPICH2-GDR release. We also plan to extend this support for other collectives like MPI_Reduce and MPI_Allreduce to support the full spectrum of parallel DNN training.

Acknowledgment

This research is supported in part by National Science Foundation grants #CCF-1565414 and #CNS-1513120.

References