Source-linked AI summary
HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism
Jay H. Park, Gyeongchan Yun, Chang M. Yi, Nguyen T. Nguyen, Seungmin Lee, Jaesik Choi, Sam H. Noh, Young-ri Choi
TL;DR
Large DNN training increasingly must use heterogeneous GPU clusters, including GPUs that cannot train the models alone. HetPipe combines pipelined model parallelism within virtual workers with data parallelism across them, adds WSP synchronization with a convergence proof, and achieves up to 49% faster convergence than Horovod in the reported setting.
Problem
The paper asks how to train large DNN models on heterogeneous GPU clusters when some GPUs cannot support training independently.
Method
HetPipe forms virtual workers from multiple heterogeneous GPUs, uses pipelined model parallelism within them and data parallelism across them, and synchronizes parameters with WSP.
Results
49% faster convergence is reported for VGG-19 and 39% faster convergence for ResNet-152 than Horovod, using all GPUs including whimpy ones.
Takeaways & Limitations
HetPipe makes large-DNN training possible with aggregated heterogeneous GPUs and reports convergence of the resulting system.
Takeaways & Limitations
HetPipe uses parameter servers that may incur synchronization and communication overhead, and decentralized optimization is left for future work.
Abstract
from arXiv · showhide
Deep Neural Network (DNN) models have continuously been growing in size in order to improve the accuracy and quality of the models. Moreover, for training of large DNN models, the use of heterogeneous GPUs is inevitable due to the short release cycle of new GPU architectures. In this paper, we investigate how to enable training of large DNN models on a heterogeneous GPU cluster that possibly includes whimpy GPUs that, as a standalone, could not be used for training. We present a DNN training system, HetPipe (Heterogeneous Pipeline), that integrates pipelined model parallelism (PMP) with data parallelism (DP). In HetPipe, a group of multiple GPUs, called a virtual worker, processes minibatches in a pipelined manner, and multiple such virtual workers employ data parallelism for higher performance. We also propose a novel parameter synchronization model, which we refer to as Wave Synchronous Parallel (WSP) to accommodate both PMP and DP for virtual workers, and provide convergence proof of WSP. Our experimental results on a given heterogeneous setting show that with HetPipe, DNN models converge up to 49% faster compared to the state-of-the-art DP technique.
1 Introduction
HetPipe addresses large-DNN training on heterogeneous GPU clusters, including GPUs too weak to train models independently, by combining pipelined model parallelism with data parallelism. It introduces WSP synchronization and reports faster convergence than state-of-the-art data parallelism.
- DNN models and training batches have grown, while heterogeneous GPUs are increasingly unavoidable because GPU architectures evolve rapidly.
- Aggregating lower-class GPUs can enable large-model training and add resources to higher-class systems through virtual workers.A virtual worker is a group of aggregated GPUs that satisfies the model’s resource constraints.
- HetPipe integrates pipelined model parallelism within each virtual worker with data parallelism across multiple virtual workers.
- HetPipe proposes Wave Synchronous Parallel, a parameter synchronization model adapted for heterogeneous virtual workers, and provides a convergence proof.
- 49% faster convergence is achieved for VGG-19 and 39% faster convergence for ResNet-152 than Horovod, using all GPUs including whimpy ones.The ResNet-152 model is too large to load into the four whimpy GPUs individually.
2 Background
The background distinguishes data and model parallelism, then motivates pipelined model parallelism as a way to improve utilization when layers are distributed across GPUs. Prior systems apply pipeline execution in homogeneous settings, while Figure 1 illustrates minibatch execution across partitions and GPUs.
- Parallelism fundamentals: Data parallelism replicates the DNN model across workers, while model parallelism partitions layers across GPUs for models too large for one GPU.
- Parameter synchronization: Bulk Synchronous Parallel waits for all workers, whereas Asynchronous Parallel permits stale weights; Stale Synchronous Parallel bounds that staleness.
- Model parallelism: Basic model parallelism sends activations forward and gradients backward between partition GPUs but leaves only one GPU actively executing at a time.
- Pipeline execution: Pipelined model parallelism feeds successive minibatches into the first GPU so multiple GPUs can process different minibatches concurrently.
- Prior pipeline systems: PipeDream and GPipe use pipelined model parallelism for single virtual workers in homogeneous settings, with different approaches to weights, microbatches, and updates.
- Pipeline execution: Figure 1 labels minibatch p in partition k as M_p,k, assigns it to GPU_k, and uses yellow and green to denote forward and backward passes.
3 System Overview
HetPipe trains large DNNs on heterogeneous GPU clusters by combining pipelined model parallelism within virtual workers with data parallelism across them. It uses Wave Synchronous Parallel to synchronize updates while accommodating pipeline and global staleness.
- HetPipe uses virtual workers that aggregate possibly heterogeneous GPUs, including resource-limited GPUs, to train large DNN models.Each virtual worker represents a training worker and can contain GPUs with different capabilities and memory capacities.
- Multiple virtual workers use data parallelism, increasing training parallelism beyond the GPUs assigned to one virtual worker.The architecture supports data parallelism through the virtual-worker abstraction.
- Each virtual worker processes minibatches with pipelined model parallelism to accommodate large models and improve GPU-resource utilization.Model parallelism divides processing across GPUs within a virtual worker, while pipelining keeps execution overlapping.
- Wave Synchronous Parallel synchronizes aggregated updates from minibatch waves rather than every minibatch, reducing communication overhead.Within a wave, minibatches can proceed without waiting for preceding minibatches, creating bounded local staleness.
- HetPipe mitigates parameter-server overhead through global staleness and continued pipeline processing while waiting for parameter updates.The paper notes that decentralized synchronization could provide further optimization but leaves it for future work.
4 Pipelined Model Parallelism Within a VW
Within each virtual worker, HetPipe partitions a DNN across heterogeneous GPUs and executes minibatches in an overlapped pipeline. The design balances memory constraints, execution time, and pipeline staleness.
- The pipeline processes up to Nm minibatches concurrently, with Nm constrained by the model’s memory requirements and the minimum capacity across virtual workers.The actual pipeline depth satisfies Nm ≤ Maxm and must be shared by every virtual worker.
- The partitioning algorithm assigns model layers to k GPUs while satisfying memory requirements and minimizing the maximum partition execution time.Execution time includes both computation and communication for activations and local gradients.
- GPU scheduling preserves minibatch order separately for forward and backward tasks, with FIFO scheduling among available tasks.Forward task p waits for earlier forward tasks, and backward task p waits for earlier backward tasks.
- Pipeline execution creates local weight staleness because later minibatches can begin before earlier minibatches finish updating weights.The local staleness threshold is slocal = Nm − 1, and larger Nm can improve pipeline throughput.
- Unlike PipeDream’s weight-stashing approach, HetPipe’s local staleness is inherent to pipelined execution within a virtual worker.The paper contrasts this with SSP, where staleness arises from different processing speeds among workers.
5 Data Parallelism with Multiple VWs
HetPipe extends data parallelism to virtual workers composed of heterogeneous, potentially resource-limited GPUs. Its synchronization scheme bounds global staleness while allowing waves and local pipelines to proceed asynchronously.
- Virtual workers enable data parallelism even when individual heterogeneous GPUs lack sufficient resources to train the DNN model.Aggregating multiple GPUs addresses the resource limitation that can prevent conventional worker-level data parallelism.
- WSP advances synchronization in clock units, where each clock unit corresponds to completing one minibatch wave.A wave contains slocal + 1 concurrently executed minibatches.
- Each virtual worker pushes aggregated updates once per wave, and the parameter server advances its global clock after all virtual workers contribute.This replaces per-minibatch update transmission with wave-level aggregation.
- WSP permits virtual workers to continue without retrieving global weights every wave, so local copies may be globally stale.The protocol bounds this staleness rather than requiring immediate synchronization after every wave.
- The maximum clock distance between virtual workers is bounded by user threshold D, while local pipelined processing can continue during synchronization waits.A worker may proceed without updates from up to D recent waves, subject to the bound.
- The global staleness bound sglobal combines the permitted wave delay with the local pipeline staleness.The paper defines sglobal as the maximum number of missing recent global updates allowed for minibatch execution.
6 Convergence Analysis
The convergence analysis represents WSP’s delayed updates through a noisy weight sequence and bounds its deviation from synchronized updates. Under stated convexity and bounded-distance assumptions, the paper derives a regret bound for distributed pipeline staleness.
- The analysis decomposes each noisy weight into guaranteed worker updates, recent local updates, and extra updates from other workers within the global staleness range.The decomposition uses sglobal and slocal + 1 to characterize WSP’s delayed-update structure.
- The reference and noisy weight sequences differ because pipeline and global staleness omit or include updates at different times.Rt contains missing reference updates, while Qt contains extra updates in the noisy sequence.
- Lemma 1 bounds the combined missing and extra update sets by (2sg + sl)(N − 1) and places their earliest index at max(1, t − (sg + sl)N).These bounds quantify the update discrepancy induced by distributed pipeline staleness.
- The convergence proof assumes convex component functions with bounded subgradients and bounded distances between parameter vectors.The assumptions introduce constants L > 0 and M > 0 for these bounds.
- Theorem 1 uses a diminishing step size ηt = σ/√t with σ determined by M and L to bound regret for the noisy distributed pipeline updates.The theorem compares the noisy-update parameter with the parameter learned from synchronized updates.
- The paper states that its theoretical results resemble prior non-pipelined staleness analyses while incorporating distributed pipeline staleness through Lemma 1 and Theorem 1.
7 Partitioning Algorithm
HetPipe profiles heterogeneous GPUs and communication costs, then uses constrained optimization to partition DNN layers for pipeline performance while supporting concurrent minibatches.
- Partitioning Algorithm: Layer execution and memory usage are profiled on each GPU type to model heterogeneous partition costs.Communication time is estimated from layer data sizes and PCIe bandwidth, adjusted by a measured scaling-down constant.
- Partitioning Algorithm: The partitioning objective minimizes the maximum partition execution time while satisfying each partition’s memory requirement.The memory constraints support Nm concurrent minibatches in the pipeline.
- Partitioning Algorithm: CPLEX solves the partitioning optimization problem using memory requirements as constraints.The optimizer divides model layers across the allocated heterogeneous GPUs.
8 Experimental Results
Experiments evaluate HetPipe across heterogeneous GPU allocations, single and multiple virtual workers, and convergence settings. Results show that allocation, communication placement, concurrent minibatches, and added whimpy GPUs materially affect throughput and convergence.
- Methodology: The 16-GPU testbed uses four nodes with four homogeneous GPUs per node, each node containing a different GPU type.HetPipe is implemented by modifying TensorFlow 1.12 and evaluated on two DNN models.
- Methodology: NP minimizes intra-worker communication but creates heterogeneous worker performance, ED equalizes workers but increases communication, and HD combines the two strategies.These policies determine how GPUs are grouped into virtual workers.
- Single virtual worker: As Nm increases, normalized throughput and maximum GPU utilization generally increase, although the effect depends on allocation policy and DNN model.Some larger-Nm configurations cannot run because GPU memory is insufficient.
- Multiple virtual workers: For VGG-19, Horovod outperforms NP, ED, and HD, whereas ED and HD perform similarly to Horovod for ResNet-152.The comparison uses VGG-19’s 548MB parameter size and ResNet-152’s 230MB parameter size.
- Multiple virtual workers: ED-local achieves 1.8 times Horovod’s performance for VGG-19 and 40% higher throughput for ResNet-152.ED-local reduces cross-node transfer for VGG-19 and processes many minibatches concurrently.
- Multiple virtual workers: Adding whimpy GPUs increases performance, with HetPipe reaching up to 2.3 times speedup through more concurrent minibatches.The experiments compare HetPipe and Horovod across different heterogeneous GPU sets.
- Convergence: HetPipe converges 35% faster than Horovod with the same GPUs and 39% faster after adding four whimpy GPUs for ResNet-152.The comparison uses top-1 accuracy and evaluates HetPipe with 12 and 16 GPUs.
- Convergence: For VGG-19, D=4 makes convergence 49% faster than Horovod, while D=32 degrades convergence by 4.7% relative to D=4.Increasing D reduces synchronization waiting but can increase global staleness.
9 Related Work
Related work covers pipelining, decentralized heterogeneous training, and combinations of data and model parallelism. HetPipe differs by integrating pipelined model parallelism with data parallelism for heterogeneous GPUs and large models.
- Pipelining: Prior pipelining systems hide communication, address backpropagation cost, or mitigate staleness in machine-learning training.PipeDream and GPipe provide detailed comparison points for HetPipe.
- Heterogeneous training: Decentralized heterogeneous training methods study heterogeneous environments but do not integrate data parallelism with pipelined model parallelism for large models.The cited methods include decentralized parameter updates, bounded staleness, and backup workers.
- Parallel training: Earlier systems combine data and model parallelism or use model parallelism, but the cited work does not jointly address pipelining and heterogeneous GPUs in the same way.Project Adam, Pal et al., STRADS, and FlexFlow represent related parallelism approaches.
10 Conclusion
HetPipe integrates pipelined model parallelism with data parallelism across virtual workers composed of heterogeneous GPUs, enabling efficient training of large DNN models. The paper proves convergence and reports fast convergence with HetPipe.
- HetPipe integrates pipelined model parallelism with data parallelism.
- Each virtual worker consists of multiple, possibly whimpy, heterogeneous GPUs.
- HetPipe makes it possible to efficiently train large DNN models and converges quickly.
A Appendix
The appendix analyzes regret for the training procedure under convexity, Lipschitz, and bounded-distance assumptions. It derives an upper bound after T iterations and uses the bound to establish convergence.
- The analysis assumes convex component functions and a minimizer w∗ of the convex objective.
- The regret analysis accounts for global staleness sg and local staleness sl.
- The proof uses a regret bound after T iterations under Lipschitz and bounded-distance assumptions.
- The appendix states that the expected objective difference approaches zero, establishing convergence.
- The proof bounds each term in the regret expression using an auxiliary lemma and the divergence D(w∥w′) = 1 2∥w−w′∥2.