Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training

Saptadeep Pal; Eiman Ebrahimi; Arslan Zulfiqar; Yaosheng Fu; Victor Zhang; Szymon Migacz; David Nellans; Puneet Gupta

Source-linked AI summary

Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training

Saptadeep Pal, Eiman Ebrahimi, Arslan Zulfiqar, Yaosheng Fu, Victor Zhang, Szymon Migacz, David Nellans, Puneet Gupta

arXiv:1907.13257v1Jul 30, 2019cs.LGcs.AIcs.DCstat.ML

TL;DR

Large models and datasets make training slower, while data parallelism eventually suffers from communication overhead and statistical-efficiency loss as device counts and global batch sizes grow. This paper combines data and model parallelism and develops a framework for selecting the crossover between them. Hybrid training is projected to outperform DP alone at scale, with speedups of at least 26.5%, 8%, and 22% for Inception-V3, GNMT, and BigLSTM.

Problem
Data parallel training scales poorly beyond a network-specific device and global-batch-size scale because synchronization overhead rises and statistical efficiency declines.
Method
The paper combines data parallelism with model parallelism and develops an analytical framework to identify the device-count crossover for a given model and system.
Results
Hybrid training is projected to provide at least 26.5%, 8%, and 22% end-to-end speedups over DP-only training at scale for Inception-V3, GNMT, and BigLSTM, respectively.
Takeaways & Limitations
When DP inefficiencies become large, combining MP with DP can continue improving training time beyond DP alone.
Takeaways & Limitations
Exact DLPlacer speedup predictions are difficult because framework-induced overheads and unmodeled operating-system effects are not fully modeled.

Abstract

from arXiv · show

Deploying deep learning (DL) models across multiple compute devices to train large and complex models continues to grow in importance because of the demand for faster and more frequent training. Data parallelism (DP) is the most widely used parallelization strategy, but as the number of devices in data parallel training grows, so does the communication overhead between devices. Additionally, a larger aggregate batch size per step leads to statistical efficiency loss, i.e., a larger number of epochs are required to converge to a desired accuracy. These factors affect overall training time and beyond a certain number of devices, the speedup from leveraging DP begins to scale poorly. In addition to DP, each training step can be accelerated by exploiting model parallelism (MP). This work explores hybrid parallelization, where each data parallel worker is comprised of more than one device, across which the model dataflow graph (DFG) is split using MP. We show that at scale, hybrid training will be more effective at minimizing end-to-end training time than exploiting DP alone. We project that for Inception-V3, GNMT, and BigLSTM, the hybrid strategy provides an end-to-end training speedup of at least 26.5%, 8%, and 22% respectively compared to what DP alone can achieve at scale.

1. Introduction

The paper examines when data parallelism (DP) stops scaling effectively and argues that combining it with model parallelism (MP) can continue reducing end-to-end training time. It introduces an analytical crossover framework, DLPlacer, and projects hybrid speedups across three networks.

1. Introduction: DP scaling degrades because larger global batches reduce statistical efficiency and more devices increase gradient synchronization overhead.Beyond a network-specific batch-size threshold, substantially more iterations may be needed to reach the desired accuracy.
1. Introduction: MP splits a model’s dataflow graph across devices processing the same mini-batch, but communication and limited available parallelism make effective placement difficult.Activation and gradient communication between dependent operations must be considered.
1. Introduction: At scale, hybrid parallelization further scales multi-device training when DP inefficiencies become large.Each data-parallel worker is model-parallelized across multiple devices.
1. Introduction: The analytical framework identifies the device-count crossover at which MP should be combined with DP for a particular model and system.Each network has a unique scale where DP degradation can be overcome by MP speedup.
1. Introduction: At least 26.5%, 8%, and 22% end-to-end speedups are projected for Inception-V3, GNMT, and BigLSTM, respectively, over DP-only training at scale.The comparison covers three distinct deep learning networks.
1. Introduction: DLPlacer uses integer linear programming to find operation-to-device placements that maximize MP speedup.For Inception-V3, its predicted two-GPU placement was within 6% of the measured 1.32x MP speedup.

2. Background

The background distinguishes data parallel training, which replicates model parameters across workers, from model parallel training, which distributes the dataflow graph across devices. It also describes pipeline parallelism as an implementation of model parallelism for sequential networks.

2. Background: Neural-network training forward-propagates a batch, back-propagates losses to compute gradients, and updates weights using the batch-average gradient.The process is described as stochastic batch gradient descent.
2. Background: Data parallelism replicates the full model on each worker, processes different input batches independently, and averages gradients before synchronized weight updates.The collection of worker mini-batches forms a global batch, and gradient sharing uses all-reduce communication.
2. Background: Model parallelism places different dataflow-graph operations on separate devices, enabling concurrent execution even when the whole model fits on one device.It was traditionally used when model parameters could not fit in a single device’s memory.
2. Background: Pipeline parallelism partitions sequential networks into device-assigned layer groups and processes smaller microbatches concurrently across devices.The paper treats pipeline parallelism as an implementation instance of model parallelism.

3. Decomposing End-to-End Training Time

End-to-end training time combines per-step time, steps per epoch, and epochs to convergence; DP and MP change these terms differently. Hybrid parallelism preserves DP’s batch-related terms while adding MP’s per-step speedup, but its benefit depends on communication, statistical efficiency, and model-specific scaling.

End-to-end training time: Training time is modeled as the product of average time per step, steps per epoch, and epochs required to converge.These factors are denoted T, S, and E, respectively.
Data Parallel Training: DP scaling efficiency falls because all-reduce communication makes the N-device step time exceed single-device step time.The ratio of single-device to N-device step time is called DP scaling efficiency.
Data Parallel Training: Larger DP device counts reduce steps per epoch but can increase epochs to convergence as global batch size grows and statistical efficiency declines.Beyond a network-specific batch-size threshold, convergence may require substantially more epochs even with hyper-parameter tuning.
Model Parallel Training: MP reduces per-step time without changing global batch size, steps per epoch, or convergence epochs, but achievable speedup is limited by model parallelism and communication costs.In hybrid training, grouping M devices per DP worker preserves the N-way DP global batch size and convergence terms while multiplying speedup by the MP factor.

4. Methodology

The evaluation methodology measures convergence, scaling, and model-parallel speedups across Inception-V3, GNMT, and BigLSTM using four-GPU systems and projected larger configurations. It combines measured epoch counts, modeled scaling efficiency, and two-way model-parallel implementations to estimate hybrid-training benefits.

Models and systems: The study evaluates Inception-V3, GNMT, and BigLSTM with their specified datasets, implementations, and training targets.GNMT is trained on WMT’16 German-English to BLEU 21.8, while BigLSTM uses the one-billion-word dataset to perplexity 67.
Models and systems: Experiments use NVIDIA DGX systems with four V100 GPUs connected by NVLink, with larger BigLSTM memory requirements accommodated by 32GB GV100 cards.Gradient sharing uses NCCL2.0 all-reduce.
Projection methodology: The projections measure convergence epochs and DP scaling efficiency across GPU counts, then combine them with MP speedups for two-GPU model-parallel workers.The methodology uses M = 2 for the evaluated models.
Projection methodology: Larger global batch sizes are emulated beyond the four-GPU system, while convergence behavior is evaluated under tuned and untuned learning-rate conditions.Even with tuning, convergence epochs increase rapidly beyond a certain global batch size.
Projection methodology: The analysis conservatively assumes DP scaling efficiency of 1 because larger-GPU DP efficiency cannot be measured directly on the four-GPU system.This assumes communication and synchronization overhead is negligible relative to forward and backward computation.
Model-parallel configurations: Inception-V3 uses traditional operation splitting, whereas GNMT and BigLSTM use pipeline parallelism because their implementations provide optimized libraries and fused RNN kernels.Splitting beyond two ways provides marginal per-step speedup for these configurations because of kernel overheads and pipeline imbalance.

5. Evaluation

The evaluation finds that increasing global batch size can sharply increase convergence epochs and limit DP-only scaling. Across the three networks, projected hybrid MP-DP training outperforms DP-only training at network-specific scales.

Inception-V3: Inception-V3’s convergence requirement rises from 7 epochs beyond global batch size 2048 to 23 epochs at batch size 16384, corresponding to 32 and 256 GPUs.This sharp epoch increase makes DP-only speedup saturate beyond 32 GPUs.
Overall results: At scale, hybrid parallelization is projected to beat DP-only training by at least 26.5% for Inception-V3, 8% for GNMT, and 22% for BigLSTM.These projections compare hybrid training with DP-only training at the evaluated large-scale configurations.
Inception-V3: At 256 GPUs, the Inception-V3 hybrid strategy is projected to outperform DP-only training by at least 26.5%.Using additional devices for two-way MP is already projected to improve over DP-only by at least 15.5% when moving from 32 to 64 GPUs.
GNMT: GNMT scales well with DP initially, but its DP-only speedup slows beyond 64 GPUs and drops sharply from 128 to 256 GPUs.A 2-way MP and 128-way DP hybrid configuration outperforms 256-way DP by 8%.
BigLSTM: BigLSTM stops scaling well beyond 16 GPUs because increasing global batch size rapidly reduces statistical efficiency and raises the required epoch count.The hybrid policy provides a 1.22x speedup over the best DP-only scale, which occurs at 16 GPUs.
Interpretation: The projected hybrid gains would be larger under realistic DP scaling-efficiency losses, which are often below 0.9 for large LSTM networks.The evaluation conservatively assumes SEN = 1.

6. Maximizing MP Performance

DLPlacer uses integer-linear programming to assign model operations, schedules, and communication routes across hardware. Its Inception-V3 placement predictions closely match silicon measurements and expose substantial two-GPU model-parallel speedup.

DLPlacer objective: DLPlacer maximizes model-parallel speedup by assigning DFG operations to devices while minimizing communication overhead.The ILP extracts operation parallelism and optimizes resource utilization across the hardware graph.
Inputs and scope: DLPlacer predicts training speedup from profiled operation execution times, memory footprints, dependency edges, and communication data volumes.The model can support finer-grained operation splitting, although that requires framework support and is outside this work’s focus.
Optimization formulation: The ILP maps DFG vertices to compute nodes, dependency edges to physical routes, and operations to execution start times under placement, routing, scheduling, memory, and overlap constraints.The hardware graph includes compute nodes, routers, and physical links with bandwidth attributes.
Optimization formulation: Communication time is modeled from routed edge data, link bandwidth, latency, and the number of traversed links.The framework represents edge communication time as dependent on tensor volume and physical-link characteristics.
Inception-V3 case study: For Inception-V3, measured DLPlacer speedups are within 6% of silicon results, and a two-GPU placement achieves 1.32x speedup.That two-GPU result is nearly the optimal speedup obtained with three or four GPUs because the placement exploits the network’s limited parallelism.
Inception-V3 case study: DLPlacer finds placements in approximately 11–18 minutes on an 18-core Xeon-E5 system, avoiding the non-trivial task of identifying high-performing placements by manual inspection.Its placement objective effectively shortens the critical path among feasible solutions.

7. Related Work

Prior work addresses data- and model-parallel scaling through asynchronous updates, hyper-parameter methods, model averaging, alternative parallelization, and learned placement, but leaves strategy selection across device counts insufficiently systematic.

Hybrid parallelization: Prior hybrid-parallel studies report benefits, but do not systematically identify the best strategy at different device counts.Das et al. maintain global batch size on CPUs, while Yadan et al. show a 2-way DP, 2-MP approach outperforming MP-only and DP-only on AlexNet.
Placement strategies: DLPlacer’s Inception-V3 placement solution assigns operations across devices, while Figure 8 compares its estimated and experimentally obtained normalized model-parallel speedups.The placement directives are implemented with TensorFlow device assignment and validated against real hardware performance.
Alternative model-parallel strategies: Layer-wise and multidimensional tensor parallelism provide alternatives that can be combined with or used instead of model parallelism to improve data-parallel scaling.The cited work assigns individual parallelization strategies to CNN layers and explores combinations of tensor dimensions.
Alternate techniques to improve DP scaling: Asynchronous SGD, hyper-parameter tuning, and model averaging target synchronization or statistical-efficiency problems but introduce effectiveness, debugging, convergence, or framework-support limitations.These methods can require problem-specific expertise, suffer poor statistical efficiency, make debugging difficult, or require custom model reimplementation.
Placement strategies: Reinforcement-learning placement can speed model-parallel training, whereas DLPlacer supplies optimal placement solutions without a notion of optimality in the RL approach.DLPlacer can still be compute intensive for complex dataflow graphs and large system graphs.

8. Conclusion

The paper combines model and data parallelism to address data-parallel scaling and statistical-efficiency losses at large global batch sizes. It projects substantial hybrid-training speedups over data parallelism alone for three networks.

Conclusion: The proposed strategy combines model parallelism with data parallelism to overcome scaling and statistical-efficiency losses in data-parallel training.Model parallelism is used when growing global batch size causes data-parallel speedup to drop significantly.
Conclusion: Hybrid training is projected to improve end-to-end speed by at least 26.5% for Inception-V3, 8% for GNMT, and 22% for BigLSTM over DP alone at scale.These projections compare the hybrid strategy with data-parallel-only training.