Source-linked AI summary
PipeDream: Fast and Efficient Pipeline Parallel DNN Training
Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, Phil Gibbons
TL;DR
Large DNNs expose communication and utilization limits in conventional data- and model-parallel training. PipeDream combines automated model partitioning with pipelining and selective data parallelism. Across five DNNs and two clusters, it is up to 5× faster in time to target accuracy than state-of-the-art approaches.
Problem
Growing DNN models make communication costly, while traditional model parallelism can underutilize compute resources and requires difficult partitioning.
Method
PipeDream combines model parallelism, aggressive minibatch pipelining, and selective data parallelism, using automated layer partitioning to balance computation and minimize communication.
Results
PipeDream is up to 5× faster in time to target accuracy than state-of-the-art approaches across five DNNs on two clusters.
Takeaways & Limitations
Pipeline-parallel training addresses communication overheads that bottleneck data-parallel training of very large DNNs.
Takeaways & Limitations
PipeDream’s default semantics exclude vertical synchronization because it requires more metadata to be stored at every pipeline stage.
Abstract
from arXiv · showhide
PipeDream is a Deep Neural Network(DNN) training system for GPUs that parallelizes computation by pipelining execution across multiple machines. Its pipeline parallel computing model avoids the slowdowns faced by data-parallel training when large models and/or limited network bandwidth induce high communication-to-computation ratios. PipeDream reduces communication by up to 95% for large DNNs relative to data-parallel training, and allows perfect overlap of communication and computation. PipeDream keeps all available GPUs productive by systematically partitioning DNN layers among them to balance work and minimize communication, versions model parameters for backward pass correctness, and schedules the forward and backward passes of different inputs in round-robin fashion to optimize "time to target accuracy". Experiments with five different DNNs on two different clusters show that PipeDream is up to 5x faster in time-to-accuracy compared to data-parallel training.
1 Introduction
Growing DNN models make conventional parallelization increasingly communication-bound and can leave model-parallel hardware underused. PipeDream addresses these limits by combining automated partitioning, pipelining, and selective data parallelism for faster training.
- Models with tens to hundreds of layers and 10–20 million parameters stress DNN training and cause common parallelization approaches to break down.
- Up to 85% of VGG-16 training time can be communication overhead, limiting the scalability of data-parallel training as models grow.
- Traditional model parallelism can severely underutilize compute resources because workers may operate one at a time or fail to overlap computation and communication.
- PipeDream injects multiple inputs to keep workers concurrently processing different minibatches, while using data parallelism for selected layer subsets to balance load.
- PipeDream reduces inter-worker communication by up to 95% relative to data-parallel training and speeds training by up to 6.76x across five evaluated DNNs.Reported speedups include 1.45x for Inception-v3, 5.12x for VGG16, 1.21x for Resnet-50, 6.76x for AlexNet, and 3x for S2VT.
- PipeDream automatically partitions layers into balanced, communication-minimizing stages and addresses uneven worker assignments and bidirectional training.
2 Background & Related Work
Distributed DNN training trades statistical and hardware efficiency under data or model parallelism. Communication stalls, resource underutilization, and difficult partitioning motivate PipeDream’s pipelined combination of parallelization strategies.
- DNN training time to target accuracy is the product of statistical efficiency, measured in epochs, and hardware efficiency, measured in time per epoch.
- Data Parallelism: Data parallelism replicates the model across GPUs and synchronizes weight updates, with communication proportional to model size.
- Data Parallelism: BSP preserves low weight staleness but forces GPUs to stall for other gradients, reducing hardware efficiency.
- Data Parallelism: Communication overhead increases with both the number of data-parallel workers and GPU compute speed, across all evaluated models.
- Data Parallelism: ASP reduces GPU idle time but can compute gradients using stale weights, lowering statistical efficiency.
- Model Parallelism: Traditional model parallelism assigns consecutive layer groups to machines, but only one stage is active for each minibatch at an instant.
- Model Parallelism: Model-parallel partitioning is often left to programmers, while reinforcement-learning placement methods are time- and resource-intensive and do not seamlessly combine parallelization forms.
- Related Work: Naive minibatch pipelining does not address the statistical efficiency, scale, and generality required by large real-world models.
3 Parallel Training in PipeDream
PipeDream combines pipelining, model parallelism, and selective data parallelism to keep GPUs productive while reducing communication and addressing partitioning, scheduling, and learning-consistency challenges.
- Pipeline Parallelism: Pipeline parallelism partitions consecutive model layers into stages mapped to GPUs, each performing forward and backward passes.Multiple minibatches are injected so different stages process different inputs concurrently.
- Pipeline Parallelism: >90% reduction in communication is reported for VGG16 because stages exchange layer outputs rather than all model parameters.Asynchronous activation and gradient transfers overlap with subsequent minibatch computation.
- Partitioning Layers Across Machines: PipeDream combines pipelined model parallelism with data parallelism for selected layer subsets to balance computation across machines.Its partitioning approach must jointly limit load imbalance and inter-stage communication.
- Partitioning Layers Across Machines: PipeDream profiles per-layer computation, activation size, and parameter size, then optimizes stage boundaries and replication factors across available machines.The profile uses a short run of 1000 minibatches; the partitioning dynamic program has total time complexity O(N^2M^2).
- Work Scheduling: The scheduler must choose between forward and backward work in a bidirectional pipeline, and PipeDream proposes a mechanism to avoid idle machines.The design targets high throughput while ensuring learning progresses.
- Effective Learning: Weight stashing ensures each minibatch uses the same parameter version for its forward and backward pass within a stage.Without weight stashing, the update is not a valid gradient for any single weight vector; vertical synchronization is omitted by default because it requires more metadata and has negligible experimental impact.
4 Implementation
PipeDream’s runtime profiles and partitions the model, assigns stages to GPUs, and manages the data, parameters, intermediate state, and synchronization needed for pipelined execution.
- Runtime workflow: PipeDream profiles the model and optimizes a partition into k stages, optionally replicating stages before assigning each stage to a GPU.Its input includes the model architecture, training dataset, and number of GPUs.
- GPU and stage state: The runtime initializes each stage’s layers and statically allocates GPU memory for activations, weights, gradients, and intermediate state.Intermediate state includes input activations and stashed weights for active minibatches.
- Execution: Workers process forward and backward minibatches through their assigned layers, with the input stage starting work and each machine following the 1F1B schedule.The ML worker retrieves pointers for inputs, parameters, outputs, gradients, and intermediate state through the PipeDream API.
- Parameter synchronization: For replicated stages, PipeDream copies weight updates to host memory, while a distributed parameter server synchronizes parameters for data-parallel stages.Gradients are communicated as soon as computed, rather than after all layers finish.
- State management: PipeDream retains forward-pass intermediate data until the corresponding stage’s backward pass completes, then releases backward-pass data after processing.Intermediate buffers are tracked by unique blob IDs.
- Fault tolerance and integration: The system supports periodic local checkpointing, uses ZeroMQ for inter-machine communication, and exposes a C++ library extensible to multiple ML frameworks.The current implementation uses Caffe and can also work with TensorFlow, MXNet, and CNTK.
5 Evaluation
Across two clusters and CNN, RNN, and sequence-to-sequence workloads, PipeDream generally outperforms data- and model-parallel alternatives, especially when communication bottlenecks scaling.
- Experimental setup: PipeDream’s evaluation covers ILSVRC12 and MSVD datasets on Titan X and V100 GPU clusters with different network bandwidths.The models include VGG16, Inception-v3, and S2VT, with additional throughput results for ResNet-50 and AlexNet.
- Configuration comparison: PipeDream’s best configurations usually combine pipelining, model parallelism, and data parallelism, outperforming any single strategy alone.The optimizer instead selects pure data parallelism for Inception-v3 with eight machines in Cluster-A.
- PipeDream vs. data parallelism: 95% communication reduction lets PipeDream improve VGG16 training by 7.04× over one machine and 2.99× over BSP on eight Cluster-A machines.VGG16’s BSP communication overhead is 72% in this setting.
- Cluster effects: On Cluster-B, PipeDream’s VGG16 speedup over BSP rises from 2.99× to 5.12×, while Inception-v3 training improves by 45% over BSP.Cluster-B combines faster V100 GPUs with a slower 10 Gbps interconnect than Cluster-A.
- Asynchronous comparison: PipeDream reaches 48% accuracy 7.4× faster than four-machine ASP data parallelism, despite ASP having no communication overhead.The passage attributes ASP’s disadvantage to poor statistical efficiency.
- Recurrent neural networks: For S2VT, PipeDream reduces communication overhead by 95% and achieves 3.34× speedup over single-machine training, compared with 3.01× over BSP.BSP communication overhead for four machines on Cluster-A is 70%.
6 Conclusion
PipeDream addresses communication bottlenecks in large-DNN data-parallel training by automatically partitioning and aggressively pipelining execution across workers.
- Conclusion: PipeDream is up to 5× faster in time to target accuracy than state-of-the-art approaches across five DNNs and two clusters.Its pipeline-parallel design targets communication overheads that bottleneck data-parallel training of very large DNNs.