Source-linked AI summary
Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters
Hao Zhang, Zeyu Zheng, Shizhen Xu, Wei Dai, Qirong Ho, Xiaodan Liang, Zhiting Hu, Jinliang Wei, Pengtao Xie, Eric P. Xing
TL;DR
Distributed deep learning can be slowed by large, bursty parameter synchronization despite GPUs’ high computational throughput. Poseidon overlaps communication with layer computation and chooses communication schemes by layer and cluster properties. It achieves near-linear scaling across frameworks and models, including 31.5x speedup on 32 TensorFlow nodes for Inception-V3.
Problem
Large DL models require distributed GPU training, but existing implementations can scale poorly because GPU throughput drives frequent synchronization of large parameter updates.
Method
Poseidon overlaps layer-level communication with computation and uses a hybrid scheme that selects parameter-server or sufficient-factor broadcasting per layer.
Results
31.5x speedup was achieved for Inception-V3 with TensorFlow on 32 nodes, a 50% improvement over original TensorFlow’s 20x speedup.
Takeaways & Limitations
Poseidon delivers near-linear speedups across multiple neural networks, datasets, frameworks, and computation engines, including under limited bandwidth.
Takeaways & Limitations
The evaluation assumes synchronous replication of model parameters and includes an implementation that parallelizes TensorFlow’s single-node version rather than using its distributed engine.
Abstract
from arXiv · showhide
Deep learning models can take weeks to train on a single GPU-equipped machine, necessitating scaling out DL training to a GPU-cluster. However, current distributed DL implementations can scale poorly due to substantial parameter synchronization over the network, because the high throughput of GPUs allows more data batches to be processed per unit time than CPUs, leading to more frequent network synchronization. We present Poseidon, an efficient communication architecture for distributed DL on GPUs. Poseidon exploits the layered model structures in DL programs to overlap communication and computation, reducing bursty network communication. Moreover, Poseidon uses a hybrid communication scheme that optimizes the number of bytes required to synchronize each layer, according to layer properties and the number of machines. We show that Poseidon is applicable to different DL frameworks by plugging Poseidon into Caffe and TensorFlow. We show that Poseidon enables Caffe and TensorFlow to achieve 15.5x speed-up on 16 single-GPU machines, even with limited bandwidth (10GbE) and the challenging VGG19-22K network for image classification. Moreover, Poseidon-enabled TensorFlow achieves 31.5x speed-up with 32 single-GPU machines on Inception-V3, a 50% improvement over the open-source TensorFlow (20x speed-up).
1 Introduction
Poseidon addresses poor distributed deep-learning scalability caused by large, bursty parameter synchronization. Its layered scheduling and hybrid communication design support framework-independent scaling across GPU clusters.
- Large DL models can take days to weeks to train on one GPU, motivating distributed GPU-cluster training.
- Distributed implementations can scale poorly because GPU throughput increases the frequency of network synchronization for large models.On VGG19-22K, open-source TensorFlow on 32 machines can be slower than a single machine.
- Large gradient matrices saturate bandwidth, while iterative updates arrive in bursts separated by low network utilization.
- Poseidon overlaps layer-level communication with independent computation and selects between parameter-server and sufficient-factor broadcasting schemes per layer.Selection accounts for layer mathematical properties and cluster configuration.
- 31.5x speedup was achieved for Inception-V3 on 32 nodes with TensorFlow, improving 50% over original TensorFlow’s 20x speedup.
- Poseidon maintains linear throughput scaling on 16 machines while reducing communication overhead through layer-specific method specialization.
2 Large-scale Deep Learning
Large-scale deep learning uses layered neural networks trained iteratively with data parallelism across workers. Shared parameters require communication, and GPU computation can make network synchronization the primary bottleneck.
- Neural networks are hierarchical models composed of many layers that transform inputs into predictions through successive representations.Networks may range from 5–10 layers to hundreds of layers.
- Training commonly alternates feed-forward and backpropagation passes, updating parameters from gradients until a stopping criterion is reached.
- Data parallelism partitions training data across workers, which compute local gradients that are aggregated to update shared parameters.
- Parameter Server: Parameter servers provide shared model access through worker updates, server-side aggregation, and a consistency mechanism.
- Single-node frameworks such as Caffe and Torch can be parallelized across distributed GPUs using parameter-server or sufficient-factor-broadcasting communication.
- GPU-based matrix computation produces gradients much faster than naive network synchronization can transfer them, bottlenecking distributed training.For AlexNet on eight nodes, the example requires transferring 840M float parameters per second.
3 Poseidon Design
Poseidon’s design exploits layer-wise independence in backpropagation to overlap communication with computation and selects communication methods per layer to reduce synchronization overhead.
- Wait-free backpropagation: Poseidon decomposes synchronization into independent per-layer operations, enabling communication to be scheduled separately rather than after the entire computation step.This targets the sequential communication pattern of naive distributed training.
- Wait-free backpropagation: Wait-free backpropagation overlaps each layer’s synchronization with independent backward computations after that layer’s gradients are generated.Communication can proceed concurrently with backward operations for lower layers without blocking them.
- Wait-free backpropagation: WFBP is especially beneficial when communication concentrates in upper fully connected layers while computation concentrates in lower convolutional layers.For VGG and AdamNet, the paper reports roughly 90% concentration for both communication and computation in these respective layer groups.
- Hybrid communication: Hybrid communication chooses among synchronization methods independently for each layer using layer properties and cluster configuration.The optimal method can be estimated before communication because neural-network structure is fixed during training.
- Hybrid communication: For a 4096 × 4096 fully connected layer on an 8-worker, 8-server cluster, SFB transfers approximately 3.7 million parameters per worker versus approximately 34 million with PS.Convolutional layers instead use PS when their updates are indecomposable and sparse.
- Hybrid communication: HybComm’s optimal communication choice varies with layer type, layer dimensions, batch size, and cluster size.The strategy dynamically adjusts methods when doing so reduces communication overhead.
4 Implementation
Poseidon is implemented as a communication library that integrates with existing deep-learning frameworks through coordinators, a key-value store, and client-side synchronization APIs.
- System architecture: Poseidon’s architecture comprises a coordinator, a shared-memory key-value store, and a client library for parameter communication.The coordinator maintains model and cluster configuration, while the key-value store supports parameter-server communication.
- System architecture: The coordinator’s BestScheme API selects the optimal communication scheme for a given layer using the hybrid-communication strategy.It exposes cluster information and layer-specific communication decisions to clients.
- Client library: The client library uses CPU threads and GPU streams to manage asynchronous memory movement, gradient transformations, updates, and synchronization jobs.Its Move, Send, and Receive APIs coordinate these operations across model replicas.
- Client library: Send is nonblocking and starts layer communication during backpropagation, while Receive obtains updated matrices or sufficient factors before Move applies them.Poseidon maintains bulk-synchronous consistency by waiting for all syncers and tracking update counts at the key-value store.
- Framework integration: Poseidon can be integrated into existing frameworks by inserting synchronization APIs between gradient generation and update application.The paper reports modifications of 150 LoC for Caffe and 250 LoC for TensorFlow.
5 Evaluation
Poseidon is evaluated across multiple neural networks, datasets, frameworks, bandwidth settings, and GPU configurations. The results show low integration overhead, near-linear scaling, improved GPU utilization, and communication benefits that are strongest for large models and constrained bandwidth.
- Evaluation scope: Poseidon achieves near-linear throughput speedups across multiple neural networks and frameworks on up to 32 GPU-equipped machines.The evaluation covers Caffe and TensorFlow with models including GoogLeNet, Inception-V3, VGG19, VGG19-22K, and ResNet-152.
- Caffe scalability: Poseidon-Caffe preserves single-node performance while outperforming Caffe+PS, which incurs memory-copy overheads.On one node, Poseidon-Caffe processes 257, 35.5, and 34.2 images per second for GoogLeNet, VGG19, and VGG19-22K, respectively.
- Caffe scalability: 29.5x speedup is achieved for VGG19-22K on 32 nodes with Caffe-WFBP plus HybComm, up from 21.5x with Caffe-WFBP.The improvement is strongest because the three fully connected layers contain 91% of model parameters.
- TensorFlow scalability: 30x speedup is achieved by Poseidon over 22x for TensorFlow-WFBP on 32 nodes, while GPUs remain busy for most of training.Poseidon addresses coarse parameter partitioning and reduces message sizes through HybComm.
- Multi-GPU and statistical performance: Poseidon extends to multi-GPU workers with minor modifications and reaches 32x and 28x speedups for GoogLeNet and VGG19 on 32 GPUs.The implementation aggregates gradients locally before sending full-matrix updates over the network.
- Bandwidth experiments: Near-linear speedup is achieved on 16 machines with 10GbE for VGG19 and VGG19-22K, whereas standard PS reaches only 8x for VGG19.Poseidon reduces communicated message sizes and can fall back to PS when that requires less communication.
6 Related Work
Prior distributed deep-learning systems use varied strategies, but limitations remain around framework specialization and communication bottlenecks on GPU clusters.
- CPU-oriented parameter-server systems do not address the more challenging setting of distributed GPU clusters.
- Coates et al. use model parallelism with specialized model structures and hardware requirements such as InfiniBand networking.
- Poseidon explicitly overlaps communication and computation through a client library, enabling use with non-graph-based frameworks.
- MXNet and TensorFlow do not address limited network bandwidth for large models with dense layers such as big softmax.
- GeePS addresses limited GPU memory but not limited network bandwidth, leaving its technique potentially complementary to Poseidon.
- SparkNet reports 4-5 times speedup with 10 machines, indicating less scalability than the results reported for Poseidon.
7 Conclusion
Poseidon is presented as a scalable, efficient communication architecture for large-scale deep learning on distributed GPUs. The paper reports linear speedups across diverse configurations, including up to 32 nodes and limited bandwidth, with favorable comparisons to Adam and Microsoft CNTK.
- Poseidon is a scalable and efficient communication architecture for large-scale deep learning on distributed GPUs.
- Poseidon delivers linear speedups using up to 32 nodes and limited bandwidth across varied neural networks, datasets, and computation engines.
- Poseidon compares favorably to Adam and Microsoft CNTK.