Source-linked AI summary
How to scale distributed deep learning?
Peter H. Jin, Qiaochu Yuan, Forrest Iandola, Kurt Keutzer
TL;DR
Distributed deep-learning training can be a major workflow bottleneck, while synchronous and parameter-server SGD have different scaling weaknesses. The paper compares these approaches on ResNet/ImageNet and proposes gossiping SGD. Asynchronous methods converge faster at fewer nodes, whereas all-reduce scales more reliably to larger node counts.
Problem
Deep-learning training can dominate workflow time, while synchronous SGD has synchronization weaknesses and parameter-server SGD has communication bottlenecks.
Source: §1 · p. 1Method
The paper compares synchronous all-reduce, elastic averaging, and asynchronous gossiping SGD for ResNet training on ImageNet.
Source: §1 · p. 2, §3.1 · p. 3Results
Up to 16–32 nodes asynchronous methods converge faster, while all-reduce more consistently reaches high accuracy at scales up to 100 nodes.
Source: §5 · p. 7Takeaways & Limitations
Which distributed SGD approach converges faster depends on node count and step size.
Source: §1 · p. 2, §5 · p. 7Experiments cover a local GPU cluster and a GPU supercomputer; other platforms may yield somewhat different results.
Source: §5 · p. 7
Abstract
from arXiv · showhide
Training time on large datasets for deep neural networks is the principal workflow bottleneck in a number of important applications of deep learning, such as object classification and detection in automatic driver assistance systems (ADAS). To minimize training time, the training of a deep neural network must be scaled beyond a single machine to as many machines as possible by distributing the optimization method used for training. While a number of approaches have been proposed for distributed stochastic gradient descent (SGD), at the current time synchronous approaches to distributed SGD appear to be showing the greatest performance at large scale. Synchronous scaling of SGD suffers from the need to synchronize all processors on each gradient step and is not resilient in the face of failing or lagging processors. In asynchronous approaches using parameter servers, training is slowed by contention to the parameter server. In this paper we compare the convergence of synchronous and asynchronous SGD for training a modern ResNet network architecture on the ImageNet classification problem. We also propose an asynchronous method, gossiping SGD, that aims to retain the positive features of both systems by replacing the all-reduce collective operation of synchronous training with a gossip aggregation algorithm. We find, perhaps counterintuitively, that asynchronous SGD, including both elastic averaging and gossiping, converges faster at fewer nodes (up to about 32 nodes), whereas synchronous SGD scales better to more nodes (up to about 100 nodes).
1 Introduction
The paper compares distributed SGD methods because deep-learning training can dominate workflows, while synchronous and asynchronous approaches each face scaling weaknesses. It finds that which approach converges faster depends on node count and step size.
1 Introduction
- Training large deep-learning datasets can take weeks or more on one CPU/GPU system, making training time a major workflow bottleneck.The motivating self-driving-car example starts from at least 750 MB/s of gathered data.Source: §1 · p. 1
- Synchronous SGD is hindered by stragglers and failing nodes, whereas parameter-server SGD faces communication bottlenecks that slow convergence.The paper frames the comparison around weaknesses in both approaches at scale.Source: §1 · p. 1
- The study asks how synchronous and asynchronous convergence changes across training stages and node counts, a choice relevant to practitioners with finite resources.It compares methods on ResNet trained for ImageNet and includes asynchronous gossiping SGD.Source: §1 · p. 2
- Asynchronous SGD scales best at larger step sizes and smaller scales (up to around 32 distributed nodes); at smaller step sizes and larger scales, all-reduce consistently converges to the most accurate solution faster.Source: §1 · p. 2
2 Background
The background contrasts lock-step all-reduce with asynchronous parameter-server SGD, then introduces elastic averaging as a modified asynchronous approach that still uses a central server.
2 Background
- The notation distinguishes local parameters, center parameters, step size, momentum, node and iteration indices, and per-node versus aggregate minibatch sizes.The section uses θ for parameters, ˜θ for the center parameter, α for step size, and µ for momentum.Source: §2 · p. 2
2.1 Synchronous All-Reduce SGD
- Synchronous all-reduce alternates local gradient computation with collective gradient aggregation, whose barrier makes faster nodes wait for the slowest.The passage attributes the communication overhead chiefly to synchronization rather than gradient-transfer bandwidth.Source: §2.1 · p. 2
2.2 Asynchronous Parameter-Server SGD
- Asynchronous parameter-server SGD lets workers update independently, but all-to-one communication is limited by the server's reception bandwidth.Increasing delays between communication rounds to relieve the bottleneck greatly decreases convergence rate.Source: §2.2 · p. 3
2.3 Elastic Averaging SGD
- Elastic averaging modifies the stochastic-gradient objective to promote consensus with a center parameter, but retains the parameter-server communication bottleneck.Its consensus objective is related to ADMM's augmented Lagrangian; the paper selects it over earlier parameter-server methods based on published results.Source: §2.3 · p. 3
3 Gossiping SGD
Gossiping SGD replaces synchronous all-reduce aggregation with asynchronous-friendly gossip communication. The section describes pull and push variants and gives convergence bounds for pull-gossip under stated assumptions.
3.1 Algorithm
- Gossiping SGD replaces exact all-reduce aggregation with gossip communication, using a randomly selected peer to form pull-gossip or peer-directed exchanges for push-gossip.The paper also relates gossiping to elastic averaging by replacing the center parameter with the average of local parameters.Source: §3.1 · pp. 3–4
- Repeated gossip rounds reduce the diffusion potential at a fixed rate per round.Pull-gossip has each node receive one other node's parameter; push-gossip sends each node's parameter to one peer, which may receive from multiple nodes.Source: §3.1 · p. 4
3.2 Analysis
- Under strong convexity, Lipschitz gradients, and zero-mean bounded-variance noise, pull-gossip has a stated expected squared-error bound to the optimum; bounded gradients add a bound to the local-parameter mean.The theorem assumes processors can communicate with all other processors at each step.Source: §3.2 · p. 5
4 Experiments
The experiments compare distributed SGD methods for ResNet-18 on ImageNet across GPU platforms and node counts. Their results vary with scale and training phase: asynchronous methods can lead at smaller scales, while all-reduce has advantages at larger scales and after learning-rate annealing.
4.1 Implementation
- The implementation used MPI for communication and GPU compute libraries, with experiments spanning a 16-machine local cluster and a GPU supercomputer.The local cluster used one GPU per K80 machine; larger-scale tests used K20X GPU nodes on a Cray Gemini interconnect.Source: §4.1 · pp. 5–6
4.3 Results
- At p = 32, 64, and 128, gossiping iterations were faster than all-reduce iterations, but gossiping converged more slowly after step-size annealing as all-reduce showed a scaling advantage.Elastic averaging performed poorly in this environment, and pull-gossiping outperformed push-gossiping, so only pull-gossiping results were shown.Source: Figure 2 · p. 6, §4.3 · p. 7
4.2 Methodology
- The experiments trained ResNet-18 on ImageNet’s 1.28 million training images and evaluated center-crop validation loss and top-1 error.Training used multiscale image resizing, random 224 × 224 crops and flips, and pixelwise color noise.Source: §4.2 · p. 6
- Unless otherwise noted, experiments used initial learning rate α = 0.1, Nesterov momentum µ = 0.9, weight decay λ = 10^-4, and communication each iteration for all-reduce and gossiping.For aggregate minibatch size m = 256, the learning rate was annealed at 150k and 300k iterations; elastic averaging used β = 0.8/p.Source: §4.2 · p. 6
4 Experiments
- The first comparison tested all-reduce, elastic averaging, and push-gossiping at p = 8 and p = 16 with aggregate minibatch size m = 256.Figure 1 reports validation loss and top-1 error for per-node minibatch sizes b = 32 at p = 8 and b = 16 at p = 16.Source: Figure 1 · p. 5, §4.3 · p. 6
4.3 Results
- At p = 8, elastic averaging with τ = 10 performed ahead of the other methods; at p = 16, gossiping was faster than elastic averaging with τ = 10, and both beat all-reduce.At p = 8 all-reduce had practically no synchronization overhead and was as fast as gossiping; at p = 16 elastic averaging had trouble matching the other methods’ validation loss after annealing to α = 0.001.Source: §4.3 · pp. 6–7
- 68.7% top-1 validation accuracy was reached by all-reduce after over 1 million iterations, while elastic averaging often struggled to exceed 67% and gossiping did so when p > 32.At larger scales, asynchronous methods required more iterations to converge despite lower wall-clock time per iteration.Source: §4.3 · p. 7
5 Discussion
The discussion finds that asynchronous SGD can converge faster at smaller node counts, while all-reduce scales more consistently to larger counts. The conclusions are bounded by the tested platforms, supervised ImageNet setting, convolutional networks, and Nesterov-momentum SGD.
5 Discussion
- Up to around 32 nodes, asynchronous methods can converge faster than all-reduce with large step sizes; at small step sizes, gossiping can beat elastic averaging, while all-reduce is most consistent.The comparison concerns both early large-step and late small-step phases of training.Source: §5 · p. 7
- With more nodes, all-reduce can consistently reach high accuracy up to about 100 nodes, while asynchronous methods appear to plateau lower.The authors suggest that asynchrony and communication pattern, rather than communication amount, account for gossiping’s weaker scaling.Source: §5 · p. 7
- The comparison was tested on a local GPU cluster and a GPU supercomputer; CPU clusters, multi-GPU servers, and cloud instances may yield different results.The paper expects somewhat different results on these untested platforms.Source: §5 · p. 7
- The experiments cover supervised ImageNet classification with convolutional networks, so comparisons may differ in other learning settings or for networks dominated by fully connected operations.The paper specifically notes deep reinforcement learning and recurrent networks as differing settings.Source: §5 · p. 7
- The study distributed only SGD with Nesterov momentum; adaptive methods such as RMSProp and Adam may require additional considerations.Source: §5 · p. 7
6.1 Analysis
The analysis bounds how synchronous and asynchronous pull-gossip behave under strongly convex objectives with Lipschitz gradients and noisy gradient samples, examining both distance to the optimum and agreement among processors.
6.1 Analysis
- The analysis considers strongly convex functions with Lipschitz gradients, independently sampled zero-mean bounded-variance gradient noise, and a fully connected network.It bounds both local parameters’ deviation from the optimum and their deviation from the spatial mean.Source: §6.1 · p. 10
6.1.1 Synchronous pull-gossip algorithm
- In synchronous pull-gossip, each processor pulls parameter values from a uniformly random processor, producing a right-stochastic mixing matrix that need not be doubly stochastic.The random matrix has two entries per row: one on the diagonal and one in the pulled processor’s column.Source: §6.1.1 · pp. 10, 12
- Synchronous pull-gossip iterates converge to a ball around the optimum whose radius depends on processor count, step size, and gradient-noise variance.A decreasing step-size schedule αt ∼ O(1/pt) can achieve a convergence rate of O(1/t).Source: §6.1.1 · pp. 10, 12
6.1.2 Asynchronous pull-gossip algorithm
- Asynchronous pull-gossip updates one processor when its local Poisson clock ticks, with the master clock ticking at rate p.The update mixes that processor’s gradient-adjusted parameter with the parameter it pulls; the other processors’ parameters remain unchanged.Source: §6.1.2 · pp. 12–13
- Asynchronous pull-gossip has SGD-like convergence bounds with additional dependence on processor count; bounding disagreement also assumes uniformly bounded gradients.The analysis models updates with rate-1 local Poisson clocks and notes that the coordination needed to adjust the step size is unrealistic in practice.Source: §6.1.2 · pp. 12–14
6.2 Gossip variants
The section presents two gossiping SGD variants, distinguished by whether gossip uses parameters before or after the gradient step.
6.2.1 Gossip with stale parameters (gradient step and gossip at the same time)
- In the stale-parameter variant, gradient descent and gossip occur at the same time, and the distributed mean is replaced by an unbiased one-sample estimate from a uniformly selected processor.The sampled parameter is selected uniformly across the p processors.Source: §6.2.1 · p. 16
6.2.2 Gossip with fresh parameters (gradient step before gossip)
- In the fresh-parameter variant, the gradient step precedes gossip, with the distributed mean replaced by a one-sample estimate to derive the gossiping SGD update.Source: §6.2.2 · p. 16
6.3 Implementation details
Implementation details cover ImageNet image scaling and the ResNet-18 implementation used for training.
6.3 Implementation details
- The section provides additional details about the implementation of deep convolutional neural network training.Source: §6.3 · p. 16
6.3.1 ImageNet data augmentation
- The training pipeline resizes images on the GPU with CUDA linear and cubic interpolation, after preprocessing ImageNet images so their largest dimension is at most 480 pixels.Multi-scale training’s image-resizing overhead was a performance bottleneck even with multithreading and asynchronous data loading.Source: §6.3.1 · p. 16
6.3.2 ResNet implementation
- The ResNet-18 implementation achieved about 68.7% center-crop top-1 ImageNet validation accuracy, while an independent implementation reported 69.6% using the same evaluation methodology.The compared implementation used a more sophisticated data-augmentation scheme.Source: §6.3.2 · p. 16