Source-linked AI summary
Revisiting Distributed Synchronous SGD
Jianmin Chen, Xinghao Pan, Rajat Monga, Samy Bengio, Rafal Jozefowicz
TL;DR
Distributed training must balance synchronous methods’ straggler delays against asynchronous methods’ stale gradients and associated accuracy concerns. The paper proposes synchronous optimization with backup workers, which uses a subset of workers while avoiding gradient staleness, and reports faster convergence and better test accuracies than asynchronous training. The authors conclude that this approach is viable and scalable, while noting ongoing work on other datasets and communication strategies.
Problem
Distributed training faces a trade-off between synchronous waiting for stragglers and asynchronous computation using stale gradients that can negatively affect test accuracy.
Method
The paper proposes synchronous stochastic optimization with backup workers, computing each mini-batch gradient from a subset of workers while avoiding gradient staleness.
Results
The proposed synchronous training method converges faster and reaches better test accuracies than asynchronous training.
Takeaways & Limitations
Synchronous stochastic optimization with backup workers is presented as a viable and scalable strategy for distributed training.
Takeaways & Limitations
The authors are still experimenting with different datasets and exploring communication improvements and time-outs as alternatives to backup workers.
Abstract
from arXiv · showhide
Distributed training of deep learning models on large-scale training data is typically conducted with asynchronous stochastic optimization to maximize the rate of updates, at the cost of additional noise introduced from asynchrony. In contrast, the synchronous approach is often thought to be impractical due to idle time wasted on waiting for straggling workers. We revisit these conventional beliefs in this paper, and examine the weaknesses of both approaches. We demonstrate that a third approach, synchronous optimization with backup workers, can avoid asynchronous noise while mitigating for the worst stragglers. Our approach is empirically validated and shown to converge faster and to better test accuracies.
1 INTRODUCTION
As distributed training becomes necessary for larger models and datasets, synchronous optimization can lose time to stragglers while asynchronous methods use potentially stale information. The paper revisits this trade-off and proposes synchronous optimization with backup workers, reporting faster convergence and better test accuracy than asynchronous training.
- Distributed training becomes necessary when datasets and deep-learning models exceed the practical scale of a single GPU.
- Synchronous stochastic optimization waits for the slowest machine in each batch, reducing practical speed.
- Asynchronous optimization avoids waiting by computing with potentially stale information, introducing gradient staleness.
- The proposed method synchronously computes each mini-batch gradient from only a subset of workers, mitigating stragglers without gradient staleness.
- The evaluation measures both convergence speed and test accuracy because faster convergence can reach a poorer local optimum in non-convex deep learning.
- The proposed synchronous method converges faster and reaches better test accuracies than asynchronous training.
2 ASYNCHRONOUS STOCHASTIC OPTIMIZATION
Asynchronous stochastic optimization lets workers update shared parameters independently, but this creates inconsistent reads and stale gradients. Experiments show that greater staleness, especially in deep models and at larger scale, degrades accuracy and training stability.
- Workers independently fetch parameters, compute mini-batch gradients, and send updates to parameter servers.
- Asynchronous updates can use inconsistent parameter reads and gradients computed from outdated parameters.Gradient staleness is measured by the number of updates occurring between a worker’s read and update operations.
- In an 18-layer Inception model, mean gradient staleness increased from ∼14.5 in the top layer to ∼39.0 in the bottom layer.
- The best MNIST CNN test classification error of 0.36% increased to 0.47% at 20-step average staleness and 0.79% at 50 steps.
- Staleness above 15 steps significantly deteriorated results and made training less stable, while larger scales could produce poorer trained models.
3 REVISTING SYNCHRONOUS STOCHASTIC OPTIMIZATION
Synchronous optimization avoids stale gradients but can waste time waiting for stragglers. Backup workers reduce this delay while preserving enough gradients for convergence, with N = 96 and b = 4 estimated fastest for 100 machines.
- Synchronous optimization: Synchronous optimization waits for gradients before updating, ensuring a true mini-batch stochastic gradient descent update without staleness.The effective batch size equals the sum of the workers’ mini-batch sizes.
- Straggler effects: Waiting for every worker makes update time depend on the slowest machine, whose delays can arise from hardware failures, resource contention, or preemption.
- Backup workers: Backup workers stop synchronization after gradients arrive from any N workers, dropping the slowest b gradients to mitigate stragglers.
- Straggler effects: With N = 100 workers and b = 0 backups, most mean collection times are 1.4s–1.8s, except for the final few gradients.The observed maximum aggregation time was 310s, although Figure 3 displays times only through 6s.
- Convergence trade-off: 137.5e3 to 76.2e3 iterations: doubling N from 50 to 100 nearly halves iterations to convergence, but increases the effective batch size.
- Convergence trade-off: N = 96, b = 4 converges fastest in the estimated 100-machine configuration, balancing shorter iterations against the need for more gradients.The estimate combines interpolated convergence iterations with mean iteration times.
4 EXPERIMENTS
The experiments compare convergence speed with final test quality, showing that Sync-Opt with backup workers outperforms Async-Opt on Inception and PixelCNN.
- Metrics of comparison: Convergence speed and final test quality are evaluated together because faster convergence can lead to a poorer local optimum.The comparison uses test error or accuracy and speed of convergence.
- Metrics of comparison: Lower initial learning rates converge faster but can produce poorer test accuracies or fail to reach higher precisions.With γ0 = 1.125, 75% precision is reached 1.5× faster than with γ0 = 4.5, but higher precisions are slower or unattained.
- Inception: Sync-Opt attains ∼0.5% better test precision than Async-Opt for comparable worker counts on Inception.The experiments vary N + b from 53 to 212 workers.
- Inception: Sync-Opt converges 6h and 18h faster than Async-Opt with 106 and 212 workers, respectively, but is 3h slower with 53 workers.The speed difference is largely due to fewer epochs and comparable or better epoch time.
- PixelCNN: Sync-Opt achieves lower negative log likelihood than Async-Opt on PixelCNN and reaches ϵ = 2.183 in < 13h instead of 40h.Async-Opt does not achieve this NLL and is outperformed even by serial RMSProp with one worker.
5 RELATED WORK
Related work spans asynchronous and synchronous distributed optimization, including methods that reduce but do not eliminate gradient staleness and analyses that optimize system configurations.
- Asynchronous and synchronous optimization: Prior work includes asynchronous optimization algorithms and implementations, alongside attempts to improve synchronous SGD.The cited approaches cover both algorithmic and systems perspectives.
- Softsync: Softsync batches gradients before asynchronous updates, reducing effective staleness while avoiding waits for the slowest worker.Unlike the proposed approach, softsync still permits stale gradients.
- Distributed systems: Other distributed systems avoid parameter servers or optimize configurations, but approaches like Async-Opt still suffer from staleness.Workers in Sync-Opt can also execute parameter-server updates in principle.
- Batch-size effects: The paper notes that its effective batch size grows linearly with N but did not observe the poorer-generalization effect reported for larger batch sizes.The authors attribute this to not yet being in the large-batch regime examined by prior work.
6 CONCLUSION AND FUTURE WORK
The paper argues that synchronous and asynchronous distributed optimization each have weaknesses, and presents backup-worker synchronization as a viable scalable strategy while identifying ongoing extensions.
- Conclusion: Distributed training becomes increasingly important as deep-learning datasets grow.
- Conclusion: Synchronous optimization suffers from stragglers, whereas asynchronous optimization suffers from gradient staleness.
- Conclusion: Synchronous stochastic optimization with backup workers is presented as a viable and scalable strategy.
- Future work: Future work explores datasets with sparse embedding layers, communication-overhead reductions, and time-outs as an alternative to backup workers.
A.1 MNIST CNN, SECTION 2.1
The MNIST CNN experiment uses a four-layer convolutional model with normalized weights and evaluates an exponential moving average of its parameters.
- Model: The model is a 4-layer CNN using 3x3 filters, max-pooling, and weight normalization in every layer.
- Training: Training uses SGD for 25 epochs with an initial learning rate of 0.1, annealed linearly to 0 during the last 10 epochs.
- Evaluation: Performance is evaluated on the exponential moving average θ̄ with decay rate α = 0.9999, using small image rotations and zooms for augmentation.
A.2 INCEPTION, SECTION 3.1
The Inception straggler experiments used a 10-parameter-server setup with one K40 GPU per worker and RMSProp-based training. Test precision was evaluated using an exponential moving average of the model parameters.
- The experiments trained Inception on ImageNet using 10 parameter servers, with each worker equipped with a K40 GPU.
- RMSProp with momentum used decay 0.9 and momentum 0.9, with mini-batches of size B = 32.
- Test precision was evaluated on the exponential moving average θ̄ using α = 0.9999.
A.3 INCEPTION, SECTION 4.2
The Inception comparison used different parameter-server counts for three worker configurations and matched learning-rate decay across asynchronous and synchronous optimization. Synchronization also enabled pipelined layer updates without gradient clipping.
- For N + b = 53, 106, and 212 workers, the experiments used 17, 27, and 37 parameter servers, respectively.
- Synchronous training avoided gradient clipping, enabling concurrent parameter updates for top layers while computing lower-layer gradients.
- Async-Opt used initial learning rate 0.045, while Sync-Opt used the rule-of-thumb 0.045N for this model.
- Both methods used exponential learning-rate decay with β = 0.94, scaled so rates were comparable after processing the same number of datapoints.
- Test precisions were evaluated on the exponential moving average θ̄ using α = 0.9999.
A.4 PIXELCNN, SECTION 4.3
The PixelCNN experiments trained on CIFAR-10 across worker configurations from one to sixteen, using one backup worker for synchronous optimization. They used RMSProp with momentum and a gradually reduced learning rate.
- PixelCNN was trained on CIFAR-10 with N + b = 1, 8, and 16 workers, each using a K80 GPU and 10 parameter servers.
- Synchronous optimization always used b = 1 backup worker.
- RMSProp with momentum used decay 0.95 and momentum 0.9, with mini-batches of size B = 4.
- The initial learning rate γ0 was 1e-4 and decreased to 3e-6 after 200,000 iterations.