Source-linked AI summary
Staleness-aware Async-SGD for Distributed Deep Learning
Wei Zhang, Suyog Gupta, Xiangru Lian, Ji Liu
TL;DR
Distributed ASGD is difficult to tune because removing synchronization introduces stale gradients and couples optimization hyperparameters with system design choices. The paper adjusts the learning rate per gradient according to staleness, proves SGD-consistent convergence, and reports near-linear speedup while maintaining convergence despite varying staleness.
Problem
ASGD’s asynchronous updates introduce gradient staleness, while interacting hyperparameters and system choices make accuracy and runtime difficult to tune.
Method
The paper modulates each ASGD gradient’s learning rate according to its staleness and analyzes the resulting algorithm theoretically.
Results
The method converges at the same rate as SSGD and achieves close to linear speedup for up to 30 learners while maintaining that convergence rate.
Takeaways & Limitations
Staleness-dependent learning rates provide a supported approach for maintaining model accuracy and convergence in distributed ASGD across CIFAR10 and ImageNet.
Takeaways & Limitations
The implementation assumes homogeneous processors and fast interconnects, hardware conditions that help bound gradient staleness.
Abstract
from arXiv · showhide
Deep neural networks have been shown to achieve state-of-the-art performance in several machine learning tasks. Stochastic Gradient Descent (SGD) is the preferred optimization algorithm for training these networks and asynchronous SGD (ASGD) has been widely adopted for accelerating the training of large-scale deep networks in a distributed computing environment. However, in practice it is quite challenging to tune the training hyperparameters (such as learning rate) when using ASGD so as achieve convergence and linear speedup, since the stability of the optimization algorithm is strongly influenced by the asynchronous nature of parameter updates. In this paper, we propose a variant of the ASGD algorithm in which the learning rate is modulated according to the gradient staleness and provide theoretical guarantees for convergence of this algorithm. Experimental verification is performed on commonly-used image classification benchmarks: CIFAR10 and Imagenet to demonstrate the superior effectiveness of the proposed approach, compared to SSGD (Synchronous SGD) and the conventional ASGD algorithm.
1 Introduction
Distributed training trades SSGD’s synchronization cost for ASGD’s gradient staleness, which complicates accuracy and learning-rate tuning. The paper proposes staleness-dependent learning rates, analyzes their convergence, and evaluates them on image-classification benchmarks.
- SSGD makes workers wait for the slowest participant, reducing scalability and runtime performance.
- ASGD removes explicit synchronization but introduces stale gradients, often producing worse models than SSGD at a fixed iteration count.
- Distributed-training accuracy depends on coupled choices including learning rate, mini-batch size, synchronization protocol, and learner count.
- The proposed ASGD variant divides the learning rate by each gradient’s staleness value.
- Experiments on CIFAR10 and ImageNet report that staleness-dependent learning rates are necessary for good model accuracy during distributed training.
- The paper builds on prior theory to prove a convergence rate for staleness-aware ASGD consistent with SGD.
2 System architecture
The system uses a parameter-server architecture with hardsync and n-softsync protocols, where n-softsync controls gradient staleness without explicit barriers. It modulates each incoming gradient’s learning rate using measured staleness while relying on hardware and communication choices that help bound delays.
- 2.2 Synchronization protocols: Hardsync, equivalent to SSGD, aggregates gradients from all learners before each update and provides an accuracy baseline at poor runtime performance.
- 2.1 Architecture Overview: The parameter-server system maintains global weights, accumulates learner gradients, and applies learning-rate-scaled updates.
- 2.1 Architecture Overview: Gradient staleness τ_i,l is the difference between the server’s current weight timestamp and the timestamp used to compute learner l’s gradient.
- 2.2 Synchronization protocols: N-softsync updates weights after collecting at least floor(λ/n) gradients, allowing learners to operate asynchronously without explicit synchronization barriers.
- 2.3 Implementation Details: MPI blocking communication, homogeneous processors, fast interconnects, and multiple learners per node help bound gradient staleness.
- 2.4 Staleness analysis: In hardsync, all gradients used for an update have staleness 0.
- 2.4 Staleness analysis: With λ = 30, average staleness remains close to n for 15-softsync and 30-softsync, while staleness generally lies between 0 and 2n.
- 2.2 Synchronization protocols: The implementation sets each incoming gradient’s learning rate using its staleness, with α0 typically taken from the best-known SSGD learning rate.
3 Theoretical Analysis
The analysis models staleness-aware ASGD as a generic optimization procedure and establishes convergence under assumptions and learning-rate conditions. It further relates convergence to staleness distribution and shows consistency with SGD’s rate, supporting linear speedup.
- The analysis studies convergence of the proposed ASGD algorithm and how gradient staleness affects its convergence rate.
- The objective is the average loss F(θ) over N samples, with θ as the parameter vector and f_i(θ) as the ith-sample loss.
- The distributed update procedure aggregates learner-computed mini-batch gradients at the parameter server, while τ_i,l records their staleness.
- Theorem 1 gives a convergence rate for the algorithm under commonly used assumptions and constants depending on F(θ).
- The convergence prerequisites can be met with a sufficiently small initial learning rate or sufficiently large iteration count, but the mini-batch size cannot be too large.
- For fixed total staleness, evenly distributed staleness yields better convergence, and the constant-staleness case has a rate consistent with SGD and supports linear speedup.
4 Experimental Results
Experiments on CIFAR10 and ImageNet evaluate runtime speedup and the effect of gradient staleness under hardsync and n-softsync protocols. Staleness-dependent learning-rate tuning preserves model accuracy across synchronization settings while enabling substantial speedup.
- Runtime Evaluation: 22x-28x speedup is achieved across CIFAR10 and ImageNet benchmarks with different batch sizes, while ASGD runs 50% faster than SSGD on average.Speedup is measured in training time per epoch for up to 30 learners.
- CIFAR10: CIFAR10 experiments use 30 learners and mini-batches of 4 per learner to compare hardsync with n-softsync protocols under increasing gradient staleness.The corresponding figure reports training and test error alongside the staleness-dependent learning rate.
- CIFAR10: With a fixed learning rate, CIFAR10 convergence progressively degrades as staleness increases, and 15- and 30-softsync fail to converge with 90% test error.Using the staleness-dependent scheme makes the error curves virtually indistinguishable across protocols and yields approximately 18% test error.
- ImageNet: ImageNet experiments evaluate 18 learners and different n-softsync protocols using training and top-1 validation error.The ImageNet model contains approximately 72 million trainable parameters and has a total model size of 289 MB.
- ImageNet: With the fixed SSGD learning rate, ImageNet convergence deteriorates with increasing staleness and fails for 9- and 18-softsync, whereas automatic staleness-based tuning produces nearly identical behavior across protocols.The learning rate is tuned per gradient as α = α0/τ.
- Overall Evaluation: Knowing the initial SSGD learning rate α0 allows ASGD to automatically tune its learning rate, achieve accuracy comparable to SSGD, and retain near-linear runtime speedup.This removes the need for laborious manual learning-rate tuning during distributed ASGD training.
5 Conclusion
The paper addresses gradient staleness in distributed ASGD through a staleness-dependent learning-rate scheme. It proves SGD-consistent convergence and demonstrates effectiveness on CIFAR10 and ImageNet with near-linear speedup up to 30 learners.
- Conclusion: The paper studies how to counter gradient staleness in distributed ASGD and develops a learning-rate strategy that adjusts updates according to staleness.The contribution is framed around improving distributed training despite varying stale-gradient effects.
- Conclusion: The proposed staleness-dependent learning-rate scheme gives ASGD the same convergence rate as SSGD.This is established theoretically in the paper.
- Conclusion: Experiments quantify gradient-staleness distributions and demonstrate the strategy on CIFAR10 and ImageNet.The benchmarks span substantially different data and model sizes.
- Conclusion: Near-linear speedup is achieved for up to 30 learners while maintaining the same convergence rate despite varying staleness.The conclusion reports this pattern across the evaluated benchmarks and system settings.