Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training

Yujun Lin; Song Han; Huizi Mao; Yu Wang; William J. Dally

Source-linked AI summary

Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training

Yujun Lin, Song Han, Huizi Mao, Yu Wang, William J. Dally

arXiv:1712.01887v3Dec 5, 2017cs.CVcs.DCcs.LGstat.ML

TL;DR

Distributed training is limited by costly gradient exchange, particularly under mobile networking constraints. DGC compresses gradients while correcting accumulation and staleness effects, achieving 270-600× compression across multiple tasks without losing accuracy. The paper reports that this approach reduces communication requirements for distributed training.

Problem
Gradient exchange consumes substantial bandwidth and limits scalable distributed training, with mobile devices facing lower throughput, intermittent connections, and expensive data plans.
Method
DGC combines gradient sparsification with momentum correction, local gradient clipping, momentum factor masking, and warmup training to preserve convergence under reduced communication.
Results
DGC achieves 270-600× gradient compression across CNN and RNN workloads, with reported accuracy maintained across image, language, and speech tasks.
Takeaways & Limitations
DGC reduces required communication bandwidth and supports distributed training with inexpensive commodity networking infrastructure.
Takeaways & Limitations
Without momentum correction and local gradient clipping, extremely sparse updates can harm convergence and reduce Cifar10 accuracy by more than 1.0%.

Abstract

from arXiv · show

Large-scale distributed training requires significant communication bandwidth for gradient exchange that limits the scalability of multi-node training, and requires expensive high-bandwidth network infrastructure. The situation gets even worse with distributed training on mobile devices (federated learning), which suffers from higher latency, lower throughput, and intermittent poor connections. In this paper, we find 99.9% of the gradient exchange in distributed SGD is redundant, and propose Deep Gradient Compression (DGC) to greatly reduce the communication bandwidth. To preserve accuracy during compression, DGC employs four methods: momentum correction, local gradient clipping, momentum factor masking, and warm-up training. We have applied Deep Gradient Compression to image classification, speech recognition, and language modeling with multiple datasets including Cifar10, ImageNet, Penn Treebank, and Librispeech Corpus. On these scenarios, Deep Gradient Compression achieves a gradient compression ratio from 270x to 600x without losing accuracy, cutting the gradient size of ResNet-50 from 97MB to 0.35MB, and for DeepSpeech from 488MB to 0.74MB. Deep gradient compression enables large-scale distributed training on inexpensive commodity 1Gbps Ethernet and facilitates distributed training on mobile. Code is available at: https://github.com/synxlin/deep-gradient-compression.

1 INTRODUCTION

Distributed training reduces computation time through data parallelism, but gradient exchange becomes a communication bottleneck, especially on mobile devices. DGC addresses this bottleneck with gradient compression and techniques that preserve model performance.

Gradient exchange can outweigh computation savings, making network bandwidth a significant bottleneck for scaling distributed training.
Mobile and federated-learning settings face lower bandwidth, intermittent connections, and expensive data plans.
DGC combines gradient sparsification with momentum correction and local gradient clipping to maintain model performance.
Momentum factor masking and warmup training address staleness caused by reduced communication.
Gradients can be compressed up to 600× without loss of accuracy across image classification, language modeling, and speech recognition tasks.

2 RELATED WORK

Prior work reduces communication through asynchronous updates, gradient quantization, or sparsification, while DGC targets higher compression without changing model structure. The section also introduces gradient sparsification and momentum-correction mechanisms underlying DGC.

Asynchronous SGD removes gradient synchronization and updates parameters when nodes complete back-propagation.
Gradient quantization reduces transfer size by representing gradients with fewer precision levels, with prior methods demonstrating convergence in selected settings.
Gradient sparsification sends only selected updates, but prior Gradient Dropping requires layer normalization to maintain convergence speed.
DGC reaches up to 600× compression for the whole model without extra layer normalization or model-structure changes, while maintaining accuracy.

3 DEEP GRADIENT COMPRESSION

Deep Gradient Compression sparsifies gradient updates, accumulates omitted gradients locally, and adds four techniques to preserve convergence and accuracy under extreme sparsity. These methods address distortion from momentum, clipping, delayed updates, and early-training instability.

3.1 Gradient Sparsification: Gradient sparsification transmits only gradients above a magnitude threshold while accumulating smaller gradients locally until they become transmissible.The encoder packs 32-bit nonzero gradient values and 16-bit zero run lengths.
3.1 Gradient Sparsification: Local accumulation makes sparse updates equivalent to increasing the batch size from Nb to NbT over an interval T, with learning-rate scaling automatically satisfied.The factors T in the learning rate and batch size cancel in the update.
3.2 Improving the Local Gradient Accumulation: Without correction, extreme sparsity caused more than 1.0% accuracy loss on Cifar10; momentum correction and local gradient clipping mitigated the problem.The authors report that these techniques improved word error rate from 14.1% to 12.9% on AN4.
3.2 Improving the Local Gradient Accumulation: Directly applying momentum to sparse gradients breaks equivalence with dense momentum SGD because delayed updates omit the accumulated discounting factor.Momentum correction locally accumulates momentum-adjusted gradients before sparsification and communication, without adding a hyperparameter.
3.3 Overcoming the Staleness Effect: 99.9% sparsity can delay most parameter updates by 600 to 1000 iterations, making stale updates slow convergence and degrade model performance.Momentum factor masking and warm-up training are used to mitigate this staleness.
3.3 Overcoming the Staleness Effect: DGC combines gradient sparsification and local accumulation with momentum correction, local gradient clipping, momentum factor masking, and warm-up training to maintain accuracy at higher compression ratios.Table 1 summarizes the roles of these techniques in improving accumulation and alleviating staleness.

4 EXPERIMENTS

Experiments evaluate Deep Gradient Compression across image classification, language modeling, and speech recognition using multiple datasets and models. Across these settings, DGC maintains accuracy while achieving high gradient compression ratios.

Experiment settings: DGC is evaluated on image classification, language modeling, and speech recognition across Cifar10, ImageNet, Penn Treebank, AN4, and Librispeech.The experiments use CNN, RNN, LSTM, and GRU-based models, including ResNet, AlexNet, and DeepSpeech architectures.
Image classification: 99.9% sparsity preserves ResNet-110 accuracy on Cifar10 while DGC learning curves closely follow the baseline.Gradient Dropping performs worse because of staleness, while momentum correction, momentum factor masking, and warm-up training restore baseline-like behavior.
Image classification: 75× better compression than TernGrad is achieved on ImageNet AlexNet with no loss of accuracy.For ResNet-50, DGC reaches 277× compression versus 597× for the comparison setting, with a slight increase in accuracy.
Language modeling: 462× gradient compression is achieved in Penn Treebank language modeling with a slight reduction in perplexity.Training loss and validation perplexity closely match the baseline at 99.9% gradient sparsity.
Speech recognition: 608× gradient compression improves recognition on both clean and noisy Librispeech speech tests.The speech-recognition learning curves show the same improvement from DGC techniques as the image-classification experiments.

5 SYSTEM ANALYSIS AND PERFORMANCE

The system analysis examines sparsification overhead, training scalability, and model behavior under 99.9% gradient sparsity. Deep Gradient Compression reduces communication demands particularly when bandwidth is limited or communication is relatively costly.

System analysis: O(n) top-k selection over millions of gradient elements motivates sampling-based threshold estimation for 99.9% sparsity.The proposed sampling procedure selects 0.1% to 1% of gradients before estimating the full-population threshold.
System analysis: 99.9% gradient sparsity is evaluated through perplexity and training-loss curves for an LSTM language model on PTB.
System analysis: 99.9% gradient sparsity is also evaluated through WER and training-loss curves for a 5-layer LSTM on AN4.
System analysis: The scalability analysis combines single-node profiling with analytical communication modeling using an all-reduce communication model.The analysis accounts for worst-case density doubling at each aggregation step.
System analysis: Deep Gradient Compression makes 1Gbps Ethernet training competitive with conventional training using 10Gbps Ethernet.The comparison concerns multi-node speedup relative to single-node training; conventional training performs substantially worse at 1Gbps.
System analysis: The benefit of Deep Gradient Compression increases as the model’s communication-to-computation ratio rises and network bandwidth falls.

6 CONCLUSION

The conclusion presents Deep Gradient Compression as a highly sparse gradient-communication method that preserves convergence through several training safeguards and improves distributed-training scalability.

6 CONCLUSION: 270-600× compression is achieved across a wide range of CNNs and RNNs without slowing convergence.
6 CONCLUSION: Momentum correction, local gradient clipping, momentum factor masking, and warm-up training support compression without slowing convergence.
6 CONCLUSION: Hierarchical threshold selection is proposed to speed up gradient sparsification.
6 CONCLUSION: Deep Gradient Compression reduces required communication bandwidth and improves distributed-training scalability.

A SYNCHRONOUS DISTRIBUTED STOCHASTIC GRADIENT DESCENT

Synchronous distributed SGD lets nodes process different minibatches with a shared model, then aggregate gradients so their models remain synchronized.

A SYNCHRONOUS DISTRIBUTED STOCHASTIC GRADIENT DESCENT: Each training node performs forward-backward computation on different batches sampled from the training dataset using the same network model.
A SYNCHRONOUS DISTRIBUTED STOCHASTIC GRADIENT DESCENT: Gradients from all nodes are summed to optimize the models.
A SYNCHRONOUS DISTRIBUTED STOCHASTIC GRADIENT DESCENT: Gradient aggregation keeps models on different nodes the same during training.
A SYNCHRONOUS DISTRIBUTED STOCHASTIC GRADIENT DESCENT: Aggregation can use parameter servers as intermediaries that store parameters among several servers.
A SYNCHRONOUS DISTRIBUTED STOCHASTIC GRADIENT DESCENT: The distributed synchronous SGD algorithm takes a dataset, per-node minibatch size, node count, optimization function, and initialized parameters as inputs.
A SYNCHRONOUS DISTRIBUTED STOCHASTIC GRADIENT DESCENT: The algorithm samples data, accumulates local gradients, and then performs an all-reduce operation.

B GRADIENT SPARSIFICATION WITH NESTROV MOMENTUM CORRECTION

This section contrasts conventional Nesterov momentum SGD with sparse updates and introduces momentum correction for the compressed-gradient setting.

B GRADIENT SPARSIFICATION WITH NESTEROV MOMENTUM CORRECTION: The conventional update rule uses Nesterov momentum SGD.
B GRADIENT SPARSIFICATION WITH NESTEROV MOMENTUM CORRECTION: The conventional update is expressed in terms of the momentum, learning rate, update state, and gradient.
B GRADIENT SPARSIFICATION WITH NESTEROV MOMENTUM CORRECTION: The sparse update is first specified before momentum correction.
B GRADIENT SPARSIFICATION WITH NESTEROV MOMENTUM CORRECTION: After correction, the update tracks local momentum and an additional accumulated state before updating the model parameters.

C LOCAL GRADIENT CLIPPING

Local gradient clipping is applied before accumulated gradients are updated, with thresholds defined separately for origin and local gradient norms and scaled by N^-1/2.

C LOCAL GRADIENT CLIPPING: Local clipping occurs before the current gradient is added to the previous accumulation.For recurrent neural network training, clipping is performed locally before accumulation.
C LOCAL GRADIENT CLIPPING: thrG denotes the threshold for the origin gradient L2-norm, while thrGk denotes the threshold for the local gradient L2-norm.
C LOCAL GRADIENT CLIPPING: N^-1/2 scales the threshold to represent the current node’s fraction of the global threshold.The scaling assumes N training nodes have independent, identically distributed gradient distributions with variance σ^2, so their summed gradients have variance Nσ^2.

D DEEP GRADIENT COMPRESSION ALGORITHM

The DGC algorithms initialize per-node state, compute minibatch gradients, optionally clip them, and apply momentum-based updates across training iterations.

D DEEP GRADIENT COMPRESSION ALGORITHM: Algorithms 3 and 4 operate on node k using a dataset, minibatch size, momentum, node count, optimization function, and initial parameters.Algorithm 3 uses vanilla momentum SGD, while Algorithm 4 uses Nesterov momentum SGD.
D DEEP GRADIENT COMPRESSION ALGORITHM: The training loop initializes U^k and V^k, iterates over training steps, and resets the minibatch gradient G^k before sampling data.
D DEEP GRADIENT COMPRESSION ALGORITHM: Each minibatch gradient is formed from sampled examples and can undergo gradient clipping before subsequent processing.