Source-linked AI summary
TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, Hai Li
TL;DR
Distributed training is limited by the communication needed to synchronize gradients and parameters. TernGrad reduces this overhead with ternary gradients, convergence analysis, and bound-guided techniques, while experiments report accuracy preservation or small degradation and higher throughput.
Problem
High communication cost for synchronizing gradients and parameters is a bottleneck in distributed training.
Method
TernGrad ternarizes gradients to {-1, 0, 1}, proves convergence under a gradient bound, and uses layer-wise ternarizing and gradient clipping to improve convergence.
Results
TernGrad increases training throughput across three DNNs, with speedup depending on communication-to-computation ratio, GPU count, and bandwidth.
Takeaways & Limitations
TernGrad can converge to similar accuracy within the same iterations while reducing communication time per iteration and improving distributed training throughput.
Takeaways & Limitations
Experiments did not include more DNN models such as VggNet-A or distributed training beyond 8 workers.
Abstract
from arXiv · showhide
High network communication cost for synchronizing gradients and parameters is the well-known bottleneck of distributed training. In this work, we propose TernGrad that uses ternary gradients to accelerate distributed deep learning in data parallelism. Our approach requires only three numerical levels {-1,0,1}, which can aggressively reduce the communication time. We mathematically prove the convergence of TernGrad under the assumption of a bound on gradients. Guided by the bound, we propose layer-wise ternarizing and gradient clipping to improve its convergence. Our experiments show that applying TernGrad on AlexNet does not incur any accuracy loss and can even improve accuracy. The accuracy loss of GoogLeNet induced by TernGrad is less than 2% on average. Finally, a performance model is proposed to study the scalability of TernGrad. Experiments show significant speed gains for various deep neural networks. Our source code is available.
1 Introduction
Distributed data-parallel training synchronizes gradients through a parameter server, making communication a bottleneck. TernGrad addresses this bottleneck by ternarizing gradients and adding convergence-oriented techniques and analysis.
- Distributed data parallelism trains model copies on different data subsets while a parameter server averages gradients and updates parameters.
- Sparse and quantized networks have been widely studied for inference, but these approaches can increase training effort.
- Dense-model distributed training can reduce communication through gradient sparsification or low-precision gradient quantization.
- TernGrad quantizes gradients to {-1, 0, 1}, proves convergence under a statistical gradient bound, and proposes layer-wise ternarizing and clipping.It also introduces scaler sharing, parameter localization, and a performance model for compressed-gradient training.
2 Related work
Prior work reduces distributed-training communication through sparsification or reduced precision. TernGrad follows gradient quantization while differing from sparsity methods and extending contemporaneous quantization work with a statistical-bound analysis and practice-guiding techniques.
- Gradient sparsification: Gradient sparsification transmits only larger gradients, reducing communication and achieving 22% speed gain on 4 GPUs without affecting translation quality.
- Gradient quantization: Prior quantization methods reduce weight, activation, or gradient precision, with reported applications ranging from single-worker acceleration to energy-efficient hardware.DoReFa-Net reported 9.8% accuracy loss in its single-worker acceleration setting.
- Gradient quantization: Compared with QSGD, TernGrad analyzes convergence through a statistical gradient bound and uses that bound to motivate practical quantization techniques.
3 Problem Formulation and Our Approach
TernGrad reduces distributed-training communication by stochastically quantizing gradients to three levels, while parameter localization and scaler sharing limit synchronization overhead. Its convergence analysis assumes bounded gradients, motivating layer-wise ternarizing and clipping, which preserve AlexNet accuracy in the reported experiments.
- 3.1 Problem Formulation and TernGrad: Each worker computes gradients on a different data subset, synchronizes them at a parameter server, and updates a locally stored parameter copy.Parameter localization maintains worker-side parameter copies and replaces floating-point parameter synchronization with quantized gradient pulling.
- 3.1 Problem Formulation and TernGrad: TernGrad randomly quantizes each gradient to the ternary levels {−1, 0, +1} using a scaler and stochastic binary rounding.The stochastic rounding has an unbiased expectation, supporting convergence analysis.
- 3.1 Problem Formulation and TernGrad: 20.18× is the theoretical worker-to-server traffic reduction from encoding ternary gradients instead of 32-bit gradients.Even a 2-bit ternary encoding yields a 16× reduction factor under the paper’s comparison.
- 3.1 Problem Formulation and TernGrad: Scaler sharing limits the averaged gradient to at most 2N + 1 numerical levels and reduces server-to-worker communication by 32/log2(1 + 2N), unless N ≥230.The sharing overhead is transferring 2N floating scalars.
- 3.2 Convergence Analysis and Gradient Bound: TernGrad’s stochastic ternary gradient is unbiased in expectation, equaling the gradient of the minimization target.This property connects the quantized update to the convergence argument.
- 3.2 Convergence Analysis and Gradient Bound: Under the stated learning-rate, objective, and gradient-bound assumptions, stochastic ternary updates converge almost surely to the minimum w∗.The proof adapts the convergence analysis of the General Online Gradient Algorithm.
- 3.3 Feasibility Considerations: The TernGrad gradient bound is stronger than standard GOGA’s, especially when a large network-wide maximum gradient dominates most gradients.Layer-wise ternarizing and gradient clipping are proposed to reduce this gap.
- 3.3 Feasibility Considerations: With c = 2.5 clipping, gradient length changes by 1.0%−1.5% and direction by 2°−3°, while AlexNet reaches standard-SGD accuracy when clipping and layer-wise ternarizing are both applied.Removing clipping caused a reported 3% top-1 accuracy loss.
4 Experiments
Experiments evaluate TernGrad across optimizers, batch sizes, worker counts, and large-scale ImageNet models. It generally preserves accuracy, improves AlexNet accuracy in a large-batch setting, and incurs less than 2% average loss on GoogLeNet.
- 4.1 Integrating with Various Training Schemes: TernGrad converges to similar accuracy within the same iterations under momentum SGD and vanilla SGD, with accuracy changes ranging from a 0.15% gain to a 0.22% loss.Communication time per iteration is reduced, and no degradation is observed with 64 workers.
- 4.1 Integrating with Various Training Schemes: TernGrad reaches baseline accuracy with less than 1% degradation across CifarNet experiments, while larger mini-batches reduce accuracy for both methods.The reported accuracy gap is smaller at mini-batch size 2048, and TernGrad’s noise may help reach flatter minimizers.
- 4.2 Scaling to Large-scale Deep Learning: TernGrad converges to approximate accuracy levels regardless of mini-batch size in AlexNet experiments.The experiments fix mini-batch size per worker at 128 and adjust learning rates for larger batches.
- 4.2 Scaling to Large-scale Deep Learning: 0.92% higher top-1 accuracy is achieved by TernGrad at mini-batch size 1024.The paper attributes this improvement to inherent randomness encouraging escape from poorer sharp minima.
- 4.2 Scaling to Large-scale Deep Learning: Less than 2% average accuracy loss is reported for GoogLeNet using TernGrad.The TernGrad runs reuse baseline-tuned hyper-parameters except for dropout ratio and weight decay; further tuning may improve accuracy.
5 Performance Model and Discussion
TernGrad uses three numerical gradient levels to reduce communication time, and its performance model evaluates training speed across neural networks, GPU counts, and cluster bandwidths.
- Three numerical levels {−1, 0, 1} enable TernGrad to aggressively reduce communication time.
- The experiments did not train additional DNN models or evaluate distributed training beyond 8 workers.
- The performance model combines lightweight single-machine profiling with analytical modeling to estimate distributed GPU training speed.
- TernGrad effectively increases training throughput for AlexNet, GoogLeNet, and VggNet-A across the evaluated GPU clusters.
- Speedup depends on the DNN's communication-to-computation ratio, GPU count, and communication bandwidth.
- On a 128-node InfiniBand and NVLink system, TernGrad doubles VggNet-A training speed.
Appendix A Convergence Analysis of TernGrad
The convergence analysis proves Theorem 1 by evaluating the ternarized update under the stated distributional and gradient assumptions, and extends the proof to mini-batch SGD.
- The proof evaluates the expected change in the objective between successive iterates using the ternarized update.
- Conditioning on X_t, the update's expected direction is linked to the gradient ∇wC(w_t).
- The squared ternarized-update norm is bounded using the Bernoulli distribution of b_tk and Assumption 3.
- The resulting bound satisfies Lemma 1's condition and proves Theorem 1.
- The proof extends to mini-batch SGD by treating z as a mini-batch of observations instead of one observation.
Appendix B Performance Model
The performance model estimates distributed training time by combining profiled computation with modeled communication, supporting both strong and weak scaling across GPU machines.
- The model supports strong scaling, which fixes total problem size, and weak scaling, which fixes the samples trained per worker.
- Strong scaling: Strong-scaling analysis estimates throughput for j machines with i GPUs per machine and mini-batch size K.
- Communication: Within a machine, all-reduce communication across i GPUs has complexity log2i before transfer to the CPU.
- Communication: Across machines, all-reduce communication time is modeled as (Cncost + |g| Cnwd) ∗ log2j.
- Computation: Computation time is estimated by profiling a single CPU and GPU training a mini-batch of K images.
- Strong scaling: Mini-batch training time is decomposed into computation and communication components: Tstrong = Tcomp + Tcomm.
- Weak scaling: Weak scaling replaces the mini-batch size with N ∗ K because each worker always trains K samples.
- Assumptions: The model assumes symmetric system architecture and dedicated GPU-to-GPU communication bandwidth.