Source-linked AI summary
AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training
Chia-Yu Chen, Jungwook Choi, Daniel Brand, Ankur Agrawal, Wei Zhang, Kailash Gopalakrishnan
TL;DR
Distributed deep-learning training faces increasing communication constraints as model and accelerator scales grow, while existing compression methods are not robust across neural-network layers. AdaComp uses localized selection of residual gradients with an adjustable threshold, achieving high compression across architectures and conditions with negligible accuracy and convergence loss.
Problem
Distributed training needs compression that is computationally friendly, applies across layer types, adapts to training variations, and preserves convergence.
Method
AdaComp uses local sampling of residual gradients with an adjustable threshold that adapts compression across layers, minibatches, epochs, optimizers, and learner counts.
Results
AdaComp achieves approximately 200× compression for fully-connected and recurrent layers and 40× for convolutional layers with negligible accuracy and convergence-rate loss across broad evaluations.
Takeaways & Limitations
AdaComp supports simultaneous compression of different neural-network layer types with one new hyper-parameter and adapts to distributed-training variations.
Takeaways & Limitations
The evaluation reports convergence and compression, but not the runtime impact of the pack and unpack functions or communication overheads.
Abstract
from arXiv · showhide
Highly distributed training of Deep Neural Networks (DNNs) on future compute platforms (offering 100 of TeraOps/s of computational capacity) is expected to be severely communication constrained. To overcome this limitation, new gradient compression techniques are needed that are computationally friendly, applicable to a wide variety of layers seen in Deep Neural Networks and adaptable to variations in network architectures as well as their hyper-parameters. In this paper we introduce a novel technique - the Adaptive Residual Gradient Compression (AdaComp) scheme. AdaComp is based on localized selection of gradient residues and automatically tunes the compression rate depending on local activity. We show excellent results on a wide spectrum of state of the art Deep Learning models in multiple domains (vision, speech, language), datasets (MNIST, CIFAR10, ImageNet, BN50, Shakespeare), optimizers (SGD with momentum, Adam) and network parameters (number of learners, minibatch-size etc.). Exploiting both sparsity and quantization, we demonstrate end-to-end compression rates of ~200X for fully-connected and recurrent layers, and ~40X for convolutional layers, without any noticeable degradation in model accuracies.
Introduction
Distributed deep-learning training increases communication demands as models and accelerator throughput scale, motivating compression that works across layers without harming convergence. AdaComp addresses this need with a computationally friendly scheme, while prior methods lack robustness across neural-network diversity.
- Distributed training requires high-bandwidth interconnects to communicate parameters across CPUs or GPUs, limiting utilization and training time.
- Accelerators exceeding 100s of TeraOps/s will make inter-accelerator bandwidth a greater constraint for distributed training.
- Future compression must handle convolutional, fully-connected, and recurrent layers while adapting to architectures and hyper-parameters with minimal overhead.
- Existing compression schemes are not robust enough for the diversity of typical neural networks.
- AdaComp is a computationally friendly gradient compression scheme based on simple local sampling that self-adapts its compression rate across minibatches and layers.
- AdaComp achieves approximately 200× compression in fully-connected and LSTM layers and 40× in convolutional layers with negligible accuracy and convergence-rate loss.
Residual gradient compression
Prior gradient-compression methods primarily target fully-connected layers and can fail when convolutional and fully-connected layers are compressed together. AdaComp instead uses localized, residual-aware selection with self-adjusting thresholds to adapt compression across layers and training conditions.
- Prior methods: Existing methods include one-bit quantization, thresholding, and combined schemes that preserve quantization errors but primarily address fully-connected layers.
- Prior methods: 16× compression without sparsity and approximately 1.5% degradation on large networks show limitations of prior ternary CNN-gradient compression.
- Observed limitations: Compressing convolutional gradients with 1-bit quantization alongside fully-connected compression can significantly worsen accuracy and even cause complete model divergence.
- Design observations: Effective pruning should consider mini-batch input-feature activity as well as residue magnitude because high-value residues may miss important gradients linked to active features.
- AdaComp design: The scheme prioritizes localized, accelerator-friendly operations because global residue sorting would add excessive computational overhead.
- AdaComp design: AdaComp divides each layer’s residue vector into fixed-length bins and locally samples residues within each bin using a new bin-size hyper-parameter LT.
- AdaComp design: Its self-adjusting threshold allows different bins to transmit different numbers of gradients and sends more gradients early, when residues are smaller.
- Implementation: The implementation combines sign-bit quantization with a layer-wide scale value derived from the average absolute gmax values.
Experiments
AdaComp is evaluated across diverse neural-network architectures, datasets, learner counts, and optimizers using standard distributed-training integration. The experiments report near-baseline convergence and accuracy alongside effective compression rates of 40× for convolutional layers and 200× for fully connected and recurrent layers.
- Experimental setup: AdaComp is inserted between gradient backpropagation and weight updates through pack() and unpack(), while exchange() handles topology-dependent gradient communication.The training flow collects, compresses, exchanges, decompresses, averages, and locally applies gradients.
- Convergence and accuracy: 1 to 128 learners produced almost identical test errors to the non-compressed baseline across tested networks, datasets, and layer types.Detailed convergence curves are reported in Figure 2 and aggregate results in Table 2.
- Convergence and accuracy: Adam retained similar compression ratios with no impact on convergence or test error: baseline 18.1% versus compression 18.3%.Adam converged faster initially than SGD but reached similar final accuracy.
- Compression results: 40× compression was achieved for convolutional layers and 200× for fully connected and recurrent layers, with negligible accuracy and convergence loss.The effective rates combine sparsity, indexed representation, and ternary quantization relative to 32-bit floating-point gradients.
Discussions
AdaComp remains robust at compression rates where Local Selection and Dryden’s method become unstable or diverge. Its adaptive soft-threshold and local sampling control residual-gradient growth across compression, minibatch, and learner variations.
- Robustness of the AdaComp Technique: Compression rates above 2000× produce only 22% CIFAR10 test error with AdaComp, while Dryden’s method and LS degrade substantially as compression increases.All methods remain near baseline below 250× compression; Adam shows even higher resilience than SGD with momentum at high rates.
- Robustness of the AdaComp Technique: LT=300 causes LS residual gradients and gradients to increase exponentially, producing complete divergence, whereas LT=200 remains stable with 17.84% test error.The paper attributes this to positive feedback from insufficiently transmitted gradients and accumulated residuals.
- Robustness of the AdaComp Technique: AdaComp’s adaptive soft-threshold sends additional residual gradients near local maxima in each bin, adapting transmission to the number of important gradients.This avoids the exponential residual-gradient and gradient growth observed in LS at high compression.
- Robustness of the AdaComp Technique: At epoch 120, LS residual gradients span -240K to 239K with long tails, while AdaComp drastically shortens the tails and prevents residual gradients from increasing over epochs.AdaComp sends gradients with large residuals together, reducing accumulated residual magnitude by many orders of magnitude relative to LS.
- Impact of mini-batch size and number of learners: Increasing minibatch size reduces achievable compression for both methods, but AdaComp achieves approximately 5–10× better compression by locally selecting important residues.More learners enhance compression because each learner processes a smaller local minibatch and therefore sees lower feature activity.
Conclusions
AdaComp is presented as a gradient-compression algorithm for distributed deep-learning training. It combines local sampling with an adjustable soft threshold to support high compression across layer types and system variations without noticeable test-accuracy degradation.
- Conclusions: AdaComp combines local sampling and an adjustable soft threshold to account for input-feature activity and accumulated residual gradients.The scheme is designed to adapt across layers, minibatches, epochs, optimizers, and learner counts.
- Conclusions: AdaComp achieves approximately 200× compression in fully-connected and recurrent layers and approximately 40× in convolutional layers without noticeable test-accuracy degradation.Figure 7 examines compression-rate changes across minibatch sizes and learner counts while keeping test-error degradation below 1%.
- Conclusions: AdaComp’s minibatch-size comparisons report compression-rate behavior while preserving model fidelity across different minibatch sizes.The comparison is shown for different minibatch sizes in Figure 7(a).