Source-linked AI summary
Decentralized Deep Learning with Arbitrary Communication Compression
Anastasia Koloskova, Tao Lin, Sebastian U. Stich, Martin Jaggi
TL;DR
Decentralized deep-learning training is constrained by limited communication bandwidth, despite its relevance to data privacy, on-device learning, and scalable computation. The paper applies compressed communication through CHOCO-SGD, extending its analysis to non-convex objectives and evaluating a momentum variant across decentralized settings. It reports linear worker speedup under arbitrary high compression and practical results in peer-to-peer and datacenter scenarios, while noting limitations in baseline theory and compression assumptions.
Problem
Decentralized deep-learning training needs communication compression because network bandwidth is limited, while data locality and scalable computation remain important.
Method
The paper uses CHOCO-SGD and a momentum variant, combining compressed neighbor communication with modified gossip averaging and error aggregation.
Results
The leading convergence term matches centralized exact-communication baselines and achieves linear speedup in worker count, while experiments cover peer-to-peer and datacenter training.
Takeaways & Limitations
CHOCO-SGD expands decentralized deep-learning training to strongly communication-restricted, data-local settings and supports arbitrary high communication compression.
Takeaways & Limitations
Baseline methods have mismatched theoretical support across unbiased and biased compression schemes, and some quantization-error bounds can be restrictive for unbounded input vectors.
Abstract
from arXiv · showhide
Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks, as well as for efficient scaling to large compute clusters. As current approaches suffer from limited bandwidth of the network, we propose the use of communication compression in the decentralized training context. We show that Choco-SGD $-$ recently introduced and analyzed for strongly-convex objectives only $-$ converges under arbitrary high compression ratio on general non-convex functions at the rate $O\bigl(1/\sqrt{nT}\bigr)$ where $T$ denotes the number of iterations and $n$ the number of workers. The algorithm achieves linear speedup in the number of workers and supports higher compression than previous state-of-the art methods. We demonstrate the practical performance of the algorithm in two key scenarios: the training of deep learning models (i) over distributed user devices, connected by a social network and (ii) in a datacenter (outperforming all-reduce time-wise).
1 INTRODUCTION
The paper targets decentralized deep-learning training under bandwidth constraints, extending communication-compressed optimization to practical peer-to-peer and datacenter settings. It combines convergence analysis with experiments spanning on-device learning, scalable training, and data locality.
- Distributed learning combines computational scalability with data locality, keeping each device’s training data local during joint optimization.
- Prior decentralized compression methods restrict the allowed compression operators, particularly by requiring unbiased compression.
- The study examines generalization on test sets and challenging peer-to-peer settings where data remains distributed across devices.
- CHOCO-SGD is extended theoretically to non-convex objectives with communication compression.
- O(1/√(nT)) convergence matches centralized exact-communication baselines in its leading term and provides linear speedup in worker count n.
- The momentum variant is evaluated for on-device peer-to-peer training and datacenter deep-learning training, with bandwidth reduction and time-to-accuracy as practical concerns.
2 RELATED WORK
Related work covers decentralized optimization, gradient compression, and quantization, while highlighting challenges caused by compressed communication. CHOCO-SGD is positioned as supporting arbitrary compression ratios and, experimentally, higher test accuracy than DeepSqueeze under equal tuning.
- Decentralized SGD with gossip averaging typically depends on the mixing matrix’s spectral gap ρ.
- The leading convergence term of decentralized SGD can match centralized mini-batch SGD, while spectral-gap effects remain asymptotically smaller.
- Quantization research progressed from unbiased to biased compression, with error-correction schemes often providing strong practical and theoretical guarantees.
- Gossip averaging can diverge or converge incorrectly when quantization noise is present.
- DCD and ECD support centralized-baseline rates only for constant compression ratios, whereas CHOCO-SGD addresses arbitrary high compression ratios.
- Under the same amount of tuning, CHOCO-SGD achieves higher test accuracy than DeepSqueeze.
3 CHOCO-SGD
CHOCO-SGD combines stochastic gradient updates with modified gossip averaging and compressed neighbor communication. Its error-correction mechanism preserves iterate averages despite quantization noise, while the framework accommodates broad compression operators over decentralized graphs.
- Distributed Setup: The decentralized objective allows local, possibly non-convex and non-identical loss functions sampled from node-specific data distributions.
- Communication: Communication occurs only between local neighbors defined by a weighted graph topology.
- Mixing matrix: The mixing matrix is symmetric and doubly stochastic, with spectral gap ρ = 1 − |λ2(W)|.
- Compression: Compression operators formalize transmission of quantized or sparsified messages.
- Compression: Unlike unbiased-only quantizers, the compression-operator definition permits biased operators and therefore a larger class of compressors.
- CHOCO-SGD: Each worker performs a stochastic gradient step and modified gossip averaging while communicating compressed updates to neighbors.
- CHOCO-SGD: Compression errors are aggregated in local variables, preserving averages even in the presence of quantization noise.
4 CONVERGENCE OF CHOCO-SGD ON SMOOTH NON-CONVEX PROBLEMS
The convergence analysis extends CHOCO-SGD to smooth non-convex problems under assumptions on smoothness and stochastic-gradient variance. The leading rate gives linear worker speedup, while compression and graph topology affect only higher-order terms.
- CHOCO-SGD’s analysis is extended to smooth non-convex functions.
- The stochastic-gradient variance is bounded on every worker under the stated assumptions.
- The theorem uses a constant SGD stepsize and a consensus stepsize determined by compression quality, spectral properties, and β = ∥I − W∥2.
- The convergence rate has a leading term that yields linear speedup over single-node SGD, while compression and graph topology affect only the higher-order second term.
- With exact averaging and δ = 1, c = ρ and the convergence rate improves to the exact-averaging case.
5 COMPARISON TO BASELINES FOR VARIOUS COMPRESSION SCHEMES
The comparison evaluates momentum-based CHOCO-SGD against decentralized and centralized baselines across unbiased and biased compression schemes. CHOCO-SGD maintains generalization with substantially lower communication, while competing methods can become unstable under high compression.
- Method: Momentum is added to CHOCO-SGD by replacing the stochastic-gradient update with local momentum and weight decay.The modified algorithm introduces a local momentum memory, momentum factor β, and weight decay factor λ.
- Experimental setup: The experiments compare gsgdb quantization, randoma sparsification, topa sparsification, and other compression choices against decentralized and all-reduce baselines.The setup uses a ring of n = 8 nodes, ResNet20 on Cifar10, and fine-tuned momentum implementations.
- Results: ECD and DCD perform well mainly at small compression ratios and sometimes diverge at high ratios.DCD performs better with biased topa sparsification than with unbiased randoma sparsification, although that combination is not theoretically supported.
- Caveat: Quantization-error bounds can be restrictive for unbounded input vectors, potentially explaining instability under high compression.This caveat concerns the absolute error bounds used in prior work, not CHOCO-SGD's reported result.
- Results: CHOCO-SGD generalizes across scenarios with at most a 1.65% accuracy drop for a fixed training budget.The comparison emphasizes test-set generalization rather than only training performance.
- Results: Sign compression achieves state-of-the-art accuracy while requiring approximately 32× fewer bits per weight than the full-precision baseline.Table 1 reports top-1 test accuracy averaged over three runs and compares against an all-reduce baseline reaching 92.64 accuracy with 1.04 MB gradient transmission per iteration.
6 USE CASE I: ON-DEVICE PEER-TO-PEER LEARNING
The on-device peer-to-peer setting keeps data local while making communication efficiency central. Across scaling and social-network experiments, CHOCO-SGD offers strong accuracy under fixed communication budgets, though decentralized methods can converge more slowly than centralized training.
- On-device peer-to-peer learning: Peer-to-peer training keeps each device’s data local, while limited bandwidth makes reducing communication especially important.This setting is intrinsically decentralized and applies to devices such as mobile phones, sensors, and hospitals.
- Scaling to large number of nodes: CHOCO-SGD’s scaling is slowed by graph topology and compression, whereas centralized SGD performs well across the tested node counts.The observed degradation is attributed to slower convergence rather than a generalization issue.
- Scaling to large number of nodes: 1000 MB: CHOCO-SGD achieves the best testing accuracy, while decentralized and centralized SGD require significantly more communication for reasonable accuracy.Torus topology helps at larger network sizes, but its higher node degree can offset the benefit for small networks.
- Experiments on a real social network graph: On a 32-node social network, experiments evaluate ResNet20 on CIFAR-10 and a three-layer LSTM on WikiText-2 with permanently local data partitions.The image-classification and language-modeling tasks use distinct datasets distributed across workers.
- Experiments on a real social network graph: For image classification, CHOCO-SGD significantly outperforms exact decentralized training at equal transmitted data, although centralized training has the highest test accuracy.At equal epoch budgets, decentralized training has the best training accuracy, followed by centralized and quantized decentralized methods.
- Experiments on a real social network graph: For language modeling, CHOCO-SGD achieves the best perplexity at fixed data volume and outperforms centralized SGD in test perplexity.Both decentralized schemes show a drop in training loss when evaluation reaches the epoch budget.
7 USE CASE II: EFFICIENT LARGE-SCALE TRAINING IN A DATACENTER
In a datacenter, CHOCO-SGD uses decentralized, compressed communication to reduce training time for large-scale deep learning. On an 8-node ResNet-50/ImageNet-1k setup, it achieves a time-wise gain over all-reduce while incurring a slight accuracy loss.
- CHOCO-SGD targets datacenter scalability through decentralized communication, complementing prior work on decentralized optimization and large-GPU training.
- The experiment used sign compression for CHOCO-SGD and centralized SGD, with all-reduce for uncompressed centralized SGD and all-gather for sign-quantized centralized SGD.The topology comprised 8 nodes, each accessing 4 GPUs.
- The benefits of CHOCO-SGD can become more pronounced as the number of nodes increases.
- 20% time-wise gain over all-reduce was demonstrated on the commodity hardware cluster.The comparison used ResNet-50 training on ImageNet-1k across eight machines.
- 1.5% accuracy loss accompanied CHOCO-SGD's shorter time for the same number of epochs.The reported result compares CHOCO-SGD with all-reduce in the datacenter experiment.
8 CONCLUSION
The paper extends CHOCO-SGD to communication-constrained decentralized deep learning, combining theoretical guarantees with practical studies under data locality and arbitrary high compression. Its framework covers stochastic gradient updates followed by averaging schemes, while experiments and analysis establish linear speedup and identify scaling difficulties shared by decentralized methods.
- The paper proposes CHOCO-SGD and its momentum version for decentralized deep learning in bandwidth-constrained environments.
- Theoretical analysis covers non-convex objectives and shows linear speedup in the number of nodes.
- The paper studies image classification and language modeling across ImageNet-1k, CIFAR-10, and WikiText-2, including data-local and communication-constrained settings.
- The method supports arbitrary high communication compression while respecting local training-data constraints.
- A CONVERGENCE OF CHOCO-SGD: The convergence proof treats decentralized SGD as local stochastic gradient updates followed by an averaging step with linear convergence.CHOCO-SGD is recovered by choosing CHOCO-GOSSIP as the consensus averaging scheme.
A.2 PROOFS
The appendix establishes convergence for decentralized SGD with arbitrary averaging, then specializes the guarantees to CHOCO-SGD and several compression operators. The resulting bounds separate linear-speedup terms from higher-order effects due to averaging, compression, and graph topology.
- General convergence framework: Algorithm 3 combines local stochastic-gradient updates with an averaging scheme that preserves iterates’ average and converges linearly.The proof framework covers arbitrary averaging steps satisfying the stated assumptions.
- General convergence framework: Theorem A.3 bounds the averaged iterates under Assumptions 1–3 with constant stepsize η < 1/(4L).The bound uses the convergence rate c of the underlying averaging scheme.
- Convergence consequences: The leading convergence term provides linear speedup over single-node SGD, while the averaging scheme contributes only a second-order term.This separation is stated for the general decentralized averaging framework.
- Convergence consequences: CHOCO-SGD with CHOCO-GOSSIP supports arbitrary high compression, although its dependence on the mixing-matrix eigengap ρ is worse than with exact averaging.The weaker ρ dependence may reflect the proof technique or the cost of supporting arbitrary high compression.
- Local-weight guarantees: The appendix also gives guarantees for individual local weights, addressing settings where averaging all distributed parameters is expensive or impossible.These guarantees complement the result for averaged parameters.
- Compression operators: The implementation section defines unbiased and biased gsgdb, randoma, topa, and sign compression operators with their associated encoding requirements.For sign compression, transmission requires d + 32 bits: one bit per entry and 32 bits for the ℓ1 norm.
E ERROR FEEDBACK INTERPRETATION OF CHOCO-SGD
CHOCO-SGD can be rewritten as an error-feedback method: local stochastic updates are followed by modified gossip averaging, while quantization errors remain in internal memory for later correction.
- Error-feedback mechanism: CHOCO-SGD transmits the change in each local variable rather than the variable itself.The transmitted difference represents the local variable’s evolution at the current step.
- Error-feedback mechanism: Before compression, stored internal memory is added to the transmitted difference to correct previous quantization errors.The memory is then updated after compression.
- Algorithmic procedure: Algorithm 4 organizes each iteration around modified gossip averaging and local stochastic-gradient updates performed in parallel across workers.The algorithm is parameterized by communication graph, mixing matrix, and separate consensus and SGD stepsizes.
F DETAILED EXPERIMENTAL SETUP AND TUNED HYPERPARAMETERS
The experiments tune CHOCO-SGD and baselines across social-network and datacenter settings, using deep-learning models, fixed training protocols, and repeated cloud runtime measurements.
- Social-network setup: The social-network comparison evaluates sign-compressed CHOCO-SGD against decentralized and centralized SGD without compression.The models are ResNet20 on Cifar10 and a three-layer LSTM for language modeling.
- Training protocol: Both models train for 300 epochs unless otherwise specified, with per-node mini-batch size 32; momentum with factor 0.9 is used only for ResNet20.These settings define the common training protocol for the two social-network tasks.
- Training protocol: Learning rates warm up for five epochs, then decay by a factor of 10 at 50% and 75% of total training epochs.The initial learning rate and consensus stepsize are selected through grid searches.
- Runtime measurement: Runtime estimates in Figure 5 come from three Google Cloud Kubernetes Engine trials measuring time per mini-batch during the first two training epochs.The same runtime procedure is applied across the evaluated methods.
- Social-network setup: The social-network topology uses 32 nodes, disjoint local data subsets, and per-node mini-batches of 32.The data partition remains fixed during training, with maximum node degree 14.
G ADDITIONAL PLOTS
The additional plots extend evaluation to 64-node scaling, social-network learning curves, and datacenter accuracy curves, with tables recording matched epoch and transmitted-bit budgets.
- Large-scale scaling: Figure 6 shows CHOCO-SGD with sign compression scaling to 64 devices on Cifar10.Vertical lines mark the epoch and bits budgets used in Figure 1.
- Budget reporting: Tables 7 and 8 report the exact epoch and transmitted-bit quantities used for matched-budget comparisons in Figure 1.The two tables separate epoch budgets from transmitted bits in megabytes.
- Social-network learning curves: Figures 7 and 8 visualize learning curves for decentralized ResNet-20 on CIFAR-10 and LSTM on WikiText2 over a 32-node social-network topology.Each node accesses a disjoint dataset subset and uses local mini-batch size 32.
- Datacenter learning curves: Figure 9 provides datacenter learning curves for ResNet-50 on ImageNet.The plotted metrics include training top-1, training top-5, and test top-5 accuracy.