1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed

1. Introduction

• Error-compensated compression recovers information lost during compression, but existing approaches do not work for Adam because its updates depend nonlinearly on gradients.The paper proposes Adam warmup followed by communication-compressed momentum SGD preconditioned with Adam; it also presents convergence analysis and a custom MPI collective primitive.Source: §1 · pp. 1–2
• 1-bit Adam matches Adam’s convergence behavior and final accuracy while using up to 5× less communication and achieving 3.3× faster end-to-end throughput, including warmup.Experiments cover large-scale ML tasks; the reported throughput comparison is against uncompressed algorithms.Source: §1 · p. 2

2. Related Work

• Prior work studies unbiased and biased compression, while error compensation can preserve asymptotic convergence and combine with decentralized training, local SGD, and acceleration.The related-work section also reviews Adam and adaptive-learning-rate methods.Source: §2 · p. 2

3.1. Communication overhead affects the efficiency of distributed training

• Allreduce accounts for up to 94% and 75% of training time on the two tested clusters, with communication overhead increasing for more nodes, smaller batches or accumulation, and lower bandwidth.These are the settings identified as offering the most benefit from compression.Source: §3.1 · p. 3

3.2. Basic compression affects Adam’s convergence

• Basic error-compensated 1-bit compression of Adam’s gradients substantially slows its convergence, motivating a strategy that addresses Adam’s nonlinear gradient dependence.The comparison is for BERT-Large pre-training against vanilla Adam.Source: §3.2 · p. 3, Figure 1 · p. 3

3.3. Adam’s variance becomes stable during training

• When Adam’s variance is constant, Adam is equivalent to momentum SGD with a coordinate-dependent learning rate.The learning rate depends on the variance and the small constant η.Source: §3.3 · p. 3
• For BERT-Large pre-training, the fused variance norm stabilizes after about 23K steps, motivating freezing the variance as a precondition during 1-bit compression.The method uses Adam during warmup before the compression phase.Source: §1 · p. 2, §3.3 · p. 3, Figure 2 · p. 4

4. 1-bit Adam Algorithm

• The section frames 1-bit Adam as a response to the incompatibility between existing error-compensated compression and Adam’s nonlinear dependence on gradients.Adam is used for models such as BERT, for which vanilla or momentum SGD has been less effective.Source: §4 · p. 4, §4.2 · p. 5

4.1. Why error compensation works for SGD

• Error compensation cancels each step’s compression error at the next step in SGD, preventing error accumulation across updates.Without compensation, history compression error accumulates and can slow convergence.Source: §4.1 · pp. 4–5

4.2. Why Adam cannot be combined with error compensation

• Adam’s nonlinear variance update leaves a quadratic error term uncancelled, making the variance difficult to estimate accurately.Adam also requires a correction factor that depends on the variance after compression, which is unavailable.Source: §4.2 · p. 5

4.3. 1-bit Adam

• 1-bit Adam warms up with Adam, freezes its variance term as a precondition, then uses error-compensated 1-bit momentum communication.The method’s 1-bit representation uses the sign of each momentum element; its scaling factor preserves the compressed momentum’s magnitude.Source: §4.3 · p. 5

5. Theoretical Analysis

• The compression stage is analyzed as error-compensated momentum SGD with a coordinate-dependent learning rate, using the variance term fixed after warmup.The algorithm initializes the fixed variance after running Adam for the warmup steps.Source: §5 · pp. 5–6
• The convergence analysis assumes Lipschitzian gradients, bounded stochastic-gradient variance, and bounded compression error.The algorithm description also states a bound on the global error.Source: §5 · pp. 5–6
• O(1/nT) is the asymptotic convergence rate stated for 1-bit Adam, which the paper says permits linear speedup with the number of workers.The paper relates this rate to distributed SGD.Source: §5 · p. 6

6. Efficient system design for compressed communication

• Compressed allreduce uses MPI all-to-all, averaging compressed local momentums, and allgather; its basic version supports Ethernet, while CUDA-Aware requires InfiniBand.The basic version copies data between GPU and CPU buffers.Source: §6 · pp. 6–7

7. Experiments

• Up to 3.3 times faster than uncompressed algorithms under limited bandwidth; SQuAD fine-tuning reaches up to 2.9x higher throughput.The throughput gains are reported for 1-bit Adam; the SQuAD result is end-to-end.Source: §7 · p. 7, §7.1 · p. 8

7.1. BERT pre-training and fine-tuning

• For BERT-Base and BERT-Large across both sequence-length phases, 1-bit Adam matches baseline convergence speed while using 6% of the original communication volume during compression.GLUE accuracy is similar to the uncompressed baseline, and SQuAD F1 is 93.32 versus HuggingFace’s 93.33.Source: §7.1 · p. 7

7.3. Deep Convolutional Generative Adversarial Networks

• For DCGAN training on CelebA, 1-bit Adam achieves almost the same training accuracy as Adam.The comparison uses the training loss and generated images.Source: §7.3 · p. 9

8. Conclusions

• 1-bit Adam combines communication efficiency with Adam’s convergence speed by preconditioning momentum SGD with Adam.The paper reports theoretical analysis showing linear speedup with worker count and robustness to any compression method.Source: §8 · p. 9
• Across BERT, ResNet, and DCGAN tasks on up to 256 GPUs, 1-bit Adam matches Adam’s sample-wise convergence, reduces communication by up to 5x, and runs up to 3.3 times faster.The authors identify reinforcement learning and other wider task varieties as future work.Source: §8 · p. 9
• The authors leave evaluation on a wider variety of tasks, such as reinforcement learning, for future work.Source: §8 · p. 9

9. Bert Pre-training: speedup under different network bandwidths

• Figure 9 compares Adam with 1-bit Adam’s compression-stage throughput on 256 V100 GPUs across network bandwidths from 50Mbits to 3Gbits.The bandwidth axis uses a log scale.Source: Figure 9 · p. 13

10. ResNet: comparing with additional communication efficient algorithms

• The ResNet-18 comparison includes DoubleSqueeze, which applies 1-bit error-compensated compression to stochastic gradients.The supplementary section also compares 1-bit Adam with Local SGD and momentum-based methods.Source: §10 · p. 13

• For ResNet-18, EF Momentum SGD and Local Momentum SGD converge faster than 1-bit Adam, which the authors attribute to uncompressed Momentum SGD running faster than Adam.Source: §2 · p. 14
• The tested variance alternatives fail to achieve convergence speed comparable with Adam, leaving 1-bit Adam as the only solution the authors found to match vanilla Adam.Source: §2 · p. 14
• The variance tests include n-bit compression, which cannot converge when n ≤ 8, and lazy variance averaging every τ steps.The lazy-update variant averages the variance term across workers every τ steps while updating it continuously using local gradients.Source: §2 · p. 14, Figure 12 · p. 15, Figure 13 · p. 15

11. Proof to the updating form

• The analysis assumes a parameter-server model for communication on each gradient chunk, without requiring the tensor to be divided into chunks.This equivalence is adopted for simplicity.Source: §11 · p. 15
• At iteration t + 1, the momentum update is described as a sequence of communication steps.The passage introduces the steps but does not provide their full sequence here.Source: §11 · p. 15
• Each worker locally updates mt and applies error-compensated compression.Source: §11 · p. 15

2. All workers send its m(i)

• The server averages the workers’ messages and compresses the result again.Source: §2 · p. 15
• The server broadcasts mt+1 to all workers, which then update their local models.Source: §2 · p. 16
• The update rules for mt+1 and mt are summarized after the communication steps.The equations themselves are not present in these passages.Source: §2 · p. 16
• V is defined as a diagonal matrix with entries 1/√v1 through 1/√vd.Source: §2 · p. 16

12. Proof to Theorem 1

• The proof treats coordinate-specific learning rates as a scaled loss h with a uniform learning rate, then studies error-compensated momentum SGD.This equivalence is the setup for the proof of Theorem 1.Source: §12 · p. 17
• The proof introduces yt = xt − γ/(1 − β)(mt + δt−1) and derives its update using the momentum and compression-error terms.Source: §12 · p. 18
• The proof invokes L-Lipschitz smoothness and combines intermediate inequalities with Lemma 1 to obtain a bound.The displayed bound is not specified in these passages.Source: §12 · p. 19
• The analysis replaces coordinate-specific learning rates with a constant-rate loss transformation to apply the momentum-SGD lemmas.Source: §12 · pp. 17, 20
• For h, the effective Lipschitz coefficient is LVmax, and the compression error is transformed to V^(1/2)δt.The proof identifies these as changes needed when moving to the scaled-loss view.Source: §12 · p. 20

13. Proof to Corollary 1

• The proof treats f(y1) − f* , β, and L as constants when deriving the concluding inequality.Source: §13 · p. 21
• The proof ends after deriving the inequality.Source: §13 · p. 21