PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Quantization of Model Updates for Communication-Efficient Federated Learning

Divyansh Jhunjhunwala, Advait Gadhikar, Gauri Joshi, Yonina C. Eldar

arXiv:2102.04487v1cs.LGcs.DCstat.ML

TL;DR

Federated learning is constrained by limited client uplink bandwidth, while quantization can reduce communication at the cost of a higher error floor. AdaQuantFL adapts quantization levels during training; experiments report reaching target training loss with fewer communicated bits than fixed quantization methods.

  • Problem

    Federated learning's limited client uplink bandwidth makes communicating model updates a major bottleneck, while compression can increase the error floor through higher update variance.

    Source: §1 · p. 1

  • Method

    AdaQuantFL uses convergence analysis in communicated bits to adapt the number of quantization levels based on the current training loss.

    Source: §1 · p. 1

  • Results

    AdaQuantFL reaches a given target training loss using much fewer bits than fixed quantization methods.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The authors identify applying AdaQuantFL to stochastic rotated and universal vector quantizers as a natural extension.

    Source: §6 · p. 4

    The convergence analysis assumes an unbiased quantizer with bounded variance and L-smooth local objectives.

    Source: §3 · pp. 2–3

Abstract

from arXiv · show

Communication of model updates between client nodes and the central aggregating server is a major bottleneck in federated learning, especially in bandwidth-limited settings and high-dimensional models. Gradient quantization is an effective way of reducing the number of bits required to communicate each model update, albeit at the cost of having a higher error floor due to the higher variance of the stochastic gradients. In this work, we propose an adaptive quantization strategy called AdaQuantFL that aims to achieve communication efficiency as well as a low error floor by changing the number of quantization levels during the course of training. Experiments on training deep neural networks show that our method can converge in much fewer communicated bits as compared to fixed quantization level setups, with little or no impact on training and test accuracy.

1. INTRODUCTION

Federated learning moves training onto clients that share model updates with a central server, making limited uplink bandwidth a major bottleneck. AdaQuantFL adapts quantization levels during training to balance communication efficiency with a low error floor.

1. INTRODUCTION

  • Federated learning distributes data across mobile clients, which train on-device and share model updates with a central server over bandwidth-limited uplinks.The setting raises communication challenges for distributed machine learning.Source: §1 · p. 1
  • Local updates reduce communication frequency, while compression reduces update size but can raise the training error floor by increasing update variance.Choosing quantization levels therefore requires balancing error against communication.Source: §1 · p. 1
  • AdaQuantFL adapts the number of quantization levels for model updates using convergence analysis in communicated bits and current training loss.The approach is intended to achieve a low error floor alongside communication efficiency.Source: §1 · p. 1

2. SYSTEM MODEL

The system trains a common global model from client datasets using local SGD and server aggregation. Clients communicate quantized updates, with the number of quantization levels trading communication cost against quantization variance.

2. SYSTEM MODEL

  • The system has n clients with individual datasets and trains a shared parameter vector by minimizing a data-weighted objective formed from client empirical risks.The dataset sizes determine the clients' fractions in the objective.Source: §2 · p. 1
  • Each client performs τ local SGD updates per round, then sends its quantized model update to the central server for global-model aggregation.The clients use stochastic gradients computed from local mini-batches.Source: §2 · p. 2
  • A quantized scalar update uses one sign bit, ⌈log2(s + 1)⌉ bits for its quantization level, and a 32-bit representation of its norm.The resulting per-round client communication cost is denoted C_s.Source: §2 · p. 2
  • The stochastic uniform quantizer is parameterized by s quantization levels and is unbiased, with variance decreasing as s increases.The passage states the variance property and identifies the quantizer used in the system model.Source: §2 · p. 2
  • Increasing quantization levels raises bits communicated per round but decreases the quantizer's variance upper bound.The paper uses this trade-off to motivate adapting the level during training.Source: §2 · p. 2

3. TRADE-OFF BETWEEN ERROR AND THE NUMBER OF BITS COMMUNICATED

AdaQuantFL analyzes convergence against communicated bits rather than training rounds, revealing how quantization level trades communication cost against the error bound. The bound motivates changing quantization as training progresses.

3. TRADE-OFF BETWEEN ERROR AND THE NUMBER OF BITS COMMUNICATED

  • Smaller s communicates fewer bits per round, allowing more rounds at a fixed bit budget and a faster initial training-loss drop, despite worse loss convergence by rounds.This motivates starting with small s and gradually increasing it to reach a lower error floor.Source: §3 · p. 2
  • AdaQuantFL bounds convergence error in communicated bits, whereas the prior result it builds on bounds error by training rounds.The total bits communicated by a client to the server up to a given time are denoted B.Source: §3 · p. 3
  • Under Assumptions 1–4 and a learning-rate condition, Theorem 1 gives an error upper bound in terms of B.The stated assumptions include unbiased quantization and stochastic gradients, smooth local objectives, and i.i.d. client data.Source: §3 · pp. 2–3
  • As s decreases, the A1 log2(4s) term in the bound falls while the A2/s^2 term rises, exposing the quantization error–variance trade-off.The authors identify the second term's increase with the added variance of quantized updates.Source: §3 · p. 3

4. PROPOSED ADAQUANTFL STRATEGY

AdaQuantFL selects quantization levels adaptively across fixed-bit communication intervals, using an error bound to guide the schedule. For the Vanilla CNN under non-i.i.d. data, it reaches an error floor of 0.02 in 0.3Gb, while the 2-bit method takes 1.8Gb.

4. PROPOSED ADAQUANTFL STRATEGY

  • AdaQuantFL divides training into communication intervals of B0 bits and chooses s to minimize the error upper bound at each cumulative bit count B.The schedule is based on the bound in Theorem 1.Source: §4 · p. 3
  • Setting the derivative of the error bound to zero gives a closed-form optimal s for each communication interval.At the start of an interval, clients treat the current model as a new initialization for selecting s over the next B0 bits.Source: §4 · p. 3
  • As the training objective f(wk) becomes smaller, the optimal s increases, matching the strategy of raising quantization levels as training progresses.The authors note that parameters needed for the optimal schedule, including L and σ^2, are unknown in practice.Source: §4 · p. 3
  • The method extends its adaptive schedule to a changing learning rate, and the initial s0 can be selected by grid search.The authors report s0 = 2 as a good choice in their experiments.Source: §4 · p. 3
  • 0.02 error floor in 0.3Gb for AdaQuantFL, versus 1.8Gb for the 2-bit method, on the Vanilla CNN with non-i.i.d. data.The figure caption identifies AdaQuantFL as achieving the lowest error floor among the compared methods.Source: Figure 3 · p. 4, §4 · p. 4

5. EXPERIMENTAL RESULTS

Experiments compare AdaQuantFL with fixed quantization across two datasets, two model architectures, and i.i.d. and non-i.i.d. client data; AdaQuantFL reaches an error floor using fewer bits in most cases.

5. EXPERIMENTAL RESULTS

  • The evaluation uses CIFAR-10 with ResNet-18 and Fashion MNIST with a Vanilla CNN, across i.i.d. and non-i.i.d. client data.Source: §5 · p. 4
  • Experiments use 10 local updates, learning rate 0.1, four ResNet-18 clients, and eight CNN clients.In the variable-learning-rate setting, the rate is reduced by 0.9 every 100 rounds.Source: §5 · p. 4
  • AdaQuantFL reaches an error floor using much fewer bits in most cases.The experiments compare it with fixed quantization schemes.Source: §5 · p. 4

6. CONCLUSION

The conclusion presents AdaQuantFL as an adaptive quantization strategy for compressed federated-learning updates, based on error-versus-bits convergence analysis. Experiments show it needs fewer bits to converge during training.

6. CONCLUSION

  • AdaQuantFL adapts quantization levels for compressed model updates in federated learning.Source: §6 · p. 4
  • The strategy is based on an error-versus-bits convergence analysis.Source: §6 · p. 4
  • Experiments show AdaQuantFL requires fewer bits to converge during training.Source: §6 · p. 4
  • The conclusion identifies stochastic rotated and universal vector quantizers as possible extensions.Source: §6 · p. 4

A. PROOF OF THEOREM 1

The proof adapts an existing convergence result and expresses it in terms of communicated bits. It substitutes the stochastic uniform quantizer's bit cost and variance bound into the resulting inequality.

A. PROOF OF THEOREM 1

  • The proof adapts a result from that holds after K rounds of training.The averaged model across clients at each step is denoted by k,t.Source: A. PROOF OF THEOREM 1 · p. 7
  • The proof uses the definition of B to express the number of rounds K in terms of communicated bits.Source: A. PROOF OF THEOREM 1 · p. 7
  • Substituting the quantizer's bit cost and variance bound gives the inequality, using ⌈log2(s + 1)⌉ ≤ log2(4s) for s ≥ 1.The proof then defines constants A1, A2, and A3.Source: A. PROOF OF THEOREM 1 · p. 7

B. PROOF OF EQN. (10)

The proof of Eqn. (10) minimizes an objective in s, while the convergence-guarantee section states a bound for adaptive quantization and variable learning rates under stated assumptions.

B. PROOF OF EQN. (10)

C. CONVERGENCE GUARANTEE FOR ADAQUANTFL

C.1. Proof of Convergence:

The proof states assumptions, gives a sufficient condition for the upper bound to vanish as K tends to infinity, and concludes convergence.

  • A sufficient condition is stated for the upper bound in (16) to converge to zero as K tends to infinity.Source: §C.1 · p. 9
  • The proof notes that s_k is at least 1 for every training round.Source: §C.1 · p. 9
  • The passage concludes, “This completes the proof of convergence.”Source: §C.1 · p. 9

D. ADDITIONAL RESULTS

Additional results report test accuracy comparisons with the 16-bit method for ResNet-18 and CNN experiments. Under a variable learning rate, AdaQuantFL does well on ResNet-18 but does not outperform the 4-bit CNN setting.

Loading 2102.04487v1…