PaperlayerView exact revision on arXiv

Source-linked AI summary

Federated Learning With Quantized Global Model Updates

Mohammad Mohammadi Amiri, Deniz Gunduz, Sanjeev R. Kulkarni, H. Vincent Poor

arXiv:2006.10672v2cs.ITcs.DCcs.LG

TL;DR

Federated learning must reduce communication beyond device-to-server aggregation because wireless links can constrain global-model broadcasting. The paper proposes LFL, which quantizes global-model updates and local updates, and reports communication savings with performance close to fully lossless FL and better than prior PS-to-device schemes.

  • Problem

    Wireless FL has limited communication capacity, while prior work mainly assumed perfect PS-to-device global-model broadcasting.

  • Method

    LFL quantizes the global-model update relative to devices’ previous estimates and quantizes local model updates with error compensation.

  • Results

    LFL significantly reduces communication cost, shows no visible performance gap from fully lossless FL, and significantly outperforms prior PS-to-device quantization schemes.

  • Takeaways & Limitations

    Quantizing global-model updates rather than global models provides a more accurate device estimate at the same communication load and supports lossy broadcasting in wireless FL.

  • Takeaways & Limitations

    The convergence analysis assumes that devices send their local updates accurately, and the impact of imperfect broadcasting depends on quantization level and ε.

Abstract

from arXiv · show

We study federated learning (FL), which enables mobile devices to utilize their local datasets to collaboratively train a global model with the help of a central server, while keeping data localized. At each iteration, the server broadcasts the current global model to the devices for local training, and aggregates the local model updates from the devices to update the global model. Previous work on the communication efficiency of FL has mainly focused on the aggregation of model updates from the devices, assuming perfect broadcasting of the global model. In this paper, we instead consider broadcasting a compressed version of the global model. This is to further reduce the communication cost of FL, which can be particularly limited when the global model is to be transmitted over a wireless medium. We introduce a lossy FL (LFL) algorithm, in which both the global model and the local model updates are quantized before being transmitted. We analyze the convergence behavior of the proposed LFL algorithm assuming the availability of accurate local model updates at the server. Numerical experiments show that the proposed LFL scheme, which quantizes the global model update (with respect to the global model estimate at the devices) rather than the global model itself, significantly outperforms other existing schemes studying quantization of the global model at the PS-to-device direction. Also, the performance loss of the proposed scheme is marginal compared to the fully lossless approach, where the PS and the devices transmit their messages entirely without any quantization.

1 Introduction

Federated learning coordinates local training while keeping data on devices, but wireless bandwidth and channel limitations make communication efficiency essential. This paper addresses the underexplored PS-to-device direction by quantizing global model updates rather than the global model itself.

  • Federated learning trains a global model from device-local data under parameter-server coordination without moving the data off devices.
  • Wireless FL communication is constrained by limited bandwidth, power, and channel impairments, making reduced communication requirements important.
  • Prior communication-efficient FL work mainly optimized device-to-PS links while assuming perfect global-model broadcasting to devices.
  • The paper introduces LFL, which broadcasts a quantized global model update so devices form a lossy global-model estimate for local training.
  • The proposed scheme significantly reduces communication cost, nearly matches fully lossless performance, and outperforms prior PS-to-device quantization schemes.

2 Lossy Federated Learning (LFL) Algorithm

LFL compresses both PS-to-device global-model updates and device-to-PS local updates. Its key design broadcasts updates relative to devices’ previous estimates, while local updates use quantization with error compensation.

  • Global model broadcasting: At each iteration, the PS broadcasts a compressed global-model representation, and devices recover an estimate before performing local SGD.
  • Global model broadcasting: LFL quantizes the update θ(t) − ˆθ(t − 1) rather than the global model itself.
  • Global model broadcasting: The quantization error parameter ε increases with skewness in entry magnitudes, equals 0 for equal magnitudes, and equals 1 for a one-nonzero-entry vector.
  • Local update aggregation: After τ-step local SGD, each device forms its local model update and transmits it using quantization with accumulated error compensation.
  • Local update aggregation: The method does not use PS-side error compensation because the update-based broadcast naturally accumulates quantization error at the PS.
  • Local update aggregation: The framework is not restricted to a specific quantization technique; any compression technique can be used within it.

3 Convergence Analysis of LFL Algorithm

The convergence analysis characterizes LFL under smoothness, strong convexity, bounded stochastic gradients, and accurate device-to-server updates. With decreasing learning rates and sufficiently small quantization distortion, the expected loss converges to the optimum.

  • The analysis assumes smooth and strongly convex local losses, bounded expected squared stochastic gradients, and lossless transmission of local updates.
  • With a decreasing learning rate satisfying lim_t→∞ η(t) = 0 and sufficiently small ε, the expected loss converges asymptotically to the minimum loss.
  • The analysis reports numerically that ε ≈ 10^-3 satisfies the required inequality for LFL because update entries are not expected to have highly diverse magnitudes.
  • The convergence bound includes a broadcasting-error term that decreases with q1 and increases linearly with ε.

4 Numerical Experiments

Experiments evaluate LFL against lossy and lossless broadcasting schemes on MNIST and CIFAR-10 under iid and non-iid data distributions. LFL maintains comparable accuracy while reducing broadcast communication.

  • Experimental setup: Experiments use CNNs on MNIST and CIFAR-10 with 40 devices, evaluating test accuracy under iid and non-iid data distributions.MNIST is used for non-iid data and CIFAR-10 for iid data.
  • Compared approaches: The compared methods include lossy transformed global model, lossy global model, lossless broadcasting, fully lossless communication, and LFL.LTGM applies a linear transform before quantization, whereas LGM directly quantizes the global model with error compensation.
  • MNIST results: On MNIST, LFL with (q1, q2) = (2, 2) performs as well as fully lossless and LB approaches while saving a factor of 12.77 in broadcast bits versus LB.The result is reported for the iid scenario described in the experiment passage.
  • CIFAR-10 results: For CIFAR-10, LFL with (q1, q2) = (5, 3) uses 9.2× less communication than LB without visible performance degradation relative to fully lossless and LB.The comparison uses q2 = 3 for LB.

5 Conclusion

The conclusion addresses FL's bandwidth bottleneck by broadcasting quantized global model updates and aggregating quantized local updates. Convergence analysis and experiments support the efficiency of LFL under accurate local updates at the server.

  • Conclusion: LFL reduces communication by broadcasting quantized global model updates that devices use to estimate the current global model for local SGD.The PS aggregates quantized local model updates to update the global model.
  • Conclusion: The paper derives convergence guarantees for LFL assuming accurate local model updates at the parameter server.
  • Conclusion: Numerical experiments show that LFL provides an accurate global-model estimate to devices while remaining close to fully lossless performance.

A Stochastic quantization

The stochastic quantizer operates coordinatewise by mapping normalized values to neighboring quantization levels with probabilities chosen from the value's fractional position. The resulting vector representation has an explicit bit cost.

  • Vector quantization: The vector quantizer applies the scalar quantizer independently to every coordinate, producing Q(x,q) = [Q(x1,q), ..., Q(xd,q)]^T.
  • Quantizer definition: For x in [0,1], Q(x,q) selects one of the two neighboring levels l/q and (l+1)/q.The interval index l satisfies x ∈ [l/q, (l+1)/q).
  • Quantizer definition: The upper level is selected with probability xq − l, while the lower level is selected with probability 1 − (xq − l).
  • Communication cost: The quantized representation requires RQ = 64 + d(1 + log2(q + 1)) bits.The count includes 64 bits for xmax and xmin, d sign bits, and d log2(q+1) quantization bits.
  • Normalization: Unlike standard QSGD normalization by ||x||2, this scheme normalizes entries using xmax − xmin.

B Proof of Lemma 1

The proof uses the bounded product of the stochastic quantizer's interpolation probabilities and the definition of the normalization error parameter to establish the stated bounds.

  • Proof step: The proof bounds the product (xq − l)(1 − xq + l) by 1/4.
  • Proof step: It then invokes the definitions of the quantizer bounds and ε = (xmax − xmin)^2 / ||x||2 to obtain the subsequent result.

C Proof of Theorem 1

The proof of Theorem 1 derives successive bounds on terms in the relevant inequality and invokes the stated assumptions and convexity properties. Substitution of these bounds yields the theorem under a bounded step-size condition.

  • The proof bounds the last two terms on the right-hand side of (24) using convexity and successive algebraic rewrites.
  • Assumption 3 supplies one intermediate inequality, while Assumption 2 and F* ≤ F justify additional terms in the bound.
  • The proof is located in Appendix D and Appendix E.
  • Substituting inequalities (27) and (28) into (26), together with (25), produces an upper bound used in (24).
  • Theorem 1 follows from inequality (32) when the step size satisfies 0 < η(t) ≤ min.

D Proof of Lemma 2

The proof of Lemma 2 separately bounds the first and second terms on the right-hand side of (33). These bounds are combined through inequalities (35), (36), and (39) to complete the lemma.

  • The first term on the right-hand side of (33) is bounded using an inequality that invokes Assumption 3.
  • The second term on the right-hand side of (33) is bounded using expressions involving gradients and Assumption 2.
  • Cauchy–Schwarz yields an intermediate bound, which is substituted into (37) to obtain (39).
  • Using inequalities (35) and η(t) ≤ 1, substituting (36) and (39) into (33) completes the proof of Lemma 2.

E Proof of Lemma 3

The proof of Lemma 3 upper-bounds an expectation-related quantity through algebraic substitution and convexity. It also defines ε as the maximum of ε(t) over t.

  • The proof uses the equality following from (23a) before upper-bounding E.
  • It defines ε ≜ max_t{ε(t)} in the derivation associated with inequality (44).
  • An inequality derived using Assumption 3 reduces (42) before being substituted into (40).
Loading 2006.10672v2…