Source-linked AI summary
Communication-Efficient Learning of Deep Networks from Decentralized Data
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Agüera y Arcas
TL;DR
Training models from rich mobile-device data is constrained by its privacy-sensitive, decentralized nature. This paper introduces Federated Learning through iterative model averaging and finds it robust to unbalanced, non-IID data while substantially reducing communication rounds versus synchronized SGD.
Problem
Mobile devices contain rich, largely private data, creating a need to train shared models without centrally storing users’ raw training data.
Method
Federated Learning combines local stochastic gradient descent on participating devices with server-side averaging of client model updates.
Results
FedAvg trains high-quality models across multiple architectures using relatively few communication rounds and reaches higher test accuracy than FedSGD across three model classes.
Takeaways & Limitations
Federated learning provides a practical way to train deep models on decentralized data while limiting the need for direct access to raw training data.
Takeaways & Limitations
The experiments use a controlled setting with synchronous communication rounds, a fixed client set, and fixed local datasets.
Abstract
from arXiv · showhide
Modern mobile devices have access to a wealth of data suitable for learning models, which in turn can greatly improve the user experience on the device. For example, language models can improve speech recognition and text entry, and image models can automatically select good photos. However, this rich data is often privacy sensitive, large in quantity, or both, which may preclude logging to the data center and training there using conventional approaches. We advocate an alternative that leaves the training data distributed on the mobile devices, and learns a shared model by aggregating locally-computed updates. We term this decentralized approach Federated Learning. We present a practical method for the federated learning of deep networks based on iterative model averaging, and conduct an extensive empirical evaluation, considering five different model architectures and four datasets. These experiments demonstrate the approach is robust to the unbalanced and non-IID data distributions that are a defining characteristic of this setting. Communication costs are the principal constraint, and we show a reduction in required communication rounds by 10-100x as compared to synchronized stochastic gradient descent.
1 Introduction
Federated Learning trains shared models from privacy-sensitive, decentralized mobile data without uploading clients’ raw datasets, using coordinated local updates and server aggregation. The setting is defined by non-IID and unbalanced client data, massive distribution, and communication constraints that dominate computation.
- Federated Learning: Federated Learning lets clients jointly train a shared model while their local training datasets remain on-device and only model updates are sent to a coordinating server.This decouples model training from direct raw-data access and can reduce privacy and security risks by limiting the attack surface to devices rather than devices and the cloud.
- Contributions: FederatedAveraging combines local stochastic gradient descent on clients with server-side model averaging as a practical algorithm for decentralized training.The paper identifies decentralized mobile-data training, a practical algorithm, and extensive empirical evaluation as its primary contributions.
- Federated Learning: Ideal federated-learning problems use mobile-device data that improves on data-center proxy data, is privacy sensitive or large, and supports supervised learning.Image classification and language modeling are presented as examples, including photo selection, voice recognition, and text entry.
- Privacy: Federated learning transmits minimal model-improvement updates rather than persisted raw data, but privacy benefits depend on update content and attacks remain possible.Sparse bag-of-words gradients can reveal entered words, whereas aggregated dense-model gradients are harder targets for attackers.
- Federated Optimization: Federated optimization is typically non-IID, unbalanced, and massively distributed because clients reflect individual usage patterns and vary in local data volume.The paper emphasizes non-IID and unbalanced data together with critical communication constraints, while leaving many deployment issues beyond scope.
- Federated Optimization: Communication costs dominate federated optimization, with clients constrained by upload bandwidth of 1 MB/s or less and participation tied to charging, power, and unmetered Wi‑Fi.The experimental setting uses synchronous communication rounds with random client participation from a fixed client set.
2 The FederatedAveraging Algorithm
The section develops FederatedAveraging from federated SGD by replacing one server-side update with multiple local client updates followed by weighted model averaging. It motivates this design through communication efficiency and evidence that averaging models from a shared initialization can reduce loss, while noting a potential non-convex-objective limitation.
- Communication efficiency: 50000 steps on minibatches of size 60 illustrate that naively performing one batch-gradient calculation per communication round can require very large training-round counts.This approach is computationally efficient but may produce good models only after many rounds.
- Federated SGD baseline: Large-batch synchronous SGD selects a C fraction of clients each round and aggregates gradients over their local data as the federated baseline.The fraction C controls global participation in each communication round.
- FederatedAveraging: FedAvg lets each client perform multiple local gradient-descent updates before the server averages the resulting models.This extends the one-step local update used by FedSGD and increases client-side computation per communication round.
- Algorithm parameters: FedAvg computation is controlled by C, the fraction of clients, B, the local minibatch size, E, the number of local epochs, and η, the learning rate.Algorithm 1 defines K clients and these local-training parameters.
- Model averaging: For general non-convex objectives, averaging models in parameter space can produce an arbitrarily bad model.The section contrasts this limitation with the favorable shared-initialization MNIST result.
- Model averaging: Shared-initialization averaging produces significantly lower full-training-set loss than either independently trained parent model on MNIST.The same shared starting model is used across FedAvg rounds, motivating this parameter-averaging behavior.
3 Experimental Results
Across MNIST, Shakespeare, and a large-scale next-word prediction task, FedAvg substantially reduced communication rounds while often achieving higher accuracy than FedSGD, including under non-IID and unbalanced data. However, very large local training epochs could cause FedAvg to plateau or diverge.
- Increasing computation per client: 35× and 46× fewer rounds were required on IID MNIST for the CNN and 2NN, respectively, when increasing computation per client.Speedups on pathologically partitioned non-IID MNIST were smaller but still substantial at 2.8–3.7×.
- Increasing computation per client: 95× speedup was achieved on unbalanced, non-IID Shakespeare data, compared with 13× on balanced IID data.The paper conjectures that larger local datasets for some roles make increased local training especially valuable.
- Accuracy and convergence: FedAvg converged to higher test accuracy than FedSGD across all three model classes; the MNIST CNN reached 99.44% in 300 rounds versus FedSGD’s 99.22% after 1200 rounds.The comparison used B = 10, E = 20 for FedAvg and B = ∞, E = 1 for FedSGD.
- Limitations: Very large numbers of local epochs could cause FedAvg to plateau or diverge on the Shakespeare LSTM problem.This behavior motivated caution about large E, particularly during later convergence stages.
- Comparison with standard SGD: 197,500 minibatch updates produced 86% accuracy for centralized SGD, whereas FedAvg reached 85% after only 2,000 communication rounds.The centralized baseline trained on the full unpartitioned training set with minibatches of size 100.
- Large-scale LSTM experiments: 23× fewer communication rounds were needed for FedAvg to reach 10.5% accuracy than FedSGD in the large-scale next-word prediction experiment.FedSGD required 820 rounds with η = 18.0, while FedAvg reached 10.5% in 35 rounds with η = 9.0.
4 Conclusions and Future Work
The experiments indicate that federated learning is practical: FedAvg trains high-quality models with relatively few communication rounds across diverse architectures. Future work includes stronger privacy guarantees and further communication-cost reductions.
- Conclusions: FedAvg trains high-quality models with relatively few communication rounds across five model architectures.The architectures include a multi-layer perceptron, two convolutional NNs, a two-layer character LSTM, and a large-scale word-level LSTM.
- Future Work: Stronger privacy guarantees through differential privacy, secure multi-party computation, or their combination remain an interesting direction for future work.Both technique classes apply most naturally to synchronous algorithms such as FedAvg.
- Future Work: Subsequent work introduced efficient secure aggregation and algorithms for further decreasing communication costs.Bonawitz et al. presented the secure aggregation protocol, while Konečný et al. presented communication-reduction algorithms.
A Supplemental Figures and Tables
The supplemental figures and table examine convergence, accuracy, communication speedups, minibatch efficiency, and local-epoch effects across MNIST and language-model experiments. They vary client fraction, local epochs, learning rates, and data heterogeneity while comparing FedAvg with relevant baselines.
- Convergence: Figure 6 examines MNIST CNN training-set convergence with C = 0.1 over a longer training range than Figure 2.The y-axis is logarithmic.
- Accuracy across data distributions: Figure 7 compares MNIST 2NN test accuracy versus communication rounds for IID and pathological 2-digits-per-client non-IID data with C = 0.1 and optimized η.The two data settings occupy separate columns.
- Local epochs: Figure 8 studies MNIST CNN training loss when many local epochs are performed between averaging steps, fixing B = 10 and C = 0.1.Learning rates and y-axis scales differ because of the difficulty of the pathological non-IID MNIST dataset.
- Communication efficiency: Table 4 reports communication-round speedups for FedAvg versus FedSGD on MNIST 2NN to reach 97% target accuracy.FedSGD is listed in the first row as the comparison baseline.
- Computation efficiency: Figure 9 compares test accuracy against minibatch gradient computations for sequential SGD and FedAvg across client fractions C and local epochs E.The experiments use B = 50, and C = 0 denotes one client per round.
- Language-model learning curves: Figure 10 shows word-LSTM learning curves evaluated every 20 rounds, with FedAvg performing better at E = 1 than E = 5 and showing lower accuracy variance than FedSGD.The comparison concerns evaluation-round accuracy and variance.