Source-linked AI summary
One-Shot Federated Learning
Neel Guha, Ameet Talwalkar, Virginia Smith
TL;DR
Federated learning faces communication, privacy, and device-heterogeneity challenges while conventional methods require repeated communication. This paper proposes one-shot learning using local-model ensembles, model selection, and distillation, achieving a 51.5% relative AUC gain over local baselines and reaching 90.1% of the unattainable ideal global model. The work is preliminary and identifies future directions including few-shot learning, privacy guarantees, personalization, and non-convex models.
Problem
Federated learning involves distributed devices with communication bottlenecks, privacy constraints, non-IID data, and heterogeneous capacities, while current methods require numerous communication rounds.
Method
The paper uses one communication round, local models trained to completion, ensemble selection, and distillation with unlabeled proxy data.
Results
51.5% relative ROC-AUC gain over local baselines brings the approaches within 90.1% of the unattainable ideal global model.
Takeaways & Limitations
Ensemble methods and distillation yield promising results for one-shot federated learning and support communication-efficient global modeling.
Takeaways & Limitations
The work is a preliminary investigation, with future work needed for personalization, formal privacy guarantees, few-shot learning, and non-convex models.
Abstract
from arXiv · showhide
We present one-shot federated learning, where a central server learns a global model over a network of federated devices in a single round of communication. Our approach - drawing on ensemble learning and knowledge aggregation - achieves an average relative gain of 51.5% in AUC over local baselines and comes within 90.1% of the (unattainable) global ideal. We discuss these methods and identify several promising directions of future work.
1 Introduction
One-shot federated learning learns a global model from distributed device data with one communication round, replacing iterative communication with local models and ensemble-based aggregation. The approaches achieve substantial gains over local baselines while approaching the ideal global model.
- Federated learning trains across large IoT-device networks, but iterative optimization requires numerous device-server communication rounds.
- One-shot federated learning learns a global model using only a single communication round between devices and the central server.
- Local models trained to completion enable ensemble methods that capture global information across device-specific models.
- 51.5% relative ROC-AUC gain over local baselines brings the proposed approaches within 90.1% of the unattainable ideal global model.
2 Related Work
Federated learning must address communication bottlenecks, privacy, non-IID device data, and heterogeneous device capacities. The paper motivates ensemble learning and distillation as alternatives to naive averaging for communication-efficient global modeling.
- Federated learning faces communication bottlenecks, privacy requirements, non-IID device data, and heterogeneous storage, computation, and communication capacities.
- Current federated approaches rely on iterative optimization, repeatedly exchanging updates until convergence.
- Ensemble learning is proposed for federated settings because many devices have few local data points and model averaging can be complicated for deep or kernelized models.
- Unlabeled proxy data enables distillation to reduce global-model size and potentially provide privacy guarantees for models such as dual SVMs.
3 Methods
The methods use completed local models to build selected prediction ensembles for supervised one-shot federated learning, then distill those ensembles when proxy data are available. Experiments compare ensemble strategies, baselines, and distilled models across federated datasets.
- Ensemble (Supervised): Local models are sent to the server, which curates an ensemble of k ≤ m models by averaging their predictions.The ensemble may include only selected device models because local-model quality can vary substantially.
- Ensemble Selection: Three selection protocols use validation performance, local training-data size, or random sampling to choose device models for the ensemble.Cross-validation and data selection apply server-defined thresholds before selecting models.
- Distillation (Semi-Supervised): With unlabeled proxy data, distillation compresses the ensemble into a smaller model that approximately matches ensemble performance using relatively few proxy samples.The proxy-data comparison averages results over five trials.
- Distillation (Semi-Supervised): Distillation minimizes the L2 difference between student and teacher predictions on proxy data and can reduce communication and support privacy-preserving learning.The privacy motivation is especially noted for dual SVMs whose local support vectors might otherwise be shared.
- Experimental Setup: The empirical study evaluates ensembles with k = 1, 10, 50, and 100 against local and unattainable ideal global classifiers on EMNIST, Sentiment140, and Gleam.The datasets cover handwriting, tweet sentiment, and Google Glass activity classification.
- Results: Ensemble approaches outperform local baselines, while selected ensembles generally outperform full ensembles except on Gleam, which has relatively few devices.On Sent140, ensembles match high-performing local models while improving on moderate-to-poor local models.
5 Conclusion and Future Work
The paper presents a preliminary investigation of one-shot federated learning and finds ensemble methods and distillation promising in federated settings. It identifies personalization, formal privacy analysis, few-shot communication, and non-convex models as future directions.
- Conclusion: The work is a preliminary investigation whose experiments indicate promising results for ensembles and distillation in federated learning.The authors describe both techniques as unexplored in federated settings.
- Future Work: Future work includes device cohorts for personalized ensembles, formal privacy guarantees for distillation, few-shot federated learning, and non-convex models.The examples include geographically similar device groups and deep neural networks.