PaperlayerView exact revision on arXiv

Source-linked AI summary

One-Shot Federated Learning

Neel Guha, Ameet Talwalkar, Virginia Smith

arXiv:1902.11175v2cs.LGstat.ML

TL;DR

Federated learning faces communication, privacy, and device-heterogeneity challenges while conventional methods require repeated communication. This paper proposes one-shot learning using local-model ensembles, model selection, and distillation, achieving a 51.5% relative AUC gain over local baselines and reaching 90.1% of the unattainable ideal global model. The work is preliminary and identifies future directions including few-shot learning, privacy guarantees, personalization, and non-convex models.

  • Problem

    Federated learning involves distributed devices with communication bottlenecks, privacy constraints, non-IID data, and heterogeneous capacities, while current methods require numerous communication rounds.

    Source: §1 · p. 1, §2 · p. 1

  • Method

    The paper uses one communication round, local models trained to completion, ensemble selection, and distillation with unlabeled proxy data.

    Source: §1 · p. 1, §3 · pp. 2–3

  • Results

    51.5% relative ROC-AUC gain over local baselines brings the approaches within 90.1% of the unattainable ideal global model.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Ensemble methods and distillation yield promising results for one-shot federated learning and support communication-efficient global modeling.

    Source: §4 · p. 4, §5 · p. 4

    The work is a preliminary investigation, with future work needed for personalization, formal privacy guarantees, few-shot learning, and non-convex models.

    Source: §5 · p. 4

Abstract

from arXiv · show

We present one-shot federated learning, where a central server learns a global model over a network of federated devices in a single round of communication. Our approach - drawing on ensemble learning and knowledge aggregation - achieves an average relative gain of 51.5% in AUC over local baselines and comes within 90.1% of the (unattainable) global ideal. We discuss these methods and identify several promising directions of future work.

1 Introduction

One-shot federated learning learns a global model from distributed device data with one communication round, replacing iterative communication with local models and ensemble-based aggregation. The approaches achieve substantial gains over local baselines while approaching the ideal global model.

  • Federated learning trains across large IoT-device networks, but iterative optimization requires numerous device-server communication rounds.Source: §1 · p. 1
  • One-shot federated learning learns a global model using only a single communication round between devices and the central server.Source: §1 · p. 1
  • Local models trained to completion enable ensemble methods that capture global information across device-specific models.Source: §1 · p. 1
  • 51.5% relative ROC-AUC gain over local baselines brings the proposed approaches within 90.1% of the unattainable ideal global model.Source: §1 · p. 1

2 Related Work

Federated learning must address communication bottlenecks, privacy, non-IID device data, and heterogeneous device capacities. The paper motivates ensemble learning and distillation as alternatives to naive averaging for communication-efficient global modeling.

  • Federated learning faces communication bottlenecks, privacy requirements, non-IID device data, and heterogeneous storage, computation, and communication capacities.Source: §2 · p. 1
  • Current federated approaches rely on iterative optimization, repeatedly exchanging updates until convergence.Source: §2 · p. 1
  • Ensemble learning is proposed for federated settings because many devices have few local data points and model averaging can be complicated for deep or kernelized models.Source: §2 · p. 2
  • Unlabeled proxy data enables distillation to reduce global-model size and potentially provide privacy guarantees for models such as dual SVMs.Source: §2 · p. 2

3 Methods

The methods use completed local models to build selected prediction ensembles for supervised one-shot federated learning, then distill those ensembles when proxy data are available. Experiments compare ensemble strategies, baselines, and distilled models across federated datasets.

Ensemble (Supervised)

  • Local models are sent to the server, which curates an ensemble of k ≤ m models by averaging their predictions.The ensemble may include only selected device models because local-model quality can vary substantially.Source: §3 · p. 2

Ensemble Selection

  • Three selection protocols use validation performance, local training-data size, or random sampling to choose device models for the ensemble.Cross-validation and data selection apply server-defined thresholds before selecting models.Source: §3 · p. 2

Distillation (Semi-Supervised)

  • With unlabeled proxy data, distillation compresses the ensemble into a smaller model that approximately matches ensemble performance using relatively few proxy samples.The proxy-data comparison averages results over five trials.Source: §3 · p. 3, §4 · p. 4, Figure 3 · p. 4
  • Distillation minimizes the L2 difference between student and teacher predictions on proxy data and can reduce communication and support privacy-preserving learning.The privacy motivation is especially noted for dual SVMs whose local support vectors might otherwise be shared.Source: §3 · p. 3

Experimental Setup

  • The empirical study evaluates ensembles with k = 1, 10, 50, and 100 against local and unattainable ideal global classifiers on EMNIST, Sentiment140, and Gleam.The datasets cover handwriting, tweet sentiment, and Google Glass activity classification.Source: §4 · pp. 3–4

Results

  • Ensemble approaches outperform local baselines, while selected ensembles generally outperform full ensembles except on Gleam, which has relatively few devices.On Sent140, ensembles match high-performing local models while improving on moderate-to-poor local models.Source: Figure 1 · p. 3, §4 · p. 4

5 Conclusion and Future Work

The paper presents a preliminary investigation of one-shot federated learning and finds ensemble methods and distillation promising in federated settings. It identifies personalization, formal privacy analysis, few-shot communication, and non-convex models as future directions.

Conclusion

  • The work is a preliminary investigation whose experiments indicate promising results for ensembles and distillation in federated learning.The authors describe both techniques as unexplored in federated settings.Source: §5 · p. 4

Future Work

  • Future work includes device cohorts for personalized ensembles, formal privacy guarantees for distillation, few-shot federated learning, and non-convex models.The examples include geographically similar device groups and deep neural networks.Source: §5 · p. 4

Related papers

Loading 1902.11175v2…