PaperlayerView exact revision on arXiv

Source-linked AI summary

Salvaging Federated Learning by Local Adaptation

Tao Yu, Eugene Bagdasaryan, Vitaly Shmatikov

arXiv:2002.04758v3cs.LGcs.AIcs.DCstat.ML

TL;DR

The paper asks whether federated models benefit individual participants and how participants can improve them locally without changing the federated-learning framework. It evaluates local adaptation under privacy and robustness protections and finds that adaptation improves models for participants across the studied settings, including those whose local models initially outperform federated models.

  • Problem

    Federated models may be less accurate than independently trained local models for many participants, undermining their incentive to participate, especially under privacy and robustness protections.

  • Method

    The paper evaluates fine-tuning, multi-task learning, and knowledge distillation as local adaptation methods that individual participants can deploy without changing aggregation or coordinating with others.

  • Results

    For 80% of word-prediction participants, the adapted robust model outperforms their local models, while participants with inaccurate local models obtain the biggest adaptation improvements.

  • Takeaways & Limitations

    Local adaptation can improve participant-specific federated models and restore an incentive to participate for users whose local models initially outperform federated models.

  • Takeaways & Limitations

    Some word-prediction participants, especially with ROBUST-FED, never recover local-model accuracy after adaptation, whereas adapted image-classification models always exceed local models.

Abstract

from arXiv · show

Federated learning (FL) is a heavily promoted approach for training ML models on sensitive data, e.g., text typed by users on their smartphones. FL is expressly designed for training on data that are unbalanced and non-iid across the participants. To ensure privacy and integrity of the fedeated model, latest FL approaches use differential privacy or robust aggregation. We look at FL from the \emph{local} viewpoint of an individual participant and ask: (1) do participants have an incentive to participate in FL? (2) how can participants \emph{individually} improve the quality of their local models, without re-designing the FL framework and/or involving other participants? First, we show that on standard tasks such as next-word prediction, many participants gain no benefit from FL because the federated model is less accurate on their data than the models they can train locally on their own. Second, we show that differential privacy and robust aggregation make this problem worse by further destroying the accuracy of the federated model for many participants. Then, we evaluate three techniques for local adaptation of federated models: fine-tuning, multi-task learning, and knowledge distillation. We analyze where each is applicable and demonstrate that all participants benefit from local adaptation. Participants whose local models are poor obtain big accuracy improvements over conventional FL. Participants whose local models are better than the federated model\textemdash and who have no incentive to participate in FL today\textemdash improve less, but sufficiently to make the adapted federated model better than their local models.

1. Introduction

The paper examines federated learning from individual participants’ perspective, finding that privacy and robustness protections can reduce their accuracy and incentive to participate. It evaluates local adaptation as a unilateral way to improve participant-specific models.

  • Federated learning trains models on sensitive, unbalanced, non-iid data distributed across participants.
  • Participants may lack incentive to participate when federated models are less accurate than models trained independently on their own data.
  • Privacy and robustness protections further reduce federated-model accuracy for many participants, worsening this incentive problem.
  • Local adaptation converts one federated model into individual models without changing aggregation or requiring cooperation from other participants.
  • 80% of word-prediction participants receive adapted robust models that outperform their local models.
  • The paper evaluates fine-tuning, multi-task learning, and knowledge distillation, with the largest improvements for participants whose local models are inaccurate.

2. Related Work

Related work has examined privacy, robustness, individual accuracy, personalization, and federated-model adaptation, but leaves several participant-level and purely local questions open.

  • Federated-learning updates can leak training data, while malicious participants can inject unwanted behaviors into the joint model.
  • Differential privacy limits training-data leakage, and robust aggregation limits individual participants’ influence on the model.
  • Most prior work does not measure federated-model accuracy separately for individual non-iid participants.
  • Existing personalization methods include speaker adaptation and multi-task learning, but many require ensembles, speech-specific settings, or framework changes.
  • Prior personalization papers do not jointly investigate individual incentives, privacy and integrity effects, and purely local adaptation beyond fine-tuning.

3. Background

The background describes federated learning as server-coordinated distributed training, then explains how differential privacy and robust aggregation modify aggregation to protect data and model integrity.

  • Federated learning trains a shared model across multiple participants’ data in a distributed learning paradigm.
  • Each round, the server sends the current global model to selected participants, receives locally updated models, and aggregates them into the next global model.
  • The framework targets millions of participants with non-iid data distributions, including predictive-keyboard and biomedical applications.
  • Differential privacy clips participant updates and adds random noise before aggregation.
  • The privacy guarantee depends on selecting the clipping bound S and noise σ for a specified (ϵ, δ) guarantee.
  • Median aggregation replaces averaging with an element-wise median to limit the influence of malicious participants and outliers.

4. Tasks

The experiments evaluate federated learning on next-word prediction and non-iid CIFAR-10 image classification using averaging, differential privacy, and robust median aggregation.

  • The study compares BASIC-FED, DP-FED, and ROBUST-FED on next-word prediction and CIFAR-10 image classification.
  • DP-FED clips updates at S = 15 and adds Gaussian noise with σ = 0.01, while ROBUST-FED uses coordinatewise medians instead of means.
  • The Reddit word-prediction task uses 80,000 participants with 150–500 posts each, preserving unbalanced and heterogeneous user data.
  • Word-prediction models use two-layer LSTMs with 200 hidden units and 10 million parameters, trained for 5,000 rounds with 100 participants per round.
  • Figure 1 reports federated-model accuracy improvements over local models trained from scratch for word prediction and image classification.
  • CIFAR-10 is distributed across 100 participants using a Dirichlet distribution with α = 0.9 to simulate non-iid data.

5. Privacy and robustness destroy individual accuracy

Individual accuracy, rather than aggregate accuracy, determines whether participants benefit from federated learning. Privacy and robustness protections can worsen this mismatch, leaving federated models less accurate than local models for many participants.

  • Individual accuracy: Global holdout accuracy may not represent whether a federated model is accurate for a specific participant with non-iid, idiosyncratic data.The paper therefore evaluates federated models on individual participants’ data.
  • Unprotected federated learning: 9.22% (7377) of participants had worse word-prediction accuracy with BASIC-FED than with their local models, while BASIC-FED outperformed all but one image-classification local model.These comparisons use local models trained independently by individual participants.
  • Privacy protection: 21.16% (16,931) of word-prediction participants and 11% (11) of image-classification participants had lower accuracy with DP-FED than with local models.The comparison is unfavorable for federated learning when differential privacy is applied.
  • Robustness protection: 52.15% (41,720) of word-prediction participants and 34% (34) of image-classification participants had lower accuracy with ROBUST-FED than with local models.Robust aggregation produces the largest reported disadvantage relative to independently trained local models.
  • Underlying tradeoff: Protecting privacy and integrity requires limiting individual contributions, but that limitation can produce an inaccurate joint model for participants.Learning a model accurate for every participant instead requires incorporating contributions from all participants.

6. Local adaptation

The paper adapts a federated model locally using fine-tuning, multi-task learning, or knowledge distillation. These methods reuse the federated model while tailoring it to each participant’s data and addressing overfitting on small local datasets.

  • Adaptation mechanisms: Local adaptation uses fine-tuning, multi-task learning, and knowledge distillation to modify the federated model for an individual participant.The local word-prediction and image-classification adaptation experiments use separate training budgets and learning rates.
  • Fine-tuning: Fine-tuning retrains all federated-model parameters on local data, while Freezebase freezes base layers and updates only the top layer.Fine-tuning leverages the federated model’s feature-extraction network instead of learning it from scratch.
  • Multi-task learning: Multi-task learning optimizes local adaptation for a participant while retaining performance on the union of all participants’ data.The method initializes the adapted model from the federated model and uses Fisher information to weight the global task against the local task.
  • Knowledge distillation: Knowledge distillation treats the federated model as teacher and the adapted model as student, with both models sharing the same structure and initialization.The local student dataset is a small subset of the teacher’s training dataset, and logit similarity is used to mitigate overfitting.
  • Knowledge distillation: The distillation loss combines cross-entropy with a KL-divergence term over the teacher and adapted models’ outputs.The prose defines GT(x) and A(x) as pre-softmax logits; α weights the losses and K is the temperature constant.

7. Local adaptation gives an incentive to participate in federated learning

Local adaptation improves federated models for individual participants across aggregation methods and can make participation worthwhile, especially for those with inaccurate local models. The gains vary by adaptation technique, participant profile, and aggregation method, with some word-prediction participants still failing to recover local accuracy.

  • 7.1. Results of adaptation: Local adaptation evaluates fine-tuning, federated backpropagation, multi-task learning, and knowledge distillation for BASIC-FED, DP-FED, and ROBUST-FED.Adaptation techniques are compared by the accuracy improvement they produce for individual participants.
  • 7.1. Results of adaptation: 2.32%, 2.12%, and 2.12% are the mean word-prediction accuracy improvements for BASIC-FED, DP-FED, and ROBUST-FED, respectively.These gains make up the losses attributed to differential privacy and robust aggregation.
  • 7.1. Results of adaptation: 2.98%, 6.83%, and 6.34% are the mean image-classification accuracy improvements for BASIC-FED, DP-FED, and ROBUST-FED, respectively.The corresponding privacy and robustness losses are -7.83% and -11.89%.
  • 7.1. Results of adaptation: Only 28 (0.04%) participants have adapted BASIC-FED below their local word-prediction models, while adapted BASIC-FED beats all local image-classification models.For DP-FED, the corresponding word-prediction count is 1465 (1.85%).
  • 7.1. Results of adaptation: 18.72% of participants have adapted ROBUST-FED below their local word-prediction models, although adapted ROBUST-FED beats all local image-classification models.The reported count is 14809 participants.
  • 7.2. Analysis: Adaptation gives accurate-local-model participants an incentive to participate because adapted federated models outperform their local models, although gains are smaller than for low-accuracy participants.The largest improvements accrue to participants with inaccurate local models, while adaptation also improves already-accurate federated models.
  • 7.2. Analysis: Adaptation helps most participants with simple, small datasets, where low federated accuracy is attributed to limited local data.Vocabulary size and total word count are used to characterize data complexity and size.
  • 7.2. Analysis: For BASIC-FED, the simplest FB technique performs best; for DP-FED and ROBUST-FED, multi-task learning performs better for tail participants.The comparison is based on cumulative accuracy improvements across adaptations.

8. Conclusion

Federated learning is promising for large-scale training on sensitive data, but privacy and robustness protections reduce the accuracy of federated models.

  • Federated learning is a promising approach to large-scale model training on sensitive data, but differential privacy and robust aggregation reduce federated-model accuracy.

A.1. Adapting, then aggregating again

Aggregating locally adapted models can improve the federated model, but the benefit depends on adaptation settings and increases each participant’s training cost.

  • A.1. Adapting, then aggregating again: 91.15%, 92.64%, and 89.22% test accuracy result from aggregating models fine-tuned for 5, 50, and 100 epochs, respectively.The conventional federated baseline reaches 90.44% test accuracy; fine-tuning uses a learning rate of 0.001.
  • A.1. Adapting, then aggregating again: Aggregating adapted models can produce a more accurate federated model with an appropriate learning rate and number of epochs.
  • A.1. Adapting, then aggregating again: The potential accuracy improvement comes at the cost of significantly increasing training time for each participant.The authors leave these tradeoffs for future work.

A.2. Removing disincentivized participants

Removing participants whose local models outperform the federated model barely changes overall word-prediction accuracy, while those participants’ data are simpler and outlying.

  • A.2. Removing disincentivized participants: 7,377 participants have more accurate local models than the federated model and therefore no incentive to participate.They are removed from the 80,000-participant word-prediction population.
  • A.2. Removing disincentivized participants: 20.008% mean and 19.570% median accuracy after retraining on 72,623 participants compares with 20.021% mean and 19.563% median for the original model.The comparison indicates similar overall accuracy despite removing the disincentivized participants.
  • A.2. Removing disincentivized participants: The removed participants have simpler, fewer words and sentences that are outliers relative to the other participants.The authors conjecture that the remaining data are more regular but still sufficiently complex for training.
  • A.2. Removing disincentivized participants: Figure 7 measures participant-level data imbalance using total words and vocabulary size for word prediction, and total images for image classification.
Loading 2002.04758v3…