Source-linked AI summary
Survey of Personalization Techniques for Federated Learning
Viraj Kulkarni, Milind Kulkarni, Aniruddha Pant
TL;DR
Federated learning must reconcile decentralized private data with clients whose non-IID local data can make individually trained models more accurate than one shared model. This paper surveys techniques for personalizing global federated models, finding that personalized models usually perform better for individual clients, while privacy and robust aggregation can prevent them from matching local models.
Problem
Non-IID client data can make a shared global model less accurate than local models, weakening the benefit of federated-learning participation for some clients.
Method
The paper surveys techniques that adapt collaboratively trained global models to individual clients, including transfer learning, multi-task learning, and meta-learning.
Results
Personalized models usually show better performance for individual clients than global or local models, but may not match local models when differential privacy and robust aggregation are implemented.
Takeaways & Limitations
Individual-client performance should be measured when federated models are intended for subsequent personalization rather than evaluated only on aggregated data.
Takeaways & Limitations
The conditions under which shared global models outperform individual local models remain an open theoretical question.
Abstract
from arXiv · showhide
Federated learning enables machine learning models to learn from private decentralized data without compromising privacy. The standard formulation of federated learning produces one shared model for all clients. Statistical heterogeneity due to non-IID distribution of data across devices often leads to scenarios where, for some clients, the local models trained solely on their private data perform better than the global shared model thus taking away their incentive to participate in the process. Several techniques have been proposed to personalize global models to work better for individual clients. This paper highlights the need for personalization and surveys recent research on this topic.
1 Introduction
Federated learning keeps decentralized data on users’ devices while coordinating shared-model training through server-mediated updates. It addresses privacy and communication concerns, but non-IID data can make participation less beneficial for clients with strong local models.
- Centralizing user data compromises privacy and security and can impose prohibitively expensive communication overheads.Regulatory policies including GDPR and HIPAA make moving data to a central repository difficult.
- Federated learning lets clients collaboratively train a shared global model without moving their data from local devices.A server distributes the current model, collects local updates, and updates the shared model over multiple rounds.
- Federated learning has shown good performance and robustness for next-word prediction on mobile devices and supports large-scale mobile deployments.
- For many tasks, some clients gain no benefit because the global shared model is less accurate than locally trained models.The concern is linked to highly non-IID data distributions across clients.
2 Need for Personalization
Personalization is needed because clients differ in devices, data, models, and labels, while privacy mechanisms can reduce individual-user performance. These challenges motivate adapting shared models to client-specific conditions rather than relying on one global model.
- Personalization addresses device, data, model, and label heterogeneity across federated-learning clients.Clients may require models customized to their environments or assign different labels to the same data.
- Privacy protection introduces a conflict between protecting participant data and achieving higher performance for individual users.Differential privacy reduces accuracy, with underrepresented or tail participants affected worst.
- Most personalization techniques first build a global model collaboratively and then adapt it for each client using private data.Optimizing solely for global accuracy can produce models that are harder to personalize.
3 Techniques
The surveyed techniques personalize federated models through contextual features, transfer learning, multi-task learning, meta-learning, distillation, and architectures that separate shared and local components. They differ in how global knowledge is adapted and how personalization is integrated into training.
- Contextual Personalization: Context featurization can enable personalized predictions from a shared global model, but effectively incorporating context remains an open problem.Most public datasets lack contextual features.
- Transfer Learning: Transfer learning initializes local personalization from a trained global model instead of learning from scratch.Retraining too long can cause catastrophic forgetting, while freezing base layers and adapting top layers is a variant.
- Multi-task Learning: Multi-task learning jointly solves related tasks so federated models can learn shared commonalities and task-specific differences.MOCHA addresses communication, straggler, and fault-tolerance challenges, but requires all clients to participate every round.
- Meta-Learning: Meta-learning treats federated training as meta-training and personalization as meta-testing, motivating modifications that optimize both stages simultaneously.Jiang et al. report that careful fine-tuning can produce accurate global models that are easily personalized, whereas naive global optimization can hurt personalization.
- Meta-Learning: Per-FedAvg incorporates MAML to seek global models that perform well after each user updates them for their own loss function.ARUBA is another federated meta-learning algorithm inspired by online convex optimization.
- Knowledge Distillation: Knowledge distillation treats the global model as a teacher and the personalized model as a student to address overfitting during personalization.The passage emphasizes this challenge for clients with small local datasets.
- Personalized Architectures: FedPer trains base layers centrally with Federated Averaging and personalization layers locally.This architecture is designed to temper adverse effects of statistical heterogeneity.
- Global–Local Formulations: The standard formulation learns one global model, whereas an alternative formulation learns a mixture of the global model and each device’s local model.The alternative formulation is solved with a variant of gradient descent called Loopless Local.
4 Discussion
Personalized models usually outperform global or local models for individual clients, but their advantage is not universal. The relative value of shared and local models depends on dataset size and distribution, leaving theoretical conditions for global-model superiority unresolved.
- When local datasets are small and IID, global models typically outperform local models and most clients benefit from participation.With sufficiently large private datasets and non-IID distributions, local models can outperform the shared model.
- Determining when shared global models outperform individual local models remains an open theoretical question.
- Evaluating only global performance is inadequate when a global model will subsequently be personalized for individual clients.The survey notes that most prior work measured performance on aggregated data rather than as seen by individual clients.
- Personalized models usually perform better for individual clients than global or local models.In some cases they fail to match local models, especially with differential privacy and robust aggregation.