Source-linked AI summary
Recent Advances on Federated Learning: A Systematic Survey
Bingyan Liu, Nuoyan Lv, Yuanchun Guo, Yawen Li
TL;DR
Federated learning enables collaborative model training without centrally collecting sensitive party data, but existing surveys often cover only specific aspects or miss recent progress. This paper systematically surveys recent methods and applications through a taxonomy based on the FL pipeline and challenges, alongside frameworks, deficiencies, and future directions. Its supported outcome is a structured overview of federated learning methods, practical frameworks, and open research directions.
Problem
Existing FL surveys often focus on specific aspects or fail to capture the latest progress, leaving a need for a comprehensive understanding of federated learning.
Method
The paper conducts a systematic survey, proposes a pipeline- and challenge-based taxonomy, summarizes methods, and reviews prevalent FL frameworks.
Results
The survey organizes federated learning methods by aggregation optimization, heterogeneity, privacy protection, and fairness, and characterizes practical frameworks.
Takeaways & Limitations
The paper provides a structured overview of recent FL development while identifying potential deficiencies and future directions for the field.
Takeaways & Limitations
Conventional FL approaches assume stable client data, whereas continuously changing local data can cause catastrophic forgetting; combining FL with incremental learning remains open.
Abstract
from arXiv · showhide
Federated learning has emerged as an effective paradigm to achieve privacy-preserving collaborative learning among different parties. Compared to traditional centralized learning that requires collecting data from each party, in federated learning, only the locally trained models or computed gradients are exchanged, without exposing any data information. As a result, it is able to protect privacy to some extent. In recent years, federated learning has become more and more prevalent and there have been many surveys for summarizing related methods in this hot research topic. However, most of them focus on a specific perspective or lack the latest research progress. In this paper, we provide a systematic survey on federated learning, aiming to review the recent advanced federated methods and applications from different aspects. Specifically, this paper includes four major contributions. First, we present a new taxonomy of federated learning in terms of the pipeline and challenges in federated scenarios. Second, we summarize federated learning methods into several categories and briefly introduce the state-of-the-art methods under these categories. Third, we overview some prevalent federated learning frameworks and introduce their features. Finally, some potential deficiencies of current methods and several future directions are discussed.
1 INTRODUCTION
Federated learning addresses the challenge of combining knowledge across parties without centrally collecting sensitive data. This survey reviews recent methods, applications, frameworks, limitations, and future directions through a systematic perspective.
- Motivation and FL pipeline: Centralized data collection is problematic because party data may be privacy-sensitive, including medical images subject to privacy restrictions.GDPR is also cited as emphasizing privacy protection when organizations share information.
- Motivation and FL pipeline: Federated learning lets multiple parties collaboratively train a DNN by exchanging local models rather than uploading local data.A central server aggregates local updates and distributes a global model for further local training.
- Survey scope: Existing FL surveys often focus on a specific aspect, leaving a need for more comprehensive coverage of the field.The paper positions its survey as addressing this limitation while incorporating recent progress.
- Survey scope: The survey introduces a pipeline- and challenge-based taxonomy, summarizes methods, reviews frameworks, and discusses deficiencies and future directions.Its taxonomy covers aggregation optimization, heterogeneity, privacy protection, and fairness.
2 PRELIMINARIES
Federated learning formalizes collaborative optimization in which clients train local models and a server aggregates them instead of receiving private data. Its key challenges include heterogeneity, privacy risks, and fairness.
- FL formulation: Clients train local models on private datasets, upload models rather than data, and receive an aggregated global model for subsequent training.Differential privacy and homomorphic encryption may further protect uploaded models.
- FL formulation: The FL objective minimizes a global loss formed from client-local losses weighted by each client’s importance.The aggregation algorithm determines the values allocated to the client weights.
- Key challenges: Federated learning faces data, model, and system heterogeneity because parties differ in data distributions, models, hardware, and memory limitations.Non-independent and identically distributed local data can affect learning across clients.
- Key challenges: Uploaded model parameters can still expose private information, motivating stronger encryption or obfuscation methods.The original privacy design transfers models instead of local data, but attackers may exploit those parameters.
- Key challenges: Fair federated models should consider fairness performance in addition to average accuracy because participants have different data resources.The survey relates individual fairness to substantially different outcomes for people with similar preferences and characteristics.
3 APPROACHES OF FEDERATED LEARNING
The survey organizes its discussion by first presenting a taxonomy of federated learning approaches and then explaining how methods in each category achieve their goals.
- 3 APPROACHES OF FEDERATED LEARNING: Federated learning approaches are allocated into taxonomy categories, with each category receiving a detailed account of how its methods achieve their goals.
3.1 Taxonomy
The proposed taxonomy classifies federated learning methods according to the FL pipeline and its major challenges. It covers aggregation, heterogeneity, security, and fairness-oriented approaches.
- Taxonomy: The taxonomy classifies existing federated learning methods according to the federated learning pipeline and its challenges.The survey presents the taxonomy in Figure 3.
- Secure federated learning: Secure federated learning concerns methods for protecting information exchanged during federated learning.
- Aggregation optimization: Aggregation optimization studies methods for combining local models into an improved global model, including FedAvg, FedMA, and FedProx.The survey motivates this category by the typically large number of participants.
- Heterogeneous federated learning: Heterogeneous federated learning addresses differences in data, models, and systems, including variation in hardware and client capabilities.The survey discusses techniques such as meta-learning and multi-task learning for related heterogeneity issues.
- Fair federated learning: Fair federated learning includes minimax optimization and sample reweighting approaches for addressing fairness.
3.2 Aggregation optimization
Aggregation optimization improves federated learning by combining local models into a better global model. The survey reviews weight-level, feature-level, and robust aggregation approaches, including their mechanisms and limitations.
- Aggregation optimization targets improving the performance of the final global model, the core output of federated learning.
- Weight-level aggregation: FedAvg aggregates local models through coordinate-based weight averaging and has shown strong performance on public datasets such as MNIST and CIFAR-10.
- Weight-level aggregation: Weight divergence from skewed client data can make corresponding coordinates mismatch, degrading the accuracy of directly averaged global models.
- Weight-level aggregation: PFNM matches subsets of neurons across local models using Bayesian nonparametric machinery, but performs well mainly on simple architectures.
- Weight-level aggregation: FedMA aligns advanced CNNs and LSTMs layer-wise by searching for permutation matrices with the Hungarian matching algorithm.
- Feature-level aggregation: Fed2 uses group convolution for structure adaptation and feature-paired averaging to align neuron features while avoiding expensive distance-based optimization.
3.3 Heterogeneous federated learning
Heterogeneous federated learning addresses differences in client data, models, and systems through methods including multi-task, transfer, clustering, and meta-learning. System heterogeneity specifically motivates resource-aware participation and training strategies.
- Heterogeneity in federated learning arises from differences in data distributions, models, or device systems.
- Multi-task learning based methods: Multi-task learning treats clients with different data distributions as related tasks and uses task relationships to facilitate learning.MOCHA formulates federated multi-task learning as a dual optimization problem.
- Multi-task learning based methods: Federated multi-task learning has been extended from convex models to non-convex models using hierarchical Bayesian inference and variational methods.VIRTUAL constructs a hierarchical Bayesian network linking the server and clients.
- Multi-task learning based methods: Federated multi-task learning can assume each local distribution is a mixture of unknown underlying distributions and use Federated Expectation-Maximization to generalize to unseen clients.
- Meta-learning based methods: Meta-learning methods seek shared models that can rapidly adapt to local data, including Per-FedAvg, which incorporates MAML into federated pipelines.Per-FedAvg targets fitting local data with one or a few gradient-descent steps.
- Transfer learning based methods: Personalized and transfer-learning methods adapt federated models to heterogeneous client data through fine-tuning, knowledge distillation, adversarial adaptation, or alternating proximal updates.FADA addresses domain shift between labeled source data and unlabeled target data.
- Clustering-based methods: Clustering-based federated learning partitions clients with similar distributions, while IFCA alternates cluster assignment and cluster-model optimization until memberships stabilize.
- System heterogeneity: System heterogeneity reflects clients’ differing hardware and memory limitations, requiring federated learning methods that accommodate unequal capabilities.
3.4 Secure federated learning
Secure federated learning responds to attacks that can extract private information or manipulate models despite raw data remaining local. The survey reviews attack types and defenses based on differential privacy, homomorphic encryption, and trusted execution environments.
- Keeping raw data local does not prevent attackers from extracting privacy information from uploaded models or gradients.
- Attack methods: Backdoor attacks inject triggers so federated models produce incorrect predictions when those triggers appear.Aggregation can destroy triggers, motivating federated-specific attack designs such as constrain-and-scale and distributed backdoors.
- Attack methods: Gradient attacks reconstruct private training data by optimizing dummy inputs and labels to match publicly shared gradients.Aggregated updates can also reveal participant information.
- Defense methods: Differential-privacy defenses add noise to obfuscate model information and make sensitive properties harder to infer.
- Defense methods: Homomorphic-encryption defenses allow client models to be aggregated without exposing concrete model parameters to the server.BatchCrypt is presented as an efficient homomorphic-encryption system for cross-silo federated learning.
- Defense methods: Trusted execution environments protect federated learning through isolated hardware regions that preserve data and program confidentiality and integrity.TEE-based approaches are described as more computationally efficient than traditional encryption schemes.
- Defense methods: More than 10× throughput promotion is reported for TEESlice at the same level of strong security guarantee.
3.5 Fair federated learning
Fair federated learning addresses unequal treatment of clients across selection, optimization, incentives, and contribution evaluation. The surveyed approaches adjust participation, customize client workloads, optimize fairness objectives, and assess contributions more equitably.
- Fairness problems can arise throughout federated learning, including client selection, model optimization, incentive distribution, and contribution evaluation.
- Client selection: Client-selection unfairness includes over-representation, under-representation, and never-representation of clients with different performance levels.
- Client selection: Fairness factors increase participation opportunities for rarely selected clients, including those with lower computational abilities or smaller datasets.Methods use bandits, dynamic queues, or Exp3-based selection mechanisms.
- Client customization: Client customization assigns adaptable model settings or procedures so lower-capability clients can continue participating in training.Federated Dropout distributes sub-models sized for each client’s computational resources.
- Client customization: Poor networks can cause retransmissions, delays, and biased aggregation; TRA accelerates training by ignoring a few lost packets.
- Model optimization: Objective-based fairness methods modify global or local objectives to avoid overfitting specific clients and can jointly target fairness and robustness.AFL optimizes for a target distribution formed from a mixture of clients, but works only for a small number of clients.
- Model optimization: Group-fairness methods can achieve accuracy and fairness on unknown test distributions but may require prior knowledge to design re-weighting functions.
- Fair contribution evaluation: Contribution evaluation is important because unfair assessments can enable free-riding; Shapley-value methods account for data contribution independently of client joining order.Shapley-value computation has complexity O(2^n), motivating efficiency improvements.
4 PREVALENT FRAMEWORKS OF FEDERATED LEARNING
The survey reviews federated-learning frameworks spanning customizable research components, scalable edge-device experimentation, diverse algorithmic configurations, production deployment, and realistic evaluation. These frameworks target different practical and experimental needs.
- FedLab: FedLab provides flexible, customizable server and client modules with interfaces for implementing varied federated-learning scenarios.
- Flower: Flower supports large-scale federated-learning experiments across mobile and edge-device scenarios and helps connect experimental research with system research.
- FedML: FedML addresses diverse computing paradigms, configurations, standardized algorithm implementations, and benchmarks through high- and low-level APIs.Its Mobile and IoT testbeds run on real-world hardware platforms.
- FATE: FATE is a production-oriented federated-AI platform that provides private set intersection and distributed computation through Eggroll.
- FedScale: FedScale combines realistic federated-learning datasets with an automated runtime for simplifying and standardizing evaluation in realistic environments.Its datasets are collected from varied sources, consistently processed, and exposed through standardized APIs.
5 DISCUSSION
The discussion identifies unresolved challenges for federated learning, including changing client data, server dependence, scalability, and the lack of unified benchmarks.
- Dynamic federated learning: Current FL approaches assume stable client data, but continuously changing sensor data can cause catastrophic forgetting.Combining federated learning with incremental learning is identified as a future research direction.
- Decentralized federated learning: Traditional FL depends on a central server, creating security risks when the third-party server is not honest.General decentralized federated learning remains an open problem because existing work targets logistic regression and has insufficient experiments.
- Scalability of federated learning: FL scalability remains underexplored despite the possibility of cooperation among a huge number of parties.Future work should investigate how federated learning operates and improves as participant numbers increase.
- Unified benchmark: A unified benchmark is lacking because federated goals and FL types use different datasets, limiting fair comparison across results.The passage contrasts datasets used for goals such as personalization and robustness and for horizontal versus vertical FL.
6 CONCLUSION
The paper surveys recent federated learning development by organizing methods around pipeline and challenge-based aspects, examining practical frameworks, and identifying future directions.
- 6 CONCLUSION: The survey proposes a taxonomy covering different federated learning aspects by analyzing the FL pipeline and its challenges.It also investigates practical FL frameworks and characterizes their features.
- 6 CONCLUSION: The paper concludes limitations and future directions intended to promote the evolution of the federated learning community.