Source-linked AI summary
Asynchronous Federated Learning on Heterogeneous Devices: A Survey
Chenhao Xu, Youyang Qu, Yong Xiang, Longxiang Gao
TL;DR
Heterogeneous devices and uneven data distributions create efficiency, convergence, and model-bias challenges for federated learning, while privacy and security remain concerns. This survey organizes asynchronous federated learning research across these challenges and applications, identifies existing findings, and outlines unresolved research directions.
Problem
Classical federated learning on heterogeneous devices can suffer from device unreliability and aggregation delays, while non-IID data can affect asynchronous federated learning convergence and model quality.
Method
The survey conducts an in-depth exploration of state-of-the-art asynchronous federated learning research and summarizes application scenarios on heterogeneous devices.
Results
The survey identifies device heterogeneity, data heterogeneity, and security and privacy issues as three fundamental challenges in asynchronous federated learning, and categorizes corresponding research and applications.
Takeaways & Limitations
The survey provides insights into asynchronous federated learning and indicates that emerging schemes can be appended and categorized as the field develops.
Takeaways & Limitations
No comprehensive analysis yet balances multiple performance-improvement strategies in asynchronous federated learning, and combining strategies can reduce efficiency.
Abstract
from arXiv · showhide
Federated learning (FL) is a kind of distributed machine learning framework, where the global model is generated on the centralized aggregation server based on the parameters of local models, addressing concerns about privacy leakage caused by the collection of local training data. With the growing computational and communication capacities of edge and IoT devices, applying FL on heterogeneous devices to train machine learning models is becoming a prevailing trend. Nonetheless, the synchronous aggregation strategy in the classic FL paradigm, particularly on heterogeneous devices, encounters limitations in resource utilization due to the need to wait for slow devices before aggregation in each training round. Furthermore, the uneven distribution of data across devices (i.e. data heterogeneity) in real-world scenarios adversely impacts the accuracy of the global model. Consequently, many asynchronous FL (AFL) approaches have been introduced across various application contexts to enhance efficiency, performance, privacy, and security. This survey comprehensively analyzes and summarizes existing AFL variations using a novel classification scheme, including device heterogeneity, data heterogeneity, privacy, and security on heterogeneous devices, as well as applications on heterogeneous devices. Finally, this survey reveals rising challenges and presents potentially promising research directions in this under-investigated domain.
1. Introduction
Federated learning enables collaborative model training without direct access to local data, but synchronous aggregation struggles with unreliable, heterogeneous devices and uneven data. This survey addresses the gap through comprehensive classification and analysis of asynchronous federated learning (AFL).
- Federated learning trains a global model by aggregating local-model parameters while keeping participants’ training data inaccessible to the central server.
- Synchronous FL on heterogeneous devices waits for slow or offline participants, reducing aggregation efficiency and resource utilization.Device and data heterogeneity create stragglers, while node-selection inefficiencies leave capable devices underused.
- AFL aggregates immediately when a local model arrives, mitigating device unreliability, reducing waits for slow devices, and allowing devices to train at their own pace.
- Existing FL surveys had not exhaustively investigated, classified, or summarized AFL before this study.
- The survey reviews 125 papers from 2019–2022 and categorizes AFL by device heterogeneity, data heterogeneity, privacy and security, and application scenarios.It also identifies promising research topics for further investigation.
2. Background Knowledge
This section introduces federated learning, blockchain, and differential privacy, then explains conventional and asynchronous FL workflows for heterogeneous devices. It also presents non-IID data and privacy–utility considerations relevant to AFL.
- Federated Learning: Federated learning is a distributed machine-learning framework in which local models are trained on separate nodes and averaged into a global model.Unlike conventional distributed ML, FL is designed for decentralized data silos where privacy limits data sharing.
- Federated Learning: In conventional FL, the server broadcasts an initial model, collects local models, and computes a dataset-size-weighted average before the next iteration.The local model on node k is weighted according to its training-sample count n_k relative to total samples n.
- Data Heterogeneity: Non-IID data means nodes collect differently distributed samples, limiting the utility of gradients learned at one hospital for prediction at another.
- Federated Learning: AFL aggregates each newly received local model immediately, reducing aggregation waiting time and improving overall efficiency.Heterogeneous devices finish local training at different times and transmit models separately.
- Blockchain: Blockchain can store training models, motivate participation, and use immutable ledgers, consensus, and smart contracts to support model and node trustworthiness.
- Differential Privacy: Differential privacy adds controllable randomized noise to protect sensitive data, but sanitization reduces data utility and can diminish model accuracy.Personalized privacy parameters may improve utility, while zero-mean Laplace noise is discussed for mitigating concerns in FL.
3. Device Heterogeneity
Device heterogeneity in AFL motivates methods that improve resource utilization while limiting stale-model effects. Research spans node selection, weighted aggregation, compression, scheduling, and semi-asynchronous designs.
- AFL research targets resource utilization and stale local models, which can reduce training efficiency and global-model performance on heterogeneous devices.
- Node Selection: Node-selection methods prioritize device resilience, computational capacity, or model-accuracy changes, but must balance robustness against global-model overfitting.
- Node Selection: Random scheduling performed best on non-IID data, while an appropriate age-aware aggregation policy also improved results in comparative simulations.
- Weighted Aggregation: Weighted aggregation reduces stale-model influence by emphasizing recent updates, while other methods additionally use dataset size, gradient similarity, or branch-level accuracy.
- Gradient Compression: Gradient compression reduces communication costs but must accommodate resource-constrained devices and frequent AFL aggregation.
- Communication and Aggregation: Communication protocols and semi-asynchronous schemes further address slow nodes by scheduling transmissions or aggregating buffered models after a defined timeframe.
4. Data Heterogeneity
Data heterogeneity, especially non-IID and imbalanced data, can bias AFL models and cause divergence or overfitting. Surveyed approaches constrain updates, cluster nodes, validate models distributively, or optimize training parameters.
- Non-IID and imbalanced data can make frequent uploads from particular nodes drive global-model divergence and overfitting to specific datasets.
- AFL research addresses non-IID data through aggregation constraints, clustered FL, distributed validation, and mathematical parameter optimization.
- Constraint Terms for Aggregation: Constraint-based methods limit local updates toward the global model, while tiering nodes by update frequency helps prevent local-model divergence.
- Clustered FL: Clustering methods group nodes using loss-surface geometry or optimized data distributions; experiments report improved test accuracy and convergence speed.
- Distributed Validation: Distributed validation reserves 5% of each node's local training samples to evaluate models from other nodes, producing a better-generalized global model.
- Mathematical Optimization: Mathematical approaches use Taylor expansion for higher-precision gradients or optimize hyperparameters in two-stage AFL training.
- Empirical and Theoretical Findings: Experiments report that AFL can match traditional FL's convergence rate with lower communication requirements and outperform classic FL under disparate resources and non-IID data.
- Vertical FL: On vertical FL, feature skew and model interdependence create additional challenges addressed through compact embeddings, variance reduction, gradient prediction, and sparse compression.
5. Privacy and Security on Heterogeneous Devices
Privacy and security remain concerns in AFL because attacks threaten local-data confidentiality and global-model integrity, while many defenses are costly for heterogeneous devices. Surveyed solutions use differential privacy, blockchain, compression, and reputation mechanisms.
- AFL faces membership, property, inversion, gradient-leakage, poisoning, and backdoor attacks, while existing defenses can be computationally demanding for constrained devices.
- Privacy: Differential-privacy schemes protect local models, with flexible or local settings intended to balance privacy and data utility across heterogeneous devices.
- Security: Blockchain supports secure storage and transmission of local models while promoting trust and reducing dependence on a single aggregation server.
- Blockchain-Based AFL: Staleness coefficients reduce latency-device contributions by comparing global-model and stale-model versions during aggregation.
- Reputation: Blockchain-based reputation mechanisms rank participants using training time, sample size, update correlation, and cheating-related metrics.
- Attack Resistance: Experiments report efficiency and poisoning-attack prevention for a reputation-based scheme, while SignSGD-based compression theoretically resists privacy and security attacks at a convergence or accuracy cost.
6. Applications on Heterogeneous Devices
AFL is applied across heterogeneous-device settings where asynchronous training supports efficiency, privacy, and real-time operation. Surveyed applications include smart transportation, fault diagnosis, IIoT, concept drift, and geolocation services.
- AFL application studies emphasize efficient and adaptable training on heterogeneous devices while preserving local-data privacy.
- Smart Transportation: In smart transportation, AFL supports vehicular data sharing, resource caching, autonomous-driving prediction, and time-sensitive training requirements.
- Smart Transportation: Vehicular experiments report that DAG blockchain can provide both performance and security, while reinforcement-learning-based device selection achieved higher accuracy and lower time cost.
- Fault Diagnosis: Fault-diagnosis systems asynchronously aggregate local models produced from private data using Extended and Sequential Kalman Filters.
- Industrial IoT: In IIoT, AFL mitigates slow-device effects through adaptive aggregation frequency and supports privacy-preserving real-time analysis and decision-making.
- Other Applications: A concept-drift scheme improves predictive performance for the worst 20% of devices while maintaining the best test performance for the top 20%.
7. Research Challenges and Future Directions
The survey identifies open AFL challenges spanning optimization, heterogeneous data, privacy, security, applications, and real-world evaluation. It proposes research directions that balance efficiency, accuracy, resource use, privacy, and security across heterogeneous devices.
- Device Heterogeneity: AFL needs optimization frameworks that balance performance gains from node selection, weighted aggregation, and cluster FL against their additional time costs.The survey suggests dynamic gaming models based on Markov Decision Processes and lightweight convex optimization methods.
- Device Heterogeneity: Generalized AFL solutions should adapt performance strategies to computing disparities, data distributions, and the effects of local models on the global model.The survey gives semi-asynchronous aggregation, selection and compression, and flexible optimization frameworks as possible directions.
- Data Heterogeneity: Non-IID data, unequal dataset sizes, and vertical feature distributions can bias AFL models or make some local information underrepresented.Proposed directions include similarity-based clustering, verifiable dataset-size weighting, server-side caching, and model splitting.
- Privacy and Security on Heterogeneous Devices: Differential privacy and blockchain-based security introduce unresolved trade-offs between privacy or attack resistance and model utility, efficiency, or resource consumption.The survey also identifies lightweight distributed cryptography and more efficient, secure consensus algorithms as research directions.
- Applications on Heterogeneous Devices: AFL research should expand beyond limited current applications and simulation-heavy evaluations toward dedicated systems and scalable testbeds on real heterogeneous devices.Suggested application areas include smart grids and smart farms, while testbeds should evaluate efficiency, security, and privacy.
8. Conclusion
The survey synthesizes AFL research around device heterogeneity, data heterogeneity, and security and privacy issues, while summarizing application scenarios on heterogeneous devices. It provides a perspective for organizing future AFL designs and identifying promising research directions.
- The survey examines AFL challenges involving device heterogeneity, data heterogeneity, and security and privacy on heterogeneous devices.
- It summarizes AFL application scenarios that could increase the technology’s impact and adoption on heterogeneous devices.
- The proposed categorization offers insights for future designs and may motivate coherent compositions of AFL techniques.