Source-linked AI summary
Federated Learning: Challenges, Methods, and Future Directions
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, Virginia Smith
TL;DR
Federated learning must address scale, systems heterogeneity, and statistically non-IID data beyond the capabilities of many classical methods. This article surveys the paradigm, its challenges and existing approaches, and identifies open research problems requiring interdisciplinary effort.
Problem
Federated learning lacks approaches that fully address massive-network scale alongside systems and statistical heterogeneity.
Method
The article surveys federated learning’s properties, challenges, classical and federated approaches, and open directions for future research.
Results
The article provides an overview of federated learning, an extensive survey of relevant results, and a set of open problems for future research.
Takeaways & Limitations
Addressing federated learning’s open problems will require interdisciplinary effort from a broad set of research communities.
Takeaways & Limitations
One-shot and few-shot communication heuristics remain theoretically unanalyzed and unevaluated at scale in massive or statistically heterogeneous networks.
Abstract
from arXiv · showhide
Federated learning involves training statistical models over remote devices or siloed data centers, such as mobile phones or hospitals, while keeping data localized. Training in heterogeneous and potentially massive networks introduces novel challenges that require a fundamental departure from standard approaches for large-scale machine learning, distributed optimization, and privacy-preserving data analysis. In this article, we discuss the unique characteristics and challenges of federated learning, provide a broad overview of current approaches, and outline several directions of future work that are relevant to a wide range of research communities.
1 Introduction
Federated learning trains statistical models across remote devices or organizations while keeping data localized and communicating only intermediate updates. Its distinctive setting combines privacy-sensitive edge data with communication, systems, and statistical heterogeneity, requiring advances across machine learning, distributed optimization, privacy, and systems.
- Applications: Federated learning supports privacy-sensitive applications across smartphones, hospitals, and Internet-of-Things systems, including personalization, predictive healthcare, and autonomous-vehicle adaptation.These settings keep sensitive data local while reducing network strain or enabling models to adapt to changing systems.
- Federated learning setup: The canonical problem learns one global model from tens to potentially millions of devices, processing data locally and periodically communicating intermediate updates to a central server.The objective aggregates device-specific local objectives into a global model.
- Core challenges: Federated networks face expensive communication because communication may be many orders of magnitude slower than local computation, while privacy concerns require raw data to remain local.Networks may include millions of smartphones, making communication a critical bottleneck.
- Core challenges: Systems heterogeneity arises from differences in hardware, connectivity, and power, with only a small fraction of devices active at once, such as hundreds in a million-device network.Methods must anticipate low participation, tolerate heterogeneous hardware, and remain robust to dropped devices.
- Core challenges: Statistical heterogeneity occurs because devices collect non-identically distributed data, vary substantially in data volume, and may exhibit structure linking devices with their associated distributions.Mobile phone users, for example, may use language differently in next-word prediction tasks.
2 Survey of Related and Current Work
Federated learning draws on methods from privacy, large-scale machine learning, distributed optimization, and signal processing, but its scale and heterogeneity require approaches beyond those methods. Local updating schemes address communication costs by shifting additional computation to devices before global aggregation.
- Federated learning challenges resemble classical problems in privacy, large-scale machine learning, and distributed optimization.
- Existing methods for expensive communication typically cannot fully handle federated networks’ scale or systems and statistical heterogeneity.
- Local-updating schemes reduce communication by applying device updates immediately and delaying server aggregation until after a variable number of local updates.Devices perform additional work locally before global aggregation.
2.1 Communication-efficiency
Communication is a key bottleneck in federated networks, motivating local updating, message compression, and decentralized training. The article focuses on star-network federation while considering decentralized topologies as an alternative when server communication is limiting.
- Communication-efficiency: Communication efficiency in federated networks centers on local updating methods, compression schemes, and decentralized training.These are the article’s three broad directions for addressing communication bottlenecks.
- Communication-efficiency: Flexible local updating and low client participation have made federated optimization methods the de facto solvers.FedAvg, the most commonly used method, averages clients’ local stochastic gradient descent updates and works well empirically, particularly for non-convex problems, but lacks convergence guarantees.
- Communication-efficiency: Model compression schemes—including sparsification, subsampling, and quantization—reduce the size of messages communicated at each round.Local updating reduces the total number of communication rounds, whereas compression reduces per-round message size.
- Communication-efficiency: The star network, with a central server connected to all devices, is the predominant federated communication topology.The article focuses on this setting while briefly discussing decentralized topologies, where devices communicate only with neighbors, as a potential alternative.
2.2 Systems Heterogeneity
Systems heterogeneity in federated networks arises from variation in hardware, connectivity, and battery power, making stragglers more prevalent than in typical data centers. Key responses include asynchronous communication, active device sampling, and fault tolerance.
- Systems Heterogeneity: Devices vary in hardware, network connectivity, and battery power, making stragglers significantly more prevalent than in typical data center environments.The paper groups responses to systems heterogeneity into asynchronous communication, active device sampling, and fault tolerance.
- Asynchronous Communication: Synchronous schemes provide simplicity and serial-equivalent computation but are more susceptible to stragglers under device variability, whereas asynchronous schemes can mitigate stragglers.Both schemes are commonly used for parallelizing iterative optimization algorithms, with different advantages and disadvantages.
- Active Device Sampling: Because only a small subset of devices typically participates per round, active sampling can select participants based on systems resources rather than leaving participation entirely passive.Existing work also considers systems overheads when designing incentives for devices with higher-quality data to participate.
- Active Device Sampling: Existing sampling and incentive approaches assume static network characteristics, leaving real-time, device-specific fluctuations in computation and communication delays unresolved.These methods primarily focus on systems variability when performing active sampling.
- Fault Tolerance: Fault tolerance is especially critical over remote devices because participating devices commonly drop out before completing a training iteration.Coded computation introduces algorithmic redundancy; gradient coding replicates data blocks and their gradient computations to enable exact or inexact recovery despite stragglers.
2.3 Statistical Heterogeneity
Statistical heterogeneity in federated data creates challenges for both modeling and convergence analysis. Approaches address these challenges through personalized or related models, fairness-aware objectives, and convergence-oriented modifications to federated optimization.
- 2.3 Statistical Heterogeneity: Non-identically distributed device data complicates both federated model design and analysis of associated training-procedure convergence.These challenges arise from statistical heterogeneity across devices.
- 2.3 Statistical Heterogeneity: Federated modeling can learn separate, global, or related-but-distinct models depending on data, network, and application properties.MOCHA exemplifies personalization by learning separate but related device models through a shared representation.
- 2.3 Statistical Heterogeneity: Aggregate-loss objectives may disadvantage devices or groups by biasing models toward data-rich devices or commonly occurring groups.Fairness-aware modeling approaches aim to reduce this variance.
- 2.3 Statistical Heterogeneity: FedAvg can diverge in practice under non-identically distributed device data, because standard convergence analyses assume identical local stochastic processes.FedProx modifies FedAvg to help ensure convergence in statistically heterogeneous settings.
2.4 Privacy
Federated learning can leak sensitive information through shared model updates despite keeping raw data local. Privacy-preserving approaches include differential privacy, homomorphic encryption, and secure function evaluation or multiparty computation, but federated methods must also address efficiency, dropped devices, and accuracy.
- Privacy risks: Keeping raw data local does not prevent model updates from leaking sensitive information, including specific credit card numbers extracted from trained language models.The paper identifies model-update sharing as a privacy risk and gives sensitive text-pattern extraction as an example.
- Privacy mechanisms: Three main privacy strategies are differential privacy, homomorphic encryption, and secure function evaluation or multiparty computation.These strategies respectively communicate noisy data sketches, compute on encrypted data, or use privacy-preserving protocols.
- Differential privacy: Differential privacy is most widely used because it offers strong information theoretic guarantees, algorithmic simplicity, and relatively small systems overhead.It limits how much changing one input element can alter the output distribution, preventing conclusions about whether a specific sample was used.
- Cryptographic approaches: Homomorphic encryption has limited applications, while SMC can hide individual updates yet reveal exact aggregated results to the central server.Homomorphic encryption has been applied to training linear models or settings with only a few entities; SMC is described as lossless.
- Federated challenges: Federated privacy methods must be computationally cheap, communication-efficient, tolerant to dropped devices, and accurate while providing rigorous guarantees.Privacy definitions in federated learning are typically classified as global privacy or local privacy.
- Privacy trade-offs: Global differential privacy requires careful hyperparameter selection affecting communication and accuracy, whereas relaxed local privacy provides stronger guarantees than global privacy.Adaptive gradient clipping is proposed to alleviate hyperparameter-selection issues, while relaxed local privacy limits potential adversaries’ power.
3 Future Directions
Future work in federated learning must address communication, heterogeneity, and privacy while extending methods beyond supervised learning. It must also resolve production challenges and strengthen realistic, reproducible benchmarks.
- Communication: Communication research should characterize extreme communication schemes and the accuracy–communication Pareto frontier, including how local updating and compression compose.The behavior of one-shot and divide-and-conquer schemes remains poorly understood in massive or statistically heterogeneous networks.
- Systems heterogeneity: Federated systems need device-centric, event-triggered asynchrony because most undedicated devices are inactive during any given iteration.Each device may decide when to wake up and interact with the central server.
- Statistical heterogeneity: Researchers should develop pre-training diagnostics for statistical heterogeneity, since existing measures such as local dissimilarity and earth mover’s distance are difficult to calculate beforehand.Open questions include whether simple diagnostics can be obtained before training over the federated network.
- Privacy: Privacy research should support granular constraints that differ across devices or individual data points, including sample-specific rather than user-specific guarantees.Such sample-specific guarantees provide a weaker form of privacy than user-specific guarantees.
- Learning settings: Federated learning should move beyond supervised learning to address unlabeled or weakly labeled data and exploratory analysis tasks.Existing methods generally assume labels exist for all data in the federated network.
- Production and evaluation: Deployment research must handle concept drift, diurnal variations, and cold starts, while shared benchmarks and tools improve reproducibility and real-world grounding.The paper highlights LEAF and TensorFlow Federated as existing implementations and benchmarking tools.
4 Conclusion
The article surveys federated learning as distributed edge-based model training, covering its distinctive properties, challenges, classical and recent approaches, and open research problems. Addressing these problems will require interdisciplinary effort from broad research communities.
- Conclusion: The article presents an overview of federated learning, where statistical models are trained at the edge in distributed networks.It frames federated learning as a distinct learning paradigm.
- Conclusion: The survey discusses federated learning’s unique properties and challenges, reviews classical and recent work, and outlines open problems for future research.The open problems are identified as worthwhile targets for future research effort.
- Conclusion: Solutions to the identified problems will require interdisciplinary effort from a broad set of research communities.The conclusion explicitly calls for collaboration across research communities.