Source-linked AI summary
Federated Learning for Computationally-Constrained Heterogeneous Devices: A Survey
Kilian Pfeiffer, Martin Rapp, Ramin Khalili, Jörg Henkel
TL;DR
On-device learning can protect local data, but single-device training limits accuracy and some devices cannot train, making computational heterogeneity a challenge for FL. This survey reviews FL under heterogeneous computation constraints, categorizes those constraints, and identifies open problems in comparability and effectiveness across heterogeneity scales and granularities.
Problem
Local-only training can limit accuracy and generalization, and some devices cannot train, constraining FL across heterogeneous devices.
Source: §1 · p. 2Method
The survey analyzes device constraints and categorizes them as hard or soft, varying across devices, rounds, and time.
Source: §6 · p. 21Results
The survey reviews current FL research under heterogeneous computation constraints and identifies open problems, including comparability.
Source: §6 · p. 21Takeaways & Limitations
Effectiveness remains an open question across heterogeneity scales and granularities.
Source: §6 · p. 21Some approaches require device resources to be known before a training round, and effectiveness under large-scale, fine-grained heterogeneity remains unclear.
Source: §4.2 · p. 14
Abstract
from arXiv · showhide
With an increasing number of smart devices like internet of things (IoT) devices deployed in the field, offloadingtraining of neural networks (NNs) to a central server becomes more and more infeasible. Recent efforts toimprove users' privacy have led to on-device learning emerging as an alternative. However, a model trainedonly on a single device, using only local data, is unlikely to reach a high accuracy. Federated learning (FL)has been introduced as a solution, offering a privacy-preserving trade-off between communication overheadand model accuracy by sharing knowledge between devices but disclosing the devices' private data. Theapplicability and the benefit of applying baseline FL are, however, limited in many relevant use cases dueto the heterogeneity present in such environments. In this survey, we outline the heterogeneity challengesFL has to overcome to be widely applicable in real-world applications. We especially focus on the aspect ofcomputation heterogeneity among the participating devices and provide a comprehensive overview of recentworks on heterogeneity-aware FL. We discuss two groups: works that adapt the NN architecture and worksthat approach heterogeneity on a system level, covering Federated Averaging (FedAvg), distillation, and splitlearning-based approaches, as well as synchronous and asynchronous aggregation schemes.
1 INTRODUCTION
On-device learning can protect data privacy and reduce centralized communication, but local-only training limits accuracy and generalization, while device heterogeneity complicates real-world FL. The survey reviews computation-aware FL and organizes its challenges and approaches.
1 INTRODUCTION
- On-device learning keeps local samples on devices and avoids uploading training data, but local-only models can lose accuracy and generalization.Some deployed devices cannot train at all, and capable devices may have only a tiny subset of a potential centralized dataset.Source: §1 · p. 2
- FL distributes training across devices while letting them collaborate by sharing knowledge, commonly through neural-network weights.The introduction describes FL as improving privacy relative to centralized cloud training while supporting accuracy and generalization.Source: §1 · p. 2
- Device capability differences make real-world FL challenging, and only a few production use cases, including GBoard [100], evidence its success.NN training can take weeks for complex tasks, while IoT devices, smartphones, and embedded systems remain constrained.Source: §1 · p. 2
- Computation heterogeneity can exclude devices holding essential data, while data and communication capabilities also vary across devices.The introduction notes that fairness or fair representation may require learning from constrained devices.Source: §1 · p. 3
1.1 Scope and Contribution
- This survey focuses on computation heterogeneity, while treating communication and data heterogeneity when they overlap with computation.It excludes hardware-design approaches not specific to FL and refers readers elsewhere for inference-only federated neural architecture search.Source: §1.1 · p. 3
- The survey categorizes approaches by NN-architecture versus system-level adaptation, then by FL paradigm or synchronous versus asynchronous aggregation.It distinguishes constraint types, heterogeneity types, scale, and granularity, and includes distillation-based approaches to computation heterogeneity.Source: §1.1 · pp. 3–4
2 BASICS OF FEDERATED LEARNING
The section lays out federated learning’s distributed objective and baseline knowledge-sharing approaches. It describes how device selection, asynchronous updates, distillation, and split learning address different aspects of collaborative training.
2.1 Problem Formulation
- FL minimizes a global loss across devices with disjoint local data; heterogeneity-aware methods target faster convergence and high final accuracy despite constrained devices.The accuracy bounds contrast centralized training with learning from local data alone.Source: §2.1 · pp. 4–5
2.2 Baseline Federated Averaging (FedAvg)
- Synchronous FedAvg distributes current weights, trains participating devices for a fixed local workload, then averages their uploaded models at the server.The figure depicts this cycle as round-by-round sharing of averaged weights.Source: §2.2 · p. 5, Figure 1 · p. 5
2.3 Client Selection in FedAvg
- Client selection uses availability and resource expectations, plus per-round deadlines, to target faster convergence, fewer stragglers, or lower energy use.A slow device can delay synchronous aggregation or be dropped from the round.Source: §2.3 · p. 6
2.4 Asynchronous FedAvg
- Asynchronous FedAvg aggregates updates as they arrive, avoiding stragglers but allowing stale updates that can reduce convergence speed and maximum reachable accuracy.Stale devices submit weights based on an older global-model state.Source: §2.4 · p. 6
2.5 Distillation for Federated Learning
- FedMD shares knowledge through soft labels on a shared public dataset rather than model weights, allowing devices to use different model architectures.Distillation uses predicted class-probability distributions, rather than only class labels, to transfer model knowledge.Source: §2.5 · p. 6, Figure 2 · p. 7
2.6 Split Learning
- Split learning sends device-computed activations and labels to a server model, then returns gradients; it can reduce device computation and speed convergence, at high communication cost.The device uses the returned gradient to update its model before the process repeats.Source: §2.6 · p. 7
3 COMPUTATION HETEROGENEITY IN FEDERATED LEARNING
The section focuses on differences in devices’ ability to train neural networks and organizes computation heterogeneity by constraint type and how it varies across a system. It also considers communication and data quantity where they overlap with computation.
3.1 Computational Resources in End-Devices
- Devices vary in training resources, and recent neural-network topologies are unrealistic for all devices, particularly memory-limited embedded systems.The section notes substantial variation even among smartphones.Source: §3.1 · p. 8
3.2 Categorization of Constraints and Heterogeneity
- Computation heterogeneity can occur across devices, over rounds, or within rounds over time, requiring different forms of adaptation.Device constraints may be known in advance, change between rounds, or fluctuate unpredictably during a round.Source: §3.2 · pp. 9–10
- Hard constraints can prevent training a model, while soft constraints reduce throughput and may leave clients as stragglers or stale devices.Limited memory is a prominent hard constraint; resource contention and unstable power are examples of soft constraints.Source: §3.2 · p. 9
- Heterogeneity varies in scale and granularity, so an FL technique must accommodate the resource variation present in its system.Finite device tiers differ from continuously varying contention, which may need to be quantized into supported levels.Source: §3.2 · p. 10
3.3 Communication Heterogeneity
- Communication limits can create stragglers or stale devices, and reducing NN structure can lower both computation complexity and communication burden.Training and communication throughput may also compete for shared resources such as energy.Source: §3.3 · p. 11
3.4 Data Quantity Heterogeneity
- Devices with more data require more mini-batch training per round, while limited-throughput techniques can help prevent them becoming stragglers or stale devices.Source: §3.4 · p. 11
4 COMPUTATION HETEROGENEITY-AWARE FEDERATED LEARNING
This section surveys computation-aware federated learning methods, spanning architecture- and system-level adaptations for heterogeneous devices. It also examines asynchronous aggregation, its straggler and stale-update tradeoffs, and the need for further convergence analysis.
4.1 Categorization of Techniques
- The survey distinguishes architecture-level techniques, which vary model complexity or exchange distilled knowledge, from system-level techniques, which adapt rounds, grouping, updates, or aggregation timing.Table 3 compares techniques by hard or soft constraints and heterogeneity across devices, rounds, and time.Source: §4.1 · p. 11, Table 3 · p. 12
4.2 NN Architecture Heterogeneity based on FedAvg
- FedAvg-based methods fit device budgets with changing or nested submodels, but the effectiveness of very small models and fine-grained subset scales remains untested.FedAvg-derived approaches also use low-rank factorization to reduce device computation and communication; subsets remain embedded in the full model and are updated during training.Source: §4.2 · pp. 13–14
4.3 NN Architecture Heterogeneity based on Distillation
- Distillation permits device-tailored architectures and per-class representations, but often trails FedAvg, requires stateful participation, and adds computation or public-data burdens.FedDF improves robustness in non-iid settings and permits more local steps without performance degradation, but requires server data for distillation.Source: §4.3 · pp. 15–16
4.4 NN Architecture Heterogeneity based on Other Techniques
- AdaSplit [20] accommodates device heterogeneity by varying the device-server split and using local contrastive training to reduce server interaction, but split learning requires server computation and stateful participation.FedGKT combines a lightweight device feature extractor with a server model and exchanges knowledge asynchronously, but does not support heterogeneous splits or aggregation.Source: §4.4 · p. 16
4.5 System Level Awareness Through Client Selection and Flexible Aggregation
- Uniform TiFL tier selection matches baseline FedAvg accuracy while reducing training time by 50%, while active resource-based selection can hurt performance when device resources correlate with non-iid data.System-level methods also use client impact to vary round lengths, permit partial local updates, and optimize communication, computation, or energy trade-offs.Source: §4.5 · pp. 17–18
4.6 System-Level Awareness Through Asynchronous Aggregation
- FedAT aggregates synchronously within performance tiers and asynchronously across them, outperforming baseline FedAvg, TiFL, and pure asynchronous schemes in experiments with 100 devices.Asynchronous convergence guarantees are limited, with most theoretical results covering convex asynchronous distributed SGD rather than asynchronous FedAvg.Source: §4.6 · p. 19
- Asynchronous aggregation allows devices to upload at any time, accommodating arbitrarily high heterogeneity granularity and fully addressing stragglers, while stale updates limit the scale of heterogeneity.Source: §4.6 · p. 19
- Neither synchronous nor asynchronous aggregation is known to be ultimately preferable; depending on assumptions, both can have advantageous properties.Source: §4.6 · pp. 19–20
- Further theoretical work is needed to study convergence beyond convex utility cases.Source: §4.6 · p. 20
5 OPEN PROBLEMS AND FUTURE DIRECTIONS
The survey identifies open problems in evaluating and sustaining computation-heterogeneity-aware FL, including unclear effectiveness across heterogeneity scales and a lack of comparable, device-representative scenarios. It also highlights unexplored resource trade-offs and resource-correlated non-iid data.
Problem 1: Maintaining effectiveness under fine-grained granularity or large-scale heterogeneity
- Effectiveness at high heterogeneity scales remains unclear; reported resource scales range from 4×−25× to 100×−250×.Studies often omit trivial baselines such as dropping devices or deploying a low-complexity model, and the effects of combining NN-level and system-level approaches on convergence and accuracy remain unclear.Source: §5 · p. 20
Problem 2: Comparability
- Comparability is limited by inconsistent resource measures, heterogeneity groupings, device counts, and data shares across evaluations.The survey calls for benchmarks that represent device constraints and for heterogeneity support in popular FL frameworks.Source: §5 · pp. 20–21
- Current benchmarks do not represent real-world device constraints, and popular FL simulation frameworks lack specific support for device heterogeneity, memory, throughput, or energy constraints.The survey calls for more device-representative benchmarks and heterogeneity support in popular frameworks.Source: §5 · p. 21
Problem 3: Unexplored trade-offs and non-iid data scenarios
- Energy, communication, computation, and memory trade-offs remain underexplored in heterogeneous FL, despite limited energy in many embedded and IoT settings.Devices may trade energy between communication and computation, or memory use against dynamically recomputing intermediate results.Source: §5 · p. 21
- Resource-correlated non-iid data is largely unstudied, although weak devices may hold data needed for fair representation of user groups.The survey calls for research on how these correlations affect the global model and how their effects can be mitigated.Source: §5 · p. 21
6 CONCLUSION
The survey reviews FL under computation heterogeneity among participating devices, categorizes hard and soft computational constraints, and surveys techniques addressing them. It identifies open problems in comparability, effectiveness across heterogeneity scales and granularities, and unexplored trade-offs.
6 CONCLUSION
- The survey categorizes device constraints as hard or soft, with soft constraints varying across devices, rounds, and time and producing different heterogeneity scales and granularities.It reviews research on FL under heterogeneous computation constraints and identifies open problems in comparability, effectiveness, and unexplored trade-offs.Source: §6 · p. 21