Source-linked AI summary
No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices
Ruixuan Liu, Fangzhao Wu, Chuhan Wu, Yanlin Wang, Lingjuan Lyu, Hong Chen, Xing Xie
TL;DR
Heterogeneous devices make it difficult to train large federated models without excluding weak clients or limiting model size. InclusiveFL assigns differently sized models to clients and transfers knowledge across them; across datasets, it performs best among approaches applicable to heterogeneous devices, though AllLarge is overall best.
Problem
Weak clients may not support large-model training, while excluding them underrepresents their data and using a small global model limits representation ability.
Source: §1 · pp. 1–2Method
InclusiveFL assigns differently sized models by client capability, shares bottom layers through heterogeneous aggregation, and transfers knowledge from larger to smaller models with momentum distillation.
Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 5Results
Across all datasets, AllLarge achieves the overall best performance, while InclusiveFL performs best among approaches applicable across heterogeneous devices.
Source: §4.1 · pp. 6–7Takeaways & Limitations
InclusiveFL offers a way to train a large global model with contributions from all clients, avoiding the problems attributed to excluding weak clients or using a small model.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Federated learning (FL) is an important paradigm for training global models from decentralized data in a privacy-preserving way. Existing FL methods usually assume the global model can be trained on any participating client. However, in real applications, the devices of clients are usually heterogeneous, and have different computing power. Although big models like BERT have achieved huge success in AI, it is difficult to apply them to heterogeneous FL with weak clients. The straightforward solutions like removing the weak clients or using a small model to fit all clients will lead to some problems, such as under-representation of dropped clients and inferior accuracy due to data loss or limited model representation ability. In this work, we propose InclusiveFL, a client-inclusive federated learning method to handle this problem. The core idea of InclusiveFL is to assign models of different sizes to clients with different computing capabilities, bigger models for powerful clients and smaller ones for weak clients. We also propose an effective method to share the knowledge among multiple local models with different sizes. In this way, all the clients can participate in the model learning in FL, and the final model can be big and powerful enough. Besides, we propose a momentum knowledge distillation method to better transfer knowledge in big models on powerful clients to the small models on weak clients. Extensive experiments on many real-world benchmark datasets demonstrate the effectiveness of the proposed method in learning accurate models from clients with heterogeneous devices under the FL framework.
1 INTRODUCTION
Heterogeneous client resources make large-model federated learning difficult: weak devices may not support local training, while common workarounds sacrifice client representation or model capacity. InclusiveFL assigns different-sized models across clients and shares knowledge between them.
1 INTRODUCTION
- Conventional FL assumes every client can train the same architecture, but weak devices may not support local training of large models [25] [31].Client devices can differ substantially in computing power and memory.Source: §1 · p. 1
- Dropping weak clients underrepresents their data and can reduce accuracy, while using a small model limits representation ability.These trade-offs motivate including heterogeneous clients without restricting the global model to the smallest architecture.Source: §1 · p. 2
- Prior heterogeneous-model methods either require a shared public proxy dataset or risk breaking model structure and misaligning feature spaces [5] [20] [36] [7].The paper identifies parameter sharing alone as insufficient to transfer useful knowledge from stronger to weaker models [15].Source: §1 · p. 2
- InclusiveFL assigns larger models to powerful clients and smaller models to weak clients, while sharing shallow bottom layers through layer-wise heterogeneous aggregation.The shared layers are intended to reduce mismatch; momentum distillation transfers knowledge from large models to small ones.Source: §1 · p. 2
- The paper proposes momentum knowledge distillation and reports experiments to assess learning accurate global models from heterogeneous clients.Its stated goal is for clients with different capabilities to contribute to learning a large, powerful global model.Source: §1 · p. 2
2 BACKGROUND AND RELATED WORK
This section reviews standard federated optimization and prior approaches to heterogeneous models and communication constraints. It contrasts proxy-data distillation and width-based parameter sharing with InclusiveFL’s focus on heterogeneous model training.
2 BACKGROUND AND RELATED WORK
- Federated learning trains across decentralized devices or data-holding servers without exchanging local data, and FedAvg is a conventional aggregation protocol [25].FedAdam applies adaptive methods on the server and SGD on clients.Source: §2 · p. 2
- Prior distillation approaches train a server or local models from prediction ensembles, but depend on public data that may be difficult to guarantee or store [5] [20] [21].The paper notes that proxy datasets can be impractical for weak clients with limited hardware.Source: §2 · pp. 2–3
- HeteroFL [7] shares averaged overlapping parameter submatrices across width-scaled models, but pruning and parameter mismatch can limit knowledge transfer.InclusiveFL instead reduces the number of layers for smaller models.Source: §2 · p. 3
- Communication-cost methods compress or subsample updates or drop activations, but do not address heterogeneous-model training and are compatible with InclusiveFL [20] [33].The cited work includes approaches to both uplink and server-to-client communication.Source: §2 · p. 3
3 METHODOLOGY
InclusiveFL trains models of different depths for client groups with different capabilities, then combines within-group and cross-group learning. It shares bottom encoder layers while keeping selected top layers separate, and uses gradient momentum to transfer knowledge from larger to smaller models.
3.1 Layer-wise Heterogeneous Aggregation Framework
- InclusiveFL assigns models of increasing depth to weak, medium, and strong clients, aggregates within each group, then performs heterogeneous aggregation across groups.The server samples clients and groups them by device type in each communication round.Source: §3.1 · p. 3
- The framework shares bottom encoder layers but isolates the last encoder and task-specific top layers to address architectural mismatch and support different target labels.The shared module contains bottom encoding layers; the isolated module contains the last encoder and task-specific layers.Source: §3.1 · p. 4
3.2 Momentum Distillation
- Momentum distillation aligns a small model’s isolated encoder layer with corresponding top layers in a larger model, where hidden-loss alignment is unsuitable across separated client data.For small and medium models, the method injects gradient momentum averaged over a stack of layers in the larger model; the momentum is initialized to zero and scaled by β.Source: §3.2 · pp. 4–5
4 EXPERIMENTS
Experiments span IID GLUE text classification and non-IID medical NER across clients assigned 4-, 8-, and 12-layer models. InclusiveFL leads the heterogeneous-device baselines, while momentum distillation reduces rounds to best performance.
4 EXPERIMENTS
- InclusiveFL is evaluated on IID GLUE classification and non-IID medical NER, with weak, medium, and powerful clients assigned models of 4, 8, and 12 layers.The medical setting uses SMM4H, CADEC, and ADE; GLUE is split across 1,000 clients.Source: §4 · p. 5
4.1 Performance Evaluation
- Across GLUE and medical NER, InclusiveFL achieves the best performance among approaches implemented across heterogeneous devices; AllLarge is best overall.Table 2 evaluates the final large model for global inference, while Table 3 evaluates local models for medical NER.Source: §4.1 · pp. 6–7, Table 2 · p. 6, Table 3 · p. 6
- HeteroFL performs worse than AllSmall and ExclusiveFL on every GLUE dataset and is also inferior to their average on medical datasets.The authors attribute this in part to cropping that does not fully use pretrained parameters, whereas layer-wise sharing better maintains pretrained-model utility.Source: §4.1 · p. 7
- ExclusiveFL performs better than AllSmall on GLUE, while Local performs worse than AllSmall, showing that data inclusion and larger-model capacity are difficult to trade off by task.Source: §4.1 · p. 7
- Momentum distillation accelerates convergence: on IID MRPC, InclusiveFL reaches F1-score 89.43 within 400 rounds versus 550 without distillation.On non-IID medical NER, it reaches accuracy 60.5 within 650 rounds versus 950 without distillation.Source: Figure 3 · p. 6, §4.1 · p. 7
4.2 Ablation Study
- Excluding any device type performs worse than client-inclusive training; powerful, medium, and weak clients contribute in decreasing order, though some medical ablations are inapplicable.The authors report that those medical tasks cannot be trained without their respective local datasets.Source: §4.2 · p. 8, Figure 5 · p. 8
4.3 Hyper-Parameter Analysis
- InclusiveFL's performance advantage persists across the two tested device-type proportions, but a 7:2:1 ratio with few powerful clients slows largest-model convergence and destabilizes aggregation.For this minority-powerful-client case, the variant using momentum distillation without layer-wise parameter sharing is slightly worse than ExclusiveFL's Matthew’s correlation of 36.87.Source: Table 5 · p. 7, §4.3 · p. 8
- A momentum factor β around 0.5 is optimal for powerful-client models, while β above 0.8 may harm smaller models by injecting too much momentum.Medium- and weak-client models also perform well at β=0.2 or 0.5.Source: §4.3 · p. 8, Figure 6 · p. 8
5 CONCLUSIONS
InclusiveFL trains a large global model across heterogeneous devices by assigning models of different sizes to clients. It combines heterogeneous aggregation and momentum distillation, with experiments reporting accuracy improvements for both large and smaller models.
5 CONCLUSIONS
- InclusiveFL assigns different-sized models to clients by capability, allowing all clients to contribute to training a large global model.The framework aims to avoid problems from excluding weak clients or using only a small model.Source: §5 · p. 8
- Layer-wise heterogeneous aggregation shares knowledge across differently sized local models, while momentum distillation transfers knowledge from large models to smaller ones.Source: §5 · p. 8
- Experiments under IID and Non-IID settings report accuracy improvements for InclusiveFL's large global model and smaller models used for local inference.Evaluation covers benchmark data and real-world medical corpora.Source: §5 · p. 8