Source-linked AI summary
FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout
Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos I. Venieris, Nicholas D. Lane
TL;DR
Federated learning must accommodate clients with widely varying hardware and network resources without excluding devices or constraining the global model to the least capable participant. FjORD uses Ordered Dropout and self-distillation to train nested, width-adaptive submodels, and reports performance gains over competing techniques while preserving flexibility across environments. The paper also identifies sustainability concerns when computation shifts to users’ household devices.
Problem
Client system heterogeneity can exclude devices or force a global model sized for the least capable participants, affecting participation and model capacity.
Method
FjORD combines Ordered Dropout, which extracts nested submodels without retraining, with self-distillation from a maximal supported submodel.
Results
FjORD’s performance exceeds competing techniques in local and federated settings while maintaining flexibility across different environment setups.
Takeaways & Limitations
FjORD supports heterogeneous-device participation and dynamically scaled models while retaining a nested structure.
Takeaways & Limitations
On-device training shifts non-negligible computation costs to users, and household electricity may impede sustainability in the short run.
Abstract
from arXiv · showhide
Federated Learning (FL) has been gaining significant traction across different ML tasks, ranging from vision to keyboard predictions. In large-scale deployments, client heterogeneity is a fact and constitutes a primary problem for fairness, training performance and accuracy. Although significant efforts have been made into tackling statistical data heterogeneity, the diversity in the processing capabilities and network bandwidth of clients, termed as system heterogeneity, has remained largely unexplored. Current solutions either disregard a large portion of available devices or set a uniform limit on the model's capacity, restricted by the least capable participants. In this work, we introduce Ordered Dropout, a mechanism that achieves an ordered, nested representation of knowledge in deep neural networks (DNNs) and enables the extraction of lower footprint submodels without the need of retraining. We further show that for linear maps our Ordered Dropout is equivalent to SVD. We employ this technique, along with a self-distillation methodology, in the realm of FL in a framework called FjORD. FjORD alleviates the problem of client system heterogeneity by tailoring the model width to the client's capabilities. Extensive evaluation on both CNNs and RNNs across diverse modalities shows that FjORD consistently leads to significant performance gains over state-of-the-art baselines, while maintaining its nested structure.
1 Introduction
FjORD addresses federated learning’s device heterogeneity by adapting model size to client capabilities. Ordered Dropout enables nested submodels, while evaluation reports accuracy benefits without retraining.
- Federated learning trains models without moving users’ data off their devices.
- Ordered Dropout creates an ordering in which left-most features are used by more devices during training.
- Heterogeneous devices force deployments either to exclude low-tier clients or restrict global model capacity.These choices can introduce training bias or degrade accuracy.
- FjORD dynamically adapts model size, computation, memory, and data exchange to available client resources.It uses Ordered Dropout to train nested submodels while allowing devices with different capabilities to contribute to the global model.
- FjORD produces models that can be dynamically scaled during inference according to device hardware and load constraints.
- FjORD shows significant accuracy benefits over baselines while extracting submodels with varying FLOPs and sizes without retraining.
2 Motivation
Federated learning in the wild is constrained by heterogeneous, resource-limited client hardware. This heterogeneity creates stragglers and can exclude devices and their data from training.
- Mobile hardware differences produce varying processing speeds and longer waits when client updates are aggregated.These slower participants become stragglers in the training process.
- Mid- and low-tier devices may be unable to support larger models because of memory limits or slow processing.
- Unsupported or timed-out devices can be excluded from training together with their unique data.
- Network communication can become an additional substantial bottleneck for federated training.
3 Ordered Dropout
Ordered Dropout constructs nested subnetworks by retaining an ordered prefix of each layer, allowing model footprint to match device capabilities. The formulation supports post-training extraction, self-distillation, and a theoretical connection to SVD.
- Ordered Dropout orders knowledge representation in nested submodels that can be extracted from an original network.The method is introduced as an importance-based pruning mechanism for subnetworks with different computational and memory footprints.
- OD samples a candidate p value and drops adjacent model components rather than random neurons.Each p corresponds to a submodel with a specific computational and memory footprint.
- 3.1 Ordered Dropout Mechanics: For each layer of width K_l, a p-subnetwork retains indices 0 through ⌈p · K_l⌉−1 and removes the remaining units and unnecessary connections.A subnetwork can be obtained after training without fine-tuning or labelled data access.
- 3.2 Training OD Formulation: OD training samples p, computes on the corresponding reduced network, and updates its weights; knowledge-distillation training additionally uses the maximal supported submodel as teacher.
- 3.2 Training OD Formulation: The distillation loss combines KL divergence from the teacher with cross-entropy on the teacher output.The reported best distillation values are α = T = 1.
- 3.3 Ordered Dropout exactly recovers SVD: For linear mappings, OD recovers the best b-rank approximation of A, where b = ⌈p · K⌉.This establishes an ordered importance representation connected to SVD.
- OD reduces FLOPs and model size by skipping forward, backward, and gradient computations for pruned units.
- Each device can be assigned a maximum supported submodel according to capabilities such as memory, processing throughput, or energy budget.
4 FjORD
FjORD builds on Ordered Dropout to train nested subnetworks across heterogeneous clients, adapting model capacity to device constraints while reducing computation and communication.
- Framework workflow: FjORD extends Ordered Dropout into a federated framework for heterogeneous clients.Each client trains a submodel suited to its capabilities, while the server aggregates updates into the global model.
- Framework workflow: Each device cluster receives a maximum model width determined by its latency or memory capacity.Clients sample narrower widths during local iterations from the dropout distribution, conditional on their maximum supported width.
- Federated aggregation: FjORD aggregates updates by parameter subsets, averaging each subset only across clients capable of updating it.The aggregation rule accounts for nested subnetworks and weights client contributions by local data volume.
- Efficiency: Ordered Dropout reduces computation and communication by transmitting and updating only the unpruned units of each submodel.For downstream and upstream transfers, the reduction is approximately 1/(p_i^max)^2 for clients using smaller models.
- Additional design choices: FjORD’s ordered neuron importance also mitigates permutation-related aggregation problems without computationally expensive neuron matching.Knowledge distillation is available for memory- or network-bound clients, but may be omitted when computation is the bottleneck because of its overhead.
5 Evaluation of FjORD
The evaluation tests FjORD across vision and text tasks, baselines, dropout distributions, and device allocations. FjORD improves performance while retaining elasticity across model widths and client populations.
- Experimental setup: FjORD is evaluated on CIFAR10, FEMNIST, and Shakespeare using CNN and RNN models.The setup includes two vision tasks and one text-prediction task, with experiments implemented on a cloud GPU cluster.
- Performance evaluation: FjORD outperforms eFD across datasets, with gains of 1.53-34.87 pp on CIFAR10, 1.57-6.27 pp on FEMNIST, and 0.01-0.82 points on Shakespeare.The comparison uses the uniform-5 setup with drop scale 1.0 and evaluates performance across dropout rates.
- Performance evaluation: FjORD requires one training process and outperforms eFD’s best model by 2.98 pp on CIFAR10, 2.73 pp on FEMNIST, and 0.13 p on Shakespeare.Obtaining eFD’s best model requires training multiple models, one for each device cluster.
- Knowledge-distillation ablation: Knowledge distillation improves FjORD’s global-model performance by 0.71 pp on CIFAR10, 2.63 pp on FEMNIST, and 0.10 p on Shakespeare.Its benefits are consistent for p > 0.4, while the impact fades for p ≤0.4; alternative weighted strategies remain future work.
- Elasticity and adaptability: FjORD sustains performance when increasing granularity from uniform-5 to uniform-10 device clusters.With more high-tier devices, larger subnetworks are more accurate, while smaller subnetworks do not significantly degrade because local rounds sample p values.
6 Related Work
The paper contrasts Ordered Dropout with conventional and nested dropout techniques and with pruning methods that impose a common model across clients. It also examines FjORD’s scalability across device clusters and adaptability to device distributions.
- Dropout Techniques: Ordered Dropout uses structured, ordered dropping to reduce computational and memory costs during training and inference.Unlike Random Dropout, it primarily targets tunable resource reduction rather than regularisation.
- Scalability: FjORD’s scalability is evaluated with respect to the number of device clusters.
- Adaptability: FjORD’s adaptability is evaluated across different device distributions.
- Dropout Techniques: Nested Dropout is the closest related technique, but prior applications focus on ordered representations in autoencoders or feature extractors.
- Computation-Communication Co-optimisation: PruneFL uses a common pruned model for all clients in a round, so the model must satisfy the least capable devices and cannot let stronger devices update more weights.
7 Conclusions & Future Work
The conclusion presents FjORD as a method for heterogeneous-device federated training built on Ordered Dropout. It reports flexibility across environments and identifies future work on deployment, personalisation, and dynamic inference.
- Conclusions: FjORD extracts smaller-footprint submodels from a main model so training a part also participates in training the whole.
- Conclusions: Ordered Dropout is equivalent to SVD for linear mappings, while FjORD exceeds competing techniques in local and federated settings.
- Future Work: Future work will investigate lifelong deployment, system and data heterogeneity for personalisation, and alternative dynamic inference techniques.
Broader Impact
The paper describes broader impacts for privacy, participation fairness, and the environment. FjORD can include less capable devices, but shifting computation to users raises consent and sustainability concerns.
- Fairness in Participation: FjORD enables less capable devices to participate in training, increasing representation of low-tier devices and correlated demographic groups.
- Privacy and Environment: On-device federated training keeps data on device premises and can add deployed client devices’ compute cycles to the distributed system.
- Limitations: Moving computation to users creates a non-negligible burden, requiring awareness when devices contribute to models users do not own.
Funding Disclosure
The work was entirely performed at and funded by Samsung AI.
- Funding Disclosure: The work was performed at Samsung AI.
- Funding Disclosure: Samsung AI funded the work.
- Funding Disclosure: The disclosure identifies Samsung AI as both the institution and funder.
Supplementary Material
The supplementary analysis characterizes Ordered Dropout through its linear-map connection to SVD and shows that the resulting optimization problem remains well behaved.
- Ordered Dropout’s linear-map analysis assumes data and responses linked by y = Ax and uses SVD notation for A.
- For uniformly distributed inputs on the unit ball, the objective can be simplified using the network structure and the Frobenius norm.
- Because each sampled width b has nonzero probability, expectation over dropout widths can be computed explicitly.
- The Eckart–Young theorem is used because the relevant approximation has rank at most b.
- Optimization Perspective: With strong convexity parameter µ and smoothness parameter L, the Ordered Dropout objective has µ′ ≥ µ and L′ ≤ L.
C.1 Datasets and Models
The experiments evaluate FjORD and related Ordered Dropout methods across vision and language tasks, comparing accuracy, convergence, and resource-scaled submodels against multiple baselines.
- Datasets and Models: The study uses EMNIST, FEMNIST, CIFAR10, and Shakespeare across centralized and federated settings, including IID and naturally heterogeneous partitions.
- Datasets and Models: CIFAR10 is federated across 100 clients with 500 data points each and uses ResNet18 with independent batch-normalization layers for each Ordered Dropout width.
- Federated Setup: Federated experiments sample 10 clients per communication round, run one local epoch, and train for 500 global rounds using SGD without momentum.
- SVD Recovery: The SVD experiment trains factor matrices on a full-rank 5 × 5 matrix with distinct singular values for 10k iterations using SGD at learning rate 0.1.
- Ordered versus Random Dropout: Random Dropout causes catastrophic performance drops when extracting submodels from the original model, unlike Ordered Dropout.
- Convergence: FjORD with or without distillation converges more smoothly and achieves lower losses than eFD across three model–dataset combinations over 500 rounds.
- Federated Dropout Comparison: eFD exceeds FD by 27.13–33 percentage points on CIFAR10, 4.59–9.04 on FEMNIST, and 1.51–6.96 on Shakespeare.
- Federated Dropout Comparison: On higher-end devices, FD can reduce computation, upstream bandwidth, and energy, but this may waste available computation and reduce global accuracy.
E Limitations
The limitations section identifies evaluation and deployment boundaries: simplified dropout sampling, absent on-device benchmarks, and externally specified device clusters.
- The evaluation uses uniform sampling of p values and keeps p constant across network layers for simplicity and tractability.
- The authors state that this simplified sampling choice does not degrade the generality of the approach.
- The method has not been benchmarked on target mobile devices, and in-the-wild deployment at scale is deferred to future work.
- Device clusters and their loads are modeled, but the device-to-cluster association is outside the scope of the work.