Source-linked AI summary
FedDUAP: Federated Learning with Dynamic Update and Adaptive Pruning Using Shared Data on the Server
Hong Zhang, Ji Liu, Juncheng Jia, Yang Zhou, Huaiyu Dai, Dejing Dou
TL;DR
Federated learning must make effective use of limited device data and computing capacity, while privacy restrictions constrain data sharing and existing server-data methods can be costly. FedDUAP combines dynamic updates using server and device data with layer-adaptive pruning; experiments report gains in accuracy, efficiency, and computational cost over state-of-the-art baselines.
Problem
Server-data approaches can incur high communication costs or low efficiency, while uniform pruning overlooks layer differences and may yield poor approximations.
Source: §1 · pp. 1–2Method
FedDUAP combines dynamic server updates using server and device data with adaptive layer pruning that accounts for layer dimensions, importance, and non-IID degrees.
Source: §1 · p. 2Results
Up to 4.8% higher accuracy, up to 2.8 times faster efficiency, and up to 61.9% smaller computational cost than state-of-the-art baseline approaches.
Source: §5 · p. 6Takeaways & Limitations
The reported advantage spans accuracy, training efficiency, and computational cost across the evaluated models and real-life datasets.
Source: §5 · p. 6The method assumes server and devices can share statistical meta information Pk and nk, although the data itself cannot be transferred between them.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Despite achieving remarkable performance, Federated Learning (FL) suffers from two critical challenges, i.e., limited computational resources and low training efficiency. In this paper, we propose a novel FL framework, i.e., FedDUAP, with two original contributions, to exploit the insensitive data on the server and the decentralized data in edge devices to further improve the training efficiency. First, a dynamic server update algorithm is designed to exploit the insensitive data on the server, in order to dynamically determine the optimal steps of the server update for improving the convergence and accuracy of the global model. Second, a layer-adaptive model pruning method is developed to perform unique pruning operations adapted to the different dimensions and importance of multiple layers, to achieve a good balance between efficiency and effectiveness. By integrating the two original techniques together, our proposed FL model, FedDUAP, significantly outperforms baseline approaches in terms of accuracy (up to 4.8% higher), efficiency (up to 2.8 times faster), and computational cost (up to 61.9% smaller).
1 Introduction
Federated learning faces constraints from limited device data and computing capacity, while privacy restrictions complicate pooling sensitive data. FedDUAP combines dynamic server updates with layer-adaptive pruning to use server and device data and improve efficiency.
1 Introduction
- Device data may be limited and devices have constrained computing and communication capacity; sensitive data transfer is also restricted, while server-data methods can incur communication costs or low efficiency.The introduction also notes that uniform pruning can overlook layer dimensions and importance, potentially producing poor approximations and increasing FL training failure probability [Zhao et al., 2018].Source: §1 · pp. 1–2
- FedDUAP combines dynamic federated updates and adaptive pruning, using server and device data while accounting for their non-IID degrees.The paper describes non-IID degree as the difference between a dataset's distribution and the distribution across all devices.Source: §1 · p. 2
- FedDU dynamically adjusts the global model using its accuracy, non-IID degrees, and normalized stochastic gradients.The method uses both server and device data.Source: §1 · p. 2
- FedAP adapts pruning operations to layer dimensions and importance based on non-IID degrees, aiming to improve efficiency while retaining accuracy comparable to the original model.Source: §1 · p. 2
- FedDUAP significantly outperforms state-of-the-art approaches on four typical models and two datasets in accuracy, efficiency, and computational cost.The introduction reports extensive experiments but gives no numerical gains here.Source: §1 · p. 2
2 Related Work
Related work addresses performance using device data, server data, and pruning, but identifies trade-offs in server-data use and limitations in pruning for federated learning.
2 Related Work
- Prior server-data approaches use knowledge transfer for objectives such as heterogeneous models, label-free data, or one-shot training, but may have low accuracy or long training time; transferring server data to devices incurs significant communication costs.Server data is also used for client selection, an approach described as orthogonal to and combinable with FedDUAP.Source: §2 · p. 2
- Unstructured pruning can preserve accuracy and reduce communication costs, but offers limited computational savings on general-purpose hardware; structured pruning can reduce both costs but is difficult to configure and not well adapted to FL.Source: §2 · p. 2
3 Method
FedDUAP combines collaborative training on server and device data with FedDU for dynamic server updates and FedAP for layer-adaptive pruning. The method uses non-IID information to guide both updates and pruning.
3.1 System Model
- The system pairs a server with edge devices, uses their locally held data for training, and represents distribution differences with Jensen–Shannon divergence.The server and devices may share statistical information, including label distributions and sample counts, rather than transferring the data.Source: §3.1 · pp. 2–3
3.2 Server Update
- FedDU adjusts server-update steps using model accuracy and server/device non-IID degrees, while normalizing server gradients to reduce objective inconsistency.The server-data influence is prominent early in training and becomes smaller later as the model approaches the stated training objective.Source: §3.2 · pp. 3–4
3.3 Adaptive Pruning
- FedAP estimates pruning from Hessian eigenvalue gaps, aggregates rates across non-IID sites, then sets layer rates and preserves higher-ranked feature maps on the server.The pruning operation is performed at a specific training round and uses server-side feature-map ranks to select filters for retention.Source: §3.3 · p. 4, Figure 2 · p. 5
4 Experiments
Experiments compare FedDU, FedAP, and their combination in federated learning, reporting accuracy, training-time, and computational-cost results.
4 Experiments
- The evaluation compared FedAvg, Data-sharing, Hybrid-FL, HRank, IMC, and PruneFL with the proposed methods.FedDU comparisons include Data-sharing, which sends server data to devices, and Hybrid-FL, which treats the server as an ordinary FedAvg device.Source: §4 · p. 5, Evaluation on FedDU · p. 5
4.1 Experimental Setup
- The experiments used a parameter server and 100 devices, randomly selecting 10 devices per round, across CIFAR-10, CIFAR-100, CNN, ResNet18, VGG11, and LeNet.Reported results focus on CNN with CIFAR-10 and ResNet with CIFAR-100; other experimentation results were described as consistent.Source: §4.1 · p. 5
Evaluation on FedDU
- FedDU improved accuracy over FedAvg by up to 4.9%, Data-sharing by 4.1%, and Hybrid-FL by 19.5% for CNN and ResNet at p = 5% and 10%.On CNN, Data-sharing took up to 15.7 times longer than FedDU to reach accuracy 0.6.Source: Evaluation on FedDU · p. 5
- Dynamic adjustment of τeff outperformed a static value by up to 1.7% in accuracy, and improvement was up to 6.1 times faster with server-data non-IID degree 9.0 ∗10−6 than 0.31.The reported server-data distribution comparison indicates faster global-model improvement at the smaller non-IID degree.Source: Evaluation on FedDU · p. 5
Evaluation on FedAP
- FedAP selected average pruning rates of 37.8% for CNN and 72.0% for VGG, reaching up to 1.8 times faster training with up to 0.1% accuracy reduction versus unpruned FedAvg.The experiments report model-specific suitable pruning rates; FedAP also reduced computational cost by up to 55.7%.Source: Evaluation on FedAP · p. 5, Figure 4 · p. 6
Evaluation on FedDUAP
- FedDUAP achieved 4.8% higher accuracy than FedAvg, 12.0% than Data-sharing, 18.4% than Hybrid, 6.9% than IMC, and 6.3% than PruneFL.It also trained up to 2.8 times faster than FedAvg and reduced computational cost by up to 61.9% across the listed baselines.Source: Evaluation on FedDUAP · p. 6, Table 2 · p. 6
5 Conclusion
FedDUAP combines dynamic federated updates using server and device data with adaptive pruning; experiments report gains in accuracy, efficiency, and computational cost.
5 Conclusion
- FedDUAP outperformed state-of-the-art baselines by up to 4.8% in accuracy, trained up to 2.8 times faster, and used up to 61.9% less computational cost.The framework combines FedDU with FedAP and accounts for non-IID degrees in server and device data.Source: §5 · p. 6