Source-linked AI summary
When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methods
Zhuo Zhang, Yuanhang Yang, Yong Dai, Lizhen Qu, Zenglin Xu
TL;DR
Federated PLM fine-tuning is costly because large models create communication and local adaptation burdens under privacy-sensitive data constraints. The paper benchmarks parameter-efficient tuning in FL through FedPETuning, finding that lightweight parameter exchange can preserve acceptable performance while reducing resource costs and supporting privacy preservation.
Problem
Full PLM fine-tuning in FL faces prohibitive communication and local adaptation costs, motivating resource-efficient methods for privacy-sensitive NLP.
Method
The paper empirically evaluates representative PETuning methods in FL and introduces FedPETuning, which locally tunes and globally aggregates lightweight parameters.
Results
FedAP reduces communication by 97.4% with 0.7% performance degradation versus FedFT, while FedPETuning also improves resistance to gradient inversion attacks.
Takeaways & Limitations
FedPETuning provides a benchmark and practical framework for studying efficient, privacy-preserving PLM adaptation under federated resource constraints.
Takeaways & Limitations
The paper does not validate FedPETuning on large-scale models such as T5, LLaMA, or Vicuna, whose deployment may exceed client resources.
Abstract
from arXiv · showhide
With increasing privacy concerns on data, recent studies have made significant progress using federated learning (FL) on privacy-sensitive natural language processing (NLP) tasks. Much literature suggests fully fine-tuning pre-trained language models (PLMs) in the FL paradigm can mitigate the data heterogeneity problem and close the performance gap with centralized training. However, large PLMs bring the curse of prohibitive communication overhead and local model adaptation costs for the FL system. To this end, we introduce various parameter-efficient tuning (PETuning) methods into federated learning. Specifically, we provide a holistic empirical study of representative PLMs tuning methods in FL. The experimental results cover the analysis of data heterogeneity levels, data scales, and different FL scenarios. Overall communication overhead can be significantly reduced by locally tuning and globally aggregating lightweight model parameters while maintaining acceptable performance in various FL settings. To facilitate the research of PETuning in FL, we also develop a federated tuning framework FedPETuning, which allows practitioners to exploit different PETuning methods under the FL training paradigm conveniently. The source code is available at \url{https://github.com/iezhuozhuo/FedETuning/tree/deltaTuning}.
1 Introduction
The paper studies parameter-efficient tuning for federated PLM training, addressing the communication, computation, storage, and privacy constraints that make full fine-tuning costly.
- 1 Introduction: Full PLM fine-tuning in FL creates excessive communication overhead and local computation, memory, and storage costs.Clients may need to exchange millions or billions of parameters while adapting task-specific model instances.
- 1 Introduction: PETuning freezes most PLM parameters and updates only lightweight or partial parameters, making it suitable for resource-constrained FL clients.The paper considers adapter tuning, prefix tuning, LoRA, and BitFit as representative methods.
- 1 Introduction: FedPETuning reduces communication by 97.4% with only 0.7% performance degradation compared with FedFT.The framework exchanges lightweight tuned parameters rather than full PLM updates.
- 1 Introduction: FedPETuning benchmarks PETuning methods across privacy attacks, performance, and resource costs in federated settings.Its contributions include a holistic benchmark and baselines for efficient PLM tuning under FL.
- 1 Introduction: FedPETuning reports stronger resistance to gradient inversion, reducing recovered-word precision by an average of 40.7% compared with FedFT.This privacy result is reported alongside the communication and performance comparison.
2 Related Work
Prior work shows that PLMs can reduce FL’s non-IID accuracy gap, but comprehensive evidence on PETuning under federated resource constraints remains limited.
- 2 Related Work: Earlier FL research addressed non-IID effects through data, model-update, and training optimizations, while PLMs were shown to narrow the centralized-training accuracy gap.These studies did not fully resolve the resource costs of large PLMs.
- 2 Related Work: Large PLMs still impose substantial communication, computation, and storage burdens on federated clients.The related work identifies these costs as barriers to practical federated training.
- 2 Related Work: PETuning reduces FL communication by freezing most PLM parameters and updating only lightweight parameters or a parameter subset.This principle has also been explored in computer vision through FedPEFT.
- 2 Related Work: NLP studies found acceptable federated performance with prompt tuning and preserved accuracy and privacy with low-rank adaptation.However, these studies did not provide a comprehensive comparison of PETuning methods for PLMs in FL.
- 2 Related Work: FedPETuning addresses the gap by providing a comprehensive FL benchmark and releasing code and data for further research.The stated aim is to support exploration of efficient federated NLP.
3 Federated Parameter-Efficient Tuning
FedPETuning combines lightweight PLM adaptation with federated aggregation: clients train efficient parameters locally while the server exchanges and aggregates only those parameters.
- 3.1 PETuning Methods: PETuning divides into addition-based, specification-based, and reparameterization-based methods, with most PLM parameters frozen and few parameters updated.The taxonomy includes adapters and prompts, BitFit, and low-rank reparameterization such as LoRA.
- 3.1 PETuning Methods: Adapter tuning updates only adapter layers, representing about 0.5% to 8% of the full model parameters.The adapter layers use down-projection and up-projection components while the PLM remains frozen.
- 3.1 PETuning Methods: Prompt tuning attaches trainable prompt vectors to the input without modifying the model architecture, using at most 10% of PLM parameters.The paper uses prefix tuning as its representative prompt-tuning method.
- 3.1 PETuning Methods: LoRA reparameterizes weight updates as low-rank matrices B and A with rank r much smaller than the original dimensions, using less than 1% trainable parameters.This reduces the number of optimized parameters while targeting the weight-update subspace.
- 3.2 FedPETuning: FedPETuning initializes the backbone PLM and efficient parameters, sends efficient parameters to sampled clients, aggregates their updates, and repeats local tuning and global aggregation.Clients assemble the full model locally with frozen PLM parameters and private data, then return only updated efficient parameters.
4 Experiments
Experiments show that FedPETuning substantially reduces federated resource costs while generally preserving acceptable performance, but it is more vulnerable to severe data heterogeneity and small-data instability than FedFT.
- 4.2 Privacy-Preserving Results: FedPETuning defends against data reconstruction attacks because attackers receive only a fraction of the model parameters.Among PETuning methods, FedAP is more vulnerable, possibly because adapter layers better retain input information.
- 4.3 Performance Comparison: FedPETuning maintains more than 95% of FedFT’s performance while substantially reducing communication overhead.FedAP reduces communication by 60 times with only 0.7% performance degradation compared with FedFT.
- 4.3 Performance Comparison: In centralized experiments, CenAP outperforms CenFT on five of six datasets, while CenAP and CenLR achieve better average performance.All tuning methods decline under federated training, with FedPETuning showing a pronounced drop in heterogeneous FL; the authors suggest FL-tailored PETuning as future work.
- 4.3 Performance Comparison: FedPETuning reduces communication overhead enough to make federated training more practical, especially under communication constraints.Communication budgets rank as FedFT ≫ FedPF > FedAP > FedLR > FedBF, and higher overhead increases upload and download time.
- 4.3 Performance Comparison: FedPETuning lowers local storage costs by 12~190 times and lets clients share a PLM across tasks while storing only task-specific parameters.
- 4.4 Resource Costs: FedPETuning degrades more sharply than FedFT as data heterogeneity increases from α = 1.0 to α = 0.1.Performance changes little from α = 1.0 to α = 10.0 because those settings have similar data heterogeneity.
- 4.4 Resource Costs: Most tuning methods benefit from more local epochs, but all methods become more unstable on small datasets.FedLR’s standard-deviation performance exceeds 4.0% on RTE and MRPC, compared with no more than 0.5% on other datasets; increasing training steps does not resolve this instability in FL.
5 Conclusion
The paper’s experiments show that FedPETuning reduces communication and storage costs while maintaining acceptable performance and providing strong privacy-preserving capacity across FL settings.
- FedPETuning achieves acceptable performance while reducing communication overhead and local storage cost, and provides strong privacy-preserving capacity across FL settings.
Limitation
The paper does not validate FedPETuning on large-scale PLMs, although parameter efficiency may become more valuable as models grow.
- FedPETuning is not validated on large-scale models such as T5, LLaMA, or Vicuna.The authors note that deploying such models is challenging because FL clients have limited computational resources.
- The study identifies a need for FL-friendly PETuning methods as modern PLMs continue to increase in size.
A Non-IID Partitionings Results
The non-IID partitioning analysis finds that clients with α=0.1 have substantially more separated data distributions, whereas α=1.0 and α=10.0 are more uniform.
- α=0.1 produces large distances between clients’ data distributions, while α=1.0 and α=10.0 produce more uniform distributions.The larger distribution gap at α=0.1 affects performance.
B Extra Results
Extra experiments show that PETuning can achieve comparable accuracy with much lower communication, while performance depends on non-IID severity and dataset size when varying training epochs.
- Communication Analysis: PETuning methods reduce communication by several orders of magnitude while providing comparable performance; most achieve acceptable accuracy across tasks.FedPETuning’s acceptable-accuracy threshold is defined relative to fine-tuning performance in the supplied figure caption.
- Non-IID Analysis: All methods perform better at α=1.0 than α=0.1, while changing α from 1.0 to 10.0 has little impact in most circumstances.
- Local Training Epoch Analysis: More local epochs benefit relatively large datasets but can slightly reduce accuracy on smaller datasets because of overfitting.