Source-linked AI summary
UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning
Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Wen-tau Yih, Madian Khabsa
TL;DR
PELT methods reduce trainable parameters but differ substantially across tasks, making method selection difficult. UniPELT combines multiple PELT methods as gated submodules that adapt to the data or task setup. On GLUE, it improves over incorporated methods, outperforms fine-tuning under different setups, and generally exceeds the per-task best-submodule upper bound.
Problem
Different PELT methods perform differently on the same task, making it difficult to select an appropriate method as the number of methods and tasks grows.
Method
UniPELT incorporates different PELT methods as submodules and uses gating to activate the combinations best suited to the current data or task setup.
Results
UniPELT consistently improves its best incorporated PELT submodule by 1~4 points, outperforms fine-tuning under different setups, and generally surpasses the per-task upper bound.
Takeaways & Limitations
A mixture of multiple PELT methods involving different parts of the PLM may be more effective and robust than single methods.
Takeaways & Limitations
2BitFit is excluded from UniPELT because it typically performed worst in preliminary experiments.
Abstract
from arXiv · showhide
Recent parameter-efficient language model tuning (PELT) methods manage to match the performance of fine-tuning with much fewer trainable parameters and perform especially well when training data is limited. However, different PELT methods may perform rather differently on the same task, making it nontrivial to select the most appropriate method for a specific task, especially considering the fast-growing number of new PELT methods and tasks. In light of model diversity and the difficulty of model selection, we propose a unified framework, UniPELT, which incorporates different PELT methods as submodules and learns to activate the ones that best suit the current data or task setup via gating mechanism. On the GLUE benchmark, UniPELT consistently achieves 1~4% gains compared to the best individual PELT method that it incorporates and even outperforms fine-tuning under different setups. Moreover, UniPELT generally surpasses the upper bound that takes the best performance of all its submodules used individually on each task, indicating that a mixture of multiple PELT methods may be inherently more effective than single methods.
1 Introduction
PELT reduces trainable parameters but methods vary substantially across tasks, complicating selection. UniPELT unifies multiple methods with gating and achieves stronger, robust performance across GLUE setups.
- <1% of PLM parameters are task-specific in many recent PELT approaches, while conventional fine-tuning requires separate modified model replicas per task.
- Different PELT methods perform rather differently on the same task, making task-specific method selection nontrivial as methods and tasks multiply.
- UniPELT incorporates PELT methods as submodules and learns through gating to activate combinations suited to the current data or task setup.
- UniPELT combines adapter, prefix-tuning, LoRA, and BitFit as representative PELT methods for comparative analysis.
- 1~4 points: UniPELT consistently improves over its best incorporated submodule on GLUE and also outperforms fine-tuning under different setups.Experiments cover 32 setups and more than 1,000 runs.
- UniPELT generally surpasses the per-task upper bound formed by the best individually used submodule, while retaining negligible losses in model efficiency.
2 Preliminaries
PELT methods preserve most pretrained parameters while introducing small task-specific components or tuning selected existing parameters. The section describes adapter, prefix-tuning, LoRA, and BitFit as representative approaches.
- BitFit tunes only PLM bias terms and can approach fine-tuning performance on certain tasks with limited training data.
- Adapter: Adapter adds a trainable bottleneck layer after each Transformer feedforward network, shrinking and recovering token hidden-state dimensions.
- Adapter: Adapter uses down and up projections with a nonlinear activation, while some variants also fine-tune layer-normalization parameters and the prediction head.
- Prefix-tuning: Prefix-tuning prepends trainable vectors to each Transformer layer’s attention keys and values, which original tokens can attend to as virtual tokens.
- Prefix-tuning: After prefix reparameterization, the feedforward network can be discarded, leaving 2Nlayer trainable prefix matrices of size RDhidden×L.
- LoRA: LoRA adds trainable low-rank matrices to the query and value projections and scales task-specific differences with a fixed scalar α.
3 Unifying PELT Methods
UniPELT unifies multiple PELT methods as gated submodules, dynamically weighting them for each task or data setup. This avoids exhaustive method selection and can benefit from combining methods that modify different parts of the language model.
- UniPELT incorporates multiple PELT methods as submodules and dynamically activates or downweights them for different task and data setups.The framework uses trainable gates for each submodule in every Transformer layer.
- Different PELT methods can be combined because they modify different parts of the PLM architecture, such as prefix-tuning before attention and adapters after feedforward layers.The paper presents this architectural separation as making combination feasible without directly interfering with one another.
- UniPELT generally exceeds the best individual submodule performance on each task, suggesting compounding benefits from jointly modifying different PLM components.The authors attribute improvement over single methods both to selective activation and to possible compounding effects across multiple methods.
- Gates estimate submodule importance and control each method’s contribution, allowing adapter, prefix-tuning, and LoRA to be selectively weighted.Adapter and prefix-tuning gates scale their outputs or vectors, while LoRA uses a learnable per-layer scaling factor.
- Naively combining PELT methods can produce mixed or worse performance, motivating UniPELT’s learned gating rather than an ungated hybrid.The paper notes that submodule interdependencies and multilayer compounding effects make their interplay complicated.
4 Experiments
Experiments on GLUE examine individual PELT methods and UniPELT across low- and high-resource setups. UniPELT generally provides stronger, more robust performance while retaining parameter efficiency.
- Experiment Setup: The study evaluates 8 GLUE tasks across four data sizes, seven methods, and five runs per setup, totaling more than 1,000 runs.The low-resource settings use 100, 500, and 1,000 training samples; high-resource experiments use all training samples.
- Individual PELT Methods: Performance differences between PELT methods can reach 5~9 points on individual tasks even when average performance is similar.The reported examples are STS-B and MNLI with K = 500.
- Individual PELT Methods: Adapter performance is generally stable but slightly below fine-tuning, while increasing its bottleneck to Dmid = 256 improves CoLA performance substantially.Dmid = 256 uses 5.3× the trainable parameters of the original Dmid = 48 adapter.
- Individual PELT Methods: Prefix-tuning is weak with K = {100, 500}, reaches parity with fine-tuning at K = 1000, and can remain unstable in low-resource settings despite more trainable parameters.With prefix length L = 50, performance improved on three of four previously weak tasks but worsened significantly on STS-B.
- Individual PELT Methods: LoRA is sensitive to its scaling factor, with no single value working well across multiple task and data setups.The finding motivates more fine-grained and dynamic gating in UniPELT.
- UniPELT: UniPELT improves over the best individual incorporated submodule by 1~4% in low-resource settings and performs best or second best on most tasks without being worst.UniPELT performs best or second best on 7/6/7 of 8 tasks at 100/500/1,000 samples, respectively, and can outperform fine-tuning on difficult low-resource tasks.
- UniPELT: UniPELT matches or exceeds the submodule upper bound, while high-resource gains are smaller and ungated combination performs worse than gated UniPELT.In high-resource experiments, UniPELT is best or second best on all eight tasks; UNIPELT-NoGate averages -0.89 versus UNIPELT.
- UniPELT: UniPELT retains parameter efficiency with 0.99%~1.26% of fine-tuning’s trainable parameters.The framework can also substitute more parameter-efficient variants of its component methods.
5 Related Work
Related work frames parameter-efficient tuning as a response to the cost of adapting large PLMs. It distinguishes approaches that train subsets of existing parameters from approaches that introduce new trainable parameters.
- Parameter-Efficient Tuning of PLMs: Parameter-efficient tuning seeks to adapt large PLMs without training and storing full task-specific copies.The related-work discussion presents this efficiency goal as increasingly critical as PLMs grow larger.
- Parameter-Efficient Tuning of PLMs: Existing PELT methods can be broadly divided by whether they introduce new trainable parameters.The passage contrasts training a subset of PLM parameters with adding new trainable components.
- Parameter-Efficient Tuning of PLMs: Unlike typical homogeneous or identical MoE experts, UniPELT uses diverse PELT methods as its experts.This distinction motivates viewing UniPELT as a mixture of heterogeneous adaptation mechanisms.
6 Conclusion
The paper studies representative PELT methods and introduces UniPELT, which combines them as task- or data-adaptive submodules. UniPELT consistently outperforms fine-tuning and its individual submodules, often exceeding the per-task upper bound from selecting one submodule.
- UniPELT incorporates different PELT methods as submodules and learns to activate the most appropriate ones for each task or data setup.
- UniPELT consistently outperforms conventional fine-tuning and its incorporated submodules across different setups.
- UniPELT generally surpasses the upper bound obtained by using the best individual submodule on each task.
- The findings suggest that mixing PELT methods affecting different parts of the PLM may improve effectiveness and robustness.
A Prefix-tuning vs. Prompt-based Fine-tuning
Prefix-tuning differs from prompt-based fine-tuning in parameter efficiency, where prompts are inserted, and prompt representation. Prefix-tuning updates only continuous prefix parameters added throughout the Transformer, whereas prompt-based fine-tuning updates the full model using input prompts.
- Prefix-tuning updates only the prefix matrix, while prompt-based fine-tuning updates all model parameters.
- Prompt-based fine-tuning uses prompts only in the model input, whereas prefix-tuning adds vectors to every Transformer layer.
- Prompt-based fine-tuning typically uses designed natural-language prompts, while prefix-tuning uses continuous prompts represented as virtual tokens.
B Implementation Details
The experiments construct controlled training and development splits using fixed sample counts and multiple data seeds, then standardize training settings across methods. Learning rates are selected separately for methods whose NLU performance had not previously been evaluated.
- The training set uses the first K shuffled samples, while the next 1,000 samples form the development set across five data seeds.
- Experiments use input length 128, batch size 16, 50 epochs, and early stopping after 10 non-increasing epochs.
- Learning rates are 2e-5 for fine-tuning, 1e-4 for adapters, 2e-4 for prefix-tuning, and 5e-4 for UniPELT.
C BART Results
A preliminary BARTlarge experiment evaluates average GLUE performance with 1,000 training examples over five runs. UniPELT achieves the best performance and shows notably smaller variance, consistent with results using BERTbase.
- With BARTlarge and 1,000 training examples, results are averaged over five runs by varying model or data seeds.
- UniPELT achieves the best average performance on the GLUE development set with BARTlarge, excluding QQP.
- BARTlarge results are largely consistent with BERTbase results, while UniPELT has notably smaller variance.
D Detailed Performance
Table 6 reports detailed GLUE development- and test-set results across training regimes of K = {100, 500, 1000} samples. The observations and findings are largely consistent across the two evaluation splits.
- D Detailed Performance: The observations and findings are largely consistent between the development and test evaluation splits.
- D Detailed Performance: The evaluation metrics are Matthew’s Correlation for CoLA, F1 for MRPC and QQP, Spearman’s correlation for STS-B, and accuracy for the remaining tasks.MNLI is evaluated on the matched dataset.