PaperlayerView exact revision on arXiv

Source-linked AI summary

Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models

Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Hai-Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juanzi Li, Maosong Sun

arXiv:2203.06904v2cs.CLcs.AIcs.LG

TL;DR

Large-scale PLM fine-tuning is costly because it updates all parameters and requires separate task-specific instances. This paper surveys delta tuning, unifies it into three method categories, develops optimization and optimal-control frameworks, and evaluates representative methods across more than 100 NLP tasks. The study finds that no single method consistently dominates, while combinations, larger backbones, and task similarity support useful performance, convergence, or transfer patterns.

  • Problem

    Fine-tuning large PLMs requires updating all parameters and storing separate task-specific instances, creating prohibitive computation and memory costs.

  • Method

    The paper formally defines delta tuning, surveys methods in three categories, develops optimization and optimal-control frameworks, and evaluates representative approaches across more than 100 NLP tasks.

  • Results

    Across more than 100 NLP tasks, no single delta tuning algorithm predominantly outperforms the others; combining methods is usually more effective, scale improves performance and convergence, and similar tasks show transferability.

  • Takeaways & Limitations

    Delta tuning provides a parameter-efficient route to adapting large PLMs while also offering theoretical perspectives on model adaptation.

  • Takeaways & Limitations

    Soft prompt tuning is difficult to optimize and generally becomes more challenging as data volume and model size decrease, with slower convergence than full-parameter fine-tuning and other delta tuning methods.

Abstract

from arXiv · show

Despite the success, the process of fine-tuning large-scale PLMs brings prohibitive adaptation costs. In fact, fine-tuning all the parameters of a colossal model and retaining separate instances for different tasks are practically infeasible. This necessitates a new branch of research focusing on the parameter-efficient adaptation of PLMs, dubbed as delta tuning in this paper. In contrast with the standard fine-tuning, delta tuning only fine-tunes a small portion of the model parameters while keeping the rest untouched, largely reducing both the computation and storage costs. Recent studies have demonstrated that a series of delta tuning methods with distinct tuned parameter selection could achieve performance on a par with full-parameter fine-tuning, suggesting a new promising way of stimulating large-scale PLMs. In this paper, we first formally describe the problem of delta tuning and then comprehensively review recent delta tuning approaches. We also propose a unified categorization criterion that divide existing delta tuning methods into three groups: addition-based, specification-based, and reparameterization-based methods. Though initially proposed as an efficient method to steer large models, we believe that some of the fascinating evidence discovered along with delta tuning could help further reveal the mechanisms of PLMs and even deep neural networks. To this end, we discuss the theoretical principles underlying the effectiveness of delta tuning and propose frameworks to interpret delta tuning from the perspective of optimization and optimal control, respectively. Furthermore, we provide a holistic empirical study of representative methods, where results on over 100 NLP tasks demonstrate a comprehensive performance comparison of different approaches. The experimental results also cover the analysis of combinatorial, scaling and transferable properties of delta tuning.

1 Introduction

Large pre-trained language models make conventional fine-tuning costly because adaptation updates all parameters and stores separate task-specific instances. The paper introduces delta tuning as a parameter-efficient alternative, categorizes its methods, develops theoretical perspectives, and evaluates them across more than 100 NLP tasks.

  • Motivation: Conventional fine-tuning updates all model parameters and produces separate instances for different tasks, making large PLM adaptation costly in computation and memory.The paper motivates delta tuning partly through the practical difficulty of deploying and experimentally validating large PLMs.
  • Delta tuning: Delta tuning updates only a small portion of parameters while leaving the remainder of the pre-trained model untouched.The paper describes the tuned parameters as a task-specific “delta” and links this design to parameter-efficient adaptation.
  • Categorization: The paper organizes delta tuning into addition-based, specification-based, and reparameterization-based methods according to their operations on delta parameters.This categorization provides a unified framework for reviewing recent approaches.
  • Theory: The paper develops theoretical frameworks that interpret delta tuning through subspace optimization and optimal control.The optimization perspective connects delta tuning to intrinsic low-dimensional structure, while the optimal-control perspective treats it as seeking controllers for PLMs.
  • Experiments: Experiments across over 100 NLP tasks examine performance, combinability, scale, and transferability of representative delta tuning methods.The empirical study is designed as an extensive and systematic comparison of these properties.
  • Findings: Combining methods is usually more effective than using one method, scaling improves performance and convergence across methods, and related tasks show non-trivial transferability.No single representative algorithm predominantly outperforms the others, and the best combination depends on the PLM backbone, task, and data scale.

2 Preliminaries

Transformers underpin mainstream pre-trained language models, with encoder, decoder, and sequence-to-sequence variants differing in architecture and training objectives. Transformer layers combine attention, feed-forward computation, residual connections, and normalization, while delta tuning can target positions throughout the model.

  • 2.1 Transformer: Transformer layers stack attention and feed-forward components with residual connections and layer normalization.Attention uses multi-head self-attention, and feed-forward layers apply two linear transformations with a nonlinear activation.
  • 2.1 Transformer: Self-attention derives query, key, and value representations from the same feature matrix, while multi-head attention uses multiple parameterized projections.The outputs of multiple heads are concatenated and projected into a new feature space.
  • 2.1 Transformer: Decoder blocks mask future positions and add cross-attention over encoder outputs for sequence generation.Cross-attention uses decoder representations for queries and encoder representations for keys and values.
  • 2.1 Transformer: Delta tuning methods can generally be applied at any position in a Transformer model.The paper presents this flexibility in its illustration of a Transformer block.
  • 2.2 Pre-trained Language Models: Mainstream PLMs use encoder-only, decoder-only, or complete encoder-decoder Transformer architectures.Figure 3 organizes PLMs by these different modeling strategies.
  • 2.2 Pre-trained Language Models: Encoder models such as BERT and RoBERTa use masked-language objectives, decoder models such as GPT use autoregressive prediction, and T5 and BART use sequence-to-sequence corruption.Bidirectional models support contextual understanding, autoregressive models support generation, and sequence-to-sequence models address both understanding and generation.

3 Delta Tuning

Delta tuning adapts pre-trained language models by modifying only a small parameter subset, organizing methods into addition-based, specification-based, and reparameterization-based approaches. Representative methods add modules or prompts, selectively train existing parameters, or reparameterize them efficiently, often reducing adaptation costs while retaining strong performance.

  • Definition: Delta tuning defines adaptation through a parameter change ΔΘ, with far fewer involved parameters than vanilla fine-tuning.Vanilla fine-tuning updates all model parameters, whereas delta tuning modifies only a small subset.
  • Categorization: The paper categorizes delta tuning as addition-based, specification-based, or reparameterization-based according to operations on delta parameters.Addition introduces parameters, specification selects existing trainable parameters, and reparameterization transforms existing parameters into an efficient form.
  • Addition-based Methods: Addition-based methods inject lightweight modules or prompts while keeping the original model parameters frozen.Adapters add down- and up-projection modules, while prompt methods add context such as prefixes or soft prompts.
  • Addition-based Methods: Adapter tuning can involve 0.5%∼8% of whole-model parameters, train 60% faster than vanilla fine-tuning, and incur only 4%-6% slower inference.Adapter methods also support multiple task-specific instances and have reported robustness advantages in few-shot, cross-lingual, and adversarial settings.
  • Addition-based Methods: Adapters encapsulate task information in lightweight modules, but practical sharing and reuse remain substantially difficult.AdapterHub is presented as an initial platform for deploying adapters in Transformer models.
  • Prompt-based Tuning: Prompt tuning narrows its performance gap with full fine-tuning as model size increases, reaching comparable SuperGlue performance on an 11-billion-parameter T5-XXL.However, prompt-based methods can be difficult to optimize and converge more slowly than full fine-tuning and other delta tuning methods.

4 Theoretical Perspectives of Delta Tuning

The paper interprets delta tuning through optimization and optimal-control frameworks, explaining how low-dimensional parameter updates approximate or steer full-model adaptation. These perspectives identify conditions for effectiveness and suggest principled designs for new methods.

  • Optimization Perspective: Delta tuning is analyzed as optimizing a small parameter subset to match full fine-tuning while reducing memory usage.The optimization perspective studies delta tuning under a low-dimensional assumption.
  • Optimization Perspective: The delta objective ˜F(θ, δ) separates the original model parameters θ from the tuned parameters δ, with initialization at the pre-trained model and δ's starting value.When delta tuning is disabled, the modified objective reduces to the original objective F.
  • Optimization Perspective: Effective delta tuning relies on low-dimensional structure, either updating solutions in a smaller subspace or approximating objectives within a smaller functional subspace.These approaches tend to work when initialization is near a local minimizer, where fewer search directions matter or simpler local approximations suffice.
  • Optimization Perspective: Low-dimensional solution-space representations can recover the exact fine-tuning solution when approximation error ϵ is zero.With nonzero error, the discrepancy depends on representation approximation, objective conditioning, and training stability.
  • Optimization Perspective: Practical constructions of the functional approximation include freezing selected network parts, injecting low-rank modules, and augmenting the feature space with prompts or prefixes.BitFit, Adapter, prompt tuning, and prefix tuning exemplify these construction choices, which differ in representation capacity and trainable components.
  • Optimal Control Perspective: The optimal-control framework unifies different delta tuning scenarios by treating delta parameters as control variables and connecting propagation with co-state calculations.The analysis frames terminal and running losses as parts of a discrete-time control problem and motivates controller-inspired, potentially interpretable designs.

5 Comparisons and Experimental Discoveries

The paper compares representative delta tuning methods with fine-tuning across more than 100 NLP tasks, examining performance, convergence, efficiency, combinations, scale, and transferability. Results show that method structure, model scale, and task compatibility substantially affect outcomes.

  • Combinations: Combining delta methods generally improves downstream performance, but the best combination varies across PLM backbones and experimental settings.The study tests simultaneous and sequential combinations, including low-resource settings and manual templates.
  • Experimental Setting: The study evaluates prompt tuning, prefix-tuning, LoRA, adapter, and fine-tuning across over 100 diverse NLP tasks.Tasks include classification, question answering, conditional generation, and dialogue.
  • Performance: FT ranks first on average, followed by LR, AP, PF, and PT, although PF, LR, and AP can outperform FT on individual tasks.Delta methods narrow the average performance gap with fine-tuning despite tuning far fewer parameters.
  • Convergence: Convergence ranks as FT > AP ≈ LR > PF, while prompt tuning lags far behind in both convergence and performance.The experiments also report that performance and convergence depend more on method structure than on the number of tunable parameters within a broad range.
  • Efficiency: Delta tuning reduces GPU-memory requirements and training time by avoiding gradient computation for most parameters, although it can require more steps to converge.The convergence penalty becomes less significant as model scale increases.
  • Scale: Larger PLM backbones improve both performance and convergence for delta tuning, while prompt tuning performs especially poorly on small T5 models.Last-layer tuning and selective-module tuning show promising results as additional scale-sensitive approaches.
  • Transferability: Transfer works best among tasks of the same type and generally performs poorly across different task types.The study measures relative zero-shot transfer performance between source and target tasks.

6 Applications

The paper presents delta tuning as useful for large-model applications because lightweight parameter updates reduce training, communication, and storage demands. It highlights applications in fast training, shareable checkpoints, multi-task learning, catastrophic-forgetting mitigation, and model-as-a-service settings.

  • Scale: Across T5 scales, larger backbones improve delta-tuning performance and convergence, while prompt tuning is especially weak on T5SMALL and T5BASE.Figure 11 compares MNLI, QNLI, and SST-2 across model scales.
  • Fast Training and Shareable Checkpoints: Delta tuning can speed training by reducing backward-propagation computations for tunable parameters, despite sometimes slower convergence.Time consumption is compared with fine-tuning, BitFit, adapter, and prompt tuning under different input lengths.
  • Fast Training and Shareable Checkpoints: Lightweight delta checkpoints require less storage and are easier to share, supporting experiments with extremely large models.The paper mentions communities of shareable checkpoints such as AdapterHub.
  • Multi-task Learning: Adapter-based tuning can place multiple adapter instances on one pretrained model, supporting multi-task learning.The paper describes adapter modules inserted in parallel with the self-attention module.
  • Transferability: Zero-shot transfer is generally stronger within the same task category than across different task types.Figure 12 reports relative target-task performance for transfers between source and target tasks.
  • Catastrophic Forgetting Mitigation: Because delta tuning updates minimal parameters, it is presented as a potential way to mitigate catastrophic forgetting during sequential multi-task training.The motivation is that updating all PLM parameters can cause forgetting without regularization.
  • Language Model as Services and In-batch Parallel Computing: Delta tuning is positioned as competitive for language-model services because users can adapt released models without editing source code or exchanging full model parameters.The paper connects this setting to reduced communication costs between users and service providers.

7 Conclusion

The paper surveys delta tuning, unifies methods into three categories, develops optimization and optimal-control perspectives, and evaluates representative methods across more than 100 NLP tasks. It concludes that delta tuning has substantial potential for large PLMs while leaving method mechanisms and setting-dependent choices for future study.

  • Contributions: The paper categorizes delta tuning methods as addition-based, specification-based, and reparameterization-based.It first formalizes the delta tuning problem and then uses the categories to organize the survey.
  • Theory: It proposes theoretical frameworks based on optimization and optimal control to analyze delta tuning and model adaptation.The paper presents these frameworks as ways to investigate mechanisms underlying PLMs and delta tuning.
  • Empirical Study: Experiments across 100+ NLP tasks examine performance, combinations, scale effects, and transferability of delta tuning methods.The empirical study is designed as a broad comparison and exploration of delta tuning properties.
  • Conclusion: The paper concludes that delta tuning shows significant potential for stimulating extremely large PLMs.It frames further theoretical and empirical study as an open direction.

Broader Impacts

Delta tuning can reduce the resource burden of adapting large language models, but its effects on model hazards remain uncertain. The paper highlights both potential bias-correction uses and risks of misuse.

  • Potential Hazards: Large pre-trained language models can exhibit biases and generate attacks, profanities, and insults because their training data may contain inherent bias.The paper identifies corpus normalization and representation or output modification as possible mitigation directions.
  • Potential Hazards: There is no clear evidence that delta tuning either mitigates or exacerbates potential hazards inherited from the base language model.The paper notes that delta tuning may retain the base model’s risks.
  • Potential Benefits and Risks: Delta tuning could potentially correct model bias and modify factual errors by changing only a small number of parameters.The paper also warns that similar strategies could further contaminate models and produce undesirable predictions.
  • Environmental Issues: Delta tuning substantially reduces computational cost, although methods tend to require more time to converge; this becomes less significant as model scale increases.The analysis examines memory and time consumption alongside convergence behavior.

Contributions

The authors divide responsibilities across the paper’s conceptual, theoretical, and empirical components. These roles include initiating the study, developing technical sections, verifying derivations, and conducting experiments.

  • Research Organization: Ning Ding, Yujia Qin, and Zhiyuan Liu initiated and organized the research, while Ning Ding drafted the abstract, introduction, and delta tuning section.The term “delta tuning” was coined by Shengding Hu and recognized by the other authors.
  • Technical Development: Multiple authors contributed technical content to the addition-based and reparameterization-based delta tuning sections.The listed contributors were Shengding Hu, Xiaozhi Wang, Yujia Qin, Weilin Zhao, Ning Ding, and Yulin Chen.
  • Empirical Study: Yujia Qin led the empirical study, while several authors drafted the experiments section and conducted overall-performance and combination experiments.Yusheng Su and Chi-Min Chan handled transferability and scaling experiments.
  • Verification and Advising: Ding verified the formula derivation, and senior advisors suggested the theoretical and empirical study and participated in the project.The contribution statements also identify authors responsible for applications and experiments.

A Implementation Details

The appendix describes dataset splitting and optimization procedures used to evaluate several delta tuning methods. Evaluation uses held-out development and test sets with early stopping and checkpoint selection.

  • Dataset Preparation: Datasets without public test sets are split by dividing development data evenly, while datasets lacking both development and test sets use an 8:1:1 training split.These procedures create new development and test sets for evaluation.
  • Optimization: PF, LR, AP, and FT use AdamW, up to 20,000 training steps with early stopping, and checkpoints saved every 100 steps.The best development-set checkpoint is subsequently evaluated on the test set.

A.2 Combinations of Delta Tuning Methods

The appendix specifies settings for individual and combined delta tuning methods, including tunable-token, adapter, and bias configurations. It also describes a regression-to-classification conversion for STS-B and identifies the scaling subsection.

  • Method Configurations: Prompt tuning prepends 10 tunable virtual tokens, adapters use reduction factor 16, and BitFit optimizes all bias components in the pre-trained models.These settings define the method configurations used in the combinations experiments.
  • STS-B Conversion: For STS-B, the regression output is reformulated as a weighted combination of two class probabilities, with weights given by the original lower and upper bounds.Optimization minimizes KL-divergence between the upper-class prediction probability and the normalized ground-truth value.
  • Sequential Combination: Sequential combinations use batch size 64 with learning rates of 1 × 10^-2 for prompt tuning, 1 × 10^-4 for BitFit, and 1 × 10^-5 for adapters.The appendix separately labels a subsection on the power of scale for delta tuning.
  • Learning-Rate Settings: Table 8 reports learning-rate settings for RoBERTaLARGE on 16-shot GLUE datasets.The supplied table passage provides the table’s topic but no numerical entries.

A.3 The Power of Scale for Delta Tuning

This table reports the percentage of parameters tuned during training, relative to all parameters in each PLM.

  • A.3 The Power of Scale for Delta Tuning: The table measures the share of PLM parameters participating in optimization during training.The percentage is defined as tuned parameters divided by all parameters in the PLM.
  • A.3 The Power of Scale for Delta Tuning: The reported quantity is a percentage rather than an absolute parameter count.
  • A.3 The Power of Scale for Delta Tuning: The measure is intended to characterize how many model parameters are tuned during delta-tuning experiments.

A.4 Task-level Transferability Evaluation

The transferability evaluation spans 12 tasks across five task types and reports relative performance using task-appropriate metrics.

  • Task coverage: The transferability experiments use 12 tasks spanning five different task types.
  • Sentiment Analysis: Sentiment analysis includes SST-2, Amazon/Polarity, and Rotten Tomatoes.
  • Inference and paraphrase: Natural language inference includes MNLI, SICK, and SciTail, while paraphrase identification includes QQP and MRPC.
  • Question answering and summarization: Question answering is evaluated with MathQA and AQUA-RAT, and summarization with Multi-News and SAMSum.
  • Evaluation Metrics: Accuracy is used for sentiment, inference, and paraphrase tasks, while question answering uses F1 and summarization uses ROUGE-L.Results are reported as relative performance: transferring zero-shot performance divided by original performance, in percent.

B Tasks Evaluated in Experiments

The experiments cover a broad set of NLP tasks organized by task ontology, including classification, question answering, generation, linguistic phenomena, and other specialized evaluations.

  • Task organization: The experiment task inventory is organized according to a task ontology.
  • Question answering: Question-answering and knowledge tasks include FreebaseQA, LAMA variants, NumerSense, SearchQA, WebQuestions, and SuperGLUE Record.
  • Reasoning: The inventory includes commonsense and reasoning benchmarks such as CosmosQA, DREAM, HellaSwag, OpenBookQA, QASC, Quarel, Quartz, RACE, COPA, SWAG, and WinoGrande.
  • Generation and linguistic phenomena: Generation and linguistic evaluations include ELI5, Multi-News, SAMSum, XSum, Spider, WikiBio, WikiSplit, and BLiMP phenomena.
  • Other tasks: Additional evaluations cover explanation generation, dosage and effect extraction, entity linking, acronym identification, ASLG, domain crawling, and ProtoQA.
Loading 2203.06904v2…