PaperlayerView exact revision on arXiv

Source-linked AI summary

Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment

Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, Fu Lee Wang

arXiv:2312.12148v1cs.CL

TL;DR

The paper addresses the computational and memory costs of adapting increasingly large PLMs and LLMs with full fine-tuning. It systematically reviews and classifies PEFT methods, discusses applications, and evaluates representative approaches. The experiments report improved parameter efficiency, generally comparable or better performance, and lower memory usage, with QLoRA producing especially large memory reductions.

  • Problem

    Growing PLMs and LLMs make task-specific full fine-tuning computationally and memory intensive, creating adaptation challenges in resource-limited environments.

  • Method

    The paper systematically reviews PEFT methods, organizes them into five categories, discusses applications and future directions, and experimentally evaluates representative methods.

  • Results

    Most evaluated PEFT methods improve parameter efficiency, achieve comparable or better performance than full fine-tuning, and reduce memory usage; QLoRA drastically lowers computational memory requirements.

  • Takeaways & Limitations

    PEFT provides a practical way to adapt PLMs and LLMs while using fewer trainable parameters and less memory than full fine-tuning.

Abstract

from arXiv · show

With the continuous growth in the number of parameters of transformer-based pretrained language models (PLMs), particularly the emergence of large language models (LLMs) with billions of parameters, many natural language processing (NLP) tasks have demonstrated remarkable success. However, the enormous size and computational demands of these models pose significant challenges for adapting them to specific downstream tasks, especially in environments with limited computational resources. Parameter Efficient Fine-Tuning (PEFT) offers an effective solution by reducing the number of fine-tuning parameters and memory usage while achieving comparable performance to full fine-tuning. The demands for fine-tuning PLMs, especially LLMs, have led to a surge in the development of PEFT methods, as depicted in Fig. 1. In this paper, we present a comprehensive and systematic review of PEFT methods for PLMs. We summarize these PEFT methods, discuss their applications, and outline future directions. Furthermore, we conduct experiments using several representative PEFT methods to better understand their effectiveness in parameter efficiency and memory efficiency. By offering insights into the latest advancements and practical applications, this survey serves as an invaluable resource for researchers and practitioners seeking to navigate the challenges and opportunities presented by PEFT in the context of PLMs.

I. INTRODUCTION

The growth of PLMs and LLMs makes full fine-tuning increasingly costly, motivating PEFT methods that reduce trainable parameters and memory while retaining comparable performance. This survey organizes PEFT approaches, reviews their applications, and evaluates representative methods.

  • Motivation: Full fine-tuning becomes increasingly expensive as pretrained models grow from BERT’s 110 million to T5’s 770 million parameters and beyond.Fine-tuning Falcon-180B may require at least 5120GB of computational resources.
  • Motivation: PEFT reduces trainable parameters by updating only additional or selected parameters, while preserving pretrained knowledge and reducing overfitting risk.The approach is intended to maintain performance comparable to full fine-tuning while lowering computational demands.
  • Scope and contributions: The survey addresses gaps in earlier reviews by covering newer work, providing detailed method explanations, and conducting quantitative experiments.Earlier surveys either covered fewer recent methods and experiments or omitted experiments altogether.
  • Scope and contributions: PEFT methods are classified into additive, partial, reparameterized, hybrid, and unified fine-tuning categories.This taxonomy provides a structured framework for understanding PEFT approaches.
  • Scope and contributions: The study evaluates representative methods for parameter efficiency and memory usage and discusses applications including multitask learning, cross-lingual transfer, and backdoor attack and defense.The experiments primarily examine NLU, machine translation, and natural language generation settings.

II. PRELIMINARIES

Transformer-based PLMs use encoder-decoder, encoder-only, or decoder-only architectures built from self-attention and feed-forward layers. These components provide the main locations where PEFT methods reduce trainable parameters.

  • Transformer architecture: The Transformer uses stacked encoder and decoder layers containing multi-head self-attention, feed-forward networks, residual connections, and layer normalization.This architecture underlies many pretrained language models.
  • Self-attention: Self-attention derives query, key, and value representations by applying learned linear transformations to the input sequence.The transformations use W_q, W_k, and W_v together with bias vectors.
  • Self-attention: Multi-head self-attention computes attention outputs for h heads in parallel and concatenates their results through an output projection.The formulation is represented as MHA(Q, K, V) = Concat(head1, · · · , headh)W_O.
  • Feed-forward network: The feed-forward network applies two linear transformations separated by a nonlinear ReLU activation.W1, b1, W2, and b2 parameterize the two transformations.
  • PEFT targets: PEFT methods primarily target self-attention and feed-forward layers across encoder-based RoBERTa, encoder-decoder T5, and decoder-based LLaMA models.Full fine-tuning instead updates all pretrained parameters on task-specific data.

III. PARAMETER-EFFICIENT FINE-TUNING METHODS

PEFT methods adapt pretrained models by adding trainable modules or soft prompts while leaving most pretrained parameters frozen. The section describes adapter-based and soft prompt-based approaches and their variants.

  • Additive fine-tuning: Additive fine-tuning introduces extra trainable parameters, including adapter modules and soft prompts, without modifying pretrained parameters.These methods are grouped as adapter-based and soft prompt-based fine-tuning.
  • Adapter-based Fine-tuning: Sequential Adapter inserts low-rank adapter networks after self-attention and feed-forward layers and updates only their projection parameters during fine-tuning.Each adapter contains down-projection, nonlinear activation, up-projection, and a residual connection.
  • Adapter-based Fine-tuning: Adapter variants modify placement or composition, including residual and parallel insertion, layer removal through AdapterDrop, and multi-task composition through AdapterFusion.AdapterSoup selects adapters for new domains after training multiple domain-specific adapters.
  • Soft Prompt-based Fine-tuning: Soft prompt fine-tuning optimizes trainable continuous vectors inserted into model inputs or hidden states rather than manually designed discrete prompts.Prompt parameter cost depends on prompt length and token embedding dimension.
  • Soft Prompt-based Fine-tuning: Prefix-tuning prepends soft prompts to attention hidden states, concatenating prefix key and value vectors with the original key and value representations.Only prefix vectors and the parameterizing FFN are optimized, while other PLM parameters remain frozen.
  • Soft Prompt-based Fine-tuning: Prompt transfer and decomposition methods extend soft prompting through source-task prompts, shared prompts, and task-specific low-rank factors.SPOT transfers prompts across tasks, while MPT decomposes each source prompt into shared and task-specific components.

3) Others:

Other additive approaches introduce small trainable components alongside pretrained transformers, including side networks, activation-scaling vectors, special-token adaptations, and attention-based fusion.

  • Other approaches: The section groups additional approaches into side networks, attention rescaling, special-token vectors, late fusion, and joint importance weights.These mechanisms add trainable components alongside the transformer rather than updating all pretrained parameters.
  • Side networks: LST trains a ladder side network from intermediate pretrained activations, so back-propagation uses the side network and ladder connections instead of the pretrained network.Structural pruning and layer dropping further reduce the side network’s size.
  • Activation scaling: (IA)3 optimizes three learned vectors per transformer block to rescale attention keys, values, and feed-forward hidden activations.The key and value vectors can be integrated into corresponding linear layers, leaving only the feed-forward vector as additional overhead.
  • Special-token adaptation: PASTA trains only a vector for selected special tokens, modifying their representations while leaving other token adaptations at zero.The method focuses on [CLS] and [SEP] because their representations receive substantial attention and [CLS] provides a global input representation.
  • Attention-based fusion: AttentionFusion uses late fusion and extra trainable attention weights to dynamically adjust the importance of token representations for each task.Its additional parameter count is determined by the query vector size, which matches the pretrained encoder’s hidden dimension.

B. Partial Fine-tuning

Partial fine-tuning reduces adaptation cost by updating only selected pretrained parameters, using bias updates or masks based on weight importance.

  • B. Partial Fine-tuning: Partial fine-tuning selects a subset of pretrained parameters considered critical to downstream tasks while discarding the remainder.The survey divides these methods into bias update, pretrained weight masking, and delta weight masking.
  • Bias Update: Bit-Fit updates transformer bias terms and the task-specific classification layer while keeping most pretrained parameters frozen.The updated biases occur in attention, feed-forward, and layer-normalization components.
  • Pretrained Weight Masking: Pretrained weight masking uses pruning criteria such as thresholds or Fisher information to identify important weights and construct binary masks.Threshold-Mask selects attention and feed-forward weights through element-wise multiplication with the mask.

3) Delta Weight Masking:

Delta weight masking restricts updates to sparse task-specific parameter changes selected by pruning or optimization criteria, while retaining the pretrained model as the base.

  • 3) Delta Weight Masking: LT-SFT first compares fully fine-tuned and pretrained weights, then selects the top-k parameters with the greatest absolute differences for subsequent updates.It resets the model to pretrained weights and iteratively fine-tunes only the selected parameters using a binary mask.
  • 3) Delta Weight Masking: Delta weight masking generally constructs binary masks through pruning techniques or criteria to reduce the number of trainable parameters.Unlike pretrained-weight masking, delta-weight pruning typically updates the mask or selected changes during training.
  • 3) Delta Weight Masking: Child-Tuning constructs a mask by marking the parameters in the updateable child network as one and the remaining parameters as zero.This implements delta masking through a parameter subnetwork selected for updating.
  • 3) Delta Weight Masking: Diff Pruning learns a sparse task-specific diff vector while freezing pretrained parameters, using a learnable binary mask as an L0-style regularizer.The method suits multitask deployment with limited storage but can consume more memory than traditional fine-tuning as model size grows.
  • 3) Delta Weight Masking: SAM represents the fine-tuned parameters as W = W0 + M ⊙ ΔW and uses a second-order loss approximation to select sparse delta weights.It computes gradient information and selects the top ⌊mp⌋ delta weights for optimization.

1) Low-rank Decomposition:

Low-rank decomposition and LoRA derivatives reparameterize updates with compact structures, extending parameter efficiency through dynamic ranks, quantization, pruning, and module composition.

  • 1) Low-rank Decomposition: Reparameterization represents high-dimensional delta weights with lower-rank matrices or transformations to reduce computational complexity and memory usage.The survey includes Fastfood, low-rank down-up projections, and Kronecker-product projections.
  • 1) Low-rank Decomposition: Intrinsic SAID estimates a task’s intrinsic dimensionality and optimizes a low-dimensional parameter vector projected into the original high-dimensional space.Its Fastfood transform has O(d) memory complexity, and updating all model parameters limits practicality for larger networks.
  • 1) Low-rank Decomposition: LoRA updates a pretrained matrix with two trainable low-rank factors, ΔW = W_downW_up, applied in parallel with attention query, key, and value matrices.The low-rank update combines orthogonal vectors and avoids storing full-rank input activations simultaneously, reducing activation-memory burden.
  • 2) LoRA Derivatives: LoRA improvements include dynamic rank adjustment, quantization-aware adaptation, kernel-wise parameter assignment, Bayesian calibration, and other architectural modifications.The survey organizes these approaches as LoRA derivatives alongside low-rank adjustment and LoRA-based multitask fine-tuning.
  • 2) LoRA Derivatives: DyLoRA trains across a range of ranks, while IncreLoRA allocates higher ranks to important modules and lower or zero ranks to less important ones.Both methods adapt parameter allocation without requiring exhaustive fixed-rank selection.
  • 2) LoRA Derivatives: LoRA derivatives also update pretrained weights, prune structured channels or heads, and merge LoRA weights into pretrained weights without extra inference computation.These variants use low-rank parameters and gradients to guide updates or identify redundant structures.
  • 2) LoRA Derivatives: QLoRA uses 4-bit NF4 quantization, double quantization, and a paged optimizer to reduce GPU requirements for LLM fine-tuning.QA-LoRA integrates group-wise quantization with low-rank adaptation so pretrained and auxiliary weights remain quantized after fine-tuning.
  • 2) LoRA Derivatives: LoRA-based multitask methods combine task-specific modules through weighted composition, mixture-of-experts gating, linearized fusion, or model-merging strategies.LoRAHub searches for module weights, MOELoRA uses task-motivated gates, and L-LoRA fine-tunes linearized modules in tangent space.

D. Hybrid Fine-Tuning

Hybrid fine-tuning combines multiple PEFT techniques to exploit complementary strengths, while manual designs allocate methods and parameters across transformer components.

  • Hybrid fine-tuning combines adapters, prefix-tuning, and LoRA to leverage complementary strengths and mitigate individual weaknesses.
  • Manual Combination: Manual combinations integrate one PEFT method’s structure or features into another to improve performance while retaining parameter efficiency.
  • Manual Combination: MAM Adapter combines scaled parallel adapters with prefix-tuning, assigning smaller bottlenecks to attention and more parameters to FFN representations.
  • Compacter combines adapters, low-rank optimization, and parameterized hypercomplex multiplication, replacing projection matrices with PHM-based constructions.
  • UniPELT combines sequential adapters, prefix-tuning, and LoRA, using gates to control each submodule’s activation and weighting.

2) Automatic Combination:

Automatic combination searches how PEFT modules should be configured across transformer layers, while unified methods provide cohesive adaptation frameworks, often with pruning or shared parameters.

  • Automatic Combination: Automatic combination configures adapters, prefix-tuning, BitFit, and LoRA across transformer layers through structure search and optimization.
  • Automatic Combination: S3Delta-M performs differentiable structure search over LoRA, Compacter, BitFit, and LNFit while explicitly controlling sparsity.
  • Automatic Combination: S4 searches layer groupings, parameter allocations, tunable groups, and PEFT assignments rather than applying one module uniformly across all layers.
  • Unified Fine-Tuning: Unified fine-tuning typically uses one PEFT method within a cohesive framework, unlike hybrid fine-tuning’s combination of different methods.
  • Unified Fine-Tuning: SparseAdapter prunes adapter and LoRA parameters using target sparsity and scores, and its plug-in pruning improves performance over full fine-tuning in the Large-Sparse setting.
  • Unified Fine-Tuning: ProPETL shares a prototype network across layers and tasks and uses binary masks to construct layer-specific subnetworks, reducing storage requirements.

IV. EXPERIMENTS

The experiments evaluate representative PEFT methods across encoder-only, encoder-decoder, and decoder-only models on GLUE, WMT16 En-Ro, and MMLU-related settings. On GLUE, most methods match or exceed full fine-tuning while reducing trainable parameters, with substantial variation among methods.

  • PLMs and Datasets: The study evaluates RoBERTa on GLUE, T5 on WMT16 En-Ro, and LLaMA models fine-tuned with Alpaca on MMLU.
  • Experimental Setup: Eleven representative PEFT methods are selected, including adapters, prompt-tuning, prefix-tuning, BitFit, LoRA, AdaLoRA, QLoRA, MAM Adapter, and ProPETL.
  • RoBERTa Base/Large on GLUE: All PEFT methods reduce trainable parameters, and most achieve performance matching or better than full fine-tuning on GLUE.
  • RoBERTa Base/Large on GLUE: 1.50% of trainable parameters lets ProPETLAdapter achieve optimal average GLUE performance, outperforming RoBERTa-base and RoBERTa-large full fine-tuning by about 1.30% and 1.65%.
  • RoBERTa Base/Large on GLUE: MAM Adapter outperforms prefix-tuning but consumes a large number of trainable parameters.
  • RoBERTa Base/Large on GLUE: Prompt-tuning with virtual marker length 20 uses the fewest trainable parameters but averages about 10% below full fine-tuning on GLUE.
  • RoBERTa Base/Large on GLUE: Child-TuningD outperforms full fine-tuning for RoBERTa-base but performs poorly for RoBERTa-large on MNLI, which the authors attribute speculatively to learning rate.

2) T5 Base/Large on WMT16 En-Ro Dataset:

The experiments compare PEFT methods with full fine-tuning across translation and language-model settings, emphasizing parameter reductions alongside performance differences. Results show that IA3 and LoRA can use very small trainable-parameter fractions while retaining comparable BLEU performance, whereas LLaMA PEFT methods trade some MMLU accuracy for efficiency.

  • T5 Base/Large on WMT16 En-Ro Dataset: LoRA uses 0.39% of trainable parameters and achieves BLEU 0.36 higher than full fine-tuning on T5-base.For T5-large, LoRA uses 0.32% of trainable parameters and achieves BLEU 0.01 lower than full fine-tuning.
  • LLaMA on MMLU: Full fine-tuning produces better 5-shot MMLU test accuracy than PEFT methods for both LLaMA-7B and LLaMA-13B.The comparison uses Alpaca-fine-tuned models evaluated on MMLU.
  • LLaMA on MMLU: IA3 uses 0.02% of full fine-tuning parameters but performs 2–4% lower than full fine-tuning on 5-shot MMLU accuracy.Among the listed PEFT methods, IA3 performs best in the LLaMA comparison.
  • LLaMA on MMLU: LoRA and QLoRA use about 2% of full fine-tuning parameters and achieve 5-shot MMLU accuracy about 2% lower than full fine-tuning.QLoRA uses half as many trainable parameters as LoRA while achieving comparable performance.

C. Memory Efficiency

The memory-efficiency experiments compare peak GPU usage for full fine-tuning and PEFT methods across several model families and benchmarks. PEFT reduces GPU memory in LLaMA models, with QLoRA providing the largest reductions, while MMLU accuracy can fluctuate during training.

  • Memory Usage Results: IA3, LoRA, and QLoRA all significantly reduce GPU footprint compared with full fine-tuning for LLaMA-7B and LLaMA-13B.The study reports reductions for all three methods relative to full fine-tuning.
  • Memory Usage Results: In LLaMA-7B-Alpaca, IA3, LoRA, and QLoRA reduce memory usage by 24.08%, 26.30%, and 66.66%, respectively.These percentages are measured against full fine-tuning memory usage.
  • Memory Usage Results: In LLaMA-13B-Alpaca, IA3, LoRA, and QLoRA reduce memory usage by 33.55%, 39.46%, and 76.86%, respectively.The reported reductions are larger for the 13B model than for the corresponding 7B model.
  • Training Stability: Five-shot MMLU dev accuracy fluctuates as evaluation steps increase during IA3, LoRA, and QLoRA fine-tuning.The fluctuations are shown for LLaMA-7B-Alpaca models on the MMLU dev set.
  • Memory Usage Results: QLoRA fine-tuning requires only 1/3 of full fine-tuning memory for LLaMA-7B and less than 1/4 for LLaMA-13B.The paper links this reduction to QLoRA’s particularly low GPU memory consumption.

VI. FURTHER DIRECTIONS

The paper identifies future directions spanning hybrid PEFT design, LoRA-derived methods, implementation support, theoretical understanding, and applications beyond NLP. These directions focus on improving combinations, explaining effectiveness, and extending PEFT to vision and multimodal learning.

  • Lightweight Hybrid PEFT Methods: Future work should explore broader combinations of PEFT methods and automatically search architectures and module placements while minimizing added parameters and memory.Existing combinations mainly involve adapters, LoRA, prefix-tuning, and BitFit, and combining methods can increase parameter and memory usage.
  • More LoRA-based PEFT Methods: LoRA-based methods increasingly incorporate adaptive rank adjustment, pruning, quantization, and multitask integration.The paper highlights pruning and weight quantization as priorities for further LoRA-derived development.
  • Implementation Support: PEFT and AdapterHub libraries simplify applying methods such as prefix-tuning, LoRA, and AdaLoRA through direct invocation and examples.The paper presents these libraries as support for the practical use of numerous PEFT methods.
  • Explainability of PEFT Methods: Theoretical research remains needed to explain generally why PEFT methods can achieve comparable performance while reducing trainable parameters.Existing analyses address sparsity and LoRA expressivity but do not yet provide a general account.
  • Computer Vision and Multimodal Learning: PEFT in computer vision and multimodal learning is a promising direction, including possible cross-modality transfer through fine-tuning pretrained models.The paper identifies language-image and image-audio learning as domains with substantial room for further exploration.
  • Conclusions: The paper’s experiments across RoBERTa, T5, and LLaMA report improved parameter efficiency, comparable or better performance, and lower memory footprints for most PEFT methods.QLoRA is singled out for drastically reducing computational memory requirements during LLM fine-tuning.
Loading 2312.12148v1…