PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning

Vladislav Lialin, Vijeta Deshpande, Xiaowei Yao, Anna Rumshisky

arXiv:2303.15647v2cs.CL

TL;DR

Fine-tuning increasingly large models creates resource constraints, motivating methods that train only a small subset of parameters. This paper surveys and compares PEFT methods, finding strong efficiency benefits but important discrepancies between reported parameter counts and practical costs.

  • Problem

    The rapid growth of language models has made fine-tuning increasingly difficult, motivating methods that reduce the number of trained parameters.

  • Method

    The paper provides a taxonomy and systematic comparison of 30 PEFT methods across more than 50 papers, including efficiency-focused analysis and experiments.

  • Results

    Low-rank and Kronecker-product reparameterizations reduce trainable parameters with minimal extra computation, while some methods incur substantial practical overhead.

  • Takeaways & Limitations

    PEFT should be evaluated using memory consumption and training speed alongside parameter counts, with reparameterization techniques remaining a promising direction.

  • Takeaways & Limitations

    Reported trainable, changed, and rank-based parameter counts are not equivalent, complicating comparisons of PEFT methods.

Abstract

from arXiv · show

This paper presents a systematic overview of parameter-efficient fine-tuning methods, covering over 50 papers published between early 2019 and mid-2024. These methods aim to address the challenges of fine-tuning large language models by training only a small subset of parameters. We provide a taxonomy that covers a broad range of methods and present a detailed method comparison with a specific focus on real-life efficiency in fine-tuning multibillion-scale language models. We also conduct an extensive head-to-head experimental comparison of 15 diverse PEFT methods, evaluating their performance and efficiency on models up to 11B parameters. Our findings reveal that methods previously shown to surpass a strong LoRA baseline face difficulties in resource-constrained settings, where hyperparameter optimization is limited and the network is fine-tuned only for a few epochs. Finally, we provide a set of practical recommendations for using PEFT methods and outline potential future research directions.

1 Introduction

The paper surveys PEFT as a response to the widening gap between model growth and available computing resources, then evaluates methods under practical fine-tuning constraints. Its findings emphasize efficiency trade-offs, implementation details, and unresolved reporting and performance challenges.

  • Motivation: Model size has grown far faster than computational resources, making fine-tuning the largest models infeasible for most users.Publicly available models grew from 350 million to 176 billion parameters in five years, while RAM increased less than tenfold.
  • Motivation: PEFT addresses this challenge by training only a small subset of existing or newly added parameters.Methods differ in parameter and memory efficiency, training speed, final model quality, and inference costs.
  • Contributions: The survey covers 30 PEFT methods across more than 50 papers from early 2019 to mid-2024 and identifies unresolved theoretical, performance, and reporting issues.The authors provide a systematic overview, comparison, and taxonomy.
  • Contributions: The experimental comparison evaluates 14 methods and variations across five datasets and model sizes of 0.7B, 3B, and 11B.Efficiency is assessed using GPU memory consumption and throughput alongside downstream performance.
  • Findings: Kronecker-based reparametrizations can improve training and inference speed with efficient implementation but do not improve memory efficiency over matrix-product counterparts.Layer Norm tuning also performs exceptionally well in the study.

2 Background: Transformer

The Transformer is introduced through its basic MHA-plus-FFN block, with residual connections and Layer Normalization supporting trainability. Attention transfers information between tokens through query-key similarities and value-weighted aggregation.

  • Transformer block: The core Transformer block consists of multi-head attention followed by a fully connected feed-forward layer.Both sublayers incorporate residual connections and Layer Normalization.
  • Attention: Attention computes a softmax-normalized weighted average of input tokens using pairwise dot products between token keys and queries.Keys and queries are typically produced by linear projections.
  • Attention: Methods that modify W_K, W_Q, and W_V directly affect how information is passed between tokens and which values are transmitted.These matrices provide the main mechanism for token-to-token information transfer.
  • PEFT context: Most Transformer PEFT methods rely on the basic MHA-plus-FFN structure and can be adapted to architectural variations.Examples include cross-attention in sequence-to-sequence models and Pre-LN configurations.

3 Taxonomy of PEFT: a birds-eye view

The taxonomy organizes PEFT methods by whether they add parameters, selectively tune existing parameters, or reparameterize updates. These families make different trade-offs across efficiency, modularity, and model quality.

  • Taxonomy: PEFT methods are categorized as addition-based, selection-based, or reparametrization-based according to how they modify the model.The taxonomy also considers objectives such as memory reduction, storage efficiency, and modularity.
  • Addition-based methods: Addition-based methods train newly introduced parameters or layers, chiefly through adapter-like methods and soft prompts.Adapters add small networks after Transformer sublayers, while soft prompts tune input embeddings.
  • Addition-based methods: Added parameters can reduce memory by shrinking gradients and optimizer states, even though they introduce extra network components.Training commonly requires 12–20 times more GPU memory than model weights because of these states and gradients.
  • Selection-based methods: Selection-based methods tune chosen existing parameters, such as top layers, biases, particular rows, or sparsely selected updates.Sparse updates also create engineering and efficiency challenges.
  • Reparametrization-based methods: Reparametrization-based methods use low-rank representations to minimize trainable parameters, including LoRA’s matrix decomposition and Kronecker-product updates.LoRA uses δW = W_downW_up, while Kronecker methods use δW = A ⊗B.
  • Hybrid methods: Hybrid methods combine categories to target specific trade-offs, such as combining adapters with prompt tuning or LoRA.UniPELT combines LoRA, prefix-tuning, and adapters; Compacter and KronAB use Kronecker products in adapters.

4 A Deep Dive into PEFT

The paper’s deep-dive sections summarize PEFT methods and document the model scales and parameter counts associated with their published evaluations. This structure supports comparison of methods across their reported applicability and efficiency.

  • Method analysis: The paper presents detailed distinctions and trade-offs among PEFT approaches using the efficiency dimensions introduced earlier.It also provides easy-to-understand pseudo-code for important algorithmic components where feasible.
  • Reported parameters: Table 2 reports the model sizes evaluated and typical trainable-parameter amounts reported in the literature.Updated trainable-parameter counts are provided through Table 5.
  • Model scale: Each method description indicates whether it has been applied below 1B, below 20B, or above 20B parameters.The authors prefer explicit parameter counts because labels such as “small” and “large” change meaning quickly.

5 Additive methods: Adapters

Adapters add small fully connected networks between Transformer layers to achieve parameter-efficient fine-tuning. Variants trade off performance, training memory, and inference cost through placement, expert selection, and regularization.

  • 5 Additive methods: Adapters: Adapters add fully connected networks between Transformer attention and feed-forward layers, typically using a smaller hidden dimension than the input.They were introduced for NLP as an extension of related image-classification methods.
  • 5 Additive methods: Adapters: Tuning adapters can match full fine-tuning performance while updating less than 4% of the model parameters.
  • 5 Additive methods: Adapters: Inserting one adapter after the self-attention layer can achieve similar performance to using two adapters per Transformer block.
  • 5 Additive methods: Adapters: AdaMix randomly activates a single expert per forward pass and averages expert weights after training to reduce inference costs.Consistency regularization minimizes symmetrized KL divergence between forward passes using different experts.
  • 5 Additive methods: Adapters: AdaMix achieves better performance than regular adapters at the same inference cost but may require more training memory.Its smaller adapter hidden states can amortize trainable-parameter overhead across approximately 4–8 experts, while consistency regularization stores two hidden-state and gradient versions.

6 Additive Methods: Soft Prompts

Soft-prompt methods optimize trainable prompt representations instead of the model’s full parameters, with variants placing prompts at the input or throughout the network. Their efficiency and effectiveness improve with model scale in some settings, but convergence, sequence-length cost, and autoencoder overhead remain important constraints.

  • 6 Additive Methods: Soft Prompts: Soft prompts replace impractical discrete prompt optimization or extensive in-context learning with continuously optimized representations.
  • 6.1 Prompt Tuning: Prompt tuning prepends trainable soft-prompt embeddings to input embeddings and optimizes them directly with gradient descent.
  • 6.1 Prompt Tuning: T5-11B reaches the same SuperGLUE performance with either 5 or 150 soft-prompt tokens, indicating greater parameter efficiency at larger scale.
  • 6.1 Prompt Tuning: T5-large performance saturates at 20K trainable parameters, while T5-XL saturates at the same 20K parameters using five prompt tokens.Prompt tuning becomes comparable with full fine-tuning only at approximately the 10B model scale, and longer sequences increase quadratic transformer computation.
  • 6.2 Prefix Tuning: Prefix tuning prepends the same learned prefix to hidden states at every layer, using an FFN parameterization to stabilize optimization.It is similar to prompt tuning but applies soft prompts throughout the network rather than only at the input.
  • 6.2 Prefix Tuning: Prefix tuning achieved performance close to full fine-tuning on BART generation tasks while training only 0.1% of the parameters.The experiments used soft-prompt lengths ranging from 10 to 200 tokens.
  • 6.3 Adaptive Prefix Tuning: Adaptive prefix tuning uses layer-specific gating and scaling to select or deselect pseudo tokens in different layers.
  • 6.3 Adaptive Prefix Tuning: Adaptive prefix tuning consistently outperforms standard prefix tuning and surpasses full fine-tuning in most BERT and DeBERTa cases.

7 Additive Methods: Other Approaches

Other additive approaches attach a side network or learn a few scaling vectors rather than updating the backbone. These designs can reduce training memory and compute, with accuracy and performance depending on the method and comparison setting.

  • 7 Additive Methods: Other Approaches: Ladder-Side Tuning trains a small Transformer beside the pretrained network and combines backbone hidden states with side-network states.
  • 7 Additive Methods: Other Approaches: LST uses the pretrained model as a feature extractor, so backpropagation runs only through the side network, saving training memory and compute.Its side network is initialized from structurally pruned pretrained parameters and uses half as many layers.
  • 7 Additive Methods: Other Approaches: LST reduces RAM threefold versus full fine-tuning and twofold versus LoRA on T5-Base, with a small accuracy degradation.It outperforms those methods when RAM usage is controlled.
  • 7 Additive Methods: Other Approaches: (IA)3 learns three vectors that rescale key, value, and FFN activations in each Transformer block.The vectors are integrated into corresponding linear layers, creating minimal inference overhead.
  • 7 Additive Methods: Other Approaches: For T0-3B, (IA)3 updates about 0.02% of parameters and outperforms methods including Compacter and LoRA despite LoRA using 16 times more trainable parameters.

8 Selective Methods

Selective methods update chosen existing parameters, including biases, sparse masks, selected matrix components, Fisher-ranked parameters, and diffusion-model scaling terms. Their efficiency benefits are accompanied by setting-specific limitations such as architecture dependence, training overhead, or sparse-operation inefficiency.

  • 8 Selective Methods: Selective methods fine-tune subsets of existing parameters according to layer depth, layer type, or individual parameter selection.
  • 8 Selective Methods: BitFit updates only biases, training about 0.05% of parameters and matching or exceeding full fine-tuning on low- and medium-data BERT settings.
  • 8 Selective Methods: For models larger than 1B parameters, BitFit significantly underperforms full fine-tuning and other PEFT methods.
  • 8 Selective Methods: Bias-dependent methods have limited applicability because T5 largely lacks biases and LLaMA has no bias terms throughout its network.
  • 8 Selective Methods: DiffPruning learns a sparse binary-masked weight update with L0-style regularization and matches full fine-tuning while modifying 0.5% of parameters below 1B.It is useful for storage-limited edge deployment but optimizes all parameters and the mask during training, increasing memory use.
  • 8 Selective Methods: FAR identifies important matrix rows, splits each layer into trainable and frozen components, and removes inference overhead by reconfiguring parameters after training.Its training procedure adds compute overhead.
  • 8 Selective Methods: FAR matches fine-tuning performance on GLUE and SQuAD 2.0 while updating 6% of DistilBERT parameters in edge-focused experiments.
  • 8 Selective Methods: FishMask selects parameters with the highest Fisher information, then optimizes only those selected parameters.It performs on par with adapters but below LoRA and (IA)3, while sparse-operation support limits hardware efficiency.

9 Reparametrization-based methods

Reparametrization-based PEFT methods reduce trainable parameters by expressing updates through low-rank or structured transformations, while differing in flexibility, efficiency, and resource costs. The reviewed methods include low-rank updates, Kronecker parameterizations, magnitude-direction decoupling, generalized scaling, adaptive rank allocation, and gradient-based projections.

  • Low-rank reparametrization: Low-rank reparametrization reduces trainable parameters while supporting updates to high-dimensional pretrained weight matrices.Intrinsic SAID instead updates all model parameters through a low-rank transformation.
  • Intrinsic SAID: Intrinsic SAID uses the Fastfood transform to reparametrize weight updates in O(D log d) time and O(D) memory.The method operates in a low-rank subspace and updates all model parameters rather than selecting a subset.
  • LoRA: LoRA freezes pretrained parameters and trains only two low-rank matrices, which can later be merged into the original weight matrix.LoRA is commonly applied to attention projections, while applying it to all weight matrices can improve performance.
  • Kronecker methods: KronA replaces LoRA’s matrix factorization with a Kronecker product, improving the rank-to-parameter tradeoff and enabling efficient matrix-vector products.Its rank satisfies rank(A ⊗ B) = rank A·rank B, and the update need not be represented explicitly.
  • Kronecker methods: At 0.07% trainable parameters on GLUE, KronA methods perform on par or better than adapters, LoRA, and Compacter while being significantly faster at inference.The evaluation covered only models with fewer than 1B parameters.
  • DoRA: DoRA separates directional and magnitude changes in LoRA-style fine-tuning, achieving consistent improvements over LoRA across evaluated tasks and models.Its scaling can produce large directional changes with small magnitude adjustments, or the reverse.
  • GLoRA: GLoRA extends LoRA by learning parameters that scale or shift weights and activations, increasing trainable parameters but allowing all updates to merge after fine-tuning.Its experiments report better performance and sample efficiency than LoRA with comparable trainable parameters.
  • AdaLoRA: AdaLoRA adaptively prunes singular-value components according to loss-impact scores, but tracking pruning variables can create significant memory overhead as model size or rank grows.Experiments on DeBERTa-base and BART-large report better performance than LoRA and other PEFT methods under a parameter budget.

10 Hybrid Approaches

Hybrid PEFT methods combine adapters, prompts, LoRA, Kronecker parameterization, or layer-wise strategies to balance parameter efficiency and performance. Reported benefits include stronger low-data performance and competitive results at very small parameter budgets, but hardware and model-size boundaries remain important.

  • Overview: Hybrid methods combine multiple PEFT techniques to exploit complementary strengths and mitigate weaknesses.Examples include MAM Adapter, UniPELT, Compacter, and S4.
  • Large-Sparse: Large-Sparse adapters prune about 40% of values at initialization and consistently outperform nonsparse counterparts with the same trainable parameter count.Training and inference costs may increase depending on sparse-tensor hardware support.
  • MAM Adapters: MAM Adapter combines scaled parallel FFN adapters with soft prompts, which modify attention using only 0.1% of the parameters.Its reported configuration uses a 200-token soft prompt and 7% extra parameters.
  • MAM Adapters: With a 200 soft prompt length and 7% extra parameters, MAM consistently outperforms LoRA, Adapters, and Prefix Tuning on evaluated tasks.The experiments used models with fewer than 1B parameters.
  • UniPELT: UniPELT gates LoRA, Prefix Tuning, and Adapters, applying them to attention projections, layer keys and values, and after feedforward layers.Each gate uses a linear projection, sigmoid activation, and sequence-length averaging.
  • UniPELT: With 1.3% trainable parameters on sub-billion-parameter BERT models, UniPELT improves over individual methods in low-data settings with only 100 examples.In higher-data scenarios, it performs on par or better than the compared approaches.
  • Compacter: Compacter uses Kronecker products, low-rank matrices, and parameter sharing across layers to construct adapter weights.Compacter++ uses a single adapter after the feedforward layer.
  • Compacter: With 0.05% additional parameters, Compacter++ performs on par or better than adapters with 0.8% additional parameters.Evaluations included T5 Base and T0-3B models.

11 Comparison of PEFT Methods

The comparison evaluates PEFT methods across performance, parameter, memory, computation, and inference dimensions, revealing substantial trade-offs. Houlsby Adapters, LoRA, and Layer Norm Tuning are especially competitive, while several alternatives are sensitive to training conditions or incur efficiency costs.

  • Evaluation dimensions: Parameter efficiency alone does not guarantee lower RAM use or faster training, so PEFT methods require multidimensional evaluation.The study considers storage, memory, computation, inference overhead, and downstream performance because improvements on one axis may not transfer to others.
  • Downstream performance: Houlsby Adapters and LoRA consistently match or exceed full-tuning performance with little hyperparameter tuning.Layer Norm tuning is also reported as competitive with full fine-tuning for T5-Large and T5-11B and is easy to implement.
  • Downstream performance: Hybrid methods such as MAM Adapters and UniPELT are difficult to train and highly hyperparameter-sensitive in compute-limited settings.The authors attribute their weak performance partly to Prompt Tuning, which performs poorly when compute is constrained.
  • Downstream performance: Many methods previously reported to outperform Adapters or LoRA underperform in this setup, with Parallel Adapter, Compacter, and KronA remaining competitive in several 11B cases.The comparison therefore exposes differences between prior reported results and performance under limited tuning and compute.
  • Efficiency: Added parameters can slow training and inference: LoRA slows T5-Large training by 20%, while additive methods incur 33–55% inference slowdowns for T5-Large.Reparametrization weights can be merged into the network to restore regular fine-tuning inference speed, whereas Kronecker methods may improve speed without materially reducing memory.

12 Challenges and guidelines

The section identifies reporting, evaluation, implementation, and comparability challenges in PEFT, then proposes guidelines for more informative and reproducible assessments.

  • Reporting parameter count: Parameter counts may denote trainable parameters, changed parameters, or update rank, and these quantities are not equivalent.Trainable-parameter count is the most reliable memory-efficiency predictor, but it remains imperfect.
  • Reporting efficiency: Efficiency cannot be inferred from parameter count alone because memory consumption and training speed vary non-linearly across methods.The section recommends reporting RAM, token throughput, inference speed, and storage requirements, including metrics for each stage of multi-stage methods.
  • Model sizes: Evaluations should span multiple model scales, especially models above 1B and below 20B parameters, because PEFT’s usefulness increases with backbone size.The guideline also recommends reporting results for model sizes commonly used by the research community.
  • Method implementation: Published PEFT implementations are often difficult to identify, reuse, or install because copied code lacks documentation, compatibility, and examples.LoRA and Compacter are highlighted as more user-friendly and adaptable implementations.
  • Comparison: PEFT comparisons are hindered by absent standard benchmarks and metrics, differing model–dataset combinations, and inconsistent evaluation settings.These differences make meaningful cross-method conclusions difficult.
  • Comparison: The section calls for community-developed standardized benchmarks and competitions to improve PEFT evaluation.Standardization is presented as a response to inconsistent comparisons across methods.

13 Discussion

The discussion frames PEFT as a way to modify large models under compute constraints while noting persistent performance and hyperparameter challenges. It highlights reparameterization and layer-adaptive methods as directions for improving efficiency and accuracy.

  • Discussion: Low-bit quantization has helped democratize inference, while PEFT extends efficiency-oriented access from using large models to modifying them.The paper connects both developments to experimentation with relatively modest compute budgets.
  • Discussion: Adapters, Prompt Tuning, LoRA, and (IA)3 have shown practicality at scale, but matching full fine-tuning remains challenging.The paper attributes part of this challenge to PEFT’s sensitivity to hyperparameters and different optimal settings.
  • Discussion: Low-rank and Kronecker-product reparameterizations reduce trainable parameters while requiring minimal extra computation.The paper identifies reparameterization techniques with favorable trainable-parameter-to-rank ratios as a future direction.
  • Discussion: Adaptive parameter allocation across transformer layers could improve parameter efficiency and accuracy by reflecting layer-dependent text processing.Most existing PEFT methods apply parameters uniformly across the model.
  • Discussion: Large-model fine-tuning shares memory, computation, and energy constraints with edge machine learning, creating opportunities for ideas to transfer between the areas.The discussion specifically relates quantization and pruning in edge machine learning to large language models.

Appendix B. PEFT Comparison: full experimental results

The appendix reports seed-level PEFT scores across T5 models, tasks, and methods, alongside full-tuning reference results. The results show substantial variation across methods, datasets, scales, and random seeds.

  • T5 3B results: 90.61 T5 3B Scaled Parallel RTE seed 0 demonstrates strong performance among the reported T5 3B results.The same method scores 91.34 for seed 1 and 89.89 for seed 42 on RTE.
  • T5 3B results: 89.00 T5 3B Ln tuning COPA seeds 0 and 1 exceeds the reported T5 3B Compacter++ COPA scores for those seeds.Compacter++ reports 79.00 for seed 0 and 82.00 for seed 1 on COPA.
  • T5 3B results: 1.00 T5 3B IA3 COPA is reported for seeds 0, 1, and 42, while its BoolQ scores are 87.28 across all three seeds.These task-dependent results illustrate large variation within one method.
  • Reference results: 27.32 T5 11B Full tuning CNN provides the reported full-tuning reference score for that task and model.T5 11B Scaled Parallel CNN seed 0 is reported at 29.21.
  • T5 11B results: 92.78 T5 11B LoRa (all) RTE seed 0 is higher than its seed 1 and seed 42 scores of 91.70 and 90.97.The appendix also reports 94.00 for T5 11B LoRa (all) COPA seeds 0 and 1.
Loading 2303.15647v2…