PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning

Vladislav Lialin, Vijeta Deshpande, Xiaowei Yao, Anna Rumshisky

arXiv:2303.15647v2cs.CL

TL;DR

Fine-tuning increasingly large language models makes full-parameter training difficult, motivating parameter-efficient alternatives. This paper surveys and compares PEFT methods and finds that low-rank reparameterization is especially successful, while efficiency requires evaluation beyond trainable-parameter counts.

  • Problem

    The rapid growth of language models creates challenges for fine-tuning, while PEFT still has limited theoretical understanding, performance gaps, and reporting issues.

    Source: §1 · pp. 1–2

  • Method

    The paper systematically surveys, taxonomizes, and compares 30 PEFT methods across more than 50 papers published from early 2019 to mid-2024.

    Source: §1 · p. 2

  • Results

    Low-rank and Kronecker-product reparameterizations successfully reduce trainable parameters while requiring minimal extra computation.

    Source: §13 · p. 36

  • Takeaways & Limitations

    PEFT efficiency should be assessed using memory consumption and training speed rather than parameter count alone.

    Source: §12 · p. 33

    Inconsistent reporting makes PEFT parameter counts difficult to interpret because trainable, changed, and difference-rank counts are not equivalent.

    Source: §12 · p. 33

Abstract

from arXiv · show

This paper presents a systematic overview of parameter-efficient fine-tuning methods, covering over 50 papers published between early 2019 and mid-2024. These methods aim to address the challenges of fine-tuning large language models by training only a small subset of parameters. We provide a taxonomy that covers a broad range of methods and present a detailed method comparison with a specific focus on real-life efficiency in fine-tuning multibillion-scale language models. We also conduct an extensive head-to-head experimental comparison of 15 diverse PEFT methods, evaluating their performance and efficiency on models up to 11B parameters. Our findings reveal that methods previously shown to surpass a strong LoRA baseline face difficulties in resource-constrained settings, where hyperparameter optimization is limited and the network is fine-tuned only for a few epochs. Finally, we provide a set of practical recommendations for using PEFT methods and outline potential future research directions.

1 Introduction

As model sizes have grown far faster than hardware resources, fine-tuning has become infeasible, motivating PEFT methods that train only a small parameter subset. This survey systematically reviews PEFT methods, evaluates their efficiency and performance, and identifies unresolved challenges and practical research directions.

Motivation

  • Models grew from 350 million to 176 billion parameters in five years, while RAM increased less than tenfold, making fine-tuning the largest models infeasible.Published literature includes models up to 1 trillion parameters, while model size scales almost two orders of magnitude faster than computational resources.Source: §1 · pp. 1–2
  • PEFT addresses this challenge by training only a small subset of existing or newly added parameters, trading off efficiency, speed, quality, and inference cost.The survey focuses on minimizing RAM consumption and training time without sacrificing performance, particularly for large language models.Source: §1 · p. 2

Scope and contributions

  • The survey covers 30 PEFT methods and over 50 papers from early 2019 to mid-2024, includes selected pre-training methods, and highlights theoretical, performance, and reporting gaps.It presents a systematic overview, comparison, and taxonomy while identifying limited theoretical understanding, the PEFT–fine-tuning performance gap, and reporting issues.Source: §1 · p. 2

Experiments

  • Across 14 methods and variations, five datasets, and 0.7B–11B models, methods previously outperforming LoRA struggled under resource constraints, while hybrid methods showed high hyperparameter sensitivity.The comparison measures GPU memory consumption and throughput in addition to task performance.Source: §1 · p. 2

Findings

  • Kronecker reparametrizations improved training and inference speed without improving memory efficiency, while Layer Norm tuning performed exceptionally well and reported parameter counts often understated actual costs.These discrepancies contributed to unexpectedly high computational costs for Prompt Tuning and Hybrid methods.Source: §1 · pp. 2–3

Future directions

  • The survey recommends standardized benchmarks, deeper hyperparameter and interpretability studies, analysis of reparametrized training dynamics, greater efficiency, and PEFT with quantized backbones.These directions target unresolved methodological and practical limitations in PEFT research.Source: §1 · p. 3

2 Background: Transformer

Transformers center on a multi-head attention (MHA) and fully connected (FFN) block with residual connections and Layer Normalization. Attention transfers information between tokens through softmax-normalized, query-key-weighted values, while most Transformer PEFT methods target this basic structure and adapt to architectural variations.

Transformer architecture

  • A basic Transformer block combines multi-head attention with a fully connected layer, and both incorporate residual connections and Layer Normalization.The block is illustrated in Figure 1.Source: §2 · p. 3, Figure 1 · p. 3

Attention mechanism

  • Attention computes a softmax-normalized weighted average of input tokens using pairwise query-key dot products and value representations.Keys, queries, and values are typically formed through linear projections; the matrices W_K, W_Q, and W_V control information transfer between tokens.Source: §2 · pp. 3–4

PEFT applicability

  • Most Transformer PEFT methods rely on the basic MHA + FFN structure and can be adapted to variants such as cross-attention or Pre-LN architectures.Although some techniques apply broadly across neural networks, Transformer-specific methods are especially valuable because Transformers are the largest neural networks ever trained.Source: §2 · pp. 3–4

3 Taxonomy of PEFT: a birds-eye view

The taxonomy organizes PEFT methods by whether they add, select, or reparameterize trainable components, while hybrid methods combine categories for specific tradeoffs. It covers over 30 methods and motivates later detailed discussions.

3 Taxonomy of PEFT: a birds-eye view

  • PEFT methods are categorized by approach into addition-based, selection-based, reparametrization-based, and hybrid methods, with Figure 2 and Sections 3.1–3.4 covering over 30 methods.The taxonomy also distinguishes objectives including minimizing memory footprint, improving storage efficiency, and adding modularity.Source: §3 · p. 4

3.1 Additiion-based methods

  • Addition-based methods train newly introduced parameters or layers, chiefly through Adapter-like methods and Soft Prompts, while reducing gradients and optimizer-state memory.Adapters add small fully connected networks after Transformer sub-layers; Soft Prompts optimize input embeddings continuously rather than searching discrete prompts.Source: §3.1 · pp. 4–5

3.2 Selection-based methods

  • Selection-based methods fine-tune only selected layers, parameter types, rows, or sparse subsets, but unrestricted unstructured sparsity remains impractical on contemporary hardware.Recent work addresses sparse-update engineering through parameter reconfiguration and NxM sparsity.Source: §3.2 · p. 6

3.3 Reparametrization-based methods

  • Reparametrization-based methods use low-rank representations, including LoRA’s low-rank weight update δW = W_downW_up, to reduce trainable parameters and scale to models up to 175B parameters.Kronecker-product reparametrization offers a more favorable tradeoff between rank and parameter count.Source: §3.3 · p. 6

3.4 Hybrid methods

  • Hybrid methods combine categories to optimize specific algorithmic tradeoffs, exemplified by MAM Adapter’s Adapters plus Prompt tuning and UniPELT’s addition of LoRA.Compacter and KronAB reduce adapter parameter counts using Kronecker products.Source: §3.4 · pp. 6–7

4 A Deep Dive into PEFT

This section details diverse parameter-efficient fine-tuning approaches, emphasizing their distinctions, tradeoffs, evaluated model scales, and typical trainable-parameter counts. It also provides concise method summaries and, where feasible, pseudo-PyTorch implementations of core algorithmic components.

Method overview

  • The section compares PEFT methods by their distinctions and tradeoffs across the dimensions defined in Section 11.Each method includes a bold one-sentence summary to support skimming.Source: §4 · p. 7
  • Method descriptions specify whether approaches have been applied to models below 1B, below 20B, or above 20B parameters, with Table 2 summarizing model sizes and typical trainable-parameter counts.The authors prefer explicit parameter counts because the meanings of “small” and “large” change rapidly.Source: §4 · p. 8, Table 2 · p. 8
  • Where feasible, each method includes a brief pseudo-PyTorch implementation of its most important algorithmic component.Source: §4 · p. 8

5 Additive methods: Adapters

Adapters are a major additive PEFT family that inserts small fully connected networks between model layers. They can match full fine-tuning performance with fewer than 4% of model parameters, while AdaMix improves adapter performance at the cost of higher training memory.

5.1 Adapters

  • Houlsby et al. (2019) adapted earlier adapter concepts to NLP by inserting smaller fully connected networks after Transformer attention and FFN layers.The approach originated in related image-classification work by Rebuffi et al. (Rebuffi et al., 2017, 2018).Source: §5.1 · p. 8

5 Additive methods: Adapters

  • Adapters add fully connected networks between Transformer layers and can achieve full fine-tuning performance by tuning less than 4% of model parameters.Their hidden dimension is typically smaller than the layer input dimension.Source: §5 · p. 8, §5.1 · p. 9

5.1 Adapters

  • Inserting an adapter only after the self-attention layer, after normalization, achieves similar performance to using two adapters per Transformer block.This finding is reported by Pfeiffer et al. (2020a).Source: §5.1 · p. 9

5.2 AdaMix

  • AdaMix randomly activates one expert per forward pass, uses independently selected up- and down-projections, and averages expert weights after training to reduce inference costs.Consistency regularization stabilizes training by minimizing symmetrized KL divergence between forward passes using different expert sets.Source: §5.2 · p. 9
  • AdaMix achieves better performance than regular adapters at the same inference cost, but consistency regularization increases training memory by retaining two hidden-state and gradient versions.Using roughly 4–8 experts can amortize trainable-parameter overhead through smaller adapter hidden states.Source: §5.2 · pp. 9–10

6 Additive Methods: Soft Prompts

Soft-prompt methods replace impractical discrete prompt optimization with trainable continuous prompts, but their efficiency and practicality vary with model scale, inference cost, and added overhead. Prompt tuning benefits larger models, prefix variants can approach or exceed full fine-tuning, while intrinsic prompt tuning introduces substantial autoencoder costs.

6.1 Prompt Tuning

  • Prompt tuning prepends input embeddings as soft prompts optimized directly through gradient descent, converting discrete prompt search into continuous optimization.Soft prompts address the impracticality of optimizing discrete natural-language prompts or relying on in-context learning with many examples (Li and Liang, 2021; Lester et al., 2021; Liu et al., 2021).Source: §6.1 · p. 10, §6 · p. 10
  • Prompt tuning becomes more parameter-efficient as model size increases: T5-11B matches SuperGLUE performance with either 5 or 150 soft prompt tokens.T5-large saturates at 20 tokens and 20K trainable parameters (0.002%), while T5-XL saturates at 5 tokens and 20K parameters (0.0002%).Source: §6.1 · p. 10
  • Despite extreme parameter efficiency, prompt tuning adds inference overhead and becomes comparable with full fine-tuning only around the 10B model scale.Adding 20–100 sequence tokens can significantly increase computation because transformer complexity is quadratic, making soft prompts more applicable to larger models.Source: §6.1 · p. 10

6.2 Prefix Tuning

  • Prefix tuning achieves performance close to full fine-tuning on BART generation tasks while training only 0.1% of parameters.The study uses soft prompt lengths ranging from 10 to 200 tokens and prepends shared trainable parameters to every layer’s hidden states.Source: §6.2 · pp. 10–11

6.3 Adaptive Prefix Tuning

  • Adaptive prefix tuning consistently outperforms standard prefix tuning and surpasses full fine-tuning in most BERT and DeBERTa cases.It uses layer-specific sigmoid gating and scaling to adaptively select or deselect soft tokens, adding computation, memory, and storage overhead.Source: §6.3 · p. 11

6.4 Intrinsic Prompt Tuning (IPT)

  • Prompt tuning is slow to converge, and pretraining soft prompts improves performance and convergence speed without reducing per-task parameter counts.These approaches therefore address optimization speed rather than the number of parameters required for each task (Vu et al., 2021).Source: §6.4 · p. 11
  • Intrinsic Prompt Tuning compresses learned soft prompts into a low-dimensional task subspace, then trains only decoder inputs for new tasks.However, its BART-base autoencoder has 78 million parameters for prompt length 100, exceeding 56% of the model and limiting practical efficiency.Source: §6.4 · p. 12

7 Additive Methods: Other Approaches

This section presents additive PEFT methods that attach trainable components to a frozen backbone, reducing training costs while maintaining competitive performance. LST reduces memory substantially, while (IA)3 updates only about 0.02% of T0-3B parameters and outperforms comparable methods.

7.1 Ladder-Side Tuning (LST)

  • LST trains a small side transformer that combines its hidden states with frozen backbone hidden states, using the pretrained model only as a feature extractor.Backpropagation therefore runs only through the side network; the side network is initialized from structurally pruned pretrained parameters and uses half as many layers.Source: §7.1 · p. 12
  • LST achieves three-fold lower RAM than full fine-tuning and two-fold lower RAM than LoRA on T5-Base, with a small accuracy degradation, and wins at matched RAM.The comparison is reported in Section 9.2 and concerns fine-tuning T5-Base.Source: §7.1 · p. 13

7.2 (IA)3

  • (IA)3 learns vectors lv, lk, and lff that rescale value, key, and hidden FFN activations in each transformer block.This design achieves high parameter efficiency by training only three vectors per block.Source: §7.2 · p. 13
  • (IA)3 updates about 0.02% of T0-3B parameters and outperforms Compacter and LoRA, while adding minimal overhead.It trains three vectors per transformer block, with only the FFN vector contributing runtime overhead because the attention vectors can be merged into linear layers.Source: §7.2 · p. 13

8 Selective Methods

Selective methods train only chosen existing parameters, ranging from biases and sparse weight updates to selected matrix components and Fisher-informed parameters. Their efficiency and practicality vary: some match full fine-tuning with few updated parameters, while others incur memory, compute, or hardware limitations.

8 Selective Methods

  • Selective methods choose existing parameters by layer depth, layer type, or individual parameter, including trainable and frozen matrix components.FAR identifies important rows using partial fine-tuning and L1 distance, then reconfigures matrices into independently computed trainable and frozen parts; training adds compute overhead, but inference overhead is removed after reconfiguration.Source: §8 · p. 13, §8.3 · pp. 14–15

8.1 BitFit

  • BitFit updates only network biases, training about 0.05% of parameters; it matches or exceeds full fine-tuning in low- and medium-data BERT settings but underperforms beyond 1B parameters.Ben-Zaken et al. (2021) leaves every weight W unchanged and trains only b; later work reports underperformance against full fine-tuning and other PEFT methods for larger models.Source: §8.1 · p. 14
  • Bias-based selection is unavailable or limited in architectures lacking network bias terms, such as T5 and LLaMA.T5 retains bias terms only in relative attention weights, whereas LLaMA does not use bias terms throughout the network.Source: §8.1 · p. 14

8.2 DiffPruning

  • DiffPruning matches full fine-tuning in sub-1B scenarios while modifying only 0.5% of parameters, supporting storage-limited multitask deployment on edge devices.Guo et al. (2020) learns a sparse weight update using a differentiable approximation to the L0 norm, but training requires optimizing all parameters plus a binary mask and therefore more memory.Source: §8.2 · p. 14

8.3 Freeze and Reconfigure (FAR)

  • FAR achieved performance similar to fine-tuning on GLUE and SQuAD 2.0 while updating only 6% of DistilBERT parameters.The original study targeted edge scenarios, applied FAR to feed-forward layers, and used a 66M-parameter DistilBERT model.Source: §8.3 · p. 15

8.4 FishMask

  • FishMask selects the highest-Fisher-information parameters for optimization, generally matching adapters but performing below LoRA and (IA)3.It requires gradients for all parameters across several batches and is computationally intensive on contemporary hardware because sparse operations lack support; it has been evaluated on BERT and T0-3B.Source: §8.4 · p. 15

8.5 DiffFit

  • DiffFit outperforms other PEFT approaches for diffusion models while enabling faster tuning and minimal degradation in inference and storage efficiency.It tunes biases, layer normalization, and embeddings while adding separate residual-mixing scaling parameters for self-attention and FFN layers.Source: §8.5 · pp. 15–16

9 Reparametrization-based methods

Reparametrization-based methods reduce trainable dimensionality through low-rank weight or gradient transformations, while extending this idea to adaptive rank allocation, magnitude-direction decoupling, and quantized backbones. Their tradeoffs span efficiency, expressivity, memory overhead, inference speed, and performance across model scales.

9.1 Intrinsic SAID

  • Intrinsic SAID reparametrizes updates in a low-rank subspace with Fastfood, but its O(D) memory complexity and updates to every parameter make it impractical for large networks.Larger models require lower-rank subspaces than smaller models to reach the same fine-tuning performance.Source: §9.1 · p. 16

9.2 LoRA

  • LoRA freezes pretrained weights and trains two low-rank matrices whose product updates the original weight matrix, achieving strong performance across models up to 175B parameters.It is commonly applied to attention projections, although applying it to all model weight matrices can improve performance; it outperforms BitFit and Adapters.Source: §9.2 · p. 17

9.3 KronA

  • KronA replaces LoRA’s matrix factorization with a Kronecker product, improving the rank-to-parameter tradeoff and enabling faster inference through implicit matrix-vector products.On GLUE, KronA performs on par or better than adapters, LoRA, and Compacter at 0.07% trainable parameters, while being significantly faster at inference; evaluation used models below 1B parameters.Source: §9.3 · p. 17

9.4 DoRA

  • DoRA decouples weight-update magnitude from direction, consistently improving over LoRA across tasks and models, especially in the low-rank (4, 8) regime.Its scaling can produce large directional changes with small magnitude adjustments, or the reverse, and it improves sample efficiency on instruction-tuning tasks.Source: §9.4 · p. 18

9.5 GLoRA

  • GLoRA extends LoRA by scaling and shifting weights or activations, and its merged parameters incur no additional inference cost while outperforming LoRA across vision and language tasks.With equal trainable parameters, GLoRA outperforms LoRA on VTAB-1K and shows better sample efficiency than LoRA and comparable PEFT methods.Source: §9.5 · pp. 18–19

9.6 AdaLoRA

  • AdaLoRA adaptively prunes singular-value components using loss-based importance scores, outperforming LoRA and other PEFT methods under parameter budgets but incurring memory overhead from tracking pruning variables.Experiments with DeBERTa-base and BART-large indicate that initial transformer layers need not use high-rank updates.Source: §9.6 · p. 19

9.7 GaLore

  • GaLore trains all model parameters while regularizing gradients in lower-dimensional spaces, reducing optimizer-state memory without approximating parameter matrices.For 60M–1B pretraining models, it achieves better C4 validation perplexity than LoRA and ReLoRA, and comparable but slightly worse perplexity than full-rank training with substantially lower memory use.Source: §9.7 · pp. 20–21

9.8 Quantization and LoRA

  • QLoRA reduces frozen-backbone memory through 4-bit NormalFloat, double quantization, and CPU-GPU optimizer-state paging, while later methods address low-precision performance discrepancies.Group-wise quantization avoids post-training quantization, and related methods include IR-QLoRA, LoftQ, L4Q, and LQ-LoRA.Source: §9.8 · p. 21

10 Hybrid Approaches

Hybrid PEFT methods combine complementary techniques to improve performance and efficiency while mitigating individual weaknesses. The section covers sparse adapters, mixed adapter–prompt designs, gated combinations, parameter sharing, and layer-wise method searches.

10 Hybrid Approaches

  • Hybrid methods combine the strengths of multiple techniques to improve PEFT performance and efficiency while mitigating their weaknesses.Source: §10 · p. 21

10.1 SparseAdapter

  • Large-Sparse adapters consistently outperform equally sized non-sparse adapters, but may incur higher training and inference costs depending on sparse-operation hardware.The method uses a large hidden dimension and prunes around 40% of values at initialization; computing the pruning mask may require gradients for all added parameters.Source: §10.1 · pp. 21–22

10.2 MAM Adapters

  • MAM Adapter outperforms BitFit and PromptTuning by a large margin and consistently surpasses LoRA, Adapters, and Prefix Tuning with 7% extra parameters.MAM combines a scaled parallel FFN adapter with soft prompts of length 200; experiments used models with fewer than 1 billion parameters.Source: §10.2 · p. 22

10.3 UniPELT

  • UniPELT significantly improves over individual LoRA, Adapters, and Prefix Tuning in low-data settings, while matching or exceeding them with more data.UniPELT gates LoRA, Prefix-tuning, and Adapters, and uses 1.3% trainable parameters on sub-billion-parameter BERT models.Source: §10.3 · pp. 22–23

10.4 Compacter

  • Compacter++ performs on par or better than adapters while adding only 0.05% parameters, compared with 0.8% for adapters.Compacter uses Kronecker products, low-rank matrices, and parameter sharing across layers; evaluations covered T5 Base and T0-3B.Source: §10.4 · pp. 23–24

10.5 S4

  • S4 consistently outperforms individual BitFit, Prefix Tuning, LoRA, and Adapters across architectures, model sizes, and tasks.S4 uses a spindle-shaped four-group layer allocation with different PEFT combinations and was evaluated with 0.5% trainable parameters in its search experiments.Source: §10.5 · p. 24

11 Comparison of PEFT Methods

The comparison evaluates PEFT methods across storage, memory, computation, inference overhead, and downstream performance, using fixed experiments spanning diverse methods, tasks, and T5 scales. Houlsby Adapters and LoRA generally match full tuning, while efficiency gains involve important speed and scalability trade-offs.

11.1 Experimental Comparison: Setup

  • The study compares 14 diverse PEFT methods plus full tuning across SuperGLUE and CNN-Dailymail on T5 models ranging from 0.7B to 11B parameters.The evaluation measures memory consumption, training speed, inference speed, and downstream performance under a fixed setup.Source: §11.1 · pp. 25–27

11.2 Comparison Results: Downstream Performance

  • Houlsby Adapters and LoRA consistently achieve full-tuning performance with little to no hyperparameter tuning, while several competing methods underperform their reported values.Parallel Adapter, Compacter, and KronA perform on par with the best methods in several cases, especially for 11B models.Source: §11.2 · pp. 28, 30, Table 4 · p. 29
  • Prefix Tuning and Prompt Tuning differ substantially: both converge slowly, but Prompt Tuning never surpasses the constant-prediction baseline and is highly seed-sensitive.Prefix Tuning reparametrizes its prefix through a fully connected network, whereas Prompt Tuning directly optimizes it.Source: §11.2 · p. 29
  • Pfeiffer Adapters perform up to 15 points worse than Houlsby Adapters, with the gap increasing as model scale grows.This contrasts with prior observations that inserting adapters only after feed-forward layers achieved similar performance.Source: §11.2 · p. 30

11.3 Comparison Results: Efficiency

  • All PEFT methods significantly reduce memory consumption, with savings ranging from 4GB for UniPELT on T5-Large to 71.5GB for Compacter++ on T5-11B.The largest saving enables T5-11B fine-tuning on one 40GB GPU instead of two 80GB GPUs and increases training speed by more than twofold.Source: §11.3 · p. 30, Table 5 · p. 31
  • PEFT can slow training on smaller models: LoRA applied to all T5-Large parameters is 20% slower than full tuning, while MAM, Compacter, and UniPELT are 5%, 20%, and 40% slower.These methods nevertheless improve memory consumption.Source: §11.3 · p. 30
  • Adding trainable parameters slows inference by 20–60% across model scales, although unmerged reparametrization methods would match regular inference speed after merging.Among additive methods, Pfeiffer adapters and (IA)3 provide the best inference speeds.Source: §11.3 · pp. 30, 32, Table 5 · p. 31
  • Kronecker-based Compacter and KronA optimize fewer than 0.5 million parameters versus LoRA’s 20 million on T5-11B, yet all use approximately 28.6GB of GPU memory.KronA nevertheless improves training and inference speed over LoRA, likely through efficient Kronecker-vector products.Source: §11.3 · p. 32

12 Challenges and guidelines

The paper identifies inconsistent reporting, evaluation, implementation, and comparison practices as barriers to reliable PEFT assessment. It recommends explicit efficiency reporting, evaluation across model scales, reusable implementations, and standardized benchmarks.

Challenges

  • PEFT comparisons are hindered by inconsistent reporting of parameter counts, efficiency measures, model scales, implementations, and evaluation settings.The paper frames these inconsistencies as challenges to direct comparison and meaningful conclusions.Source: §12 · pp. 32–35

Reporting parameter count

  • Parameter counts should distinguish trainable parameters, changed parameters, and update rank, because these quantities are not equivalent and trainable count only imperfectly predicts memory efficiency.IntrinsicSAID changes all model parameters despite learning a low-rank transformation, while DiffPruning updates 0.5% but trains 200%; Ladder-side Tuning uses less RAM than LoRA or BitFit despite training more parameters.Source: §12 · p. 33

Reporting efficiency

  • Efficiency evaluations should report training RAM, training and inference token throughput, storage, and stage-wise metrics rather than parameter count alone.Memory and speed can vary substantially at similar parameter counts or RAM usage; inference overhead and multi-task storage also differ across PEFT categories.Source: §12 · pp. 33–34, Table 6 · p. 35

Model sizes

  • Efficiency and downstream performance should be measured across multiple model scales, especially above 1B and below 20B parameters.Layer Norm tuning was the only consistently efficient method across scales in the experiments, while other methods varied considerably; larger models may require fewer updated parameters.Source: §12 · p. 34

Method Implementation

  • PEFT papers should provide easy-to-use, readable, reusable implementations, since copied or poorly documented codebases hinder adoption and reproducibility.LoRA and Compacter are highlighted as user-friendly and adaptable implementations.Source: §12 · pp. 34–35

Comparison

  • Methods should be compared with popular and conceptually similar approaches using standardized benchmarks and unified metrics to enable meaningful conclusions.KronA (Edalati et al., 2022), UniPELT (Mao et al., 2021), LST (Sung et al., 2022), and MAM (He et al., 2022a) report useful but still non-comparable metric combinations; the paper calls for community benchmarks and competitions.Source: §12 · pp. 34–35

13 Discussion · Appendix A. Prompts

The discussion identifies low-rank reparameterization as a highly successful route to parameter-efficient fine-tuning while emphasizing persistent hyperparameter sensitivity and opportunities for adaptive, cross-disciplinary methods. Appendix A lists prompts used for BoolQ, CB, COPA, and RTE experiments.

13 Discussion

  • Low-bit quantization has democratized large-model inference, while PEFT extends this accessibility by enabling model modification under modest compute budgets.The discussion frames PEFT as the next step beyond inference toward modifying large language models.Source: §13 · pp. 35–36
  • Despite practical scalability for Adapters, Prompt Tuning, LoRA, and (IA)3, PEFT often struggles to match full fine-tuning because of hyperparameter sensitivity.Optimal PEFT hyperparameters can differ substantially from full-fine-tuning settings, with generally higher optimal learning rates; tuning may be costly at 20-100B scale.Source: §13 · p. 36
  • The discussion recommends investigating hyperparameter effects, establishing reasonable defaults, and reducing sensitivity through approaches such as pre-training new parameters (Vu et al., 2021).These efforts are motivated by the cost of PEFT hyperparameter optimization for large models.Source: §13 · p. 36
  • Low-rank reparameterizations, including LoRA-style and Kronecker-product methods, reduce trainable parameters with minimal extra computation.The paper proposes exploring reparameterizations with favorable trainable-parameter-count-to-rank ratios.Source: §13 · p. 36
  • Adaptive parameter allocation across transformer layers could improve parameter efficiency and accuracy because models process inputs differently at different layers.Most PEFT methods apply parameters uniformly, motivating layer-aware systems with varying parameter counts.Source: §13 · p. 36
  • Because large-language-model fine-tuning shares memory, computation, and energy constraints with edge machine learning, cross-disciplinary exchange may accelerate PEFT innovation.Quantization and pruning are cited as techniques transferred from edge machine learning to large language models.Source: §13 · p. 36

Appendix A. Prompts

  • Appendix A provides experimental prompts for BoolQ, CB, COPA, and RTE, covering yes/no identification, textual entailment, plausible cause-or-effect selection, and entailment classification.The prompts specify the task instructions used in the experimental comparison.Source: Appendix A · p. 50

Appendix B. PEFT Comparison: full experimental results

Appendix B reports raw scores across T5 Large, 3B, and 11B models, multiple PEFT methods, four datasets, and several random seeds. Learning rates were swept over 1e-3, 1e-4, and 5e-5, while weight decay changed results by less than 0.01.

Experimental setup

  • Each model-method-dataset combination used a learning-rate sweep over 1e-3, 1e-4, and 5e-5, while weight-decay sweeps changed results by less than 0.01.The reported raw results were used to produce Tables 4 and 5.Source: Appendix B · p. 50

T5 3B

  • For T5 3B, Scaled Parallel scored 90.61–91.34 on RTE, 87.00 on COPA, and 88.90–89.02 on BoolQ, whereas Ln tuning scored 89.00 on COPA.The reported CNN scores were 30.03 for Scaled Parallel and 28.43 for Ln tuning.Source: Appendix B · p. 51

T5 11B

  • T5 11B LoRa (all) reached 94.00 on COPA and 93.56 on BoolQ, while scoring 92.78 on RTE and 28.75 on CNN.These raw results cover seeds 0, 1, and 42 where reported; some CNN entries are missing.Source: Appendix B · p. 53
  • Across T5 11B methods, Compacter and Compacter++ reached 91.09 and 91.58 on BoolQ, while Prefix_tuning and MAM remained near 53–71 across RTE, COPA, and BoolQ.Compacter scored 29.52 on CNN, and Compacter++ scored 29.50 where reported.Source: Appendix B · pp. 54–55

T5 Large

  • On T5 Large, Full tuning scored 87.00 on RTE, 73.00 on COPA, 85.38 on BoolQ, and 26.79 on CNN at seed 42.Other methods varied substantially, including Compacter++ at 85.56 on RTE and 84.07 on BoolQ, and Prefix_tuning at 14.92–17.73 on CNN.Source: Appendix B · pp. 55–57

Related papers

Loading 2303.15647v2…