Revisiting Parameter-Efficient Tuning: Are We Really There Yet?

1 Introduction

• Full-model tuning of hundreds of millions or billions of parameters per task is described as time-consuming and resource-intensive.PETuning instead adjusts lightweight trainable parameters while keeping most pretrained parameters frozen.Source: §1 · p. 1
• Using development data for both early stopping and reporting results compromises fair comparisons and can produce misleading conclusions.The authors identify this practice as a failure to assess generalization.Source: §1 · p. 2
• The study re-examines PETuning performance across tasks and resource levels and investigates method stability.It also proposes a model-selection practice less prone to overfitting.Source: §1 · p. 2
• PETuning can suit low-resource tasks, but no method consistently outperforms finetuning; performance is weaker in medium- and high-resource settings.The paper reports that PETuning methods fall behind finetuning across high-resource tasks.Source: §1 · p. 2
• All examined PETuning methods show seed-related instability, with variation arising from weight initialization and training-data order.The paper identifies trainable-parameter count and training iterations as factors in stability.Source: §1 · p. 2

2 The Broken Protocol

• The proposed rigorous protocol strictly separates validation and testing to examine the effects of development-set checkpoint selection and testing.The prior practice was common partly because it enabled quick results from large-scale experiments.Source: §2 · p. 3
• The controlled comparison splits RTE's original development set into dev.1 and dev.2, then uses dev.2 as test data in both protocols.RTE1−2 selects checkpoints on dev.1; RTE2−2 selects them on dev.2, with 50 epochs and early stopping after 10 non-improving epochs.Source: §2 · p. 3
• Using one set for both development and testing substantially boosts PETuning results, but this boost does not establish better generalization.For prefix tuning, the reported gain is approximately 10% with either evaluation loss or accuracy as the stopping metric.Source: §2 · p. 3, Table 1 · p. 3
• Reusing the test set during training leaks test information and can erase the gap between checkpoint selection and performance on unseen data.The authors characterize that gap as a measure of generalization ability.Source: §2 · pp. 3–4

3.1 Experimental Setup

• Experiments cover 12 GLUE and SuperGLUE datasets divided into low-, medium-, and high-resource groups, with separate dev and test sets and multiple random-seed runs.The training set is split 90%/10% into new training and development sets, while the original development set is used for testing; runs number 20 for low- and medium-resource tasks and 10 for high-resource tasks.Source: §3.1 · p. 4

3.2 Analysis of Performance

• Across resource levels, PETuning methods often outperform finetuning on low-resource tasks, are at best comparable on medium-resource tasks, and fall behind on high-resource tasks.Table 3 summarizes the resource-stratified comparison across multiple runs.Source: Table 3 · p. 5, §3.2 · p. 5
• Adapter, LoRA, and BitFit outperform finetuning on low-resource tasks, while only Adapter and BitFit remain comparable in medium-resource tasks and all PETuning methods trail finetuning in high-resource tasks.LoRA gains about 8% on average in low-resource tasks; Adapter obtains the highest scores among PETuning methods on high-resource tasks.Source: §3.2 · pp. 5–6
• Prefix tuning's best runs are competitive with finetuning on many tasks, but its average performance falls as results vary across random seeds.The authors attribute its poor average performance to instability and note that prior work often reported a single run.Source: Figure 3 · p. 5, §3.2 · p. 6

3.3 Analysis of Stability

• Changing either weight initialization or training-data order can individually produce large standard deviations, making PETuning instability sensitive to both randomness sources.The study separates these sources using different random seeds for initialization and data order.Source: §3.3 · p. 7
• Increasing PETuning trainable parameters broadens the performance distribution and is associated with greater instability.The probability-density curves flatten as parameter scale increases; the authors conjecture that smaller models may converge quickly to rough global minima but underfit the data manifold.Source: §3.3 · pp. 7–8, Figure 5 · p. 7
• PETuning standard deviations decline after an initial ascent as training steps increase, while fixed-epoch training gives smaller datasets fewer iterations and can leave instability higher.Prefix tuning has a longer ascent stage and may need more iterations for stability, but longer training on small datasets does not necessarily improve performance.Source: §3.3 · p. 8, Figure 6 · p. 8

• Prior studies examined Adapter effectiveness across scales and proposed a robust Adapter training strategy.Source: §4 · p. 8
• Other work proposed unified views connecting different PETuning methods.Source: §4 · p. 8
• Reliable comparisons of off-the-shelf PETuning methods for stability and effectiveness remained lacking.The paper identifies this gap as its contribution.Source: §4 · p. 8

• PETuning methods are not consistently competitive with finetuning; prefix tuning performs poorly across tasks, and most methods fare worse in higher-resource settings.Source: §5 · p. 9
• Using the same dataset for early stopping and evaluation can produce unreliable conclusions, especially alongside PETuning instability.The authors report inflated results and overly optimistic estimates under this practice.Source: §5 · p. 9
• A protocol that strictly separates validation and test sets underpins the paper’s re-evaluation of PETuning performance and stability.Source: §5 · p. 9
• PETuning instability is sensitive to both weight initialization and training-data order.Source: §5 · p. 9
• Fewer trainable parameters and more training iterations can usually improve PETuning stability.Source: §5 · p. 9

Limitations

• 20 runs for low- and medium-resource tasks and 10 for high-resource tasks estimate stability, measured by standard deviations across runs.The authors note that more runs would make the stability estimates more precise and could consolidate the conclusions.Source: Limitations · p. 9
• The study simulated a standard train/dev/test split using available GLUE and SuperGLUE train and dev sets.Access to the real test data would make the conclusions more comparable to existing work.Source: Limitations · p. 9
• The study covers four representative PETuning methods, and its conclusions may not generalize to all existing or upcoming models in this fast-moving field.Source: Limitations · p. 9

A PETuning Methods

• PETuning methods keep most pretrained parameters frozen and tune lightweight additions or a fraction of the model’s parameters for downstream tasks.Source: A PETuning Methods · p. 12
• The methods either add trainable parameters at model, feature, or parameter level, or tune partial base-model parameters.Figure 7 contrasts Adapter, Prompt-tuning, Diff Pruning, LoRA, and BitFit across these designs.Source: A PETuning Methods · p. 12, Figure 7 · p. 12

A.1 Model-Level

• Adapters insert newly initialized modules at each transformer layer, typically using a bottleneck down-projection and up-projection, while keeping the pretrained model fixed during tuning.Source: §A.1 · p. 12

A.2 Feature-Level

• Prompt-tuning adds tunable prefix or suffix vectors to input features or transformer-layer inputs and tunes only those prompts.Source: §A.2 · p. 12

A.3 Parameter-Level

• Diff-pruning adds a sparse trainable diff vector to pretrained parameters, while LoRA approximates weight updates with trainable low-rank matrices.Source: §A.3 · p. 12

A.4 Partial Finetuning

• BitFit tunes pretrained-model bias terms without introducing additional parameters.Partial finetuning tunes a fraction of the model’s parameters.Source: §A.4 · p. 12

• General task and hyperparameter settings are introduced here, while specific data and evaluation setups are deferred to Sections 2 and 3.Source: B General Experimental Setup · p. 13

B.1 Task Setup

• The study covers 12 GLUE and SuperGLUE tasks spanning inference, question answering, paraphrasing, sentiment, similarity, word-sense disambiguation, and coreference.The tasks are divided into low-, medium-, and high-resource groups by training-set size.Source: §B.1 · p. 13
• Low-resource tasks have fewer than 1k training examples, medium-resource tasks have 1k–10k, and high-resource tasks have more than 10k.The groups include CB, COPA, and WSC; RTE, MRPC, WiC, STS-B, and BoolQ; and SST-2, QNLI, QQP, and MNLI, respectively.Source: §B.1 · p. 13

B.2 Hyperparameter Setup

• Hyperparameters are grid-searched across learning rates and batch sizes, with method-specific searches for Adapter reduction factor, prefix-tuning prompt length, and LoRA scaling factor and rank.All PETuning additions are randomly initialized to avoid giving methods an initialization advantage from extra post-pretraining.Source: §B.2 · p. 13
• Training uses up to 50 epochs and stops after 10 worse-performing epochs on a new development set; evaluation loss is the common stopping metric in Section 3.Section 2 also examines evaluation accuracy as a stopping metric for RTE.Source: §B.2 · p. 13

C.1 The Same Task with Different Training Data Sizes

• Standard deviation is reported to be substantially proportional to training-data size across the sampled scales.Figure 8 plots means and standard deviations across data scales; Table 5 reports results for the sampled training-data settings.Source: §C.1 · p. 13, Table 5 · p. 14

C.2 Finetuning is More Prone to Overfit

• Finetuning converges faster than PETuning on low-, medium-, and high-resource examples, especially on COPA, where training takes fewer than 100 steps.The authors suggest faster convergence may contribute to overfitting on low-resource tasks and poorer performance there.Source: §C.2 · p. 14, Figure 9 · p. 14

C.3 High Stability on Fewer Trainable Parameters.

• Adapter and prefix tuning have their lowest standard deviations at the small parameter scale, while LoRA’s standard deviation is slightly lower at the medium scale.Across 20 runs, results generally suggest higher PETuning stability with fewer trainable parameters, although potential outliers under the limited run count may increase variance.Source: §C.3 · pp. 14–15, Figure 10 · p. 15, Figure 11 · p. 15