PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Effectiveness of Parameter-Efficient Fine-Tuning

Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, Nigel Collier

arXiv:2211.15583v1cs.CLcs.AIcs.LG

TL;DR

Parameter-efficient fine-tuning performs well and is often more stable, but the reasons for these properties and the best tunable-parameter choices remain unclear. The paper unifies existing methods as sparse fine-tuned models, analyzes their stability and generalization, and proposes SAM to select parameters analytically. Experiments report strong performance for SAM and support the theoretical analysis.

  • Problem

    The effectiveness, stability, and tunable-parameter selection of parameter-efficient fine-tuning methods remain insufficiently understood, with selection being NP-hard.

  • Method

    The paper categorizes existing methods, models them as sparse fine-tuned models, analyzes stability and generalization, and uses SAM to select parameters through an analytically solvable approximation.

  • Results

    SAM achieves the best data perturbation stability and overall performance among the evaluated models, while experiments support the theoretical analysis.

  • Takeaways & Limitations

    Sparsity is presented as a regularization that controls stability and contributes to better generalization, while SAM addresses data use and projection discontinuity in parameter selection.

  • Takeaways & Limitations

    The sparse fine-tuned model definition assumes the fine-tuned model has the same structure as the pre-trained model, requiring equivalent models for structurally modified methods.

Abstract

from arXiv · show

Fine-tuning pre-trained models has been ubiquitously proven to be effective in a wide range of NLP tasks. However, fine-tuning the whole model is parameter inefficient as it always yields an entirely new model for each task. Currently, many research works propose to only fine-tune a small portion of the parameters while keeping most of the parameters shared across different tasks. These methods achieve surprisingly good performance and are shown to be more stable than their corresponding fully fine-tuned counterparts. However, such kind of methods is still not well understood. Some natural questions arise: How does the parameter sparsity lead to promising performance? Why is the model more stable than the fully fine-tuned models? How to choose the tunable parameters? In this paper, we first categorize the existing methods into random approaches, rule-based approaches, and projection-based approaches based on how they choose which parameters to tune. Then, we show that all of the methods are actually sparse fine-tuned models and conduct a novel theoretical analysis of them. We indicate that the sparsity is actually imposing a regularization on the original model by controlling the upper bound of the stability. Such stability leads to better generalization capability which has been empirically observed in a lot of recent research works. Despite the effectiveness of sparsity grounded by our theory, it still remains an open problem of how to choose the tunable parameters. To better choose the tunable parameters, we propose a novel Second-order Approximation Method (SAM) which approximates the original problem with an analytically solvable optimization function. The tunable parameters are determined by directly optimizing the approximation function. The experimental results show that our proposed SAM model outperforms many strong baseline models and it also verifies our theoretical analysis.

1 Introduction

Parameter-efficient fine-tuning reduces storage demands by tuning only a small portion of a pre-trained model, while achieving strong performance and motivating questions about its stability and parameter selection.

  • Motivation: Fine-tuning increasingly large models separately for each downstream task makes storing fully fine-tuned parameters inefficient.The paper highlights models growing to billions or trillions of parameters.
  • Motivation: Parameter-efficient methods address this inefficiency by tuning and storing only a small part of the original parameters for each task.
  • Open Questions: Existing parameter-efficient methods often achieve surprisingly good performance and greater stability than fully fine-tuned models.Some studies also report better overall scores on particular tasks.
  • Contributions: The paper categorizes these methods as random, rule-based, or projection-based according to how they select tunable parameters.
  • Contributions: Its contributions combine a unified sparse fine-tuned model, theoretical analyses of stability and generalization, the SAM selection method, and experiments.

2 Unified View of Parameter Efficient Fine-tuning

The paper unifies parameter-efficient methods as sparse fine-tuned models, including structurally modified methods through equivalent models, and groups parameter selection into three categories.

  • 2.1 Sparse Fine-tuned Model: A sparse fine-tuned model changes at most a fraction p of a pre-trained model’s parameters while retaining the original structure.The paper later uses equivalent models to extend this analysis to methods with additional structures.
  • 2.1 Sparse Fine-tuned Model: The unified parameterization represents tuned parameters as θ = θ0 + M∆θ, with a binary diagonal mask selecting which parameters change.The mask contains exactly ⌊mp⌋ selected parameters for sparsity coefficient p.
  • 2.1 Sparse Fine-tuned Model: Equivalent models preserve the original model’s outputs for every input without requiring the same structure.This lets the sparse fine-tuning analysis cover Adapter and LoRA.
  • 2.2 Method Categories: The paper classifies parameter-efficient methods as random, rule-based, or projection-based according to how they choose the tunable mask.Problem (1) is NP-hard because the mask’s feasible region is nonconvex.
  • 2.2 Method Categories: Random approaches select parameters independently of task-specific data, whereas rule-based approaches use predefined choices such as biases or large-magnitude weights.The rule-based category includes BitFit, MagPruning, Adapter, and LoRA.
  • 2.2 Method Categories: Projection-based approaches use task-specific data to optimize a mask, then project it onto a feasible L0-ball region.Examples include DiffPruning and ChildPruning.
  • 2.2 Method Categories: Because the L0-ball is nonconvex, small mask perturbations can produce very different projections and unstable parameter selections.For p = 0.5, nearly identical masks can project to diag{0, 1} versus diag{1, 0}.

3 Theoretical Analysis of the Sparse Fine-tuned Model

The paper unifies parameter-efficient methods as sparse fine-tuned models and argues that sparsity regularizes the original model, improving stability and potentially generalization.

  • Unified Sparse Fine-tuned Model: Sparse fine-tuned models update only a fraction of pre-trained parameters, and the analysis applies to random, rule-based, and projection-based categories matching this definition.The theory assumes the fine-tuned model retains the pre-trained model’s structure.
  • Sparse Fine-tuned Model as a Regularizer: Optimizing the sparse fine-tuning objective implies optimizing an upper bound containing a regularization term that penalizes changes to untuned parameters.The term is ∥(I − M)(θ − θ0)∥2.
  • Stability Analysis: The stability analysis uses Pointwise Hypothesis Stability, which measures how model output changes when a training sample is removed.The analysis assumes A(Si) is close to A(S) and a positive-semidefinite Hessian at A(S).
  • Stability Analysis: As the sparsity parameter p decreases, the upper bound on pointwise hypothesis stability also decreases.The paper interprets this as sparse models having better stability.
  • Generalization Analysis: The generalization-error upper bound decreases with sparsity, but insufficient tunable parameters can increase empirical error and produce a decrease-then-increase pattern.The bound depends on both empirical error and sparsity.

4 Second-order Approximation Method

The paper proposes SAM to select tunable parameters using task-specific data while avoiding the projection discontinuity problem. It approximates the optimization analytically and fixes the selected mask during training.

  • Motivation: Random and rule-based methods are robust to noise perturbation but use the same fixed parameters across tasks without task-specific data information.Projection-based approaches are introduced as another existing category, while SAM targets the parameter-selection problem.
  • Second-order Approximation Method: SAM uses task-specific data information to choose a parameter mask while avoiding the projection discontinuity problem.It replaces random or rule-based selection with a second-order approximation of the original optimization problem.
  • Second-order Approximation Method: The method approximates the Hessian with a diagonal positive-semidefinite matrix because computing the full or diagonal Hessian is expensive for large neural models.The diagonal approximation is denoted H = diag{h1, h2, · · ·, hn}.
  • Optimization Formulation: The parameter mask is constrained to select exactly ⌊mp⌋ diagonal entries, with each mask entry restricted to 0 or 1.The off-diagonal mask entries are fixed at zero.
  • Parameter Selection: SAM selects the top ⌊mp⌋ parameters according to |∇L(θ0)i^2| and keeps the selected parameters fixed during optimization.The selection uses the gradient at the pre-trained parameters θ0.

5 Experiments

Experiments evaluate the setup, theoretical predictions, projection discontinuity, stability–performance relationship, sparsity effects, and robustness to data perturbations.

  • Experimental Setup: Table 1 reports means, standard deviations, and score ranks across experiments run 10 times with different random seeds.The original development set is used as the test set, while 10% of training data is reserved for early stopping.
  • Main Experiment: Most parameter-efficient models outperform FullTuning and are more stable, while Random also exceeds FullTuning, supporting sparsity’s effectiveness.SAM outperforms several baselines on several tasks and ranks in the top three on most tasks.
  • Projection Discontinuity Problem: Changing DiffPruning’s mask every 600 steps returns training error to nearly its initial loss, demonstrating severe disruption from projection discontinuity.The result is illustrated on the CB task.
  • Relation between Stability and Overall Performance: Stability rank and overall-performance rank are positively correlated; Spearman’s ρ = 0.4356 with p-value= 0.000014 < 0.05.Figure 4 plots stability rank against overall performance rank, with line length showing standard deviation.
  • Effectiveness of Sparsity: As sparsity decreases, most tasks become more stable with better generalization, but below a threshold instability and performance degradation increase sharply.The experiment varies SAM sparsity across {0.0002, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2}.
  • Data Perturbation Stability: Across 10 datasets formed by removing 10% of training samples, SAM retains the best data-perturbation stability and overall performance.The setting otherwise matches the main experiments.

6 Related Works

Related work frames parameter-efficient fine-tuning as a response to the storage and stability costs of full-model tuning and groups methods by parameter-selection strategy.

  • Background: Full-model fine-tuning creates a separate large model for each task and has been reported as unstable.This motivates parameter-efficient alternatives.
  • Parameter-Efficient Fine-Tuning: The paper categorizes parameter-efficient methods into random, rule-based, and projection-based approaches.Examples include Random and Mixout; BitFit, MagPruning, Adapter, and LoRA; and DiffPruning and related methods.
  • Related Generalization Methods: Other related approaches regularize parameters, use adversarial training, or exploit task information to improve generalization.These lines situate the paper among broader generalization-oriented methods.

7 Conclusions

The paper unifies parameter-efficient methods as sparse fine-tuned models, analyzes their stability and generalization, and introduces SAM to address parameter-selection weaknesses.

  • Conclusions: Most parameter-efficient methods are categorized as random, rule-based, or projection-based and unified as sparse fine-tuned models.The unified view supports theoretical analysis of stability and generalization.
  • Conclusions: Random and rule-based methods do not use task data, whereas projection-based methods suffer from projection discontinuity.SAM is proposed to alleviate both problems.
  • Conclusions: Extensive experiments support the theoretical analysis and the effectiveness of the proposed SAM models.

A.1 Proof of Proposition 1

The proof reformulates the constrained sparse fine-tuning problem through its Lagrangian and derives an upper-bounded regularized objective.

  • Proof of Proposition 1: Proposition 1 states that optimizing Problem (2) implies optimizing an upper bound of a regularized problem.
  • Proof of Proposition 1: The proof writes the Lagrangian of Problem (2), introducing λ as the Lagrangian multiplier.
  • Proof of Proposition 1: The constrained problem is therefore equivalent to solving the resulting regularized optimization problem.

A.2 Proof of Theorem 1

The proof derives pointwise hypothesis stability for a regularized learning algorithm by comparing solutions with and without one training sample. It uses Lipschitz loss, local closeness, and positive-semidefinite Hessian assumptions.

  • The proof replaces a convexity assumption with Taylor expansion, making the argument more suitable for neural networks but tying the bound to local-minimum geometry.The resulting constant is determined by the specific shape around the local minima.
  • The analysis assumes a ρ-Lipschitz loss, close solutions A(S_i) and A(S), and a positive-semidefinite Hessian at A(S).The Hessian is represented through its singular values, including Λ_min.
  • The algorithm minimizes empirical loss plus a quadratic penalty λ∥w−w0∥2 around the original parameters.This regularized objective defines A(S) through an arg-min formulation.
  • The proof compares the regularized objectives on S and S_i, differing only through the removed sample’s loss contribution.It then substitutes the minimizers A(S_i) and A(S) into the comparison.

A.3 Proof of Theorem 2

The generalization proof invokes a stability-based lemma for bounded losses and applies it to relate empirical and expected error. Its scope is expressed through pointwise hypothesis stability and a high-probability guarantee.

  • The proof uses the earlier stability result together with the stability-to-generalization lemma to obtain a high-probability bound.The supplied theorem statement specifies a constant C and probability 1−δ.
  • Lemma 2 provides a high-probability generalization statement for any learning algorithm with pointwise hypothesis stability β and bounded loss 0≤c(y,y′)≤C.The guarantee holds with probability 1−δ.
  • Theorem 2 defines generalization error as expected loss and empirical error as the average training loss.These quantities are denoted R(A,S) and R̂(A,S), respectively.

A.4 Proof of Theorem 3

This section covers the method’s computational and statistical evaluation, including training time, tunable-parameter ratios, significance tests, and stated limitations. SAM is described as competitive but incurs additional training cost under the current masked implementation.

  • Training Time Analysis: Adapter and LoRA outperform FullTuning in training-time analysis because they tune only a few newly added parameters.Other parameter-efficient models can take longer than FullTuning when implemented with masks.
  • Training Time Analysis: SAM outperforms all other models except Adapter and LoRA in the reported training-time comparison because it converges faster.Despite faster convergence, SAM still takes longer than FullTuning because it requires a mask.
  • Training Time Analysis: Parameter-efficient models may require more training time while using less storage because only changed parameters are stored.The paper identifies this storage advantage as useful when there are many downstream tasks.
  • Training Time Analysis: Table 3 reports training time in hours for the compared models.The analysis accounts for early stopping, so models may use different numbers of running steps.
  • Experimental Setup: The experiments aim to keep tunable-parameter ratios comparable, using a 0.5% tunable ratio for SAM and several other models.Adapter and BitFit have fixed tunable parameters, while LoRA follows its official setting.
  • Significance Tests: The t-tests compare SAM with other models using t-statistics and p-values, treating p<0.05 as statistically significant.Tables 5 and 6 report these comparisons for the main and perturbation-stability experiments.
  • Limitations and Future Directions: The authors identify two limitations: second-order approximation may leave room for improvement, and gradient masking makes SAM slightly slower than FullTuning.Future work includes alternative target-function assumptions and faster implementation strategies.
Loading 2211.15583v1…