PaperlayerView exact revision on arXiv

Source-linked AI summary

Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning

Naibin Gu, Peng Fu, Xiyu Liu, Bowen Shen, Zheng Lin, Weiping Wang

arXiv:2406.03792v1cs.CL

TL;DR

Existing PEFT fine-tuning remains inefficient because large foundation models and empirically added trainable modules contain task-specific redundancy. Light-PEFT estimates and prunes redundant parameters in both components during early training, achieving faster, lower-memory fine-tuning with comparable performance across diverse tasks and models.

  • Problem

    Existing PEFT methods retain redundant foundation-model computation and trainable-module parameters, limiting fine-tuning efficiency.

    Source: §1 · pp. 1–2

  • Method

    Light-PEFT uses Masked Early Pruning of the Foundation Model and Multi-Granularity Early Pruning of PEFT to estimate and structurally prune redundancy early in training.

    Source: §1 · p. 2

  • Results

    Light-PEFT maintains comparable performance while improving training and inference speed and reducing memory across GLUE, SuperGLUE, QA tasks, and multiple models.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The framework produces lighter foundation models and PEFT modules that retain PEFT’s task-specific, plug-and-play use while improving fine-tuning efficiency.

    Source: §1 · p. 2

    The study focuses primarily on single-task fine-tuning; extending early redundancy estimation and pruning to multi-task learning remains future work.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Parameter-efficient fine-tuning (PEFT) has emerged as the predominant technique for fine-tuning in the era of large language models. However, existing PEFT methods still have inadequate training efficiency. Firstly, the utilization of large-scale foundation models during the training process is excessively redundant for certain fine-tuning tasks. Secondly, as the model size increases, the growth in trainable parameters of empirically added PEFT modules becomes non-negligible and redundant, leading to inefficiency. To achieve task-specific efficient fine-tuning, we propose the Light-PEFT framework, which includes two methods: Masked Early Pruning of the Foundation Model and Multi-Granularity Early Pruning of PEFT. The Light-PEFT framework allows for the simultaneous estimation of redundant parameters in both the foundation model and PEFT modules during the early stage of training. These parameters can then be pruned for more efficient fine-tuning. We validate our approach on GLUE, SuperGLUE, QA tasks, and various models. With Light-PEFT, parameters of the foundation model can be pruned by up to over 40%, while still controlling trainable parameters to be only 25% of the original PEFT method. Compared to utilizing the PEFT method directly, Light-PEFT achieves training and inference speedup, reduces memory usage, and maintains comparable performance and the plug-and-play feature of PEFT.

1 Introduction

Large foundation models make fine-tuning increasingly costly, while existing PEFT methods retain redundancy in both the foundation model and added trainable modules. Light-PEFT addresses these inefficiencies by estimating and pruning redundant parameters early, achieving faster and more memory-efficient fine-tuning with comparable performance.

Motivation

  • Growing model scale increases fine-tuning cost and makes maintaining a separate full-model copy for each task expensive.These costs motivate parameter-efficient alternatives.Source: §1 · p. 1

Background

  • PEFT reduces computational requirements by freezing most foundation-model parameters and fine-tuning only a small number of task-specific parameters.It also avoids storing a complete model copy for each task and supports inference by switching PEFT modules.Source: §1 · p. 1

Motivation

  • Existing PEFT methods remain inefficient because large foundation models are redundantly used for task-specific fine-tuning, while quantization does not reduce parameter count and requires dequantization during training.Structured-pruning methods have often prioritized inference efficiency and may increase training costs.Source: §1 · p. 1
  • As foundation models grow, empirically added trainable modules also grow and introduce redundant parameters during fine-tuning.Uniformly assigning the same rank to every weight is unnecessary for each task, motivating task-specific allocation and pruning.Source: §1 · pp. 1–2

Proposed framework

  • Light-PEFT simultaneously estimates redundant foundation-model and PEFT-module parameters early in training, then removes them through structured pruning.Its two methods are Masked Early Pruning of the Foundation Model and Multi-Granularity Early Pruning of PEFT.Source: §1 · p. 2

Empirical results

  • 39% lower training memory and 1.6× faster training were achieved across models, PEFT structures, and GLUE, SuperGLUE, and question-answering benchmarks.The framework also reduced inference memory by 48% and increased inference speed to 1.6× while maintaining comparable performance.Source: §1 · p. 2

2 Related Works

Related work improves PEFT modules, training memory, or inference efficiency, but commonly estimates rank allocation during training or incurs additional training costs. Light-PEFT instead uses early-stage structured pruning to reduce PEFT training and inference costs together.

2.1 Parameter-Efficient Fine-Tuning

  • Earlier PEFT improvements adapt ranks or configurations during training, but their continual estimation does not improve actual fine-tuning training efficiency.Light-PEFT estimates rank allocation early and then trains with pruned modules.Source: §2.1 · p. 2
  • Prior pruning-and-PEFT methods mainly target inference efficiency and may require higher training time and memory than original PEFT methods.APT reduces the training cost of CPET, but the cited comparison still places these methods above original PEFT training costs.Source: §2.1 · p. 2
  • Unstructured pruning combined with PEFT often provides little practical hardware acceleration, whereas structured pruning is more directly suited to hardware speedups.The related methods include PST, DSEE, SPAs, CPET, and APT.Source: §2.1 · p. 2
  • Light-PEFT applies early-stage structured pruning to reduce original PEFT training speed and memory costs while improving inference efficiency.The approach is orthogonal to memory-reduction methods based on quantization, activation reduction, or reversible structures.Source: §2.1 · pp. 2–3

2.2 Structured Pruning of Models

  • Structured pruning targets components such as Transformer attention heads and feed-forward dimensions, but many methods incur extra training costs to obtain smaller inference models.Early-ticket studies motivate exploring early pruning for PEFT and generative foundation models.Source: §2.2 · p. 3

3 Preliminaries

The paper studies Adapter and LoRA as representative PEFT structures and examines how foundation-model size and PEFT-module organization affect training efficiency. Larger foundation models increase forward-pass time and memory, while adding more modules is more costly than increasing one module’s rank.

3.1 Parameter-Efficient Fine-Tuning

  • Adapter inserts bottleneck MLPs after each attention and feed-forward sub-layer, using down- and up-projections with bottleneck dimension r much smaller than hidden size d.The module applies a nonlinear activation between its projections.Source: §3.1 · p. 3
  • LoRA adds trainable down- and up-projection modules in parallel to a fixed weight matrix, with a scaling factor s applied to the update.The input to the linear weight matrix is denoted X.Source: §3.1 · p. 3

3.2 PEFT Training Efficiency

  • As foundation-model size increases, forward and backward pass time and required memory increase in OPT FP16 experiments.PEFT reduces gradient computation but still propagates through all foundation-model parameters, while model weights and activations remain memory costs.Source: Figure 1 · p. 3, §3.2 · p. 3
  • Adding more structured PEFT modules increases forward and backward time significantly more than increasing the rank of one module with the same module count.The Figure 2 comparison uses RoBERTa-Large FP32 and distinguishes Q, K, V, and O attention matrices.Source: Figure 2 · p. 4, §3.2 · p. 4

4 Methodology

Light-PEFT estimates redundancy early in both the frozen foundation model and trainable PEFT modules, then prunes them to enable more efficient fine-tuning. Its methods combine masked structured pruning of foundation-model components with coarse- and fine-grained pruning of PEFT modules and ranks.

4.1 Overview of Light-PEFT

  • Light-PEFT simultaneously estimates redundancies during early training and prunes both the foundation model and PEFT modules before continued fine-tuning.The early estimation stage uses t′ steps with t′ ≪ t, producing lighter components while preserving PEFT's plug-and-play use.Source: §4.1 · p. 4, Figure 3 · p. 5

4.2 Masked Early Pruning of Foundation Model

  • Masked Early Pruning introduces trainable masks for attention heads and FFN intermediate dimensions, jointly optimized with PEFT modules using L1 regularization.Masks begin at 1, and structured pruning removes components according to mask magnitudes using layerwise attention-head and global intermediate-dimension pruning ratios.Source: §4.2 · pp. 4–5

4.3 Multi-Granularity Early Pruning of PEFT

  • Multi-Granularity Early Pruning targets both coarse-grained PEFT modules and fine-grained ranks because module removal can improve training speed beyond rank allocation alone.The method performs PEFT pruning in the early training stage to maximize efficiency gains.Source: §4.3 · p. 5

4.3.1 Modules Pruning

  • Module pruning estimates importance from the change a LoRA or Adapter module contributes to the original information, then globally removes modules with low importance.LoRA importance is based on module output relative to the underlying weight output, while Adapter importance measures change to the preceding sub-layer output; LoRA modules are temporarily added at all weights for estimation.Source: §4.3.1 · pp. 5–6

4.3.2 Ranks Pruning

  • Rank pruning uses first-order Taylor importance for parameters connected to each rank, sums corresponding W_down and W_up contributions, and globally removes unimportant ranks.This fine-grained step reduces trainable parameters by allowing different modules to retain different rank allocations.Source: §4.3.2 · p. 6

5 Experiments

Experiments across NLU and QA benchmarks evaluate Light-PEFT’s early pruning of foundation-model and PEFT parameters, showing comparable performance with lower training and inference costs.

5.2.1 Experiments on NLU Tasks

  • Light-PEFT achieves comparable or better benchmark performance while pruning foundation-model and PEFT parameters across GLUE, SuperGLUE, and QA tasks.On GLUE, it retains 72% or 67% of foundation-model parameters with 0.3M trainable parameters; on SuperGLUE it retains 76%.Source: §5.2.1 · p. 7, §5.2.2 · pp. 7–8
  • 1.6× training speedup is achieved on GLUE while retaining 67% of foundation-model parameters and 0.3M trainable parameters, with slightly better performance than baselines.At a lower pruning rate, Light-PEFT achieves 1.4× speedup while retaining 72% of foundation-model parameters.Source: §5.2.1 · p. 7

5.2.2 Experiments on QA Tasks

  • QA experiments retain 64% of OPT-1.3B foundation-model parameters and 1.5M trainable parameters with performance comparable to original LoRA, while outperforming LoRA at equal 1.57M trainable parameters.Light-PEFT also outperforms task-agnostic and task-specific LLM-Pruner implementations and remains better than baselines when pruning to 54%.Source: §5.2.2 · p. 7
  • On OPT-6.7B QA tasks, Light-PEFT achieves performance comparable to the original method with 5.2M trainable parameters and maintains good performance after reducing them to 2M.These results support removing redundant parameters from both the foundation model and trainable modules while preserving performance.Source: §5.2.2 · pp. 7–8

5.3.1 Ablation Study

  • Module pruning and rank pruning generally improve generalization, and combining them preserves comparable performance despite pruning more trainable parameters.The ablation isolates coarse-grained module pruning, fine-grained rank pruning, and their combination.Source: §5.3.1 · p. 8, Table 4 · p. 8

5.3.2 Training and Inference Efficiency

  • Training efficiency improves with 39% lower peak memory and 2.2× faster forward-backward passes, while inference reaches up to 1.6× speedup and 48% lower loading memory.The training measurements use RoBERTa-Large with 67% of foundation-model parameters and 0.3M trainable parameters; inference measurements use OPT-6.7B.Source: §5.3.2 · pp. 8–9, Figure 4 · p. 8, Table 5 · p. 9

6 Conclusion

The conclusion presents Light-PEFT as an early-pruning framework that reduces redundancy in foundation models and PEFT modules while improving efficiency and preserving comparable performance.

6 Conclusion

  • Light-PEFT simultaneously estimates and prunes redundant foundation-model and PEFT parameters early in training.It combines Masked Early Pruning of Foundation Model with Multi-Granularity Early Pruning of PEFT.Source: §6 · p. 9
  • Across GLUE, SuperGLUE, and QA tasks with various models, Light-PEFT improves training and inference efficiency, reduces memory usage, and maintains comparable performance.The conclusion describes these benefits as preserving the plug-and-play feature of PEFT.Source: §6 · p. 9

Limitations

Light-PEFT primarily addresses single-task fine-tuning, leaving multi-task learning as a future direction.

Limitations

  • The framework is evaluated primarily in single-task fine-tuning, while estimating and pruning redundancy for multi-task learning remains future work.The authors identify multi-task learning as a direction for extending efficient fine-tuning.Source: Limitations · p. 9

Ethics Statement

The paper states that Light-PEFT aims to improve training efficiency and reduce computational resource costs.

Ethics Statement

  • Light-PEFT aims to enhance training efficiency and reduce computational resource costs.The paper describes these goals as having positive impacts.Source: Ethics Statement · p. 9

A.1 Details of Experimental Setup

The experiments use specified NVIDIA hardware, established implementation components, task-specific data-processing procedures, and AdamW optimization.

A.1 Details of Experimental Setup

  • NLU and OPT experiments use NVIDIA TITAN RTX, RTX 3090, and A800 GPUs according to model and task.TITAN RTX and RTX 3090 support NLU and OPT-1.3B QA experiments, while A800 supports OPT-6.7B QA experiments.Source: §A.1 · p. 13
  • The implementation builds on Transformers, LLM-Adapters, and EarlyBERT, with SuperGLUE and QA preprocessing following prior procedures.The cited preprocessing sources are Liu et al. (2022a) for SuperGLUE and Xiao et al. (2023) for QA tasks.Source: §A.1 · p. 13
  • AdamW is used for optimization, while detailed NLU and QA hyperparameters are provided in Tables 7–9.Tables 7, 8, and 9 cover NLU tasks, OPT-1.3B QA tasks, and OPT-6.7B QA tasks, respectively.Source: §A.1 · p. 13

A.2 The impact of the pruning rate on the foundation model.

The appendix examines pruning-rate, estimation-step, and mask-penalty settings, identifying ranges that preserve performance and a default penalty configuration.

A.2 The impact of the pruning rate on the foundation model.

  • Above 62.5% pruning on WiC causes only a relatively minor performance decrease, whereas exceeding this threshold produces a significant decline.The result suggests that pruning within the stated range removes redundant foundation-model parameters.Source: §A.2 · p. 13, Figure 5 · p. 13

A.3 The impact of the estimation steps of early pruning

  • On BoolQ, estimation beyond 6.8% of total training steps yields no further performance improvement.This supports identifying redundant foundation-model and PEFT parameters during early training.Source: §A.3 · p. 13, Figure 6 · p. 13

A.4 The settings of mask learning penalty

  • The experiments keep λA and λF equal and set both to 1 × 10^-4 in the main experiments.This setting is selected from pilot experiments as providing good task performance.Source: §A.4 · p. 13, Table 6 · p. 13
  • The appendix lists the hyperparameters used for NLU tasks and QA tasks with OPT-1.3B and OPT-6.7B.These settings are documented in Tables 7–9.Source: Table 7 · p. 14, Table 8 · p. 14, Table 9 · p. 14
Loading 2406.03792v1…