PaperlayerView exact revision on arXiv

Source-linked AI summary

Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies

Luping Wang, Sheng Chen, Linnan Jiang, Shu Pan, Runze Cai, Sen Yang, Fei Yang

arXiv:2410.19878v3cs.CLcs.AIcs.LG

TL;DR

Large models are difficult to adapt because their scale demands substantial computation and GPU memory, especially on limited hardware. This survey organizes PEFT methods and applications, and reports that PEFT supports more resource-efficient model adaptation, with related models achieving strong performance at reduced costs.

  • Problem

    Large models' parameter scale creates substantial computational and GPU-memory demands for adapting them to downstream tasks on limited hardware.

  • Method

    The survey reviews PEFT principles, algorithms, applications, classifications, and future research directions across language, visual, multimodal, and diffusion settings.

  • Results

    DeepSeek-R1 delivers performance comparable to OpenAI-o1 on mathematical and logical reasoning tasks while requiring only 2% of the computational cost.

  • Takeaways & Limitations

    PEFT provides a practical approach for adapting large pretrained models while minimizing additional parameters and computational resources.

Abstract

from arXiv · show

The large models, as predicted by scaling raw forecasts, have made groundbreaking progress in many fields, particularly in natural language generation tasks, where they have approached or even surpassed human levels. However, the unprecedented scale of their parameters brings significant computational and storage costs. These large models require substantial computational resources and GPU memory to operate. When adapting large models to specific downstream tasks, their massive parameter scale poses a significant challenge in fine-tuning on hardware platforms with limited computational power and GPU memory. To address this issue, Parameter-Efficient Fine-Tuning (PEFT) offers a practical solution by efficiently adjusting the parameters of large pre-trained models to suit various downstream tasks. Specifically, PEFT adjusts the parameters of pre-trained large models to adapt to specific tasks or domains, minimizing the introduction of additional parameters and the computational resources required. This review mainly introduces the preliminary knowledge of PEFT, the core ideas and principles of various PEFT algorithms, the applications of PEFT, and potential future research directions. By reading this review, we believe that interested parties can quickly grasp the PEFT methodology, thereby accelerating its development and innovation.

I. INTRODUCTION

This survey reviews PEFT as a way to adapt large models while reducing fine-tuning costs, and organizes methods, applications, comparisons, and future directions into a systematic framework.

  • Large models create substantial computational and memory challenges for downstream fine-tuning, while direct use can produce below-desired task performance.
  • Intuitive diagrams and structured tables explain method principles and support comparative analysis of different PEFT approaches.
  • It analyzes PEFT applications across natural language processing, computer vision, multimodal fusion, and diffusion models, including guidance for method selection and improvement.
  • The survey examines recent PEFT research and expands conventional additive, reparameterized, and subtractive categories with hybrid, quantization, and multi-task approaches.It encompasses over 100 research articles published from June 2019 to July 2024.
  • The survey identifies future directions spanning algorithmic refinements and task scenarios while offering resources for academic and practical research.
  • PEFT can reduce a full SFT process from 4 million GPU hours to 400K GPU hours or less, lowering energy use and environmental impact.

2) Prevalent LLMs:

The survey presents prominent LLM families, spanning proprietary, open-source, multimodal, and reasoning-focused models. It highlights differences in architecture, capabilities, efficiency, and application focus.

  • The GPT Family: The GPT family comprises decoder-only Transformer models ranging from GPT-1 through GPT-4o, CODEX, and WebGPT.GPT-3 introduced in-context learning, while GPT-4 and GPT-4o are multimodal LLMs.
  • The LLaMA Family: The LLaMA family provides open-source models from 1 billion to 405 billion parameters and is widely used as a base for fine-tuning research.Its qualities are extended through supervised fine-tuning and parameter-efficient fine-tuning.
  • The OpenAI o1 Family: The OpenAI o1 family targets complex reasoning in mathematics, coding, and science through Chain-of-Thought reasoning and reinforcement learning.The family includes o1-preview and the smaller, faster o1-mini.
  • The DeepSeek Family: The DeepSeek family combines mixture-of-experts architectures and efficiency-focused techniques across models including DeepSeek-V1, DeepSeek-V2, DeepSeek-V3, and R1.DeepSeek-V3 reports less than 10% of LLaMA 3-405B’s training cost, while R1 reports comparable mathematical and logical reasoning performance to OpenAI-o1 at 2% of its computational cost.

3) Multimodal Large Language Models:

Multimodal large language models extend LLMs to process inputs such as text, sound, and video for multimodal reasoning and generation. Their mainstream architecture combines a multimodal encoder, an LLM, and a modal connector.

  • Background: MLLMs extend LLMs by accepting multimodal information such as text, sound, and video to support multidimensional reasoning and text generation.They differ from earlier multimodal research through their billion-scale LLM component and associated training techniques.
  • Background: Earlier multimodal research includes representative paradigms such as CLIP and generative paradigms that transform multimodal tasks into sequence-to-sequence tasks.CLIP establishes visual and text encoders as a bridge for downstream multimodal tasks.
  • Architecture: The mainstream MLLM architecture contains three modules: a multimodal encoder, an LLM, and a modal connector.The encoder processes non-text inputs, the LLM supplies pretrained knowledge and reasoning, and the connector links modalities to the LLM.
  • Architecture: The multimodal encoder converts inputs such as images or audio into compact representations, preferably using a pretrained encoder that is aligned to preserve associated meanings.Examples include CLIP, EVA-CLIP, and ViT-G.
  • Architecture: The modal connector projects encoded multimodal information into a space the LLM can understand, transforming it into prompt tokens that guide generation.This enables the LLM to generate expected content from image, video, or other modality information.

B. Optimization , Datasets, and Evaluation of Large Language Models

The paper reviews instruction tuning, alignment tuning, RLHF workflows, and datasets used across pre-training, SFT, and RLHF. It emphasizes PEFT’s role in making instruction tuning more computationally efficient and broadly applicable.

  • 1) Instruction Tuning: Parameter-efficient fine-tuning can complement or replace few-shot in-context learning while achieving performance comparable to full-parameter tuning at lower computational cost.
  • 2) Alignment Tuning and RLHF: Alignment tuning targets helpfulness, honesty, and harmlessness to address hallucination and align language-model outputs with human preferences.
  • 2) Alignment Tuning and RLHF: RLHF combines a pre-trained language model, a reward model learned from human feedback, and a reinforcement-learning algorithm.
  • 2) Alignment Tuning and RLHF: RLHF first uses supervised examples to teach expected behaviors, then trains a reward model from human-ranked outputs before reinforcement-learning fine-tuning.
  • 3) Datasets for LLM: Pre-training uses broad textual sources, SFT uses specialized annotated examples, and RLHF uses human-feedback data to improve alignment with preferences.

4) LLM evaluation:

The evaluation discussion organizes LLM assessment around diverse general and specialized benchmarks, while the PEFT discussion categorizes methods and compares their efficiency, applications, limitations, and performance.

  • 4) LLM evaluation: LLM evaluation should combine benchmarks that represent real-world scenarios, diverse domains, linguistic complexity, extended dialogue, and varied tasks.
  • 4) LLM evaluation: General benchmarks assess broad knowledge and reasoning, whereas specialized benchmarks target domains such as mathematics, coding, and language understanding.
  • III. PEFT TAXONOMY: PEFT methods are grouped into additive, reparameterized, selective, hybrid, quantization, and multi-task categories.
  • III. PEFT TAXONOMY: The survey summarizes PEFT methods by core idea, trainable-parameter count, applications, limitations, and performance across base models and tasks.
  • A. Additive PEFT: Additive PEFT introduces trainable components while keeping the original model parameters unchanged, reducing storage, memory, and computation requirements.

1) Adapter:

The survey presents adapters and soft prompts as additive PEFT mechanisms that adapt frozen pre-trained models through small task-specific modules or trainable continuous context.

  • 1) Adapter: Adapters insert small trainable layers into frozen pre-trained models to learn task-specific transformations with minimal overhead.
  • 1) Adapter: Standard adapters typically use a down-projection, nonlinear activation, and up-projection to adapt representations for downstream tasks.
  • 1) Adapter: AdapterDrop removes adapters from lower transformer layers during training and inference, improving inference speed in multi-task settings with minimal task-performance impact.
  • 2) Soft Prompt: Soft prompts append trainable continuous vectors to model inputs, guiding task-specific outputs while leaving most pre-trained parameters unchanged.
  • 2) Soft Prompt: Prefix-tuning and prompt-tuning prepend trainable vectors, while p-tuning uses continuous embeddings encoded with a prompt encoder and optimized by backpropagation.
  • 2) Soft Prompt: Instance-dependent prompt methods generate task-specific prompts from each input representation before inserting them into the model input.

3) Scale and Shift:

Scale-and-shift methods adapt pre-trained representations by learning lightweight feature transformations or modifying selected special-token representations, avoiding full fine-tuning.

  • 3) Scale and Shift: SSF modifies deep features with learnable scale and shift parameters to match the distribution of a target dataset.
  • 3) Scale and Shift: SSF requires tuning far fewer parameters than full fine-tuning.
  • 3) Scale and Shift: PASTA adapts pre-trained transformers by modifying the representations of special tokens at each layer with trainable vectors.
  • 3) Scale and Shift: For PASTA, non-special-token positions remain unchanged, while each special-token position receives its corresponding trainable vector.

4) Others:

This section presents PEFT approaches that adapt large pretrained models through alternative parameterizations, side networks, and lightweight modules while preserving efficient inference. It highlights low-rank updates and related adapter designs that avoid updating the full model.

  • Alternative PEFT methods: Inference-time Policy Adapters combine a base model’s output distribution with a trainable adapter policy optimized through reinforcement learning.The two distributions are merged during decoding according to a mixture coefficient.
  • Alternative PEFT methods: Ladder Side-Tuning uses a side network with shortcuts from intermediate backbone activations, avoiding backpropagation through the entire backbone.The side network predicts outputs from representations produced inside the pretrained model.
  • Reparameterization: Reparameterization constructs a low-rank learnable matrix, fine-tunes it during training, and combines it with pretrained parameters at inference without affecting inference speed.The learned low-rank update adapts the model while leaving the pretrained parameters available for the final computation.
  • Reparameterization: LoRA updates a pretrained weight matrix with the low-rank product ∆W = BA, then uses W = W0 + ∆W for inference without additional latency.The trainable matrices A and B have dimensions d×r and r×k, respectively.
  • Reparameterization: KronA adapts Transformer weight matrices by adding a scaled Kronecker product of two learnable matrices to the original weights.The scaling factor controls the contribution of the Kronecker-product update.

2) LoRA Derivatives:

LoRA derivatives modify rank allocation, parameter selection, optimization, uncertainty handling, or update structure to improve flexibility, efficiency, stability, or calibration. These methods retain low-rank adaptation while introducing dynamic, sparse, tensorized, or Bayesian variants.

  • Dynamic Rank: DyLoRA trains LoRA blocks across a range of ranks and sorts learned representations by rank, avoiding rank selection and additional training.AdaLoRA dynamically allocates the parameter budget among weights, while related methods adjust or freeze low-rank components during training.
  • Dynamic Rank: SoRA uses a proximal-gradient-optimized gate to dynamically control LoRA rank and removes zero-gated blocks during inference.This produces a concise rank-optimal module after training.
  • LoRA Improvement: LoRA+ applies different learning rates to the down- and up-projection matrices, setting ηB = ληA with λ greater than 1.The method focuses tuning on ηA to improve model adaptability.
  • LoRA Improvement: LoRA-FA updates only the direction component through a low-rank approximation, preserving the original weight magnitude for efficient and stable learning.The approximation uses ∆W = BA with rank smaller than the weight dimensions.
  • LoRA Improvement: Laplace-LoRA applies Bayesian inference to LoRA by estimating predictive uncertainty around a maximum a posteriori solution.The method improves calibration without a separate validation set and is intended for small datasets.
  • LoRA Improvement: PeriodicLoRA periodically accumulates low-rank updates into the backbone, increasing learning capacity through progressively higher-rank updates.Only LoRA weights are updated during each stage before being unloaded into the backbone.

C. Selective PEFT

Selective PEFT adapts a pretrained model by updating a small parameter subset selected through masks rather than adding broad trainable modules. Its methods use unstructured or structured selection, pruning, importance scores, and task-specific sparse masks.

  • Selective PEFT: Selective PEFT uses a parameter masking matrix to fine-tune a very small subset of pretrained parameters, with unstructured and structured masking as its two divisions.The distinction depends on how parameters are masked.
  • Unstructural Masking: U-Diff pruning adds a task-specific diff vector to fixed pretrained parameters and adaptively prunes the vector with a differentiable L0 approximation.Only a small fraction of parameters is modified for each task.
  • Unstructural Masking: PaFi selects low-magnitude pretrained parameters for fine-tuning, while FishMask selects parameters using Fisher information and updates only the resulting sparse subset.Both methods construct sparse parameter-selection masks before or during training.
  • Unstructural Masking: LT-SFT composes task-specific and language-specific sparse masks to support zero-shot cross-lingual transfer.The masks reduce parameter overlap and interference, improving modularity and preventing overfitting.
  • Unstructural Masking: SAM approximates the original optimization problem with a second-order Taylor expansion and solves the resulting approximation to determine which parameters to optimize.Its formulation uses a parameter mask, the loss gradient, and an approximated diagonal Hessian.
  • Unstructural Masking: Child-tuning updates only a masked child network by blocking gradients for the remaining parameters during backpropagation.The child network is represented by a binary mask over model parameters.
  • Unstructural Masking: LoRAPrune estimates parameter importance from low-rank adapter gradients and performs iterative structured pruning to reduce model size while maintaining performance.The pruning is structured and progressive rather than selecting isolated parameters.

2) Structural Masking:

This section covers structured masking, hybrid PEFT, and quantization-based methods that selectively update, combine, or compress model parameters. The methods aim to reduce memory, storage, or trainable-parameter costs while preserving adaptation capability.

  • Structural Masking: S-Diff pruning removes local groups of weight parameters collectively, while S-Bitfit learns whether each bias parameter receives an update.Both methods impose structured decisions on which parameters are adapted.
  • Structural Masking: FAR selectively activates important parameters while freezing the rest, reducing memory usage and fine-tuning time with minimal performance impact.The active set includes selected parameters in attention mechanisms and feed-forward networks.
  • Hybrid PEFT: Hybrid PEFT methods combine different PEFT types because their performance varies across tasks; MAM-Adapter is presented as a representative example.UniPELT dynamically activates suitable PELT submodules through a gating mechanism.
  • Hybrid PEFT: ProPETL shares one prototype network across layers and tasks while learning binary masks to prune subnetworks, reducing parameter storage and improving efficiency.Its design combines shared structure with task- and layer-specific selection.
  • Quantization PEFT: Quantization PEFT improves computational efficiency and reduces memory usage by training or adapting models with low-precision representations.Examples include low-precision adapter training, quantized fine-tuning deltas, and trainable quantization scales.
  • Quantization PEFT: QLoRA quantizes pretrained weights to 4-bit NF4 precision and combines double quantization with a paged optimizer to prevent gradient-checkpointing memory spikes.NF4 is described as information-theoretically optimal for normally distributed data and empirically stronger than 4-bit integer and float representations.
  • Quantization PEFT: LoftQ alternates quantization and singular value decomposition to approximate original weights, while LQ-LoRA decomposes weights into a fixed quantized component and trainable low-rank components.LQ-LoRA optimizes W ≈ Q + L1L2 while keeping Q fixed during fine-tuning.
  • Quantization PEFT: QDyLoRA combines quantization from QLoRA with DyLoRA’s dynamic training across a spectrum of ranks for efficient large-language-model tuning.The method integrates quantized weights with dynamic low-rank adaptation.

F. Multi-task PEFT

Multi-task PEFT extends parameter-efficient adaptation beyond single downstream tasks by sharing, composing, or routing compact task-specific modules across tasks.

  • Multi-task PEFT methods include adapter-based AdaMix, soft-prompt-based ATTEMPT, and LoRA-based MOELoRA.
  • Adapter-based: AdapterFusion extracts knowledge into task-specific adapters before composing it to exploit multi-task representations without destructive interference.
  • Adapter-based: AdaMix mixes multiple adaptation modules within each Transformer layer while retaining most of the pretrained model.
  • Adapter-based: Hyperformer uses shared hypernetworks to generate task-specific adapter layers, enabling knowledge sharing across tasks.
  • Soft Prompt-based: Soft-prompt methods transfer or decompose prompts across tasks while updating only prompt-related components and preserving the language model.
  • LoRA-based: LoRA-based methods dynamically compose low-rank modules or route inputs among low-rank experts for adaptation to unseen or multiple tasks.
  • The survey organizes applications across vision models, diffusion models, and multimodal large language models, with recommended PEFT techniques summarized in Figure 12.

A. PEFT in Vision Models

Vision-model PEFT adapts pretrained backbones to classification and dense prediction by updating lightweight prompts, adapters, factors, or hypernetwork components while freezing most backbone parameters.

  • Vision-model PEFT targets image classification, segmentation, object detection, depth estimation, and other downstream vision tasks.
  • Image Classification: VPT, AdapterFormer, and FacT represent soft-prompt, adapter, and LoRA-based strategies for image classification.
  • Image Classification: Visual prompting adapts pretrained models by learning input perturbations without updating model parameters, achieving competitive performance with linear probes.
  • Image Classification: AdaptFormer inserts trainable fully connected layers alongside the feed-forward network while freezing the pretrained ViT parameters.
  • Image Classification: FacT tensorizes ViT weights into a 3D tensor and approximates weight increments with lightweight factors that alone are updated and stored.
  • LION inserts implicit layers around a frozen backbone and prunes parameters to reduce computational burden during downstream adaptation.
  • Dense Prediction: Polyhistor decomposes adapter weights into template and scaling kernels combined by a Kronecker product to reduce parameters while preserving accuracy.

B. PEFT in Diffusion Models

Diffusion-model PEFT supports few-shot personalization and controllable generation through compact embeddings, low-rank or structured adaptations, and lightweight condition-processing modules.

  • Diffusion models are adapted for personalized generation, reference-based customization, and other image-generation applications.
  • Generation by Few-shot Finetuning: DreamBooth fine-tunes a diffusion model on a few subject images using class-specific prior preservation to bind an identifier while preserving class diversity.
  • Generation by Few-shot Finetuning: Textual Inversion represents a concept as a learned pseudo-word embedding, enabling compositional text-to-image generation without retraining the model.
  • Generation by Few-shot Finetuning: DreamArtist uses positive and negative prompt embeddings to improve controllability and generate diverse images from a single reference.
  • Generation by Few-shot Finetuning: Diffusion PEFT methods use compact parameter spaces, truncated decompositions, or orthogonal transformations to reduce trainable parameters while preserving generation quality.
  • Controllable Generation: ControlNet freezes the pretrained network, adds a trainable copy, and injects spatial conditions through zero-initialized convolutions.
  • MLLM: MLLM PEFT primarily trains a model connector that converts multimodal embeddings into text-prompt tokens while keeping LLM input consistent.
  • MLLM: Q-Former is more complex than the MLP-based LLaVA connector, but it supports explicit modality alignment and can reduce multimodal data-quality requirements.

V. FUTURE DIRECTIONS

The survey identifies future PEFT directions spanning task objectives, data and modality challenges, model design, learning mechanisms, and foundational shortcomings.

  • Future PEFT research should address multi-objective settings that balance accuracy with privacy, fairness, and latency.
  • Manually weighted multi-objective losses make selecting suitable trade-off coefficients non-trivial, motivating flexible task-adaptive methods.
  • Hybrid LoRA-adapter methods often rely on trial-and-error combinations within predefined design spaces rather than principled search.
  • Continual-learning PEFT must adapt to shifting data distributions while mitigating catastrophic forgetting.
  • Fine-tuned LLMs can become overconfident on modest datasets, motivating improved calibration for safety-critical and data-scarce applications.
  • Privacy-preserving PEFT remains an open direction requiring methods that preserve privacy while maintaining performance and controlling computational costs.
  • The survey presents PEFT as a way to reduce added parameters and computational requirements while approaching or surpassing full-parameter fine-tuning performance.
Loading 2410.19878v3…