PaperlayerView exact revision on arXiv

Source-linked AI summary

Parameter-Efficient Fine-Tuning for Foundation Models

Dan Zhang, Tao Feng, Lilong Xue, Yuandong Wang, Yuxiao Dong, Jie Tang

arXiv:2501.13787v1cs.CLcs.AIcs.LG

TL;DR

Foundation models span diverse modalities and tasks, but adapting them efficiently requires methods that reduce fine-tuning costs while preserving downstream performance. This survey systematically reviews PEFT categories, mechanisms, applications, and trends across foundation-model families. It concludes that PEFT is diversifying across domains, while multimodal foundation models remain relatively underexplored and require further study.

  • Problem

    Existing PEFT insights are scattered across studies, leaving insufficiently unified understanding of techniques, trends, and applications across diverse foundation models.

  • Method

    The survey reviews PEFT development, categorizes methods, examines designs across foundation-model structures and downstream applications, and identifies future research directions.

  • Results

    PEFT research is growing across language, vision, and multimodal domains, with LLMs and VFMs dominating activity while MFMs remain relatively underexplored.

  • Takeaways & Limitations

    A systematic understanding of how PEFT adapts to diverse foundation models supports more effective and efficient model adaptation across tasks and domains.

Abstract

from arXiv · show

This survey delves into the realm of Parameter-Efficient Fine-Tuning (PEFT) within the context of Foundation Models (FMs). PEFT, a cost-effective fine-tuning technique, minimizes parameters and computational complexity while striving for optimal downstream task performance. FMs, like ChatGPT, DALL-E, and LLaVA specialize in language understanding, generative tasks, and multimodal tasks, trained on diverse datasets spanning text, images, and videos. The diversity of FMs guides various adaptation strategies for PEFT. Therefore, this survey aims to provide a comprehensive overview of PEFT techniques applied to diverse FMs and address critical gaps in understanding the techniques, trends, and applications. We start by providing a detailed development of FMs and PEFT. Subsequently, we systematically review the key categories and core mechanisms of PEFT across diverse FMs to offer a comprehensive understanding of trends. We also explore the most recent applications across various FMs to demonstrate the versatility of PEFT, shedding light on the integration of systematic PEFT methods with a range of FMs. Furthermore, we identify potential research and development directions for improving PEFTs in the future. This survey provides a valuable resource for both newcomers and experts seeking to understand and use the power of PEFT across FMs. All reviewed papers are listed at \url{https://github.com/THUDM/Awesome-Parameter-Efficient-Fine-Tuning-for-Foundation-Models}.

I. INTRODUCTION

Foundation models support diverse modalities and tasks, while PEFT reduces fine-tuning costs by updating far fewer parameters. This survey organizes PEFT methods, applications, and trends across foundation-model families, highlighting rapid growth alongside underexplored multimodal models.

  • Foundation models are trained on large-scale, multimodal datasets for language understanding, code generation, visual understanding, and content generation.
  • PEFT minimizes trainable parameters and computational overhead while targeting performance near full fine-tuning, with adaptation strategies varying across model architectures.
  • LoRA trains 4.7M or 37.7M parameters instead of GPT-3’s 175B, saving over 99.97% of parameters while improving results by 0.1% to 0.5% over full fine-tuning.
  • PEFT research is growing across language, vision, and multimodal domains, with LLMs and VFMs currently dominating activity while MFMs remain relatively underexplored.
  • Existing surveys provide focused insights, but the field lacks a unified account of PEFT development across diverse foundation models.
  • The survey reviews PEFT trends, categories, designs across five model structures, downstream applications, and future research directions.

A. Overview of Foundation Models

Foundation models are categorized into five groups according to input modalities and functions. These groups span language, vision, vision-language, visual content generation, and multimodal processing.

  • The survey roughly categorizes foundation models into five groups based on their input modalities and functions.
  • Large Language Models: Large language models understand, generate, and manipulate text across tasks such as translation, summarization, question-answering, and text generation.
  • Vision Foundation Models: Vision foundation models process visual data for tasks including image classification, object detection, and segmentation.
  • Vision Language Models: Vision-language models connect visual and textual modalities for grounding, image captioning, and visual question answering.
  • Multi-Modal Foundation Models: Multimodal foundation models extend language models to process and generate combinations of text, images, and audio.

B. Development of Parameter-Efficient Fine-Tuning

PEFT development reduces fine-tuning costs by updating only selected parameters or adding small trainable components. Its taxonomy has diversified into selective, additive, prompt, reparameterization, and hybrid strategies.

  • PEFT reduces computational and storage costs by limiting the number of parameters updated during foundation-model fine-tuning.
  • Selective PEFT: Selective PEFT fine-tunes a subset of model parameters while freezing the remainder, using layer-wise freezing or learned parameter-selection strategies.
  • Additive PEFT: Additive PEFT inserts small adapter networks between foundation-model layers and updates those bottleneck parameters while largely preserving the original model.
  • Prompt PEFT: Prompt PEFT learns soft commands represented as sequences of embedding vectors to guide task performance.
  • Reparameterization PEFT: Reparameterization PEFT decomposes or re-represents existing parameters so only part of them is adjusted during fine-tuning.
  • Hybrid PEFT: Hybrid PEFT combines adapters, prompts, and parameterization methods, seeking configurations suited to different tasks and scenarios.
  • The field’s evolution is characterized by diversification and integration as model sizes and multitask-learning demands increase.

III. METHODOLOGY

The survey analyzes important PEFT categories across language, vision, vision-language, multimodal, and visual-generation foundation models, including their advantages and disadvantages.

  • The methodology organizes PEFT categories across five foundation-model types and analyzes the advantages and disadvantages of each category.

A. Selective PEFT

Selective PEFT fine-tunes a subset of existing parameters or adds only minimal trainable parameters, using predetermined or automatically selected parameters while retaining the rest.

  • Selective PEFT trains a subset of model parameters or introduces minimal additional parameters instead of fully fine-tuning the model.
  • Table II compares PEFT methods by modality, category, structural changes, fine-tuned position, inference efficiency, added parameters, and trainable-parameter percentage.
  • The category includes specific selection, which chooses predetermined parameters, and automatic selection, which determines trainable parameters through the model.

1) Specific Selection:

Specific-selection methods fine-tune chosen layers, neurons, biases, or tokens while leaving most parameters unchanged. PASTA illustrates the efficiency of this approach by matching similar fine-tuning performance with a very small trainable subset.

  • Specific Selection: Specific-selection methods choose particular layers or neurons for fine-tuning, including Freeze Layers, BitFit, and PASTA.
  • Specific Selection: Freeze Layers fine-tunes only the last few foundation-model layers, while BitFit adjusts some or all model bias terms.
  • Specific Selection: 0.029% of total parameters enabled PASTA with performance similar to full fine-tuning in natural language understanding tasks.
  • Specific Selection: On CoNLL2003 named entity recognition, PASTA with RoBERTa achieved a 90.8% F1 score, outperforming P-tuning v2 by 0.6% with 20 times fewer trainable parameters.
  • Automatic Selection: Automatic-selection methods use algorithms such as masking, Diff-Pruning, FISH, AutoFreeze Layers, and CHILD-TUNING to determine trainable parameters.
  • Automatic Selection: Masking methods train binary masks with a straight-through estimator to selectively update foundation-model parameters and a randomly initialized classifier layer.

A.3 Pros and Cons

Selective PEFT avoids adding parameters and can preserve inference efficiency, but some methods introduce memory or training-time costs. Adapter methods add task-specific modules and can improve transferability while creating inference overhead.

  • Selective PEFT: Selective PEFT avoids adding new parameters, helping control model complexity and preventing increased downstream inference time.
  • Selective PEFT: Masking matrices in methods such as FISH and CHILD-TUNING can increase memory usage in memory-constrained scenarios.
  • Selective PEFT: Special selection mechanisms such as Diff-Pruning can lengthen training and offset the benefits of using fewer trainable parameters.
  • Adapters: Adapters transform one layer’s output into the next layer’s task-specific input without modifying the foundation model’s original parameters.
  • Bottleneck Adapters: Bottleneck adapters down-project, apply a nonlinear activation, up-project, and use a residual connection, but their added parameters can slow inference.
  • Multi-Adapters: Multi-adapter methods add modules to integrate knowledge across tasks, reduce forgetting, and improve bottleneck-adapter transferability.
  • Multi-Adapters: Adapter Fusion learns task-specific adapters first, then trains a fusion module to combine them for a target task without changing adapter structures or parameters.
  • Multi-Adapters: AdaMix replaces gating with random expert averaging to reduce gating-unit computation and parameters and avoid overloading one expert, but requires more training memory.

3) Adapter Sparsity:

Adapter sparsity improves PEFT efficiency by selectively dropping adapter components or exploiting sparse parameterization while retaining competitive performance. Comparisons emphasize backpropagation costs and inference overhead across additive PEFT methods.

  • Adapter sparsity: AdapterDrop randomly drops adapters during training to reduce computation and memory requirements while encouraging reliance on the original transformer layers.It proposes specialized adapter dropout, which retains only the top n layers during training.
  • Adapter sparsity: Table IV compares additive PEFT methods by backpropagation-cost reduction and inference overhead, including FFN-specific or other added overhead.
  • Adapter sparsity: AdapterBias uses only a vector and linear layer for token-dependent hidden-state shifts and remains competitive with Bottleneck Adapter using 40 times fewer parameters.
  • Adapter sparsity: SparseAdapter maintains the same parameter budget through Large-Sparse pruning and can match or outperform standard adapters at 80% sparsity.
  • Adapter sparsity: LST trains a small transformer network alongside a pretrained network, using the foundation model only as a feature extractor.

B.2 Additive PEFT in More FMs

Additive PEFT adapts foundation models by adding lightweight components or learned prompts while preserving most pretrained weights. Across language, vision, and generative models, these methods reduce trainable parameters but may introduce inference or configuration costs.

  • Adapters: Adapter methods add lightweight modules between foundation-model layers while leaving most pretrained weights unchanged.Bottleneck adapters down-project, apply a nonlinear transformation, and up-project, with a residual connection.
  • Adapters: Vision adapters span convolutional bypasses, lightweight recognition modules, image-specific inductive biases, and side networks for segmentation or object recognition.Convpass uses less than 0.5% trainable parameters, while AdaptFormer uses less than 2% of ViT parameters.
  • Prompt PEFT: Prompt PEFT inserts hard or continuous prompts into model inputs or layers to align inputs with pretrained distributions and guide outputs.The surveyed prompt types are Hard Prompt, AutoPrompt, and Soft Prompt; hard prompts require manual templates, while soft prompts use machine-learned continuous vectors.
  • Soft Prompt: Prefix Tuning freezes foundation-model parameters and optimizes a task-specific continuous prefix, achieving performance comparable to full fine-tuning with about 0.1% of parameters.Prompt Tuning instead appends task-specific prompt tokens only at the input layer, while P-Tuning learns flexible virtual-token embeddings through an MLP and LSTM.
  • Reparameterization: LoRA and related reparameterization methods adapt models through compact matrix updates, while variants target memory reduction, dynamic parameter allocation, or tensor decomposition.LoRA updates low-rank matrices A and B; LoRA-FA reduces activation memory, and MPO decomposes parameter matrices into tensor-network components.

D.3 Pros and Cons

Reparameterization PEFT is flexible and can adapt diverse foundation models without extra inference overhead. Its main constraints are sensitivity to hyperparameters and the assumption that low-rank updates adequately represent task adaptations.

  • Advantages: Reparameterization PEFT can be applied across mainstream models, including language models and visual generation models, enabling rapid adaptation to tasks and domains.The passage specifically identifies LoRA as usable for models such as GPT and Stable Diffusion.
  • Advantages: These methods introduce no extra inference overhead while preserving their flexibility across model types.The stated advantage concerns the reparameterization category’s deployment behavior.
  • Limitations: The rank of inserted adaptation matrices significantly affects task adaptation, making reparameterization methods sensitive to hyperparameter choices.Rank selection is identified as a key configuration factor.
  • Limitations: Low-rank adaptation may limit expressiveness and produce suboptimal performance when the task feature space is highly complex.This limitation follows from assuming that model adaptations can be represented with low-rank matrices.
  • Hybrid PEFT: Hybrid PEFT combines methods such as LoRA, BitFit, and P-Tuning within one framework to draw on their complementary strengths.The survey presents this integration as a strategy for constructing a more comprehensive framework.

E.1 Hybrid PEFT in Basics

Hybrid PEFT integrates multiple adaptation methods to combine their strengths, but this flexibility can increase complexity and produce performance trade-offs. The survey situates hybrid methods across language, vision, and multimodal foundation models.

  • Hybrid PEFT methods: Hybrid PEFT combines methods such as adapters, prefix tuning, and LoRA within coordinated frameworks.UniPELT uses gating to regulate each integrated module, while NOAH combines adapter, LoRA, and VPT modules in Transformer blocks.
  • Hybrid PEFT methods: UniPELT improved performance by 1% to 4% over the standalone PEFT methods it integrates.
  • Hybrid PEFT methods: COMPACTER reduces adapter parameters through hypercomplex multiplication and Kronecker products of low-rank matrices.It adds only 0.05% to 0.2% of the original model’s parameters and was evaluated on GLUE and SuperGLUE.
  • Hybrid PEFT methods: Hybrid designs include unified, searched, grouped, bias-based, and scaling-based approaches such as S4, NOAH, and DiffFit.S4 groups layers for selective training, NOAH searches for prompt-module architectures, and DiffFit tunes bias terms with scaling factors.
  • Strengths and limitations: Hybrid PEFT can improve flexibility and robustness, but added complexity, labeling costs, hyperparameter interactions, and unforeseen combinations may limit outcomes.NOAH requires extra super-network training and additional labeling efforts.
  • Applications across foundation models: Hybrid PEFT methods are applied across causal and prefix LLMs, vision transformers, visual models, and prompted vision-language models.The surveyed applications cover language, image, video, and multimodal downstream tasks.

A. Observation of Current Trend

The survey identifies unresolved challenges in PEFT reliability, interpretability, benchmarking, and adaptation to changing domains and architectures. It proposes future directions spanning continual learning, architecture-aware design, scaling laws, interdisciplinary constraints, and brain-inspired approaches.

  • Current challenges: PEFT hyperparameters are sensitive, and their optimal settings can differ substantially from full fine-tuning.The survey notes that PEFT often requires a much higher learning rate than full fine-tuning.
  • Current challenges: PEFT interpretability remains difficult because learned prompts and adapter parameters are hard to relate to understandable model mechanisms.The survey highlights unordered token-based prompts and the relationship between learned adapter parameters and layers.
  • Current challenges: PEFT lacks comprehensive benchmarks because varied datasets and task setups produce inconsistent performance assessments.Standardized baselines are identified as a current trend toward fairer comparisons.
  • Future directions: Future PEFT research may incorporate domain-specific constraints, continual learning, architecture-aware schemes, and scaling-law analyses.These directions address applications across disciplines, sequential tasks, model components, and trainable-parameter ranges.
  • Future directions: Brain-inspired PEFT connects efficient adaptation with hierarchical processing, efficient coding, and selective synaptic modification.
  • Conclusion: The survey concludes that systematic understanding of PEFT across diverse foundation models can support efficient adaptation across tasks and domains.
Loading 2501.13787v1…