PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Prompt Tuning

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, Ser-Nam Lim

arXiv:2203.12119v2cs.CV

TL;DR

Adapting large vision Transformers by full fine-tuning is effective but costly because each task requires storing a separate backbone copy. VPT instead learns a small set of input-space prompts with a linear head while freezing the backbone, and it often surpasses full fine-tuning across downstream tasks while reducing storage.

  • Problem

    Full fine-tuning of large pre-trained vision Transformers requires a separate backbone copy for every task, creating an effectiveness-versus-efficiency adaptation problem.

  • Method

    VPT inserts task-specific learnable prompts into Transformer inputs, updates them with a classification head, and keeps the pre-trained backbone frozen.

  • Results

    Across 24 downstream recognition tasks, VPT surpassed full fine-tuning in 20 cases while using less than 1% of backbone parameters for each task.

  • Takeaways & Limitations

    VPT offers a storage-efficient adaptation strategy that can outperform other tuning protocols, often including full fine-tuning, across vision Transformer settings.

  • Takeaways & Limitations

    On self-supervised ViTs, comparisons were less conclusive: Partial-1 outperformed VPT for MAE, and VPT was competitive but not best for MoCo v3.

Abstract

from arXiv · show

The current modus operandi in adapting pre-trained models involves updating all the backbone parameters, ie, full fine-tuning. This paper introduces Visual Prompt Tuning (VPT) as an efficient and effective alternative to full fine-tuning for large-scale Transformer models in vision. Taking inspiration from recent advances in efficiently tuning large language models, VPT introduces only a small amount (less than 1% of model parameters) of trainable parameters in the input space while keeping the model backbone frozen. Via extensive experiments on a wide variety of downstream recognition tasks, we show that VPT achieves significant performance gains compared to other parameter efficient tuning protocols. Most importantly, VPT even outperforms full fine-tuning in many cases across model capacities and training data scales, while reducing per-task storage cost.

1 Introduction

Large pre-trained vision Transformers are effective but expensive to adapt because full fine-tuning requires a separate backbone copy per task. VPT instead tunes a small input-space parameter set while freezing the backbone.

  • Motivation: Full fine-tuning stores and deploys a separate backbone copy for every downstream task, which is costly for large Transformer models.ViT-Huge has 632M parameters, compared with 25M for ResNet-50.
  • Results: 20 of 24 downstream recognition cases favored VPT over full fine-tuning while VPT used less than 1% of model parameters.The comparison spans a wide range of downstream classification tasks using a pre-trained ViT-B backbone, with mean and standard deviation reported.
  • Motivation: Existing subset-tuning strategies, including classifier-head, bias, and adapter approaches, generally under-perform full fine-tuning in accuracy.These strategies alter only selected parameters or add residual modules to the backbone.
  • Visual Prompt Tuning: VPT adds task-specific learnable parameters to the Transformer input while freezing the pre-trained backbone and learning the prompts with a linear head.The prompts are prepended to the input sequence of each Transformer layer during downstream fine-tuning.

2 Related Work

Transformer models have become important across language and vision, motivating efficient adaptation methods. Related approaches add modules, update limited parameters, or optimize prompts, while VPT applies prompting to vision Transformer inputs.

  • Vision Transformers: Transformers have achieved strong results across vision tasks including classification, detection, segmentation, video understanding, and few-shot learning.They are also widely used in recent self-supervised pre-training methods.
  • Parameter-Efficient Tuning: Adapters add lightweight down-projection, nonlinear activation, up-projection, and residual modules inside Transformer layers.Bias-only tuning instead updates bias terms while freezing the remaining backbone parameters.
  • Prompting: Prompting prepends task instructions or continuous task-specific vectors to inputs and can optimize those vectors through gradients.VPT draws on this prompting direction while targeting downstream vision Transformer adaptation.

3 Approach

VPT inserts learnable prompt embeddings into a pre-trained Vision Transformer’s input space and updates them with the classification head while freezing the backbone. It supports shallow insertion at the first layer or deep insertion across layers, reducing task-specific storage.

  • Preliminaries: A Vision Transformer embeds image patches into d-dimensional vectors, combines them with a classification token, and processes the sequence through N Transformer layers.Each layer contains multiheaded self-attention, feed-forward networks, layer normalization, and residual connections; the final [CLS] embedding feeds the classification head.
  • Visual-Prompt Tuning: VPT introduces p continuous d-dimensional prompt embeddings after the embedding layer and updates only task-specific prompts while freezing the Transformer backbone.The method has shallow and deep variants depending on how many Transformer layers receive prompts.
  • VPT-Shallow: VPT-Shallow inserts learnable d-dimensional prompt tokens into the first Transformer layer only.Later layers process the resulting prompt, patch, and classification-token features without new prompts being inserted.
  • VPT-Deep: VPT-Deep introduces prompt sets at every Transformer layer’s input.Each layer receives its own collection of input learnable prompts, rather than sharing the single insertion used by VPT-Shallow.
  • Storage: For a ViT-Base with 86M parameters, 50 shallow prompts add 0.038M parameters and 50 deep prompts add 0.46M parameters.These correspond to 0.04% and 0.53% of all model parameters, respectively, while reusing the original pre-trained Transformer.
  • Architecture Variants: For Swin and other architectures without a [CLS] token, VPT follows the architecture’s design by using globally pooled features as the classification-head input.This is an architecture-specific adaptation of the same prompt-tuning framework.

4 Experiments

Experiments across diverse recognition tasks, data scales, backbone sizes, and design variants show that VPT provides strong accuracy with a small tuning scope. VPT-deep generally outperforms full fine-tuning and other parameter-efficient methods, while prompt placement and depth materially affect performance.

  • Main Results: VPT-deep outperforms Full on 20 of 24 tasks while requiring 1.18× versus 24.02× total model parameters.It also outperforms all other parameter-efficient protocols across task groups.
  • Main Results: VPT-deep consistently outperforms Full across training-data scales, while Linear and Adapter reverse their low-data advantage when more data is available.Bias offers similar advantages but marginally under-performs VPT-deep across the data scales.
  • Main Results: VPT-deep significantly outperforms Full on VTAB Natural and Structured groups across ViT-B, ViT-L, and ViT-H, while performing nearly equivalently on Specialized.Its advantage over VPT-shallow and Linear also holds across all three backbone choices and task groups.
  • Ablation on Model Design Variants: Prepending prompts in latent embedding space generally outperforms adding them element-wise or inserting them at the pixel level.Prepending shallow prompts before projection reduces VTAB-Natural accuracy by 6.9%, while channel concatenation can reduce it by up to 30 accuracy points.
  • Ablation on Model Design Variants: Even one prompt lets VPT-deep outperform two baselines and remain competitive with or better than Full on VTAB-Structured and Natural, although optimal prompt length varies by task.VPT performance generally improves with prompt depth, but inserting prompts from top to bottom reduces accuracy, indicating earlier layers matter more.
  • Ablation on Model Design Variants: Average pooling over image-patch outputs produces essentially unchanged results relative to the default [CLS] output, whereas pooling final prompt outputs reduces accuracy.For VTAB-Specialized, Image-pool and the default configuration achieve 82.4 and 82.3, respectively.

5 Analysis and Discussion

VPT produces useful task representations with a frozen backbone, but its effectiveness varies across task types, pre-training objectives, and backbone architectures. The analyses show strong results for supervised ViTs and larger ConvNets, alongside clear limits for some self-supervised and smaller-backbone settings.

  • Visualization: VPT-deep enables linearly separable representations while using fewer parameters than full fine-tuning.On Clevr/count, VPT-deep and Full recover the task’s underlying manifold structure, unlike VPT-shallow and Linear.
  • Visualization: Extra tunable parameters at every Transformer layer improve performance over VPT-shallow, which inserts prompts only at the first layer.
  • More vision tasks: On ADE20K semantic segmentation, VPT is comparable with Bias but does not match Full, while using 15M rather than 64M parameters against DeepLab v3+.The comparison uses SETR-PUP with a ViT-L backbone and reports validation mIoU with single- and multi-scale inference.
  • More pre-training methods: For self-supervised ViTs, both VPT variants surpass Linear, but Partial-1 outperforms VPT for MAE and VPT is not best for MoCo v3.The authors state that the reasons for these differences from supervised ViTs remain open questions.
  • ConvNets: ConvNet VPT adds learnable prompt pixels by padding image height and width, using p=5 compared with 263 in prior adversarial-attack work.
  • ConvNets: On ConvNeXt-B, VPT outperforms Full in 8 of 19 cases, whereas its advantage diminishes on ResNet-50 with no clear winner across 19 tasks.

6 Conclusion

The paper concludes that Visual Prompt Tuning efficiently adapts large vision Transformers by learning task-specific input prompts while keeping the backbone fixed. It reports that VPT can outperform fine-tuning protocols while reducing storage, and identifies broader transfer and pre-training questions for future work.

  • VPT introduces task-specific learnable prompts in the input space while keeping the pre-trained backbone fixed.
  • VPT can surpass other fine-tuning protocols, often including full fine-tuning, while dramatically reducing storage cost.
  • The experiments raise questions about fine-tuning dynamics across vision-Transformer pre-training objectives and efficient transfer to broader vision tasks.

A Implementation Details

The experiments were implemented in PyTorch and run on NVIDIA A100-40GB GPUs.

  • All experiments used PyTorch and NVIDIA A100-40GB GPUs.

A.1 Classification Experiments

The classification experiments tune VPT primarily through prompt length and use task-specific implementation and optimization settings across datasets and backbones. The appendix documents datasets, pretrained models, comparison methods, and table-level configurations for the 24 evaluated classification tasks.

  • Prompt length is the only VPT-specific hyper-parameter tuned, with candidate ranges of {1, 5, 10, 50, 100, 200} for ViT and {1, 5, 10, 50} for Swin.The maximum prompt length is approximately the number of image patch tokens within each MSA: 196 for ViT and 49 for Swin.
  • VPT-deep uses dropout of 0.1, while ConvNet experiments use a separate prompt-length search range.
  • Adapters add lightweight down-projection, nonlinear activation, up-projection, and residual modules inside Transformer layers.The implementation follows the setup that inserts adapters after the FFN “Add & LayerNorm” sub-layer.
  • Training uses standard image augmentation, with ImageNet normalization and task-dependent resizing, cropping, and horizontal flipping.VTAB-1k uses resizing to 224×224 without other augmentations under its default settings.
  • Learning rate and weight decay are selected by grid search on each task’s validation set, with optimization configurations summarized in Table 6.
  • The classification study evaluates 24 tasks and documents dataset specifications alongside pretrained backbone details, including feature-extractor parameter counts and batch-size groupings.

A.2 Semantic Segmentation Experiments

The segmentation setup applies VPT only within SETR’s ViT encoder, while learning the PUP decoder head and prompts with the backbone frozen. Experiments use ADE20K and MMSegmentation with specified decoder and optimization settings.

  • ADE20K provides 20,210 training images, 2,000 validation images, and 150 fine-grained scene-parsing labels.
  • SETR-PUP uses a progressive upsampling decoder selected as the best decoder choice in the MMSegmentation reproduction.
  • VPT inserts prompts only into SETR’s ViT encoder, learning the prompts and PUP head while freezing the ViT backbone.
  • The segmentation experiments use learning rate 0.005 for all methods and sweep prompt lengths from 1 to 200.

B Extended Analysis

The extended analysis examines prompt placement, sharing, initialization, parameter combinations, resolution, data scale, and computational cost. Across these studies, VPT’s gains depend on prompt design, while random initialization and frozen-backbone tuning remain effective.

  • Prompt sequence: Updating prompt embeddings, rather than merely expanding the input sequence, drives VPT’s advantage over Full on 20 of 24 evaluated tasks.
  • Sharing prompts: Shared-intra prompts remain competitive with one prompt, whereas Shared-all degrades performance but still exceeds linear probing across three VTAB subgroups.
  • Prompt initialization: Prototype-based prompt initialization does not improve on random initialization, which works best generally across VTAB subgroups.
  • Prompt depth and length: Prompt-length sensitivity varies with prompt depth, especially when prompts occupy nine layers, making depth-specific tuning consequential.
  • Combining tuning methods: Combining VPT with bias tuning does not generally improve results and undermines VPT-deep across all three task subgroups.
  • Computational cost: Doubling sequence length produces about 2× rather than 4× inference latency and peak memory, while fewer prompts reduce training latency.

C Further Discussion

Further discussion positions VPT against adversarial reprogramming and textual prompting while extending the analysis to downstream data size. These comparisons emphasize VPT’s parameter efficiency and differing visual-prompt behavior.

  • VPT versus adversarial reprogramming: Adversarial reprogramming injects nearly 20 times more learned input parameters than VPT, 264k versus 13k.
  • Visual versus textual prompts: VPT differs from textual prompting by outperforming full-model fine-tuning in 20 of 24 cases and favoring random initialization and earlier-layer prompts.
  • Downstream data size: For FGVC tasks, VPT-deep consistently surpasses Full across downstream data scales, while parameter-efficient methods outperform Full in small-to-medium data regimes.

D Supplementary Results

Supplementary results provide per-task classifications, additional visualizations, and detailed VTAB-1k and FGVC tables. They reinforce VPT-deep’s broad performance pattern across datasets and task groups.

  • Per-task results: Per-task results for the 24 classification tasks evaluated in Table 1 are reported in Tables 13 and 14.
  • Training-data ablations: VPT-deep consistently surpasses Full across data scales for five FGVC datasets, including beyond the small-to-medium data regime.
  • Visualizations and tables: Additional t-SNE visualizations cover VTAB datasets with at most 20 target classes, alongside detailed VTAB-1k and FGVC per-task tables.
Loading 2203.12119v2…