PaperlayerView exact revision on arXiv

Source-linked AI summary

Pro-tuning: Unified Prompt Tuning for Vision Tasks

Xing Nie, Bolin Ni, Jianlong Chang, Gaomeng Meng, Chunlei Huo, Zhaoxiang Zhang, Shiming Xiang, Qi Tian, Chunhong Pan

arXiv:2207.14381v3cs.CV

TL;DR

Fine-tuning pre-trained vision models is costly because it updates all model parameters for each task and depends heavily on high-quality downstream data. Pro-tuning instead learns task-specific prompts while freezing the vision model; it is reported to outperform fine-tuning across diverse vision tasks and scenarios.

  • Problem

    Fine-tuning updates and stores all model parameters separately for each downstream task and relies heavily on high-quality downstream data.

    Source: §1 · p. 1

  • Method

    Pro-tuning adapts frozen pre-trained vision models by learning task-specific prompts for downstream images, blending them with intermediate features using lightweight prompt blocks.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Pro-tuning exceeds fine-tuning across sixteen challenging benchmarks; on ADE20K, it gains +0.6 mIoU over full fine-tuning (41.3 mIoU v.s., 40.7 mIoU).

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported results cover CNNs and vision transformers, image classification across five scenarios, object detection, and semantic segmentation.

    Source: §1 · p. 2

    Pro-tuning applies to closed-set categories, relies on a strong pre-trained model, and retains the full inference computational cost of that model, potentially limiting edge-device use.

    Source: §6 · p. 11

Abstract

from arXiv · show

In computer vision, fine-tuning is the de-facto approach to leverage pre-trained vision models to perform downstream tasks. However, deploying it in practice is quite challenging, due to adopting parameter inefficient global update and heavily relying on high-quality downstream data. Recently, prompt-based learning, which adds a task-relevant prompt to adapt the downstream tasks to pre-trained models, has drastically boosted the performance of many natural language downstream tasks. In this work, we extend this notable transfer ability benefited from prompt into vision models as an alternative to fine-tuning. To this end, we propose parameter-efficient Prompt tuning (Pro-tuning) to adapt frozen vision models to various downstream vision tasks. The key to Pro-tuning is prompt-based tuning, i.e., learning task-specific vision prompts for downstream input images with the pre-trained model frozen. By only training a few additional parameters, it can work on diverse CNN-based and Transformer-based architectures. Extensive experiments evidence that Pro-tuning outperforms fine-tuning in a broad range of vision tasks and scenarios, including image classification (generic objects, class imbalance, image corruption, adversarial robustness, and out-of-distribution generalization), and dense prediction tasks such as object detection and semantic segmentation.

1 Introduction

Fine-tuning is costly to adapt across tasks and sensitive to imperfect downstream data, while existing vision-language prompt strategies do not apply to pre-trained vision-only models. Pro-tuning adapts frozen CNNs and vision transformers with learned prompts, and is reported to outperform fine-tuning across diverse vision tasks and scenarios.

1 Introduction

  • Fine-tuning updates and stores all model parameters for each task and relies heavily on high-quality downstream data; on long-tailed CIFAR-100, it loses more than 12% validation accuracy.The reported drop is relative to performance on the original CIFAR-100.Source: §1 · p. 1
  • Prompt-based learning reformulates a downstream task with a task-relevant input description, but vision-language prompt strategies are tailored to multimodal models.The introduction poses adapting pre-trained vision-only models with lightweight prompts as the remaining question.Source: §1 · p. 2
  • Pro-tuning learns task-specific prompts for downstream images while freezing the pre-trained model, blending prompts with intermediate feature maps using lightweight added parameters.It is presented as a unified approach for CNNs and vision transformers.Source: §1 · p. 2
  • On COCO val2017, Pro-tuning reaches 42.3 AP v.s., 42.2 AP under HTC while using 60.4M rather than 80.0M trainable parameters.On ADE20K, it reaches 41.3 mIoU v.s., 40.7 mIoU under UperNet, with 46.6M rather than 66.5M trainable parameters.Source: §1 · p. 2
  • Pro-tuning is presented as plug-and-play across vision models, with minor additional computational costs.The figure caption identifies comparisons among tuning paradigms under various pre-trained vision models, including CIFAR-100-LT-100 with imbalance ratio 100.Source: §1 · p. 2, Figure 1 · p. 2
  • Pro-tuning exceeds fine-tuning across sixteen challenging benchmarks spanning image classification, object detection, and semantic segmentation.The classification evaluations cover five scenarios, including class imbalance, image corruption, adversarial robustness, and out-of-distribution generalization.Source: §1 · p. 2

2 Related Work

The related work contrasts full-model fine-tuning with prompt-based adaptation. It motivates parameter-efficient prompting specifically for frozen vision models, beyond strategies developed for multimodal models.

2.1 Model Fine-Tuning

  • Fine-tuning the entire pre-trained vision model remains dominant, but parameter-efficient adaptation is motivated by increasing model scale and the potential of larger pre-trained models for transfer.The work explores how far frozen vision models can be exploited without the advantages of large-scale pre-training.Source: §2.1 · p. 3

2.2 Prompt-based Learning

  • Prompt-based learning adds task-relevant descriptions to downstream inputs instead of adapting the pre-trained model, with GPT-3 pioneering masked-language-model task responses.CLIP and ALIGN use visual categories as text prompts, but their strategies are tailored to multimodal models.Source: §2.2 · p. 3
  • The paper targets the gap left by multimodal prompting: parameter-efficient prompts for adapting frozen pre-trained vision models across a broad distribution of downstream tasks.Vision-language models such as CLIP and ALIGN achieve vision-task performance without fine-tuning, but their prompt methods are inapplicable to pre-trained vision-only models.Source: §2.2 · p. 3

3 Pro-tuning

Pro-tuning adapts frozen pre-trained vision models, including CNNs and vision transformers, using task-specific prompts generated for downstream inputs. It blends prompts with intermediate features and trains only the added modules and prediction layer.

3 Pro-tuning

  • Pro-tuning integrates with pre-trained CNNs and vision transformers to adapt them to downstream tasks.Source: §3 · p. 3, §3.1 · p. 3

3.1 Overall Architecture

  • Stage-wise prompt blocks generate task-specific prompts from intermediate features, blend them with corresponding feature maps, and pass them through later frozen stages for prediction.An added fully connected layer produces the final prediction; the default places one prompt block after each stage.Source: §3.1 · pp. 3–4

3.2 Optimization

  • Training updates only the additional modules while freezing the pre-trained model, using cross-entropy loss and back-propagating gradients through the model.The added modules include prompt blocks and a fully connected prediction layer.Source: §3.2 · p. 4

4 Experiment

The experiments evaluate Pro-tuning across image-classification scenarios, dense prediction, and ablations, using CNN- and Transformer-based models. They compare it with common tuning protocols and examine few-shot performance and prompt-block design.

4.1 Experimental Setting

  • The experimental setting compares Pro-tuning with fine-tuning, partial fine-tuning, head retraining, and training from scratch across the reported benchmarks.The experiments use seven ImageNet-pretrained CNN and Transformer models; the paper reports fifteen long-standing benchmarks.Source: §4.1 · pp. 5–6

4.2 Image Classification

  • On long-tailed classification, Pro-tuning outperforms fine-tuning in many settings, while using substantially fewer trainable parameters; it also achieves competitive or better accuracy on generic-object datasets.For long-tailed CIFAR-100 at imbalance ratio 100, it reaches 63.9% validation accuracy versus 61.8% for fine-tuning and 58.9% for head retraining under ResNet-101.Source: Table 1 · p. 5, §4.2 · pp. 6–7, Table 2 · p. 6
  • Pro-tuning outperforms fine-tuning on image-corruption, adversarial-robustness, and out-of-distribution tests, often with fewer trainable parameters.Reported gains include +1.2% validation accuracy on CIFAR-100-C under ResNet-50 and +1.2% on ImageNet-R under Swin-S.Source: §4.2 · p. 7, Table 3 · p. 7, Table 4 · p. 8

4.3 Dense Prediction

  • For object detection, Pro-tuning achieves comparable performance with significantly fewer trainable parameters; for semantic segmentation, it consistently outperforms fine-tuning and head retraining.The evaluations use Cascade Mask RCNN and HTC for detection, and UperNet and FCN for segmentation, with ImageNet-pretrained ResNet-50.Source: §4.3 · p. 8, Table 5 · p. 8, Table 6 · p. 8

4.4 Ablation Study

  • In few-shot transfer, Pro-tuning beats fine-tuning by 2.1% on CIFAR-10 with 4 shots and improves over baselines on CIFAR-100 and Caltech-101 at higher shot counts.The reported gains are +3.4% and +2.9% over head retraining and fine-tuning on CIFAR-100 with 8 shots, and +5.9% over head retraining on Caltech-101 with 16 shots.Source: §4.4 · pp. 8–9, Figure 4 · p. 9
  • Jointly training prompt blocks and the pretrained model degrades performance toward fine-tuning, while adding prompt blocks brings little transferability benefit over one three-layer block.The prompt-block comparison covers CIFAR-100 and long-tailed CIFAR-100 at imbalance ratio 100 under ResNet-50.Source: §4.4 · p. 9, Table 7 · p. 9, Table 8 · p. 9
  • Multi-stage prompt insertion performs best; learnable β beats fixed settings, and lower β values improve performance in the tested long-tailed setting.For insertion positions, uniform insertion and insertion after the first layer generalize better than insertion only after the last layer.Source: §4.4 · pp. 9–10, Table 9 · p. 10, Table 10 · p. 10
  • Performance generally improves as prompt-block kernel size increases, then saturates at k = 5; larger kernels offer little benefit and are harder to optimize.The authors use kernel size 5 in experiments unless otherwise specified.Source: §4.4 · p. 10

5 Discussion

Pro-tuning is framed as learning a task-specific prior that addresses downstream-data deviations from the pre-trained model’s image manifold. The discussion also examines prompt-block kernel sizes on long-tailed CIFAR-100.

5 Discussion

  • Table 11 compares prompt-block kernel sizes on long-tailed CIFAR-100 at imbalance ratio 100 under ResNet-50.Source: Table 11 · p. 10
  • Pro-tuning learns a task-specific prior to guide perturbed downstream data back toward the pre-trained model’s learned manifold.The authors attribute its performance gain to this capability.Source: §5 · p. 10
  • Fine-tuning the whole manifold on limited downstream data is described as prone to overfitting and difficult optimization.The authors characterize the learned manifold as potentially fine-detailed and rich in high-level semantics.Source: §5 · p. 10

6 Conclusion

Pro-tuning adapts frozen CNNs and vision transformers through stage-wise prompt blocks, with experiments across fifteen challenging vision datasets. Its stated boundaries include closed-set categories, reliance on a strong pre-trained model, and unchanged inference cost; the authors also note potential misuse in harmful downstream tasks.

6 Conclusion

  • Pro-tuning uses stage-wise prompt blocks to adapt feature distributions in frozen pre-trained CNNs and vision transformers.Experiments on fifteen challenging vision datasets are reported as evidence of effectiveness and generalizability.Source: §6 · p. 11
  • The method applies to closed-set categories, relies on a strong pre-trained model, and retains the full inference cost of pre-trained models.The authors note that this cost may limit development on edge devices.Source: §6 · p. 11
  • The authors warn that the method may be misused for privacy theft or economic fraud and say deployment on public platforms warrants further consideration.Source: §6 · p. 11

Related papers

Loading 2207.14381v3…