PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Visual Prompts for Adapting Large-Scale Models

Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, Phillip Isola

arXiv:2203.17274v2cs.CV

TL;DR

The paper asks whether visual prompts can adapt large-scale vision models without changing their parameters. It learns a single input-agnostic perturbation for frozen models and maps outputs according to the model type. Visual prompting is particularly effective for CLIP, competitive with linear probes, and robust to distribution shift, while offering adaptation without model access at test time.

  • Problem

    The paper investigates whether large-scale vision models can be adapted through input-space prompts instead of changing model parameters or accessing model outputs.

  • Method

    The method learns a single input-agnostic image perturbation by backpropagation with model parameters frozen, using text prompts for CLIP and hard-coded mappings for vision models.

  • Results

    Visual prompting is particularly effective for CLIP, achieves performance competitive with linear probes, and remains robust to distribution shift across broad experiments.

  • Takeaways & Limitations

    Visual prompting provides a new input-space perspective for adapting and using pre-trained vision models, including settings where users cannot control the model at test time.

  • Takeaways & Limitations

    The study does not aim to outperform state-of-the-art performance on specific tasks, and visual prompting underperforms linear probes in some cases.

Abstract

from arXiv · show

We investigate the efficacy of visual prompting to adapt large-scale models in vision. Following the recent approach from prompt tuning and adversarial reprogramming, we learn a single image perturbation such that a frozen model prompted with this perturbation performs a new task. Through comprehensive experiments, we demonstrate that visual prompting is particularly effective for CLIP and robust to distribution shift, achieving performance competitive with standard linear probes. We further analyze properties of the downstream dataset, prompt design, and output transformation in regard to adaptation performance. The surprising effectiveness of visual prompting provides a new perspective on adapting pre-trained models in vision. Code is available at http://hjbahng.github.io/visual_prompting .

1 Introduction

Visual prompting adapts frozen vision models by learning a shared pixel-space perturbation and reformulating inputs or outputs for the pre-trained task. The study finds this approach particularly effective for CLIP and competitive with linear probes across broad evaluations.

  • 1 Introduction: Prompting reformulates downstream inputs and/or outputs into the format of the pre-trained task without updating model parameters.The paper frames visual prompting as data-space adaptation rather than parameter adaptation.
  • 1 Introduction: Visual prompting connects prompting with adversarial reprogramming, which repurposes models through input perturbations despite different terms and motivations.The paper positions both as forms of adapting models by modifying pixels.
  • 1 Introduction: A single visual prompt is learned by backpropagation to transform all input images, while outputs are mapped to downstream labels.CLIP uses a discrete text prompt, whereas vision models use a hard-coded mapping.
  • 1 Introduction: Visual prompting adapts model inputs and therefore does not require model access at test time after the prompt is acquired.This differs from fine-tuning and linear probing, which require access to model parameters or outputs.
  • 1 Introduction: Experiments across four pre-trained models and 15 image classification datasets find visual prompting particularly effective for CLIP and robust to distribution shift.Performance is reported as competitive with, and sometimes beyond, standard linear probes.

2 Related Work

Prior work adapts frozen models through language prompts, soft prompts, visual prompts, or input perturbations. The paper distinguishes visual prompting from parameter- and output-based adaptation because it operates on inputs and can be used without model control at test time.

  • 2 Related Work: Natural-language prompting reformulates downstream data as a masked language-modeling problem so frozen models can perform new tasks.Hand-crafting suitable templates and label words requires domain expertise and significant effort.
  • 2 Related Work: Prefix tuning and prompt tuning learn continuous task-specific prompts through backpropagation while keeping language-model parameters fixed.Prompt tuning simplifies prefix tuning by prepending tunable tokens only to the input.
  • 2 Related Work: Earlier image-prompting methods extend frozen language models or create visual grounding prompts, rather than broadly adapting large-scale vision models.The cited approaches include image-conditional visual prefixes and colored-block prompts.
  • 2 Related Work: Adversarial reprogramming uses a single class-agnostic perturbation to repurpose a model for a new task, but prior vision work pursued adversarial goals or smaller-scale settings.The paper revisits this framework as visual prompting for large-scale vision adaptation.
  • 2 Related Work: Fine-tuning changes model parameters, linear probing learns a layer over model outputs, and prompting reformulates the downstream dataset.Visual prompting, like domain adaptation, modifies inputs and does not require model control at test time.

3 Methods

Visual prompting performs data-space adaptation by learning a pixel-space perturbation while keeping the pre-trained model frozen. The study compares prompt designs and output transformations across vision models and CLIP.

  • Prompting Framework: Visual prompting combines input transformation with output transformation to adapt a model to a downstream task.The input prompt modifies pixels, while output transformation maps model outputs to target labels.
  • Pre-trained Models: The experiments evaluate three vision models and CLIP, spanning different input modalities, pre-training dataset sizes, and architectures.The selected models are Instagram-pretrained ResNeXt, BiT-M, RN50, and CLIP.
  • Prompt Optimization: A single task-specific visual prompt v_φ is learned by backpropagation while the model parameters remain frozen.The prompt is added to each input image during training and evaluation, and optimized parameters are restricted to φ.
  • Prompt Design: The study compares random-location patches, fixed-location patches, and padding across multiple prompt sizes.Padding with p = 30 achieves the best performance among the evaluated design choices.
  • Output Transformation: Vision models use hard-coded class-index mappings, whereas CLIP uses discrete text prompts to transform outputs into downstream labels.For CLIP, prompted text strings processed by the text encoder provide weights for a linear classifier.
  • Evaluation: Visual prompting is evaluated against linear probing using average test accuracy, with Figure 3 contrasting vision models and CLIP.Figure 4 measures accuracy gain or loss relative to text-prompted zero-shot CLIP, while Table 1 reports CLIP performance across 12 datasets.

4 Experimental Setup

The experiments assess visual prompting across standard image-classification datasets and distribution-shift benchmarks, comparing it with established adaptation baselines.

  • Datasets: Visual prompting is evaluated on 12 image-classification datasets spanning natural images, remote sensing, digits, scenes, textures, actions, and other categories.The datasets include CIFAR100, CIFAR10, Flowers102, Food101, EuroSAT, SUN397, DTD, UCF101, SVHN, OxfordPets, Resisc45, and CLEVR.
  • Distribution Shift: Robustness to distribution shift is measured on Camelyon17, FMoW, and iWildCAM from the WILDS benchmark.These evaluations use differing training and test distributions, including changes in hospitals, regions, years, and cameras.
  • Baselines: The comparison includes fine-tuning, linear probes, and text prompting as baseline adaptation methods.Fine-tuning updates all model parameters, linear probing learns a layer over frozen outputs, and text prompting represents zero-shot transfer.

5 Results

Visual prompting is effective for CLIP and remains comparatively robust under distribution shift. Its performance varies substantially across datasets, with competitive or superior results in several settings.

  • Effectiveness of CLIP: Visual prompting achieves competitive performance with linear probing for CLIP, unlike prompting with the evaluated vision models.For CLIP, visual prompting outperforms linear probing on EuroSAT, SVHN, and CLEVR by 1.1%, 23%, and 15.4%, respectively.
  • Effectiveness of CLIP: 24% average performance gain is achieved by learning a visual prompt compared with text-prompted zero-shot CLIP.The magnitude of the gain varies across downstream datasets.
  • Robustness to Distribution Shift: 4.5% and 3.5% are the average performance gaps between visual prompting and linear probing or fine-tuning, respectively, under distribution shift.These results come from WILDS evaluations using training and test sets from different domains.

6 Understanding Visual Prompts

Visual prompting performance depends on downstream dataset properties, prompt design, and output transformation. Gains are generally larger for out-of-distribution or perceptually less diverse datasets, while moderate padding prompts perform best in the reported classification experiments.

  • 6.1 Downstream Dataset: Visual prompting performance varies substantially across downstream datasets, from a +83.2% accuracy gain on the best-performing dataset to a -1% accuracy loss on the worst.
  • 6.1 Downstream Dataset: Performance gain generally increases as datasets become more out-of-distribution from ImageNet and less perceptually diverse.
  • 6.1 Downstream Dataset: A single visual prompt achieves better performance gains on datasets with low perceptual diversity, measured using average pairwise LPIPS.
  • 6.2 Prompt Design: Prompt design strongly affects performance: fixed-location templates, including padding and fixed patches, outperform random-location patches across tested prompt sizes.
  • 6.2 Prompt Design: A single pixel added to EuroSAT improves frozen CLIP accuracy by +3%.
  • 6.2 Prompt Design: Padding with p = 30 achieves the best performance in the reported EuroSAT experiments.
  • 6.3 Output Transformation: For vision models, output transformation uses hard-coded mappings, whereas CLIP uses discrete text prompts.
  • 6.3 Output Transformation: For CLIP, visual prompting yields larger accuracy gains when the text prompt has lower zero-shot accuracy, compensating for low-quality text prompts.

7 Discussion

The discussion frames visual prompting as input-space adaptation that can steer pretrained models without modifying them. It is especially relevant when model access is restricted, though the method can underperform linear probes and is not presented as state-of-the-art.

  • Visual prompts can steer a pretrained model by modifying its input space, including potentially changing visual style for image-to-image models.
  • Input-conditional prompts may be less universal but perhaps more accurate than the universal prompts explored in this work.
  • Visual prompting is useful when fine-tuning is infeasible because the model is exposed through a third-party API.
  • The method underperforms a linear probe in some cases, and the paper leaves ensembling, hybrid adaptation, and larger pretrained models for future work.

8 Conclusion

The conclusion presents visual prompting as a practical way to adapt frozen models through input perturbations. Experiments identify CLIP as particularly suitable and motivate further study of pixel-space adaptation and improved prompts.

  • Visual prompting learns a single, input-agnostic perturbation with gradients to repurpose a frozen model for downstream tasks.
  • CLIP is particularly suitable for visual prompting, achieving results competitive with linear probes across experiments.
  • The findings motivate research into when pixel-space adaptation effectively steers deep networks and how to develop better visual prompts.

A Appendix

The appendix provides an overview of pretrained models and reports performance across datasets, including out-of-distribution evaluation and an architecture ablation on CIFAR100.

  • Table 3 summarizes the pretrained models used in the study.
  • Table 4 reports performance across 12 datasets using vision pretrained models and marks cases where visual prompting outperforms linear probing.
  • Table 5 reports out-of-distribution test accuracy on WILDS and marks a case where visual prompting outperforms both linear probing and fine-tuning.
  • Figure 9 presents a model architecture ablation on CIFAR100.

A.1 Ablation on Model Architecture

The architecture ablation compares multiple ResNet, ResNeXt, BiT-M, and CLIP models on CIFAR100. Larger models do not consistently improve vision-model performance, while CLIP ViTs outperform CLIP ResNets.

  • Model architectures: The CIFAR100 comparison includes four ImageNet-pretrained ResNets and two Instagram-pretrained ResNeXts.The ResNet variants are 18, 50, 101, and 152; the ResNeXt variants are 32x8d and 32x16d.
  • Model architectures: The study evaluates four BiT-M models, two ResNet-based CLIP models, and two ViT-based CLIP models.The CLIP models include ResNet-50, ResNet-101, ViT-B/32, and ViT-B/16.
  • Findings: Larger vision models do not necessarily achieve higher performance.This conclusion is reported across the evaluated vision-model architectures rather than tied to a single model family.
  • Findings: ViT-based CLIP models outperform ResNet-based CLIP models.The comparison covers two models from each CLIP architecture family.

A.2 Dataset Statistics

The dataset description specifies split sources for CLIP adaptation and documents a revision that reinterprets visual prompting as an exploration of practical frozen-model adaptation rather than a methodologically novel contribution.

  • Dataset splits: OxfordPets, Flowers102, Food101, SUN397, DTD, EuroSAT, and UCF101 use the splits provided by [49].These datasets are explicitly identified as using the cited data splits.
  • Dataset splits: The remaining datasets use their officially provided data splits.The passage contrasts these datasets with the seven datasets using splits from [49].
  • Scope and framing: The revised paper acknowledges that visual prompting for vision models is essentially the same as adversarial reprogramming.The authors state that this connection was overlooked in the first version.
  • Scope and framing: The paper removes methodological-novelty claims and reframes visual prompting as a practical adaptation method for modern large-scale models.This reframing is presented as an exploration of the viability of visual prompts.
Loading 2203.17274v2…