PaperlayerView exact revision on arXiv

Source-linked AI summary

Reprogramming Vision-Language Models via Structured Prompt Reparameterization

Zizhao Li, Chengyi Cai, Mohammed Yaqoob Ansari, Feng Liu, Joseph West, Kourosh Khoshelham

arXiv:2609.36680v1cs.CV

TL;DR

Fine-grained categories can share attribute descriptions and semantic directions, while existing visual reprogramming methods do not explicitly model class relationships. RVP combines within-class prompt aggregation with residual corrections across classes; across benchmarks and CLIP backbones, it improves over prior visual reprogramming methods, with its largest gains on fine-grained tasks.

  • Problem

    Existing CLIP visual reprogramming methods aggregate prompts within classes but do not explicitly model class relationships, despite fine-grained categories sharing attributes and semantic directions.

    Source: §1 · p. 1

  • Method

    RVP aggregates multiple text descriptions within each class and refines the resulting logits with structured residual corrections across classes.

    Source: §1 · p. 2

  • Results

    Across 11 benchmarks and multiple CLIP backbones, RVP consistently improves over prior visual reprogramming methods, with the largest gains on fine-grained tasks.

    Source: §8 · p. 9

  • Takeaways & Limitations

    The results highlight the benefit of modeling class relationships in visual reprogramming and motivate evaluating RVP beyond CLIP.

    Source: §8 · p. 9

    RVP’s advantage is less pronounced on Food101, where high within-class variation weakens the consistency of text-based class relationships.

    Source: §7 · p. 9

Abstract

from arXiv · show

Visual reprogramming adapts pretrained models to downstream tasks by modifying their input and output interfaces while keeping the backbone fixed. In vision-language models, existing methods mainly rely on intra-class prompt aggregation and do not explicitly model relationships among classes. However, fine-grained categories often exhibit highly overlapping attribute descriptions and strong inter-class correlation in the text embedding space, where discriminative cues lie in subtle low-variance components. We propose Reparameterized Inter-Class Visual Reprogramming (RVP), a structured framework that aggregates multiple text prompts within each class and applies residual correction across classes. We also show that CLIP-based visual reprogramming with input-independent linear output aggregation can be expressed as a linear mapping from frozen image embeddings to downstream logits, and use this view to design a structured reparameterization that models shared semantic components and class-specific differences. RVP uses only a single visual prompt and can be reparameterized at inference into a frozen backbone followed by a linear classifier, incurring nearly zero computational overhead. Across 11 few-shot classification benchmarks and four CLIP backbones, RVP consistently improves over prior visual reprogramming methods with comparable or better inference efficiency.

1 INTRODUCTION

Visual reprogramming preserves pretrained backbones, but existing CLIP methods aggregate prompts within classes without explicitly modeling relationships among classes. RVP combines intra-class aggregation with inter-class correction, and the paper reports improved few-shot performance with efficient inference.

1 INTRODUCTION

  • Fine-grained classes share highly similar attribute descriptions and dominant semantic directions, while discriminative cues lie in subtle, low-variance components.This motivates modeling class relationships beyond independent intra-class prompt selection.Source: §1 · p. 1, Figure 1 · p. 2
  • RVP jointly aggregates attribute descriptions within each class and refines class logits through an inter-class relation matrix.Its structured mapping keeps the classifier anchored in CLIP’s text-embedding subspace.Source: §1 · p. 2
  • CLIP-based visual reprogramming can be viewed as a linear mapping from normalized image embeddings to downstream logits, unifying prompt aggregation and label mapping.This view supports the structured reparameterization used by RVP.Source: §1 · p. 2
  • Across multiple CLIP backbones, RVP improves few-shot visual reprogramming while preserving efficient inference.The paper reports consistent improvements across benchmarks, especially on fine-grained datasets, with comparable or lower inference cost.Source: §1 · p. 2

2 RELATED WORK

Related work distinguishes interface-level model reprogramming from prompt learning and feature adaptation. In visual reprogramming, RVP builds on methods that use multiple class descriptions by adding structured transformations over text-derived responses.

Model Reprogramming

  • Model reprogramming adapts pretrained models through input and output transformations while keeping internal parameters fixed.Prior work spans vision, graph, acoustic, and language tasks.Source: §2 · pp. 2–3

Input Visual Reprogramming and Feature Adapter

  • Visual reprogramming adapts image classifiers with learnable input patterns, while recent VLM methods use multiple class attribute prompts or multiple visual prompts.RVP instead learns structured transformations over text-derived responses.Source: §2 · p. 3

Prompt Learning

  • Prompt learning adds trainable parameters to pretrained models through textual, visual, internal-token, or cross-modal prompts.It is often architecture-dependent.Source: §2 · p. 3

3 PRELIMINARIES AND INSIGHTS

The section formulates CLIP classification as image–text similarity and visual reprogramming as an input transformation followed by frozen encoding. For input-independent linear aggregation, description-level similarities map linearly from normalized image embeddings to class logits.

3 PRELIMINARIES AND INSIGHTS

  • CLIP maps images and text descriptions into a shared embedding space, where classification uses image–text similarity.Downstream classes can be represented by sets of textual descriptions.Source: §3 · p. 3
  • Visual reprogramming transforms downstream images with a trainable visual prompt before processing them with frozen CLIP encoders.The text descriptions supply the downstream class representations.Source: §3 · p. 3
  • When description-level aggregation uses input-independent linear weights, normalized image embeddings map linearly to downstream class logits.This formulation provides a unified view of prompt aggregation and label mapping.Source: §3 · p. 4

4 REPARAMETERIZED INTER-CLASS VISUAL REPROGRAMMING

RVP structures visual reprogramming around within-class description weighting and residual interactions across classes while keeping CLIP frozen. It trains these components jointly and folds the text embeddings and learned mappings into one linear classifier for inference.

4 REPARAMETERIZED INTER-CLASS VISUAL REPROGRAMMING

  • RVP replaces a dense mapping with structured intra-class and inter-class components, using fewer parameters and one visual prompt with one image-encoder pass.The formulation uses C × M intra-class weights and C × C inter-class parameters instead of C^2M parameters for the dense mapping.Source: §4 · p. 4

4.1 TRAINING PHASE: DYNAMIC AGGREGATION AND MESSAGE PASSING

  • During training, a softmax-normalized matrix learns weights for aggregating each class’s multiple text descriptions into base logits.The visual prompt transforms the image before frozen CLIP encoding.Source: §4.1 · pp. 4–5
  • A residual inter-class matrix redistributes evidence across correlated classes, suppressing shared semantics while enhancing subtle class-specific differences.The identity term preserves CLIP-induced logits, while a zero-initialized correction matrix learns residual changes rather than replacing the pretrained structure.Source: §4.1 · p. 5
  • The visual prompt and both class-weighting matrices are optimized jointly while the CLIP image and text encoders remain frozen.Final logits are supervised with downstream cross-entropy loss.Source: §4.1 · p. 5

4.2 INFERENCE PHASE: LINEAR REPARAMETERIZATION

  • At inference, the learned aggregation and class relations are absorbed into one classifier, producing logits from one normalized image embedding with a single linear projection.This exact reparameterization adds nearly zero inference overhead.Source: §4.2 · pp. 5–6

5 EXPERIMENTS

Experiments evaluate RVP with pretrained CLIP models across four visual encoder architectures in a 16-shot classification setting. The 11-dataset benchmark spans diverse visual domains, and reported accuracies average three random seeds.

Experimental Setup and Benchmarks

  • The benchmark covers 11 publicly available datasets spanning textures, actions, and natural scenes.The suite includes FGVC Aircraft, StanfordCars, Flowers102, UCF101, and RESISC45, among others.Source: §5 · p. 6
  • Experiments use four CLIP image encoder architectures, including ResNet and Vision Transformer variants.Source: §5 · p. 6
  • All experiments use a 16-shot downstream classification setting, with accuracy averaged across three independent random seeds.Source: §5 · p. 6

6 RESULTS

RVP improves visual reprogramming results across CLIP backbones and few-shot settings, with notable gains on fine-grained datasets. Its structured inter-class mapping also retains low inference latency, while ablations and qualitative comparisons highlight the role of inter-class modeling.

Quantitative Results

  • The comparison evaluates RVP against VP, AR, AttrVR, and DVP-cls using the same text prompt set as DVP.DVP-cls is used to isolate reprogramming performance without external LLM-generated descriptions.Source: §6 · p. 6
  • 72.3% average accuracy on RN50 exceeds DVP by 6.3 points and AttrVR by 7.7 points.The passage reports particularly strong gains on Aircraft and Cars.Source: §6 · p. 7, Table 2 · p. 7
  • +6.3 on RN50, +4.2 on RN101, +4.8 on ViT-B/32, and +3.0 on ViT-B/16 are RVP's gains over DVP, with larger gains on weaker backbones.Source: §6 · p. 7
  • RVP achieves the highest average accuracy across evaluated backbones and the best result on 9 of 11 datasets with ViT-B/16 CLIP.On ViT-B/16, it exceeds DVP by 3.0 points and AttrVR by 4.2 points, with gains of +7.4 on Aircraft and +14.0 on Cars over DVP.Source: §6 · p. 6, Table 3 · p. 7

Few-shot Classification Performance

  • At 32-shot, RVP reaches 50.7%, versus 41.2% for DVP and 38.4% for AttrVR on Aircraft.RVP leads prior visual reprogramming baselines at every evaluated shot setting; its improvement is modest at 1-shot and larger with more labeled samples.Source: Figure 3 · p. 7, §6 · p. 7

Accuracy and Inference Speed

  • RVP maintains latency close to single-prompt methods while achieving the highest Aircraft accuracy across four backbones.A single visual prompt and inference-time reparameterization into one linear classifier avoid DVP's multiple image-encoder forward passes.Source: Figure 4 · p. 8, §6 · p. 8

Ablation Study

  • Removing inter-class matrix E drops average accuracy from 82.7% to 77.9%, the largest architectural ablation drop.The effect is especially strong on Aircraft and Cars; the visual prompt is also essential, while intra-class matrix P adds a smaller consistent gain.Source: §6 · pp. 8–9, Table 4 · p. 8

Qualitative Analysis

  • For an aircraft image, RVP ranks the ground-truth Cessna 525 first, while DVP ranks it fifth despite both matching attributes from other classes.RVP's inter-class modeling suppresses misleading shared semantics in this example.Source: §6 · p. 9, Figure 5 · p. 9

7 LIMITATIONS

RVP is less advantageous on Food101, where compositional food images vary substantially and class semantics are less stable.

7 LIMITATIONS

  • RVP performs less strongly on Food101 than on most benchmarks.The passage attributes this to food images combining a main dish with varying side dishes, garnish, sauces, and ingredients.Source: §7 · p. 9
  • Food images’ compositional contents make class semantics less stable across samples.Source: §7 · p. 9
  • High intra-class variation weakens text-based class relationships and reduces the benefit of structured inter-class modeling.Source: §7 · p. 9

8 CONCLUSION

RVP adapts frozen vision-language models through structured intra-class aggregation and inter-class modeling, and can be reparameterized as a single linear classifier at inference. Across 11 benchmarks and multiple CLIP backbones, it improves over prior visual reprogramming methods, with its largest gains on fine-grained tasks.

8 CONCLUSION

  • RVP adapts frozen vision-language models through structured intra-class aggregation and inter-class modeling.Source: §8 · p. 9
  • At inference, RVP admits exact reparameterization into a single linear classifier.Source: §8 · p. 9
  • Across 11 benchmarks and multiple CLIP backbones, RVP consistently improves over prior visual reprogramming methods, with the largest gains on fine-grained tasks.Source: §8 · p. 9

A DATASET INFORMATION

The appendix describes an 11-dataset, 16-shot benchmark and reports RVP results across additional CLIP backbones and broader adaptation comparisons. It also details the methods’ parameter costs.

A DATASET INFORMATION

  • The benchmark follows a 16-shot protocol across 11 datasets spanning fine-grained and generic objects, textures, scenes, actions, and remote sensing.The datasets range from 10 classes in EuroSAT to 397 in SUN397.Source: A DATASET INFORMATION · p. 13, Table 5 · p. 13

B.1 ADDITIONAL RESULTS ON DIFFERENT BACKBONES

  • RN101 and ViT-B/32 results give RVP average accuracies of 73.0% and 75.8%, beating DVP by 4.2 and 4.8 points and leading on 9 of 11 datasets with each backbone.The largest listed RN101 gains over DVP are Cars (+13.5); for ViT-B/32, Cars leads at +17.6.Source: §B.1 · pp. 13–14, Table 6 · p. 13, Table 7 · p. 13

B.2 BROADER COMPARISON

  • 82.7% average accuracy: RVP is competitive across the 11-dataset comparison with prompt learning, feature adaptation, task residual learning, and linear probing.Source: §B.2 · p. 14, Table 8 · p. 14
  • On StanfordCars, RVP slightly improves over LDC while using about 1.5% of LDC’s trainable parameters.The passage attributes this efficiency to exact reparameterization of RVP’s learned mapping into a single linear head at inference.Source: §B.2 · p. 14, Table 9 · p. 15

B.3 COMPUTATION COST

  • RVP uses 39,936 visual-prompt parameters, compared with 119,808 for DVP’s three prompts; its structured output mapping adds C × M intra-class and C × C inter-class parameters.The passage gives 69,840 visual-prompt parameters for VP and 39,936 for AR and AttrVR.Source: §B.3 · pp. 14–15

B.4 ADDITIONAL ANALYSIS

The additional analyses report consistent RVP gains under controlled prompt comparisons and examine inter-class correction, qualitative examples, and theoretical properties. They also identify benchmark settings where its advantage is less pronounced.

B.4 ADDITIONAL ANALYSIS

  • Across 11 datasets, normalized spectral rank r90/C is negatively associated with the accuracy gain from inter-class correction (partial Spearman ρ = −0.625, p = 0.040, controlling for class count).Leave-one-dataset-out partial correlations remain negative, ranging from −0.772 to −0.510.Source: §B.4 · p. 15
  • On StanfordCars, enabling E changes 36.2% of predictions; its stable rank is 15.6 versus 50.1 ± 1.0 under an entry-permutation null, and its correction is predominantly off-diagonal.The reported relative correction is 0.456; the diagonal-to-total Frobenius norm ratio is 0.075, near the null value of 0.072.Source: §B.4 · pp. 15–16
  • On Aircraft, RVP achieves the best accuracy with 51,936 total parameters, while DVP uses substantially more parameters and underperforms RVP.RVP uses the same number of prompt parameters as AR and AttrVR, with only modest additional structured-aggregation overhead.Source: §B.4 · p. 16, Table 11 · p. 16
  • Under the same text-prompt setting, RVP achieves 82.7% average accuracy, exceeding DVP by 4.0 points and AttrVR by 4.8, with DVP margins of 9.7 points on Aircraft and 15.7 on Cars.DVP is restricted to a single group of trainable visual prompts.Source: §B.4 · p. 16, Table 12 · p. 16
  • 82.7% average accuracy: RVP outperforms DVPlite by 2.9 points without directional prompt decomposition or LLM-based cause grouping.Source: §B.4 · pp. 16–17, Table 13 · p. 16

B.5 ADDITIONAL VISUALIZATION

  • In the aircraft and car examples, misleading top-matched attributes from similar classes accompany correct RVP predictions, while DVP fails.The passage attributes RVP’s advantage to suppressing confusing evidence from correlated classes.Source: §B.5 · p. 17, Figure 6 · p. 17

B.6 ADDITIONAL ERROR ANALYSIS

  • Food101’s compositional variation weakens text-based class relationships, while EuroSAT’s low-resolution satellite crops create visual and spatial ambiguity that inter-class modeling addresses less directly.The EuroSAT examples include visually similar Sea or lake, River, and Highway/Road classes.Source: §B.6 · p. 18, Figure 7 · p. 18, Figure 8 · p. 18

C.1 THEORETICAL JUSTIFICATION OF RVP

  • RVP’s inference classifier remains in the span of pretrained CLIP text embeddings, while the theory describes a residual correction that can remove shared low-rank semantics and bounds margin degradation by 2∥fE∥∞.The structured mapping restricts the dense hypothesis class to intra-class aggregation followed by inter-class correction, a low-complexity family for few-shot adaptation.Source: §C.1 · pp. 19–21

C.2 PROOF OF REMOVING VISUAL EMBEDDING NORMALIZATION AT INFERENCE

Visual embedding normalization can be omitted at inference without changing the predicted class when only classification decisions matter. The normalized and unnormalized logits differ by a positive, sample-specific factor, so their scale and calibration are not preserved.

C.2 PROOF OF REMOVING VISUAL EMBEDDING NORMALIZATION AT INFERENCE

  • Top-1 predictions are unchanged when visual embedding normalization is omitted at inference, provided only classification decisions matter.The normalized and unnormalized logits differ by a positive scalar shared across classes for each sample.Source: §C.2 · pp. 21–22
  • The shared positive factor preserves pairwise logit ordering, making the unnormalized inference rule equivalent in form to a standard linear classifier on backbone features.The proof derives this from the unchanged signs of pairwise logit differences.Source: §C.2 · p. 22
  • Removing normalization does not preserve absolute logit scale or calibrated confidence and may affect softmax probabilities or procedures that depend on logit scale.The change is a sample-dependent multiplicative factor.Source: §C.2 · p. 22

D IMPLEMENTATION DETAILS.

The implementation fixes training settings and reports resource use for RVP. The appendix also catalogs abbreviations and notation used in the paper.

D IMPLEMENTATION DETAILS.

  • Training uses SGD with learning rate 40, momentum 0.9, cosine annealing for 200 epochs, batch size 64, and learning rate 10−3 for P and E.The experiments use M=20 prompts per class; temperature τ is inherited from pretrained CLIP and kept fixed, and the method adds no hyperparameters.Source: D IMPLEMENTATION DETAILS · p. 22
  • 6.37 GB of GPU memory is used during training; the experiments run on one 48 GB NVIDIA L40S GPU and take about 47.5 hours across 11 datasets.Figure 9 presents host memory on the left and GPU memory on the right.Source: D IMPLEMENTATION DETAILS · p. 22, Figure 9 · p. 23

E NOTATIONS

  • The appendix catalogs abbreviations and notation, including generic visual-reprogramming notation and notation for the linear-mapping view.Table 14 lists abbreviations; Tables 15 and 16 cover the two notation scopes.Source: E NOTATIONS · p. 23, Table 14 · p. 24, Table 15 · p. 24

Related papers

Loading 2609.36680v1…