PaperlayerView exact revision on arXiv

Source-linked AI summary

You Only Reprogram Once: Rethinking Prolonged Training for Visual Reprogramming

Zizhao Li, Mohammed Yaqoob Ansari, Xinyu Su, Jiayang Ao, Joseph West, Kourosh Khoshelham

arXiv:2609.36661v1cs.CV

TL;DR

Visual reprogramming commonly relies on lengthy prompt optimization, and it is unclear whether that training is necessary when the frozen response may already contain useful downstream structure. YORO constructs a predictor from full frozen responses in one forward-only traversal using Bayesian Discriminant Mapping. Across three full-data settings, it improves average accuracy over the strongest prior gradient-free mapping, while YORO-FP optionally refines prompts when adaptation helps.

  • Problem

    Visual reprogramming can require repeated backpropagation through a frozen backbone for hundreds of epochs, raising whether effective adaptation requires prolonged prompt optimization.

    Source: §1 · p. 1

  • Method

    YORO uses Bayesian Discriminant Mapping to turn complete frozen responses into an affine downstream predictor from streaming class statistics in one forward-only traversal.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Across three full-data backbone–input settings, YORO improves average accuracy over the strongest prior gradient-free label mapping by 18.4–24.4%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support checking how much downstream structure is recoverable from frozen responses before adding prompt optimization, with YORO-FP covering cases where input adaptation helps.

    Source: §1 · p. 2, §6 · p. 9

    YORO assumes useful target-label structure remains in frozen responses; experiments focus on image classification, leaving other tasks and modalities for future work.

    Source: §5 · p. 9

Abstract

from arXiv · show

Visual reprogramming is a parameter-efficient method for adapting pretrained models, yet its training can remain computationally expensive: even with a frozen backbone, visual prompts are often optimized through the full model for hundreds of epochs. Before changing what the pretrained model sees, we ask whether we are fully using what it already tells us. We find that modeling the full source response can already yield strong downstream predictions without prompt optimization. Motivated by this observation, we introduce You Only Reprogram Once (YORO), which constructs a downstream predictor from the frozen response space in a single forward-only traversal. Its Bayesian Discriminant Mapping (BDM) derives a covariance-aware affine mapping from streaming class statistics, requiring no backpropagation, optimizer updates, or repeated visits to the training set. When further input adaptation helps, YORO-FP optionally refines the visual prompt for 20 epochs. BDM also extends naturally to CLIP by treating attribute-prompt similarities as source responses. Across three full-data settings, YORO improves average accuracy over the strongest prior gradient-free mapping by 18.4--24.4\%. On 16-shot CLIP, it raises the four-backbone average from 71.4\% to 77.2\%. YORO-FP provides further gains on selected tasks, while validation often retains the one-pass predictor. These results suggest a different default for visual reprogramming: read out the frozen response first, and optimize the input only when needed.

1 INTRODUCTION

Visual reprogramming often trains prompts for hundreds of epochs, but weak initial performance may reflect an incomplete readout of the frozen model rather than insufficient downstream information. YORO reads the full frozen response in one pass, and its results show that prompt optimization need not be the default.

1 INTRODUCTION

  • Prior mappings use one-to-one correspondences, predicted labels, or selected probabilities, potentially discarding source-response information that separates downstream classes.Prompt optimization and label mapping are usually evaluated together, so poor initial performance may reflect either insufficient downstream information or an insufficient readout.Source: §1 · p. 1
  • With the same ResNet–18 and watermark-style prompt, a stronger readout of the initial frozen response outperforms a weaker mapping paired with prolonged optimization.On CIFAR-100, one traversal lets YORO outperform BLM trained for 200 epochs; continued prompt training can reduce training loss without improving validation accuracy.Source: §1 · p. 1, Figure 1 · p. 2
  • YORO fixes the input transformation at initialization and constructs a predictor from labeled responses in one forward-only traversal, without backpropagation or prompt updates.It needs no intermediate features or model parameters and can use output-access black-box models when full response vectors are available.Source: §1 · p. 2
  • YORO-FP optionally refines the visual prompt when the initial frozen response lacks sufficient downstream structure, making adaptation's value assessable against the one-pass predictor.The full-data and few-shot CLIP results also show additional gains on selected tasks, while validation frequently retains the original one-pass predictor.Source: §1 · p. 2
  • 18.4–24.4%: Across three full-data backbone–input settings, YORO improves average accuracy over the strongest prior gradient-free label mapping.It also exceeds the reported learned linear mapping by 6.3–12.3%.Source: §1 · p. 2

2 RELATED WORK

Prior visual-reprogramming mappings range from one-to-one label assignments to dense mappings based on predicted labels, while prompt optimization often remains multi-epoch. YORO instead maps complete pretrained responses directly and constructs its predictor in one traversal.

Visual reprogramming and label mapping

  • RLM, FLM, and ILM use one-to-one source–target assignments, while BLM and BLM+ use top-1 predictions or top-K probabilities; these pipelines still rely on multi-epoch prompt optimization.RLM uses random assignments, FLM selects frequently predicted labels, and ILM updates the mapping during prompt training.Source: §2 · p. 3
  • Recent CLIP-based methods improve image–text alignment using class attributes, multiple visual prompts, or semantic label hierarchies and multi-scale representations.These methods include AttrVR (Cai et al., 2025a), DVP (Cai et al., 2025b), and Dual-Granularity Alignment.Source: §2 · p. 3

Training-efficient adaptation

  • BDM models complete pretrained responses rather than compressed predictions or selected similarities, and YORO builds a streaming one-pass interface with dataset-size-independent memory.Unlike Gaussian discriminant analysis in CLIP visual-feature space, BDM does not require intermediate-feature access; YORO-FP tests a 20-epoch prompt budget.Source: §2 · p. 3

3 METHOD

YORO constructs an affine downstream predictor from the frozen source response in one training traversal, with BDM using class statistics and shared covariance. If the frozen response lacks sufficient downstream structure, YORO-FP optionally refines the visual prompt.

3.1 SETTING AND THE ONE-PASS QUESTION

  • YORO holds the input prompt fixed and constructs a downstream predictor from classifier logits or CLIP attribute-prompt similarities using forward passes only.For CLIP, the attribute prompts and both encoders remain fixed, and the similarities are flattened into the response vector used by BDM.Source: §3.1 · pp. 3–4

3.2 BAYESIAN DISCRIMINANT MAPPING

  • BDM models the full response with target-class means and shared within-class covariance, yielding an affine Bayesian decision rule that accounts for correlations among response dimensions.Its covariance-adjusted class means determine mapping weights, while class-conditional locations and priors contribute to the bias.Source: §3.2 · p. 4
  • When high-dimensional covariance estimates are noisy or singular, BDM shrinks covariance toward a trace-scaled identity to stabilize estimation while retaining supported covariance structure.The shrinkage parameter trades empirical covariance structure against isotropic stability; an added term is used only when required for stable factorization.Source: §3.2 · pp. 4–5
  • BDM selects its shrinkage parameter by validation accuracy, reusing streaming statistics and cached validation responses across candidates.Candidate-specific work is limited to forming the shrunk covariance, solving the mapping, and scoring cached responses.Source: §3.2 · p. 5
  • BDM recovers exactly the same affine mapping from streaming statistics as joint fitting, so YORO needs neither to store responses nor revisit the training set.The exact recovery holds for fixed responses and shrinkage.Source: §3.2 · p. 5

3.3 YORO-FP: CONTROLLED FEW-PASS REFINEMENT

  • When frozen responses do not preserve sufficient target structure, YORO-FP starts from the one-pass predictor and allows 20 prompt epochs while keeping the backbone frozen.Each epoch updates only the prompt; detached minibatch logits supply statistics for a candidate mapping that is blended with the previous mapping using β = 0.9.Source: §3.3 · p. 5

4 EXPERIMENTS

Across the evaluated visual-reprogramming and few-shot CLIP settings, YORO’s one-pass readout outperforms prior mappings on average, while prompt refinement helps selectively. The experiments compare accuracy, adaptation cost, validation-selected refinement, and mapping-update strategies.

4 EXPERIMENTS

  • The protocols evaluate full-data visual reprogramming on image classifiers and 16-shot adaptation on 11 datasets with four frozen CLIP backbones.Full-data settings use ResNet–18 or ResNeXt–101 and compare against prior mappings, including a learned linear mapping; YORO-FP uses 20 additional prompt-training epochs.Source: §4 · pp. 5–6

4.1 RESULTS ON IMAGE CLASSIFIERS

  • YORO outperforms prior gradient-free label mappings on average across all three full-data settings, without iterative prompt optimization.Tables 1 and 2 compare padding and watermarking settings.Source: §4.1 · p. 6, Table 1 · p. 6, Table 2 · p. 6
  • On watermarking with ResNet–18, YORO reaches 69.1% average accuracy versus 50.7% for BLM+ and 60.8% for Deep, while YORO-FP reaches 71.9%.YORO-FP’s largest gains are on GTSRB and SVHN; across the image-classifier results, YORO also outperforms BLM+ under legacy padding.Source: §4.1 · p. 7

4.2 FEW-SHOT RESULTS ON CLIP

  • On 16-shot CLIP, YORO or YORO-FP achieves the highest average accuracy across all four backbones, with stronger gains on weaker backbones.YORO improves DVP from 26.1% to 36.6% on FGVC Aircraft and from 56.5% to 71.8% on StanfordCars with ViT-B/32; YORO-FP adds little on average.Source: §4.2 · p. 7, Table 3 · p. 7, Table 4 · p. 7

4.3 DOES VISUAL PROMPT TRAINING STILL HELP?

  • Epoch 0 is selected more often than refined checkpoints in five of seven backbone–setting combinations, while refinement is selected more often for ResNeXt–101 padding and ViT-B/16 CLIP.SVHN shows dataset-level gains from refinement across all three full-data settings, and gains are also observed on GTSRB.Source: §4.3 · p. 8, Figure 3 · p. 8

4.4 ACCURACY AND TRAINING COST.

  • On Oxford Flowers with CLIP ViT-B/32, YORO achieves 94.0% accuracy in 17.1 seconds, versus 91.8% for DGA in 6562.1 seconds.The compared methods were rerun under the same hardware protocol; YORO constructs its mapping from a single forward traversal.Source: §4.4 · p. 8, Table 5 · p. 8

4.5 ABLATION STUDY

  • The mapping-update ablations use 14-dataset full-data averages and 11-dataset few-shot CLIP averages to compare Frozen, Refresh, and Momentum strategies.The full-data study adds Caltech101 and RESISC45 to the 12-dataset published-baseline comparison.Source: §4.5 · pp. 8–9, Table 6 · p. 9, Table 7 · p. 9
  • For optional YORO-FP refinement, Momentum performs best in all three full-data settings and improves average accuracy over one-pass YORO by 0.6%, 2.4%, and 1.1% across the listed settings.Refresh generally improves over Frozen; Momentum smooths mapping updates. On few-shot CLIP, Momentum is strongest overall among YORO-FP update rules, but YORO remains best across the four-backbone average at 77.2%.Source: §4.5 · p. 9, Table 6 · p. 9, Table 7 · p. 9

5 LIMITATIONS

YORO relies on useful target-label structure remaining in the frozen response space, an assumption that may weaken when target labels differ substantially from source labels.

5 LIMITATIONS

  • YORO assumes the frozen response space retains useful structure for the target labels.The assumption may weaken when the target label space differs substantially from the source label space, as illustrated by SVHN.Source: §5 · p. 9
  • When that assumption weakens, visual prompt adaptation remains useful for reshaping responses.The experiments focus on image classification; extending the one-pass principle to other tasks and modalities remains an open direction.Source: §5 · p. 9
  • The evaluation scope is limited to image classification, with other tasks and modalities left for future work.Source: §5 · p. 9

6 CONCLUSION

The conclusion presents YORO as a strong one-pass baseline that improves prior label mappings by reading frozen responses with BDM. It recommends assessing recoverable downstream structure before adding prompt optimization, while retaining YORO-FP for cases where input adaptation helps.

6 CONCLUSION

  • YORO uses BDM to improve prior label mappings across full-data visual reprogramming and few-shot CLIP without a backward pass or prompt update.Source: §6 · p. 9
  • The paper concludes that one labeled-data traversal can produce a strong downstream predictor when frozen responses are read out effectively.Source: §6 · p. 9
  • The conclusion recommends first assessing recoverable downstream structure in frozen responses before adding prompt optimization.YORO-FP addresses the complementary case where additional input adaptation is useful.Source: §6 · p. 9

A IMPLEMENTATION AND EVALUATION DETAILS

The implementation uses a resized, centered image with a zero-initialized trainable watermark for the full-data protocol, and evaluates YORO and optional YORO-FP refinement. Few-shot CLIP uses 16 examples per class and complete attribute-prompt similarity responses.

Full-data protocol

  • The full-data setup resizes images to 192 × 192 in a 224 × 224 canvas, with a zero-initialized additive watermark containing 39,936 trainable RGB values.YORO-FP uses random resized crops and horizontal flips, except for GTSRB and SVHN; batch sizes are 256 for ResNet–18 and 64 for ResNeXt.Source: A IMPLEMENTATION AND EVALUATION DETAILS · p. 13
  • The full-data prompt is optimized with Adam at learning rate 0.01, while the backbone remains frozen.The experiments use three random seeds; training-dynamics and update ablations include Caltech101 and RESISC45 alongside the 12 published-baseline tasks.Source: A IMPLEMENTATION AND EVALUATION DETAILS · p. 13

Few-shot CLIP protocol

  • The 16-shot CLIP protocol evaluates four frozen backbones across 11 datasets, sampling 16 training examples and at most four validation examples per class.Each downstream class has 20 fixed attribute descriptions, and BDM uses the complete temperature-scaled image-to-attribute-prompt similarity vector.Source: A IMPLEMENTATION AND EVALUATION DETAILS · p. 13
  • For CLIP, equivalent statistics are accumulated in normalized visual-feature space using an exact low-rank identity instead of constructing the full logit covariance.YORO-FP uses prompt learning rate 0.1 and the same weak crop-and-flip family as the full-data protocol before CLIP normalization.Source: A IMPLEMENTATION AND EVALUATION DETAILS · p. 13

A.1 DATASET COVERAGE

The appendix specifies dataset suites, model selection and training protocols, then details prompt configurations and legacy-padding results. YORO remains stronger than prior gradient-free mappings under legacy padding, despite that protocol’s less consistent input scale.

A.1 DATASET COVERAGE

  • The full-data principal comparison covers 12 datasets, while the training-dynamics and mapping-update ablations add Caltech101 and RESISC45 for a 14-task suite.The two added datasets are not part of the published-baseline comparison.Source: §A.1 · p. 13
  • The few-shot CLIP suite comprises 11 datasets, combines several visual domains, and uses 16 training examples and at most four validation examples per class.It treats the complete CLIP image–text similarity vector as BDM’s response and retains 20 fixed attribute descriptions per class.Source: §A.1 · pp. 13–14
  • Model selection chooses shrinkage ρ from six validation candidates using training statistics and validation responses computed once and shared across candidates.YORO-FP evaluates the unrefined predictor and epochs 1–20, then selects the first checkpoint achieving the best validation accuracy; test data are evaluated once afterward.Source: §A.1 · p. 14
  • YORO uses one labeled training-set traversal with no backward pass, whereas YORO-FP uses 21 traversals in total.All main-text results are macro-averaged across datasets and backbones.Source: §A.1 · p. 14
  • The unified algorithm reuses each minibatch’s pre-update response for BDM statistics and prompt optimization, so epoch-boundary mapping updates require no extra traversal.Source: §A.1 · p. 14

A.2 VISUAL PROMPT CONFIGURATIONS

  • Padding prompts place trainable parameters in the border around a resized image, whereas watermarking overlays the trainable pattern on the resized image itself.The legacy padding protocol instead centers the original image without first resizing it, so image area varies with native resolution and can be small for low-resolution datasets.Source: §A.2 · pp. 14–15
  • Rescaling makes downstream images occupy a more consistent fraction of the pretrained input across datasets, while legacy padding can reduce effective image content.The reported legacy-padding gains show YORO remains effective across input configurations and under the same parameterization used by BLM.Source: §A.2 · p. 15
  • Under legacy padding, YORO’s average accuracy is 54.7% with ResNet–18 versus 45.3% for BLM and 46.7% for BLM+, and 67.4% with ResNeXt–101–32x8d.For ResNeXt–101–32x8d, YORO improves over BLM and BLM+ by 14.2 and 13.6 points, respectively, without iterative prompt optimization.Source: §A.2 · p. 15

B DERIVATION AND STREAMING EQUIVALENCE

The derivation connects the Gaussian decision rule to an affine label mapping and shows that streaming class statistics recover the same mapping as batch statistics. Under the legacy padding setting, YORO improves substantially over BLM and BLM+ across both pretrained backbones.

B DERIVATION AND STREAMING EQUIVALENCE

  • YORO substantially improves over BLM and BLM+ across both pretrained backbones under the legacy padding setting.Downstream images are padded directly to the pretrained input size without resizing.Source: Table 9 · p. 16
  • Removing the class-common quadratic term yields the affine class score and mapping parameters.The derivation identifies 2z⊤Σ−1z as common to all classes.Source: B DERIVATION AND STREAMING EQUIVALENCE · pp. 15–16
  • Streaming class counts and response sums recover batch class means and priors, producing the same affine mapping parameters.The proof states that deterministic shrinkage and a linear solve yield identical weights and biases.Source: B DERIVATION AND STREAMING EQUIVALENCE · p. 16

B.1 OPTIONAL EXACT LOW-RANK CLIP IMPLEMENTATION

For CLIP, YORO can compute BDM through an exact low-dimensional implementation that avoids forming the dense response-space covariance. The optional YORO-FP refinement adds substantial training cost for small aggregate accuracy changes.

B.1 OPTIONAL EXACT LOW-RANK CLIP IMPLEMENTATION

  • The CLIP shortcut solves a d×d system and forms the final mapping while producing exactly the same affine mapping as dense response-space BDM.It uses visual-feature statistics only as an implementation shortcut; the BDM formulation is unchanged.Source: §B.1 · p. 16
  • For the largest response, SUN397 with K = 7,940, a dense K × K second moment would occupy approximately 481 MiB, but the implementation never materializes it.The implementation stores O(d2 +kTd+Kd) values for CLIP and solves a d×d system.Source: §B.1 · p. 17
  • The implementation enforces one training-set traversal for YORO while keeping the backbone frozen.The authors state they will release code, configurations, per-seed results, and evaluation scripts.Source: §B.1 · p. 17

C DETAILED RESULTS AND ABLATIONS

  • YORO-FP uses 21 labeled passes and about 18–21× YORO’s training time, raising full-data mean accuracy by 0.6% but lowering few-shot CLIP mean accuracy by 0.2%.Validation retains the original YORO checkpoint in 54.8% and 58.3% of runs, respectively.Source: C DETAILED RESULTS AND ABLATIONS · p. 17, Table 10 · p. 17

C.1 ADDITIONAL COVARIANCE AND RESPONSE-SELECTION ABLATIONS

Covariance regularization improves FS-CLIP accuracy over unregularized LDA, while response pruning does not outperform full-response BDM. Validation generally favors retaining most response coordinates.

C.1 ADDITIONAL COVARIANCE AND RESPONSE-SELECTION ABLATIONS

  • On FS-CLIP, default BDM reaches 77.2%, compared with 72.7% for empirical LDA and 76.4% for OAS and RBLW without validation-based shrinkage selection.The ablation averages results across 11 datasets and four CLIP backbones.Source: §C.1 · p. 17, Table 11 · p. 17
  • Adaptive retains 97.3% of response coordinates on average, indicating that validation generally favors keeping most coordinates.Adaptive jointly selects the retention ratio and covariance-shrinkage coefficient using validation data.Source: §C.1 · p. 18
  • Conditional 95% reaches 77.0% and Adaptive reaches 76.9%, below BDM’s 77.2% on FS-CLIP.The authors retain the complete response by default and use shrinkage rather than explicit coordinate pruning.Source: Table 11 · p. 17, §C.1 · p. 18

C.2 CONTEXTUAL COMPARISON WITH EFFICIENT CLIP ADAPTATION

YORO compares favorably with an efficient CLIP adaptation baseline, and the results distinguish settings where prompt refinement helps from those where it does not. The supplied notation passages define symbols used by YORO and YORO-FP.

C.2 CONTEXTUAL COMPARISON WITH EFFICIENT CLIP ADAPTATION

  • On ten overlapping datasets, YORO exceeds Tip-Adapter by 2.8% using the complete attribute-prompt response as its only source representation.The comparison is intended as an efficiency-oriented context because training protocols differ.Source: §C.2 · p. 18, Table 12 · p. 18
  • With BDM frozen, prompt training raises SVHN accuracy by 3.7%, 6.8%, and 22.8% across the three tested settings.The particularly large watermarking gain is reported as evidence that input adaptation reshapes the response geometry.Source: §C.2 · p. 18, Table 13 · p. 18
  • In full data, Momentum improves the mean by 0.6% and produces the largest number of wins, while validation still retains YORO in 54.8% of runs.The comparison reports that Frozen and Refresh improve the mean by 0.2% and 0.1%, respectively.Source: §C.2 · p. 19, Table 14 · p. 19
  • In few-shot evaluation, all three update rules reduce aggregate accuracy, and validation selects no refinement in 58.3–66.7% of runs.The paper recommends spending the few-pass budget only when held-out validation shows that response-space adaptation transfers.Source: §C.2 · p. 19, Table 14 · p. 19

D NOTATION

  • The notation tables define symbols for YORO and YORO-FP, including affine mapping weights and bias, class-mean matrices, and minibatch response vectors.Source and target superscripts denote source and target, respectively.Source: D NOTATION · p. 21, Y Minibatch labels · p. 22
Loading 2609.36661v1…