PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Models

Yang Chen, Zhan Zhuang, Yanbin Wei, Zebin Chen, Hua Liu, Yu Zhang

arXiv:2608.17306v1cs.CVcs.AI

TL;DR

Adversarial prompt tuning can overfit robust generalization on seen classes, limiting robustness to adversarial examples from unseen classes. ADAPT disentangles pseudo-robust shortcuts from target features with decoy prompts and an orthogonal loss; across 10 datasets, ADAPTM surpasses FAP by 1.73% in ‘Ha’.

  • Problem

    Adversarial prompt tuning can improve robustness on seen classes but exhibits robust-generalization overfitting, limiting adversarial robustness on unseen classes.

    Source: §1 · p. 2, §3.1 · p. 3

  • Method

    ADAPT uses a target prompt and decoy-prompt pool, training decoys to absorb pseudo-robust shortcuts and an orthogonal loss to disentangle target features.

    Source: §1 · p. 2

  • Results

    Across 10 datasets, ADAPTM outperforms previous SOTA FAP by 1.73% in ‘Ha’ under a diverse attack suite.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    The findings identify reliance on pseudo-robust features as the source of robustness-generalization failure in this setting.

    Source: §1 · p. 2, §7 · p. 8

    Alternative-backbone, computational-cost, adversarial cross-dataset, and adversarial domain-generalization results are deferred to Appendix C due to page limits.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

While adversarial prompt tuning can enhance robustness of vision-language models efficiently, we find that existing methods aggravate robust generalization overfitting on seen classes, leading to a rapid degradation in performance against adversarial examples of unseen classes as training progresses. We empirically identify that this degradation stems from the tendency of the model to learn pseudo-robust features (i.e., non-generalizable shortcuts). To mitigate this, we propose ADAPT (Adversarial Disentangled Prompt Tuning), a robust prompt tuning framework following the philosophy of ``Learning What Not to Learn''. Specifically, ADAPT uses a dual-prompt mechanism with a target prompt and a pool of decoy prompts. During training, the decoy prompts are guided to entrap diverse pseudo-robust features, while the target prompt is constrained to be orthogonal to the decoys in the embedding space to learn robust features. By disentangling the robust features from the pseudo-robust features, ADAPT effectively prevents robust generalization overfitting. We further provide an analysis showing that the orthogonal loss bounds the effect of shifts in pseudo-robust features on unseen classes, yielding a testing error guarantee. Empirically, extensive experiments demonstrate that ADAPT substantially improves the robustness of the target prompt on unseen classes. The code is available at https://github.com/cheny02/ADAPT-ACMMM2026.

1 Introduction

Vision-language models deliver strong zero-shot performance but are vulnerable to imperceptible adversarial perturbations [12, 35, 50]. Adversarial prompt tuning offers an efficient robustness approach compared with costly full-parameter adversarial fine-tuning [6, 22, 24, 37, 44, 49, 52].

Introduction

  • On unseen classes, robust test loss rises and the robust–clean test-loss gap widens during training, while both remain stable on seen classes.The authors call this failure to transfer robustness “robust generalization overfitting.”Source: Figure 1 · p. 2, §1 · p. 2
  • The authors attribute robust generalization overfitting to pseudo-robust shortcuts that fit seen-class attack patterns but fail to generalize semantically to unseen classes [11].These features can defend against attacks on seen data while limiting robustness on unseen categories.Source: §1 · p. 2
  • ADAPT uses decoy prompts to absorb diverse pseudo-robust features and an orthogonal loss to separate them from a target prompt learning generalizable robust semantics.The target prompt is thereby discouraged from using shortcuts captured by the decoys.Source: §1 · p. 2
  • ADAPT’s theoretical guarantee bounds unseen-class generalization error by minimizing the target prompt’s projection onto the shortcut subspace.Source: §1 · p. 2
  • Across 15 datasets, ADAPT outperforms state-of-the-art methods in adversarial base-to-new generalization and mitigates the robustness gap between seen and unseen classes.Source: §1 · p. 2

2 Related Work

Related work establishes CLIP as the foundational backbone and traces adversarial robustness research from costly full-model training to efficient prompt tuning, whose unseen-class weakness motivates ADAPT.

CLIP-based VLMs

  • CLIP [32], pretrained on about 400 million image-text pairs, established transferable vision-language representations, and this work uses it as the baseline backbone.Later CLIP-style extensions include ALIGN [18], OpenCLIP, SigLIP [47], MetaCLIP [46], DFN [8], and EVA-CLIP-18B [39] [40].Source: §2 · p. 2

Adversarial training of VLMs

  • Full-parameter adversarial fine-tuning methods including TeCoA [27], FARE [36], PMG-AFT [43], SLADE, and AdvSimplex can defend VLMs but incur prohibitively high computational costs.Adversarial training [25] is presented as an effective defense against image perturbation attacks [12] [35] [50].Source: §2 · p. 2

Adversarial prompt tuning

  • Prompt tuning [19] [51] enables more efficient adversarial robustness methods, but AdvPT [49], APT [22], and FAP [52] still struggle on unseen classes, motivating ADAPT’s disentangled dual-prompt design.ADAPT guides decoy prompts to absorb pseudo-robust features while enforcing orthogonality on the target prompt.Source: §2 · p. 3

3 Methodology

The methodology motivates ADAPT by identifying robust generalization overfitting in adversarial prompt tuning, then separates shortcut-like features from robust ones using decoy and target prompts. It also gives conditions under which this disentanglement bounds unseen-class testing error.

3.1 Preliminaries

  • Adversarial prompt tuning improves robustness on seen classes but suffers from robust generalization overfitting.The section identifies this failure as the motivation for ADAPT.Source: §3.1 · p. 3

3.2 A Motivating Example

  • In the toy task, the shortcut margin is 6.0 versus 3.0 for semantics, and flipping shortcut correlation yields 0% robustness for APT on new classes.The example holds the semantic feature invariant while shifting new-class centroids so the shortcut correlation flips.Source: §3.2 · p. 3

3.3 ADAPT

  • ADAPT pairs a target prompt with a pool of K decoys, sampling decoys during training to cover diverse shortcut directions and using only the target at inference.Each decoy is independently initialized and updated only when selected; inference adds no computational overhead versus standard APT.Source: §3.3 · p. 4
  • Each sampled decoy is optimized on adversarial examples generated by the current target while being pushed away from hand-crafted semantic anchors, making it a shortcut trap.The decoy combines adversarial classification with a loss that encourages dissimilarity from the semantic anchor.Source: §3.3 · p. 4
  • The target is trained to be orthogonal to decoys while retaining semantic-anchor consistency and adversarial classification, with stop-gradient preventing orthogonality updates to decoys.ADAPT alternates updates to the sampled decoy and target prompts.Source: §3.3 · p. 4, Figure 3 · p. 5

3.4 Theoretical Analysis

  • When γ > 2ε_sem + 2Bε_U, the target classifier is invariant to arbitrary shortcut shifts and its unseen-class testing error is bounded by ζ.The guarantee applies when the target is sufficiently close to the semantic anchor and orthogonal to the shortcut subspace.Source: §3.4 · p. 5

4 Experiments

ADAPT improves adversarial transfer performance across unseen classes, few-shot learning, and stronger attacks, while ablations support the contribution of its prompt components.

4.1 Setups

  • Experiments span four adversarial transfer settings, using 11 datasets for the first three and ImageNet-to-OOD evaluation for domain generalization.The compared methods include TeCoA, APT, and FAP; adversarial training and evaluation use PGD under the ℓ∞ threat model, with 1/255 and 4/255 budgets.Source: §4.1 · pp. 5–6

4.2 Main Results

  • ADAPT exceeds APT on unseen classes by 8.85% accuracy and 4.69% robustness, improving Hb by 5.58% without sacrificing base-class performance.ADAPTM exceeds FAP on new classes by 1.62% accuracy and 1.59% robustness. ADAPT achieves 60.04% average base-class accuracy, while ADAPTM achieves 28.52% average base-class robustness.Source: Table 1 · p. 6, §4.2 · pp. 6–7

4.1 Setups

  • ADAPTM extends ADAPT to multimodal prompting, coupling target and decoy deep visual prompts to projected textual prompts layer-by-layer.A lightweight cross-modal projection maps the visual prompts into text embedding space, and both prompt types enter the corresponding transformer blocks.Source: §4.1 · p. 6

4.2 Main Results

  • In 1-shot evaluation, ADAPT beats APT (TeCoA) by 0.90% (1.9%) robustness and 1.64% (2.7%) H, while ADAPTM leads all settings.The few-shot experiments tune on 1, 4, or 16 shots per class and evaluate on remaining samples across 11 datasets.Source: Figure 4 · p. 7, §4.2 · p. 7
  • Across CW and AA evaluations, ADAPTM exceeds FAP by 1.73% Ha, while ADAPT exceeds APT by 5.23% Ha.Table 2 covers 10 datasets, excluding ImageNet; Ha is the harmonic mean of accuracy and robustness measures across base and new classes.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 Ablation Studies

  • Removing Lorth lowers Hb by 2.33%, while removing Lsem causes the largest degradation; removing Ldis or Pd also harms new-class generalization.Lsem tethers the target prompt to the general semantic space, while Lorth enforces independence between target and decoy prompts.Source: Table 3 · p. 7, §4.3 · p. 8

5 Analysis on Hyperparameter Sensitivity

Across 11 datasets under adversarial base-to-new generalization at ε=4/255, the sensitivity analysis finds that ADAPT performs best with two decoy prompts and moderate loss coefficients. It is relatively insensitive to α and λ within moderate ranges, while β must be large enough to preserve semantic knowledge.

Effect of K

  • Increasing K from 1 to 2 improves performance, but larger K adds no benefit and slightly complicates optimization.Multiple decoy prompts better capture diverse pseudo-robust features.Source: §5 · p. 8

Effect of α

  • ADAPT is relatively insensitive to α within [0.005, 0.1], with best performance at α=0.01.α controls the dissimilarity loss Ldis.Source: §5 · p. 8

Effect of β

  • Small β substantially degrades performance, which stabilizes for β∈[30, 40], highlighting the importance of preserving general semantic knowledge.β weights the semantic loss Lsem.Source: §5 · p. 8

Effect of λ

  • Performance improves substantially as λ increases from 0 to 0.05, peaks at λ=0.05, and declines only slightly at larger values.The orthogonal loss Lorth disentangles target and decoy prompts; performance remains relatively stable for larger λ values.Source: §5 · p. 8

6 Visualization of Prompt Embeddings

The t-SNE visualization shows target prompts aligned with general semantic anchors across base and new classes, while decoy prompts occupy distinct, non-semantic subspaces. Decoys perform well on base classes but degrade on new classes, consistent with diverse class-specific shortcuts that ADAPT traps to help the target learn purified robust features.

6 Visualization of Prompt Embeddings

  • Target prompts align with general semantic anchors across base and new classes, while decoys occupy distinct non-semantic subspaces separated from the anchors and one another.The visualization is on Caltech101.Source: §6 · p. 8, Figure 6 · p. 8
  • Decoy prompts perform well on base classes but degrade significantly on new classes, indicating class-specific shortcuts that lack transferability.The decoys diverge into distinct non-semantic subspaces, suggesting that different prompts converge to diverse shortcuts.Source: §6 · p. 8
  • By trapping diverse shortcuts with a pool of decoy prompts, ADAPT allows the target prompt to learn purified robust features.The visualization supports the role of the semantic loss in enforcing semantically invariant features in the target prompts.Source: §6 · p. 8

7 Conclusion · Contents of the Appendix · A Theoretical Analysis

The paper attributes robust generalization overfitting to reliance on pseudo-robust features and presents ADAPT as a way to separate them from robust features. Its conclusion reports state-of-the-art benchmark performance, particularly for base-to-new generalization.

7 Conclusion

  • The paper formally defines robust generalization overfitting in adversarial prompt tuning.This identifies the central problem addressed by the work.Source: §7 · p. 8
  • The failure of robustness generalization stems from the model’s reliance on pseudo-robust features.Source: §7 · p. 8
  • ADAPT uses decoy prompts to entrap non-generalizable pseudo-robust features and an explicit orthogonal loss to guide the target prompt toward robust features.The method follows the philosophy of “Learning What Not to Learn”.Source: §7 · p. 8
  • Experiments on benchmark datasets show that ADAPT achieves state-of-the-art performance, particularly in base-to-new generalization settings.Source: §7 · p. 8

Contents of the Appendix

  • The appendix is organized into theoretical analysis (A), setting details (B), and additional experimental results (C).Source: Contents of the Appendix · p. 11
  • Appendix C covers alternative VLM backbones, base-to-new and adversarial classification results, domain and cross-dataset generalization, computational cost, semantic–shortcut decomposition, training settings, and adaptive attacks.Source: Contents of the Appendix · p. 11

A.1 Assumptions and Main Results (restated)

The restated analysis gives a new-class testing-error guarantee under assumptions that separate semantic features from bounded pseudo-robust features whose distribution may shift. It also requires semantic anchors and a probabilistic margin condition for new classes.

A.1 Assumptions and Main Results (restated)

  • All CLIP image and text embeddings are ℓ2-normalized unless stated otherwise.Source: §A.1 · p. 11
  • The analysis decomposes image features into semantic S and pseudo-robust U components, bounds ∥u(x)∥₂ ≤ B, and allows U’s distribution to shift arbitrarily from base to new classes.The decomposition is assumed for possibly adversarial inputs from either base or new classes.Source: §A.1 · p. 11
  • Hand-crafted text anchors lie in S, and new-class samples satisfy the anchor-margin condition with probability at least 1 − ζ.The stated margin is γ > 0.Source: §A.1 · p. 11
  • When the theorem’s stated condition holds, every new-class sample with m_H(x,y) ≥ γ is classified correctly regardless of bounded u(x), yielding new-class testing error at most ζ.The guarantee applies when ∥u(x)∥₂ ≤ B.Source: §A.1 · p. 11

A.2 Proof of Theorem A.2 … C.1 Generalization to Alternative VLM Backbones

Theorem A.2 bounds new-class error when the anchor margin exceeds semantic and shortcut shifts, and experiments show ADAPT’s robustness gains transfer across alternative VLM backbones. The merged material also specifies the evaluation and implementation settings.

A.2 Proof of Theorem A.2

  • For new-class samples with anchor margin at least 𝛾, the score difference is at least 𝛾−2𝜀sem−2𝐵𝜀U, ensuring correct classification when 𝛾>2𝜀sem+2𝐵𝜀U.Because the margin bound holds with probability at least 1−𝜁, it yields the new-class error guarantee; it is also invariant to base-to-new shifts in 𝑢(x) within ∥𝑢(x)∥2≤𝐵 when 𝑠(x) stays fixed.Source: §A.2 · p. 11

B Setting Details

  • The evaluation covers four adversarial settings; the first three use 11 classification datasets, while domain generalization trains on ImageNet and tests on four OOD variants.Experiments use ViT-B/32 CLIP; AutoAttack combines APGD-CE, APGD-T, FAB-T, and Square with one restart each.Source: B Setting Details · p. 12

C.1 Generalization to Alternative VLM Backbones

  • On ViT-L/14, ADAPT raises new-class robustness from APT’s 32.47% to 38.01% and Hb from 48.18% to 52.87%; ADAPTM reaches 53.94% Hb versus FAP’s 52.38%.The comparison is for adversarial base-to-new generalization across 10 datasets at 𝜖=4/255.Source: §C.1 · p. 12
  • On the FARE backbone, ADAPT improves new-class robustness from APT’s 13.49% to 20.31% and Hb from 28.25% to 35.54%; ADAPTM reaches 39.24% Hb versus FAP’s 36.65%.These averages cover 10 datasets under a perturbation budget of 𝜖=4/255.Source: §C.1 · p. 12
  • On OpenCLIP ViT-B/32, ADAPT raises new-class accuracy and robustness from APT’s 39.11% / 13.69% to 47.13% / 19.09%.The evaluation averages 10 datasets under 𝜖=4/255; the gains transfer to a CLIP-like model with a different pretraining pipeline.Source: §C.1 · p. 12
  • Across alternative VLMs, ADAPT consistently improves over APT, supporting its broad applicability to enhancing robustness across different models.The paper attributes these gains to addressing standard adversarial prompt tuning’s difficulty with pseudo-robust features.Source: §C.1 · p. 12

C.2 Comprehensive Results of Adversarial Base-to-new Generalization · C.3 Full Results of Adversarial Few-shot Classification

Under adversarial base-to-new generalization, ADAPT mitigates APT’s overfitting to seen classes, while ADAPTM achieves the strongest reported new-class results. In adversarial few-shot classification, ADAPTM records the highest average H score across the reported 1-, 4-, and 16-shot settings.

C.2 Comprehensive Results of Adversarial Base-to-new Generalization

  • ADAPT’s adversarial base-to-new results at 𝜖=1/255 show robustness transferability consistent with results at 𝜖=4/255, while APT overfits to seen classes and drops on new classes.ADAPT outperforms APT on new classes by 7.06% in accuracy and 6.35% in robustness on average across 11 datasets.Source: §C.2 · pp. 12–13
  • ADAPTM achieves the highest average new-class accuracy of 61.21% and robustness of 45.89%, surpassing FAP by 4.59% and 2.55%, respectively.The results support the effectiveness of disentangling robust from pseudo-robust features.Source: §C.2 · p. 13

C.3 Full Results of Adversarial Few-shot Classification

  • Adversarial few-shot classification evaluates performance on 11 datasets using 1, 4, and 16 shots per class.The reported comparison describes the proposed methods as data-efficient relative to baselines.Source: §C.3 · p. 13
  • ADAPTM achieves the highest average H score in the 1-shot setting, at 19.96%.The paper attributes few-shot effectiveness to the dual-prompt mechanism with orthogonal loss, which mitigates shortcut reliance under scarce labeled data.Source: §C.3 · p. 13
  • ADAPTM achieves the highest average H score in the 4-shot setting, at 24.78%.The reported results characterize the approach as effective at learning robust representations with scarce labels.Source: §C.3 · p. 13
  • ADAPTM achieves the highest average H score in the 16-shot setting, at 30.11%.The proposed dual-prompt mechanism with orthogonal loss is reported to mitigate reliance on shortcuts.Source: §C.3 · p. 13

C.4 Results of Adversarial Domain Generalization · C.5 Results of Adversarial Cross-dataset Generalization

ADAPT achieves strong adversarial domain-generalization averages and transfers to unseen datasets, where ADAPTM records the highest average harmonic mean. The evaluations use ImageNet as the source domain and test transfer across target domains or datasets.

C.4 Results of Adversarial Domain Generalization

  • On the source domain, ADAPT achieves the highest accuracy of 41.34% and competitive robustness compared with baselines.Source: §C.4 · p. 13
  • For domain generalization, models are tuned with 16-shot samples from each of ImageNet’s 1,000 classes and evaluated on four target domains.The targets are ImageNet-V2, ImageNet-Sketch, ImageNet-A, and ImageNet-R.Source: §C.4 · p. 13
  • ADAPT achieves average accuracy of 23.92%, robustness of 7.84%, and H of 11.81% across adversarial target domains.The reported averages are the best average performance among the compared methods.Source: §C.4 · p. 13
  • ADAPT surpasses APT in H score by 0.40% on ImageNet-Sketch and 0.85% on ImageNet-R.The paper attributes OOD transfer to disentangled features rather than overfitting source-distribution shortcuts.Source: §C.4 · p. 13

C.5 Results of Adversarial Cross-dataset Generalization

  • ADAPTM achieves the highest average harmonic mean of 17.43% in adversarial cross-dataset generalization.The methods consistently outperform baselines in overall performance.Source: §C.5 · p. 13
  • For cross-dataset generalization, models are trained on ImageNet with 16 shots per class and evaluated directly on 10 other datasets.The results indicate transferability to unseen domains.Source: §C.5 · p. 13

C.6 Computational Cost Analysis … C.9 Decoy-Guided Adaptive Attack

The appendices assess ADAPT’s computational practicality, validate the semantic–shortcut decomposition, test whether robust generalization overfitting persists across training settings, and evaluate a decoy-aware adaptive attack that reduces—but does not eliminate—robustness.

C.6 Computational Cost Analysis

  • Training memory is 27470M for ADAPT and 27530M for ADAPTM, versus 11454M for APT and 19808M for FAP, primarily due to maintaining decoy prompts.Memory scales with class count and decreases significantly on datasets with fewer categories than ImageNet; the paper considers this a justifiable training-time trade-off for generalization.Source: §C.6 · p. 13
  • ADAPT adds no inference cost over APT because decoy prompts are discarded after training, preserving inference efficiency while optimizing its dual-prompt mechanism.During training, adversarial examples are generated once using the target prompt, and only one decoy prompt is used per iteration.Source: §C.6 · p. 14
  • ADAPT achieves a highly favorable trade-off between computational cost and robustness in the ImageNet adversarial base-to-new setting.The comparison uses ε=4/255.Source: §C.6 · p. 13, Table A7 · p. 16

C.8 Robust Generalization Overfitting under Different Training Settings

  • APT’s robust loss gap between base and new classes appears across learning rates 0.001–0.004 and training durations of 50–200 epochs, though its magnitude varies.This indicates the overfitting is not tied to a single learning rate or training duration.Source: Figure A1 · p. 16, §C.8 · p. 16

C.7 Empirical Validation of the Semantic–Shortcut Decomposition

  • On DTD, the projection analysis finds that z_U performs better on base classes, while z_S transfers better to new classes, supporting the semantic–shortcut abstraction.The analysis decomposes image features into a learned base-useful component z_U and residual z_S, classified with the same hand-crafted prompt.Source: §C.7 · p. 15, Table A8 · p. 16

C.9 Decoy-Guided Adaptive Attack

  • DGAA selects three shortcut-confusing non-ground-truth classes using target-prompt competition and decoy-prompt shortcut-match scores, then runs target-prompt margin PGD toward each.It keeps the adversarial example that most reduces the target-prompt margin.Source: §C.9 · p. 16
  • DGAA lowers ADAPT robustness from 27.48%/15.65% under standard PGD to 23.62%/13.66% on base/new classes, yet leaves non-trivial robustness.The results are averaged over 11 datasets with ε=4/255.Source: Table A9 · p. 16, §C.9 · p. 16

Related papers

Loading 2608.17306v1…