PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Models

Yang Chen, Zhan Zhuang, Yanbin Wei, Zebin Chen, Hua Liu, Yu Zhang

arXiv:2608.17306v1cs.CVcs.AI

TL;DR

Adversarial prompt tuning can overfit seen classes by learning pseudo-robust shortcuts, weakening robustness on unseen classes. ADAPT uses decoy prompts and orthogonal target-prompt training to disentangle these shortcuts, achieving stronger robust generalization in experiments.

  • Problem

    Existing adversarial prompt tuning improves seen-class robustness but suffers robust generalization overfitting because pseudo-robust features fail to generalize to unseen classes.

  • Method

    ADAPT uses a target prompt and decoy-prompt pool to entrap pseudo-robust features while orthogonal loss separates the target prompt from decoys.

  • Results

    ADAPTM outperforms the prior state-of-the-art FAP by 1.73% in Ha across 10 datasets under diverse attacks.

  • Takeaways & Limitations

    ADAPT strengthens robust generalization to unseen classes without degrading base-class performance.

  • Takeaways & Limitations

    Additional results on alternative backbones, computational cost, cross-dataset generalization, and domain generalization are deferred to the appendix due to page limits.

Abstract

from arXiv · show

While adversarial prompt tuning can enhance robustness of vision-language models efficiently, we find that existing methods aggravate robust generalization overfitting on seen classes, leading to a rapid degradation in performance against adversarial examples of unseen classes as training progresses. We empirically identify that this degradation stems from the tendency of the model to learn pseudo-robust features (i.e., non-generalizable shortcuts). To mitigate this, we propose ADAPT (Adversarial Disentangled Prompt Tuning), a robust prompt tuning framework following the philosophy of ``Learning What Not to Learn''. Specifically, ADAPT uses a dual-prompt mechanism with a target prompt and a pool of decoy prompts. During training, the decoy prompts are guided to entrap diverse pseudo-robust features, while the target prompt is constrained to be orthogonal to the decoys in the embedding space to learn robust features. By disentangling the robust features from the pseudo-robust features, ADAPT effectively prevents robust generalization overfitting. We further provide an analysis showing that the orthogonal loss bounds the effect of shifts in pseudo-robust features on unseen classes, yielding a testing error guarantee. Empirically, extensive experiments demonstrate that ADAPT substantially improves the robustness of the target prompt on unseen classes. The code is available at https://github.com/cheny02/ADAPT-ACMMM2026.

1 Introduction

Vision-language models are powerful but fragile to visual adversarial perturbations, while full-parameter adversarial fine-tuning is impractical at scale. The paper identifies robust generalization overfitting on unseen classes and proposes ADAPT, which traps pseudo-robust shortcuts in decoy prompts so the target prompt learns generalizable robust features.

  • Motivation: VLMs achieve strong zero-shot performance on unseen data but can make confident, incorrect predictions under visual adversarial perturbations.
  • Motivation: Full-parameter adversarial fine-tuning is computationally prohibitive for modern large-scale VLMs, motivating parameter-efficient fine-tuning approaches.
  • Problem: Robust generalization overfitting occurs when robust losses on seen classes decrease or remain stable while robust test loss on unseen classes rises during adversarial prompt tuning.This failure specifically concerns transferring robustness to unseen semantic categories.
  • Problem: The degradation is attributed to pseudo-robust features that fit seen-class attack patterns but lack high-level semantic generalization.The model shortcuts by reverse-engineering perturbation patterns generated by the attack algorithm on base classes.
  • ADAPT: ADAPT uses a robust target prompt and sampled decoy prompts that entrap diverse pseudo-robust features, while an orthogonal loss separates the target from decoys.This design rejects shortcut features and directs the target prompt toward essential visual semantics that remain robust across unseen categories.

2 Related Work

Related work centers on CLIP-based vision-language models and parameter-efficient adversarial prompt tuning. CLIP established a dominant paradigm for transferable vision-language representations, while subsequent methods optimize prompts on adversarial examples to improve robustness.

  • CLIP-based VLMs: CLIP, pretrained on about 400 million image-text pairs through contrastive learning, established a dominant paradigm for transferable vision-language representations.Its success catalyzed a proliferation of CLIP-style architectures.
  • Adversarial Prompt Tuning: Recent research integrates adversarial training with prompt tuning, a parameter-efficient paradigm that optimizes continuous context vectors for downstream tasks.This integration addresses the efficiency constraints of adversarial training.
  • Adversarial Prompt Tuning: AdvPT and APT demonstrate that optimizing prompts on adversarial examples can enhance robustness.These methods represent a promising direction in adversarial prompt tuning.

3 Methodology

ADAPT addresses robust generalization overfitting by using decoy prompts to capture pseudo-robust shortcuts while constraining a target prompt to learn disentangled robust features. Its orthogonality-based design provides robustness-generalization guarantees to unseen classes under shortcut shifts.

  • Problem Diagnosis: APT improves robustness on seen classes but suffers from robust generalization overfitting, with performance failing on new classes when shortcut correlations change.In the toy setting, this failure reaches 0% robustness on new classes.
  • ADAPT Framework: ADAPT uses a dual-prompt mechanism in which decoy prompts capture pseudo-robust features and the target prompt learns robust features.This reverses the shortcut-learning tendency of single-prompt adversarial prompt tuning.
  • Decoy Prompt Pool: A pool of K independently initialized decoy prompts is updated through stochastic selection to learn diverse shortcut directions and cover the non-generalizable feature subspace.Only the target prompt is used during inference, so ADAPT adds no computational overhead compared with standard APT.
  • Prompt Disentanglement: Decoy prompts are pushed away from the hand-crafted semantic anchor and trained on adversarial classification loss, while target–decoy orthogonality is enforced by stop-gradient updates on the target prompt.When the orthogonality loss reaches 0, the target and decoy embeddings are orthogonal and disentangled.
  • Theoretical Guarantee: If the target prompt is close to the semantic anchor and orthogonal to the shortcut subspace, robustness carries over to unseen classes and predictions remain invariant under arbitrary shortcut shifts.Under γ > 2ε_sem + 2Bε_U, the testing error rate on new classes is bounded by ζ.

4 Experiments

Experiments evaluate ADAPT across four adversarial generalization settings and show stronger unseen-class robustness than competing prompt-tuning baselines. Ablations further support the contributions of its decoy prompts and disentanglement losses.

  • Experimental settings: Experiments cover adversarial base-to-new generalization, few-shot classification, cross-dataset generalization, and domain generalization across an 11-dataset suite or ImageNet OOD variants.The first three settings use 11 datasets, while domain generalization trains on ImageNet and evaluates on its out-of-distribution variants.
  • Adversarial base-to-new generalization: ADAPT improves new-class performance over APT by 8.85% in accuracy and 4.69% in robustness, yielding a 5.58% gain in ‘Hb’.ADAPTM also surpasses FAP by 1.62% in new-class accuracy and 1.59% in new-class robustness.
  • Adversarial base-to-new generalization: The robust generalization gains do not degrade base-class performance, and ADAPTM extends ADAPT to multi-modal prompting for comparison with FAP.ADAPT optimizes textual prompts, whereas ADAPTM adds learnable deep visual prompts for target and decoy prompts through coupling between vision and language branches.
  • Evaluation under stronger attacks: Under stronger CW and AA attacks, the proposed methods consistently outperform the baselines across 10 datasets excluding ImageNet.CW and AA are described as more potent than PGD, and ‘Ha’ aggregates accuracy and robustness across base and new classes.
  • Ablation study: Ablations report an average ‘Hb’ of 28.05%, while removing Lsem causes the most severe performance degradation.The ablation examines the decoy-prompt pool Pd, dissimilarity loss Ldis, semantic loss Lsem, and orthogonal loss Lorth; Lsem regularizes the target prompt toward general semantic space.

5 Analysis on Hyperparameter Sensitivity

The analysis evaluates K, α, β, and λ across 11 datasets under adversarial base-to-new generalization with ε = 4/255. Increasing K from 1 to 2 improves performance, while larger values add no benefit and complicate optimization; α controls the dissimilarity loss.

  • Effect of K: Increasing K from 1 to 2 improves performance by capturing more diverse pseudo-robust features, whereas larger K provides no additional benefit and slightly complicates optimization.The evaluation uses average performance over 11 datasets under adversarial base-to-new generalization with ε = 4/255.
  • Effect of α: α controls the dissimilarity loss Ldis in Eq. (4).

6 Visualization of Prompt Embeddings

t-SNE visualization shows that ADAPT disentangles target and decoy prompt embeddings: target prompts align with general semantic anchors, while decoys occupy separated non-semantic subspaces.

  • Prompt Embedding Disentanglement: The decoy prompts are separated from one another in the latent space.Figure 6 shows the decoys occupying a subspace distinct from the target prompts.
  • Prompt Embedding Disentanglement: Target prompts consistently align with general semantic anchors across base and new classes.This supports the semantic loss’s enforcement of semantically invariant features.
  • Prompt Embedding Disentanglement: Decoy prompts diverge into distinct non-semantic subspaces isolated from the semantic anchors.The visualization indicates that decoys are guided toward pseudo-robust, non-semantic representations.

7 Conclusion · Contents of the Appendix · A Theoretical Analysis

The paper identifies robust generalization overfitting as arising from reliance on pseudo-robust features and introduces ADAPT to separate these from robust features. The appendices provide theoretical analysis, setting details, and extensive additional experiments and analyses.

  • 7 Conclusion: The paper formally identifies robust generalization overfitting in adversarial prompt tuning as a failure stemming from reliance on pseudo-robust features.This phenomenon is presented as the source of robustness-generalization failure.
  • 7 Conclusion: ADAPT addresses robust generalization overfitting by entrapping non-generalizable pseudo-robust features in decoy prompts and guiding the target prompt toward robust features.It uses a dual-prompt mechanism with an explicit orthogonal loss, following the philosophy of “Learning What Not to Learn”.
  • 7 Conclusion: Experiments on benchmark datasets demonstrate the effectiveness of ADAPT.The supplied conclusion passage states that benchmark experiments support the method, but truncates the following quantitative results.
  • Contents of the Appendix: Appendix A contains the paper’s theoretical analysis.The appendix contents also list setting details and additional experimental results as separate sections.
  • Contents of the Appendix: Appendix C covers alternative VLM backbones, base-to-new generalization, adversarial few-shot classification, and adversarial domain and cross-dataset generalization.These topics are listed as Sections C.1 through C.5.
  • Contents of the Appendix: Additional appendices analyze computational cost, semantic–shortcut decomposition, robust generalization overfitting under different training settings, and decoy-guided adaptive attacks.These topics are listed as Sections C.6 through C.9.

A.1 Assumptions and Main Results (restated)

Under Assumption A.1, normalized CLIP features decompose into semantic and pseudo-robust subspaces, with bounded shortcut components whose distribution may shift arbitrarily across classes. The main result shows that a target classifier satisfying the stated projection condition predicts new-class samples correctly whenever the semantic margin is at least γ, yielding testing error at most ζ.

  • Notation: All CLIP image and text embeddings are ℓ2-normalized unless stated otherwise.
  • Assumption A.1: Assumption A.1 decomposes each possibly adversarial feature as z(x) = s(x) + u(x) across orthogonal semantic and pseudo-robust subspaces, with ∥u(x)∥2 ≤ B.The pseudo-robust component’s distribution may change arbitrarily from base to new classes, while text anchors lie in the semantic subspace.
  • Assumption A.1: The new-class anchor-margin assumption provides γ > 0 with probability at least 1 − ζ over a new-class sample.
  • Main result: If the target classifier satisfies the stated orthogonal-projection condition, any new-class sample with m_H(x,y) ≥ γ is classified correctly regardless of u(x).This guarantee holds as long as ∥u(x)∥2 ≤ B.
  • Main result: The testing error rate of the target classifier on new classes is at most ζ.

A.2 Proof of Theorem A.2 … C.1 Generalization to Alternative VLM Backbones

Theorem A.2 establishes correct new-class prediction when γ > 2ε_sem + 2Bε_U, with error at most ζ and invariance to shortcut-only shifts. Experiments use four adversarial evaluation settings and show ADAPT improves robustness across TeCoA, FARE, and OpenCLIP backbones.

  • A.2 Proof of Theorem A.2: The score difference decomposes into semantic and shortcut components under z(x) = s(x) + u(x), separating robust semantics from shortcut effects.The decomposition follows Assumption A.1(1), with s(x) in S and u(x) in U.
  • A.2 Proof of Theorem A.2: For every competing class c ≠ y, Δ_y,c(x) ≥ γ − 2ε_sem − 2Bε_U; thus γ > 2ε_sem + 2Bε_U guarantees correct prediction.The bound is obtained by combining semantic and shortcut lower bounds.
  • A.2 Proof of Theorem A.2: The classifier’s new-class error is at most ζ because the margin event m_H(x,y) ≥ γ holds with probability at least 1 − ζ.Correctness was shown whenever the margin event holds.
  • A.2 Proof of Theorem A.2: The guarantee is invariant to base-to-new shifts that alter only u(x) within ||u(x)||2 ≤ B while keeping s(x) unchanged.The lower bound depends on the shortcut component only through its norm bound.
  • B Setting Details: Experiments cover adversarial base-to-new generalization, few-shot classification, cross-dataset generalization, and domain generalization across 11 image-classification datasets.The first three settings use the diverse 11-dataset suite.
  • B Setting Details: AutoAttack combines APGD-CE, APGD-T, FAB-T, and Square, with each attack using the default one-restart setting.The suite includes untargeted, targeted, and black-box attacks.
  • C.1 Generalization to Alternative VLM Backbones: On TeCoA ViT-L/14, ADAPT raises new-class robustness from APT’s 32.47% to 38.01% and harmonic mean Hb from 48.18% to 52.87%.ADAPTM further reaches Hb = 53.94%.
  • C.1 Generalization to Alternative VLM Backbones: On FARE, ADAPT improves new-class robustness from 13.49% to 20.31% and Hb from 28.25% to 35.54%, while ADAPTM reaches Hb = 39.24%.The results show similar improvements on the FARE backbone.

C.2 Comprehensive Results of Adversarial Base-to-new Generalization · C.3 Full Results of Adversarial Few-shot Classification

Across adversarial base-to-new generalization and few-shot classification, ADAPT methods improve robustness transferability, mitigate seen-class overfitting, and achieve strong data efficiency. ADAPTM attains the best reported average performance across evaluated datasets and shot settings.

  • C.2 Comprehensive Results of Adversarial Base-to-new Generalization: At 𝜖=1/255, ADAPT demonstrates robustness transferability consistent with results under the larger 𝜖=4/255 budget.
  • C.2 Comprehensive Results of Adversarial Base-to-new Generalization: APT exhibits significant overfitting to seen classes, causing a large performance drop on new classes, whereas ADAPT effectively mitigates this issue.
  • C.2 Comprehensive Results of Adversarial Base-to-new Generalization: ADAPTM achieves the highest average new-class performance with 61.21% in accuracy and 45.89% in robustness.These results are reported for the multi-modal prompting setting.
  • C.2 Comprehensive Results of Adversarial Base-to-new Generalization: ADAPTM outperforms FAP by 4.59% in accuracy and 2.55% in robustness on new classes.
  • C.2 Comprehensive Results of Adversarial Base-to-new Generalization: The reported results attribute ADAPT’s effectiveness to disentangling robust features from pseudo-robust features.
  • C.3 Full Results of Adversarial Few-shot Classification: Across 11 datasets, ADAPTM achieves the highest average ‘H’ score for every evaluated shot setting.The evaluated settings are 1, 4, and 16 shots per class.
  • C.3 Full Results of Adversarial Few-shot Classification: ADAPTM’s average ‘H’ scores are 19.96% for 1-shot, 24.78% for 4-shot, and 30.11% for 16-shot classification.

C.4 Results of Adversarial Domain Generalization · C.5 Results of Adversarial Cross-dataset Generalization

ADAPT generalizes robustly beyond ImageNet’s source distribution, achieving the strongest reported domain-generalization averages and the highest cross-dataset harmonic mean. Its gains support disentangled robust features rather than source-shortcut overfitting.

  • C.4 Results of Adversarial Domain Generalization: ADAPT achieves 41.34% accuracy on the ImageNet source domain, with competitive robustness against baselines.
  • C.4 Results of Adversarial Domain Generalization: ADAPT attains the best average accuracy of 23.92% across ImageNet-V2, ImageNet-Sketch, ImageNet-A, and ImageNet-R.
  • C.4 Results of Adversarial Domain Generalization: ADAPT achieves the best average robustness of 7.84% across the four target domains.
  • C.4 Results of Adversarial Domain Generalization: ADAPT obtains the best average ‘H’ of 11.81%, indicating strong combined accuracy and robustness across target domains.
  • C.4 Results of Adversarial Domain Generalization: ADAPT surpasses APT on ImageNet-Sketch and ImageNet-R, gaining 0.40% and 0.85% in ‘H’, respectively.
  • C.4 Results of Adversarial Domain Generalization: These domain-generalization results indicate that ADAPT’s robustness stems from disentangled features rather than overfitting source-distribution shortcuts.
  • C.5 Results of Adversarial Cross-dataset Generalization: On adversarial cross-dataset generalization, the proposed methods consistently outperform baselines when trained on ImageNet and evaluated directly on 10 target datasets.
  • C.5 Results of Adversarial Cross-dataset Generalization: ADAPTM achieves the highest average harmonic mean of 17.43% across the cross-dataset evaluation.

C.6 Computational Cost Analysis … C.9 Decoy-Guided Adaptive Attack

The supplementary analyses show that ADAPT trades higher, class-proportional training memory for unchanged inference cost, while empirical projections support its semantic–shortcut decomposition. Robust generalization overfitting persists across training settings, and DGAA lowers but does not eliminate ADAPT’s robustness.

  • C.6 Computational Cost Analysis: ADAPT and ADAPTM require 27470M and 27530M training memory, respectively, exceeding APT’s 11454M and FAP’s 19808M.The increase primarily comes from maintaining the decoy-prompt pool and scales with the number of classes.
  • C.6 Computational Cost Analysis: ADAPT adds no inference cost over APT because decoy prompts are discarded after training.Adversarial examples are generated once per iteration with the target prompt, preserving inference efficiency despite dual-prompt training resources.
  • C.7 Empirical Validation of the Semantic–Shortcut Decomposition: On DTD, image-side projection separates each CLIP feature into a learned base-useful component z_U and residual component z_S using a projection trained on base classes.The CLIP encoders remain frozen, and both components are classified with the same hand-crafted prompt.
  • C.7 Empirical Validation of the Semantic–Shortcut Decomposition: z_U performs better on base classes, whereas z_S transfers better to new classes, supporting the proposed decomposition of real CLIP features.This result provides empirical support for distinguishing a base-useful component from a more transferable residual component.
  • C.8 Robust Generalization Overfitting under Different Training Settings: APT exhibits a robust loss gap between base and new classes across learning rates from 0.001 to 0.004 and training durations from 50 to 200 epochs.The gap’s magnitude varies, but its emergence across settings indicates that robust generalization overfitting is not tied to one configuration.
  • C.9 Decoy-Guided Adaptive Attack: DGAA selects three shortcut-confusing non-ground-truth classes by combining target-prompt competition with decoy-prompt shortcut matching, then applies target-prompt margin PGD toward each.It retains the adversarial example that most strongly reduces the target-prompt margin.
Loading 2608.17306v1…