PaperlayerView exact revision on arXiv

Source-linked AI summary

Few-Shot Adversarial Prompt Learning on Vision-Language Models

Yiwei Zhou, Xiaobo Xia, Zhiwei Lin, Bo Han, Tongliang Liu

arXiv:2403.14774v2cs.CVcs.CLcs.CRcs.LG

TL;DR

Adversarial robustness methods using vision-language models face high adaptation costs, static text supervision, and risks to natural generalization. FAP uses few-shot prompt learning with adversarially learned text supervision and a feature-balancing objective. It reports improved performance across benchmark settings and comparable zero-shot robustness with 1% training data.

  • Problem

    Prior vision-language adversarial adaptation has high data and computational costs, static text prompts with limited adversarial information, and objectives that may neglect natural generalization.

    Source: §1 · p. 2

  • Method

    FAP few-shot tunes prompts, learns adversarially correlated text supervision from adversarial examples, and balances cross-modal consistency with uni-modal differentiation.

    Source: §1 · p. 2, §3 · p. 4, §3.2 · p. 5

  • Results

    Across 11 benchmark datasets, FAP significantly outperforms state-of-the-art adversarial prompt methods in three settings, and a 16-shot ImageNet-1K subset achieves comparable zero-shot robustness using 1% training data.

    Source: §1 · p. 2

  • Takeaways & Limitations

    FAP matches state-of-the-art performance with reduced computational demand in few-shot adversarial prompt tuning on vision-language models.

    Source: §5 · p. 10

    The work does not consider full-finetuning alternatives, and a systematic theoretical analysis and proof remain elusive.

    Source: G Limitations · p. 28

Abstract

from arXiv · show

The vulnerability of deep neural networks to imperceptible adversarial perturbations has attracted widespread attention. Inspired by the success of vision-language foundation models, previous efforts achieved zero-shot adversarial robustness by aligning adversarial visual features with text supervision. However, in practice, they are still unsatisfactory due to several issues, including heavy adaptation cost, suboptimal text supervision, and uncontrolled natural generalization capacity. In this paper, to address these issues, we propose a few-shot adversarial prompt framework where adapting input sequences with limited data makes significant adversarial robustness improvement. Specifically, we achieve this by providing adversarially correlated text supervision that is end-to-end learned from adversarial examples. We also propose a novel training objective that enhances the consistency of multi-modal features while encourages differentiated uni-modal features between natural and adversarial examples. The proposed framework gives access to learn adversarial text supervision, which provides superior cross-modal adversarial alignment and matches state-of-the-art zero-shot adversarial robustness with only 1% training data. Code is available at: https://github.com/lionel-w2/FAP.

1 Introduction

The introduction frames adversarial vulnerability as a safety concern and identifies limits in prior vision-language adaptation: high data and computation costs, static text prompts, and risks to natural generalization. FAP addresses these issues through few-shot prompt tuning, learned adversarial text supervision, and an objective balancing natural and adversarial representations.

1 Introduction

  • Adversarial examples can mislead state-of-the-art DNNs with perturbations imperceptible to humans, limiting safety-critical applications including medicine and autonomous driving.Source: §1 · p. 1
  • Earlier vision-language approaches used static CLIP text supervision to align adversarial visual features and achieve zero-shot robustness in downstream tasks.Source: §1 · p. 2
  • Prior methods rely on large-scale adaptation, static prompts without adversarial hints, and objectives that may neglect natural generalization.The introduction links these limitations to heavy computation, inadequate adversarial supervision, and potential robustness failures when natural generalization is constrained.Source: §1 · p. 2
  • FAP learns adversarially correlated text supervision end-to-end from adversarial examples and uses an objective that aligns multimodal features while differentiating unimodal features.The framework is presented as few-shot prompt learning for pretrained vision-language models.Source: §1 · p. 2
  • Across 11 benchmark datasets, FAP significantly outperforms state-of-the-art adversarial prompt methods in three settings; a 16-shot ImageNet-1K subset achieves comparable zero-shot robustness using 1% training data.The settings are adversarial few-shot learning, adversarial zero-shot transfer, and adversarial base-to-new generalization.Source: §1 · p. 2

2 Preliminary

The preliminary material describes CLIP-based image–text classification and prompt learning, then reviews adversarial visual prompting and its limitations. It also previews FAP’s learnable adversarial text supervision and feature-balancing objective.

2 Preliminary

  • CLIP prompt learning replaces hand-crafted prompts with learnable text tokens optimized on downstream examples using cross-entropy over text-image logits.Source: §2 · p. 3
  • Adversarial visual prompting tunes visual prompts using adversarial examples and text-image contrastive training, while retaining the pretrained model.The cited prior method uses hand-crafted text prompts and reports downstream zero-shot adversarial robustness.Source: §2 · p. 3
  • Prior approaches face substantial data and prompt costs, static prompts without adversarial hints, and potential risks to natural generalization.The cited examples include large-scale ImageNet adaptation and prompts with 200 token-level prompts.Source: §2 · pp. 3–4
  • FAP tunes prompt tokens and deep image-to-text projections while freezing the rest of the model, promoting cross-modal consistency and uni-modal differences between natural and adversarial examples.Its natural and adversarial image-encoder forward processes share parameters.Source: Figure 1 · p. 4

3 Method

FAP adapts lightweight visual and text prompts on pre-trained CLIP with few-shot data, combining cross-modal consistency for natural and adversarial examples with a visual-space distinction between them.

3.1 Learnable Text Supervision for Adversarial Examples

  • FAP introduces text prompt tokens learned end-to-end from adversarial examples and links them to visual prompts through a linear projection.The visual prompt is incorporated into the image embedding, while the text prompt is inserted into word embeddings.Source: §3.1 · p. 5

3.2 Balancing Natural and Adversarial Generalization in Few-Shot Adversarial Prompt

  • The surrogate contrastive objective separates natural classification error from adversarial boundary error, balancing natural generalization with cross-modal adversarial consistency.The authors state that this balance alleviates potential robustness failures caused by discrepancies in natural generalization.Source: §3.2 · p. 5

3.3 Uni-Modal Adversarial-Aware Mechanism

  • The adversarial-aware visual mechanism minimizes similarity between natural and adversarial features while text supervision maintains consistent text-image outputs across those distinct visual features.The design preserves distributional differences in the uni-modal visual space while enforcing consistency in the text-image joint space.Source: §3.3 · pp. 5–6

3.4 Overall Learning Objective

  • Inner maximization uses the adversarial term as a surrogate loss to generate adversarial examples, while prompt tokens are adapted on few-shot set S under objective (7).The general PGD formulation with cross-entropy loss is also applicable.Source: §3.4 · p. 6

3.5 Intuition behind Objective Design

  • Without the adversarial-aware term, some adversarial embeddings resemble natural examples; with it, Figure 2b separates them and counters the minimal-perturbation shortcut.The visualization uses natural-image, adversarial-image, and text embeddings from same-class Caltech101 images.Source: §3.5 · p. 6, Figure 2 · p. 6

4 Experiments

Experiments assess few-shot adversarial prompt learning across diverse datasets, comparing prompt baselines and testing generalization, benchmark matching, and design choices. The results highlight strong performance with limited adaptation data, alongside trade-offs and dataset-dependent instability in deep prompt interaction.

4.1 Setups

  • Evaluation spans 11 datasets, compares hand-crafted and learnable text-prompt baselines, and uses ViT-B/32 CLIP with 2-step training and 100-step PGD evaluation.The datasets cover multiple vision tasks; experiments average results over three random seeds.Source: §4.1 · p. 7

4.2 Main Results

  • Static text prompts struggle in few-shot adversarial learning, while end-to-end learned adversarial text prompts gain robustness and FAP achieves superior average natural and adversarial accuracy.Figure 3 plots natural accuracy as dashed lines and robust accuracy as solid lines across shot numbers and 11 datasets.Source: §4.2 · p. 7, Figure 3 · p. 8
  • Across 11 datasets, FAP surpasses counterparts on robust metrics and natural generalization and has lower standard deviation, including when few-shot samples have a natural generalization gap.Table 1 reports average base/new natural and adversarial accuracy; the prose reports improved stability and performance despite a natural generalization gap.Source: §4.2 · p. 7, Table 1 · p. 8
  • With 1.25% of ImageNet-1K examples, FAP aligns with benchmark performance and accelerates training by over 97%; 16-to-32-shot training and 12-layer prompts exceed prior results.The benchmark uses average natural and robust accuracy across downstream datasets; running time is measured on one NVIDIA RTX A40 GPU.Source: §4.2 · p. 8, Table 2 · p. 9

4.3 More Analysis

  • As λ increases, natural accuracy declines while adversarial robustness improves, but increasing λ beyond 2.5 brings no further robustness gains.Table 3 reports adversarial base-to-new generalization performance across λ values.Source: Table 3 · p. 9, §4.3 · p. 9
  • FAP’s objective combines natural and adversarial terms, using KL divergence for cross-modal consistency and cosine similarity for uni-modal diversity, and yields the best robustness among tested loss designs.The ablation is conducted in the adversarial base-to-new setting.Source: §4.3 · p. 9, Table 5 · p. 10
  • Increasing prompt depth or length improves performance, with depth producing larger gains than length and larger improvements in robustness than natural accuracy.Table 4 evaluates 16-shot adversarial prompt learning on StanfordCars.Source: Table 4 · p. 9, §4.3 · p. 9
  • Text-to-image deep prompt interaction can destabilize generalization, with severity and λ varying by dataset; the proposed minimal refinement switches interaction to image-to-text.Figure 4 analyzes DTD, OxfordPets, and Caltech101; the authors report particularly unacceptable degradation on some fine-grained datasets.Source: §4.3 · pp. 9–10, Figure 4 · p. 10

5 Conclusion

The conclusion presents few-shot adversarial prompt tuning for vision-language models, combining learnable adversarial text supervision with a new training objective. The appendices outline the learning and testing pipelines, related work, and implementation environment.

5 Conclusion

  • The paper introduces learnable adversarial text supervision and a new objective for few-shot adversarial prompt learning, reporting state-of-the-art performance with reduced computational demand.The stated focus is adversarial prompt tuning on vision-language models for zero-shot downstream adversarial robustness.Source: §5 · p. 10

A Pipelines of Adversarial Prompt Learning and Testing

  • The learning pipeline takes a few-shot dataset, pretrained CLIP, prompt vectors, text descriptions, and λ, then updates prompt vectors using the final loss.The prompts are used to form clean representations and generate adversarial examples during training.Source: A Pipelines of Adversarial Prompt… · p. 17
  • The testing pipeline compares clean and adversarial image-text representations and reports natural and adversarial accuracy.The algorithm counts correct predictions for each before calculating the two accuracies.Source: A Pipelines of Adversarial Prompt… · p. 17

B Related Work

  • Related work covers adversarial training and image-text alignment [11] [60], few/zero-shot classification [61], and VLM prompt learning such as CoOp [28] and MaPLe [30].The review also describes VLM pretraining approaches and the susceptibility of VLMs to adversarial perturbations.Source: B Related Work · p. 18

C Additional Implementation Details

C.1 Additional Implementation Details for Baselines

The section specifies baseline prompt implementations and hand-crafted templates, then reports cross-dataset transfer results and a few-shot failure risk linked to weak natural generalization.

C.1 Additional Implementation Details for Baselines

  • Baselines retain their source-paper prompt designs, while their losses are replaced with TeCoA for adversarial training and evaluation.The visual, text, multimodal, and vision-language baselines use their respective prompt designs.Source: §C.1 · p. 19

C.2 Hand-crafted Prompt Templates

  • The hand-crafted templates used for initialization are reported for 11 image-recognition datasets, with an additional pixel-level pad prompt noted.The section also points to Figure 6 for potential failure cases and solutions from 8-shot DTD experiments.Source: §C.2 · p. 19, Table 7 · p. 19, Figure 6 · p. 20

D.1 Detailed Results on Adversarial Cross-Dataset Evaluation

  • With 16-shot ImageNet-1K adaptation and no downstream tuning, the method outperforms counterparts on 8/11 datasets and the baseline on all 11.Robust adaptation also reduces zero-shot natural accuracy on downstream datasets relative to original CLIP.Source: §D.1 · p. 20, Table 8 · p. 20

D.2 Natural Generalization Gap Hinders Robust Adapting

  • Under TeCoA loss, some few-shot trials quickly reach a stable loss, while normal trials continue to decrease during training.The authors associate this failure risk with insufficient natural generalization on the sampled dataset.Source: §D.2 · p. 20
  • Natural-example tuning restores robust fine-tuning, and a dual-term objective serves as a surrogate for that two-stage approach.The authors report that the dual-term loss avoids potential failures from the natural-generalization barrier in end-to-end training.Source: §D.2 · pp. 20–21

D.3 Incremental Changes from AdvMaPLe

The section evaluates two changes to AdvMaPLe and applies the proposed objective to AdvVLP, reporting gains in adversarial robustness and few-shot transfer.

D.3 Incremental Changes from AdvMaPLe

  • The two AdvMaPLe changes are a more stable projection direction and an objective that accounts for natural generalization and differentiates learning across modalities.The objective uses CLIP's structure to design a differentiated robust-learning strategy between modalities.Source: §D.3 · p. 21
  • The objective alone enhances performance, while changing projection direction alone has subtle effects; combining changes adds gains by repairing instability on some datasets.The combined method's final-row results are identified as a significant improvement over AdvMaPLe.Source: §D.3 · p. 21, Table 9 · p. 21

D.4 Case Study: Improving AdvVLP with Our Learning Objective

  • With AdvVLP, the proposed objective improves zero-shot adversarial robustness on 10 of 11 datasets relative to TeCoA loss.The comparison adapts AdvVLP with either TeCoA loss or Lfinal.Source: §D.4 · p. 21
  • +11.11% base natural accuracy, +6.67% base PGD-100 accuracy, +2.03% new natural accuracy, and +0.87% new PGD-100 accuracy with the proposed objective.The authors also report superior stability across trials.Source: §D.4 · p. 21

D.5 Results on Different CLIP Architectures

The section reports cross-dataset transfer with ViT-B/16 and finds that this backbone does not improve adversarial robustness over ViT-B/32.

D.5 Results on Different CLIP Architectures

  • ViT-B/16 does not improve adversarial robustness; the authors consider ViT-B/32's loss of detailed information acceptable for adversarial prompt learning.They attribute this to the method's emphasis on cross-modal feature alignment and understanding rather than detailed features.Source: §D.5 · pp. 21–22
  • ViT-B/16 uses finer image patches, generally improves natural recognition, and incurs higher computational cost from longer input sequences.The authors caution that more complex models do not necessarily perform better on adversarial-robustness tasks [85].Source: §D.5 · p. 21
  • The transfer table reports natural and zero-shot PGD-100 accuracy on ImageNet-1K and 10 downstream target datasets.Source: Table 11 · p. 22

D.6 Zero-shot Adversarial Robustness under Different Perturbation Bounds

The section varies training and testing perturbation bounds together and reports that stronger attacks reduce zero-shot robust performance.

D.6 Zero-shot Adversarial Robustness under Different Perturbation Bounds

  • Training uses perturbation bounds ϵ = {1/255, 2/255, 4/255}, and each model is tested with the same bound used during training.Source: §D.6 · p. 22
  • Larger perturbation bounds produce stronger attacks and lower zero-shot robust performance; robustness to large attack strength requires more training data.Source: §D.6 · p. 22, Figure 8 · p. 23
  • Figure 8 compares zero-shot adversarial robustness across different perturbation bounds.Source: Figure 8 · p. 23

D.7 Zero-shot Adversarial Evaluation under Auto-Attack

Under Auto-Attack, the authors report that their model remains more robust than AdvVP, AdvVLP, and AdvMaPLe. Adapting AdvVLP with their learning objective further improves performance across all three perturbation bounds.

D.7 Zero-shot Adversarial Evaluation under Auto-Attack

  • The model retains better Auto-Attack robustness than AdvVP, AdvVLP, and AdvMaPLe despite performance degradation under the stronger attack.Table 12 evaluates zero-shot robustness on downstream datasets at perturbation bounds 1/255, 2/255, and 4/255.Source: §D.7 · p. 23, Table 12 · p. 23
  • Adapting AdvVLP with the proposed learning objective yields further performance gains at all three perturbation bounds.Source: §D.7 · p. 23, Table 12 · p. 23
  • Auto-Attack uses APGD-CE and APGD-DLR, with a fractional attack generator that automatically adjusts step size α.Source: §D.7 · p. 23

D.8 Discussions on Training-time Attack Generation

The training-time attack-generation results indicate that including Lcos in attack generation is redundant, so the authors omit it for simplicity. Separately, few-shot results show that natural and robust accuracy become less variable as the number of examples increases.

D.8 Discussions on Training-time Attack Generation

  • The authors report that adding Lcos during attack generation would focus gradients on natural–adversarial differences and produce stronger perturbations.Source: §D.8 · p. 23
  • The two training-time attack-generation methods produce quite similar results, supporting the omission of Lcos from attack generation for simplicity.The authors explain that Lcos is already incorporated in the defense objective, making its use in attack generation somewhat redundant.Source: §D.8 · pp. 23–24, Table 13 · p. 24

D.9 Detailed Results for Adversarial Few-shot Learning

  • As the shot number increases, standard deviations for both natural and robust accuracy decrease, indicating greater stability in learned adversarial robustness.At smaller shot counts, both standard deviations are relatively high, and robust representation learning depends on example quality.Source: §D.9 · p. 24

D.10 Detailed Results for Adversarial Base-to-New Generalization

Detailed base-to-new results favor the method on most datasets, including fine-grained recognition settings where the authors report more stable results. In uni-modal comparisons, neither vision nor text prompts gain robustness-related hints as examples increase, while text prompts retain preferable natural performance.

D.10 Detailed Results for Adversarial Base-to-New Generalization

  • The method performs favorably on most base-to-new datasets, with more stable results across trials on fine-grained recognition datasets such as DTD, Flowers102, OxfordPets, and FGVCAircraft.The authors report that their objective avoids potential failures associated with natural-generalization barriers in robustness learning in these settings.Source: §D.10 · p. 24, Table 16 · p. 27

D.11 Comparison between Adversarial Text and Vision Prompt

  • As available examples increase, AdvVP and AdvTP fail to acquire more robustness-correlated hints, while AdvTP maintains preferable natural performance with adversarial examples alone.The authors attribute this natural-performance behavior to the text prompt’s ability to capture semantic information.Source: §D.11 · p. 24, Figure 9 · p. 28

E Impact Statement

  • The authors identify potential misuse in security applications while suggesting minimal-adaptation robustness may suit mobile and embodied systems.Source: E Impact Statement · p. 24

G Limitations

The authors identify two limitations: they evaluate prompt learning rather than other parameter-based adaptation methods, and they lack a systematic theoretical analysis or proof of their empirical findings.

G Limitations

  • The work does not consider parameter-based adaptation methods such as full-finetuning, so its comparisons are limited to prompt-based adaptation.Source: G Limitations · p. 28
  • The authors report empirical evidence but leave a systematic theoretical analysis and proof for future work.Source: G Limitations · p. 28, §2 · p. 30

Related papers

Loading 2403.14774v2…