PaperlayerView exact revision on arXiv

Source-linked AI summary

Evolution-based Region Adversarial Prompt Learning for Robustness Enhancement in Vision-Language Models

Xiaojun Jia, Sensen Gao, Simeng Qin, Ke Ma, Xinfeng Li, Yihao Huang, Wei Dong, Yang Liu, Xiaochun Cao

arXiv:2503.12874v2cs.CV

TL;DR

VLMs remain vulnerable to adversarial examples, and prior adversarial prompt-tuning methods rely on single-gradient perturbations that can be insufficiently diverse. ER-APT combines gradient-based attacks with genetic evolution and dynamic loss weighting. The paper reports improved adversarial robustness and performance surpassing state-of-the-art adversarial prompt-tuning methods.

  • Problem

    Prior adversarial prompt-tuning methods rely on single-gradient perturbations that may miss local loss maxima, produce low-quality examples, and generalize poorly to diverse or adaptive attacks.

    Source: §1 · p. 2

  • Method

    ER-APT refines gradient-generated adversarial examples through genetic selection, mutation, and crossover, and dynamically weights clean-accuracy and robustness losses by task learning speed.

    Source: §1 · p. 2

  • Results

    Experiments across various network architectures and datasets report that ER-APT significantly improves VLM adversarial robustness and outperforms state-of-the-art adversarial prompt-tuning methods.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    ER-APT uses broader perturbation-space exploration and dynamic loss weighting to support robust prompt tuning while balancing robustness and accuracy.

    Source: §1 · p. 2, §5 · p. 9

    The theoretical analysis assumes an L-Lipschitz loss over a perturbation population in which each perturbation is within η of a maintained population member.

    Source: §3.5 · pp. 5–6

Abstract

from arXiv · show

Large pre-trained vision-language models (VLMs), such as CLIP, demonstrate impressive generalization but remain highly vulnerable to adversarial examples (AEs). Previous work has explored robust text prompts through adversarial training, achieving some improvement in both robustness and generalization. However, they primarily rely on singlegradient direction perturbations (e.g., PGD) to generate AEs, which lack diversity, resulting in limited improvement in adversarial robustness. To address these limitations, we propose an evolution-based region adversarial prompt tuning method called ER-APT, which combines gradient methods with genetic evolution to generate more diverse and challenging AEs. In each training iteration, we first generate AEs using traditional gradient-based methods. Subsequently, a genetic evolution mechanism incorporating selection, mutation, and crossover is applied to optimize the AEs, ensuring a broader and more aggressive perturbation distribution.The final evolved AEs are used for prompt tuning, achieving region-based adversarial optimization instead of conventional single-point adversarial prompt tuning. We also propose a dynamic loss weighting method to adjust prompt learning efficiency for accuracy and robustness. Experimental evaluations on various benchmark datasets demonstrate the superiority of our proposed method, outperforming stateof-the-art APT methods. The code is released at https://github.com/jiaxiaojunQAQ/ER-APT.

1. Introduction

Large VLMs are vulnerable to adversarial examples, while adversarial prompt tuning methods based on single-gradient perturbations can produce low-quality examples and overfit specific attack patterns. ER-APT combines gradient attacks with genetic evolution for region-based prompt tuning, and the paper reports that it outperforms state-of-the-art adversarial prompt tuning methods.

1. Introduction

  • Adversarial training from scratch and full-model fine-tuning are costly for large VLMs, motivating parameter-efficient prompt tuning.Prompt tuning updates learnable prompts rather than all model weights.Source: §1 · p. 2
  • Single-gradient perturbations may miss local loss maxima and produce low-quality examples, while also overfitting attack patterns and generalizing poorly to diverse or adaptive attacks.The introduction attributes this limitation to VLM loss-function nonlinearity.Source: §1 · p. 2
  • ER-APT combines gradient-based example generation with genetic selection, mutation, and crossover to refine diverse adversarial examples for region-based prompt tuning.This broadens adversarial-space exploration beyond conventional single-point perturbations.Source: §1 · p. 2
  • Dynamic loss weighting balances clean accuracy and adversarial robustness by assigning lower weight to the task whose loss decreases faster.The method treats clean accuracy and robustness improvements as two tasks.Source: §1 · p. 2
  • ER-APT outperforms state-of-the-art adversarial prompt tuning methods in experiments across various network architectures and datasets.The introduction reports significant improvements in VLM adversarial robustness.Source: §1 · pp. 2–3

2. Related Work

Prompt tuning adapts pre-trained VLMs through learnable prompts rather than updating all model weights. Adversarial prompt-tuning methods apply this approach to improve VLM robustness, using visual, text, or multimodal prompts and training strategies.

2.1. Prompt tuning for VLMs

  • Prompt tuning adapts pre-trained VLMs with learnable prompts instead of updating all model weights.The section describes textual prompts, prompts conditioned on visual features, and prompts in both vision and language branches.Source: §2.1 · p. 3

2.2. Adversarial Prompt Tuning in VLMs

  • Adversarial prompt-tuning methods improve VLM robustness through class-specific visual prompts, robust text prompts, or multimodal prompts trained with adversarial examples.The cited approaches include C-AVP [4], APT [32], AdvPT [54], and the multimodal method [61].Source: §2.2 · p. 3

3. The Proposed Method

ER-APT evolves a population of adversarial examples for prompt tuning and adapts clean-accuracy and robustness loss weights to training progress. Its theoretical analysis gives a region-based robustness bound under stated assumptions.

3.1. Preliminary Study

  • The preliminary setup represents CLIP with image and text encoders and places learnable prompts in both image and text embeddings.The learnable prompt is represented as P = {P v, P t}.Source: §3.1 · p. 3

3.2. Pipeline of The Proposed Method

  • The pipeline initializes adversarial examples with model gradients, then ranks the population by adversarial-loss fitness and retains the top N/3 individuals.The gradient update projects perturbations into the allowed [−ϵ, ϵ] set.Source: §3.2 · p. 4
  • Mutation adds random perturbations to selected individuals, while crossover blends selected parents; only selected samples from the last generation tune prompts.The mutation intensity is controlled by ϕ, and crossover uses a randomly sampled weighting coefficient.Source: §3.2 · pp. 4–5

3.3. Formulation of The Proposed Method

  • ER-APT replaces a single perturbation search with an evolved AE population, using fitness-based selection, mutation, and crossover to explore a broader attack space.The method updates prompts using losses over examples from the evolved population.Source: §3.3 · p. 5

3.4. Proposed Dynamic Loss Weighting Method

  • The loss combines clean-image cross-entropy with clean-versus-adversarial KL divergence, and its weights adapt according to each task’s learning speed.The faster a task loss decreases, the lower its assigned weight.Source: §3.4 · p. 5

3.5. Theoretical Analysis

  • Under the stated Lipschitz and population-coverage assumptions, diversity keeps η small and tightens the region-based bound γ + Lη.The bound is stronger than the standard adversarial-training bound when γ + Lη is below the maximum loss over the perturbation set.Source: §3.5 · pp. 5–6

4. Experiments

ER-APT is evaluated across few-shot, base-to-new, cross-dataset, and AutoAttack settings, with results generally exceeding FAP. Ablations and training-time comparisons assess the contributions and cost of its components.

4.1. Setups

  • The experiments use 11 image-recognition datasets and compare ER-APT with zero-shot CLIP and single-modal and multimodal adversarial prompt-learning baselines.Experiments use ViT-B/32 CLIP; robustness is evaluated with 100-step PGD, while training attacks use 2-step PGD at ϵ = 1/255.Source: §4.1 · p. 6

4.2. Adversarial few-shot learning

  • ER-APT improves natural accuracy over FAP by 1.28% on average and robustness accuracy by 1.61%, achieving the best robustness results on 10 of 11 datasets.These results use the 16-shot adversarial few-shot setting.Source: §4.2 · p. 7

4.3. Adversarial base-to-new generalization

  • On new classes, ER-APT exceeds FAP by 4.3% in New Natural Accuracy and 4.43% in New Adversarial Accuracy.In the 16-shot base-to-new setting, ER-APT also surpasses FAP on base classes by 1.24% in Base Natural Accuracy and 3.76% in Base Adversarial Accuracy.Source: §4.3 · p. 7, Table 3 · p. 7

4.4. Adversarial Cross-Dataset Evaluation

  • ER-APT surpasses the previous state-of-the-art method by 1.17% in natural accuracy and 1.31% in robust accuracy on downstream datasets.The cross-dataset evaluation tunes on ImageNet-1K and evaluates across 10 downstream datasets; Table 2 reports mean and standard deviation for natural and PGD-100 accuracy.Source: §4.4 · pp. 7–8, Table 2 · p. 7

4.5. Zero-shot Evaluation under Auto-Attack

  • ER-APT achieves an average robustness improvement of 3.29% over FAP against AutoAttack on 10 of 11 datasets.The models are tuned using PGD-generated perturbations and evaluated under AutoAttack at ϵ = 1/255.Source: §4.5 · p. 8, Table 4 · p. 8

4.6. Ablation Study

  • Removing either Evolution-based Region or Dynamic Loss Weighting reduces performance across almost all reported metrics, with the larger degradation when Evolution-based Region is removed.The ablation considers base and new class accuracy, natural accuracy, and robust accuracy.Source: §4.6 · p. 8, Figure 3 · p. 8

4.7. Computation Cost

  • ER-APT requires 1.42 to 1.68 times FAP’s training time on the four selected datasets, with the added time primarily attributed to evolution-based region generation.The authors report that ER-APT still achieves state-of-the-art robustness and generalization across tasks and datasets.Source: §4.7 · p. 8, Table 5 · p. 8

5. Conclusion

ER-APT combines gradient-based attacks with genetic evolution to diversify adversarial examples, while dynamic loss weighting balances accuracy and robustness. Experiments across benchmark datasets report stronger adversarial robustness and generalization than state-of-the-art adversarial prompt-tuning methods.

  • ER-APT combines gradient-based attacks with genetic evolution to generate diverse adversarial examples for prompt tuning.Selection, mutation, and crossover refine examples to cover a broader perturbation space.Source: §5 · p. 9
  • Dynamic loss weighting is proposed to balance robustness and accuracy during training.Source: §5 · p. 9
  • Across multiple benchmark datasets, ER-APT consistently outperforms state-of-the-art adversarial prompt-tuning methods in adversarial robustness while preserving strong generalization.Source: §5 · p. 9

A.1. Environment and Device

The appendix identifies the computing environment and points to the handcrafted text templates used for prompt initialization and static supervision.

A.1. Environment and Device

  • Experiments ran with PyTorch 1.10.1, CUDA 11.3, and Python 3.8 on an NVIDIA A100-SXM4-40GB or A100-SXM4-80GB GPU.Source: §A.1 · p. 13

A.2. Hand-crafted Prompt Templates

  • Handcrafted prompt templates for initialization across 11 image-recognition datasets are reported in Table 6.Source: §A.2 · p. 13
  • Table 6 is identified as the collection of handcrafted text templates for static text supervision across datasets.Source: Table 6 · p. 13

A.3. Comparison between APT and AdvPT

APT and AdvPT both use the CoOp framework and update learnable category-specific prompt tokens without changing model parameters. They differ in when they generate adversarial examples.

  • APT and AdvPT both update only learnable tokens in category-specific prompts, leaving model parameters unchanged.Source: §A.3 · p. 13
  • APT updates adversarial examples at each iteration as the text prompt evolves.Source: §A.3 · p. 13
  • AdvPT generates all adversarial examples with zero-shot CLIP before training, and the typical text prompt uses 16 context tokens plus a class token.Source: §A.3 · p. 13

B. Hyperparameters

The hyperparameter choices balance adversarial-example diversity and robustness against computational cost and training stability. The selected settings are N = 9, ϕ = 0.1, and T = 1.0.

B. Hyperparameters

  • N = 9 achieves OxfordPets 43.50% and Caltech101 70.53%; increasing N to 15 yields only marginal improvement while significantly increasing computational cost.The authors select N = 9 for competitive robustness and computational efficiency.Source: B. Hyperparameters · p. 13, Table 7 · p. 13
  • ϕ = 0.1 achieves Caltech101 Robust Acc 70.53% and OxfordPets Robust Acc 43.50%, while ϕ = 0.05 yields 69.97% and 41.30%, respectively.The authors describe ϕ = 0.05 as insufficiently strong and ϕ = 0.2 as potentially destabilizing, selecting 0.1 for diversity and stable training.Source: B. Hyperparameters · p. 13, Table 8 · p. 13
  • T = 1.0 yields OxfordPets Robust Acc 43.50% and DTD Robust Acc 32.03%; the authors select it for a stable balance between accuracy and robustness.T = 0.5 may destabilize training, whereas T = 2.0 weakens dynamic loss weighting.Source: B. Hyperparameters · pp. 13–14, Table 9 · p. 13

C. Training Pipeline for ER-APT

The section presents an intuitive training pipeline for ER-APT based on the detailed methodology in the main text.

C. Training Pipeline for ER-APT

D. Detailed experimental results for each dataset.

The detailed results supplement main-text summaries with per-dataset reporting across adversarial few-shot settings and base-to-new generalization.

D. Detailed experimental results for each dataset.

Related papers

Loading 2503.12874v2…