Source-linked AI summary
Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness
Sibo Wang, Jie Zhang, Zheng Yuan, Shiguang Shan
TL;DR
Large-scale pre-trained vision-language models can generalize zero-shot but remain vulnerable to adversarial examples, and adversarial fine-tuning may erode that generalization. PMG-AFT guides fine-tuning with the original pre-trained model, improving robust and clean accuracy across zero-shot datasets.
Problem
Pre-trained vision-language models are vulnerable to adversarial examples, while adversarial fine-tuning can overfit its target dataset and compromise pre-training generalization.
Source: §1 · p. 1Method
PMG-AFT adds an auxiliary branch that minimizes the distance between target and pre-trained model outputs for adversarial examples, alongside constraints from the pre-trained model and clean examples.
Source: §1 · p. 2Results
Across 15 zero-shot datasets, PMG-AFT improves average robust accuracy by up to 4.99% over state-of-the-art methods and average clean-sample accuracy by 8.72%.
Source: §1 · p. 2Takeaways & Limitations
PMG-AFT achieves a better trade-off between robust and clean accuracy and can combine with adversarial-sample generation algorithms.
Source: §1 · p. 2, §5 · p. 8The added branch increases training time by 298 seconds per epoch compared with FT-TeCoA.
Source: §4.2 · p. 7
Abstract
from arXiv · showhide
Large-scale pre-trained vision-language models like CLIP have demonstrated impressive performance across various tasks, and exhibit remarkable zero-shot generalization capability, while they are also vulnerable to imperceptible adversarial examples. Existing works typically employ adversarial training (fine-tuning) as a defense method against adversarial examples. However, direct application to the CLIP model may result in overfitting, compromising the model's capacity for generalization. In this paper, we propose Pre-trained Model Guided Adversarial Fine-Tuning (PMG-AFT) method, which leverages supervision from the original pre-trained model by carefully designing an auxiliary branch, to enhance the model's zero-shot adversarial robustness. Specifically, PMG-AFT minimizes the distance between the features of adversarial examples in the target model and those in the pre-trained model, aiming to preserve the generalization features already captured by the pre-trained model. Extensive Experiments on 15 zero-shot datasets demonstrate that PMG-AFT significantly outperforms the state-of-the-art method, improving the top-1 robust accuracy by an average of 4.99%. Furthermore, our approach consistently improves clean accuracy by an average of 8.72%. Our code is available at https://github.com/serendipity1122/Pre-trained-Model-Guided-Fine-Tuning-for-Zero-Shot-Adversarial-Robustness.
1. Introduction
Zero-shot vision-language models are vulnerable to adversarial examples, while adversarial fine-tuning can compromise their pre-trained generalization. PMG-AFT uses pre-trained-model guidance to improve zero-shot adversarial robustness and clean accuracy.
1. Introduction
- Large-scale vision-language models offer zero-shot generalization, yet imperceptible adversarial perturbations can induce failures, raising robustness concerns for security-related deployments.CLIP classifies an image by comparing its embedding with embeddings of candidate class labels.Source: §1 · p. 1
- Direct adversarial fine-tuning can overfit its dataset, causing the model to lose generalization capabilities acquired during pre-training.Fine-tuning is presented as a less resource-intensive alternative to adversarial training from scratch for large-scale models.Source: §1 · p. 2
- FT-TeCoA improves zero-shot robust accuracy across unseen datasets but substantially decreases clean accuracy; Figure 1 compares it with CLIP, FT-Standard, and PMG-AFT.Parameter-space methods may alleviate overfitting but do not optimize zero-shot robust accuracy as their goal.Source: Figure 1 · p. 1, §1 · p. 2
- PMG-AFT improves average robust accuracy by up to 4.99% and average clean-sample accuracy by 8.72% across 15 zero-shot datasets.The paper reports that it outperforms state-of-the-art methods.Source: §1 · p. 2
- PMG-AFT adds an auxiliary generalization branch that minimizes the distance between target-model and pre-trained-model outputs on adversarial examples.The objective also includes constraints from the pre-trained model and clean examples to encourage retention of pre-training generalization information.Source: §1 · p. 2
2. Related Work
Related work spans pre-trained vision-language models, adversarial defenses, and fine-tuning overfit. This paper focuses on using original-model features during adversarial fine-tuning to improve zero-shot robustness.
Pre-trained Vision-language Model
- Pre-trained vision-language models such as CLIP [36], SWAG [41], ALBEF [26], and BLIP [27] use vision-language pretraining to support downstream adaptability through zero-shot learning.The paper distinguishes its focus on adversarial robustness from earlier work aimed at standard benchmark performance.Source: §2 · p. 2
Adversarial Robustness
Finetuning and Catastrophic Overfitting
- Fine-tuning can make a target model deviate from its pre-trained version and overfit small datasets; parameter-space constraints differ from this paper’s feature-space approach.Adversarial overfitting can involve declines in both robust and clean accuracy.Source: §2 · p. 3
3. Methodology
The methodology defines zero-shot adversarial robustness and identifies overfitting as a risk of fine-tuning CLIP. PMG-AFT combines adversarial classification with supervision from the original model to retain generalization information.
3.1. Preliminaries and Problem Setup
- The setup uses CLIP image and text encoders for classification, updates only the image encoder during fine-tuning, and evaluates zero-shot robust accuracy across datasets beyond the training target.The paper introduces image classification as its example task and notes the method can extend to other tasks.Source: §3.1 · pp. 3–4
3.2. Adversarial Fine-Tuning is Prone to Overfitting
- Clean and adversarial fine-tuning can overfit: fine-tuning improves accuracy on TinyImageNet but may reduce accuracy on other zero-shot datasets, with adversarial fine-tuning deviating farther from original CLIP.The paper attributes the resulting decline in generalization capability to this overfitting.Source: §3.2 · p. 4
3.3. PMG-AFT
- PMG-AFT uses frozen pre-trained text embeddings to generate adversarial examples, then separates target-model robustness learning from preserving the original model’s generalization information.The method alternates adversarial-example generation with parameter updates and trains only the target image encoder.Source: §3.3 · p. 4, Figure 3 · p. 5
- The robustness branch minimizes cross-entropy on adversarial examples, encouraging the target image encoder to classify them correctly.The branch encourages adversarial visual features to resemble the text representation of the true class.Source: §3.3 · p. 5
- The generalization branch compares target-model and original pre-trained CLIP outputs on adversarial examples using KL divergence, aiming to mitigate overfitting.The original model is fixed, and its predictions provide supervision for the target model.Source: §3.3 · p. 5
- A label-independent regularization loss encourages adversarial and clean outputs in the target model to remain similar.The total objective combines robustness, generalization, and clean regularization losses with hyperparameters α and β.Source: §3.3 · p. 6
4. Experiments
The experiments fine-tune CLIP on TinyImageNet and evaluate it on 16 datasets, with evaluation outside TinyImageNet conducted zero-shot. They compare PMG-AFT with existing methods across robustness, clean accuracy, attack strength, trade-offs, and ablations.
4.1. Experimental Setup
- Fine-tuning uses the ViT-B/32 CLIP backbone for 10 epochs, with PGD-2 and PGD-10 training and evaluation at ε = 1/255, 2/255, and 4/255.The evaluation covers TinyImageNet and 15 additional datasets; the latter are evaluated zero-shot.Source: §4.1 · p. 6
4.2. Main Result
- PMG-AFT improves average robust accuracy by 4.99% over FT-TeCoA and achieves clean accuracy above FT-TeCoA and comparable to CLIP.The evaluation uses PGD-10 at ε = 1/255; TinyImageNet is the exception to the robust-accuracy improvement and is not a strict zero-shot test.Source: §4.2 · p. 6, Table 1 · p. 7
- On CIFAR100, PMG-AFT reaches 26.06% average robust accuracy, 4.71% above FT-TeCoA, with an additional 298 seconds per training epoch.The added auxiliary branch increases computational overhead; the authors report better robustness and clean accuracy than FT-TeCoA.Source: §4.2 · p. 7
4.3. Performance against AutoAttack
- Under AutoAttack, CLIP's average robust accuracy drops from 17.38% under PGD-10 to 1.74%, while PMG-AFT remains above CLIP and FT-TeCoA.The evaluation uses AutoAttack at ε = 1/255.Source: §4.3 · p. 7, Table 3 · p. 8
4.4. Performance against Different Attack Strength
- As perturbation bounds rise from ε = 1/255 to 4/255, robust accuracy declines for PMG-AFT and FT-TeCoA, but clean accuracy is almost unaffected.Training and testing use the same perturbation bound; PMG-AFT consistently outperforms FT-TeCoA across attack intensities.Source: Figure 4 · p. 7, §4.4 · p. 7
4.5. Trade-off between Robust and Clean Accuracy
- PMG-AFT achieves the most optimal clean–robust accuracy balance among the compared methods, outperforming all others on both averages.Figure 5 plots average clean accuracy on the x-axis and average robust accuracy on the y-axis for 16 datasets.Source: §4.5 · pp. 7–8, Figure 5 · p. 8
4.6. Ablation Study
- Adding the generalization-information branch improves average robust and clean accuracy over FT-TeCoA by 1.48% and 11.29%, respectively.Adding regularization raises robust generalization from 28.44% to 32.04%; output-layer KL divergence gives the best robust and clean accuracy among tested layer and distance choices.Source: §4.6 · p. 8, Table 4 · p. 8, Table 5 · p. 8
5. Conclusion
PMG-AFT uses constraints from the pre-trained model and clean examples to help the target model learn generalized features. Experiments across multiple zero-shot datasets report strong adversarial robustness and a better trade-off between robust and clean accuracy.
5. Conclusion
- PMG-AFT constrains the target model using the pre-trained model and clean examples to learn generalized features.The method modifies the outer minimization problem in adversarial training and can combine with adversarial sample-generation algorithms.Source: §5 · p. 8
- Experiments across multiple zero-shot datasets demonstrate strong adversarial robustness for PMG-AFT.Source: §5 · p. 8
- PMG-AFT achieves a better trade-off between robust and clean accuracy.Source: §5 · p. 8