Source-linked AI summary
Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models
Christian Schlarmann, Naman Deep Singh, Francesco Croce, Matthias Hein
TL;DR
Vision-input attacks threaten LVLM users, and CLIP’s use as a frozen encoder makes its robustness relevant to downstream models. The paper proposes unsupervised adversarial fine-tuning that preserves CLIP’s clean-image embeddings. It reports improved clean and adversarial downstream performance, with robust LVLM use requiring no downstream retraining.
Problem
Imperceptible vision attacks can steer LVLMs toward attacker-chosen outputs, creating risks of fraud and misinformation as these models are expected to see real-world use.
Source: §1 · pp. 1–2Method
FARE adversarially fine-tunes CLIP’s vision encoder without dataset-specific supervision, keeping perturbed-image features close to the original encoder’s features and preserving clean-image embeddings.
Source: §3.3 · p. 4Results
Takeaways & Limitations
Replacing an LVLM’s original CLIP encoder with FARE-CLIP yields a robust model without retraining or fine-tuning the downstream LVLM.
Source: §1 · p. 2, §5 · p. 9The supervised TeCoA baseline uses fixed ImageNet class text embeddings and cosine similarity, which can distort unseen-class or unnormalized features and harm other zero-shot tasks and LVLM performance.
Source: §3.2 · pp. 3–4
Abstract
from arXiv · showhide
Multi-modal foundation models like OpenFlamingo, LLaVA, and GPT-4 are increasingly used for various real-world tasks. Prior work has shown that these models are highly vulnerable to adversarial attacks on the vision modality. These attacks can be leveraged to spread fake information or defraud users, and thus pose a significant risk, which makes the robustness of large multi-modal foundation models a pressing problem. The CLIP model, or one of its variants, is used as a frozen vision encoder in many large vision-language models (LVLMs), e.g. LLaVA and OpenFlamingo. We propose an unsupervised adversarial fine-tuning scheme to obtain a robust CLIP vision encoder, which yields robustness on all vision down-stream tasks (LVLMs, zero-shot classification) that rely on CLIP. In particular, we show that stealth-attacks on users of LVLMs by a malicious third party providing manipulated images are no longer possible once one replaces the original CLIP model with our robust one. No retraining or fine-tuning of the down-stream LVLMs is required. The code and robust models are available at https://github.com/chs20/RobustVLM
1. Introduction
The introduction frames adversarial attacks on vision inputs as a safety risk for large vision-language models and proposes FARE, an unsupervised CLIP fine-tuning scheme. It reports that replacing CLIP with FARE-CLIP improves downstream robustness while preserving clean performance, without downstream retraining.
1. Introduction
- CLIP aligns image and text representations, supports zero-shot classification, and serves as a frozen vision encoder in models such as OpenFlamingo and LLaVA.These LVLMs connect the frozen encoder to a language model and support tasks including image captioning and visual question answering.Source: §1 · p. 1
- Imperceptible image perturbations can steer LVLMs toward attacker-chosen outputs, enabling malicious actors to spread misinformation or defraud users.The introduction cites prior demonstrations of attacks on the vision modality, including targeted attacks using imperceptible changes.Source: §1 · p. 2
- FARE fine-tunes CLIP’s vision embedding adversarially without supervision while aiming to preserve the original clean-image features, enabling replacement without downstream retraining.The stated downstream scope includes zero-shot classification and LVLM tasks.Source: §1 · p. 2
- FARE-CLIP preserves clean performance better than supervised TeCoA while improving robustness across downstream tasks, including LLaVA attacks and zero-shot classification.The introduction also reports robustness to jailbreak attacks, lower LLaVA hallucination rates, and better chain-of-thought task performance than TeCoA.Source: Figure 1 · p. 1, §1 · p. 2, Figure 2 · p. 2
2. Related Work
Related work covers adversarial robustness, attacks on large vision-language models, and unsupervised adversarial fine-tuning. The paper distinguishes FARE by applying unsupervised fine-tuning to CLIP while preserving embeddings for zero-shot downstream tasks.
Multi-modal models
- Many LVLMs use a frozen CLIP-like vision encoder and train only the interaction with a pretrained language model; evaluation focuses on OpenFlamingo and LLaVA-1.5.The two evaluated models use the same original ViT-L/14 CLIP encoder but different language models.Source: §2 · p. 2
General adversarial robustness
- Adversarial training is a prominent defense, while much prior attack research targets unimodal image or text models.The related-work section also notes research on perturbation transfer and adversarial attacks and defenses for deep metric learning.Source: §2 · pp. 2–3
Adversarial robustness of LVLMs
- LVLM adversarial work includes imperceptible targeted attacks and jailbreaks, while supervised adversarial fine-tuning of CLIP is the baseline for this paper.Some cited jailbreak attacks use large perturbation radii, unlike the imperceptible targeted attacks discussed in the section.Source: §2 · p. 3
Unsupervised adversarial fine-tuning
- Prior unsupervised adversarial fine-tuning studies target SimCLR or BYOL, whereas FARE focuses on CLIP and preserves embeddings for zero-shot downstream robustness.The cited approaches include contrastive-loss methods, self-supervised BYOL training, and two-stage SimCLR training.Source: §2 · p. 3
3. Unsupervised Adversarial Fine-Tuning for CLIP
The section defines adversarial robustness for CLIP zero-shot classification, reviews the limits of supervised TeCoA fine-tuning, and introduces FARE, which trains CLIP’s image encoder to resist perturbations while preserving its original features.
3.1. Robustness of CLIP as Zero-Shot Classifier
- CLIP zero-shot classification compares an image embedding with prompts for candidate classes and assigns the class with the highest cosine similarity; adversarial images are bounded by a perturbation radius ε.The section focuses on the ℓ∞ threat model and identifies AutoAttack as a standard attack option.Source: §3.1 · p. 3
3.2. Supervised Adversarial Fine-Tuning
- TeCoA adversarially fine-tunes CLIP on ImageNet with a cross-entropy objective, but its fixed class text embeddings can distort unseen classes and cosine loss leaves embedding magnitudes unconstrained.These issues can degrade other zero-shot tasks and LVLM performance, which uses unnormalized embeddings.Source: §3.2 · pp. 3–4
3.3. Unsupervised Adversarial Fine-Tuning of the Image Embedding
- FARE minimizes the distance between perturbed-image features and the original encoder’s clean-image features, without labels or dependence on a particular dataset or text encoder.As this distance approaches zero, the fine-tuned encoder’s clean-image embeddings approach the original embeddings, supporting use in downstream LVLMs without changing their performance.Source: §3.3 · p. 4
- Preserving CLIP image embeddings also preserves image–text cosine similarities, supporting zero-shot classification performance; the approach can extend to foundation models with an intermediate cross-modal embedding layer.The section states this preservation relationship as the motivation for Theorem 3.1.Source: §3.3 · p. 4
4. Experiments
The experiments evaluate robust CLIP encoders in zero-shot classification and LVLMs, including attacks, hallucination, and reasoning tasks. FARE generally improves robustness while retaining downstream performance, with outcomes varying by threat model and task.
4. Experiments
- The experiments use CLIP ViT-L/14 and two ImageNet fine-tuning radii; only models trained at ε = 4/255 are fully robust to targeted imperceptible LVLM attacks.FARE uses no labels, and the LVLM projection layers and language models remain fixed.Source: §4 · p. 4
4.1. Quantitative Robustness Evaluation of LVLMs
- Across OpenFlamingo and LLaVA tasks, FARE models improve on TeCoA in clean and robust performance; FARE2 stays close to original CLIP on clean data.FARE4 outperforms TeCoA2 and TeCoA4 on all LLaVA datasets in clean and most datasets in robust performance; FARE2 shows higher clean performance and better overall robustness at ε = 2/255 for OpenFlamingo.Source: Table 1 · p. 5, §4.1 · p. 5
4.2. Stealthy Targeted Attacks on LVLMs
- The targeted-attack evaluation tests six target captions on 25 sampled images using APGD with 10,000 iterations.The paper notes that 500 iterations make the attack much less successful, so it uses 10,000 to assess robustness.Source: §4.2 · p. 6
- At ε = 2/255, CLIP breaks in every targeted-attack case while robust models never do; at ε = 4/255, FARE4 and TeCoA4 are fully robust, and FARE2 breaks in 2.0% of cases.TeCoA2 breaks in 3.3% of cases at ε = 4/255.Source: §4.2 · p. 6, Table 3 · p. 7
4.3. Evaluation of Zero-Shot Classification
- Across image-classification datasets, FARE4 is most robust on average at ε = 2/255 and slightly outperforms both TeCoA models at ε = 4/255.On non-ImageNet zero-shot datasets, undefended CLIP has the best clean performance, while FARE models, especially FARE2, maintain much better clean accuracy than TeCoA. On ImageNet, TeCoA models perform best in clean and robust evaluations.Source: §4.3 · p. 6, Table 4 · p. 8
4.4. Performance on Other Tasks
- On POPE, clean CLIP has the best F1-score on every split, FARE is closest, and TeCoA has the worst average F1-score.The paper attributes TeCoA’s hallucinations to difficulty generating correct outputs even for nominal inputs.Source: §4.4 · p. 8
- On SQA-I, both FARE models outperform their respective TeCoA models by 2.4%, and FARE2 is only 1% below original CLIP accuracy.The authors conclude that FARE does not degrade the LVLM’s reasoning-task ability.Source: §4.4 · p. 9, Table 6 · p. 9
- For jailbreaking attacks, both TeCoA and FARE significantly reduce harmful outputs versus original CLIP, regardless of attack strength or prompt type.The authors call this evaluation preliminary and warn that it might overestimate robustness because jailbreaking attacks remain an active research area.Source: §4.4 · p. 9
5. Conclusion
FARE fine-tunes CLIP’s vision encoder adversarially while aiming to preserve its original embeddings, transferring robustness to downstream tasks without retraining the LVLM. The paper reports improved robustness alongside strong clean performance, while noting limits in model scope and evaluated tasks.
5. Conclusion
- FARE-CLIP improves robustness across downstream tasks while preserving clean performance, including when used in OpenFlamingo and LLaVA.The paper also reports robustness to jailbreak attacks, a lower LLaVA hallucination rate, and stronger clean and adversarial zero-shot classification than previous methods.Source: §5 · p. 9
- Replacing the original CLIP vision encoder with FARE-CLIP yields robust LVLMs without retraining the downstream models.The authors state that the method can also be applied after training LVLMs that fine-tune their vision encoder, at little extra computational cost.Source: §5 · p. 9
- The evaluation focuses on CLIP-based LVLMs and visual-input robustness, and does not examine instruction following, explainability, or perception-related tasks.The authors identify language-side defense and other LVLM types as further areas for investigation.Source: §5 · p. 9
Impact Statement
The authors frame safe deployment of widely used LVLMs as a pressing concern. Their transfer-attack results underscore the importance of robust vision encoders.
Impact Statement
- Transfer attacks in Table 2 successfully attack LVLMs that share a non-robust vision encoder, regardless of the language model or vision-language connector.The authors say this enables attacks even on closed-source LVLMs.Source: Impact Statement · p. 10
- The authors connect LVLMs’ broad deployment and strong performance across tasks with the need for safe and secure deployment.They describe their work as a step toward making LVLM deployment safer.Source: Impact Statement · p. 10
- The transfer-attack findings stress the importance of using a robust vision encoder in LVLMs.Source: Impact Statement · p. 10
A. Omitted Proof
The appendix states that preserving the ℓ2 distance between CLIP image embeddings preserves their cosine similarity, which is used in zero-shot classification. It also describes fine-tuning and evaluation settings, including class-token-only training and how to read the Figure 1 comparisons.
A. Omitted Proof
- Preserving the ℓ2 distance between original and fine-tuned embeddings preserves cosine similarity, the quantity used in zero-shot classification.The supplied proof passage introduces a theorem about the original and fine-tuned image embeddings and a text embedding, but does not state its bounds.Source: A. Omitted Proof · p. 13
B.1. General Setup
- Using only the class token in the FARE loss attained good downstream LVLM results; including all tokens required more compute and memory without improving results.Source: §B.1 · p. 13
- The main robust models were trained on ImageNet for two epochs using 10 PGD steps, with perturbation radii of 4/255 or 2/255 and step size 1/255.The reported setup also uses AdamW, a cosine-decaying learning rate with warmup, weight decay of 1e-4, and effective batch size 128.Source: §B.1 · p. 13
- Figure 1 compares CLIP, TeCoA, and FARE across zero-shot tasks on radial axes scaled from zero to each task’s maximum among the three models.The adversarial evaluations use ℓ∞ = 2/255; zero-shot classification is evaluated on CLIP, while the other evaluations use LLaVA.Source: §B.2 · p. 13
B.3. Ablation of Training Hyperparameters
The selected FARE hyperparameters use a 1e-5 learning rate and 1e-4 weight decay, favoring downstream zero-shot generalization over the ImageNet robustness gains of the larger learning rate. The appendix describes the search and reports that the final setting gives the best generalization to downstream zero-shot tasks.
B.3. Ablation of Training Hyperparameters
- The hyperparameter search used ViT-B/32 backbones to reduce the cost of selecting settings for the main paper’s ViT-L/14 encoders.The search focused on learning rate and weight decay, and evaluated clean and adversarial performance on ImageNet and average zero-shot datasets.Source: §B.3 · pp. 13–14
- +4.2% clean zero-shot performance, with similar zero-shot robustness, for LR = 1e-5 and WD = 1e-4 versus the LR = 1e-4 setup.The authors select this as the final setting; Table 8 identifies it as yielding the best generalization to downstream zero-shot tasks.Source: §B.3 · p. 14, Table 8 · p. 14
- +5% average ImageNet robustness for LR = 1e-4 models over smaller-LR models across the two perturbation radii, a gain the authors relinquish for downstream zero-shot performance.Source: §B.3 · p. 14
- Weight decay appeared to have no impact on clean performance or robustness across the tested FARE models.Source: §B.3 · p. 14
B.4. Ablation of Loss Function
This appendix tests the loss choice, TeCoA comparison, untargeted attack procedure and strength, and targeted-attack examples. It reports that the ensemble attack is stronger and faster, while FARE generalizes better than TeCoA to zero-shot classification beyond ImageNet.
B.4. Ablation of Loss Function
- The ℓ1-norm loss yields performance similar to the squared ℓ2-norm formulation, though minimizing ℓ1 can produce sparse residuals.The squared ℓ2-norm is motivated by its connection to cosine similarity and its preservation of non-normalized embeddings.Source: §B.4 · p. 14, Table 9 · p. 14
B.5. Comparison to Original TeCoA Checkpoint
- FARE generalizes better to other zero-shot classification tasks than TeCoA, with clean performance near original CLIP but slightly lower robustness.TeCoA performs better on ImageNet, where its supervised training gives it an advantage.Source: §B.5 · p. 15, Table 10 · p. 15
B.6. Untargeted Attack Details
- The untargeted attack pipeline uses half-precision APGD against multiple ground-truth captions or answers, then single-precision attacks on remaining samples.For captioning, samples below task-specific CIDEr thresholds are no longer attacked; VQA uses a similar scheme and additional targeted attacks.Source: §B.6 · pp. 15–16
B.7. Untargeted Attack: Comparison to Existing Attack
- The ensemble attack is decisively stronger and saves runtime by applying cheaper half-precision attacks before single-precision attacks on remaining samples.The comparison covers COCO and VQAv2; the attack table reports both attack strength and runtime.Source: Table 11 · p. 15, §B.7 · p. 16
B.8. Targeted Attack Details
- The six targeted captions represent possible harms including misinformation, phishing, insulting users, and eliciting an email through an LVLM agent.For targets 1–5, the attacks use 25 independently sampled COCO images each; target 6 uses 25 hand-selected stock images.Source: §B.8 · p. 16
B.9. Targeted Attack: Ablation of Attack Iterations
The appendix tests how attack iteration count affects targeted attacks against the original LLaVA-CLIP model. At ε = 2/255, 500 iterations yield lower attack success than the 10,000 iterations used in the main experiments; at ε = 4/255, 500 suffice to break the model.
B.9. Targeted Attack: Ablation of Attack Iterations
- Even the undefended LLaVA-CLIP model requires many iterations to break at ε = 2/255.The 500-iteration attack is substantially less successful at this radius.Source: §B.9 · p. 16
- The appendix compares 500 iterations with the 10,000 used in the main experiments, and reports a considerably weaker attack at ε = 2/255.This comparison cautions against judging robustness from the lower-iteration attack alone.Source: Table 12 · p. 16, §B.9 · p. 16
- At ε = 2/255, 500 iterations reduce targeted-attack success to 59.3%, versus 100% at 10,000; at ε = 4/255, 500 iterations suffice to break LLaVA-CLIP.The 500-iteration comparison uses the targeted attacks from the main experiments.Source: §B.9 · p. 16, Table 12 · p. 16
B.10. Zero-shot Evaluations
The appendix describes zero-shot classification and robustness evaluations, then reports examples of hallucination, a ScienceQA evaluation, and clean results with LLaVA-13B. In the larger model, FARE outperforms TeCoA across benchmarks and FARE2 is close to clean CLIP.
B.10. Zero-shot Evaluations
- Zero-shot classification is evaluated across 13 datasets and ImageNet; robustness uses 1,000 samples per dataset with APGD attacks at ε = 2/255 and ε = 4/255.CIFAR10, CIFAR100, and STL-10 use their original resolution; the other datasets use 224×224 images.Source: §B.10 · p. 16
C.1. Hallucination Experiments
- In illustrated POPE examples, TeCoA misses clearly visible people, while CLIP and FARE also hallucinate about a subtler knife that requires effort for humans to identify.POPE asks the model whether an object is present and compares its Yes/No answer with ground truth.Source: §C.1 · p. 17, Figure 4 · p. 17
C.3. LLaVA-13B
- All FARE models outperform TeCoA across clean LLaVA-13B benchmarks, and FARE2 has only a very small performance drop versus clean CLIP.The table reports clean vision-language task scores; adversarial robustness was not evaluated for this larger model.Source: Table 13 · p. 17, §C.3 · p. 17
C.2. Science Question Answering Evaluations
- The ScienceQA evaluation focuses on SQA-I, a 10,000-pair subset with an explanatory concept, a question, and an image as input.The paper evaluates whether robust models perform well on this reasoning task.Source: §C.2 · p. 17
C.4. Evaluation of Embedding Loss
This section compares how fine-tuning changes clean and adversarial embeddings relative to original CLIP. FARE models have the most stable embeddings, while FARE4 trades greater clean-embedding distortion than FARE2 for increased stability under attack.
C.4. Evaluation of Embedding Loss
- The clean embedding loss measures distortion from original CLIP on clean images, while the adversarial embedding loss measures distortion relative to original CLIP on adversarial inputs.These metrics are evaluated on 500 ImageNet validation images.Source: §C.4 · p. 18
- TeCoA embeddings already deviate substantially from original CLIP on clean images, consistent with its loss not preserving original embeddings and the reported downstream performance degradation.The study measures clean and adversarial embedding losses on ImageNet validation images.Source: §C.4 · p. 18
- FARE models have the most stable embeddings; FARE4 distorts clean embeddings more than FARE2 but is more stable under adversarial attacks.The comparison uses a 100-step APGD attack at ε = 4/255 on 500 ImageNet validation images.Source: §C.4 · p. 18