Breaking the weakest link to evade vision language models

Ilan Zini; Boussad Addad; Katarzyna Kapusta

Source-linked AI summary

Breaking the weakest link to evade vision language models

Ilan Zini, Boussad Addad, Katarzyna Kapusta

arXiv:2608.18938v1Aug 19, 2026cs.AIcs.LG

TL;DR

VLM robustness to imperceptible visual adversarial perturbations is insufficiently understood. This paper attacks only the vision encoder and finds that visually indistinguishable perturbations can substantially change model-generated descriptions.

Problem
The study addresses limited evidence on how adversarial image perturbations alter textual outputs from vision-language models, a security concern for multimodal systems.
Method
The paper develops a white-box gradient-based attack that optimizes perturbations exclusively through a VLM’s vision encoder rather than its entire multimodal architecture.
Results
Visually indistinguishable perturbations changed Granite-Vision’s output from “a military tank” to “an ambulance” at ε = 0.05.
Takeaways & Limitations
The findings show that current VLMs remain vulnerable to visual adversarial manipulation despite strong multimodal capabilities.
Takeaways & Limitations
Future work is needed to assess perturbation transferability across VLM architectures and investigate defense mechanisms.

Abstract

from arXiv · show

Vision Language Models (VLMs) have recently emerged as a critical component of multimodal AI systems, enabling joint reasoning over visual and textual inputs in real-world and safety-critical applications. Despite their growing deployment, the robustness of VLMs against adversarial threats remains insufficiently explored, particularly in the context of evasion attacks targeting multimodal alignment. In this work, we investigate the vulnerability of VLMs to adversarial perturbations applied to visual inputs and study two attack settings: untargeted attacks, where the goal is to disrupt the model's interpretation of the original image, and targeted attacks, where the adversary aims to force the model to generate a specific semantic description unrelated to the original image. To efficiently generate adversarial examples, we propose a gradient-based attack method that performs optimization exclusively on the vision encoder of the VLM rather than on the entire multimodal architecture. This design significantly reduces the computational cost and resource requirements of the attack while maintaining strong effectiveness. We evaluate our approach on several open-source VLMs, including Qwen2.5-VL, Granite-Vision, FastVLM, and Phi-3.5-Vision, and show that small, human-imperceptible perturbations can substantially alter the textual interpretation produced by the models. Our findings highlight the vulnerability of modern VLMs to adversarial manipulation and emphasize the need for improved robustness and security mechanisms in multimodal AI systems.

1 Introduction

Vision–Language Models support broad real-world applications but remain vulnerable to small, human-imperceptible visual perturbations that alter their textual outputs. The paper studies targeted and untargeted attacks and proposes optimizing only the vision encoder under a white-box threat model.

Background: Vision–Language Models jointly process images and text, supporting applications including image captioning, visual question answering, and multimodal reasoning.Their visual representations are combined with linguistic context to produce coherent textual outputs.
Security concerns: Small, human-imperceptible image perturbations can drastically alter model behavior, creating security risks in deployed multimodal systems.Manipulated outputs could spread misinformation, bias responses, or generate harmful content while appearing legitimate to users.
Attack settings: The study examines targeted attacks that force specific outputs and untargeted attacks that disrupt interpretation of the original image.Both settings investigate how perturbations to input images alter the textual outputs generated by the model.
Attack method: The proposed efficient approach optimizes only the VLM’s vision encoder instead of the entire multimodal architecture.It assumes a white-box attacker with full access to the vision encoder, a setting motivated by the public release of many competitive VLMs.

2 Background and related work

Prior work establishes evasion attacks as a key VLM threat because visual perturbations can silently change textual outputs. This work instead optimizes perturbations only through the vision encoder, reducing attack computation and memory requirements.

Evasion attacks modify test-time inputs, whereas poisoning attacks corrupt training data; for VLMs, visual evasion can alter textual outputs without visible image changes.
Textual adversarial attacks: Gradient-based textual attacks can bypass LLM safety alignment and transfer across models, highlighting aligned-model vulnerabilities relevant to multimodal systems.
Visual adversarial attacks on VLMs: Prior visual attacks show that optimized images can universally jailbreak aligned VLMs, while other work studies targeted and untargeted attacks on models including OpenFlamingo.
Our approach: The proposed attack optimizes adversarial perturbations exclusively through the vision encoder rather than backpropagating through the entire VLM.
Our approach: 1.6% to 18% of total model parameters belong to the vision encoder, decreasing to 0.9% for Qwen 72B sharing Qwen 3B’s vision encoder.This parameter reduction significantly lowers the attack’s computational and memory footprint.

3 Problem formulation and attack design

The attack framework models adversaries by their goals and knowledge, then generates targeted or untargeted image perturbations by optimizing only through each VLM’s vision encoder. Targeted attacks align adversarial and target-image embeddings, whereas untargeted attacks maximize divergence from the source embedding.

3.1 Threat model: The threat model distinguishes attackers by their goal and knowledge.This framework considers both attack objectives and the attacker’s access to model components.
Attacker’s goal:: Untargeted attacks make the generated description no longer reflect the original image’s main semantic content.The attack does not require a target image.
Attacker’s goal:: Targeted attacks keep the adversarial image visually identical to the source while inducing a description corresponding to a target image.The attacker starts from both a source image and a target image.
Attacker’s knowledge:: The white-box threat model grants full access to the VLM’s vision encoder.The paper motivates this assumption by noting that competitive systems such as Granite-Vision, Qwen-VL, Phi-Vision, and FastVLM are open source.
3.2 Design of our solution: A VLM processes image input I through visual encoder vϕ, projection module Mλ, and language model fθ conditioned on the image and prompt.The encoder extracts visual features, which are mapped into the language embedding space before textual generation.
3.2 Design of our solution: The method computes adversarial perturbations by minimizing a loss on vision-encoder image embeddings instead of backpropagating through the entire multimodal model.Optimization occurs directly on the image input while gradients are propagated only through the vision encoder.
3.2 Design of our solution: Targeted optimization minimizes embedding MSE so the adversarial image representation approaches the target representation and elicits a target-consistent description.The attack begins with source image Isource and target image Itarget.
3.2 Design of our solution: Untargeted optimization starts from Gaussian noise with σ = 10^-6 and maximizes distance between adversarial and source embeddings.The noisy image is clipped to [0, 1], and the untargeted procedure uses a negated MSE loss rather than a target image.

4 Experiment

Experiments evaluated vision-encoder-only adversarial attacks on four open-source VLMs using ImageNet images, showing strong targeted vulnerability and near-universal untargeted vulnerability. The optimization also substantially reduced VRAM usage compared with full-VLM backpropagation.

Experimental setup: Experiments used a diverse ImageNet subset and four open-source VLMs: Qwen2.5-VL-3B-Instruct, Granite-Vision-3.2-2B, FastVLM-7B, and Phi-3.5-Vision-Instruct.Targeted attacks used 1000 randomly selected source images paired with 1000 randomly selected target images, while untargeted attacks used the original images without targets.
Experimental setup: Adversarial examples used I-FGSM through the vision encoder for 50 iterations, and attack success was assessed by Granite-4.0-micro comparing one-sentence VLM descriptions.Untargeted optimization began from Gaussian-noise-perturbed originals, whereas targeted attacks had to bridge generally unrelated source-target image semantics.
Targeted attacks: Targeted success generally increased with ε, with Granite-Vision-3.2-2B exceeding 45% at larger perturbations, FastVLM reaching 26%–29%, and Phi-3.5-Vision remaining around 2%.Qwen2.5-VL-3B was more resilient at small budgets but increasingly vulnerable as ε grew, whereas model vulnerability varied substantially.
Untargeted attacks: 93%: untargeted success rate at ε = 0.05 for every model, with Granite-Vision and FastVLM near 99%–100% and Qwen2.5-VL and Phi-3.5-Vision above 97%.Increasing ε beyond 0.05 produced only marginal improvements, indicating that untargeted attacks largely saturated at low perturbation levels.
Resource footprint: 13%–16%: VRAM usage with vision-encoder-only optimization on Qwen2.5-VL and Granite-Vision, versus approximately 44%–47% for full-VLM backpropagation on an NVIDIA H100 80 GB.The restriction also substantially reduced GPU utilization and shortened attack generation time.

5 Conclusion and future work

The work presents an efficient vision-encoder-only gradient-based attack that reduces computational demands while retaining strong effectiveness across multiple VLMs. Future work should examine perturbation transferability across architectures and relevant defense mechanisms.

Conclusion: The proposed gradient-based attack operates exclusively on the vision encoder, avoiding backpropagation through the entire VLM.This formulation enables adversarial example generation with reduced computational requirements.
Conclusion: The vision-encoder-only formulation substantially reduces VRAM usage, GPU utilization, and attack generation time while maintaining strong effectiveness across multiple models.
Future work: Future work should investigate the transferability of adversarial perturbations across different VLM architectures.
Future work: Future work should also investigate relevant defense mechanisms.

Appendix

The appendix specifies the vision-encoder untargeted attack procedure and provides qualitative and computational evidence for adversarial manipulation. Targeted perturbations can remain hardly visible, while encoder-only optimization reduces resource demands and completes much faster than full-VLM optimization.

Untargeted attack: Algorithm 2 initializes an adversarial image with Gaussian noise, maximizes vision-embedding MSE from the source, and clips updates within an ε-bounded image range.The procedure iterates gradient updates for N steps with step size α, clipping pixel values to [0,1] and [I_b − ε, I_b + ε].
Targeted attack: With ε = 0.05 on Granite-Vision-3.2-2B, targeted attacks produce captions from adversarial images whose perturbations are hardly visible to users.The figure compares generated captions for source and adversarial images.
Compute efficiency: Encoder-only optimization substantially reduces both VRAM usage and GPU utilization compared with optimizing through the full Granite-Vision-3.2-2B VLM.The comparison uses an NVIDIA H100 80 GB.
Compute efficiency: Approximately 100 seconds: encoder-only optimization completes on Qwen2.5-VL-3B, whereas full-VLM optimization may exceed 20 minutes without producing a successful adversarial example.Both measurements are reported for an NVIDIA H100 80 GB.