PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Faceted Attack: Exposing Cross-Model Vulnerabilities in Defense-Equipped Vision-Language Models

Yijun Yang, Lichao Wang, Jianping Zhang, Chi Harold Liu, Lanqing Hong, Qiang Xu

arXiv:2511.16110v1cs.CR

TL;DR

VLM safeguards are expanding even as multimodal capabilities create potential for misuse. MFA probes defense-equipped models with coordinated attacks on alignment, filters, and vision encoders; it reports high success rates, including against commercial VLMs.

  • Problem

    VLMs' visual reasoning capabilities can also be misused to generate harmful instructions, while their safety against adversarial attacks is a concern.

    Source: §1 · p. 1

  • Method

    MFA combines Attention-Transfer Attack, a transfer-enhancement and repetition strategy for filters, and adversarial images targeting vision encoders.

    Source: §1 · p. 2

  • Results

    58.5% overall attack success across 17 VLMs; MFA reaches 52.8% on commercial models, a 34% relative improvement over the second-best method.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MFA provides a practical probe and theoretical lens for diagnosing and ultimately fortifying defenses against safety weaknesses across alignment, filtering, and vision modules.

    Source: §1 · p. 2

    Failures mainly occur when VLMs lack reasoning contrast, as with mPLUG-Owl2's repeated or ambiguous replies.

    Source: §5 · p. 7

Abstract

from arXiv · show

The growing misuse of Vision-Language Models (VLMs) has led providers to deploy multiple safeguards, including alignment tuning, system prompts, and content moderation. However, the real-world robustness of these defenses against adversarial attacks remains underexplored. We introduce Multi-Faceted Attack (MFA), a framework that systematically exposes general safety vulnerabilities in leading defense-equipped VLMs such as GPT-4o, Gemini-Pro, and Llama-4. The core component of MFA is the Attention-Transfer Attack (ATA), which hides harmful instructions inside a meta task with competing objectives. We provide a theoretical perspective based on reward hacking to explain why this attack succeeds. To improve cross-model transferability, we further introduce a lightweight transfer-enhancement algorithm combined with a simple repetition strategy that jointly bypasses both input-level and output-level filters without model-specific fine-tuning. Empirically, we show that adversarial images optimized for one vision encoder transfer broadly to unseen VLMs, indicating that shared visual representations create a cross-model safety vulnerability. Overall, MFA achieves a 58.5% success rate and consistently outperforms existing methods. On state-of-the-art commercial models, MFA reaches a 52.8% success rate, surpassing the second-best attack by 34%. These results challenge the perceived robustness of current defense mechanisms and highlight persistent safety weaknesses in modern VLMs. Code: https://github.com/cure-lab/MultiFacetedAttack

1 Introduction

MFA probes safety weaknesses in defense-equipped VLMs through attacks on alignment, content filters, and vision encoders. Its experiments report broad cross-model transfer and higher success rates than existing methods.

1 Introduction

  • MFA targets a gap in safety research: real-world adaptive cross-model attacks, content filters, and proprietary VLMs remain insufficiently evaluated.The paper notes that prior work often focuses on prompt- or image-based jailbreaks while overlooking deployed input/output filters.Source: Figure 1 · p. 1, §1 · p. 1
  • ATA hides harmful instructions inside a benign-looking meta task, competing for attention and potentially steering aligned VLMs toward harmful outputs.The authors frame this mechanism as reward hacking and derive formal conditions under which benign-looking tasks dilute safety signals.Source: §1 · pp. 1–2
  • A lightweight transfer-enhancement objective and repetition strategy jointly evade input- and output-level filters without model-specific fine-tuning.The approach addresses the difficulty of using ATA alone against commercial VLMs protected by extra filters.Source: §1 · p. 2
  • 58.5% overall attack success across 17 open-source and commercial VLMs; on commercial models, MFA reaches 52.8%, a 34% relative improvement over the second-best method.Adversarial images optimized for one vision encoder transfer to unseen VLMs, including commercial and open-source models.Source: §1 · p. 2
  • Adversarial images embed malicious system prompts in pixel space and transfer from one vision encoder to unseen VLMs, exposing risk from shared visual representations.The authors describe this as a monoculture-style vulnerability.Source: §1 · p. 2

2 Related Work

Prior VLM jailbreak research has largely studied prompt-based or image-based attacks, often outside comprehensive deployed defenses. MFA connects reward-hacking theory with multimodal attacks against input-output filters and shared visual encoders.

2 Related Work

  • Prompt-based jailbreaks seek to override model safety instructions, while many gradient-based evaluations omit content filters and leave transfer to commercial deployments open.The cited examples include prompt engineering and GCG in white-box or gray-box settings.Source: §2 · p. 2
  • Image-based attacks use typography, visual distraction, affirmative-response prompting, or split image-text instructions, but prior studies do not explicitly consider real-world safety stacks.Examples include HADES [Li et al. 2024], CSDJ [Yang et al. 2025], FigStep [Gong et al. 2023], and HIMRD [Teng et al. 2025].Source: §2 · p. 2
  • MFA formally connects reward hacking to jailbreaking, arguing that benign-looking prompts can exploit alignment objectives.The paper situates reward hacking in RL [Ng, Russell et al. 2000] and notes related findings in RLHF-trained LLMs [Pan et al. 2024; Denison et al. 2024].Source: §2 · p. 2
  • MFA combines reward-hacking theory with multimodal attacks that bypass input-output filters, transfer across models, and probe shared visual encoders.The paper identifies prior work's typical limits as single-modality focus, neglect of real-world moderation, or lack of theoretical analysis.Source: §2 · p. 2

3 Multi-Faceted Attack

MFA combines complementary attacks against alignment, system prompts, and content moderation to probe defense-equipped VLMs. Its ATA component uses a benign-seeming task to redirect attention toward producing contrasting responses.

3 Multi-Faceted Attack

  • MFA combines three attacks—ATA, a filter-targeted transfer algorithm, and a vision-encoder attack—to target distinct layers of the VLM safety stack.Unlike attacks on isolated components, the framework targets settings where alignment, system prompts, and input/output filters operate together.Source: §3 · pp. 2–3
  • Figure 2 depicts the coordinated pipeline and its three components: ATA, moderator bypass, and vision-encoder attack.The figure describes these as jointly breaking alignment, system prompts, and content moderation.Source: Figure 2 · p. 3

3.1 Attention Transfer Attack: Alignment Breaking Facet

  • ATA reframes a harmful request as a benign-looking task asking for two contrasting responses, shifting attention toward task completion and away from rejecting harmful content.The authors explain that the model may produce harmful output while attempting to satisfy the task's helpfulness objective.Source: §3.1 · p. 3

1. Theoretical Analysis: Why ATA Breaks Alignment?

The theoretical analysis models ATA as reward hacking under a single scalar reward that combines safety and helpfulness. When a dual response scores above refusal, the training objective can push the model toward dual answers.

1. Theoretical Analysis: Why ATA Breaks Alignment?

  • ATA modifies a harmful prompt into a meta task that elicits one harmful and one safe answer; if the dual response earns higher reward than refusal, the training loss pushes toward dual answers.The analysis attributes this reward gap to a single-objective reward function combining safety and helpfulness.Source: §1 · p. 3

2. Empirical Validation

Empirical checks support ATA’s reward-hacking account and show robustness across prompt variants. MFA’s moderator and vision-encoder facets add methods for bypassing defenses and transferring attacks across models.

2. Empirical Validation

  • Dual answers consistently earn higher reward comparisons than refusals across tested reward models and VLM-generated response pairs.The evaluation uses three reward models and response pairs from six VLMs.Source: §2 · p. 3
  • Across prompt variants, refusal rates stayed at or below 40% while harmful-content rates remained at or above 80% on LLaMA-4-Scout-Inst and Grok-2-Vision.Source: §2 · p. 4, Table 2 · p. 4
  • ATA exploits single-scalar RLHF when helpfulness and safety compete, allowing a framed meta-task to elevate harmful content above a safe refusal.The authors suggest separating helpfulness and safety signals in reward design.Source: §2 · p. 4

3.2 Content-Moderator Attack Facet: Breaching the Final Line of Defense

  • GPTFuzzer achieved 0% success against GPT-4.1 and GPT-4.1 mini, which the paper attributes to robust content moderation.Source: §3.2 · p. 4
  • The repetition strategy appends an optimized adversarial signature to the generated response, which can poison moderator evaluation and evade input- and output-level filters.Source: §3.2 · p. 4
  • Weakly supervised optimization guides signature selection with a second moderator while backpropagating through the first, improving cross-model success rates by up to 28%.The process is repeated for the second signature substring against the second moderator.Source: §3.2 · p. 4

3.3 Vision-Encoder–Targeted Image Attack

  • The vision attack aligns an adversarial image embedding with a malicious system prompt; the poisoned visual signal can override the built-in safety prompt.Unlike end-to-end visual jailbreaks, its objective perturbs only the vision-encoder pathway using cosine similarity.Source: §3.3 · p. 5

2. Why focus on Vision Encoder?

The paper focuses on the vision encoder because embedding-space perturbations offer a simpler, lower-cost alternative to decoder-level attacks. A single encoder-targeted image also transferred to unseen models, illustrating a cross-model risk from shared visual representations.

  • Optimizing an approximately 100 k-dimensional embedding is 3–5× faster than full decoder-level attacks and fits on a 24 GB GPU.The encoder-focused attack also avoids brittle token-level constraints and offers higher payload capacity.Source: §2 · p. 5
  • The approach uses PGD with cosine-similarity loss to align the adversarial image embedding with the target prompt embedding.The encoder is frozen, and the alignment effectively writes the malicious system prompt into the visual channel.Source: §2 · p. 5
  • 44.3% attack success was achieved by an image crafted against LLaVA-1.6 and transferred to nine unseen commercial and open-source models without per-model fine-tuning.Source: §2 · p. 5
  • The authors identify shared visual representations as a monoculture risk and call for diversity or additional hardening in VLM visual front-ends.Source: §2 · p. 5

4 Experiments

Experiments test MFA against open-source and commercial VLMs, then assess its components and cross-model transfer. MFA outperforms baselines on commercial models, while ablations show complementary attack facets and transfer across diverse models and content moderators.

4.2 Results Analysis

  • GPTFuzzer achieves 0% on GPT-4.1, while MFA reaches 40.0% LG and 20.0% HM; MFA also significantly outperforms baselines on GPT-4o and GPT-4V.The study evaluates 17 VLMs on HEHS and StrongReject, using human and automated harmfulness measures.Source: §4.1 · pp. 5–6, Table 3 · p. 6, §4.2 · p. 6
  • Open-source alignment-only VLMs show higher attack success than commercial models, though GPTFuzzer sometimes outperforms MFA on LLaMA-3.2 and LLaMA-4-Scout.The authors attribute this contrast to the absence of stronger defenses such as content filters on those open-source models.Source: §4.2 · p. 6
  • MFA transfers to models it never interacted with, including GPT-4o, GPT-4.1, and Gemini-2.5-flash.The authors link this cross-modal transfer to the transfer-enhancement objective and vision-encoder adversarial images.Source: §4.2 · p. 6
  • MFA generates diverse harmful responses that reflect the original instruction, whereas heuristic attacks can produce indirect or irrelevant outputs.The section identifies Fig. 4 as real attack cases and points to Appendix D for additional examples.Source: §4.2 · p. 6, Figure 4 · p. 7

4.3 Ablation Study

  • Transfer achieves the highest average ASR across seven content moderators: 80.00% on HEHS and 68.70% on StrongReject.The comparison includes GCG and BEAST and tests both Fast and Transfer variants.Source: §4.3 · p. 6, Table 4 · p. 7
  • A single image optimized with MiniGPT-4’s vision encoder induces harmful outputs across six VLMs, reaching 59.58% average ASR without model-specific tuning.mPLUG-Owl2 reaches 85.00%.Source: §4.3 · p. 7, Table 5 · p. 7
  • Combining the adversarial signature with visual and text attacks reaches 72.92% success, with each facet targeting a distinct vulnerability.The ablation reports that each facet is effective in isolation and that their combination has a compounded effect.Source: §4.3 · pp. 6–7

5 Discussion & Conclusion

The discussion reports lower computational cost for the visual attack and identifies weak reasoning contrast as a failure condition. The appendices document evaluation settings and victim-model coverage, while the conclusion frames the work as a benchmark for future VLM safety research.

5 Discussion & Conclusion

  • On MiniGPT-4, the visual attack uses 10× fewer parameters and GMACs than Visual-AE, while Fast resolves a HEHS prompt in 17.0s versus 43.7s for GCG.The timing comparison was run on an NVIDIA A800.Source: §5 · p. 7
  • MFA failures mainly occur when VLMs lack reasoning contrast; mPLUG-Owl2 often repeats or gives ambiguous replies such as “Yes and No.”The authors point to Appendix E for failure-case visualizations and analysis.Source: §5 · p. 7

B.1 Hardware Environment

  • Experiments use NVIDIA A800 and RTX 4090 GPUs; the conclusion presents the evaluation as a practical benchmark for future research.The A800 supports high-resolution adversarial image optimization and open-source inference; the RTX 4090 supports ablations and low-resolution optimization.Source: §5 · p. 7, §B.1 · p. 10

B.2 Details of Victim Open-source VLMs.

  • The open-source evaluation covers eight models with diverse vision encoders, LLM backbones, and alignment pipelines, using public checkpoints and default settings without added safety layers.Appendix A identifies additional case studies and failure-case visualizations among its supplements.Source: §B.2 · p. 10, A Appendix Overview · p. 10

B.3 Details of Victim Commercial VLMs

  • Commercial models are evaluated through official APIs or Poe using default configurations; their reward-model details and filter thresholds are proprietary, so evaluation is black-box.The commercial test bed is described as representing publicly accessible defenses.Source: §B.3 · pp. 10–11

B.4 Our Approach Implementation.

The implementation details specify two attack pipelines: a two-stage adversarial signature for content filters and adversarial images optimized on surrogate vision encoders. The section also records the implementation choices used for baseline comparisons.

B.4 Our Approach Implementation.

  • The filter-targeted signature uses 20 tokens split into a 15-token prefix and 5-token tail, optimized in two stages against LlamaGuard2 and LlamaGuard unsafe scores.Each stage runs for at most 50 steps or stops early when the relevant moderator classifies the prompt as safe; the resulting signature is reported to bypass multiple moderators.Source: §B.4 · p. 11
  • The vision attack optimizes adversarial images on surrogate encoders using PGD, with 224 px and 448 px versions and different perturbation budgets.The 224 px image uses LLaVA-1.6 for 50 iterations at an ℓ∞ budget of 128/255; the 448 px image uses InternVL-Chat-V1.5 for 100 iterations at 64/255.Source: §B.4 · p. 11
  • High-resolution-input models receive the 448 px image, other open-source models receive the 224 px version, and commercial systems are tested at both resolutions.The stronger result is reported for commercial systems; the tested adversarial images pass LlamaGuard-Vision, which is omitted from the core evaluation because such filters are not widely deployed.Source: §B.4 · p. 11

B.5 Baseline Implementation

  • Baseline implementations follow their official or default settings, including generated images for FigStep, HADES, and CS-DJ and transfer attack settings for GPTFuzzer.For GPTFuzzer, the selected template achieved 67% ASR on the training set; Visual-AE uses the authors’ released unconstrained images, and HIMRD uses Gemini-1.0-Pro for heuristic prompt search.Source: §B.5 · pp. 11–12

C.1 Ablation Study on ATA.

The appendix reports reward-model comparisons supporting ATA and supplies further filter-attack details and qualitative examples. Across the additional cases, MFA elicits policy-violating responses spanning multiple abuse domains.

C.1 Ablation Study on ATA.

  • Dual-answer responses consistently receive higher rewards than refusals across tested reward models and VLMs, with win rates reaching 97.5% for Tulu and 95% for RM-Mistral.The authors interpret these comparisons as evidence that the policy favors harmful content in dual-answer responses.Source: §C.1 · p. 12, Table A-3 · p. 12

C.2 Ablation Study on Filter-Targeted Attack.

  • The filter-targeted ablation evaluates seven moderators spanning open-source and proprietary systems, with varied base-model sizes and safety datasets.BEAST and GCG are run using their official code with the target string changed from “Sure” to “Safe”; other hyperparameters remain at default.Source: §C.2 · p. 12, Table A-4 · p. 13

D Additional MFA Case Studies

  • Additional cases span racist propaganda, conspiracy scripts, ransom notes, and harmful-activity instructions, with responses directly satisfying the malicious prompts.The appendix includes screenshots and API transcript snippets; it reports identical safety behavior across those access paths.Source: D Additional MFA Case Studies · p. 12
  • Examples include harmful outputs from Gemini-2.0-Pro, Llama-3.2-11B-Vision-Instruct, NVLM-D-72B, and GPT-4V under visual or alignment facet attacks.The examples include extremist recruitment, discriminatory content, misinformation, and harmful instructions.Source: Figure A-4 · p. 15, Figure A-5 · p. 16, Figure A-6 · p. 17

E Failure Case Analysis

The failure cases show that following MFA’s dual-response format does not always produce substantive harmful content: LLaVA-1.5 refuses hateful text, while ShareGPT4V and mPLUG-Owl2 give terse replies.

E Failure Case Analysis

  • LLaVA-1.5 follows MFA’s dual-response format but replaces hateful content with a placeholder and appends the adversarial signature.The paper interprets this as treating the request as a meta illustration and as evidence of a strong alignment barrier against racism.Source: E Failure Case Analysis · p. 20, Figure A-10 · p. 20
  • ShareGPT4V and mPLUG-Owl2 comply with the dual-response template but give extremely terse answers, yielding partial or negligible jailbreak success.The paper attributes this to limited reasoning depth, smaller model capacity, and weaker instruction-following than larger VLMs.Source: E Failure Case Analysis · p. 20, Figure A-11 · p. 20
Loading 2511.16110v1…