Source-linked AI summary
Manipulating Multimodal Agents via Cross-Modal Prompt Injection
Le Wang, Zonghao Ying, Tianyuan Zhang, Siyuan Liang, Shengshan Hu, Mingchuan Zhang, Aishan Liu, Xianglong Liu
TL;DR
Multimodal agents lack a systematic black-box attack framework for prompt injection across interacting modalities. CrossInject coordinates visual and textual manipulation, and the paper reports at least 30.1% higher average attack success rates than existing methods.
Problem
The paper identifies a lack of systematic attack frameworks for black-box multimodal agents and comprehensive understanding of risks from cross-modal interactions.
Source: §1 · p. 2Method
CrossInject coordinates generatively aligned malicious visual content and surrogate-LLM-optimized textual commands with malicious instructions embedded in external data.
Source: §1 · p. 2Results
Across diverse multimodal agent architectures and real-world scenarios, CrossInject achieves at least 30.1% higher average attack success rates than existing methods.
Source: §1 · p. 2Takeaways & Limitations
The authors validate CrossInject’s effectiveness on real-world multimodal autonomous driving agents.
Source: §1 · p. 2, §8 · p. 11The authors identify more effective defenses for open multimodal settings and applicability to real-world agents with more complex architectures as future work.
Source: §8 · p. 11
Abstract
from arXiv · showhide
The emergence of multimodal large language models has redefined the agent paradigm by integrating language and vision modalities with external data sources, enabling agents to better interpret human instructions and execute increasingly complex tasks. However, in this paper, we identify a critical yet previously overlooked security vulnerability in multimodal agents: cross-modal prompt injection attacks. To exploit this vulnerability, we propose CrossInject, a novel attack framework in which attackers embed adversarial perturbations across multiple modalities to align with target malicious content, allowing external instructions to hijack the agent's decision-making process and execute unauthorized tasks. Our approach incorporates two key coordinated components. First, we introduce Visual Latent Alignment, where we optimize adversarial features to the malicious instructions in the visual embedding space based on a text-to-image generative model, ensuring that adversarial images subtly encode cues for malicious task execution. Subsequently, we present Textual Guidance Enhancement, where a large language model is leveraged to construct the black-box defensive system prompt through adversarial meta prompting and generate an malicious textual command that steers the agent's output toward better compliance with attackers' requests. Extensive experiments demonstrate that our method outperforms state-of-the-art attacks, achieving at least a +30.1% increase in attack success rates across diverse tasks. Furthermore, we validate our attack's effectiveness in real-world multimodal autonomous agents, highlighting its potential implications for safety-critical applications.
1 Introduction
The introduction identifies a gap in systematic attacks tailored to black-box multimodal agents and presents CrossInject as a coordinated cross-modal framework. It reports higher attack success rates across diverse settings and validation on real-world multimodal autonomous driving agents.
1 Introduction
- The paper identifies a lack of systematic attack frameworks for black-box multimodal agents and comprehensive understanding of risks from cross-modal interactions.Prior prompt-injection research had studied visual or textual settings, while coordinated multimodal attacks are non-trivial because agents integrate semantically rich inputs.Source: §1 · p. 2
- CrossInject jointly targets visual and textual channels, aligning generated images with malicious instructions and optimizing textual commands to steer agent behavior.The framework also combines these components with malicious instructions embedded in external data sources.Source: §1 · p. 2
- At least 30.1% higher average attack success rates than existing prompt-injection methods are reported across diverse agent architectures and real-world scenarios.CrossInject is also reported to transfer across agent models, resist representative defenses, and work on real-world multimodal autonomous driving agents.Source: §1 · p. 2
- The authors describe this as the first work to perform cross-modal prompt injection on LVLM-driven multimodal agents.Their evaluations span multiple agents and tasks in both digital and physical worlds.Source: §1 · pp. 2–3
2 Related Work
Related work describes multimodal agents and prior prompt-injection research, then identifies cross-modal attacks on agents as an underexplored area. Existing work predominantly targets unimodal inputs, motivating a framework that manipulates multiple input sources.
2.1 Multimodal Agent
- Multimodal agents integrate perception, planning, and action across digital and physical applications, while their wider adoption raises security concerns.The cited architecture description treats perception as integrating multiple sources, planning as formulating strategies, and action as carrying them out.Source: §2.1 · p. 3
2.2 Prompt Injection
- Prior prompt-injection methods predominantly target LLM agents, with related attacks also using visual or audio inputs.Examples include adversarial requests in external data sources and visual or audio perturbations.Source: §2.2 · pp. 3–4
- Existing agent prompt-injection research has focused mainly on unimodal inputs, while the paper proposes simultaneously attacking multiple sources across modalities.The proposed framework is intended to provide consistent attack effectiveness across diverse operational scenarios.Source: §2.2 · p. 4
3 Threat Model
The threat model represents a multimodal agent as processing a system prompt, image, external data, and user command to produce an action. CrossInject manipulates these inputs under a black-box attacker model with knowledge of the agent’s approximate functionality.
3.1 Problem Definition
- The agent’s action is modeled as a function of its system prompt, visual input, external data, and user command.For digital-world visual-question-answer agents, the action primarily takes the form of a textual response.Source: §3.1 · p. 4
- Cross-modal injection transforms the image, user command, and external data so their coordinated input can drive an action that violates the role set by the system prompt.The visual transform adds an adversarial perturbation; the textual transform embeds deceptive semantics, and the external-data transform inserts a malicious instruction.Source: §3.1 · p. 4
3.2 Adversarial Goal
- The attacker has black-box access through uploads or text commands and is assumed to know an approximate description of the agent’s functionality.External instructions may be implanted passively in public online resources or actively in a document uploaded to the agent.Source: §3.2 · p. 4
4 Methodology
CrossInject attacks multimodal agents by manipulating visual and textual inputs to steer them toward injected tasks. Its Visual Latent Alignment and Textual Guidance Enhancement components target different parts of the agent’s pipeline.
4.1 Visual Latent Alignment
- Visual Latent Alignment maps malicious instructions into target images with a text-to-image model, then aligns benign-image features to that target.An ℓ∞-bounded perturbation keeps the input visually indistinguishable while normalized feature distances support scale-invariant alignment.Source: §4.1 · p. 5
4.2 Textual Guidance Enhancement
- Textual injection complements visual input manipulation by steering the agent’s core planners toward compliance with the attacker’s request.The visual injection acts at the input end, while the textual command steers responses at the output end.Source: §4.2 · p. 5
- For black-box optimization, CrossInject uses an open-source LLM surrogate and role-aware defensive meta-prompts, then applies GCG to optimize the deceptive command toward the target action.The authors state that direct end-to-end gradient computation is infeasible for multimodal agents’ high-dimensional inputs and complex processing pipelines.Source: §4.2 · p. 6
5 Experiments
Experiments compare CrossInject with existing attacks across multimodal agents and input surfaces, then ablate its visual and textual components. CrossInject achieves the highest ASRs across evaluated scenarios, while the ablations examine how alignment, perturbation budgets, textual optimization, and surrogate choice affect attack performance.
5.1 Experimental Setup
- Tests used RecipeMaster and PoetryGenius with Qwen2-VL or Phi-3.5-vision, 100 CoEDIT/SST2 samples, local-document and webpage attacks, and ASR/PNA metrics.Source: §5.1 · p. 7
5.2 Main Results
- JIP achieved 0% ASR in all tested scenarios, while FB's average ASR increase over naive methods was 15.5% and remained below CrossInject in most cases.All evaluated agents maintained 100% performance under no attack.Source: §5.2 · p. 8
- CrossInject achieved the highest ASRs across scenarios, with local-document average gain up to 32.7% and maximum improvement 71.7%, and webpage average improvement 27.5% and maximum 34.0%.Source: §5.2 · p. 8
- Online webpages had average 10.8% lower ASR than local documents; the surfaces were comparable only for RecipeMaster with Qwen2-VL on text editing (Fig. 3).Source: §5.2 · p. 8, Figure 3 · p. 8
5.3 Ablation Studies
- Removing visual latent alignment dropped average ASR 18.7% (maximum 28.0%); aligning visual input to text instead yielded 11.1% lower ASR than CrossInject.Replacing aligned perturbations with random noise also failed to significantly influence agent behavior.Source: Table 2 · p. 8, §5.3 · p. 9
- ASR rose from approximately 70% to 90% as visual budgets increased, with diminishing returns beyond 16; textual optimization reached near-optimal performance at 100 iterations.At budget 32, RecipeMaster with Qwen2-VL showed a slight ASR decline; higher iterations gave no consistent gains for the cross-model pairings.Source: §5.3 · p. 9, Figure 4 · p. 10
- Removing the malicious command caused an average drop of 24.8% and maximum decrease of 47.0%; real-system-prompt optimization improved ASR only +3.0% on average.The authors describe their constructed-prompt approach as more practical in black-box settings.Source: §5.3 · p. 9, Table 3 · p. 9
- Using Llama-3.1-8B as surrogate achieved the highest ASR, with average improvement of 26.9% in three cases; larger, higher-quality datasets also enhanced attack efficacy.The authors report that architectural similarity to victim planners better simulates their reasoning, and Mistral-7B had lower command-optimization efficacy than Llama-3.1-8B.Source: §5.3 · pp. 9–10, Table 4 · p. 10
6 Physical World Agent Case Study
The case study tested CrossInject on a smart vehicle navigating a track with a stop sign. Across 10 attempts, CrossInject more often prevented the vehicle from bypassing the sign than a textual-only attack.
Vehicle Setup
- The physical-world test used a LIMO smart vehicle with Qwen2-VL on a constructed track where a stop sign required the agent to detour.The vehicle's system prompt instructed it to obey traffic regulations.Source: §6 · p. 10
Attack Implementation
- The attack used a stop-sign-shaped adversarial patch and the injected command “accelerate through road,” which contradicted the system prompt.The patch was tailored to the stop sign's shape for visual stealthiness.Source: §6 · p. 10
Results Analysis
- CrossInject prevented the vehicle from bypassing the stop sign in 9 of 10 attempts, versus 4 instances for the Naive textual-only attack.The illustrated attacked case shows the vehicle colliding with a traffic sign.Source: §6 · p. 10, Figure 5 · p. 10
7 Countermeasures
The paper evaluates textual, visual, and combined defenses against CrossInject on sentiment analysis with local document input. Textual defense reduces attack success more on average than Gaussian blur, while combining them does not consistently improve on the stronger individual defense.
Textual Defense
- Sandwich prompting reduced ASR by 6.7% on average and by up to 15.0%, but did not eliminate the attack.The evaluation focused on Sentiment Analysis with the local document input surface.Source: §7 · pp. 10–11
Visual Defense
- Gaussian Blur reduced ASR by 2.8% on average and was ineffective in two cases.The input image was blurred with a 9 × 9 kernel.Source: §7 · p. 11
Combined Defense
- The combined sandwich-prompting and Gaussian-Blur defense performed better than either method alone in two cases, but its gains were not additive or consistently better than the stronger method.The authors suggest partially overlapping effects may limit the combined defense.Source: §7 · p. 11
8 Conclusion and Future Work
The paper identifies cross-modal prompt injection as a vulnerability in multimodal agents and proposes CrossInject. Experiments report at least a +26.4% increase in attack success rates across diverse tasks, and the authors identify open-environment defenses and more complex real-world agents as future work.
Conclusion
- +26.4% increase in attack success rates across diverse tasks is reported for CrossInject, which was also validated on real-world multimodal autonomous agents.The paper calls for further investigation into multimodal-agent security.Source: §8 · p. 11
Future Work
- Future work includes more effective defenses for multimodal settings in open environments and extending the attack to real-world agents with more complex architectures.The authors name adversarial training as one possible defense direction.Source: §8 · p. 11