Source-linked AI summary
WebInject: Prompt Injection Attack to Web Agents
Xilong Wang, John Bloch, Zedian Shao, Yuepeng Hu, Shuyan Zhou, Neil Zhenqiang Gong
TL;DR
WebInject addresses prompt injection against screenshot-driven MLLM web agents, where attackers seek to induce specified actions through webpage manipulation. It optimizes stealthy pixel perturbations while approximating non-differentiable rendering and resizing, and experiments show strong gains over baselines. The approach is evaluated under a threat model requiring webpage source-code access and currently leaves closed-source transferability unevaluated.
Problem
MLLM-based web agents remain vulnerable to prompt injection, in which manipulated web environments can induce attacker-chosen actions.
Method
WebInject optimizes a bounded raw-pixel perturbation to maximize the target-action probability, approximating non-differentiable webpage-to-screenshot mapping and using projected gradient descent.
Results
WebInject achieves an ASR of 0.972 against Gemma-3, compared with 0.062 for the strongest existing webpage-based attack and 0.000 for screenshot-based attacks.
Takeaways & Limitations
WebInject provides an effective, stealthy, and practical webpage-based prompt injection attack across diverse target monitors and attacker-specified actions.
Takeaways & Limitations
The threat model assumes source-code access, and transferability to closed-source MLLMs was not evaluated because multi-surrogate optimization was computationally infeasible.
Abstract
from arXiv · showhide
Multi-modal large language model (MLLM)-based web agents interact with webpage environments by generating actions based on screenshots of the webpages. In this work, we propose WebInject, a prompt injection attack that manipulates the webpage environment to induce a web agent to perform an attacker-specified action. Our attack adds a perturbation to the raw pixel values of the rendered webpage. After these perturbed pixels are mapped into a screenshot, the perturbation induces the web agent to perform the attacker-specified action. We formulate the task of finding the perturbation as an optimization problem. A key challenge in solving this problem is that the mapping between raw pixel values and screenshot is non-differentiable, making it difficult to backpropagate gradients to the perturbation. To overcome this, we train a neural network to approximate the mapping and apply projected gradient descent to solve the reformulated optimization problem. Extensive evaluation on multiple datasets shows that WebInject is highly effective and significantly outperforms baselines.
1 Introduction
WebInject targets prompt injection in MLLM-based web agents by perturbing webpage pixels so agents perform attacker-chosen actions. It optimizes these perturbations despite non-differentiable rendering and resizing, and substantially outperforms existing attacks.
- Prompt injection can induce web agents to perform attacker-chosen actions with consequences including click fraud, malware downloads, or sensitive-information disclosure.
- Existing webpage-based attacks are often heuristic-driven and trade off effectiveness against stealth, while direct source-code perturbation fails because webpage-to-screenshot mapping is nontrivial.
- WebInject manipulates webpage raw pixels so the resulting screenshot induces an MLLM-based web agent to perform a target action.The attack aims to combine effectiveness, stealthiness, and practical feasibility.
- WebInject formulates perturbation finding as probability maximization under an ℓ∞ constraint, approximates non-differentiable mappings, and uses projected gradient descent.
- 0.910 higher success rate than the best-performing baseline is achieved against Gemma-3, with evaluations spanning synthetic and real webpages.
2 Background
The background models a webpage as rendered pixels transformed by a monitor-specific ICC profile into the screenshot observed by a web agent. The agent combines the prompt, resized screenshot, and action history to generate sequential actions.
- A browser renders HTML source code into raw pixels, which are transformed by a monitor-specific ICC profile before becoming the screenshot.The raw image and screenshot are represented as wd × hd × 3 tensors.
- Different monitor sizes and ICC profiles can produce different screenshots for the same webpage.
- An MLLM-based web agent observes the webpage through its monitor screenshot and iteratively interacts with the webpage environment.
- At each step, the agent receives the user prompt, resized current screenshot, and prior-action history, then outputs the next action.
3 Threat Model
The threat model gives the attacker control of a target webpage and defines effectiveness as inducing a target action across realistic prompts and monitors while keeping modifications invisible. It assumes source-code access, shadow histories, model parameters, and monitor information, but not real histories or screenshots.
- The attacker seeks effectiveness and stealthiness: induce an attacker-specified target action while keeping webpage modifications invisible to users.
- Target prompts mimic natural user requests, and target monitors represent commonly used devices whose size and ICC profile affect agent behavior.
- The attacker is assumed able to modify the target webpage source code and construct shadow histories by sampling actions from the action space.
- The attacker is assumed to know the MLLM parameters but lacks the web agent’s real interaction history and cannot directly modify screenshots.
4 WebInject
WebInject formulates webpage perturbation as an optimization problem balancing target-action effectiveness, imperceptibility, and compatibility across monitors. It solves the resulting nondifferentiable problem with learned mapping and differentiable resizing approximations, PGD, and source-code injection.
- 4.3 Implementing the Perturbation δ: The final attack injects webpage code that adds δ to rendered raw pixels within the shared region, producing the desired modified webpage.The implementation targets I(ω′, d) = I(ω, d) + δ for each target monitor.
- 4.1 Formulating an Optimization Problem: WebInject optimizes a perturbation that increases the MLLM’s probability of generating the target action while bounding its ℓ∞-norm for imperceptibility.The formulation uses summed cross-entropy across target prompts, monitors, and shadow histories.
- 4.1 Formulating an Optimization Problem: The perturbation is restricted to the rectangular region shared by all target monitors, with zero values outside that overlap.This prevents monitor-specific dimensions from hiding part of the perturbation.
- 4.2 Solving the Optimization Problem to Obtain the Perturbation δ: WebInject approximates the non-differentiable webpage-to-screenshot mapping with a monitor-specific neural network and replaces resizing with a differentiable alternative.The mapping network is trained from rendered input-output pairs generated under target-monitor ICC profiles.
- 4.2 Solving the Optimization Problem to Obtain the Perturbation δ: Projected gradient descent initializes δ at zero, updates it using minibatch gradients, and projects it onto the stealthiness and monitor-overlap constraints.Clamping enforces elementwise values within [−ϵ, ϵ], while a mask preserves only the shared monitor region.
5 Experiments
Across ten webpage datasets and five MLLM-based web agents, WebInject consistently achieves high attack success and outperforms existing attacks. Ablations show robustness across monitors, perturbation bounds, prompt variants, and target actions.
- 5.1 Experimental Setup: The evaluation covers ten real and synthetic webpage datasets, five MLLMs, generated target prompts, user histories, and ASR as the effectiveness metric.The five evaluated models are UI-TARS, Phi-4, Llama-3.2, Qwen-2.5, and Gemma-3.
- 5.2 Experimental Results: WebInject achieves ASR 0.972 with Gemma-3, versus 0.062 for the best webpage-based attack and 0.000 for screenshot-based attacks.Table 1 averages ASR across ten webpage datasets and compares multiple attacks across MLLM-based web agents.
- 5.2 Experimental Results: Increasing the number of target monitors slightly decreases average ASR because optimization is restricted to their overlapping region, while expanding the set of users attacked.WebInject remains significantly stronger than webpage- and screenshot-based attacks as the number of target monitors increases.
- 5.2 Experimental Results: As ϵ increases from 4/255 to 32/255, WebInject’s ASR rises to nearly 1 because the optimization space becomes larger.The passage notes that ϵ ≤16/255 is generally considered stealthy in prior work.
- 5.2 Experimental Results: For semantically equivalent but textually different user prompts, WebInject achieves comparable ASR, including 0.957 versus 0.988 for Gemma-3 on synthetic blog webpages.The attack extends to prompts semantically similar to the target prompt used during optimization.
- 5.2 Experimental Results: WebInject also successfully misleads Phi-4 into generating target actions beyond click((x,y)) on the synthetic Blog dataset.The study evaluates alternative target actions in addition to the default randomly chosen click coordinate.
6 Related Work
Prior work applies prompt injection to web agents through webpage modifications or screenshot perturbations. WebInject instead optimizes perturbations that can be implemented through webpage source code.
- Prompt injection attacks to web agents: Webpage-based attacks inject deceptive or lookalike HTML elements, but they are often heuristic-driven and trade off effectiveness against stealth.Examples include misleading pop-ups and elements resembling legitimate attacker-chosen targets.
- Prompt injection attacks to web agents: Screenshot-based attacks optimize visual perturbations added directly to screenshots to increase the probability of an attacker-chosen action.These attacks use adversarial-example techniques rather than modifying the webpage source code.
- Prompt injection attacks to web agents: WebInject distinguishes itself by optimizing perturbations that are directly implementable through webpage source-code changes while targeting effectiveness, stealth, and practicality.This contrasts with prior webpage- and screenshot-based approaches.
7 Conclusion
WebInject is presented as an effective, stealthy, and practical prompt injection attack that optimizes a universal perturbation across target monitors to induce attacker-chosen actions.
- WebInject optimizes a universal perturbation across diverse target monitors to maximize the probability that web agents perform an attacker-chosen target action.
8 Limitations
The paper identifies limitations in its threat model and evaluation scope, including the ability to modify webpage source code and the absence of transferability tests on closed-source MLLMs.
- The threat model assumes attackers can modify target webpage source code, which may not apply to highly trustworthy sites such as Amazon.
- WebInject was not evaluated for transferability to closed-source MLLMs because optimizing across multiple surrogate models was computationally infeasible.
- Suggested defenses include source-code analysis, screenshot perturbation detection, and adversarial training of MLLMs.