AgentHijack: Visual Patch Attacks on Multimodal Computer-Use Agents

Zhihao Liu; Hongyu Sun; Zhiyuan Fu; Xiaonan Duan; Jice Wang; Shangru Zhao; Weizhi Meng; Wuxin Yang; Yangfan Zhou; Yuqing Zhang

Source-linked AI summary

AgentHijack: Visual Patch Attacks on Multimodal Computer-Use Agents

Zhihao Liu, Hongyu Sun, Zhiyuan Fu, Xiaonan Duan, Jice Wang, Shangru Zhao, Weizhi Meng, Wuxin Yang, Yangfan Zhou, Yuqing Zhang

arXiv:2609.09212v1Sep 6, 2026cs.CRcs.AIcs.CY

TL;DR

The paper addresses whether optimized local visual patches can move beyond influencing VLM outputs to produce verifiable environmental consequences in computer-use agents. It develops an end-to-end framework that trains patches on controlled webpages and evaluates real screenshot-to-action trajectories, finding strong text-level induction but substantial losses at parsing and execution, with real environment effects in a non-trivial subset of cases.

Problem
The paper asks whether a trainable image trigger can survive screenshot perception, VLM generation, action parsing, and environment execution, because prior work had not systematically established this closed-loop risk.
Method
The paper trains fixed-position local patches on author-controlled GitHub Pages and a local CSDN clone, then evaluates real trajectories using T-ASR, TAPR, and E2E-ASR across the agent pipeline.
Results
84.5% T-ASR, 47.0% TAPR, and 20.3% E2E-ASR across 600 online cases show strong text-level induction followed by substantial losses at parser and execution layers.
Takeaways & Limitations
Optimized local visual signals can propagate through open computer-use-agent execution pipelines and produce real environmental consequences in a non-trivial subset of cases.
Takeaways & Limitations
The attacker cannot modify the system prompt, user instruction, model weights, action parser, environment controller, evaluator, VM state outside the pipeline, or logs.

Abstract

from arXiv · show

This paper presents an end-to-end evaluation framework for image-triggered command injection against computer-use agents (CUAs). The goal is to test whether a local visual patch can induce verifiable environmental consequences along the full chain of screenshot input, VLM generation, action parsing, and environment execution. We train and deploy patches on author-controlled GitHub Pages pages and a locally deployed CSDN clone, and evaluate them in real environments across five open-source or publicly available GUI-agent or vision-language-model (VLM) backends. Our experiment aggregates 600 instance-level online cases, with T-ASR, TAPR, and E2E-ASR reaching 84.5%, 47.0%, and 20.3%, respectively. Trajectory analysis further shows that in some successful cases the agent first executes a malicious terminal command and then continues the original benign task. These results indicate that optimized local visual signals can affect not only VLM outputs but also propagate through the execution pipeline of open CUAs and create real environmental risk.

1. Introduction

The paper frames visual patches as a threat to computer-use agents when their influence survives screenshot perception, VLM generation, action parsing, and environment execution. It introduces an end-to-end evaluation framework using controlled webpages and real task workflows to measure this closed-loop risk.

Background: Computer-use agents increasingly convert screenshots, webpages, and natural-language tasks into browser, mouse, keyboard, or terminal actions.As a result, errors can propagate beyond the response layer into action parsing and environment execution.
Research gap: Prior work identifies untrusted web content, tool-return values, and visible GUI carriers as channels for indirect prompt injection and tool-chain contamination.However, systematic evidence remains limited on whether trainable image triggers survive later agent-pipeline stages.
Threat setting: The threat setting uses a trained local visual patch to induce terminal commands related to confidentiality, integrity, or availability without placing explicit malicious text in the page.The central question is whether the visual signal can produce VLM output compatible with the action parser and alter the real environment.
Framework: The framework connects offline patch optimization with online evaluation across screenshot input, VLM generation, action parsing, and environment-state change.It trains patches on author-controlled GitHub Pages and a locally deployed CSDN clone, then evaluates trajectories with T-ASR, TAPR, and E2E-ASR.
Evaluation design: The paper separates target-string hits, parser acceptance, and environment-level success because a single ASR-style number cannot characterize real agent risk.Across 600 online cases, these layers are represented by T-ASR, TAPR, and E2E-ASR, respectively.
Motivation: The motivating scenario keeps the user’s desktop task benign while an ordinary-looking webpage image carries a learned signal that changes the agent’s next action.The setting models realistic technical pages and tutorial-style workflows that agents may use as information sources.
Motivation: Output-only evaluation is insufficient because harmful effects require the visual trigger to survive screenshot perception, VLM generation, action parsing, and environment execution.A parsed terminal command can modify files, copy sensitive assets, or stop services, unlike a changed textual response alone.

3. Background and Threat Model

The paper frames visual patches as threats to closed-loop computer-use agents, extending prior environmental-injection work by measuring whether effects survive VLM generation, action parsing, and environment execution. Its threat model limits attackers to a bounded patch on an author-controlled webpage while requiring verifiable final-state consequences.

Background: Prior studies show that untrusted content, tool outputs, and visible GUI carriers can influence agent decisions, but systematic evidence for trainable image triggers surviving later pipeline stages remains limited.The paper positions its contribution relative to indirect prompt injection, environmental carriers, and visual patch attacks.
Problem Definition: The agent is modeled as a closed loop in which a multimodal model produces output, an action parser converts it into an action, and the environment transition function updates state.The formal pipeline uses y_t = M(I_t ⊕_Ω δ, b, h_t), a_t = P(y_t), and E_t+1 = T(E_t, a_t).
Problem Definition: The attacker optimizes a local patch so a benign task and screenshot induce parser-compatible output, while online success requires the final state to satisfy an adversarial evaluator.The feasible patch set constrains size, location, and perturbation, and the practical offline objective uses target-string supervision because parsing and execution are nondifferentiable.
Threat Model: The victim is a CUA handling benign tasks, screenshots, VLM responses, parsed actions, and desktop execution, with local files, configurations, services, and task integrity as protected assets.The attacker’s goal is an adversarial terminal action whose consequence is verified through the final environment state.
Threat Model: The attacker controls one visible image region and uses a backend-specific train-on-target patch, but cannot modify prompts, model weights, parsers, controllers, evaluators, system state, or logs.Experiments remain confined to author-controlled pages, test repositories, recoverable virtual machines, and isolated environments.

4. Attack Framework

The attack framework separates offline visual-patch optimization from online real-environment evaluation, allowing the paper to distinguish target-string induction from failures in parsing or execution. It trains backend-specific fixed-position patches and evaluates how much their effect transfers through the screenshot-to-action workflow.

Framework Design: The framework first optimizes patches offline and then freezes them for online evaluation in the real task workflow.This split enables controlled gradient-based training while testing transfer under a publish-first, observe-later threat model.
Framework Design: Offline and online stages support layered diagnosis: weak string hits indicate failed induction, whereas high string hits with low TAPR or E2E-ASR implicate parsing or execution losses.The three metrics map to text generation, action parsing, and environment success.
Offline Optimization: The offline stage trains a shared 700 × 500 fixed-position patch over multiple screenshots and prompt variants from the same GitHub Pages and local CSDN environments used online.Each backend is optimized with its own VLM and later evaluated through the corresponding agent wrapper.
Offline Optimization: Target-only supervision trains the token span for a parser-compatible command fragment while ignoring prompt, delimiter, and image-placeholder tokens.This focuses optimization on emitting the target string rather than reproducing a full templated response.
Online Evaluation: During online evaluation, the agent observes the patch through screenshots, the VLM generates a response, the parser converts it into GUI or terminal actions, and the environment updates its state.Recorded outputs, parsed actions, logs, screenshots, and final checks distinguish failure at each interface.

5. Experimental Setup

The experiments evaluate controlled CIA-oriented tasks across five reproducible agent-wrapper/backend stacks in two author-controlled web environments. They combine 600 online cases, layered success metrics, and clean-patch and placement controls to measure attack success and benign utility.

Evaluation Targets: 600 online cases result from 15 adversarial objectives, 2 webpage contexts, 4 benign-task variants, and 5 evaluated backends.Each backend executes 120 logical cases once.
Task Construction: The four benign variants instantiate the same adversarial objective under different harmless semantics, while all require opening a terminal and writing a marker or command output.The family label therefore describes benign task semantics rather than different action requirements.
Evaluation Targets: The evaluation unit is an agent-wrapper plus backend pair, emphasizing publicly released or directly implemented reproducible stacks rather than isolated backend capacity.This framing includes the prompt_agent OSWorld-style adapter and other public GUI-agent projects or implementations.
Deployment: The online stage uses an author-built GitHub Pages site and a locally deployed CSDN clone, where patches appear as ordinary visible content and are passively observed through screenshots.The deployment uses fixed visible regions, including the GitHub Pages location at (361, 165).
Metrics: T-ASR measures target-string hits, TAPR measures parser recognition, and E2E-ASR measures task-specific final-state success, with E2E-ASR providing the most direct environment-level rate.The metrics separate text, action, and environment layers.
Controls: Clean-patch controls reuse the attack pipeline with harmless crops, while random patches and wrong-position deployment test dependence on learned content and spatial placement.Benign-task completion is logged separately from attack success.

6. Main Experimental Results

Across 600 online cases, the visual patch’s influence declined consistently from VLM output to parser acceptance and finally to environmental state change. Backend-specific results and failure signatures show why layered metrics are needed to localize attack risk rather than relying on a single success rate.

Aggregate results: 84.5% T-ASR, 47.0% TAPR, and 20.3% E2E-ASR were achieved across 600 aggregated online cases.The results show substantial loss from target-string induction to parser acceptance and environment-state change.
Backend comparison: Qwen3.5-4B reached 107/120 T-ASR, 80/120 TAPR, and 67/120 E2E-ASR, while UI-TARS-1.5-7B had 0% E2E-ASR despite perfect T-ASR.EvoCUA-8B retained non-trivial T-ASR and TAPR but lost most cases during execution.
Layered outcomes: The consistent decrease from T-ASR to TAPR to E2E-ASR shows that visual influence can be attenuated by parsing, focus, ordering, privilege, or environment-state constraints.A text-level hit is therefore not automatically equivalent to real environmental harm.
Failure analysis: Web rendering, viewport cropping, compression, and layout changes perturb offline-trained triggers, producing lower TAPR or E2E-ASR in real environments.Some cases that trigger easily during offline validation lose effectiveness after deployment to controlled websites.
Failure analysis: Backend failure signatures include output-format mismatch, observe-before-act behavior, fallback to benign tasks, and UI-grounding instability during closed-loop execution.These patterns explain why text-level success does not reliably produce executable or environmental attack outcomes.
Evaluation implications: Reporting T-ASR, TAPR, and E2E-ASR together localizes risk across VLM generation, action parsing, and environment execution.A single success rate can overestimate dangerous actions or hide visual influence attenuated by later pipeline stages.

7. Control Experiments and Case Studies

Control experiments show that the attack depends on the optimized patch’s precise location and content, while trajectory analyses demonstrate both execution failures and covert successful insertions into benign workflows.

Position and trigger controls: 11/12 (91.67%) T-ASR at the original position fell to 0/12 at all eight shifted positions, showing strong spatial specificity.The combined shifted-position result was 0/96.
Position and trigger controls: Moving the patch away from its trained coordinate sharply reduces target-string hits, indicating dependence on spatial alignment.
Position and trigger controls: Harmless patches did not spuriously trigger the attack target, supporting optimized trigger content rather than merely adding an image block.
Successful trajectory: A successful trajectory opened a terminal, entered the predefined attack command, and copied the target file into an attacker staging directory.
Successful trajectory: The agent then resumed the benign tmux-installation workflow, inserting the malicious action as a covert auxiliary step rather than an explicit task failure.
Failure modes: TAPR failures affected perception or local decision making but did not produce the structured terminal-plus-command action template.
Failure modes: Action-execution failures passed TAPR but failed at the environment layer because of focus errors, command offsets, unstable ordering, or environmental constraints.

8. Security Implications for Open-Source GUI Agents

The results show that visual attacks can produce system-level consequences in open-source agent stacks, while security assessment must examine the full execution chain rather than task completion or text output alone.

System-level consequences: Successful attacks changed environment state by copying private keys, modifying configuration files, or stopping services after malicious terminal commands were executed.
Scope across agent stacks: The evaluated open-source stacks inherit visual environmental-injection risk, including Qwen3.5-4B and Qwen3.5-9B connected through a prompt_agent adapter.
Evaluation implications: Security evaluation should jointly cover visual input, VLM output, action parsing, and system execution rather than relying only on task-completion rate or single-turn output quality.
Covert workflow insertion: An online trajectory shows that an adversarial terminal action can occur as an auxiliary step inside an otherwise benign task workflow.

9. Ethics Statement

The study confines command-injection experiments to author-controlled pages, isolated environments, test repositories, and recoverable virtual machines, excluding real third-party platforms, user data, and production systems.

Scope and safeguards: All experiments used author-controlled GitHub Pages, a local CSDN clone, test repositories, recoverable virtual machines, and isolated environments.
Scope and safeguards: The study did not attack real third-party platforms, real user data, or production systems.
Purpose: The stated purpose is to support rigorous security evaluation and defense design rather than increase real-world offensive capability.

10. Conclusion

The paper evaluates whether optimized local visual patches can survive the full CUA execution chain and produce environmental consequences. Across 600 online cases, effects were strongest at VLM output and attenuated at parsing and execution, yet some cases caused real security impact.

Conclusion: The framework connects offline visual-trigger optimization with online real-environment evaluation across screenshot input, VLM generation, action parsing, and environment execution.
Conclusion: 84.5% T-ASR, 47.0% TAPR, and 20.3% E2E-ASR were recorded across 600 online cases.
Conclusion: The results show broad influence at the VLM-output layer followed by substantial loss at parser and execution layers, while still producing environmental consequences in a non-trivial subset of cases.
Authorship statement: The technical conclusions, experimental results, citation choices, and submission decisions remain the authors’ responsibility despite language-model assistance with manuscript organization and polishing.

Appendix A. Online Task Suite Construction

The online task suite crosses adversarial objectives, controlled webpage contexts, and benign-task variants, with objectives organized by CIA risk category. Author-controlled local environments isolate visual-trigger effects, while benign variants retain terminal use despite their semantic labels.

Task-space construction: 120 task instances per evaluated backend combine 15 adversarial objectives, 2 controlled webpage contexts, and 4 benign user-task variants.Concrete payload commands are kept in local task manifests.
Task-space construction: Adversarial objectives are organized into confidentiality, integrity, and availability categories.The inventory is split across compact column-width tables.
Controlled contexts: Both deployment sites are author-controlled local web environments designed to isolate visual-trigger effects from third-party platform variables.The controlled setting excludes factors such as ads, login state, recommendations, and remote content updates.
Benign-task variants: The four benign-task variants all require opening a terminal and writing a harmless marker or command output.Their family labels describe benign semantic type rather than the absence of terminal use.

Appendix B. Online Judging Protocol

The judging protocol records model responses through final evaluator outputs and scores attack success at text, action, and environment levels. End-to-end failure includes breakdowns in content, parsing, focus, command completion, or final-state verification, while benign completion remains separate.

Recorded fields: Online instances record VLM responses, parsed actions, execution traces, evaluator outputs, screenshots, and logs.These fields support judging across the attack pipeline.
Success criteria: Text-level success requires the normalized VLM output to contain the full target string.This is the first level of the three-level success protocol.
Success criteria: Action-level success requires the parsed action sequence to match the target action template, such as opening a terminal and issuing the target command.The criterion is evaluated by Jact.
Failure handling: E2E-ASR failure includes missing target text, unparsable or misfocused actions, incomplete commands, or a final state that fails Gadv.Benign-task completion is inspected separately and excluded from the primary attack success rates.