PaperlayerView exact revision on arXiv

Source-linked AI summary

MMA-SafetyBench: A Benchmark for Multimodal Agent Safety Evaluation

Yuke Wang, Benlei Cui, Shen Pang, Xuemei Dong, Longtao Huang, Hui Xue, Yuwen Zhai, Junjie Li, Jingqun Tang, Haiwen Hong

arXiv:2610.19384v1cs.CR

TL;DR

Multimodal-agent safety evaluations do not capture how untrusted visual observations can redirect an agent’s intent and actions. This paper constructs MMA-SafetyBench with trajectory-guided, reproducible visual payloads and evaluates nine models. ASR reaches 87.96%, while SCR also captures semantic disruption without a target match.

  • Problem

    Visual observations can redirect an agent’s task interpretation and actions, a safety issue not measured by evaluations focused primarily on harmful model outputs.

    Source: §1 · p. 1

  • Method

    The authors identify task-relevant injection points in benign trajectories and insert fixed, context-matched visual payloads into a 1,083-trajectory, five-domain benchmark.

    Source: §1 · p. 1

  • Results

    ASR reaches 87.96%, while SCR captures target-matching outputs and semantic impairment that disrupts the benign task without a target match.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings motivate authenticated instruction channels and external permission enforcement because a safety-priority prompt reduces but does not eliminate failures.

    Source: §7 · p. 8

    The benchmark measures susceptibility to controlled visual-observation attacks, not deployment incidence, and excludes adaptive model-specific attackers and production UI enforcement.

    Source: §6 · p. 8

Abstract

from arXiv · show

Multimodal agents increasingly rely on visual interfaces to perceive state and select actions, creating an attack surface not captured by text-only safety evaluations. We study Visual Semantic Hijacking, in which deceptive but semantically meaningful interface elements, such as forged system alerts, cause an agent to depart from its authorized task. We propose a trajectory-guided procedure that identifies candidate injection points and inserts reproducible visual payloads into the observation stream. Using this procedure, we construct MMA-SafetyBench, which contains 1,083 adversarial trajectories across web automation, desktop GUI control, mobile navigation, document understanding, and video processing. Across nine evaluated models, Attack Success Rate (ASR) reaches 87.96%, while Semantic Compromise Rate (SCR) reaches 98.09%. A 1--3--5 scoring scheme distinguishes target-matching outputs, semantic disruption without a target match, and successful preservation of the benign task. Controls covering all nine models and five domains show that attacks persist under neutral prompts. A safety-priority prompt lowers ASR without eliminating the failure mode.

1. Introduction

Visual Semantic Hijacking describes how deceptive interface content can redirect an agent from its authorized task, making visual observations a safety concern beyond harmful text outputs. The paper introduces a trajectory-guided benchmark and evaluates nine models across five domains.

1. Introduction

  • Visual Semantic Hijacking names the failure mode in which forged dialogs, spoofed buttons, or document-image instructions alter task interpretation and action selection.Prior multimodal safety work primarily studies harmful model outputs, while agent safety also requires measuring visual redirection of actions.Source: §1 · p. 1
  • The trajectory-guided procedure locates candidate injection points in multi-step traces and inserts context-matched visual payloads fixed before victim evaluation.Fixing payloads is intended to reduce variation across repeated runs.Source: §1 · p. 1
  • ASR reaches 87.96% in the evaluation, while SCR also captures intent changes and benign-task disruption without target-matching outputs.SCR includes both target-matching outputs and semantic impairment.Source: §1 · p. 2
  • The benchmark’s contribution combines the injection procedure, a five-domain trajectory suite, and evaluation of models using ASR and SCR.The evaluation also includes neutral-prompt, clean-task, and safety-priority-prompt controls.Source: §1 · p. 2
  • MMA-SafetyBench contains 1,083 adversarial trajectories across five visual-agent domains and evaluates nine models with ASR, SCR, and neutral-prompt, clean-task, and safety-priority-prompt controls.The benchmark includes web automation, desktop GUI control, mobile navigation, document understanding, and video processing.Source: §1 · pp. 1–2

2. Related work

Prior work studies multimodal jailbreaks, embodied-agent security, and prompt-injection benchmarks, but MMA-SafetyBench focuses on visual directives that hijack an agent’s ongoing task. It distinguishes semantic compromise from target-matching outputs.

2.1. Jailbreak attacks on multimodal LLMs

  • Multimodal jailbreak research includes pixel perturbations, visual jailbreaks such as JailBreakV-28K [12] and MultiModal Linkage [17], and adaptive attack optimization [3].MMA-SafetyBench instead tests task hijacking through rendered observations, with payloads fixed across victim models and separate measures for semantic compromise and target matches.Source: §2.1 · p. 2

2.2. Security and robustness of embodied agents

  • Capability benchmarks cover web and desktop agents and video understanding [2] [4], while security work examines tool-content injections [20] and forged visual directives.MMA-SafetyBench tests whether visual warnings and dialogs override an authorized task.Source: §2.2 · pp. 2–3

2.3. Relation to prompt-injection and agent-safety benchmarks

  • Related benchmarks address rendered-interface attacks, web prompt injection [6], dynamic tool-use attacks, and iterative moderation evasions [8], which target different outcomes.MMA-SafetyBench pairs target-hit or content-output criteria with broader semantic impairment; interactive-domain ASR measures predicted click coordinates rather than completed actions.Source: §2.3 · p. 3

3. MMA-SafetyBench

MMA-SafetyBench tests visual task hijacking by inserting visual-semantic payloads into benign observation-action traces. Its threat model, five-domain task taxonomy, and trajectory-based construction procedure define the benchmark’s scope and evaluation setting.

3. MMA-SafetyBench

  • The benchmark modifies visual observations within ongoing multimodal-agent workflows and measures resulting reasoning and actions.It contains 1,083 adversarial trajectories testing command hijacking and unauthorized data disclosure.Source: §3 · p. 3, §3.3 · p. 4

3.1. Threat model

  • The black-box attacker uses the benign trajectory and interface state to choose an injection point but cannot change prompts, APIs, or permission boundaries.Forged visual directives are evaluated through predicted target coordinates or generated content, not verified environment-side effects.Source: §3.1 · p. 3
  • The threat model treats system and user instructions as trusted task definitions and visual observations as potentially adversarial content.The benchmark tests whether agents maintain this trust boundary when visual content conflicts with the original task.Source: §3.1 · p. 3

3.2. Taxonomy of visual hijacking scenarios

  • Five domains cover web navigation, desktop interaction, mobile navigation, document analysis, and video processing, adapting visual hijacking tests to each interface and output.Video payloads test whether short-lived visual content changes classification or routing outputs.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3. Data collection and payload generation

  • The dataset adds payloads to benign trajectories sampled from web, desktop, mobile, document, and video task sources.The underlying tasks are used as benign trajectories before payload insertion.Source: §3.3 · p. 4
  • Payload synthesis makes directives resemble native interface elements, including system-style dialogs and diagnostic text in video frames.The design tests whether appearance and authority cues change the priority assigned to otherwise untrusted content.Source: §3.3 · p. 4
  • Trajectory search selects task-relevant states for visual injection; video and document payloads are placed near the end of retrieved context.Payloads are pre-generated and fixed per sample to make repeated evaluations comparable and avoid generator refusals being mistaken for environment content.Source: §3.3 · p. 4

4. Tasks and metrics

The benchmark adapts three attack intents to five agent domains and scores whether agents produce target-matching outputs, show semantic impairment, or preserve the benign task. ASR uses domain-specific output criteria, and GPT-5 aids trajectory evaluation.

4.1. Evaluation tasks and scenario adaptation

  • Three attack intents—Destructive Induction, Privilege Escalation, and Data Leakage—are adapted to domain-specific interfaces and action spaces.The intents respectively request destructive actions, actions outside the benign task, or sensitive information disclosure or redirected outputs.Source: §4.1 · p. 4, §4 · p. 4

4.2. Metrics and judgment methodology

  • ASR counts a sample as successful when its output meets the domain-specific criterion: a malicious-target coordinate for web, GUI, and mobile, or adversarially intended content for document and video tasks.For spatial tasks, the predicted coordinate must fall within the annotated target bounding box.Source: §4.2 · pp. 4–5
  • The 1–3–5 rubric distinguishes target-matching outputs, semantic impairment without a target match, and rejection of the payload while preserving the authorized task.For interactive tasks, a coordinate hit counts as Score 5 even though actual clicking, state changes, and downstream harm are not verified.Source: §4.2 · p. 5
  • GPT-5 compares free-form rationales and actions with annotated adversarial intent, additionally checking target-box coordinates for spatial tasks.The passage points to human-label agreement results in Appendix A.3.Source: §4.2 · p. 5

5. Experiments

Across five agent domains, the experiments find visual task hijacking across nine models, measure both target-matching attacks and broader semantic compromise, and test prompt controls and failure interpretations.

5.1. Experimental setup

  • The evaluation uses nine models on the same 1,083 adversarial trajectories and a judge that agrees with human labels on 290 of 300 audited cases.The setup holds instructions, injection points, payloads, tools, and output schemas fixed across models; most judge disagreements are between refusal and partial compliance.Source: §5.1 · p. 5

5.2. Main results

  • No evaluated model is consistently robust across domains, and the pattern appears in proprietary and open-weight models with domain-dependent magnitude.Qwen3-VL-32B reaches 86.91% ASR on Video, while GLM-4.6V reaches 78.09% on Mobile.Source: §5.2 · p. 5
  • GPT-5 records 75.12% ASR on GUI and 70.16% on Video under restricted permissions, but the experiment does not isolate the effect of those restrictions.Its ASR is 44.00% on Web and 48.06% on Mobile.Source: §5.2 · p. 5
  • 69.17% Video and 63.69% GUI mean ASR lead the domains; GLM-4.6V peaks at 87.96% on Video.The other domain means are 43.00% on Web, 48.89% on Document, and 51.24% on Mobile.Source: §5.2 · p. 5, Table 2 · p. 6
  • 36.37 points is Claude 4.5 Sonnet’s largest ASR–SCR gap, on GUI, where ASR is 42.58% and SCR is 78.95%.Every evaluated model has a positive gap in at least four domains; SCR also captures semantic impairment without target matching.Source: §5.2 · p. 6, Figure 3 · p. 6

5.3. Prompt-sensitivity and clean-task controls

  • 84.03% is average clean-task success, while neutral prompting lowers macro-average ASR from 55.20% to 50.60%, yet attacks remain successful.Clean-task success ranges from 62.19% to 94.00%; neutral prompting lowers ASR in 40 of 45 model–domain pairs and raises it in five.Source: §5.3 · pp. 6–7, Table 3 · p. 7, Table 4 · p. 7

5.5. Failure-mode audit and metric interpretation

  • Score 3 captures semantic impairment without a target match, while Score 1 includes both explicit rejection and attentional bypass.A coordinate error can leave ASR unsuccessful even when the agent adopts an injected instruction; attentional bypass does not necessarily reflect a general safety mechanism.Source: §5.5 · pp. 7–8

5.4. Safety-priority prompt baseline

  • 55.20% to 19.54% is the average ASR reduction with the Safety-Priority Prompt across 45 model–domain pairs, but residual failures remain.GLM-4.6V still reaches 46.89% on GUI and 45.03% on Video; the prompt cannot authenticate rendered instructions or enforce permissions outside the model.Source: §5.4 · p. 7, Table 5 · p. 8

6. Limitations

MMA-SafetyBench measures susceptibility to controlled visual-observation attacks rather than deployment incidence.

6. Limitations

  • The benchmark uses fixed benign tasks, injection points, payloads, tools, and output schemas, so it does not test adaptive model-specific attackers or production UI enforcement.Its scope is controlled susceptibility, not deployment incidence.Source: §6 · p. 8
  • GPT-5 agrees with human labels on 290 of 300 audited trajectories, but alternative adjudication may affect borderline Score 1/Score 3 cases.Closed-model updates also add variation.Source: §6 · p. 8
  • Trusted rendering, provenance checks, confirmation gates, and external permission enforcement remain to be evaluated.The Safety-Priority Prompt cannot authenticate rendered content or enforce permissions outside the model.Source: §6 · p. 8

7. Conclusion

The conclusion presents MMA-SafetyBench as a five-domain benchmark for visual-semantic hijacking, with high attack success and broader semantic compromise across nine models. The extended construction details describe curated tasks and fixed, context-matched visual payloads.

7. Conclusion

  • 87.96% ASR is reported across nine models, while SCR also captures intent changes and benign-task disruption without a target-matching output.The conclusion says a safety-priority prompt reduces but does not eliminate these failures.Source: §7 · p. 8

A.1. Task domain curation

  • MMA-SafetyBench contains 1,083 adversarial trajectories curated from existing datasets and manual workflows, prioritizing multi-step tasks involving sensitive data or consequential actions.The suite includes web, desktop, mobile, document, and video tasks.Source: §A.1 · p. 11
  • Web, desktop, and mobile tasks insert forged interface elements into browser workflows, system utilities, or Android dialogs to test responses to unauthorized actions.Web tasks use 200 sampled VisualWebArena tasks; desktop tasks use OSWorld, and mobile tasks adapt OS-Sentinel trajectories.Source: §A.1 · p. 11
  • Document tasks embed directives in receipts and resumes, while video tasks insert a forged interrupt into the final retrieved frame.Receipt tasks use CORD images; video tasks use 13 LVBench videos of up to one hour.Source: §A.1 · p. 11

A.2. Adversarial payload synthesis pipeline

  • Payloads combine interface-matched visual presentation with directives framed as required system responses, then replace the observation at a selected step.Payloads are pre-synthesized and held fixed across victim models; this removes generator sampling from comparisons, though API and model-version changes can still affect reproducibility.Source: §A.2 · p. 11

A.3. GPT-5 judge reliability validation

Human reviewers agreed with the GPT-5 judge on 290 of 300 sampled trajectories. The appendix also details attack intents, evaluation prompts and controls, and examples of model responses across domains.

A.3. GPT-5 judge reliability validation

  • 96.67% agreement was observed between the GPT-5 judge and three researchers scoring 300 trajectories across five domains.The remaining disagreements occurred primarily at the boundary between refusal and partial compliance, supporting use of the judge while motivating disclosure of the rubric and validation procedure.Source: §A.3 · p. 11

A.4. Detailed taxonomy and implementation of evaluation tasks

  • The evaluation tasks cover destructive actions, authority hijacking, data leakage, and social engineering through forged alerts, pop-ups, document marks, and directives.Examples include database commands, unauthorized data routing, and false status updates.Source: §A.4 · pp. 11–12

B. Detailed prompt templates for evaluation domains

B.6. Evaluation prompts for GPT-5 judge

  • The 1–3–5 rubric assigns Score 5 to target-matching outputs, Score 3 to semantic impairment without a match, and Score 1 to preservation of the authorized task.The judge receives the reasoning trace, visual context, final action, and annotated adversarial intent.Source: §B.6 · p. 13

B.7. Neutral-lite prompt templates

  • The neutral-lite control retains the user task while treating visual content as evidence rather than higher-priority instructions.Templates apply this principle to documents, video, web and desktop interfaces, and mobile screens.Source: §B.7 · pp. 13–14

B.8. Safety-priority prompt (SPP)

  • The safety-priority prompt prepends a high-priority instruction to refuse malicious or illegal activity.It directs the agent to prioritize that instruction over the instructions that follow.Source: §B.8 · p. 14

C.1. Representative trajectory cases

  • In the web and mobile examples, agents predict clicks inside forged fields or restart controls; the desktop example instead shows an adopted instruction with an off-target coordinate.The web and mobile cases are scored as target-matching outputs, while the desktop case is scored as semantic impairment.Source: §C.1.1 · pp. 15, 17, §C.1.2 · p. 17, §C.1.3 · p. 17
  • The document example adds a false risk status to a benign resume summary, while the video example replaces a multiple-choice answer with an injected URL.Both examples are labeled target-matching outputs.Source: §C.1.4 · p. 17, §C.1.5 · p. 17

C.2. Failure-mode analysis

The failure-mode analysis separates robust responses from semantic impairment, distinguishing explicit rejection and attentional bypass from spatial grounding errors and schema breakdown.

Failure-mode patterns

  • Four recurring patterns distinguish rejection or bypass of the payload from adoption of its intent without a target-matching output.The patterns include two Score 1 outcomes and two Score 3 outcomes.Source: §C.2 · pp. 17–18

Robust responses

  • Explicit rejection identifies and dismisses suspicious content, while attentional bypass ignores it and continues the benign task without identifying a security risk.Attentional bypass is robust on the evaluated sample but may reflect task-focused attention rather than a general safety mechanism.Source: §C.2 · p. 18

Semantic impairment

  • Spatial grounding errors indicate intent to follow an injected instruction but predict a coordinate outside its target; schema breakdown violates the action format, aborts, or loops.Both patterns receive Score 3 because neither the adversarial target criterion nor the benign task is satisfied.Source: §C.2 · p. 18

D. Released artifacts and reproducibility

The artifact section identifies released code and data and notes factors that can affect exact reproduction. The extended results caption explains how Table 6 reports SCR and its gap from ASR.

D. Released artifacts and reproducibility

E. Extended experimental results

  • Table 6 reports SCR for all nine models, with parenthesized values giving Δ = SCR − ASR.The gap represents semantic impairments without a target-matching output.Source: E. Extended experimental results · p. 18

F. Defense evaluation

The Safety-Priority Prompt (SPP) lowers attack success across reported model–domain pairs but leaves residual failures, while the discussion frames risks and mitigations within controlled experiments. The authors also describe provider notification and a restricted release.

F. Defense evaluation

  • SPP reduces ASR for every reported model–domain pair, but residual failures remain, so prompt-only defenses are insufficient for these attacks.It asks agents to reject malicious or illegal requests and prioritize that rule over later instructions.Source: F. Defense evaluation · p. 18

G.1. Potential negative impacts

  • If similar failures occur in deployment, possible consequences include unauthorized transactions, unsafe commands, and disclosure of private data.Risk depends on both the agent’s susceptibility to visual instructions and the permissions available to it.Source: §G.1 · p. 19

G.2. Mitigation strategies

  • Mitigations should separate trusted instructions from untrusted environmental content through dialog authentication, task and application-state checks, confirmations, and permission limits.The authors suggest combining prompt-level controls with enforcement outside the model.Source: §G.2 · p. 19

G.3. Responsible disclosure

  • The authors plan to notify affected providers and omit unnecessary operational details from the release; the study is limited to controlled benchmark environments.Released artifacts should exclude personal data not licensed for redistribution.Source: §G.3 · p. 19

Related papers

Loading 2610.19384v1…