Source-linked AI summary
AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructions
Zonghao Ying, Le Wang, Yisong Xiao, Jiakai Wang, Yuqing Ma, Jinyang Guo, Zhenfei Yin, Mingchuan Zhang, Aishan Liu, Xianglong Liu
TL;DR
Existing embodied-agent safety benchmarks cover limited hazards and focus mainly on final outcomes rather than the full perception–planning–execution process. AGENTSAFE combines an adversarial sandbox, risk-aware tasks, and multi-stage diagnostics; evaluations reveal failures to translate hazard recognition into safe planning and execution.
Problem
Existing safety benchmarks for embodied agents cover narrow hazard scopes and focus mainly on final outcomes instead of evaluating the perception–planning–execution process and risks to humans, environments, and agents.
Source: §1 · pp. 1–2Method
AGENTSAFE combines an interactive adversarial sandbox, a risk-aware hazardous-instruction suite, and a multi-level protocol for evaluating embodied VLM agent safety.
Source: §1 · p. 2Results
Experiments on 9 state-of-the-art VLMs and 2 embodied agent workflows reveal systematic failures to translate hazard recognition into safe planning and execution.
Source: §1 · p. 2Takeaways & Limitations
The findings underscore the need for comprehensive, multi-stage safety evaluation and the paper presents AGENTSAFE as a foundation for research on safer embodied intelligence.
Source: §1 · p. 2, §6 · p. 9The threat model focuses on inherent safety—refusing instructions that directly lead to harmful outcomes—in benign-user and malicious-actor scenarios.
Source: §3.2 · p. 3
Abstract
from arXiv · showhide
The integration of vision-language models (VLMs) is driving a new generation of embodied agents capable of operating in human-centered environments. However, as deployment expands, these systems face growing safety risks, particularly when executing hazardous instructions. Current safety evaluation benchmarks remain limited: they cover only narrow scopes of hazards and focus primarily on final outcomes, neglecting the agent's full perception-planning-execution process and thereby obscuring critical failure modes. Therefore, we present SAFE, a benchmark for systematically assessing the safety of embodied VLM agents on hazardous instructions. SAFE comprises three components: SAFE-THOR, an extensible adversarial simulation sandbox with a universal adapter that maps high-level VLM outputs to low-level embodied controls, supporting diverse agent workflow integration; SAFE-VERSE, a risk-aware task suite inspired by Asimov's Three Laws of Robotics, comprising 45 adversarial scenarios, 1,350 hazardous tasks, and 9,900 instructions that span risks to humans, environments, and agents; and SAFE-DIAGNOSE, a multi-level and fine-grained evaluation protocol measuring agent performance across perception, planning, and execution. Applying SAFE to nine state-of-the-art VLMs and two embodied agent workflows, we uncover systematic failures in translating hazard recognition into safe planning and execution. Our findings reveal fundamental limitations in current safety alignment and demonstrate the necessity of a comprehensive, multi-stage evaluation for developing safer embodied intelligence.
1 Introduction
AGENTSAFE evaluates hazardous-instruction safety across embodied agents’ perception, planning, and execution using an adversarial sandbox, risk-aware tasks, and a multi-level protocol. Experiments reveal failures to translate hazard recognition into safe planning and execution.
Motivation
- Existing benchmarks often cover narrow hazard sets and final outcomes, rather than risks to humans, environments, and agents across perception, planning, and execution.Source: §1 · pp. 1–2
SAFE-THOR
- SAFE-THOR uses a universal adapter for object grounding and action abstraction to connect high-level VLM outputs with low-level executable controls.The adapter supports integration with diverse embodied-agent workflows while aiming to preserve generalization capabilities.Source: §1 · p. 2
SAFE-VERSE
- SAFE-VERSE organizes hazardous instructions around risks to humans, the environment, and the agent, drawing inspiration from Asimov’s Three Laws of Robotics.It comprises 45 adversarial scenarios, 1,350 hazardous tasks, and 8,100 interactive hazardous instructions.Source: §1 · p. 2
SAFE-DIAGNOSE
- SAFE-DIAGNOSE evaluates safety at the perception, planning, and action stages to provide finer-grained insight into where and why failures occur.Source: §1 · p. 2
Main findings
- Experiments across 9 state-of-the-art VLMs and 2 embodied agent workflows reveal systematic failures to translate hazard recognition into safe planning and execution.The paper also proposes SAFE-AUDIT, which uses zero-shot LLM reasoning grounded in world knowledge to audit and refine action plans.Source: §1 · p. 2
2 Related Work
Prior embodied-agent safety benchmarks vary in their hazards and settings, but the paper identifies gaps in risk coverage, pipeline diagnosis, and the grounding of high-level plans into executable actions.
Prior benchmarks
Recent benchmarks
Comparison
- The paper characterizes prior work as lacking a comprehensive risk taxonomy, full pipeline failure diagnosis, and universal grounding from high-level plans to executable actions.Source: §2 · p. 3
3 Preliminaries and Threat Model
The paper models an embodied VLM agent interacting with a 3D environment through observations, instructions, reasoning, and low-level actions. Its threat model tests benign-use utility and refusal of hazardous instructions, including obfuscated attacks.
3.1 Embodied VLM Agent
- The agent-environment interaction is modeled as a partially observable Markov decision process (POMDP).Source: §3.1 · p. 3
- At each timestep, the VLM processes visual observations and a natural-language goal, reasons, and produces an executable low-level action in the 3D environment.The agent aims to generate an action sequence that completes the task safely.Source: §3.1 · p. 3
3.2 Threat Model
- The threat model evaluates inherent safety as the ability to refuse instructions that directly lead to harmful outcomes, across benign-user and malicious-actor scenarios.Source: §3.2 · p. 3
- Benign instructions establish a task-utility baseline, while hazardous instructions test safety alignment under explicit or semantically obfuscated harmful intent.The latter includes adversarially enhanced instructions that use semantic manipulation, commonly called jailbreaking [37].Source: §3.2 · p. 3
4 Benchmark Design
AGENTSAFE combines an interactive sandbox and risk-aware task suite with diagnostics across perception, planning, and execution. Its SAFE-AUIDT module audits initial reasoning before the agent produces an action plan.
4.1 SAFE-THOR Evaluation Sandbox
- SAFE-THOR uses a universal adapter to map simulator observations into VLM-usable representations and translate high-level plans into executable actions.The adapter grounds linguistic object references to simulator identifiers and maps plans to action primitives.Source: §4.1 · p. 3
4.2 SAFE-VERSE Evaluation Task Suite
- SAFE-VERSE covers human, environment, and agent risks across 45 scenes, with 9,900 instructions including 1,800 basic commands and 8,100 adversarial variants.Basic instructions include normal and baseline hazardous commands; jailbreak transformations preserve the unsafe intent of hazardous instructions.Source: §4.1 · p. 4, §4.2 · p. 5
4.3 SAFE-DIAGNOSE Evaluation Protocol
- SAFE-DIAGNOSE evaluates agent performance at perception, planning, and execution stages rather than relying on a single success metric.Stage-specific measures support diagnosis of where failures occur.Source: §4.3 · p. 5
4.3.1 Stage 1: Perception and Grounding.
- Perception and grounding compare the objects an agent perceives with scene ground truth using Grounding Recall and Hallucination Rate.Grounding Recall tracks the share of actual objects perceived; Hallucination Rate tracks perceived objects that do not exist.Source: §4.3.1 · p. 5
4.3.2 Stage 2: Planning.
- Planning evaluation measures valid refusals and complete, coherent, executable plans; these rates need not sum to one.A valid refusal can address an unsafe instruction or a task impossible in the current environment.Source: §4.3.2 · pp. 5–6
4.3.3 Stage 3: Action Execution.
- Execution evaluation measures task completion, and stage comparisons help distinguish execution problems from safety-reasoning failures during planning.Low Task Success Rate despite high Planning Success Rate can indicate execution issues, while low Planning Rejection Rate on risky tasks points to planning-stage safety-reasoning failure.Source: §4.3.3 · p. 6
4.4 SAFE-AUIDT: A Thought-Level Safety Module
- SAFE-AUIDT audits the agent’s initial thought before it becomes a multi-step plan, aiming to prevent unsafe behavior from being conceptualized.It refuses dangerous thoughts, enriches safe but suboptimal thoughts, and passes safe, robust thoughts through unchanged.Source: §4.4 · p. 6
5 Experiments and Evaluation
Across nine VLM backbones and two agent workflows, the experiments find that planning is a central safety weakness and that SAFE-AUDIT reduces hazardous execution while preserving normal-task utility.
5.1 Experimental Setup
- The setup evaluates nine representative VLMs with a thought-inclusive workflow and adds ReAct and ProgPrompt using GPT-4o; SAFE-DIAGNOSE measures perception, planning, and execution.The protocol uses GR and HR for perception, PRR and PSR for planning, and TSR for execution.Source: §5.1 · p. 6
5.2 Main Evaluation Results
- On normal tasks, agents average over 60% GR, with peak GR of 82.79% and average HR of 4.55%; action grounding converts 92.22% of valid plans into physical actions.The reported normal-instruction results appear in Table 1.Source: §5.2 · p. 6, Table 1 · p. 7
- On Human-Harm tasks, Claude-sonnet-3.5 and Claude-opus-4 achieve PRRs of 90.11% and 85.56%, respectively, while GPT-5-mini’s PRR is 18.16% and its PSR reaches 80.04%.Source: §5.2 · p. 7
- Gemini-2.5-flash's Env-Harm PSR of 65.78% accompanies a TSR of 61.56%, while Claude-opus-4's Human-Harm PSR of 12.44% accompanies a TSR of 5.89%.The authors report that planning-stage differences affect harmful execution outcomes.Source: §5.2 · p. 7
- For Env-Harm instructions, the Multilingual jailbreak improves PSR by 7.67% over baseline but lowers TSR by 2.27%, while perception varies within 8% across methods.For Self-Harm and Human-Harm, only the Multilingual method improves both PSR and TSR; other methods perform worse than baseline instructions.Source: §5.2 · p. 7
- ReAct and ProgPrompt achieve 0.00% HR on benign tasks, but ProgPrompt has zero PRR across harmful scenarios while ReAct reaches 51.28% Human-Harm PRR.The authors attribute ProgPrompt's vulnerability to its rigid executable-code plan format, which appears to bypass the model's safety guardrail.Source: §5.2 · pp. 7–8
5.3 Diagnostic Analysis
- Diagnostics distinguish normal-task grounding errors from baseline-risk planning failures and jailbreak disruptions across perception, planning, and execution; planning is identified as the primary vulnerability.Unreliable environmental grounding can still exacerbate failures, while adversarial instructions may also produce malformed or unexecutable plans.Source: §5.3 · p. 8
5.4 Defensive Performance of SAFE-AUDIT
- On normal instructions, SAFE-AUDIT raises average TSR by 2.22%; on hazardous instructions, its average PSR and TSR are 3.52% and 0.48%, respectively.The authors attribute the safety results to auditing and correcting the agent's initial thought at the planning stage; ThinkSafe lowers normal-task TSR by up to 14.96%.Source: Figure 7 · p. 8, Figure 8 · p. 8, §5.4 · p. 9
6 Conclusion and Future Work
AGENTSAFE evaluates embodied VLM safety and identifies failures to carry hazard recognition into safe planning and execution. Future work targets stronger attacks, adaptive auditing, and translation to physical environments.
6 Conclusion and Future Work
- Current agents may perceive hazards but fail to reflect that understanding in planning and execution, a vulnerability AGENTSAFE identifies in the safety pipeline.The authors envision AGENTSAFE as a foundation for research on safer embodied intelligence.Source: §6 · p. 9
- SAFE-AUDIT is a proactive planning-level module that intercepts and refines unsafe thoughts before execution.Source: §6 · p. 9
- Future work will integrate stronger multimodal attacks beyond jailbreak attacks to evaluate agent robustness.Source: §6 · p. 9
- The authors plan to extend SAFE-AUDIT into an adaptive, learning-based auditor that continuously improves safety interventions.Source: §6 · p. 9
- Translating findings to physical environments remains a long-term direction for understanding the sim-to-real gap.Source: §6 · p. 9
A.1 Comprehensive Results for Additional Backbone Models
This section supplements single-backbone main findings with evaluations of eight additional VLM backbones on adversarially enhanced instructions, reporting performance metrics across three risk types.
A.1 Comprehensive Results for Additional Backbone Models
- The supplement evaluates eight additional VLM backbones on adversarially enhanced instructions, complementing main findings focused on one backbone.Source: §A.1 · p. 10
- Table 3 reports performance metrics for the additional backbones on adversarially enhanced self-harm instructions.Source: Table 3 · p. 11
- Table 4 reports performance metrics for the additional backbones on adversarially enhanced environment-harm instructions.Source: Table 4 · p. 11
- Table 5 reports performance metrics for the additional backbones on adversarially enhanced human-harm instructions.Source: Table 5 · p. 12