SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents

1 INTRODUCTION

• Embodied-agent research has rarely examined hazardous instructions, while prior safety studies used non-interactive image-text substitutes and lacked task abstraction and length coverage.Most existing embodied-agent benchmarks focus on planning in structured, fixed-format tasks.Source: §1 · pp. 1–2
• SafeAgentBench evaluates agents executing diverse tasks with explicit and implicit hazards in an interactive simulation, using a plan-execution-assessment workflow.The benchmark assesses task completion after a controller executes the agent’s high-level plan.Source: §1 · pp. 2–3
• The dataset contains 750 tasks spanning 10 common risks and detailed, abstract, and long-horizon task types.It includes 450 hazardous tasks and 300 safe controls.Source: §1 · p. 2
• SafeAgentEnv supports multiple agents and 17 high-level actions, while evaluation combines task execution with LLM-based semantic assessment.The semantic method addresses tasks with multiple possible outcomes and simulator defects.Source: §1 · pp. 2–3
• Only 10% of hazardous detailed tasks are rejected by ReAct, the most safety-aware baseline; it rejects 32% of abstract hazardous tasks, while ProgPrompt safely completes 50% of long-horizon cases.These are the best baseline results reported for the respective task types.Source: §1 · p. 3
• GPT-4 and three open-source models differ by less than 3% in average proactive defense rate on hazardous detailed tasks, and two explored defenses did not significantly improve safety without harming planning.The defenses were a hybrid agent design and a GPT-4-based chain-of-thought filter.Source: §1 · p. 3
• Improving agents’ understanding of environments and safety risks remains a research challenge for safer embodied systems and robot deployment.The paper frames safety perception and mitigation as future directions.Source: §1 · p. 3

2.1 EMBODIED AGENTS WITH LLMS IN TASK PLANNING

• Prior embodied LLM agents use programmatic plans, environmental feedback, architectural changes, model training, memory, and multi-agent learning, while task diversity and safety risks remain overlooked.The paper positions its hazardous tasks as a way to evaluate embodied-agent safety awareness.Source: §2.1 · p. 3

2.2 SAFETY RESEARCH FOR EMBODIED LLM AGENTS

• Embodied-safety studies examine jailbreak-triggered actions, scene-graph anomaly detection, and disaster decisions, but lack holistic agent evaluation and broad task coverage.Text-level benchmarks also assess LLM safety, rather than holistic embodied-agent behavior.Source: §2.2 · p. 3

2.3 PLANNING BENCHMARK FOR EMBODIED AGENTS

• Planning benchmarks often overlook safety; EARBench uses non-interactive scenarios, while IS-Bench and SafePlan-Bench address implicit but not explicit hazards.General-purpose benchmarks also have limitations in task diversity, action support, or emphasis on planning.Source: §2.3 · p. 4

3 DATASET

• SafeAgentBench is presented as executable, large-scale, diverse, and high-quality data for assessing safe handling of hazardous instructions.Its curation aims to support evaluation across task types and risks, including explicit and implicit safety risks.Source: §3 · p. 4

3.1 DATASET COMPOSITION

• The dataset contains 750 tasks, including 450 hazardous tasks, with each task creating a scenario for testing embodied-agent planning.Figure 2a presents the dataset composition and task-category counts.Source: §3.1 · p. 4

3.1.1 HARM CATEGORIES IN EMBODIED WORLD

• The dataset categorizes 10 domestic-robot hazards into five risks to humans and five risks to property, drawing on safety standards and hazard research.This categorization is intended to test agents’ knowledge breadth and safety awareness.Source: §3.1.1 · p. 4

3.1.2 TASK TYPES FOR PLANNING

• Detailed tasks specify objects, actions, executable plans, and goal conditions; 300 hazardous tasks are paired with 300 safe controls of comparable average length, 5.03 versus 5.12 steps.Most detailed tasks require no more than eight steps.Source: §3.1.2 · p. 4
• Abstract tasks vary the abstraction of explicit hazards across four related instructions, while long-horizon tasks test implicit risk through a risky subtask and a required safety condition.The dataset includes 100 abstract tasks and 50 long-horizon tasks.Source: §3.1.2 · p. 5

3.2 TASK GENERATION

• GPT-4 generates instructions and goal conditions from scene objects, supported actions, and task-specific context, using hazardous categories or corresponding hazardous tasks as references.The paper reports using GPT-4 to generate diverse, executable tasks; task formats are not generally fixed.Source: §3.2 · p. 5

3.3 FILTERING AND ANNOTATION

• Generated tasks are checked for executability, filtered for semantic similarity, and manually reviewed; annotators also provide ground-truth executable plans for detailed tasks.The process uses GPT-4 for the first executability check and an OpenAI embedding model for similarity filtering.Source: §3.3 · p. 5

4.1 EMBODIED ENVIRONMENT

• SafeAgentEnv maps 17 high-level actions to executable low-level API sequences, with feasibility checks and support for multiple agents in domestic scenes.It supplies object types, available skills, an egocentric RGB frame, and visible-object properties while withholding privileged scene information.Source: §4.1 · pp. 5–6

4.2 EVALUATION METHODS

• Execution evaluation checks whether simulator goal conditions are met, while semantic evaluation uses GPT-4 to assess whether a plan is likely to complete the task.Semantic evaluation addresses tasks with multiple valid outcomes and simulator limitations such as the absence of a ‘wet’ state.Source: §4.2 · p. 6

4.3 EMBODIED LLM AGENT BASELINES

• The benchmark tests nine planning baselines and reports their GPT-4-powered performance across detailed, abstract, and long-horizon hazardous tasks.The baselines are also assessed with Gemini-2.5-pro, Llama3-8B, Qwen2-7B, and DeepSeek-V2.5 to study backbone effects.Source: §4.3 · p. 6, Table 2 · p. 6
• Table 3 reports rejection and risk rates across abstract hazard levels L1–L4, while Figure 4 breaks down safe- and hazardous-task performance for GPT-4 baselines.Figure 4 indicates that proactive defense is lower than planning failures.Source: Table 3 · p. 7, Figure 4 · p. 7

5.1 DETAILED TASKS REVEAL WEAK PROACTIVE SAFETY

• Only 10% is the highest rejection rate among nine GPT-4-powered baselines on detailed hazardous tasks, and most agents have a 30% risk rate, reaching 69% for MLDT.Rejection and deliberate failure remain below 23% and are lower than planning failures, which are the primary cause of low risk rates.Source: §5.1 · pp. 7–8

5.2 ABSTRACT TASKS SHOW LEVEL-DEPENDENT SAFETY AWARENESS

• As abstraction increases, reasoning-based agents reject more hazardous instructions, but six baselines show a slight risk-rate reversal at Level 4 versus Level 3.The authors suggest the wider planning space at the highest abstraction level may facilitate simple hazardous plans; ReAct has a 41% risk rate on Level 4 tasks.Source: §5.2 · p. 8

5.3 LONG-HORIZON TASKS REMAIN CHALLENGING WITH IMPLICIT RISKS

• KARMA safely completes 70% of long-horizon tasks, while five baselines have incomplete rates above 50%.The authors attribute KARMA’s performance to its memory module and characterize planning and safety awareness on implicit long-horizon risks as weak.Source: §5.3 · p. 9, Figure 6 · p. 9

5.4 VARIOUS LLMS ALL EXHIBIT POOR SAFETY AWARENESS

• Safety awareness remains low across LLMs, and agent architecture has a greater impact on safety than backbone choice; long-horizon performance variance across LLMs is below 13%.Rejection rises as hazards become more abstract, while planning ability degrades; safe-task performance follows GPT-4 > Gemini-2.5-pro > Llama3-8B > Qwen2-7B.Source: Figure 5 · p. 8, §5.4 · p. 9

5.5 SEMANTIC EVALUATION PROVES RELIABLE

• GPT-4 semantic evaluations agree with human ratings at 91.89%, 90.36%, and 90.70% across the three task types, based on 1,008 human ratings.The user study sampled baseline outputs in a 3:2:2 ratio across task types.Source: §5.5 · p. 9

5.6 PRELIMINARY DEFENSES REMAIN INEFFECTIVE

• Combining ReAct reasoning, Lota-Bench skills, and MAP scene descriptions improves long-horizon performance by 34%, while a GPT-4 Chain-of-Thought filter over-rejects safe tasks and scarcely helps long-horizon cases.The hybrid approach has little effect on rejection rates for explicit hazards; the authors conclude current defenses can compromise planning effectiveness or fail to generalize.Source: §5.6 · pp. 9–10

• SafeAgentBench is presented as the first safety-aware benchmark for embodied LLM task planning in simulation, covering explicit and implicit hazards.Source: §6 · p. 10
• State-of-the-art LLM-powered agents struggle to reject hazardous tasks.Source: §6 · p. 10
• Upgrading the LLM improves planning but has minimal impact on safety awareness; the authors call for safety integration in architecture design and LLM training.Source: §6 · p. 10

• Detailed tasks span 10 hazards, including fire, electrical shock, explosion, poisoning, slips, spills, breakage, appliance misuse, furniture damage, and small-item damage.Source: A TASK EXAMPLES · p. 13
• Detailed tasks pair unsafe instructions with safe alternatives, whereas abstract and long-horizon tasks provide only the instruction.Source: Table 5 · p. 13
• A detailed task has a unique execution plan, illustrated by heating a cellphone on a stove burner.Source: A TASK EXAMPLES · p. 13, Figure 7 · p. 14
• An abstract instruction can prompt a plan to break a vase, while a long-horizon task tests step ordering and safety awareness, such as not heating kettle water for more than two steps.Source: A TASK EXAMPLES · p. 13, Figure 8 · p. 14, Figure 9 · p. 14

• Prompts for each task type include task requirements and rules.Source: B DATASET GENERATION · p. 15
• The detailed-task generation prompt is shown in Figure 10.Source: Figure 10 · p. 15
• Generation prompts for abstract and long-horizon tasks are presented in Tables 6 and 7, respectively.Source: Table 6 · p. 15, Table 7 · p. 16

C.1 HYPERPARAMETERS AND COMPUTATIONAL RESOURCES

• No model training or tunable hyperparameters are used; planning and evaluation use temperature 0 and max_tokens 4096, with most simulation experiments conducted on CPUs.Open-source LLMs are deployed with vllm on an RTX 4090 GPU.Source: §C.1 · p. 17

C.2 AVAILABLE HIGH-LEVEL ACTIONS

• SafeAgentEnv provides 17 high-level actions, including object manipulation, appliance use, cleaning, and liquid handling.Actions include pick, put, open, close, slice, turn on, turn off, drop, throw, break, pour, cook, dirty, clean, fillLiquid, emptyLiquid, and find.Source: §C.2 · p. 17

C.3 BASELINES

• Baseline designs include Lota-Bench skill selection, ReAct reasoning interleaved with actions, and LLM-Planner plans updated through physical grounding.Lota-Bench uses context learning and greedy skill search; ReAct updates plans using reasoning traces and external information.Source: §C.3 · p. 17
• CoELA combines reasoning, language comprehension, and text generation in a modular framework for planning and cooperation.The described settings include ThreeDWorld and VirtualHome.Source: §C.3 · p. 17
• ProgPrompt uses program-like prompt specifications, while MLDT decomposes tasks into goal-, task-, and action-level steps.These methods target feasible action sequences or complex long-horizon tasks, respectively.Source: §C.3 · p. 17
• MAP uses a multi-agent architecture and one environmental image to generate plans using commonsense knowledge.The benchmark compares these differing designs under the same LLM setup to investigate their effects on safety awareness and performance.Source: §C.3 · p. 17, Figure 11 · p. 19
• Planning prompts contain no explicit safety instructions, and may support either single-shot or step-by-step planning.Additional inputs such as memory are included when a baseline design provides them.Source: §C.3 · p. 17, Table 8 · p. 18

C.3.1 ADAPTATION TO SAFEAGENTENV

• Nine baselines originally used different simulators; all were migrated to AI2-THOR v5.0 with aligned agent frameworks, data formats, actions, and a unified controller.CoELA’s multi-agent system required modifications, while methods using few-shot task-plan data needed aligned action names and formats.Source: §C.3.1 · p. 18

C.3.2 PROCESSION OF VISUAL INFORMATION

• Observation handling ranges from no real-time visual input to object lists, detected objects, and VLM-generated scene analyses.MAP’s two Miner Agents produce a scene graph and a linguistic scene description; KARMA analyzes object states in images stored in short-term memory.Source: §C.3.2 · p. 18

C.4 EVALUATION METRIC IN SEMANTIC PERSPECTIVE

• GPT-4o evaluates whether tasks have been completed successfully and safely.The paper provides separate semantic-evaluation prompts for detailed, abstract, and long-horizon tasks.Source: §C.4 · p. 20, Table 9 · p. 20, Table 10 · p. 21
• For long-horizon tasks, the evaluator classifies plans as completed and safe, completed but unsafe, or incomplete.The classification categories are specified in the prompt template.Source: Table 11 · p. 22

D ADDITIONAL EXPERIMENTS ON OTHER MODELS

• Across three hazardous-task categories, the baselines show little to no proactive defense with Llama3-8B, Qwen2-7B, and DeepSeek-V2.5.Llama3-8B and DeepSeek-V2.5 have lower risk rates than GPT-4; Qwen2-7B has the lowest risk rates among the four LLMs, alongside limited planning capability.Source: Table 12 · p. 23, Table 13 · p. 23, Table 14 · p. 24
• For abstract hazards, baselines tend to refuse more often, but the tables still report hazardous-task rejection gaps and nonzero risk rates.The reported patterns differ across Llama-3, Qwen-2, and DeepSeek-V2.5.Source: Table 15 · p. 24, Table 16 · p. 24, Table 17 · p. 25
• Less than 20% of performance breakdowns for Llama-3, Qwen-2, and DeepSeek-V2.5 involve proactive defense, which is much lower than planning failures.The figures characterize these models as weaker in both planning and safety awareness than GPT-4.Source: Figure 13 · p. 25, Figure 14 · p. 25, Figure 15 · p. 26

E USER STUDY

• The user study contains 140 questions: 60 detailed, 40 abstract, and 40 long-horizon tasks.Examples are shown in Table 18.Source: E USER STUDY · p. 27, Table 18 · p. 27

G LLM USAGE STATEMENT

• The authors report using GPT-4 only to polish manuscript language; benchmark design, annotation, environment development, experiments, and analysis were conducted by the authors.The authors take responsibility for the paper and do not consider LLMs contributors or authors.Source: G LLM USAGE STATEMENT · p. 28