HomeSafe-Bench: Evaluating Vision-Language Models on Unsafe Action Detection for Embodied Agents in Household Scenarios

Jiayue Pu; Zhongxiang Sun; Zilu Zhang; Xiao Zhang; Jun Xu

Source-linked AI summary

HomeSafe-Bench: Evaluating Vision-Language Models on Unsafe Action Detection for Embodied Agents in Household Scenarios

Jiayue Pu, Zhongxiang Sun, Zilu Zhang, Xiao Zhang, Jun Xu

arXiv:2603.11975v2Mar 12, 2026cs.CVcs.AIcs.CR

TL;DR

Existing safety evaluations do not adequately measure continuous unsafe-action detection for embodied agents in household environments. The paper introduces HomeSafe-Bench and HD-Guard, combining a multidimensional video benchmark with hierarchical FastBrain–SlowBrain monitoring. HD-Guard achieves a practical latency–safety trade-off, while the analysis identifies missed visual entities, weak temporal grounding, causal-reasoning difficulties, and limited long-context memory.

Problem
Existing evaluations are largely text-based, digital, or static, and no dedicated framework evaluates VLM detection of unsafe household actions despite embodied agents’ physical-risk limitations.
Method
The paper constructs HomeSafe-Bench with 438 annotated household hazard cases and proposes HD-Guard, which coordinates continuous FastBrain screening with asynchronous SlowBrain reasoning.
Results
HD-Guard achieves a practical latency–safety trade-off for real-time applications, while analyses identify recurrent VLM bottlenecks in visual, temporal, and causal hazard understanding.
Takeaways & Limitations
The benchmark evaluates household unsafe-action detection across diverse hazards, and the hierarchical design supports real-time monitoring with deeper reasoning for uncertain or complex cases.
Takeaways & Limitations
HD-Guard lacks long-context memory because SlowBrain processes only the last two frames, causing missed hazards whose critical physical cues appear earlier.

Abstract

from arXiv · show

The rapid evolution of embodied agents has accelerated the deployment of household robots in real-world environments. However, unlike structured industrial settings, household spaces introduce unpredictable safety risks, where system limitations such as perception latency and lack of common sense knowledge can lead to dangerous errors. Current safety evaluations, often restricted to static images, text, or general hazards, fail to adequately benchmark dynamic unsafe action detection in these specific contexts. To bridge this gap, we introduce HomeSafe-Bench, a challenging benchmark designed to evaluate Vision-Language Models (VLMs) on unsafe action detection in household scenarios. HomeSafe-Bench is contrusted via a hybrid pipeline combining physical simulation with advanced video generation and features 438 diverse cases across six functional areas with fine-grained multidimensional annotations. Beyond benchmarking, we propose Hierarchical Dual-Brain Guard for Household Safety (HD-Guard), a hierarchical streaming architecture for real-time safety monitoring. HD-Guard coordinates a lightweight FastBrain for continuous high-frequency screening with an asynchronous large-scale SlowBrain for deep multimodal reasoning, effectively balancing inference efficiency with detection accuracy. Evaluations demonstrate that HD-Guard achieves a superior trade-off between latency and performance, while our analysis identifies critical bottlenecks in current VLM-based safety detection.

1 Introduction

Household embodied agents face safety risks that existing evaluations do not adequately capture, especially for continuous physical hazards. HomeSafe-Bench addresses this gap with a video benchmark, while HD-Guard provides hierarchical real-time monitoring.

Motivation: Household environments expose embodied agents to unpredictable risks, compounded by perception latency, missed detections, and limited common-sense knowledge.These limitations can produce dangerous errors such as placing metal objects in a microwave.
Motivation: Existing safety benchmarks focus on text, digital operations, or static visual inputs, leaving continuous physical risks insufficiently evaluated.The paper identifies no dedicated framework for evaluating VLM detection of unsafe household actions.
Contributions: HomeSafe-Bench contains 438 cases across six household functional areas with annotations for key frames, hazard category, severity, and reasoning difficulty.Its hybrid construction combines LLM-generated hazard causes, physical simulation, video generation, multidimensional annotation, and quality checks.
Contributions: HD-Guard combines a lightweight FastBrain for high-frequency safety screening with an asynchronous large-scale SlowBrain for complex multimodal reasoning.FastBrain classifies frame-level states as Green, Yellow, or Red and triggers SlowBrain for uncertain cases.
Contributions: The paper reports that HD-Guard achieves a practical balance between low latency and hazard-detection quality while exposing bottlenecks in current VLMs.The analyses highlight missed visual entities, weak temporal grounding, and difficulty with causal reasoning about physical hazards.

2 Related Work

Research on embodied-agent safety has expanded beyond industrial settings, but existing approaches leave continuous, agent-specific physical-risk perception insufficiently isolated and evaluated.

Foundation models: Large language and vision-language models enable embodied agents to plan from instructions and rich visual perception in unstructured environments.Examples include PaLM-E and EmbodiedGPT.
Safety evaluation: Early embodied-agent safety efforts primarily addressed text-based policy constraints or static task-planning evaluation.These approaches did not directly evaluate continuous interactive physical risks.
Safety evaluation: Interactive benchmarks explore physical safety but may couple safety perception with action planning, preventing independent evaluation of VLM safety detectors.Existing datasets also conflate general human hazards with risks specific to embodied agents.

3 Benchmark Construction

HomeSafe-Bench constructs challenging household hazard videos through a hybrid generation pipeline and evaluates them with multidimensional taxonomies, temporal supervision, and quality assurance.

Benchmark overview: HomeSafe-Bench targets unsafe-action detection and deep multimodal reasoning in diverse household scenarios.The benchmark contains 438 video sequences across six household scenarios.
Video collection: The construction pipeline combines LLM-derived hazard sources, video descriptions, physical simulation, generative video synthesis, multidimensional annotation, and quality checks.The pipeline is designed to combine physical accuracy with visual realism.
Risk taxonomy: The benchmark organizes risks by four danger categories, four severity levels, and three reasoning tiers spanning perceptual, physical, and causal reasoning.This taxonomy is paired with balanced coverage of mechanical, thermal, and environmental risks.
Temporal annotation: Each video receives temporal supervision for the hazard lifecycle, including intent onset, point-of-no-return, intervention deadline, and impact.These key frames divide each sequence into evaluation phases that reward timely warnings.
Annotation quality assurance: Benchmark validity is supported by dual annotation, agreement measures for categorical and temporal labels, and rejection of inconsistent samples.The annotation process uses Cohen’s κ, Lin’s CCC, ICC, and MAE for reliability assessment.

4 HierarchicalStreamingDual-BrainDetectorforHouseholdEmbodiedAgents Safety

HD-Guard combines continuous FastBrain screening with asynchronous SlowBrain reasoning to detect household hazards in real time. Its hierarchical priority mechanism preserves rapid reactions while adding deeper contextual analysis for ambiguous cases.

HD-Guard combines high-frequency visual perception with multimodal reasoning for real-time unsafe-behavior detection.The architecture is designed to balance rapid reflex-like responses with deep contextual understanding.
The safety policy H coordinates a lightweight FastBrain and a large-scale SlowBrain to output either nominal operation or an intervention.The policy maps video input V_t to C_t ∈ {0, 1}, where 1 triggers intervention.
FastBrain: Real-Time Streaming and Filtering: FastBrain classifies frames as Green, Yellow, or Red, using 1 FPS for Green and 5 FPS for Yellow or Red states.Green conserves resources, while Yellow and Red increase monitoring; the 5 FPS high rate is supported by an ablation study.
SlowBrain: Deep Multimodal Reasoning: Yellow states trigger SlowBrain analysis of a temporal window using structured Chain-of-Thought reasoning and physical common sense.The SlowBrain is intended to resolve complex, long-tail hazards requiring spatial and causal analysis.
Dual-Brain Integration Strategy: During SlowBrain computation, FastBrain retains priority, immediately overriding with a safety alert if it detects a Red transition.Otherwise, the system awaits the SlowBrain’s final verdict, with the logical OR combining the immediate override and delayed reasoning outcome.

5 Experiments

Experiments compare VLM safety detection, severity calibration, latency, and error patterns on HomeSafe-Bench. HD-Guard delivers competitive safety with lower latency by combining FastBrain screening and SlowBrain reasoning, while sampling-frequency results favor 5 fps.

Main Results: Open-source models like InternVL3.5-8B outperform leading closed-source models in overall safety and detection sensitivity.
Main Results: Top-performing models often overreact with premature warnings, creating high false alarm rates that hinder practical deployment.
Main Results: Small models can outperform larger counterparts in WSS, so increasing parameter count alone does not guarantee better safety performance.
Latency-Safety Tradeoff: HD-Guard maintains competitive safety across metrics while offering a practical latency-safety trade-off for real-time applications.Against MiniCPM-o-4.5, it achieves 24.94 safety versus 18.04 at nearly identical latency: 3.10s versus 3.07s.
Fine-Grained Error Analysis: HD-Guard achieves a 0% reasoning deficit rate on D3 tasks, reducing visual entity omissions to 0.5% for D1/D2 risks.The cited baselines are 45.6% reasoning deficit for Qwen3-VL-30B and 30.4% visual omissions, respectively.
Fine-Grained Error Analysis: HD-Guard’s false alarm rate is 25.1%, lower than GPT-5.1 at 29.9% and InternVL3.5-8B at 53.2%.
Sampling Frequency Ablation: At 5 fps, HD-Guard reaches peak WSS of 25.00, whereas 10 fps lowers WSS to 24.88 and raises the False Trigger Rate to 28.54%.The study reports an inverted-U relationship: low rates miss transient hazards, while excessive sampling adds redundant information.

B Appendix B: Data Annotation Details

HomeSafe-Bench annotations organize household hazards by reasoning difficulty, temporal progression, danger category, and severity. These criteria define how cases and key frames are labeled for evaluation.

Annotation criteria cover reasoning difficulty, temporal key frames, danger categories, and severity assessment.The supplied passages identify multidimensional annotation standards and their corresponding tables.
Reasoning Difficulty: Three reasoning-difficulty levels assess the perceptual and reasoning requirements for detecting hazards.
Temporal Key Frames: Five key frames capture each video's temporal progression, with the final frame used as Impact for D3 cases lacking an accident during the video.
Danger and Severity: The taxonomy includes two danger classes and four severity levels, L1–L4, based on NEISS standards and economic restoration costs.

B.4.1 Inter-annotator Agreement Analysis

The agreement analysis evaluates annotation consistency across co-annotated videos and resolves disagreements through independent re-annotation and consensus labeling.

Agreement evaluation covers 412 co-annotated videos for validity and 236 videos for categorical and temporal annotations.
238 conflicting samples were independently re-annotated to produce final consensus labels.Re-annotation was triggered by categorical disagreement or temporal keyframe annotations exceeding tolerance.

C.1 Fine-Grained Error Definitions

The paper categorizes model failures across instruction following, perception, and physical reasoning. These definitions distinguish formatting failures, overreaction, spatiotemporal perception bottlenecks, and hidden-hazard reasoning deficits.

Failure categories span the execution chain from instruction following to high-order reasoning.
Instruction Following: FormatorInstructionError covers invalid structured outputs such as missing verdicts or timestamps.
Perception: Dynamic D1 and D2 errors can reflect spatiotemporal perception deficits when reasoning omits key entities.
Physical Reasoning: Physical Reasoning Deficit describes D3 cases where models identify entities but fail to anticipate hidden hazards.

C.2 Error Analysis across Danger Categories

Error analysis across danger categories identifies temporal delays in dynamic collisions, while HD-Guard maintains visual coverage across several categories. The analysis highlights category-specific cognitive biases.

12.0% lag in C4 for Qwen3-VL-8B illustrates warning delays associated with severe physical displacements in dynamic collisions.
0.0% visual omission across C1–C3 for HD-Guard indicates warnings consistently precede accidents in those categories.

C.3 Error Analysis across Severity Levels

Across hazard severity levels L1–L4, baseline models exhibit a trade-off between perceptual sensitivity and logical prediction that HD-Guard effectively resolves. Low-severity hazards expose visual omission, while the reported reliability remains below 0.7%.

L1–L4 severity analysis: 65.5% visual entity omission for Qwen3-VL-30B-Instruct and 48.0% for GPT-5.1 occur in low-severity L1 events.Low visual saliency makes subtle risk signs difficult for baseline models to capture.
L1–L4 severity analysis: Baseline performance reflects a trade-off between perceptual sensitivity and logical prediction across hazard severity levels L1–L4.The analysis identifies perceptual insensitivity in minor risks and reasoning bottlenecks in more severe events.
L1–L4 severity analysis: HD-Guard resolves the baseline trade-off across severity levels while maintaining reliability below 0.7%.The reported reliability threshold is presented as supporting safety-critical use.

D.1 Case I: Resolving Perception and Reasoning Bottlenecks

The case studies contrast baseline failures from perception and reasoning bottlenecks with HD-Guard’s dual-brain behavior. They also show that correct hazard recognition can still fail during deployment when accumulated latency exceeds the physical time-to-impact window.

Case I: Perception bottleneck: In Case 1A, Qwen3-8B-Instruct misses a chair in the robot’s path and incorrectly classifies the robot as stationary.The failure is attributed to low sampling frequency or temporal aliasing, leading to collision with and overturning of the obstacle.
Case I: Reasoning bottleneck: InternVL3.5-8B halts at 2.0s before the 3.6s intent onset, acting on an ambiguous trajectory and hallucinated risk.The premature intervention prevents valid task completion rather than confirming the hazardous action.
Dual-brain trade-offs: The case studies frame dual-brain synergy as a way to mitigate over-reaction while separating fast screening from slower reasoning.Figure 12 identifies over-reaction mitigation via dual-brain synergy, and Figure 13 highlights temporal misalignment as a distinct failure mode.
Case III: System latency: 1.56s of accumulated engineering latency delays the physical stop until 3.89s, after the t = 2.60s impact window.FastBrain identifies the liquid-spill hazard at t = 2.33s, while SlowBrain is delayed by 7.11s and gives an incorrect safety verdict.