PaperlayerView exact revision on arXiv

Source-linked AI summary

Can AI Perceive Physical Danger and Intervene?

Abhishek Jindal, Dmitry Kalashnikov, R. Alex Hofer, Oscar Chang, Divya Garikapati, Anirudha Majumdar, Pierre Sermanet, Vikas Sindhwani

arXiv:2509.21651v2cs.AI

TL;DR

Physical AI can cause direct harm, but evidence on how well foundation models perceive and reason about physical danger remains limited. This paper builds a multimodal benchmark from injury reports and operational constraints, evaluates frontier models, and tests thinking-trace post-training. No evaluated model achieves less than 30% constraint violation rate in joint reasoning about embodiment limits, physics, and visual cues, while thinking post-training reaches state-of-the-art safety constraint satisfaction.

  • Problem

    Physical-safety evaluation remains limited in text-only and digital-action settings, while waiting for physical-AI accidents to provide evidence is data-scarce and carries a human cost.

    Source: §1 · pp. 1–2

  • Method

    The paper builds a continuously evolving benchmark from real-world injury reports and operational constraints, then trains models with reasoning traces for embodiment-specific constraint tasks.

    Source: §1 · p. 2, §2 · p. 3, §4 · p. 8

  • Results

    No model achieves less than 30% constraint violation rate when jointly reasoning about embodiment limitations, physics, and visual cues; thinking post-training achieves state-of-the-art safety constraint satisfaction.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The paper frames modality, embodiment-adherence, and latency–accuracy gaps as areas to close for meeting rigorous safety standards.

    Source: §6 · p. 10

    The gripper example is bounded by a specified embodiment constraint: a simple clamp cannot lift objects that rotate out because of an offset center of mass.

    Source: §1 · p. 2

Abstract

from arXiv · show

When AI interacts with the physical world -- as a robot or an assistive agent -- new safety challenges emerge beyond those of purely ``digital AI". In such interactions, the potential for physical harm is direct and immediate. How well do state-of-the-art foundation models understand common-sense facts about physical safety, e.g. that a box may be too heavy to lift, or that a hot cup of coffee should not be handed to a child? In this paper, our contributions are three-fold: first, we develop a highly scalable approach to continuous physical safety benchmarking of Embodied AI systems, grounded in real-world injury narratives and operational safety constraints. To probe multi-modal safety understanding, we turn these narratives and constraints into photorealistic images and videos capturing transitions from safe to unsafe states, using advanced generative models. Secondly, we comprehensively analyze the ability of major foundation models to perceive risks, reason about safety, and trigger interventions; this yields multi-faceted insights into their deployment readiness for safety-critical agentic applications. Finally, we develop a post-training paradigm to teach models to explicitly reason about embodiment-specific safety constraints provided through system instructions. The resulting models generate thinking traces that make safety reasoning interpretable and transparent, achieving state of the art performance in constraint satisfaction evaluations. The benchmark is released at https://asimov-benchmark.github.io/v2

1. Introduction

The paper argues that physical AI can directly harm people, while existing safety evaluations often focus on text or digital actions and accident-based evidence is costly and scarce. It introduces ASIMOV-2.0, assesses frontier models, and explores safety post-training.

1. Introduction

  • Physical-safety evaluations remain limited across text-only and digital-action settings, while waiting for real accidents is data-scarce and carries a human cost.Physical AI safety involves perceiving danger, reasoning about it, and taking timely actions to avoid harm.Source: §1 · pp. 1–2

Frontier Physical Safety Benchmark

  • ASIMOV-2.0 tests embodied agents’ injury-risk and severity estimates, physical-constraint reasoning, and proactive responses to potential accidents.Its grounding samples long-tail scenarios near the edge of safety, while an automated generative pipeline supports scalability.Source: §1 · p. 2

Evaluation of Frontier Models

  • No model achieves less than 30% constraint violation rate when reasoning jointly about embodiment limitations, physics, and visual cues.The evaluation also identifies lower performance on action and image/video modalities than on text.Source: §1 · p. 2
  • Smaller, faster models suited to on-device deployment generally score lower than their larger counterparts.The paper describes this as a sizeable latency gap.Source: §1 · p. 3

Value of Safety Thinking and Post-training

  • Supervised fine-tuning and RL post-training on thinking-trace data lead to models that outperform all frontier models on safety understanding tasks.The training data teaches models to reason over embodiment-specific physical constraints before answering.Source: §1 · p. 3

2. ASIMOV-2.0: A New Physical Safety Benchmark

ASIMOV-2.0 grounds physical-safety scenarios in injury narratives and operational constraints, using a generative pipeline and human review to build evaluations across text, video, and images. It expands earlier benchmark coverage with long-tail scenarios, video understanding, and constraint-satisfaction tasks.

Benchmark motivation

  • The benchmark samples long-tail injury scenarios for embodiment-agnostic text, image, and video evaluation, and also tests embodiment-specific operational constraints.Embodiment prompts can include system instructions such as a robot’s lifting limit.Source: §2 · p. 3

Benchmark Generation

  • A Generator synthesizes scenarios and candidate actions from real-world sources, while an optional Critic gives feedback for refinement and multimedia prompts turn text scenarios into images or videos.The implementation uses Gemini-2.5 Pro, Imagen, and VEO3, with human labels for the resulting safety questions.Source: §2 · p. 3

Real-world Grounding

  • The benchmark draws injury narratives from NEISS and builds an operational safety taxonomy inspired by industrial-robotics standards, including ISO 10218-1:2025 and ISO/TS 15066:2016.The taxonomy contains representative instructions for testing compliance with real-world physical constraints.Source: §2 · pp. 3–4

Benchmark Components

  • ASIMOV-2.0 upgrades ASIMOV-1.0 with a Generator-Critic-Refine loop, improved data quality and long-tail coverage, and new video-understanding and constraint-satisfaction evaluations.The authors report higher-quality human annotations and careful data rebalancing.Source: §2 · p. 4
  • The injury component uses hospital reports to create text scenarios, while its video component uses 5-to-10-second clips of safe-to-unsafe transitions and human-rated safety questions.The video scenarios are grounded in NEISS and filtered for rater consensus, timing agreement, and quality issues.Source: §2 · pp. 4–5
  • The image-constraint component combines embodiment or task limits with scenes of objects’ physical properties and user-requested actions, retaining pairs with at least 80% consensus that a violating object is present.The filtered set contains 164 constraint-image pairs with annotated boxes for violating and non-violating objects.Source: §2 · pp. 5–6

3. Evaluations and Insights

Evaluations of frontier models reveal uneven physical-safety performance across risk perception, action assessment, video, and physical-constraint tasks. Strong text-based risk recognition contrasts with weaker action and video performance and substantial constraint violations.

Accuracy in Perceiving Risks

  • 92.3% average latent-risk accuracy and 88.7% average high-severity accuracy were achieved by GPT-5, Gemini 2.5 Pro, and Claude Opus 4.1.GPT-5-Mini and GPT-5-Nano were respectively 20% and 5% lower in latent-risk accuracy, and 19% and 17% lower in high-severity accuracy than GPT-5.Source: Figure 4 · p. 6, §3 · p. 7

Recognizing Safety Risks in Videos

  • Video risk-recognition accuracy is lower than text-scenario accuracy, with gaps of 27% for Claude Opus 4.1 and 40% for GPT-5.Gemini 2.5 Pro shows a more modest drop.Source: Figure 5 · p. 6, §3 · p. 7

Triggering Safety Interventions

  • Gemini 2.5 Pro predicts the last intervention timestamp within 0.75 seconds of ground truth on average, and within 0.5 seconds in 56% of videos.Across models, intervention metrics show high variance; Gemini 2.5 Pro tends to be more proactive than GPT-5.Source: Figure 5 · p. 6, §3 · p. 7

Action Safety

  • Top models score 74% on action-safety accuracy and 66% on post-action risk assessment, both generally below scenario-only risk and severity accuracy.Smaller models are substantially weaker on these action metrics, with Gemini 2.5 Flash an exception.Source: Figure 4 · p. 6, §3 · p. 7

Adherence to Physical Constraints

  • Constraint violation rates range from 75% to 38.6%, with the highest errors in gripper geometry and type.The authors suggest models may be underexposed to embodiment-specific hardware limitations.Source: §3 · p. 7

4. “Thinking" for Safety

The section tests whether inference-time reasoning and post-training improve safety-constraint adherence in image-based pointing tasks. More thinking generally helps, while a small supervised and reinforcement-learning post-training dataset achieves the lowest violation rate among major models with thinking enabled.

Post Training for Safety Thinking via SFT and RL

  • The post-training data uses three-step traces: enumerate image objects, label constraint satisfaction, then point to a non-violating object; SFT and RL train these behaviors.RL adds a penalty when the predicted point falls within a violating object's bounding box.Source: §4 · pp. 7–8

Impact of Thinking Effort

  • Increasing inference-time computation significantly improves performance, especially for smaller models, but GPT violation rates do not consistently decrease from medium to high effort.The study varies thought-token budgets for Claude, reasoning-effort settings for GPT, and thinking instructions for Gemini.Source: §4 · pp. 7–8, Figure 7 · p. 9

Post-training Results

  • With 200 image-constraint training pairs, post-training achieves the lowest violation rate among major models with thinking enabled.Average thought length falls by 50%, while pointing accuracy is statistically insignificantly affected.Source: §4 · p. 8, Figure 8 · p. 9

Thinking Trace Example

  • A constraint-violating baseline example is handled without violation by the fine-tuned model, whose thoughts follow the structure induced during post-training.The paper presents this as an example of the trained reasoning structure in use.Source: §4 · pp. 8–9

5. Related Work

The related work distinguishes physical safety for embodied agents from digital AI safety, and reviews multimodal evaluation and reasoning as relevant but still under-explored dimensions. This paper situates its benchmark in that gap by grounding physical-safety evaluation in real-world scenarios and constraints.

Physical Safety for Embodied AI

  • Physical safety for embodied agents is distinct from AI safety focused predominantly on digital harms.The related-work passage frames ensuring embodied agents’ physical safety as a critical challenge.Source: §5 · p. 9
  • Unlike Liu et al. (2024b), this work grounds scenarios in human injury reports and industrial standards and adds video to assess temporal hazard understanding.Liu et al. (2024b) grounded scenarios in COCO (Lin et al., 2014).Source: §5 · p. 10

Multi-modal Content Safety

  • Multimodal content-safety benchmarks address social and digital harms, but generally do not evaluate physical interaction risks.Examples include offensive-content detection in image-text pairs (Zhou et al., 2024) and pairs designed to prevent text-only safety leakage (Hu et al., 2024).Source: §5 · p. 10

Reasoning and Safety

  • Prior findings disagree on reasoning’s relationship to safety, while this benchmark examines that relationship with multimodal data, physical constraints, and grounded situations.Guan et al. (2024) report increased robustness and fewer over-refusals, whereas Huang et al. (2025a) report a trade-off; Mei et al. (2025) find deeper reasoning can increase overconfidence in incorrect answers.Source: §5 · p. 10

6. Conclusion

The paper introduces ASIMOV-2.0 for physical-safety evaluation across modalities and safety-reasoning tasks, evaluates frontier models, and reports state-of-the-art constraint-satisfaction results after thinking post-training. Its evaluation highlights modality, embodiment-adherence, and latency–accuracy gaps relevant to rigorous safety standards.

6. Conclusion

  • Thinking post-training achieves state-of-the-art safety constraint-satisfaction results on the benchmarks.The paper introduces ASIMOV-2.0 across multiple modalities and safety-reasoning tasks, then evaluates frontier AI models on it.Source: §6 · p. 10
  • The paper identifies modality and embodiment-constraint adherence gaps, as well as a latency–accuracy trade-off in which smaller models or inference budgets typically perform worse.Source: §6 · p. 10
  • The authors state that closing these gaps will enable AI systems to meet safety standards such as IEC 61508 and ISO 13849-1.The standards classify necessary safety-function reliability into Safety Integrity Levels or Performance Levels, respectively.Source: §6 · p. 10

A. Examples of Thinking Traces before and after finetuning

The appendix illustrates a physical-constraint prompt and contrasting reasoning traces for a hot, smoking rope. The examples show constraint checking followed by either identifying a hazard or selecting and localizing objects that meet the constraint.

A. Examples of Thinking Traces before and after finetuning

B. Proactive vs Reactive Interventions

Figure 10 compares proactive and reactive interventions by Gemini 2.5 Pro and GPT 5.

B. Proactive vs Reactive Interventions

  • Figure 10 compares proactive versus reactive interventions for Gemini 2.5 Pro and GPT 5.Source: Figure 10 · p. 16

C. NEISS injury types and physical constraint taxonomy

The section presents a distribution of injury types in the dataset and a taxonomy of operational safety constraints.

  • Figure 11 presents the distribution of injury types in the dataset.Source: Figure 11 · p. 16
  • Figure 12 presents a taxonomy of operational safety constraints.Source: Figure 12 · p. 17

D. Eval Datasets - descriptive statistics

The section describes dataset statistics and reports a Point Bench comparison of baseline and safety-finetuned Gemini ER 1.5.

Related papers

Loading 2509.21651v2…