Source-linked AI summary
ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs
Yizhan Li, Jianxin You, Mengyang Xiong, Yinhuan Chen, Zicheng Zhao, Dekun Wu, Dongqing Zhang, Bang Liu
TL;DR
ReactHuman addresses the lack of evaluations for whether MLLMs can turn intuitive physics into immediate, safety-critical action. It benchmarks frozen MLLMs as simulated-humanoid brains using reproducible physics, executed reaction plans, and five diagnostic metrics; across seven models, reactive safety remains far from solved, with fixed dispositions and physically unreachable actions persisting without scale improvement.
Problem
Existing evaluations test physical reasoning passively or embodied behavior over deliberate horizons, leaving immediate safety-critical action unmeasured.
Method
ReactHuman evaluates MLLMs in 17 event families and over 1,000 reproducible scenes, executing structured plans on simulated humanoids and scoring reasonability, safety, and physical grounding.
Results
Across seven models, reactive safety remains far from solved: models mishandle roughly one hazard in three, rely on fixed dispositions, and miss interception points even when selecting the correct action.
Takeaways & Limitations
ReactHuman provides a diagnostic, reproducible evaluation for whether embodied MLLMs can act safely within physics and localizes failures that answer-only protocols miss.
Takeaways & Limitations
The benchmark uses rigid-body simulator ground truth and open-loop execution, so its conclusions require validation on real hardware and closed-loop settings.
Abstract
from arXiv · showhide
Reacting to sudden physical hazards (catching a slipping plate, dodging a falling knife) is both a meaningful test of embodied intelligence and a hard requirement for deploying multimodal large language models (MLLMs) as the decision coreof household robots. Existing evaluations, however, probe intuitive physics passively through question answering over videos, or target deliberate, long-horizon tasks such as navigation and rearrangement; none measure whether a model can turn physical understanding into immediate, safety-critical action. We introduce ReactHuman, the first physics-grounded benchmark for human-like reactive decision-making, in which the evaluated MLLM acts as the brain of a simulated humanoid facing sudden household hazards; it spans 17 event families and over 1,000 bit-for-bit reproducible scenes with exact, annotation-free ground truth derived from 240 Hz rigid-body simulation, including adversarial objects whose appearance contradicts their physics (a foam anvil, a steel apple). We further design a five-metric suite that scores each reaction along three axes: reasonable, safe, and physically grounded. We physically execute every committed plan so that decisions have observable consequences. With this harness we evaluate seven representative MLLMs. Results show that reactive safety is far from solved: models mishandle roughly one hazard in three, act from fixed dispositions rather than the observed scene, trust appearance over motion, and miss interception points at meter scale even when the chosen action is correct; none of these failures shrink with model scale. ReactHuman thus offers both a fine-grained diagnosis and a scalable training signal toward physically grounded, safety-aware embodied agents. The benchmark can be found here: https://huggingface.co/datasets/Alan123/reacthuman-benchmark-scaled
1 Introduction
ReactHuman targets the missing ability to convert intuitive physics into immediate, safe, reasonable, physically grounded action. It introduces a reproducible benchmark that executes each MLLM reaction in simulation and diagnoses distinct failure modes.
- ReactHuman evaluates whether MLLMs can turn physical understanding into immediate, safety-critical action rather than passive judgments or long-horizon behavior.
- The benchmark makes an MLLM the decision-making brain of a simulated humanoid that physically executes each structured reaction plan.
- Its 17 event families isolate different links in perception, reasoning, and action, including semantic choice, interception, causal propagation, and appearance–physics conflicts.
- A five-metric suite separates reasonable action, safety, and physical grounding, distinguishing unreasonable, unsafe, and kinematically ungrounded reactions.
- The benchmark covers over 1,000 reproducible scenes with synchronized views, simulator-derived ground truth, adversarial objects, and executed decisions.
2 Related Work
Prior benchmarks separately test passive physical reasoning, deliberate embodied tasks, or control, but do not evaluate immediate reactive safety. ReactHuman combines a frozen MLLM, whole-body controller, humanoid, and deterministic simulation to test that decision layer.
- Existing physical-reasoning benchmarks mainly use forward prediction, causal reasoning, plausibility judgments, or multiple-choice video question answering.
- ReactHuman differs by making the model output an action with safety consequences while generating exact ground truth automatically through simulation.
- Embodied-AI benchmarks emphasize navigation, manipulation, rearrangement, situated question answering, or whole-body control on deliberate time scales.
- Its harness couples a frozen MLLM to a pre-trained whole-body controller and simulated humanoid without the action fine-tuning that defines VLA policies.
- The scene-generation pipeline combines LLM semantic planning with a deterministic seeded randomizer that retains control of physical parameters and reproducibility.
3 The ReactHuman Benchmark
ReactHuman freezes a simulated hazard before resolution, asks an MLLM for a structured motor plan, and executes that plan on a humanoid in the same physics scene. Deterministic semantic planning, seeded physics, and simulator labeling provide reproducible, exact evaluation across varied event families.
- 3.1 Freeze-and-Predict Protocol: At a frozen observation, the MLLM outputs a structured plan containing intent, confidence, walking command, and predicted hand-trajectory keyframes.
- 3.1 Freeze-and-Predict Protocol: A pre-trained whole-body controller converts the plan into humanoid motion and torques, making catches, misses, dodges, and impacts observable physical outcomes.
- 3.2 Action Space: The action space is recovered from motor commands and classified into Execute_Catch, Trigger_Dodge, or No_Action rather than selected from a menu.
- 3.3 Event Families: The 17 event families span distinct dynamics and decision structures, from simple semantic drops to time-varying interception and causal chain reactions.
- 3.3 Event Families: Fourteen adversarial asset variants contradict visual appearance through unusual physical parameters, requiring action selection from observed dynamics.
- 3.4 Scene Generation and Ground Truth: A frozen LLM supplies semantic scene choices, while seeded randomization controls physical and visual parameters and Genesis produces 240 Hz simulator-derived labels.
4 Metric Suite
ReactHuman scores each reaction across action choice, safety, intent, and physical hand placement rather than relying on a single label. Its metrics expose whether a model understood the hazard, chose a permissible response, and physically reached the relevant location.
- The suite uses five complementary metrics to separate correct action choice, safety, intent understanding, endpoint placement, and hand-path movement.
- Semantic Action Accuracy checks whether the recovered action matches ground truth, but a perfect score can still coexist with a safety violation.
- Safety Validity independently flags malformed outputs, dangerous catches, inaction during danger, and catch-or-inaction when Dodge is required.
- Physical Endpoint Distance measures the meter-scale gap between the final hand position and the simulator’s true impact point.
- Action–Intent Alignment distinguishes misunderstanding from a correct idea expressed through a wrong motor plan, while using keyword matching for intent.
- Hand-Distance Evolution measures the closest approach and pathwise closing toward the impact point, revealing trajectories that endpoint distance can hide.
5 Experiments
Across 2,138 decisions on 306 balanced scenes, seven MLLMs show persistent safety, adaptation, physics-grounding, and execution failures. Performance varies by hazard structure, but model scale and voting do not reliably resolve these failures.
- 5.2 Results and Analysis: 35.9% of Dodge decisions violate safety rules, with freezing accounting for 253 of 392 mistakes.Violations occur for every model, spanning 23–66%; freezing is the dominant error.
- 5.2 Results and Analysis: Six of seven models favor retreat, Claude instead catches 67% but violates safety on 34% of Dodge scenes.The empty strong-on-both-sides region indicates fixed model dispositions rather than scene-adaptive action choice.
- 5.2 Results and Analysis: Catch accuracy falls from 66.7% for one bouncing ball to 38.9% for a collapsing can stack, while Dodge accuracy falls from 91.3% to 54.8% as warning time shrinks.Nearly half of thrown-object decisions, 44%, violate a safety rule.
- 5.2 Results and Analysis: Correct Catch choices still miss the interception point by a median 0.48 m, with 89% of misses falling short because models do not step forward.In 49% of these decisions, the robot stands more than 0.8 m away, beyond arm reach.
- 5.2 Results and Analysis: At slow speeds, models freeze 42% versus roughly 21% at faster bands, while accuracy drops to 38.1% SAA.Action distributions barely change from normal to very fast, and all four-speed answers match in only 27% of scene groups.
- 5.2 Results and Analysis: 62.1% majority-vote accuracy barely exceeds the 54.0% average and does not surpass the 63.7% best single model.Unanimous agreement occurs on only 16% of scenes and is correct 80% of the time.
- 5.2 Results and Analysis: Models trust appearance over observed physics, never questioning adversarial materials across 280 decisions.Foam ceiling panels are dodged in 49 of 56 decisions, while steel cans are grasped as often as light cans, 39% versus 39%.
6 Discussion and Limitations
ReactHuman identifies recurring reactive failures, including scene-insensitive dispositions and missed physical consequences, while noting that its conclusions remain bounded by simulation and open-loop execution.
- Models freeze, apply fixed dispositions, and ignore motion or appearance–physics conflicts; these failures do not improve with scale.The paper links each failure to a distinct proposed mitigation, including safety-constrained decoding, paired catch-and-dodge training, and motion-grounded supervision.
- Ground truth comes from rigid-body simulation, so deformation and shattering are omitted and conclusions require validation on real hardware.
- The evaluation is open-loop: one frozen observation produces one committed plan, and execution is not re-planned mid-motion.Closed-loop repeated observation–action cycles are identified as a natural extension.
7 Conclusion
ReactHuman reframes physical-reasoning evaluation as reactive, safety-critical decision-making with reproducible simulation and diagnostic coverage. Across seven models, reactive safety remains far from solved, with fixed dispositions and no improvement with scale.
- ReactHuman measures whether embodied MLLMs can act safely within physics and localizes failures through 17 event families, exact ground truth, and five diagnostic metrics.Scenes are bit-for-bit reproducible, and the seed-deterministic pipeline extends the benchmark without annotation cost.