PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey: Learning Embodied Intelligence from Physical Simulators and World Models

Xiaoxiao Long, Qingrui Zhao, Kaiwen Zhang, Zihao Zhang, Dingrui Wang, Yumeng Liu, Zhengjie Shu, Yi Lu, Shouzheng Wang, Xinzhe Wei, Wei Li, Wei Yin, Yao Yao, Jia Pan, Qiu Shen, Ruigang Yang, Xun Cao, Qionghai Dai

arXiv:2507.00917v3cs.RO

TL;DR

Embodied AI must ground abstract reasoning in real-world perception and action. This survey synthesizes how physical simulators and world models jointly support embodied learning, control, planning, and decision-making, concluding that their integration supports more sample-efficient learning, long-horizon planning, and robust generalization across diverse environments.

  • Problem

    Embodied intelligence requires grounding abstract reasoning in real-world understanding and action, while robot data collection remains costly, safety-sensitive, and difficult to control and repeat.

  • Method

    The survey reviews robot learning, physical simulators, and world models while proposing an IR-L0 to IR-L4 capability framework for intelligent robots.

  • Results

    The survey concludes that integrating physical simulators and world models supports sample-efficient learning, long-horizon planning, and robust generalization across diverse environments.

  • Takeaways & Limitations

    The integration of external simulation and internal world modeling is presented as a foundation for advancing embodied intelligence toward more general-purpose autonomous robots.

  • Takeaways & Limitations

    World models face unresolved challenges in long-term memory, interpretability, and formal safety verification.

Abstract

from arXiv · show

The pursuit of artificial general intelligence (AGI) has placed embodied intelligence at the forefront of robotics research. Embodied intelligence focuses on agents capable of perceiving, reasoning, and acting within the physical world. Achieving robust embodied intelligence requires not only advanced perception and control, but also the ability to ground abstract cognition in real-world interactions. Two foundational technologies, physical simulators and world models, have emerged as critical enablers in this quest. Physical simulators provide controlled, high-fidelity environments for training and evaluating robotic agents, allowing safe and efficient development of complex behaviors. In contrast, world models empower robots with internal representations of their surroundings, enabling predictive planning and adaptive decision-making beyond direct sensory input. This survey systematically reviews recent advances in learning embodied AI through the integration of physical simulators and world models. We analyze their complementary roles in enhancing autonomy, adaptability, and generalization in intelligent robots, and discuss the interplay between external simulation and internal modeling in bridging the gap between simulated training and real-world deployment. By synthesizing current progress and identifying open challenges, this survey aims to provide a comprehensive perspective on the path toward more capable and generalizable embodied AI systems. We also maintain an active repository that contains up-to-date literature and open-source projects at https://github.com/NJU3DV-LoongGroup/Embodied-World-Models-Survey.

1 INTRODUCTION

Embodied intelligence grounds perception, reasoning, and action in physical interaction, addressing how abstract reasoning can connect to real-world understanding and action. This survey examines how physical simulators and world models complement each other in developing more autonomous, adaptable, and generalizable robots.

  • Motivation: Embodied intelligence connects perception, action, and cognition through physical interaction with the environment.Robots can continuously adjust behavior and cognition from physical-world feedback.
  • Physical Simulators: Physical simulators provide explicit, controlled environments for training, testing, and refining robot behaviors before real-world deployment.They support prediction and fine-tuning while reducing the costs and risks of physical experimentation.
  • Contributions: The paper proposes a five-level intelligent-robot grading model spanning IR-L0 to IR-L4 across autonomy, task handling, environmental adaptability, and societal cognition.The framework is intended to assess and guide robot development across technological evolution.
  • World Models: World models provide internal environmental representations that support autonomous simulation, prediction, and action planning.The survey treats them as internal complements to external simulation for adaptive decision-making.
  • Survey Focus: The survey analyzes the complementary relationship between simulators and world models in advancing embodied intelligence.It focuses on how structured external training and internal modeling jointly support autonomy, adaptability, task performance, and generalization.
  • Contributions: The survey reviews robot learning, mainstream physical simulators, and world-model architectures and applications.Coverage includes locomotion, manipulation, interaction, simulation capabilities, rendering, sensors, controllable simulation, dynamics, and reward modeling.

2 LEVELS OF INTELLIGENT ROBOT

The survey organizes intelligent robots into five progressive levels, from program-driven execution to fully autonomous social intelligence. These levels are evaluated through autonomy, task handling, environmental adaptability, and societal cognition, with progressively richer sensing, control, interaction, and learning capabilities.

  • Level Framework: The proposed grading system combines intelligent cognition and autonomous behavior across five progressive robot levels.Its core factors include autonomy, task complexity, environmental performance, and social interaction or cognition.
  • IR-L0: IR-L0 robots perform repetitive deterministic tasks through predefined programs or teleoperation with minimal perception and no real-time feedback loops.Typical systems use high-precision servomotors, rigid structures, PLC or MCU controllers, and limited sensors.
  • IR-L1: IR-L1 robots execute predefined task sequences using basic sensors and rule-based or finite-state control in closed-task environments.They cannot process complex or unforeseen events and demonstrate stability only under clearly defined rules.
  • IR-L2: IR-L2 robots respond to environmental changes, switch between task modes, and perform basic multimodal interaction while retaining essential human supervision.They combine multimodal sensing, mapping, path planning, obstacle avoidance, and modular task execution.
  • IR-L3: IR-L3 robots make autonomous decisions in complex dynamic environments, infer user intent, adapt behavior, and support sophisticated multimodal interaction within ethical constraints.Their capabilities include multimodal fusion, affective computing, adaptive policy optimization, planning, dialogue, and embedded ethical governance.
  • IR-L4: IR-L4 robots are described as fully autonomous systems capable of perception, decision-making, execution, social interaction, and long-term adaptive learning without human intervention.The level includes open-ended task handling, advanced cognition, empathy, and self-evolving ethical reasoning.

3 ROBOTIC MOBILITY, DEXTERITY AND INTERAC-

Humanoid robots are presented as physical embodiments of embodied intelligence because their human-like form supports integration into human-centered environments. The section reviews technical progress in whole-body control, locomotion, manipulation, and natural human-robot interaction.

  • Humanoid Robotics: Humanoid robots provide physical representations of embodied intelligence and can integrate into human-centered environments.Their human-like appearance supports meaningful assistance in those settings.
  • Research Directions: Recent machine-learning advances have produced breakthroughs in robot whole-body control and general manipulation.The chapter surveys fundamental methods, locomotion and manipulation developments, and natural human-robot interaction.

3.1 Related Robotic Techniques

This section reviews control and learning techniques for humanoid robots, spanning model-based optimization, whole-body coordination, reinforcement learning, imitation learning, and visual-language-action models. It also outlines their capabilities, applications, and limitations in producing adaptive robotic behavior.

  • Model Predictive Control: Model Predictive Control predicts future system behavior with a dynamic model and repeatedly optimizes control actions while explicitly handling input and state constraints.Its formulation suits complex, high-dimensional humanoid systems.
  • Whole-Body Control: Whole-body control coordinates all joints and limbs by translating prioritized motion and force objectives into constraints and optimization problems.Typical objectives include balance, trajectory following, and applying forces with the hand.
  • Reinforcement Learning: Reinforcement learning enables humanoid robots to discover actions through environmental interaction and reward or penalty feedback instead of explicit behavior programming.This makes RL suitable for high-dimensional, dynamic, and uncertain environments.
  • Imitation Learning: Imitation learning acquires robotic skills by observing demonstrations, avoiding explicit programming or handcrafted reward functions.Human motion capture and model-based reference trajectories can guide natural and stable motion in simulation, but demonstrations are costly and may limit generalization.
  • Visual-Language-Action Models, VLA: Visual-language-action models integrate visual perception, language understanding, and action generation to map natural-language instructions to physical robot actions.RT-2 introduced this paradigm to robot control using language-like action tokens, while later models still face generalization, real-time inference, grounding, bias, and computational challenges.

3.2 Robotic Locomotion

Robotic locomotion research addresses stable adaptation to unstructured terrain and agile high-dynamic movements, using model-based, optimization-based, and learning-based control. Recent systems extend locomotion toward outdoor traversal, jumping, whole-body motion, and fall recovery across varied environments.

  • Locomotion Tasks: Locomotion research separates unstructured-environment adaptation from high-dynamic movements such as running and jumping.The former emphasizes stable walking over unpredictable terrain, while the latter balances stability and agility during rapid motion.
  • Unstructured Environment Adaptation: Position-controlled robots showed limited terrain adaptability because high-impedance joints could be damaged by large impacts.Force-controlled joints with lower gear ratios improve compliance and smoother responses under impact.
  • Unstructured Environment Adaptation: Whole-body and trajectory-based controllers enabled stable walking across varied terrains, including soft gym mats and complex wave-field terrain.These approaches combined center-of-mass planning, whole-body control, path planning, and high-frequency reactive feedback.
  • Unstructured Environment Adaptation: Learning-based methods enabled outdoor locomotion, stair traversal, hurdle crossing, and jumps across gaps up to 0.8 meters wide.Domain randomization, exteroceptive sensing, height maps, perceptive internal models, and end-to-end policies supported transfer and mobility in diverse terrains.
  • High Dynamic Movements: Simplified dynamic models and MPC supported increasingly dynamic jumping, including an 18 cm jump, jumps up to 0.5 meters, and continuous jumping motions.The approaches used SLIP, LIPM, SRBM, and centroidal dynamics to reduce computational demands or stabilize motion.
  • High Dynamic Movements: Reinforcement learning and imitation learning broadened humanoid movement capabilities to running, jumping, parkour, and expressive whole-body behaviors.Imitation methods used human motion datasets and adversarial motion priors to improve naturalness and dynamic performance.
  • Fall Protection and Recovery: Fall protection and recovery research combines biomechanical control, compliant strategies, and learned whole-body behaviors to reduce impact and restore standing.HoST enabled standing from varied postures on the Unitree G1, while multi-contact methods included roll-and-stand and side-lying behaviors.

3.3 Robotic Manipulation

Robotic manipulation progresses from single-end-effector tasks to coordinated bimanual and whole-body control, with learning-based methods expanding capability beyond structured settings. Recent systems combine simulation, demonstrations, and end-to-end learning to address increasingly complex interactions.

  • Unimanual Manipulation: Unimanual manipulation spans pick-and-place, pushing, insertion, tool use, and deformable or articulated-object handling with a single end effector.
  • Unimanual Manipulation: DexCap collects human hand movements through portable motion capture, enabling robots to perform tasks from simple picks to complex tea preparation.
  • Unimanual Manipulation: Learning-based methods improve perception, affordance learning, imitation, collision avoidance, and manipulation of deformable, articulated, and dexterous objects.
  • Unimanual Manipulation: DexGraspVLA achieves a 90.8% success rate across 1,287 unseen object, lighting, and background combinations in zero-shot settings.
  • Bimanual Manipulation: Bimanual manipulation coordinates two arms for transport, assembly, and deformable-object handling while facing high-dimensional actions, collisions, and role assignment.
  • Bimanual Manipulation: The ALOHA series supports large-scale demonstrations for precise bimanual tasks, while RoboTwin 2.0 provides 50 tasks, 731 objects, and 5 embodiments through simulated data.
  • Bimanual Manipulation: Bimanual research is shifting from task-specific structural simplifications toward data-driven end-to-end frameworks, while dexterous-hand coordination remains challenging.
  • Whole-Body Manipulation Control: Whole-body manipulation uses a humanoid’s arms, torso, base, and other components to interact with objects.

3.4 Human-Robot Interaction

Human-robot interaction research organizes embodied collaboration around cognitive understanding, physical reliability, and social embeddedness. Progress combines multimodal interpretation, planning and compliant control, and simulation-based data generation, while robust operation in complex environments remains challenging.

  • Overview: Human-robot interaction is organized into Cognitive Collaboration, Physical Reliability, and Social Embeddedness.
  • Cognitive Collaboration: Cognitive collaboration uses multimodal intention learning and contextual semantic analysis to interpret human emotions, goals, actions, and environments.
  • Physical Reliability: Physical reliability coordinates force, timing, and distance so robots can adapt to human actions, avoid conflicts, and preserve safety.
  • Physical Reliability: Sampling-based and optimization-based planners, together with impedance, admittance, adaptive, and hybrid controllers, support collision-free and compliant interaction.
  • Physical Reliability: Simulation platforms support handover research by providing physics-based evaluation, dexterous grasp generation, and imitation-learning environments.
  • Physical Reliability: Physical reliability still faces high computational cost and limited robustness in complex scenarios, motivating more efficient and adaptive algorithms.
  • Social Embeddedness: Social embeddedness requires robots to adapt to social norms, cultural expectations, group dynamics, negotiation, etiquette, and emotional expression.

4 GENERAL PHYSICAL SIMULATORS

Physical simulators address robotic data bottlenecks by providing efficient, controllable, and reproducible environments for training, evaluation, and algorithm development. The survey reviews mainstream platforms spanning general-purpose simulation, articulated-object interaction, GPU acceleration, and full-stack embodied-AI infrastructure.

  • Motivation: Robot learning faces cost, safety, and hardware barriers in physical data collection, limiting generalization and applicability.
  • General Physical Simulators: Sim2Real transfer uses high-fidelity virtual environments to generate large multimodal datasets with automated pixel-level annotations and reproducible testing.
  • General Physical Simulators: Simulation supports debugging, validation, and optimization of perception, control, and planning algorithms for deployment.
  • Mainstream Simulators: Webots provides cross-platform robot modeling, multiple sensors, prebuilt models, and multi-language APIs for education, industry, and research.
  • Mainstream Simulators: Gazebo emphasizes open-source extensibility, middleware integration, and customizable models, controllers, sensors, physics, and rendering components.
  • Mainstream Simulators: MuJoCo targets contact-rich articulated dynamics with high-precision computation, generalized coordinates, biomechanical modeling, and convex contact optimization.
  • Mainstream Simulators: Isaac Sim and Isaac Lab form a GPU-accelerated, full-stack ecosystem supporting high-fidelity sensors, modular tasks, imitation learning, reinforcement learning, and sim-to-real research.
  • Mainstream Simulators: SAPIEN and its ManiSkill benchmarks emphasize physically realistic articulated-object interaction, while NVIDIA Newton targets high-fidelity multiphysics and sim-to-real transfer.

4.2 Physical Properties of Simulators

Simulator physical properties determine interaction realism and policy-transfer effectiveness across contacts, deformable materials, fluids, granular media, and differentiable dynamics. Existing platforms vary widely in native support, fidelity, and computational mechanisms for these phenomena.

  • Physical Modeling: Physical modeling capabilities influence generated-data realism and the effectiveness of policy transfer for complex robot behaviors.
  • Suction Modeling: Suction simulation models non-rigid attachment at contact interfaces and supports applications such as industrial grasping and warehouse picking.
  • Suction Modeling: MuJoCo uses user-defined suction logic, Gazebo uses plugins, and Webots, CoppeliaSim, and Isaac Sim provide native suction support.
  • External Disturbances: Random external forces simulate environmental uncertainty and help train humanoid robots for balance and disturbance resistance.
  • Deformable and Soft-Body Simulation: Deformable-object and soft-body-contact support ranges from basic models in several platforms to more advanced GPU-accelerated or high-precision approaches.
  • Fluid Simulation: Fluid simulation varies from basic buoyancy and drag in Webots and Gazebo to particle-based or native high-fidelity methods in Isaac Sim and Genesis.
  • Granular-Material Simulation: DEM models contact, collision, and friction among particles but is not natively supported by mainstream simulators, requiring indirect or non-specialized approaches.
  • Differentiable Physics: Differentiable physics computes gradients of physical states with respect to inputs, enabling gradient-based optimization and learning.

4.3 Rendering Capabilities

Simulator rendering capabilities affect visual realism, sensor fidelity, and the efficiency of large-scale robotic learning. The survey compares rendering engines, ray tracing, PBR, and parallel rendering across mainstream platforms.

  • High-fidelity rendering helps reduce the simulation-to-reality gap during robot design, validation, and optimization.
  • The comparison evaluates rendering engines, ray tracing, Physically Based Rendering, and parallel rendering across simulators.
  • Isaac Sim and Isaac Lab use NVIDIA Omniverse RTX with Hydra for high-fidelity graphics, while Isaac Gym uses Vulkan mainly for efficient debugging feedback.
  • Ray tracing improves shadows, reflections, refractions, global illumination, and physically realistic LiDAR and depth-camera simulation.
  • Webots, MuJoCo, and PyBullet lack native real-time ray tracing; CoppeliaSim supports ray-traced static images but not dynamic real-time rendering.
  • PBR models material-light interactions for consistent visual fidelity, supporting vision-based robotic learning; Webots, Gazebo, Isaac Sim, Isaac Lab, SAPIEN, and Genesis support it.
  • Isaac Gym, Isaac Sim/Lab, SAPIEN with ManiSkill, and Genesis provide strong parallel rendering, whereas older general-purpose simulators generally lack integrated high-throughput visual pipelines.

4.4 Sensor and Joint Component Types

Simulator support for sensors and joints enables repeatable perception, control, and structural testing, but simulator fidelity remains constrained by accuracy, complexity, data, and generalization challenges.

  • Realistic sensor models provide safe, repeatable bases for debugging, validation, and data collection in robot perception.
  • Most mainstream simulators support RGB vision, IMU, and contact-force sensing, while Isaac Sim and Isaac Lab provide comprehensive high-fidelity sensor simulation.
  • Joint types determine robot degrees of freedom and flexibility, making accurate joint modeling important for testing designs and optimizing control.
  • Floating, fixed, hinge, and prismatic joints respectively permit multidirectional motion, fully constrain motion, enable axial rotation, and enable axial translation.
  • Simulators provide controlled, repeatable experimentation that reduces the cost and risk of physical trials.
  • Simulator effectiveness is limited by inaccurate real-world representation, computational complexity, dependence on high-quality data, and overfitting to specific scenarios.
  • These limitations motivate world models designed to adapt to new data, handle complex systems efficiently, and reduce reliance on extensive datasets.

5 WORLD MODELS

World models represent and predict environmental dynamics through architectures ranging from compact latent-state models to video and 3D generative systems. The survey emphasizes their roles as neural simulators, dynamic models, and reward models for embodied agents.

  • World models are generative models intended to represent real-world dynamics, including physics and spatial properties.
  • They enable agents to predict future states and plan actions using internal representations of their environments.
  • Video-generation advances have increased interest in world models that synthesize high-fidelity visual sequences and model physical environments.
  • World-model architectures differ in state encoding, temporal modeling, and prediction targets, spanning latent dynamics and future video or 3D observations.
  • RSSMs encode evolving environment states in compact latent spaces and model temporal dynamics recurrently for long-horizon prediction and decision-making.
  • JEPAs predict abstract representations of missing content through self-supervised representation matching rather than reconstructing visual observations.
  • Transformer-based and autoregressive models improve sequence modeling and generate future observations conditioned on past context, actions, or language.
  • Token quantization can discard high-frequency details, reducing the visual quality of autoregressively generated videos.

6 WORLD MODELS FOR INTELLIGENT AGENTS

Autonomous driving and articulated robots are major embodied-intelligence applications with demanding world-modeling requirements. Both require rich, long-term, and safe interaction modeling for real-world deployment.

  • Autonomous driving and articulated robots are critical applications of artificial and embodied intelligence with different action-space characteristics.
  • Both application areas require precise and predictable models of rich, long-term, and safe real-world interactions.

6.1 World Models for Autonomous Driving

World models for autonomous driving address limitations of modular architectures by supporting realistic, controllable scene generation, predictive modeling, sensor-aware understanding, and planning through imagined futures.

  • World-model foundations: Video-generation world models target autonomous-driving needs including scene generation, multi-view consistency, closed-loop simulation, and reasoning.These systems prioritize traffic-specific characteristics and safety demands.
  • Neural simulation: World models support controllable, diverse driving-video generation by conditioning outputs on traffic rules, user descriptions, trajectories, and structured environmental factors.GAIA-2 and DriveDreamer-2 use ego dynamics, multi-agent configurations, weather, time of day, and road semantics for scenario control.
  • Neural simulation: ReconDreamer reduces ghosting through online restoration and progressive data updates, enabling rendering of multi-lane shifts spanning up to 6 meters.The method targets large and complex maneuvers in closed-loop simulation.
  • Dynamics and reward modeling: Driving world models improve planning and safety by learning latent dynamics, separating controllable from non-controllable factors, evaluating trajectories, and exploring multiple imagined futures.Examples include latent-space reinforcement learning, BEV safety assessment, image-based rewards, and what-if reasoning.
  • Dynamics and reward modeling: Recent systems incorporate physical reasoning, multi-agent behavior, multimodal geometry, and self-supervised latent prediction to strengthen autonomous-driving world models.These approaches address physical interactions, configurable traffic personalities, sensor fusion, and reduced reliance on perception labels.

6.2 World Models for Articulated Robots

World models for articulated robots combine compositional generation, physical simulation, predictive dynamics, and policy evaluation to improve transfer, generalization, efficiency, and adaptation across manipulation and locomotion tasks.

  • Compositional modeling: Compositional world models generalize to unseen object-action combinations and multimodal goals by factorizing video generation into reusable primitives.RoboDreamer synthesizes plans for novel tasks and outperforms monolithic baselines in simulation.
  • Compositional modeling: DreMa combines Gaussian Splatting with physics simulation for photorealistic future prediction and data-efficient imitation learning, achieving one-shot policy learning on a Franka robot.The approach uses equivariant transformations to improve accuracy and generalization.
  • Simulation and transfer: World-model systems support sim-to-real transfer through high-fidelity neural simulation, geometry-consistent multi-view video, and zero-shot adaptation across terrains and tasks.Pangu generates multimodal driving environments, RoboTransfer preserves cross-view geometry, and DWL transfers one humanoid policy across snow, stairs, and uneven ground.
  • Dynamic modeling: Latent world models improve planning and exploration by imagining future states, seeking novelty, and learning compact dynamics with fewer episodes or limited task-specific supervision.PlaNet, Plan2Explore, and DreamerV3 demonstrate these capabilities across continuous control, image-based exploration, and more than 150 tasks.

6.3 Challenges and Future Perspectives

The survey identifies unresolved challenges in world-model reliability, including partial observability, data bias, weak causal understanding, semantic reasoning, evaluation, memory, social behavior, interpretability, and compositional generalization.

  • Perception and modeling: High-dimensional, partially observed inputs create computational demands and uncertainty, requiring robust state estimation or belief-state maintenance.The challenge spans camera images, LiDAR point clouds, and radar signals.
  • Reasoning and generalization: World models often learn correlations rather than causal relationships, limiting counterfactual reasoning and robust generalization to novel situations.The survey calls for a transition toward genuine causal understanding of the environment.
  • Reasoning and generalization: Effective systems must connect fine-grained physical prediction with abstract reasoning about traffic laws, pedestrian intent, and object affordances.This fusion is presented as an open problem in semantic and context-aware behavior.
  • Evaluation and reliability: Future evaluation requires metrics beyond future-prediction error because visually sharper predictions may not yield safer or more efficient control policies.The survey emphasizes downstream-task-relevant evaluation frameworks.
  • Evaluation and reliability: Long-term forecasting remains difficult because prediction errors compound and real-world dynamics are stochastic, motivating memory architectures that retain relevant information over extended timescales.The survey gives remembering a previously seen road-work sign as an example.
  • Data and deployment: Training-data composition can transmit regional biases and underrepresent rare safety-critical events, constraining performance across environments.The survey highlights geographic variation and the long tail of infrequent scenarios as data-curation concerns.

7 CONCLUSION

The survey synthesizes physical simulators and world models as central technologies for embodied AI, combining a robot-intelligence grading framework, simulator comparisons, and analysis of world-model progress.

  • Survey contributions: The survey introduces five intelligence levels, IR-L0 to IR-L4, and compares mainstream simulators while reviewing world models from recurrent architectures to foundation-scale systems.It covers autonomy assessment, simulation platforms, and advances in autonomous driving and articulated robotics.
  • Overall conclusion: The survey concludes that simulators and world models have mitigated the sim-to-real gap while enabling sample-efficient learning, long-horizon planning, and robust generalization.These outcomes are stated across diverse environments within the survey’s synthesis.
Loading 2507.00917v3…