PaperlayerView exact revision on arXiv

Source-linked AI summary

Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI

Yang Liu, Weixing Chen, Yongjie Bai, Xiaodan Liang, Guanbin Li, Wen Gao, Liang Lin

arXiv:2407.06886v8cs.CVcs.AIcs.LGcs.MAcs.RO

TL;DR

Embodied AI lacks a comprehensive survey organizing research on aligning cyberspace and the physical world in the era of MLMs and WMs. This paper systematically reviews representative robots, simulators, four embodied tasks, datasets, challenges, and future directions, providing a comparative overview of recent development.

  • Problem

    The research community lacks a comprehensive survey organizing embodied AI studies, challenges, and future directions across cyberspace and the physical world.

  • Method

    The paper conducts a systematic survey of embodied robots, simulators, four representative embodied tasks, datasets, methodologies, challenges, and future research directions.

  • Results

    The comparative summary provides a clear picture of recent development in embodied AI across robots, simulators, datasets, and approaches.

  • Takeaways & Limitations

    The survey provides a foundational reference for research on embodied AI and its emerging direction toward sensing, perception, and interaction across cyber and physical spaces.

  • Takeaways & Limitations

    Large-scale datasets remain scarce, evaluation metrics vary across datasets, and world models struggle with real-world complexity, generalization, and sim-to-real transfer.

Abstract

from arXiv · show

Embodied Artificial Intelligence (Embodied AI) is crucial for achieving Artificial General Intelligence (AGI) and serves as a foundation for various applications (e.g., intelligent mechatronics systems, smart manufacturing) that bridge cyberspace and the physical world. Recently, the emergence of Multi-modal Large Models (MLMs) and World Models (WMs) have attracted significant attention due to their remarkable perception, interaction, and reasoning capabilities, making them a promising architecture for embodied agents. In this survey, we give a comprehensive exploration of the latest advancements in Embodied AI. Our analysis firstly navigates through the forefront of representative works of embodied robots and simulators, to fully understand the research focuses and their limitations. Then, we analyze four main research targets: 1) embodied perception, 2) embodied interaction, 3) embodied agent, and 4) sim-to-real adaptation, covering state-of-the-art methods, essential paradigms, and comprehensive datasets. Additionally, we explore the complexities of MLMs in virtual and real embodied agents, highlighting their significance in facilitating interactions in digital and physical environments. Finally, we summarize the challenges and limitations of embodied AI and discuss potential future directions. We hope this survey will serve as a foundational reference for the research community. The associated project can be found at https://github.com/HCPLab-SYSU/Embodied_AI_Paper_List.

I. INTRODUCTION

Embodied AI connects cyberspace and the physical world to support AGI-oriented systems that can perceive, reason, interact, and act. This survey addresses gaps in prior reviews by organizing embodied robots, simulators, four research tasks, and MLM- and WM-based architectures.

  • Embodied AI extends intelligence beyond abstract virtual problem-solving to navigating and interacting with unpredictable physical environments.Its applications include robotics, healthcare, and smart manufacturing.
  • Embodied agents combine language understanding, active exploration, multimodal perception, and action execution for complex tasks.MLMs align visual and linguistic representations, while WMs support simulation and comprehension of physical laws.
  • Prior surveys did not fully consider MLMs, WMs, embodied agents, and recent developments in embodied robots and simulators.The survey responds to this scarcity with coverage spanning cyber space and the physical world.
  • The paper presents a comprehensive cyber–physical survey of methodologies, benchmarks, challenges, applications, and future directions based on MLMs and WMs.It describes this perspective as the first comprehensive survey of its kind, to the authors’ knowledge.
  • The survey categorizes embodied AI into robots, simulators, embodied perception, embodied interaction, embodied agents, and sim-to-real adaptation.These categories form a taxonomy covering four representative embodied research tasks.
  • ARIO standardizes embodied-agent data through approximately 3 million episodes from 258 series and 321,064 tasks.The unified dataset is intended to facilitate robust, general-purpose embodied agents.

II. EMBODIED ROBOTS

Embodied robots span fixed-base, wheeled, tracked, quadruped, humanoid, and biomimetic forms, while simulators support safe, scalable development across general and real-scene environments.

  • Robot Categories: Embodied robots include fixed-base, wheeled, tracked, quadruped, humanoid, and biomimetic categories.
  • Simulator Roles: Embodied simulators enable controlled experimentation, data generation, standardized benchmarks, and safer algorithm development than direct physical deployment.
  • General Simulators: General-purpose simulators provide virtual environments for algorithm development and model training with cost, time, and safety advantages.
  • Real-Scene Simulators: Real-scene simulators use real-world data, photorealistic 3D assets, and engines such as UE5 and Unity, making them suited to household activities.
  • Automated Construction: Automated scene construction methods use language models, human instructions, or conditional diffusion to generate interactive simulation environments and training tasks.

IV. EMBODIED PERCEPTION

Embodied perception extends visual recognition toward 3D reasoning, environmental interaction, and task execution. Active visual perception combines localization, scene understanding, and exploration.

  • Future Direction: The survey identifies embodied-centric visual reasoning and social intelligence as future directions for visual perception.
  • Perception Scope: Embodied perception requires understanding 3D relations and dynamic environments while predicting and performing tasks from visual information.
  • Active Visual Perception: Active visual perception integrates state estimation, scene perception, and environment exploration for interaction and navigation in complex settings.
  • Visual SLAM: Visual SLAM estimates a robot’s position while constructing an environmental map, using onboard cameras as a lower-cost alternative to range-based sensing.
  • 3D Scene Understanding: 3D scene understanding identifies object semantics and locations while inferring geometric attributes from 3D scene data.

3) Active Exploration:

Active exploration complements passive perception by gathering informative observations through environmental interaction or viewpoint changes. VLN applies this perception to instruction-following navigation in unseen environments.

  • Active Exploration: Passive 3D scene understanding is complemented by active exploration, which dynamically interacts with environments or changes viewpoints to obtain information.
  • Visual Language Navigation: VLN requires agents to interpret visual observations and natural-language instructions before selecting actions in unseen environments.
  • VLN Task Variants: VLN instructions range from detailed action descriptions to rough task demands, while tasks include navigation, navigation with interaction, and sequential navigation.
  • VLN Datasets: R2R uses step-by-step instructions and Matterport3D navigation graphs, while Room-for-Room extends trajectories to require longer-range alignment.
  • VLN Datasets: ALFRED evaluates household tasks requiring environmental understanding and interaction under coarse- and fine-grained instructions.
  • VLN Datasets: Behavior-1K contains 1,000 long-sequence daily tasks requiring navigation-interaction over thousands of low-level actions.

2) Method:

VLN methods divide into memory-understanding and future-prediction approaches. The former models historical observations, while the latter predicts future states to support navigation, especially in continuous environments.

  • Method Taxonomy: VLN methods are categorized as Memory-Understanding Based or Future-Prediction Based.
  • Memory-Understanding Based: Memory-understanding methods encode historical observations or trajectories to perceive and understand the environment and guide navigation decisions.
  • Comparison: Memory-understanding methods remain mainstream because historical information matters for navigation decisions, while future prediction is gaining attention with world models.
  • Memory-Understanding Based: Graph-based memory methods represent time-step information as navigation-graph nodes and encode visual, language, or relational history.
  • Memory-Understanding Based: Large-model memory methods fuse historical observations with multimodal encodings and language-model knowledge for benchmark navigation.
  • Future-Prediction Based: Future-prediction methods model future states, rewards, observations, or movable waypoints to handle navigation in continuous environments.
  • Hybrid Methods: Hybrid approaches combine instruction targets, pixel-level interaction masks, previous navigation decisions, predicted operations, and landmark sequences.

A. Embodied Question Answering

Embodied question answering requires agents to explore environments from a first-person perspective, decide which actions gather relevant information, and determine when exploration can stop. The survey reviews EQA datasets and methods spanning neural networks and LLMs/VLMs.

  • EQA agents explore environments from a first-person perspective to gather information needed to answer questions.
  • EQA requires agents to select exploratory actions and decide when sufficient information has been gathered to answer.
  • Datasets: EQA datasets range from synthetic indoor scenes and single-object questions to multi-object comparisons and realistic 3D environments.
  • Datasets: Recent datasets extend EQA toward prior-knowledge reasoning, open-vocabulary questions, episodic memory, and active exploration.
  • Methods: EQA methods are broadly divided into neural-network approaches and LLM/VLM-based approaches.
  • Neural Network Methods: Early EQA systems used separate vision, language, navigation, and answering modules built mainly with CNNs and RNNs.

3) Limitations:

Embodied grasping combines perception, semantics, reasoning, and robot control to execute object manipulation from human instructions. Its development is constrained by limited semantic data and by the generalization demands of complex instructions.

  • Embodied interaction includes grasping and placing objects from human instructions, alongside question-answering tasks.
  • Datasets: The emergence of multi-modal models has driven extensions of grasping datasets with linguistic text for semantic grasping.
  • Datasets: Traditional grasping datasets provide RGB, depth, point-cloud, or 3D-scene data with kinematic grasp annotations but lack semantic information.
  • Language-guided Grasping: Semantic understanding combined with spatial and logical reasoning supports complex grasping from natural-language instructions.
  • Language-guided Grasping: Language-guided grasping uses explicit object instructions or implicit instructions requiring spatial and logical reasoning.

3) End-to-End Approaches:

End-to-end and modular approaches combine vision-language representations with grasp generation to connect natural-language semantics and robotic manipulation. These methods increasingly support semantic scene reasoning and generalization beyond previously seen expressions or categories.

  • End-to-End Approaches: CLIPORT combines CLIP with Transporter Net in an end-to-end dual-stream architecture for semantic understanding and grasp generation.
  • End-to-End Approaches: CROG learns grasp synthesis directly from image-text pairs using a vision-language-grasping dataset based on OCID.
  • Modular Approaches: F3RM combines precise 3D geometry with CLIP-derived semantics to localize language-specified objects before grasp generation.
  • Limitations: End-to-end and modular frameworks improve robots’ ability to perform complex grasping from natural-language instructions.
  • Embodied agents typically combine visual perception, high-level task planning, and low-level action planning.

A. Embodied Task Planning

Embodied task planning decomposes complex instructions into executable subtasks, but fixed symbolic methods can be rigid and language-model plans can fail when they omit real-world visual complexity. Integrating visual information and replanning is therefore emphasized as a way to align plans with current environments.

  • Traditional planners use explicit rules, logical reasoning, PDDL, MCTS, or A* but may not adapt well to dynamic or unforeseen changes.
  • LLMs can decompose abstract tasks into manageable steps using internal world knowledge and chain-of-thought reasoning.
  • Planned subtasks may fail when planners do not fully account for real-environment complexity or lack sufficient visual information.
  • Planning utilizing the visual information from embodied perception model: Visual labels, locations, and descriptions can help LLMs identify objects and obstacles, revise unreasonable steps, and improve replanning.
  • Planning utilizing the visual information from embodied perception model: Multimodal integration and contextual reasoning can improve environmental judgments when visual information is limited.
  • Planning utilizing the VLMs: 96% task-planning accuracy contrasted with 60% overall task completion in RoboGPT, highlighting the importance of low-level action planning.

B. Embodied Action Planning

Embodied action planning separates high-level task decomposition from uncertainty-aware low-level execution, with feedback refining the overall plan. Approaches range from tool-based policy composition to integrated multimodal foundation models and scalable multi-agent strategies.

  • B. Embodied Action Planning: Action planning addresses real-world uncertainty that task-planning subtasks cannot represent at sufficient granularity.Execution feedback is used to refine task planning.
  • B. Embodied Action Planning: LLMs can abstract pretrained policies into callable tools, while Reflexion and DEPS support execution-time adjustment and zero-shot skill composition.These approaches simplify handling complex subtasks through reusable policy interfaces and skill combinations.
  • B. Embodied Action Planning: VLA models integrate perception, decision-making, and execution to reduce communication latency and adapt to dynamic environments.The integrated architecture performs planning and action execution within one system.
  • B. Embodied Action Planning: Scalable embodied agents combine robust perception, efficient decision-making, resource optimization, hierarchical SLAM, edge computing, and multi-agent communication.Domain adaptation supports generalization to new environments.
  • B. Embodied Action Planning: Sim-to-real adaptation transfers behaviors learned in simulation to physical environments while validating their robustness and reliability.The process involves world models, data collection, and training methods.
  • B. Embodied Action Planning: World models predict future states and suit structured tasks such as autonomous driving and object sorting, but are less suited to unstructured complex tasks.They are trained from scratch on physical-world data, unlike pretrained VLA models fine-tuned with real-world data.

1) Generation-based Methods:

Embodied world models are organized into generation-based, prediction-based, and knowledge-driven methods. These approaches differ in whether they learn input-output transformations, operate through latent representations, or impose constructed world knowledge and physical constraints.

  • 1) Generation-based Methods:: Generative models can produce images, videos, point clouds, and other physically consistent formats, suggesting that they internalize world knowledge from large datasets.Their learned representations capture statistical properties and can simulate aspects of the world.
  • 1) Generation-based Methods:: Generation-based methods learn transformations between input and output spaces through autoencoder frameworks.The taxonomy also distinguishes prediction-based latent-space models and knowledge-driven models.
  • 1) Generation-based Methods:: Generative world models require substantial training resources and often lack interpretability, while efficiency and data-bias problems remain unresolved.These limitations hinder broader practical application.
  • 1) Generation-based Methods:: Prediction-based models construct internal latent representations to reconstruct features under given conditions and extract high-level semantic information.Operating in latent space is intended to support perception across environments at lower cost, though instability and interpretability limitations remain.
  • 1) Generation-based Methods:: Knowledge-driven models inject constructed knowledge, such as physical rules or common sense, to constrain outputs and improve reliability and interpretability.Real2Sim2Real uses real-world knowledge to build physics-compliant simulators for robot training.
  • 1) Generation-based Methods:: World models still struggle with high-dimensional sensing, stochastic dynamics, long-term dependencies, unseen scenarios, and sim-to-real transfer.Suggested remedies include realistic simulators, multimodal inputs, modular architectures, data efficiency, transfer learning, and real-world priors.

B. Data Collection and Training

Embodied-AI training increasingly combines real and simulated data to reduce costly collection demands, while new sim-to-real paradigms address transfer across domains. ARIO responds to fragmented multimodal, multi-robot datasets with a unified standard and large-scale collection.

  • B. Data Collection and Training: High-quality real-world demonstration collection is expensive, time-consuming, and labor-intensive, motivating efficient collection and simulation-based alternatives.The survey discusses collection and training in both real and simulated environments.
  • B. Data Collection and Training: Large, diverse robotic datasets can improve generalization and adaptability, with Open X-Embodiment, UMI, and Mobile ALOHA representing complementary collection strategies.Open X-Embodiment includes data from 22 robots, 527 skills, and 160,266 domestic tasks.
  • B. Data Collection and Training: Simulation enables automated data collection, and PyBullet-trained systems such as CLIPORT and Transporter Networks have transferred to real-world applications.Simulation-based datasets can reduce dependence on costly physical demonstrations.
  • B. Data Collection and Training: Five sim-to-real paradigms use digital twins, human corrections, domain randomization, system identification, and natural language to reduce transfer gaps.These paradigms respectively target strategy learning, corrected behavior, environmental diversity, scene similarity, and cross-domain representations.
  • B. Data Collection and Training: Existing multi-robot datasets lack simultaneous multimodal coverage, unified formats, compatible control-object representations, sufficient volume, and combined simulated-real data.These gaps complicate processing and limit large-scale pretraining and sim-to-real research.
  • B. Data Collection and Training: ARIO unifies control and motion data across robot morphologies and provides approximately 3 million episodes from 258 series and 321,064 tasks.Its standardized timestamps and variable-data support target more generalizable embodied models.

5) Real-world Deployments of Embodied AI Systems:

Embodied AI systems are deployed across healthcare, logistics, and manufacturing, but real-world operation remains constrained by transfer gaps, data scarcity, long-horizon planning, causal reasoning, evaluation, and security. The survey frames broader deployment as dependent on more capable datasets, simulators, benchmarks, and safeguards.

  • 5) Real-world Deployments of Embodied AI Systems:: Healthcare, logistics, and manufacturing deployments use embodied systems for surgical precision, supply delivery, warehousing, transportation, precision, and collaboration.Examples include Da Vinci, Moxi, Amazon Robotics, Stretch, Fanuc, and ABB.
  • 5) Real-world Deployments of Embodied AI Systems:: Sim-to-real deployment is challenged by domain shifts, dynamic interactions, limited training diversity, sensor noise, simulation overfitting, and unexpected events.These factors can undermine real-world robustness.
  • 5) Real-world Deployments of Embodied AI Systems:: Sufficient real-world robotic data remains difficult to obtain because collection is resource-intensive, while simulation alone worsens the sim-to-real gap.The survey calls for collaborative, large-scale datasets and more realistic, efficient simulators.
  • 5) Real-world Deployments of Embodied AI Systems:: Long-horizon instructions require sequences of low-level actions over extended time spans, exceeding the capabilities of many current high-level planners.“Clean the kitchen” exemplifies the need for rearranging, sweeping, wiping, and related actions.
  • 5) Real-world Deployments of Embodied AI Systems:: Data-driven agents may rely on correlations rather than causal relations among knowledge, behavior, and environment, producing biased strategies and reduced reliability.The survey motivates world-knowledge-driven causal reasoning for robust operation.
  • 5) Real-world Deployments of Embodied AI Systems:: Comprehensive evaluation requires realistic simulators and benchmarks spanning diverse skills while assessing both high-level planning and low-level control.Existing benchmarks vary in skills and often constrain scenes and objects through simulators.
  • 5) Real-world Deployments of Embodied AI Systems:: LLM-dependent embodied agents face vulnerabilities including word injection, scenario manipulation, and knowledge injection in sensitive or private spaces.Such attacks can lead to hazardous robot or vehicle behavior.
  • 5) Real-world Deployments of Embodied AI Systems:: The survey positions embodied AI’s cross-space sensing, perception, and interaction capabilities as significant for AGI and reviews the field’s major components and directions.Its comparative synthesis covers robots, simulators, datasets, approaches, and future research.
Loading 2407.06886v8…