PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Embodied AI: From Simulators to Research Tasks

Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, Cheston Tan

arXiv:2103.04918v8cs.AIcs.LG

TL;DR

Embodied AI lacks a contemporary comprehensive survey connecting simulators with its major research tasks. This paper surveys nine simulators, benchmarks them across seven features and secondary criteria, and reviews visual exploration, visual navigation, and embodied QA. It identifies simulator capabilities, task connections, and challenges relevant to selecting simulators for research.

  • Problem

    Earlier embodied AI surveys were mostly outdated, while contemporary comprehensive coverage connecting simulators and research tasks remained scarce.

  • Method

    The paper surveys nine embodied AI simulators using seven primary features and reviews visual exploration, visual navigation, and embodied question answering approaches, evaluations, and datasets.

  • Results

    The survey finds that AI2-THOR, iGibson, and Habitat-Sim possess all three secondary features and are more widely used across diverse embodied AI research tasks.

  • Takeaways & Limitations

    The comparisons support selecting embodied AI simulators according to research-task requirements while highlighting connections among simulators, datasets, and tasks.

Abstract

from arXiv · show

There has been an emerging paradigm shift from the era of "internet AI" to "embodied AI", where AI algorithms and agents no longer learn from datasets of images, videos or text curated primarily from the internet. Instead, they learn through interactions with their environments from an egocentric perception similar to humans. Consequently, there has been substantial growth in the demand for embodied AI simulators to support various embodied AI research tasks. This growing interest in embodied AI is beneficial to the greater pursuit of Artificial General Intelligence (AGI), but there has not been a contemporary and comprehensive survey of this field. This paper aims to provide an encyclopedic survey for the field of embodied AI, from its simulators to its research. By evaluating nine current embodied AI simulators with our proposed seven features, this paper aims to understand the simulators in their provision for use in embodied AI research and their limitations. Lastly, this paper surveys the three main research tasks in embodied AI -- visual exploration, visual navigation and embodied question answering (QA), covering the state-of-the-art approaches, evaluation metrics and datasets. Finally, with the new insights revealed through surveying the field, the paper will provide suggestions for simulator-for-task selections and recommendations for the future directions of the field.

I. INTRODUCTION

Embodied AI shifts learning from internet-curated datasets toward interaction with environments, motivating contemporary surveys of simulators and research tasks. This paper surveys nine simulators and three major tasks to characterize capabilities, limitations, and simulator-task relationships.

  • Embodied AI agents learn through interactions with surrounding environments rather than primarily from internet-curated images, videos, and text.
  • Existing embodied AI surveys were mostly published before the modern deep learning era, creating a scarcity of contemporary comprehensive coverage.
  • The paper surveys nine general-purpose embodied AI simulators designed to provide realistic simulated representations of rooms, apartments, and physical environments.
  • The research tasks are connected by increasing complexity, with visual exploration serving as a useful component of visual navigation.
  • The simulator selection excludes simulation-to-real research and robotics in the physical world.
  • It benchmarks simulators for realism, scalability, and interactivity, then surveys visual exploration, visual navigation, and embodied question answering.

II. SIMULATORS FOR EMBODIED AI

The survey compares embodied AI simulators through environment construction, physics, object sources and properties, controllers, actions, and multi-agent support. These features describe how simulators represent environments and enable agent interaction.

  • Evaluation features: The survey compares nine simulators using seven features: environment, physics, object type, object property, controller, action, and multi-agent support.
  • Environment: Game-based scenes use 3D assets, whereas world-based scenes use real-world scans; asset-based environments generally provide clearer object segmentation and built-in physics.
  • Physics: Basic physics covers collision, rigid-body dynamics, and gravity, while advanced physics adds cloth, fluid, and soft-body simulation.
  • Object type: Objects come from dataset-driven environments or online asset stores, trading higher collection cost and quality control against easier contribution and sustainability.
  • Object property: Object properties range from basic interaction to multiple-state changes, such as an apple becoming slices after being cut.
  • Control and action: Controllers range from Python APIs to virtual robots and virtual reality, while actions span navigation, atomic actions, and human-computer interaction.
  • Multi-agent: Only a few simulators support multi-agent setups, including avatar-based interaction in ThreeDWorld and user-based interaction in AI2-THOR.

B. Comparison of Embodied AI Simulators

The comparison evaluates simulators through realism, scalability, and interactivity, linking these properties to their suitability across embodied AI research tasks. AI2-THOR, iGibson, and Habitat-Sim emerge as especially widely used and technically strong examples.

  • Evaluation framework: The seven primary simulator features are grouped into realism, scalability, and interactivity for secondary evaluation.
  • Cross-simulator comparison: Simulators possessing all three secondary features, including AI2-THOR, iGibson, and Habitat-Sim, are more widely used across diverse embodied AI research tasks.
  • Quantitative comparison: AI2-THOR has the largest environment configurations, while Habitat-Sim and iGibson rank as the top two performers in graphic rendering.
  • Physics capability: Advanced physics features provide testbeds for embodied agents performing tasks involving sophisticated physical interactions.

III. RESEARCH IN EMBODIED AI

The survey organizes embodied AI research around visual exploration, visual navigation, and embodied QA, with tasks increasing in complexity and exploration supporting downstream navigation. It reviews exploration methods, evaluation metrics, datasets, and the relationship between simulated environments and research tasks.

  • Research tasks: Embodied AI research focuses on visual exploration, visual navigation, and embodied QA, which form a hierarchy of increasing task complexity.The paper treats each task as foundational for subsequent tasks in the hierarchy.
  • Visual exploration: Visual exploration gathers information through motion and perception to update an internal environment model for downstream tasks such as visual navigation.The internal model may be a topological graph map or another representation useful for path planning.
  • Visual exploration: Classical exploration uses SLAM for mapping, localization, and path planning, but sensor-based geometric methods are susceptible to measurement noise and require fine-tuning.The survey contrasts these approaches with learning-based methods using RGB and/or depth sensors.
  • Evaluation: The survey summarizes embodied AI tasks using metrics including ATV, downstream-task performance, SPL, SR, navigation error, goal progress, and QA accuracy.Table III lists metrics spanning exploration, navigation, and embodied question answering.
  • Visual exploration: Learning-based exploration can incorporate semantic understanding and generalize knowledge from seen environments to novel environments in an unsupervised manner.The paper states that this reduces reliance on humans and improves efficiency.

1) Approaches:

Visual exploration approaches are organized around baseline behaviors, curiosity, coverage, and reconstruction. These methods differ in whether they reward prediction difficulty, observed targets, or reconstruction-relevant views, while addressing the challenges of partial observation and stochastic environments.

  • Baselines: Common visual exploration baselines include random actions, always moving forward, collision-triggered turning, and frontier exploration using a map.Frontier exploration iteratively visits boundaries between free and unexplored space.
  • Curiosity: Curiosity-driven agents use prediction error as a reinforcement-learning reward, emphasizing intrinsic motivation when external rewards are sparse.The approach seeks states that are difficult for a forward-dynamics model to predict.
  • Curiosity: Stochasticity can produce misleading curiosity rewards, including noisy-TV behavior or rewards from random agent actions without environmental progress.Inverse dynamics addresses some environmental stochasticity but may not resolve stochasticity caused by the agent’s actions.
  • Coverage: Coverage methods maximize directly observed targets, while one hybrid approach combines analytical path planning, learned SLAM, and noise models to reduce end-to-end sample complexity and improve physical realism.Another method uses a scene-memory transformer to store encountered observations without relying on map-like inductive biases.
  • Reconstruction: Reconstruction methods reward agents for obtaining views that help recreate other views or predict semantic concepts at sampled query locations.One described method defines true reconstruction concepts using the nearest K-means cluster centroids to a query feature representation.

2) Evaluation Metrics:

The survey evaluates exploration and navigation using target coverage and downstream navigation outcomes, while describing datasets, task goals, and challenge settings. Visual navigation commonly uses point or object goals and learning-based systems that incorporate sensor data and semantic understanding.

  • Exploration metrics: Exploration metrics include the amount of targets visited and variants of visited-area coverage, such as absolute area in m2 and percentage explored.Targets may be areas or interesting objects.
  • Exploration metrics: Visual exploration can also be evaluated by its impact on downstream Image Navigation, Point Navigation, or Object Navigation.This category has appeared more commonly in recent works.
  • Datasets: Matterport3D and Gibson V1 provide photorealistic RGB data with depth and semantic segmentation, while Habitat-Sim adds configurable agents and multiple sensors.Gibson V1 was enhanced with interactions and realistic robot control to form iGibson.
  • Navigation tasks: Visual navigation sends an agent through a 3D environment toward a goal, with this survey focusing on point and object goals as common fundamental cases.Goals may also be specified as images or areas, and navigation can include external priors or natural-language instructions.
  • Navigation approaches: Learning-based navigation uses RGB and/or depth sensors to reduce case-specific hand engineering, improve robustness to measurement noise, and incorporate semantic understanding.The paper also describes generalization from previously seen environments to novel environments in an unsupervised manner.
  • Navigation benchmarks: The iGibson, Habitat, and RoboTHOR challenges benchmark point and object navigation with egocentric RGB-D observations.The 2020 iGibson challenge used 73 Gibson scenes for training and a reconstructed real-world apartment scene across multiple interaction scenarios.

1) Categories:

The surveyed categories span point, object, and language-conditioned navigation, with increasingly multimodal and interactive settings. Approaches range from end-to-end learning to modular systems using maps, auxiliary tasks, priors, and memory.

  • Point navigation: Point navigation methods learn policies from visual and action histories, with DFP processing recent color, depth, and action observations.The setting may involve unseen environments without ground-truth maps or agent poses.
  • Point navigation: BDFP adds an intermediate map-like representation to make DFP’s black-box policy more interpretable, while SplitNet uses separate decoders for auxiliary tasks and policy learning.BDFP outperforms DFP in most cases, and SplitNet can outperform comparable end-to-end methods in previously unseen environments.
  • Modular navigation: Modular navigation systems combine semantic 2.5D memory or learned mapping with recurrent policies to improve navigation in previously unseen environments.One method outperforms a learned LSTM policy without a map.
  • Point navigation: Habitat-based work uses PPO or DD-PPO, with one agent reaching within 3–5% of the shortest-path oracle after 2.5 billion learning steps.The agent receives egocentric RGB or depth observations together with GPS and compass information.
  • Point navigation: Auxiliary tasks improve sample and time efficiency, with the best agent matching prior performance at 40 million frames, 5.5X faster.The auxiliary tasks include action-conditional contrastive predictive coding, inverse dynamics, and temporal distance estimation.
  • Object navigation: Object navigation asks agents to find an object category from a random starting position in an unexplored environment, while adaptation and semantic priors support generalization.Meta-reinforcement learning can use a self-supervised interaction loss, and multimodal priors can provide semantic knowledge for navigation.
  • Language-conditioned navigation: Visual-language navigation follows natural-language instructions through long sequences requiring continual visual input and camera viewpoint manipulation.Auxiliary reasoning tasks include trajectory retelling, progress estimation, angle prediction, and cross-modal matching; vision-dialog navigation adds ongoing conversation and cross-modal memory.

2) Evaluation Metrics:

Visual navigation is primarily evaluated with success weighted by path length and success rate, supplemented by task-specific measures for instruction-following and dialogue navigation.

  • General visual navigation: Visual navigation mainly uses success weighted by path length (SPL) and success rate.SPL accounts for both task success and path length.
  • General visual navigation: Success rate measures the fraction of episodes in which the agent reaches the goal within the time budget.SPL has known issues, so both metrics are used in visual navigation evaluation.
  • Vision-and-language navigation: VLN additionally uses oracle success rate and trajectory length, with SPL described as the best general metric because it accounts for the path taken.Oracle success rate evaluates whether the agent stops at the closest point to the goal along its trajectory.
  • Vision-dialog navigation: Vision-dialog navigation adds goal progress and oracle path success rate to success rate and oracle success rate.Goal progress measures average progress toward the goal, while oracle path success rate concerns stopping at the closest point along the shortest path.

3) Datasets:

Visual navigation commonly uses Matterport3D and Gibson V1, while language-conditioned navigation relies on instruction and dialogue datasets built in Matterport3D.

  • Visual navigation: Matterport3D and Gibson V1 are the most popular visual-navigation datasets, and AI2-THOR is also used.Gibson V1 scenes are smaller and usually have shorter episodes than Matterport3D scenes.
  • Vision-and-language navigation: Most VLN work uses Room-to-Room, containing 21,567 navigation instructions averaging 29 words.The dataset is paired with the Matterport3D Simulator.
  • Vision-dialog navigation: Vision-dialog navigation uses the Cooperative Vision-and-Dialog Navigation dataset, comprising 2,050 human-to-human dialogs and over 7,000 trajectories.CVDN is also situated in the Matterport3D Simulator.
  • Embodied question answering: Embodied question answering requires capabilities spanning visual recognition, language understanding, question answering, commonsense reasoning, planning, and navigation.The paper characterizes embodied QA as the most onerous and complicated task in embodied AI.
  • Embodied question answering: EQA commonly separates navigation from question answering, because agents must explore environments to see relevant objects before answering.PACMAN uses a hierarchical planner-controller navigation module before applying QA to frames collected along the path.
  • Multi-target embodied QA: Multi-target EQA requires visiting multiple rooms or objects and comparing their attributes, such as color, size, or distance.The MT-EQA dataset contains six types of compositional comparison questions.
  • Interactive question answering: Interactive QA extends EQA by requiring object interaction, using a hierarchical controller system with navigation, scanning, manipulation, memory, and answering modules.IQA is studied in AI2-THOR, and IQUAD V1 contains 75,000 multiple-choice questions.

2) Evaluation Metrics:

Embodied QA and interactive QA evaluate navigation and answering separately, using navigation-distance, stopping, room-entry, hit, trajectory, ranking, and accuracy measures.

  • Evaluation structure: Embodied QA and interactive QA decompose evaluation into navigation and question-answering subtasks.The two subtasks use different metrics.
  • Navigation metrics: Navigation metrics include navigation error, goal progress, minimum target distance, stopping rate, target-room termination, and target-room entry.These measures capture both distance-based progress and whether agents reach relevant rooms before answering.
  • Simulator-task connections: Figure 6 connects nine simulators to research tasks and evaluation datasets, grouping visual-navigation tasks separately from other categories.Its simulator-task connections indicate that Habitat-Sim and iGibson support visual exploration and a range of visual-navigation tasks.
  • Navigation metrics: Hit accuracy based on IoU and episode length are additional navigation metrics, with episode length also used in visual navigation.The cited metrics distinguish spatial overlap from trajectory duration.
  • QA metrics: QA performance is evaluated with mean rank of the ground-truth answer and answer accuracy.Mean rank measures the position of the correct answer among predictions.

IV. INSIGHTS AND CHALLENGES

The survey links simulator capabilities to embodied research tasks and identifies realism, scalability, and interactivity as continuing challenges. It also proposes a task pyramid and directions for improving simulators and their transfer to physical-world settings.

  • Insights into Embodied AI: Habitat-Sim and iGibson support visual exploration and multiple visual navigation tasks, while simulator suitability varies across research tasks.The survey attributes this support to the high fidelity of world-based scene simulators; some simulators connect to no surveyed embodied task but still meet the classification criteria.
  • Insights into Embodied AI: Embodied question answering and visual navigation with priors require multiple-state object properties because these tasks involve interactive environments.AI2-THOR is identified as the simulator of choice for these requirements, whereas existing VLN work uses Matterport3D Simulator because it does not require interactivity.
  • Insights into Embodied AI: Visual exploration supports visual navigation, which contributes to embodied QA in a pyramid of increasingly complex embodied AI tasks.The survey hypothesizes Task-based Interactive Question Answering as a further advance requiring agents to perform tasks to answer questions.
  • Challenges in Embodied AI Simulators: Current simulators differ from conventional reinforcement-learning environments but still face challenges in realism, scalability, and interactivity.The survey describes embodied AI simulators as having advanced functionality and fidelity while identifying these three areas as ongoing challenges.
  • Challenges in Embodied AI Simulators: Only Habitat-Sim and iGibson are world-based scene simulators, while only ThreeDWorld provides the advanced physics features needed for sophisticated physical interactions.The survey also identifies a shortage of simulators combining world-based scenes with advanced physics, including cloth, fluid, and soft-body physics.
  • Challenges in Embodied AI Simulators: Scalability is constrained by the scarcity and cost of tools for collecting large-scale, high-fidelity 3D scenes and object scans.The survey points to learning-based 3D reconstruction and scene generation as potential ways to scale dataset collection.
  • Challenges in Embodied AI Simulators: Game-based simulators offer fine-grained and symbolic interactions, whereas world-based simulators provide gross motor control but generally lack multi-state object classes.The survey calls for a balance between object functionality and the complexity of actions agents can perform.
  • Challenges in Embodied AI Simulators: Real-to-sim adaptation could improve simulators by integrating tactile perception, human-level motor control, and audio with visual inputs.The proposed direction is intended to develop more realistic simulators that bridge physical and virtual worlds.

C. Challenges in Embodied AI Research

Embodied AI research becomes increasingly complex as tasks progress from visual exploration to visual navigation and embodied QA, requiring longer trajectories, multiple modalities, and stronger memory. The field also faces limited attention to multi-agent settings and addresses rising training difficulty through hybrid methods and prior knowledge.

  • Long trajectories and multiple sensor modalities make agent memory and internal representations especially important.
  • Embodied AI tasks increase in complexity from visual exploration to visual navigation and embodied QA as language understanding and question answering are added.
  • Fully learning-based approaches make newer tasks harder and lengthen training as each added component expands the agent’s learning requirements.
  • Hybrid classic-learning approaches and prior knowledge are identified as promising ways to reduce search space and sample complexity while improving robustness.
  • Multi-agent collaboration and communication remain relatively underexplored, although simulators with multi-agent features have recently increased.

V. BIOGRAPHY SECTION

The biography section profiles five authors whose backgrounds span engineering, applied mathematics, computer engineering, brain and cognitive sciences, computer vision, multimedia analysis, and embodied AI. Their affiliations include Singapore’s A*STAR research institutes and academic institutions in Singapore and the United States.

  • Jiafei Duan is a research engineer at A*STAR’s Institute of Infocomm Research working on embodied AI and computational cognitive science.
  • Samson Yu is a research engineer at A*STAR’s Institute of High Performance Computing conducting fundamental AI and embodied AI research.
  • Hui Li Tan works at A*STAR’s Institute for Infocomm Research, with interests including computer vision and multimodal deep learning.
  • Hongyuan Zhu is a research scientist at A*STAR’s Institute for Infocomm Research whose interests include multimedia content analysis and segmentation.
  • Cheston Tan is a senior scientist at A*STAR’s Institute for Infocomm Research with degrees from Berkeley and MIT.
Loading 2103.04918v8…