PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Robotics with Foundation Models: toward Embodied AI

Zhiyuan Xu, Kun Wu, Junjie Wen, Jinming Li, Ning Liu, Zhengping Che, Jian Tang

arXiv:2402.02385v1cs.ROcs.AI

TL;DR

Robots still struggle to generalize from controlled, task-specific settings to open-ended everyday tasks, and long-horizon planning remains challenging. This survey organizes foundation-model research in autonomous manipulation across planning and control, reviews its resources and challenges, and reports a field still progressing toward everyday deployment.

  • Problem

    Robots trained on task-specific data with limited model capacity struggle to generalize and transfer beyond controlled environments, while long-horizon tasks require multiple skills in sequence.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The survey categorizes foundation-model applications in autonomous manipulation across high-level planning and low-level control and reviews datasets, simulators, benchmarks, and challenges.

    Source: §1 · p. 1, §6 · p. 7

  • Results

    The survey reports significant progress toward general-purpose tasks, while noting that wide deployment in everyday scenarios remains a considerable journey.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The review concludes with key challenges and promising research directions for advancing foundation models in robotics toward generalized embodied AI.

    Source: §6 · p. 7

    Simulator-to-real deployment remains limited by appearance and dynamics distribution shifts that Sim2Real methods cannot fully compensate for.

    Source: §5.3 · p. 7

Abstract

from arXiv · show

While the exploration for embodied AI has spanned multiple decades, it remains a persistent challenge to endow agents with human-level intelligence, including perception, learning, reasoning, decision-making, control, and generalization capabilities, so that they can perform general-purpose tasks in open, unstructured, and dynamic environments. Recent advances in computer vision, natural language processing, and multi-modality learning have shown that the foundation models have superhuman capabilities for specific tasks. They not only provide a solid cornerstone for integrating basic modules into embodied AI systems but also shed light on how to scale up robot learning from a methodological perspective. This survey aims to provide a comprehensive and up-to-date overview of foundation models in robotics, focusing on autonomous manipulation and encompassing high-level planning and low-level control. Moreover, we showcase their commonly used datasets, simulators, and benchmarks. Importantly, we emphasize the critical challenges intrinsic to this field and delineate potential avenues for future research, contributing to advancing the frontier of academic and industrial discourse.

1 Introduction

Robots remain largely confined to controlled environments, while task-specific learning methods struggle to generalize and transfer to everyday tasks. This survey organizes foundation-model robotics research around high-level planning and low-level control and reviews its resources, challenges, and future directions.

1 Introduction

  • Robots remain confined to controlled environments, and task-specific data and limited model capacity constrain generalization and transfer to everyday tasks.Earlier vision-guided planning and reinforcement-learning control reduced reliance on precise environment or object models, which is impractical in dynamic real-world settings.Source: §1 · p. 1
  • Foundation models are not yet physical-world decision-makers, leaving their integration with decision-making models a key embodied-AI question.Although large language, vision, and multimodal models offer understanding, reasoning, interaction, and generative capabilities, they mainly provide auxiliary information.Source: §1 · p. 1
  • The survey categorizes autonomous-manipulation research into high-level planning and low-level control, including work that combines both.Planning processes commands and observations into stepwise decisions; control determines execution parameters such as end-effector positions or joint degrees.Source: §1 · p. 1
  • The survey reviews commonly used datasets, simulators, and benchmarks while noting that broad everyday deployment remains a considerable journey.It also emphasizes persistent challenges and proposes potential research directions.Source: §1 · pp. 1–2

2 Foundation Models for High-Level Planning

Long-horizon tasks challenge a single policy to acquire multiple skills and execute them in order from concise instructions. The survey reviews foundation-model planning by planning form and by assistants that add perception, knowledge, or feedback.

2 Foundation Models for High-Level Planning

  • Long-horizon planning remains difficult because a single policy must acquire multiple skills and execute them in sequence from concise instructions.The survey organizes planning approaches by their forms and assistants and presents a taxonomy in Figure 1.Source: §2 · p. 2

2.1 Forms of Planning

  • Structured planning formats such as PDDL improve precision and controller interpretability, but their predefined forms restrict expressiveness and may limit zero-shot tasks.Looser formats such as natural language are more expressive but can create execution challenges requiring sophisticated low-level control.Source: §2.1 · p. 2

Policy Code Planning

  • Policy code planning converts natural-language instructions into ordered code steps that can express spatial references, feedback loops, and control primitives.Examples include CaP, ProgPrompt, Instruct2Act, and OEIEA, which generate or use policy code through different prompting and perception approaches.Source: Policy Code Planning · p. 2

Natural Language Planning

  • Natural-language plans are interpretable but may be unclear enough to mislead agents and abort tasks.CAPEAM reduces ambiguity by extracting subgoals and concrete objects, while external models assess subgoal feasibility, correctness, and clarity.Source: Natural Language Planning · pp. 2–3

2.2 Assistants of Planning

  • Planning assistants add vision, external feedback, or common-sense knowledge to bridge language processing with practical perceptual information.The survey describes these additions as ways to support target identification, manipulation feedback, reasoning, and planning.Source: §2.2 · p. 3

Visual-Assisted Plan

  • Vision-assisted planning grounds language in environments, but vision-language models can fall short in fine-grained semantic understanding needed for tiny-object manipulation and dexterous operation.Methods include integrating detection outputs into language-model prompts, jointly scoring skills, and using vision-language models directly for planning.Source: Visual-Assisted Plan · p. 3

Plan with Extra Knowledge

  • Planning approaches supplement foundation models with domain knowledge through learned failures, object-location memory, action translation, or commonsense prompts.These methods address foundation models' typical lack of domain-specific knowledge in some scenarios.Source: Plan with Extra Knowledge · p. 3

Plan with Feedback

  • Interactive planning updates plans during execution using environmental success signals or human-provided information.Inner Monologue uses closed-loop language feedback, while IRAP gathers task-relevant knowledge through question-and-answer interaction.Source: Plan with Feedback · p. 3

3 Foundation Models for Low-Level Control

The survey frames low-level control as learning policies that output robot actions, including target joint positions and six-degree-of-freedom poses. It groups foundation-model approaches by the component emphasized in learning.

3 Foundation Models for Low-Level Control

  • Foundation models are used for low-level control by leveraging recognition, reasoning, and generalization to generate execution commands.The survey also notes that internet-scale data offers a promising route for training robotic foundation models.Source: §3 · p. 3
  • Low-level control aims to learn policies that output actions such as target joint positions or target 6 DoF poses.The survey distinguishes this execution focus from high-level planning.Source: §3 · pp. 3–4
  • The survey classifies low-level-control methods into policy learning, environment modeling, and representation learning.Table 1 presents a taxonomy of foundation models for low-level control and their properties.Source: §3 · p. 4, Table 1 · p. 4

3.1 Policy Learning

Policy learning addresses reinforcement learning's reward-design and sample-efficiency challenges and imitation learning's narrow-demonstration generalization problem. The survey describes foundation-model knowledge and large-scale robotic data as approaches used in these settings.

Reinforcement Learning

  • L2R uses language-model code generation to translate task descriptions into reward functions, which trajectory optimization and reinforcement learning use to learn dexterous-manipulation policies.The approach addresses the need for carefully designed rewards in reinforcement learning.Source: Reinforcement Learning · p. 4
  • FRL's FAC uses foundation-model value, reward, and policy priors for sample-efficient learning, robustness to noisy priors, and minimal human intervention.Q-Transformer explores transformer-based Q-function and multitask-policy learning from large offline datasets.Source: Reinforcement Learning · p. 4

Imitation Learning

  • RT-1 trained on 130k trajectories across 700 tasks collected from 13 robots and outperformed baselines on new test tasks and scenarios.The 17-month data collection and high-capacity transformer-based architecture with pretrained vision and language models supported reported generalization beyond training tasks.Source: Imitation Learning · p. 4

3.2 Environment Modeling

The survey treats environment models as complements to policy learning and distinguishes forward prediction of future states from inverse recovery of actions. Forward dynamics models physical rules but demands substantial data, while inverse dynamics is less data-hungry and performs better than other pre-training objectives when learning from scratch.

3.2 Environment Modeling

  • Environment models can complement policy learning by supporting reinforcement learning or trajectory optimization to generate better policies.Source: §3.2 · p. 4

Forward Dynamic Learning

  • Forward dynamic learning predicts next states from current or historical observations and actions, modeling physical rules but requiring substantial data for accuracy.SWIM learns from internet-scale human play videos using abstract target poses and waypoints, then can be fine-tuned on a small amount of robot data without rewards or task supervision.Source: Forward Dynamic Learning · p. 4

Inverse Dynamic Learning

  • Inverse dynamic learning performs better than other pre-training objectives from scratch and yields smoother, more discriminative representations with higher sample efficiency than forward dynamics.It is non-causal because it uses current and next states to recover past actions.Source: Inverse Dynamic Learning · p. 5

3.3 Representation Learning

The section describes foundation models as sources of image and text representations that can support downstream embodied AI, including text-aligned visual features. It reviews both frozen models used for specific functions and learned control models trained to handle multiple decision-making tasks.

3.3 Representation Learning

  • Vision and language foundation models provide image and instruction representations, including text-aligned features such as recognizing apples from an instruction.Source: §3.3 · p. 5

Frozen Foundation Models

  • Frozen foundation models can provide functions such as detection, segmentation, and tracking; SMS combines open-vocabulary detection with an LLM to build semantic occupancy distributions.Source: Frozen Foundation Models · p. 5

Learned Control Models

  • DualMind trains a generalist policy in two stages: self-supervised learning of common world knowledge, followed by conditional policy training on multimodal prompts.Its Encoder-Decoder Transformer uses Token-Leaner and cross-attention to enhance feature extraction.Source: Learned Control Models · p. 5

4 Datasets, Simulators, and Benchmarks

The survey organizes embodied-AI datasets into object, human, and robotic collections, alongside overviews of simulators and benchmarks. It highlights the distinct data content and collection constraints of these resources and describes simulator capabilities and benchmark scope.

4.1 Datasets

  • Embodied-AI datasets are grouped as object, human, and robotic data, with emphasis on 3D real-world information, object affordances, and robot-object interactions.Source: §4.1 · p. 5

Object Datasets

  • OmniObject3D contains 6,000 everyday objects across 190 types, with scanned shapes, high-fidelity appearance, and annotations including multi-view images and videos.Source: Object Datasets · p. 5

Human Datasets

  • Human activity videos capture real-world appearance, human-object interactions, and movement dynamics; Ego-Exo4D contains 1,422 hours across 5,625 instances and more than 800 participants.Source: Human Datasets · p. 5

Robotic Datasets

  • Robotic demonstrations are time-consuming and labor-intensive to collect, while RT-X combines data from 22 robots and reports positive zero-shot generalization and cross-morphology transfer.RT-X comprises 527 skills and 160,266 tasks collected through collaboration among 21 institutions.Source: Robotic Datasets · pp. 5–6

4.2 Simulators

  • Simulator development emphasizes realistic soft-body simulation, support for multiple controllers and action spaces, and parallel environments for efficient large-scale training.ManiSkill2 uses the material point method for soft-body interactions; ManiSkill2 and Isaac Gym are cited for computational efficiency with simultaneous environments.Source: §4.2 · p. 6

4.3 Benchmarks

  • FMB supports reproducible manipulation tests with 3D-printed objects, while ManiSkill2 provides 20 targeted operational tasks; a universal benchmark remains challenging across the field’s varied tasks and environments.Source: §4.3 · p. 6

5 Discussions and Future Directions

The survey characterizes foundation-model robotics as nascent and identifies challenges in long-horizon planning, reliability, data, computation, and safety. It reviews proposed directions while noting boundaries such as simulator-to-real distribution shifts and unresolved recovery from out-of-distribution states.

5 Discussions and Future Directions

  • The field remains in a nascent phase, with practical deployment in real-world environments still impeded by numerous challenges.Source: §5 · p. 6

5.1 Synergies of Planning and Control

  • Long-horizon tasks challenge single policies because errors accumulate; the survey calls for planning and control to collaborate and describes RT-X transfer and Skill Transformer’s joint skill-and-action prediction.Skill Transformer models long-horizon sequences and dependencies from visual observation history without isolating every subtask.Source: §5.1 · p. 6

5.2 Hallucination of Foundation Model

  • Foundation-model hallucinations can produce nonexistent objects or misleading instructions, creating potential malfunctions and safety risks when models guide physical agents.Source: §5.2 · p. 6

5.3 Efficient Data Collection

  • Robotic data collection is costly and slow; simulator training and human activity data offer alternatives, but simulation gaps persist and human-robot morphology differences complicate alignment.The survey notes that Sim2Real cannot fully compensate for shifts in appearance and dynamics; RT-1 took 17 months to collect 130,000 episodes.Source: §5.3 · pp. 6–7

5.4 Augmenting Existing Datasets

  • Image augmentation can improve generalization, but conventional transformations may damage semantic information and image-based augmentation adds no robot trajectories, leaving recovery from dangerous or OOD states unresolved.ROSIE uses a VLM to identify task-relevant image regions and a diffusion model to augment them from textual instructions.Source: §5.4 · p. 7

5.5 Efficiency on Computation and Deployment

  • Large foundation models demand substantial training and inference resources, while cloud deployment can conflict with privacy and real-time needs, motivating compression and lightweight models for edge devices.Source: §5.5 · p. 7

5.6 Safety and Interpretation of Robot

  • Real-world robot safety remains difficult because theoretical guarantees may assume simple, low-dimensional settings and OOD detection methods still struggle to generalize and provide recovery strategies.Source: §5.6 · p. 7

6 Conclusion

The survey reviews foundation-model applications in embodied AI, especially autonomous robotic manipulation, and organizes the literature around high-level planning and low-level control. It also surveys experimental resources and identifies challenges and promising research directions.

6 Conclusion

  • The review categorizes foundation-model research in autonomous manipulation by high-level planning strategies and low-level control mechanisms.Source: §6 · p. 7
  • It surveys commonly used robots, simulators, and benchmarks in experimental research on foundation models for embodied AI.Source: §6 · p. 7
  • The survey discusses key challenges and outlines promising research directions for advancing foundation models in robotics toward generalized embodied AI.Source: §6 · p. 7
  • It calls for concerted effort from academic and industrial spheres to advance foundation models in robotics.Source: §6 · p. 7

Related papers

Loading 2402.02385v1…