Source-linked AI summary
Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis
Yafei Hu, Quanting Xie, Vidhi Jain, Jonathan Francis, Jay Patrikar, Nikhil Keetha, Seungchan Kim, Yaqi Xie, Tianyi Zhang, Hao-Shu Fang, Shibo Zhao, Shayegan Omidshafiei, Dong-Ki Kim, Ali-akbar Agha-mohammadi, Katia Sycara, Matthew Johnson-Roberson, Dhruv Batra, Xiaolong Wang, Sebastian Scherer, Chen Wang, Zsolt Kira, Fei Xia, Yonatan Bisk
TL;DR
General-purpose robots remain constrained by task-specific models, labeled data requirements, and weak robustness under distribution shifts. This survey formulates foundation models for robotics, reviews both transferred and robotics-specific models, and synthesizes their methods and evidence. It finds promising applications alongside persistent limitations in generalization, embodiment, safety, reproducibility, and data coverage.
Problem
General-purpose robotics is limited by task-specific systems that require extensively labeled data and struggle to generalize robustly across real-world environments.
Method
The survey defines foundation models for robotics as both applications of existing vision and language models and robotics-specific models trained with robot-generated data, then reviews methods and meta-analyzes experiments.
Results
The reviewed field shows promising foundation-model applications, but end-to-end robotics models still lose 21%–31% performance on unseen tabletop manipulation tasks and 14%–18% under disturbances.
Takeaways & Limitations
Foundation models provide a broad framework for open-world perception, planning, and action, while practical general-purpose robotics still requires broader embodiments, data, simulators, and improved robustness.
Takeaways & Limitations
The review was checkpointed on September 1st 2024 and may omit relevant work or contain inaccuracies because the field and literature are rapidly changing.
Abstract
from arXiv · showhide
Building general-purpose robots that operate seamlessly in any environment, with any object, and utilizing various skills to complete diverse tasks has been a long-standing goal in Artificial Intelligence. However, as a community, we have been constraining most robotic systems by designing them for specific tasks, training them on specific datasets, and deploying them within specific environments. These systems require extensively-labeled data and task-specific models. When deployed in real-world scenarios, such systems face several generalization issues and struggle to remain robust to distribution shifts. Motivated by the impressive open-set performance and content generation capabilities of web-scale, large-capacity pre-trained models (i.e., foundation models) in research fields such as Natural Language Processing (NLP) and Computer Vision (CV), we devote this survey to exploring (i) how these existing foundation models from NLP and CV can be applied to the field of general-purpose robotics, and also exploring (ii) what a robotics-specific foundation model would look like. We begin by providing a generalized formulation of how foundation models are used in robotics, and the fundamental barriers to making generalist robots universally applicable. Next, we establish a taxonomy to discuss current work exploring ways to leverage existing foundation models for robotics and develop ones catered to robotics. Finally, we discuss key challenges and promising future directions in using foundation models for enabling general-purpose robotic systems. We encourage readers to view our living GitHub repository 2 of resources, including papers reviewed in this survey, as well as related projects and repositories for developing foundation models for robotics.
1 Overview
The survey examines foundation models as a potential route beyond task-specific robotic systems, reviewing their application to robotics and robotics-specific model development. It organizes prior work, analyzes real-world evidence, and identifies unresolved generalization, reproducibility, embodiment, and data challenges.
- Task-specific robotic systems require extensive labeled data and struggle with generalization when deployed across scenes, tasks, and distribution shifts.
- Foundation models from NLP and CV offer potential for open-world perception, task planning, and motion control in robotics.
- Important open challenges include poor reproducibility, limited multi-embodiment generalization, inadequate feasibility modeling, underexplored world dynamics and symbolic reasoning, and insufficient large-scale real-world data and simulators.
- The survey covers both applying existing vision and language models, mainly through zero-shot and in-context learning, and developing robotics foundation models from robot-generated data.
- Its review includes unimodal, multimodal, and robotics foundation models evaluated on physical robots, high-fidelity simulations, or real robotics datasets.
2 Problem Formulation and Preliminaries
The paper formulates a robotics foundation model as a function mapping sensory inputs and context to task-relevant outputs. It highlights unresolved questions about multimodal inputs, data and prompting, and recovery when planned actions fail.
- A robotics foundation model is defined as f, which maps sensory inputs x_t and context c to outputs y_t for downstream robotic tasks.
- Sensory input x may include visual, textual, audio, haptic, geometric, or multimodal observations, while context c specifies the task or robot embodiment.
- The output y can represent object poses, task plans, next states, rewards, or control signals such as target end-effector poses.
- When y is a task plan, the formulation often assumes perfect execution of low-level components, leaving timely recovery from execution failures unresolved.
- Open methodological questions concern collecting demonstrations, offline or online data, reward functions, prompting examples, training, and fine-tuning.
3 Challenges on General-purpose Robots
The survey identifies five broad challenges for general-purpose robots, spanning perception, data, planning and control, task specification, and uncertainty and safety. Foundation models may help, but current data coverage and safety mechanisms remain constrained.
- General-purpose robots must overcome limited perception, scarce diverse data, engineered planning and control models, difficult task grounding, and uncertainty and safety problems.
- 3.2 Data Scarcity: Robot datasets remain far from internet scale; in Open-X Embodiment, 55 of 73 datasets concern single-arm manipulation, while only one covers quadruped locomotion and one bi-manual tasks.
- 3.2 Data Scarcity: Existing data is dominated by tabletop manipulation with toy kitchen objects and human expert collection through virtual-reality or haptic devices, limiting coverage of real-world variation.
- 3.3 Planning and Control: Classical planning and control depend on engineered environment, dynamics, maps, object states, motion primitives, or predefined rules, restricting adaptation to varied tasks and settings.
- 3.4 Task Specifications: Task specifications must be grounded in the robot’s world model and capabilities, with language prompts, goal images, or rewards offering possible interfaces for resolving ambiguity.
- 3.5 Uncertainty and Safety: Provable safety methods provide bounded guarantees but can produce sub-optimal behavior, leaving safe learning from experience in novel environments open.
4 Review of Current Research Methodologies
The survey reviews foundation-model methods through two categories: applying existing vision and language models to robotic functions, and developing robotics foundation models. It further organizes methods by perception, planning, and action generation, contrasting modular and end-to-end designs.
- The review distinguishes foundation models used in robotics from robotics foundation models, including zero-shot applications and models trained or initialized for robotics.
- Methods are organized around perception, planning, and action generation, with motion planning and control treated together as action generation at different levels.
- Existing vision and language models typically serve one robotic function in modular systems, such as open-set perception plugged into downstream planners and controllers.
- Robotics foundation models more often use end-to-end differentiability, blur module boundaries, and can unify multiple robot functions.
4.1 Foundation Models used in Robotics
Existing vision-language and language foundation models are applied across robotic perception, planning, motion planning, and action generation, but effective grounding from semantic outputs to complex actions remains unresolved.
- VFMs and VLMs in Robot Perception: Foundation models support robotic perception by combining open-set semantic information with spatial representations for mapping, navigation, manipulation, and scene understanding.Methods use language-queryable maps, multimodal 3D representations, semantic masks, and grounded topological graphs.
- VFMs and VLMs in Robot Perception: Dense foundation-model features can support indoor localization and place recognition, with AnyLoc achieving broad applicability across locations, times, and viewpoints.LEXIS and FM-Loc use CLIP features for localization, while AnyLoc combines dense features with unsupervised aggregation.
- LLMs and VLMs in Task Planning: Task-level planning uses LLMs to decompose goals into actionable steps and compose pre-trained skills, sometimes searching hierarchical scene graphs to reduce planning horizons.SayCan exemplifies high-level skill composition, while SayPlan combines semantic subgraph search with classical path planning.
- LLMs and VLMs in Action Generation: Direct robot control remains difficult because individual waypoints, joint angles, and other actions lack the semantic compositionality available for high-level task plans.Proposed interfaces include frontier evaluation, affordance or value maps, few-shot joint-angle generation, contact-pattern commands, and language-generated rewards.
- Action Grounding: Pretrained skill libraries favor dexterity and precision, whereas map- or constraint-based grounding favors task flexibility but has mainly been demonstrated on simpler 2D pick-and-place tasks.The survey identifies balancing task diversity with action complexity as an unresolved interface-design goal.
- LLMs and VLMs in Task Planning: Chain-of-thought prompting exposes intermediate planning steps and can improve planning by exploiting broader context windows and autoregressive language-model structure.The proposed explanation links semantic similarity between instructions and intermediate steps to improved planning behavior.
4.2 Robotics Foundation Models (RFMs)
Robotics foundation models use robot-generated data and multimodal pretraining to support individual capabilities or multiple robotic modules, with emerging evidence of control, navigation, and multi-task generalization.
- Robotics Foundation Models (RFMs): Robotics foundation models are trained on robotics data to solve robotic tasks, ranging from single-purpose perception, planning, or control models to general-purpose multi-module systems.The survey distinguishes models dedicated to one capability from models spanning perception, control, and non-robotic tasks.
- Robotic Action Foundation Models: Robotic action foundation models map raw sensory observations such as images or videos to control outputs applied to robot end-effectors, with reported generalization in manipulation.The cited examples include RT-series models, RoboCat, and MOO.
- Imitation Learning: Imitation learning extends from single-skill imitation toward multi-task learning conditioned on goals expressed through images, human demonstrations, or language.The review presents one-shot imitation learning as a foundation for learning multiple robotic tasks.
- Reinforcement Learning at Scale: Large-scale robotic datasets have enabled offline reinforcement-learning approaches that learn policies from robot-collected data and extend toward multitask learning.QT-OPT is identified as an early large-scale offline reinforcement-learning model using Q-learning and robot-farm data.
- Vision and Language Pre-training: Vision, language, audio, and tactile pretraining provide alternative routes for learning representations or control policies from internet, egocentric, trajectory, and sensor data.RT-2 and MOO use vision-language backbones for control, while other work pretrains on robot trajectories, audio, or tactile sensing.
- Robotics Motion Planning Foundation Models: Motion-planning foundation models use large-scale heterogeneous data to predict high-level navigation actions while relying on rough topological maps rather than accurate metric maps and localization.Their reported generalization is demonstrated in visual navigation tasks.
- General-purpose Robotics Foundation Models: Gato and PaLM-E demonstrate multimodal, multi-task, and multi-embodiment models that combine robotic and non-robotic capabilities, while PACT targets several downstream navigation tasks.These systems suggest that perception and planning can be merged within shared model weights, although universal robotics coverage is not established.
4.3 How do Foundation Models Help Solve Robotics Challenges
Foundation models address robotics challenges through generalization, data-efficient learning, reduced reliance on explicit models, and more natural task specification, while uncertainty and safety remain underexplored.
- Foundation Models for Generalization: Foundation models support generalization across perception, task planning, motion planning, and control, with visual models focused on perception, LLMs on planning and control, and RFMs on planning and action generation.The survey also identifies foundation models as useful for task specification and notes that improved perception generalization can reduce the need for domain adaptation or fine-tuning.
- Foundation Models for Data Scarcity: Foundation models can help robots learn and adapt to new tasks from limited task-specific data, including through generated robot trajectories and simulations.
- Foundation Models to Relieve the Requirement of Models: Few-shot and zero-shot capabilities may reduce reliance on maps, world models, and dynamics models through language-generated plans and model-free end-to-end control policies.
- Foundation Models for Task-Specification: Foundation models enable more natural task specifications through language, images, demonstrations, rewards, sketches, and other multimodal inputs for robotic control.Some systems encode camera images, instructions, and motor commands into compact token representations for real-time mobile-manipulator control.
- Foundation Models for Uncertainty and Safety: Uncertainty and safety remain largely underexplored, although uncertainty alignment, chain-of-thought prompting, open-vocabulary learning, and hallucination recognition are identified as possible avenues.
5 Review of Current Experiments and Evaluations
The survey reviews datasets, simulators, foundation models, and evaluation practices, finding expanding but uneven empirical coverage, persistent generalization gaps, and important deployment constraints.
- Real World Robotics Datasets: Real-world robotics datasets provide diverse objects and multimodal inputs, but limited incentives for uploading sensor-action data hinder the data flywheel needed for internet-scale growth.Examples include RoboNet, Bridge-V1 with 7200 hours of demonstrations, and Bridge-V2 with 60,096 trajectories across 24 environments.
- Real World Robotics Datasets: Dataset standardization efforts unify navigation or embodiment interfaces, while real-world robotics data is growing but remains far smaller than internet-scale language and vision corpora.GNM uses a waypoint-based navigation interface, and RT-X uses a 7-DoF end-effector pose as a cross-embodiment reference.
- Analysis of Current Method Evaluation: Simulators enable large-scale parallel data collection with varied visual, physical, and interactive properties, but sim-to-real gaps and limited environment and object diversity remain barriers.
- Analysis of Current Method Evaluation: Foundation-model robotics research emphasizes tabletop and mobile pick-place manipulation, while dexterous manipulation, locomotion, and other low-level action outputs receive substantially less exploration.Existing low-level action studies mainly use tabletop manipulation with 7-DoF end-effector actions; direct joint-angle outputs remain less studied.
- Analysis of Current Method Evaluation: 21% to 31% performance drops occur on unseen tabletop tasks, while disturbances produce 14% to 18% drops, showing that generalization and robustness remain incomplete.
- Analysis of Current Method Evaluation: GPT-4 and GPT-3 are frequently used for few-shot prompting, CLIP and ViLD bridge image-text representations, PaLM-family models support planning, and RT-1 serves as a manipulation base model.
6 Discussions and Future Directions
The survey identifies open directions for robotics foundation models, including embodiment-aware control, world modeling, richer multimodal data, continual learning, and safety. It also argues for flexible architectures and improved hardware and data collection to support broader generalization.
- Grounding for Robot Embodiment: Foundation models need grounding interfaces that connect concepts to robot actions while accounting for how embodiment changes interaction strategies.Natural language and code interfaces are limited, and the same task can require different actions for different robot bodies.
- Safety and Uncertainty: Real-world robots also require multimodal sensing and safety mechanisms that reflect physical properties, social context, human intent, and formal guarantees.Visual data alone cannot capture properties such as friction or scent, while safety constraints must remain aligned with capability and context.
- Architecture and Optimization: A robotics foundation model should support flexible modular components with unified learning rather than prematurely choosing modular or end-to-end architectures.The survey distinguishes architecture from optimization and emphasizes shared representations alongside functionality-specific modules.
- World Models: Foundation world models could support model-based planning, trajectory optimization, and more explainable solutions if they achieve precise, generalizable representations.The survey links this direction to action- and language-conditioned video generation as an emerging precursor.
- Novel Robotics Platforms and Multi-sensory Information: Robotics datasets and platforms remain concentrated on gripper-based single-arm manipulation, while dexterous hardware is costly, failure-prone, and difficult to deploy beyond simulation.The survey calls for richer hardware and more efficient human-demonstration platforms to expand the conceptual space of robotics foundation models.
- Continual Learning: Continual learning must address catastrophic forgetting while meeting real-time constraints through data mixing, task inference, stable online learning, and control integration.Periodic retraining on all accumulated data can mitigate forgetting but requires substantial computation and data.
Meta Analysis Tables
The survey organizes its meta-analysis tables by major robotics task categories.
- Meta Analysis Tables: The meta-analysis covers manipulation, dexterous manipulation, mobile manipulation, locomotion, navigation, and multi-task learning.
Disclaimer
This survey is a time-bounded literature review that may omit relevant work and contain inaccuracies as the field evolves.
- Disclaimer: The review was checkpointed on September 1st 2024 and may have missed relevant work or included inaccuracies.