PaperlayerView exact revision on arXiv

Source-linked AI summary

Foundation Models in Robotics: Applications, Challenges, and the Future

Roya Firoozi, Johnathan Tucker, Stephen Tian, Anirudha Majumdar, Jiankai Sun, Weiyu Liu, Yuke Zhu, Shuran Song, Ashish Kapoor, Karol Hausman, Brian Ichter, Danny Driess, Jiajun Wu, Cewu Lu, Mac Schwager

arXiv:2312.07843v1cs.RO

TL;DR

Robotics has relied on task-specific datasets, motivating the question of how foundation models can improve general-purpose robot autonomy. This survey synthesizes applications of foundation models across perception, decision-making, and control, alongside their challenges and risks. It concludes that generalization, zero-shot, multimodal, and scalable capabilities are promising, but adoption is constrained by data, safety, uncertainty, variability, reproducibility, and deployment limitations.

  • Problem

    Robotics lacks sufficient general-purpose foundation models because robot data are scarce, settings are highly variable, multimodal alignment is difficult, and reliability in autonomous operation is uncertain.

  • Method

    The survey reviews literature using or building foundation models for robotics, covering applications, challenges, risks, and future research directions.

  • Results

    The surveyed literature reports promising foundation-model applications in robot decision-making, planning, control, and perception, including code generation, pretrained visual control, multitask control pretraining, and semantic navigation.

  • Takeaways & Limitations

    Generalization, zero-shot, multimodal, and scalability capabilities have the potential to transform robotics, provided research addresses deployment and evaluation challenges.

  • Takeaways & Limitations

    Multimodal foundation models assume tokenizable, standardized modalities and sufficient paired data, conditions that are not met for some robotic modalities such as 3D point clouds.

Abstract

from arXiv · show

We survey applications of pretrained foundation models in robotics. Traditional deep learning models in robotics are trained on small datasets tailored for specific tasks, which limits their adaptability across diverse applications. In contrast, foundation models pretrained on internet-scale data appear to have superior generalization capabilities, and in some instances display an emergent ability to find zero-shot solutions to problems that are not present in the training data. Foundation models may hold the potential to enhance various components of the robot autonomy stack, from perception to decision-making and control. For example, large language models can generate code or provide common sense reasoning, while vision-language models enable open-vocabulary visual recognition. However, significant open research challenges remain, particularly around the scarcity of robot-relevant training data, safety guarantees and uncertainty quantification, and real-time execution. In this survey, we study recent papers that have used or built foundation models to solve robotics problems. We explore how foundation models contribute to improving robot capabilities in the domains of perception, decision-making, and control. We discuss the challenges hindering the adoption of foundation models in robot autonomy and provide opportunities and potential pathways for future advancements. The GitHub project corresponding to this paper (Preliminary release. We are committed to further enhancing and updating this work to ensure its quality and relevance) can be found here: https://github.com/robotics-survey/Awesome-Robotics-Foundation-Models

I. INTRODUCTION

Foundation models extend robotics beyond task-specific training by bringing internet-scale pretraining, multimodal representations, and zero-shot capabilities to perception, decision-making, and control. Their adoption remains constrained by limited robot-relevant data, safety and uncertainty concerns, real-time requirements, and substantial training costs.

  • Motivation: Foundation models are pretrained on extensive internet-scale data and can be fine-tuned for diverse downstream robotics tasks.The survey discusses LLMs, VLMs, ALMs, and VNMs across perception, prediction, planning, and control.
  • Motivation: Traditional robotics models rely on limited, task-specific datasets, whereas foundation models may improve adaptability, generalization, and performance through knowledge transfer.The survey identifies these potential benefits based on results from other domains.
  • Robotics Applications: Multimodal foundation models can align heterogeneous sensor data into compact representations supporting robot perception, decision-making, and control.The survey also associates multimodal models with cross-modal understanding for zero-shot classification, detection, and 3D tasks.
  • Robotics Applications: LLMs and VLMs can assist high-level task specification and planning, while language-conditioned imitation and reinforcement learning may improve contextual understanding and data efficiency.Language-driven rewards can provide shaped rewards for reinforcement-learning agents.
  • Challenges: Open challenges include robot-data scarcity, high variability across embodiments and environments, uncertainty quantification, safety evaluation, and real-time inference.The survey specifically highlights hallucinations, lifecycle safety testing, and inference latency as deployment concerns.
  • Survey Scope: The survey reviews foundation-model applications, challenges, risks, and future opportunities while comparing existing applications, algorithms, and architectures from a broad robotics perspective.Its scope covers background, robotics, and robotics-adjacent papers.

B. Large Language Model (LLM) Examples and Historical Context

LLMs have scaled through larger parameter counts, context windows, and training datasets, with earlier models achieving strong benchmark performance.

  • LLMs have billions of parameters and are trained on trillions of tokens.
  • GPT-2 and BERT achieved state-of-the-art performance on the Winograd Schema and GLUE benchmarks, respectively.
  • Successors including GPT-3, LLaMA, and PaLM typically exceed 100 billion parameters, use context windows over 1000 tokens, and train on tens of terabytes of text.

C. Vision Transformers

Vision Transformers represent images as tokenized patches, while self-supervised methods such as DINO and DINOv2 provide pretrained visual features for robotics-related vision tasks.

  • Vision Transformers divide images into fixed-size patches, flatten them into vectors, and treat the patches as image tokens.
  • DINO trains Vision Transformers through label-free knowledge distillation, with a student network learning to mimic a teacher network.
  • DINOv2 offers pretrained visual models trained on LVD-142M using different Vision Transformers and a discriminative self-supervised method.
  • DINOv2 provides visual features at image level for detection and pixel level for segmentation.

D. Multimodal Vision-Language Models (VLMs)

Vision-language models process images and text together, using aligned representations to support multimodal learning and more efficient visual-text pretraining.

  • Visual-language models accept both images and text as input, while CLIP compares similarities between textual descriptions and images.
  • CLIP uses internet-scale image-text pairs to capture semantic information.
  • BLIP jointly optimizes image-text contrastive, image-text matching, and language modeling objectives during pretraining.
  • FLIP randomly masks and removes many image patches during CLIP training to improve efficiency while maintaining performance.

E. Embodied Multimodal Language Models

Embodied multimodal language models incorporate real-world sensing and actuation into pretrained language models, while related large models support visual generation and robot decision-making applications.

  • Embodied language models incorporate real-world sensor and actuation modalities into pretrained large language models.
  • Typical vision-language models are trained on general tasks such as image captioning or visual question answering.
  • Web-scale diffusion models such as DALL-E and DALL-E2 provide zero-shot text-to-image generation from hundreds of millions of image-caption pairs.
  • DALL-E2 uses a prior to generate a CLIP image embedding from a text caption, followed by a decoder conditioned on that embedding.
  • LLMs and VLMs may support robot decision-making, planning, and control.
  • LLMs can facilitate task specification by allowing robots to receive and interpret high-level human instructions.

A. Robot Policy Learning for Decision Making and Control

Robot policy learning uses language, visual representations, and foundation-model-generated data to improve task specification, scalability, and adaptation. The surveyed approaches include imitation learning, reinforcement learning, and hierarchical methods for manipulation and other sequential tasks.

  • Language-conditioned imitation learning: Language-conditioned imitation learning maps states and language instructions to actions using annotated demonstration trajectories.At test time, the policy produces actions in closed loop from observations and instructions.
  • Language-conditioned imitation learning: Play-LMP learns reusable latent plans from unlabeled teleoperated play data and decodes them into policies for user-specified tasks.The surveyed analysis reports greater robustness to perturbations than expert positive demonstrations in this setting.
  • Language-conditioned imitation learning: CACTI combines limited expert demonstrations, Stable Diffusion augmentation, pretrained visual representations, and multi-task imitation-policy training.It targets multi-task, multi-scene kitchen manipulation in simulation and the real world.
  • Language-conditioned imitation learning: Hierarchical imitation methods such as MimicPlay learn high-level latent plans from human play and low-level commands from limited teleoperation.This combination is intended to reduce training cost for long-horizon visuomotor manipulation.
  • Reinforcement learning: RL foundation-model approaches pretrain agents on diverse tasks or use LLMs to decompose complex tasks into sub-tasks for transformer-based control.These approaches combine supervised and reinforcement learning in some surveyed systems.

B. Language-Image Goal-Conditioned Value Learning

Language-image value learning connects semantic goals to robot control through pretrained visual or vision-language representations and value functions. The surveyed systems use these representations for goal specification, task grounding, and motion planning.

  • Pretrained value representations: R3M pretrains visual representations from diverse human videos for use as a frozen perception module in robot manipulation policy learning.Its purpose is to transfer visual representations to downstream manipulation tasks.
  • Pretrained value representations: VIP learns visual goal-conditioned value functions from unlabeled videos without requiring video-language alignment.Time-contrastive learning captures temporal dependencies and supports zero-shot reward specification.
  • Language-image value learning: LIV extends VIP by learning multimodal vision-language value functions for language-goal and image-goal manipulation tasks.Policy learning is decoupled from representation pretraining, and small in-domain robot datasets can ground language contextually.
  • Language-image value learning: SayCan combines LLM task grounding with learned value functions to connect language-based planning to the physical environment.The value function helps associate language-model information with locations and objects.
  • Language-image value learning: VoxPoser converts RGB-D observations and language instructions into 3D affordance and constraint maps that guide motion planning.The resulting 3D value maps serve as objective functions for synthesizing everyday manipulation trajectories without prior task training or instruction.
  • Language-based planning: LLMs can generate high-level task plans, while prompting methods such as ProgPrompt and Code-as-Policies generate action sequences or robot policy code.Code-as-Policies uses natural-language commands and few-shot prompting to invoke control primitives from perception outputs.

D. In-context Learning (ICL) for Decision-Making

In-context and transformer-based methods apply pretrained models to planning, navigation, and control using prompts, web data, robot trajectories, and multi-embodiment datasets. The surveyed results emphasize transfer across tasks, concepts, domains, and robot platforms.

  • In-context learning: In-context learning uses examples in prompts without parameter optimization, while Chain-of-Thought introduces intermediate reasoning steps for complex problems.The survey also notes challenges involving ambiguity, domain-specific knowledge, transparency, and explainability.
  • Robot control: Self-supervised visual pretraining from real-world images and in-the-wild videos produces representations effective across motor-control tasks, embodiments, and real-world robotic tasks.The cited studies report applicability without task-specific fine-tuning of the pretrained encoder in one setting.
  • Robot transformers: RT-1 trains on over 130k real-world robotic experiences spanning more than 700 tasks collected with 13 robots over 17 months.It receives images and natural-language instructions and outputs discretized base and arm actions.
  • Robot transformers: RT-2 co-fine-tunes vision-language models on web and robot data to produce low-level closed-loop control from language instructions and camera observations.Robot actions are represented as text tokens and include end-effector motion, gripper extension, and termination.
  • Robot transformers: RT-X combines data from 22 robots, 21 institutions, 527 skills, and 160266 tasks in a standardized multi-embodiment dataset.The resulting RT-1- and RT-2-based models demonstrate positive transfer across robotic domains.
  • Robot control: SMART uses self-supervised multi-task control-transformer pretraining followed by fine-tuning across sequential decision-making tasks.Experiments report improved learning efficiency, robustness to distribution shifts, and effectiveness with low-quality pretraining data.
  • Open-vocabulary navigation: Navigation systems combine pretrained language, vision-language, visual-navigation, audio-language, or image-goal models for open-vocabulary and zero-shot operation.LM-Nav demonstrates natural-language visual navigation outdoors, while AVLMaps combines multimodal cues into 3D heatmaps.

2) Open-Vocabulary Manipulation:

Open-vocabulary manipulation targets previously unseen objects and environments using multimodal prompting, generative models, and foundation-model-based perception. The surveyed systems unify diverse task specifications or generate rearrangement plans without additional training.

  • Open-vocabulary manipulation: VIMA conditions a transformer policy on multimodal prompts that combine textual and visual tokens for diverse manipulation tasks.VIMA-BENCH provides a simulator for evaluating multimodal robot learning across object and texture collections.
  • Open-vocabulary manipulation: RoboCat is a 1.18B-parameter decoder-only transformer that operates different robotic arms, learns some tasks from as few as 100 demonstrations, and improves using self-generated data.It is presented alongside VIMA as a system for broad manipulation capabilities.
  • Open-vocabulary manipulation: StructDiffusion uses segmentation, language, and partial-view point clouds to construct goal configurations for previously seen or unseen objects.A multimodal transformer combines word and point-cloud embeddings after the scene is decomposed into objects.
  • Open-vocabulary manipulation: DALL-E-Bot performs zero-shot autonomous scene rearrangement with a pretrained DALL-E2 image-diffusion model.Its pipeline represents objects using segmentation masks, captions, and CLIP visual features before generating a rearrangement description.
  • Foundation-model perception: Foundation models convert high-dimensional sensory inputs, including images, video, audio, and language, into abstract structured representations for robot understanding and interaction.Open-vocabulary object detection methods such as GLIP use language-aware visual representations to identify previously unseen objects.

1) Object Detection:

Foundation models support open-vocabulary detection, 3D classification, and promptable segmentation by aligning visual representations with language. Their flexibility broadens recognition capabilities, but real-time and fine-grained performance remain challenges.

  • Object Detection: OWL-ViT performs open-vocabulary detection using contrastive image-text pre-training and end-to-end detection fine-tuning.It supports multiple text-based or image-driven queries, unlike GLIP’s single-query formulation.
  • Object Detection: Grounding DINO extends closed-set detection to open-set detection by fusing vision and language.Its multi-stage multimodal feature fusion contributes to outperforming GLIP in open-set object detection.
  • 3D Classification: PointCLIP transfers CLIP’s 2D image knowledge to 3D point-cloud classification by aligning multiview depth maps with text.The method projects points onto predefined image planes before applying CLIP’s visual encoder.
  • 3D Classification: ULIP aligns language, images, and point clouds in a unified representation using automatically synthesized ShapeNet55 triplets.Its contrastive pretraining improves 3D recognition and remains agnostic to the downstream 3D model architecture.
  • Semantic Segmentation: SAM introduces promptable segmentation with a composable design evaluated across five zero-shot transfer tasks.However, SAM cannot run in real time for robotic applications, while FastSAM and MobileSAM offer comparable performance at faster inference speeds.

1) Language Grounding in 3D Scene:

Language grounding methods combine pretrained vision-language features with 3D representations to connect words, objects, locations, and interactions. These approaches address limited 3D-language data but inherit spatial, camera-pose, and scene-editing constraints.

  • Language Grounding in 3D Scene: CLIP can be combined with NeRFs to extract semantic information from environments, although NeRFs themselves are generally not foundation models.NeRF reconstructs 3D geometry and appearance from posed 2D images, while CLIP supplies semantic features.
  • Language Grounding in 3D Scene: LERF grounds CLIP embeddings in a dense multiscale 3D field that produces semantic relevancy maps.Its inputs include 3D position, viewing direction, and scale; outputs include color, density, DINO features, and CLIP features.
  • Language Grounding in 3D Scene: LERF and related representations inherit limitations including weak spatial-relationship reasoning and dependence on known camera poses.CLIP queries may also highlight opposite terms similarly, reflecting a bag-of-words-like weakness.
  • Language Grounding in 3D Scene: VLMaps backprojects LSeg pixel embeddings into top-down grid cells without additional training.Combined with a code-writing LLM, the representation supports navigation using landmarks and spatial references.
  • Language Grounding in 3D Scene: 3D VLM development is constrained by scarce 3D data paired with language descriptions.FeatureNeRF addresses this by distilling 2D vision foundation models into continuous 3D semantic feature volumes.
  • Language Grounding in 3D Scene: 3D-LLM uses 2D VLM backbones with mapped multiview 3D features and prompting to generate diverse 3D-language data.The resulting model accepts 3D representations and performs multiple 3D-related tasks.

D. Learned Affordances

Learned affordances and predictive world models connect visual or linguistic representations to robot interaction, planning, and control. Foundation-model approaches span generative interaction synthesis, multimodal state modeling, language-guided rewards, and embodied sequence reasoning.

  • Learned Affordances: Affordance models represent potential object or environmental interactions such as pushing, pulling, sitting, and grasping.Detecting these affordances bridges perception and action.
  • Learned Affordances: Affordance Diffusion generates hand-object interaction images through separate layout and content stages.The layout network predicts spatial arrangement, while the content network synthesizes how the interaction occurs.
  • Learned Affordances: VRB learns visual affordances from internet videos of human behavior and integrates them with four robot-learning paradigms.These include offline imitation, exploration, goal-conditioned learning, and reinforcement-learning action parameterization.
  • Learned Affordances: World models predict how world states change under agent actions and can support downstream planning through action-sequence optimization.The survey focuses on models trained with foundation-model-scale diversity or data volume.
  • Learned Affordances: EmbodiedGPT combines a language model with EgoCOT videos and instructions to improve embodied planning, control, captioning, and visual question answering.Prefix adapters augment a 7B language model for planning generation.
  • Learned Affordances: Foundation models can provide rewards or task suggestions that guide reinforcement-learning agents toward meaningful behavior.ELLM uses the agent’s current-state description to generate goals without a human in the loop.

A. Generalist AI

Generalist AI aims to deploy robots or embodied agents across diverse environments and tasks. The surveyed approaches use generative environments, shared multimodal policies, and simulators or benchmarks to support this breadth.

  • Generalist AI: Deploying robots across varied non-factory environments and tasks remains a long-standing challenge.The challenge motivates work on generalist robots.
  • Generalist AI: Generative simulators provide one proposed route toward developing generalist AI and producing realistic interactive behavior.Generative Agents connects its architecture with ChatGPT to create a game environment.
  • Generalist AI: Gato is a multi-modal, multi-task, multi-embodiment generalist policy using the same network weights across environments and tasks.Its demonstrated capabilities include Atari, dialogue, image captioning, robotic block stacking, and simulated navigation.
  • Generalist AI: High-quality simulators and benchmarks support robotics development and simulation-to-real-world generalization.Gibson, iGibson, BEHAVIOR-1K, and Habitat 3.0 expand perception, household-task, and evaluation capabilities.
  • Generalist AI: RoboTHOR supports embodied-agent development and evaluation in both simulated and physical settings.Its simulation includes 75 training and validation scenes, plus 14 test-dev and 14 test-standard scenes with physical counterparts.

A. Overcoming Data Scarcity in Training Foundation Models for Robotics

Robotics foundation models face scarce, costly, and safety-sensitive data, motivating play data, simulation, generative augmentation, and multimodal relabeling. Deployment also requires addressing physical realism, skill-distribution limits, and latency.

  • Data scarcity: Robot-specific data is scarce relative to the internet-scale data used to train foundation models.This scarcity motivates techniques including play data and data augmentation.
  • Data collection: Play data replaces fully annotated expert demonstrations with unstructured, unlabeled, inexpensive teleoperated data that remains rich for learning.It avoids scene staging, task segmentation, and resetting to an initial state.
  • Generative augmentation: Diffusion-based inpainting augments manipulation datasets with unseen objects, backgrounds, and distractors guided by text.The generated data must be both visually diverse and physically feasible; unrealistic grasps can degrade downstream training.
  • Simulation: Simulation can efficiently provide multimodal robot data and precise ground-truth labels under changing objects, lighting, and weather conditions.TartanAir is cited as a navigation dataset collected with these variations.
  • Instruction augmentation: VLM-based instruction augmentation weakly relabels offline control datasets for language-conditioned policy learning.DIAL fine-tunes a VLM on annotated manipulation trajectories, scores annotation relevance, and produces new instruction labels.
  • Skill diversity: Existing robot transformers are limited by the observed skill distribution, while video motion data is proposed to support acquisition of new movements.This addresses the constraint that robots otherwise lack the capability to generate new movements.
  • Deployment: Foundation-model inference time remains a bottleneck for reliable real-time robotic deployment.The survey identifies computational efficiency as requiring further research because real-time capability is essential for robots.

C. Limitations in Multimodal Representation

Multimodal foundation models promise shared representations across robotic modalities, but tokenization, scarce paired data, hallucinations, calibration, and distribution shift constrain reliable deployment. The survey highlights conformal methods as one route to calibrated uncertainty for language-instructed robots.

  • Representation assumptions: Multimodal representation learning assumes heterogeneous modalities can be tokenized and aligned through cross-modal attention without losing information.Robotics modalities may violate this assumption when sufficient paired data is unavailable.
  • Data limitations: Robotics modalities such as 3D point clouds often lack sufficient paired data for direct alignment with text and other modalities.Converting modalities before alignment is one proposed workaround, but 3D foundation-model training remains challenging because data is scarce.
  • Reliability: LLM hallucinations can produce outputs that are factually incorrect, logically inconsistent, or physically infeasible, creating a reliability challenge for safety-critical robots.The survey asks how to provide assurances when these models are deployed in robotics.
  • Uncertainty: Uncertainty quantification must address both individual inputs and the distribution of future inputs encountered after deployment.Instance-level estimates can inform runtime decisions, while distribution-level estimates inform deployment suitability.
  • Calibration: Uncertainty estimates should be calibrated so prediction coverage or error bounds match their stated probabilities over future inputs.The desired guarantee depends on whether uncertainty is represented by prediction sets, output distributions, or distribution-level error bounds.
  • Calibration: Safety-critical robotics often requires Frequentist guarantees, which Bayesian uncertainty estimates do not necessarily provide because they depend on the chosen prior.A collision-probability bound should constrain the observed deployed collision rate when that is the operational goal.
  • Distribution shift: Distribution shift, including shifts induced by closed-loop robot actions, can invalidate uncertainty calibration learned on the training distribution.Actions change future states and observations, potentially altering the inputs received by perception models.
  • Methods: KNOWNO uses conformal prediction to provide calibrated instance-level and distribution-level uncertainty quantification for language-instructed robots.Its purpose is to help robots recognize uncertainty and request human assistance or clarification.

1) Pre-deployment safety tests:

Safe deployment requires testing foundation models and complete robotic systems before and during operation, despite open-ended outputs, unpredictable errors, simulation gaps, and changing robot settings. The survey identifies runtime monitoring, fallback policies, simulation, and red-teaming as complementary safeguards.

  • Pre-deployment safety tests: Pre-deployment tests must use scenarios not seen during training and cover diverse cases because foundation-model errors are difficult to predict beforehand.Open-ended outputs such as natural-language plans are also difficult to evaluate automatically in isolation.
  • Pre-deployment safety tests: Human and partially automated red-teaming are identified as future directions for evaluating foundation models in robotics.Existing non-robotics deployment cycles use thorough human red-teaming, while newer work partially automates the process with foundation models.
  • System-level evaluation: End-to-end robotic-system safety should be assessed in addition to evaluating the foundation model alone.Simulation is useful, but meaningful transfer requires sufficient fidelity and adequately representative test scenarios.
  • Runtime safety: Runtime monitoring can predict failures or detect out-of-distribution situations, triggering safety-preserving fallback behavior.These approaches use scenario-level prediction or experiences from small batches in novel distributions.
  • High variability in robotic settings: Robot platforms and environments vary in physical configuration, layout, objects, and capabilities, limiting generalization across embodiments, environments, and tasks.The survey calls for task-agnostic, cross-embodiment, open-ended pretraining on diverse robotic data.
  • Benchmarking and reproducibility in robotics settings: Reliance on real hardware complicates reproducibility because repeating results may require access to the exact equipment used.Many simulators simplify or omit contact physics while emphasizing high-level tasks and visual environments.
  • Benchmarking and reproducibility in robotics settings: Low-physics simulators create a sim-to-real gap and can make real-world performance depend on how low-level planning and control handle true physics.Even physics-based simulators may lack standardized simulation environments, according to the cited discussion.
  • Conclusion: The survey concludes that generalization, zero-shot, multimodal, and scalable capabilities remain promising, while data scarcity, variability, uncertainty, safety, and real-time performance remain concerns.It frames these concerns as priorities for future research rather than resolved deployment properties.
Loading 2312.07843v1…