Source-linked AI summary
Real-World Robot Applications of Foundation Models: A Review
Kento Kawaharazuka, Tatsuya Matsushima, Andrew Gambardella, Jiaxian Guo, Chris Paxton, Andy Zeng
TL;DR
Domain-specific training alone is insufficient for the diversity of robots, tasks, and environments. This review organizes foundation-model applications by modality and robotics function, with a focus on replacing components in existing systems. It identifies active research directions alongside challenges in generalization and real-world evaluation.
Problem
Domain-specific training alone is insufficient for robots spanning diverse bodies, tasks, and environments.
Source: §1 · pp. 1–2Method
Results
The review identifies active directions in applying existing foundation models to robot movement with more modalities, building larger-data and larger-model robotic foundation models, and exploring generalization across bodies and non-Transformer models.
Source: §6.1 · p. 22Takeaways & Limitations
The authors view open-world applications as a significant advancement toward robots beyond traditional capabilities, while generalization across environments and tasks remains a future challenge.
Source: §6.2 · p. 22Generalizing robots using foundation models across environments and tasks remains a future challenge.
Source: §6.2 · p. 22
Abstract
from arXiv · showhide
Recent developments in foundation models, like Large Language Models (LLMs) and Vision-Language Models (VLMs), trained on extensive data, facilitate flexible application across different tasks and modalities. Their impact spans various fields, including healthcare, education, and robotics. This paper provides an overview of the practical application of foundation models in real-world robotics, with a primary emphasis on the replacement of specific components within existing robot systems. The summary encompasses the perspective of input-output relationships in foundation models, as well as their role in perception, motion planning, and control within the field of robotics. This paper concludes with a discussion of future challenges and implications for practical robot applications.
1. Introduction
Robots are being applied across more environments and everyday tasks, while domain-specific training alone is insufficient for diverse robots, tasks, and environments. This review surveys foundation models as replacements for components of existing robot systems and organizes their applications across modalities and robotics functions.
1. Introduction
- Robot applications now span settings beyond mass production and everyday tasks including tidying, washing, wiping, and cooking.The introduction also notes small-batch, high-variety production, indoor spaces, and disaster sites.Source: §1 · p. 1
- Domain-specific training alone is insufficient for diverse robots, tasks, and environments, motivating a single pretrained system or module applicable across them.The review identifies foundation models as one solution to this need.Source: §1 · pp. 1–2
- Foundation models are trained on large amounts of data and can serve downstream tasks through in-context learning, fine-tuning, or zero-shot use.The review focuses on replacing specific components in existing robot systems.Source: §1 · p. 2
- The paper organizes its review around modalities and downstream tasks, robotics applications, robotic foundation models, and the robots, tasks, and environments in use.Its robotics application categories include low- and high-level perception, high- and low-level planning, and data augmentation.Source: §1 · pp. 2–3, Figure 1 · p. 2
2. Foundation Models
The section classifies foundation models for robotics by modality and network input-output relationships, then describes representative model types and tasks across language, vision, audio, 3D, and other modalities.
2. Foundation Models
- Foundation models are characterized by in-context learning, scaling laws, and homogenization, and the review classifies them by their applicability to robotics.In-context learning supports new tasks from a few examples without retraining; scaling laws relate increased data, compute, and model size to performance gains; homogenization supports handling diverse modalities in some architectures.Source: §2 · p. 3
2.1. Foundation Models for Language
- Language models include LLMs that process vast training data, produce language or latent representations, and support reasoning, code handling, and retrieval.The review also notes that language models can impart knowledge to robots through in-context learning, while hallucination remains a reliability challenge.Source: §2.1 · pp. 3–5
2.2. Foundation Models for Vision
- Vision models include latent feature extractors and recognition models for segmentation or bounding boxes; recognition requires manually collected data and is less scalable.Vision-to-latent learning can be self-supervised, while Segment Anything can segment whole images or areas specified by points or boxes.Source: §2.2 · p. 5
2.3. Foundation Models for Vision and Language
- Vision-language models map image and text inputs to latent, language, vision, or recognition outputs for tasks including retrieval, captioning, visual question answering, generation, and open-vocabulary recognition.CLIP computes image-text correspondences for a single pair, whereas GLIP can match text descriptions to multiple image regions.Source: §2.3 · pp. 5–6
2.4. Foundation Models for Audio
- Audio-language models support audio-text similarity, speech-to-text, and text-to-speech or audio conversion, with models such as CLAP, Whisper, MusicLM, and VALL-E.Whisper allows users to specify easily confused words and use in-context learning.Source: §2.4 · pp. 6–7
2.5. Foundation Models for 3D Representation
- 3D foundation models map among 3D representations, vision, and language for latent similarity, 3D shape generation, and recognition in 3D space.The described applications include object classification and policy-learning features, shape comparison, and segmentation or bounding-box extraction.Source: §2.5 · p. 7
2.6. Foundation Models for Other Modalities
- Other modalities include sensors such as IMUs and object poses, handled by models supporting broader multimodal similarity, 6D object-pose estimation, tracking, or language-driven motion and gesture generation.The section names ImageBind, Meta-Transformer, FoundationPose, Human Motion Diffusion Model, T2M-GPT, and GestureDiffCLIP.Source: §2.6 · p. 7
3. Applications of Foundation Models to Robotics
The review organizes foundation-model use in robotics into five categories, following the perception-to-planning sequence and including data augmentation. It illustrates these uses with representative studies while noting that some approaches do not fit the framework.
Framework
- The five categories are low-level perception, high-level perception, high-level planning, low-level planning, and data augmentation, ordered around robotics’ typical perception, planning, and control sequence.They distinguish extracting features or representations, transforming perception into maps or objectives, abstract task planning, lower-level motion control, and augmenting learning data.Source: §3 · p. 8
Application combinations
- Figure 4 groups applications into four combinations of perception, planning, and data augmentation methods.Examples include extracting rewards and motion constraints for reinforcement learning or trajectory optimization, and generating maps or scene graphs for task planning.Source: §3 · pp. 8–9, Figure 4 · p. 9
Scope
- Some research approaches do not fit the five-category framework, and the review treats robotic foundation models that span perception and planning separately.The review selects representative papers for its category-based summary.Source: §3 · pp. 9–10
Low-level perception
- CLIPort combines CLIP semantic features with Transporter Networks’ spatial information to output pick-and-place positions, while REFLECT combines visual and sound recognition to summarize failures and generate corrective plans.These examples represent feature extraction and scene recognition, respectively.Source: §3 · p. 9
High-level perception
- VoxPoser uses LLMs and VLMs to generate affordance and constraint maps for model predictive control, while CLIP-Fields builds a point-cloud semantic map for navigation.These illustrate objective-function and constraint design, and map construction, respectively.Source: §3 · pp. 9–10
High-level planning
- TidyBot uses an LLM with learned human preferences to infer object placement, whereas Code as Policies generates Python code for predefined robot APIs.The examples represent task planning and code generation.Source: §3 · p. 10
Low-level planning
- SayTap uses an LLM to produce footstep plans and velocity before simulation-based reinforcement learning obtains joint commands, while General Pattern Machines generates sequences for reinforcement learning and manipulation actions.Both are examples of foundation models used for low-level planning.Source: §3 · p. 10
Data augmentation
- GenAug uses diffusion models to diversify images for imitation learning, while DIAL uses LLM-generated language instructions and fine-tuned CLIP for instruction augmentation.Both examples augment data used in imitation learning.Source: §3 · p. 10
3.1. Low-level Perception
The review divides low-level perception into feature extraction and scene recognition. Foundation models are used as general visual or multimodal representations and for common robot-vision tasks, including open-vocabulary recognition.
3.1. Low-level Perception
- Low-level perception applications are grouped into feature extraction and scene recognition.The review presents these as the two main uses in this category.Source: §3.1 · p. 11
3.1.1. Low-level Perception for Feature Extraction
- Pretrained models such as CLIP and R3M serve as general feature extractors, either off-the-shelf or fine-tuned for downstream tasks.R3M is trained on egocentric human-activity videos with the aim of reducing the domain gap to robot manipulation; CACTI uses it for imitation-learning features.Source: §3.1.1 · p. 11
3.1.2. Low-level Perception for Scene Recoginition
- Models such as Detic, SAM, OWL-ViT, and ViLD support open-vocabulary object detection or segmentation without further fine-tuning, while other models recognize scene states and events.Examples include REFLECT’s multimodal failure summaries, door-state classification for navigation, and ingredient-state detection in cooking robots.Source: §3.1.2 · p. 11
3.2. High-level Perception
The review divides high-level perception into objective design and map construction. Foundation models generate goals, rewards, costs, and affordances for planning, and build semantic maps that support navigation and scene queries.
3.2. High-level Perception
- High-level perception applications are grouped into objective design and map construction.These are the review’s two main uses in this category.Source: §3.2 · p. 11
3.2.1. High-level Perception for Objective Design
- Foundation models generate planning goals, rewards or costs, and manipulation affordances, using outputs such as goal images, object poses, subgoals, reward functions, or affordance maps.Examples include text-conditioned diffusion for goal generation, LLM-generated rewards, affordances learned from egocentric video, and VoxPoser’s 3D voxel representation for MPC.Source: §3.2.1 · p. 12
3.2.2. High-level Perception for Map Construction
- CLIP-based semantic maps and multimodal map representations support navigation, language queries, and complex 3D-scene queries.Examples include CLIPFields’ text-to-map retrieval, maps combining text, image, and audio embeddings, and ConceptGraphs’ open-vocabulary scene queries.Source: §3.2.2 · pp. 12–13
3.3. High-level Planning
The review presents high-level planning as a prominent use of LLMs and divides it into task planning and code generation. Examples range from selecting and updating action plans to generating executable code or structured plans.
3.3. High-level Planning
- High-level planning is divided into task planning and code generation, and the review describes it as one of the most effective uses of LLMs.The section covers both selecting plans or actions and generating code for robot tasks.Source: §3.3 · p. 13
3.3.1. High-level Planning with Task Planning
- Task-planning systems use language models to choose or revise actions, decompose long tasks, assess feasibility, and replan from feedback or failure.SayCan combines language-model action scores with visual affordance scores; other examples include LM-Nav, Inner Monologue, PIGINet, PET, and REFLECT.Source: §3.3.1 · pp. 13–14
3.3.2. High-level Planner with Code Generation
- Code-generation approaches produce Python, structured action plans, or PDDL plans, while some systems use LLM feedback to build and retrieve new skills.Examples include ProgPrompt, LLM+P, Voyager, and ChatGPT for Robotics.Source: §3.3.2 · p. 14
3.4. Low-level Planning
Low-level planners using foundation models issue either higher-level targets or direct joint commands. The reviewed examples use LLMs to generate footstep plans, transform action sequences, or produce gait commands.
3.4. Low-level Planning
- Foundation-model planners produce either higher-level commands, such as footstep plans and end-effector positions, or lowest-level joint-angle commands.Source: §3.4 · p. 14
- SayTap uses an LLM and in-context learning to generate quadruped footstep plans and target velocity, then reinforcement learning maps these to joint commands.The learned policies support diverse walking patterns.Source: §3.4 · p. 14
- General Pattern Machines applies LLMs to sequence transformation, completion, and improvement, including Cartpole reinforcement-learning tasks and manipulation actions from demonstrations.Source: §3.4 · p. 14
- Prompt2Walk uses an LLM planner to generate quadruped gaits as direct joint-angle commands rather than footstep plans.Source: §3.4 · p. 14
3.5. Data Augmentation
Generalist robot agents need varied camera views and language instructions, but ordinary image scaling can alter the relationship between visual input and behavior. The reviewed work uses foundation models to augment images and instructions for training.
3.5. Data Augmentation
- Image scaling can change the relationship between a robot’s visual input and behavior, so varied views must be prepared without scaling.Source: §3.5 · p. 15
- CACTI, ROSIE, GenAug, and FoundationPose use pretrained or foundation-model tools to augment images, including scenes, objects, textures, and backgrounds.ROSIE selects image regions with OWL-ViT; GenAug uses depth-guided text-to-image inpainting.Source: §3.5 · p. 15
- Moo removes object information from language instructions to support object generalization, while DIAL gathers instruction variants and selects those appropriate for trajectory data.DIAL uses an LLM or crowdsourcing to obtain instructions and fine-tuned CLIP to select them.Source: §3.5 · p. 15
4. Progress towards Robotic Foundation Models
Robotic foundation models span visual representations, vision-language models, and end-to-end policies or dynamics models. Their development faces costly data collection and multimodal requirements, while varied architectures, datasets, and learning objectives address distinct aspects of robot learning.
4. Progress towards Robotic Foundation Models
- Robotic foundation models face costly collection of diverse data and must handle modalities beyond images and text; design choices aim to reduce human collection labor.Source: §4 · p. 15
4.1. Approaches for Robotic Foundation Models
- The review groups robotic foundation models into pre-trained visual representations, vision-language models, and end-to-end control policies or dynamics models.Table 2 summarizes representative models and their inputs, outputs, architectures, and training objectives.Source: §4.1 · p. 15, Table 2 · p. 16
4.1.1. Pre-trained Visual Representations for Robotics
- Robotic pre-trained visual representations provide features for low-level perception, with examples trained from ImageNet, egocentric human videos, and robotic datasets.R3M uses time-contrastive learning on Ego4D videos; MVP uses masked autoencoding, and VC-1 combines ImageNet, Ego4D, and robotics data.Source: §4.1.1 · p. 16
4.1.2. Vision Language Models for Robotics
- Robotics vision-language models accept multimodal inputs or use video-and-text data to support downstream tasks such as planning and visual question answering.PaLM-E extends a language model to multimodal inputs, while RoboVQA presents a scalable data-collection scheme and diverse cross-embodiment dataset.Source: §4.1.2 · p. 16
4.1.3. End-to-End Control Policies and Dynamics Models
- End-to-end control policies and dynamics models learn from large-scale datasets, with approaches ranging from task adaptation to transformer-based control and action-conditioned video simulation.Examples include RT-X training across robots, Octo supporting new input and output modalities, and UniSim modeling interactions with the real world.Source: §4.1.3 · pp. 16–17
4.2. Model Architectures
- Transformer-based architectures unify modalities through tokenization and fusion, while models such as Perceiver-Actor map RGB-D voxels and language to poses, gripper states, and collision-planner signals.Action outputs may be discretized into bins or regressed using a diffusion policy.Source: §4.2 · p. 18
4.3. Datasets and Their Collection
- Robotics datasets include action-conditional collections for prediction or offline reinforcement learning, larger real-robot datasets, and action-free videos that transfer across robot morphologies.RT-X uses Open X-Embodiment data from 22 robots and 21 institutions, comprising more than 1M episodes; teleoperation systems support collecting demonstrations.Source: §4.3 · pp. 18–19
4.4. Learning Objectives
- Learning objectives include behavioral cloning, value learning, trajectory prediction, and masked-token prediction; masking trajectories can express behavioral cloning, offline reinforcement learning, and forward dynamics in one framework.Source: §4.4 · p. 19
5. Robots, Tasks, and Environments
The review classifies real-robot foundation-model research across navigation, manipulation, combined navigation and manipulation, locomotion, and communication. Research is extensive in manipulation but relatively scarce in locomotion and embodied-robot communication.
5. Robots, Tasks, and Environments
- The review covers real-robot studies across navigation, manipulation, combined navigation and manipulation, locomotion, and communication; Figure 5 overviews robots, tasks, and environments.Simulation studies and pre-foundation-model language-based robot work are outside this section’s scope.Source: §5 · pp. 19–20, Figure 5 · p. 20
5.1. Navigation
- Navigation work uses language-aligned embeddings for map construction and destinations, while LM-Nav, GNM, and ViNT include outdoor experiments.Source: §5.1 · p. 20
5.2. Manipulation
- Manipulation research is the most extensive, commonly addressing single-arm manipulation and grasping in everyday settings, with pick-and-place among the primary tasks.Studies also address dual-arm and in-hand manipulation, obstacle avoidance, path planning, and flexible objects.Source: §5.2 · p. 20
5.3. Navigation with Manipulation
- Navigation-and-manipulation studies are conducted indoors and combine navigation with pick-and-place grasping using language-vision models or robotic foundation models.Source: §5.3 · p. 21
5.4. Locomotion
- Foundation-model locomotion research is limited and mainly uses quadrupeds; SayTap generates footstep plans, while Language to Rewards generates rewards or constraints for control.The review notes that expressing leg movements in language is difficult and that bipedal-robot research is lacking.Source: §5.4 · p. 21
5.5. Communication
- Communication studies using actual robots are relatively scarce; examples use robot-human walking data to generate diaries or convey robots’ experiences and perspectives.The review treats diary generation as high-level perception and notes work that considers human preferences.Source: §5.5 · p. 21
6. Conclusion and Future Challenges
The review maps real-world robot applications of foundation models and their construction, then identifies current research trends and practical challenges. It highlights gaps in modalities, control, deployment settings, generalization, and evaluation.
6.1. Conclusion
- The review categorizes applications by input-output modalities and five aspects—low- and high-level perception, high- and low-level planning, and data augmentation—and summarizes their combinations.Source: §6.1 · p. 21
- It also summarizes robotic foundation-model construction through model architecture, datasets, and learning objectives, alongside the robots, tasks, and environments covered by existing research.Source: §6.1 · p. 22
- The review identifies growing work on applying existing foundation models to robot movements and building larger robotic models, while low-level motion control and realistic household and outdoor deployments remain scarce.Source: §6.1 · p. 22
6.2. Future Challenges
- Depth, force feedback, inertial sensing, and human, object, and robot motion remain underused relative to language, image, and sound; robot-skill hierarchy and API granularity are future concerns.Source: §6.2 · p. 22
- Current robotic foundation models have exceptionally slow control frequencies and are unsuitable for fine-grained force control or human collaboration; suitable action spaces and safety-risk reduction remain open questions.Source: §6.2 · p. 22
- Practical studies rarely cover outdoor settings, often use setups unlike real environments, rely on off-the-shelf robots, and mostly use position rather than torque control or soft robots.Source: §6.2 · p. 22
- Generalization across environments and tasks remains a challenge, although open-world examples include OVMM, OK-Robot [151], and GOAT [223].Source: §6.2 · p. 22
- Language-instruction performance is difficult to evaluate quantitatively: simulator benchmarks such as CALVIN and ARNOLD exist, but fair real-world evaluation and human-interaction evaluation remain challenging.Source: §6.2 · pp. 22–23