Source-linked AI summary
PaLM-E: An Embodied Multimodal Language Model
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, Pete Florence
TL;DR
PaLM-E addresses the challenge of grounding language models in real-world sensor inputs for embodied reasoning. It injects continuous observations into a pretrained LLM and jointly trains the resulting model across embodied and vision-language tasks, achieving broad transfer, strong robotics data efficiency, state-of-the-art OK-VQA performance, and retained language capabilities at scale.
Problem
Language models need grounding in visual and physical sensor modalities to support inference and decision making in real-world robotic settings.
Method
PaLM-E interleaves encoded continuous observations with language embeddings in a pretrained LLM and trains the encoders end-to-end across multiple tasks and embodiments.
Results
PaLM-E supports multiple embodied reasoning tasks and embodiments, while PaLM-E-562B achieves state-of-the-art OK-VQA performance without task-specific finetuning.
Takeaways & Limitations
Joint training transfers across domains, improving robotics data efficiency and enabling a single model to combine embodied reasoning with general vision-language and language capabilities.
Takeaways & Limitations
The control setup assumes access to low-level policies for a small vocabulary of skills, and those policies cannot solve long-horizon tasks or take complex instructions.
Abstract
from arXiv · showhide
Large language models excel at a wide range of complex tasks. However, enabling general inference in the real world, e.g., for robotics problems, raises the challenge of grounding. We propose embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts. Input to our embodied language model are multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings. We train these encodings end-to-end, in conjunction with a pre-trained large language model, for multiple embodied tasks including sequential robotic manipulation planning, visual question answering, and captioning. Our evaluations show that PaLM-E, a single large embodied multimodal model, can address a variety of embodied reasoning tasks, from a variety of observation modalities, on multiple embodiments, and further, exhibits positive transfer: the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains. Our largest model, PaLM-E-562B with 562B parameters, in addition to being trained on robotics tasks, is a visual-language generalist with state-of-the-art performance on OK-VQA, and retains generalist language capabilities with increasing scale.
1. Introduction
PaLM-E addresses the grounding gap by incorporating continuous embodied observations into a pretrained language model. A single jointly trained model supports robotic decision making and general vision-language capabilities, with transfer across tasks and data-efficient robotics learning.
- Motivation: PaLM-E targets the grounding problem by connecting language-model representations to real-world visual and physical sensor modalities.Prior interfaces provide language models with textual input, which is insufficient when scene geometry matters.
- Approach: Embodied language models inject images and state estimates into a language model’s embedding space and train the encoders end-to-end for sequential decisions.The continuous inputs are processed alongside language tokens by the Transformer-based language model.
- Evaluation: The approach is evaluated across three robotic manipulation domains, visual question answering, image captioning, and language tasks.Two robotic domains use closed-loop real-world evaluation.
- Results: PaLM-E-562B achieves state-of-the-art OK-VQA performance without task-specific finetuning while retaining broad multimodal and language capabilities.The model also demonstrates zero-shot multimodal chain-of-thought, multi-image reasoning, and other capabilities.
2. Related Work
Related work combines language, vision, and embodied control in several ways, but PaLM-E emphasizes language as a planning representation within a jointly trained, multi-embodiment generalist.
- General vision-language modeling: General vision-language models support tasks such as visual question answering, captioning, and optical character recognition by jointly processing images and text.These models provide the broader vision-language foundation relevant to PaLM-E.
- Actions-output models: Prior embodied models commonly use vision and language inputs for direct action prediction, with language serving primarily as task specification.VIMA is identified as exploring multimodal prompts similar to PaLM-E.
- Actions-output models: PaLM-E instead generates high-level instructions as text, allowing it to condition on its own predictions and leverage world knowledge in its parameters.The same text-generating interface supports embodied reasoning and question answering.
- LLMs in embodied task planning: Embodied task-planning research often uses language to express goals, while fewer approaches use natural language as the planning representation itself.The paper positions planning with language as its focus because ungrounded plans can be impossible to execute.
- LLMs in embodied task planning: Unlike approaches that keep LLM parameters largely unchanged, PaLM-E fine-tunes a generalist multi-embodiment agent and reports positive transfer from joint training across domains.The comparison is made explicitly with Gato and related planning systems.
3. PaLM-E: An Embodied Multimodal Language Model
PaLM-E embeds continuous observations as multimodal tokens interleaved with text and autoregressively generates textual answers or decisions. In control tasks, those decisions are executed through low-level skills in a robot feedback loop.
- Architecture: PaLM-E injects images, state estimates, and other continuous observations into the embedding space of a pretrained language model.The model reuses the language model’s processing pathway for embodied observations.
- Multimodal inputs: Multimodal tokens and text form the input sentence, while a decoder-only LLM generates the output autoregressively.Outputs may answer questions or specify decisions.
- Language-model foundation: A decoder-only LLM predicts text-token probabilities autoregressively and can be conditioned on a prefix without changing its architecture.The prefix supplies context for subsequent token predictions.
- Multimodal inputs: An observation encoder maps each continuous observation into a sequence of vectors matching the language-token embedding space.These vectors can be interleaved with text tokens and multiple observation encoders can be used.
- Robot control loop: For text-only tasks, the generated text is the solution; for control tasks, generated text conditions low-level policies that execute a sequence of skills.The available skills are inferred from training data and prompts rather than constrained by another output-filtering mechanism.
4. Input & Scene Representations for Different Sensor Modalities
PaLM-E maps state estimates and visual observations into the language embedding space through modality-specific encoders. It also explores object-centric representations and entity referrals to better structure and reference physical objects.
- State estimation vectors: State vectors describing object properties such as pose, size, and color are mapped into the language embedding space by an MLP.
- Vision Transformer (ViT): ViT encoders map images into token embeddings and use a learned affine transformation to match the language model’s embedding dimensionality.The projected embeddings are inserted as visual tokens for processing by the language model.
- Object-centric representations: Because ViT representations resemble static grids, PaLM-E explores structured encoders that separate visual inputs into distinct objects.This object-level structure is intended to better interface with symbol-pretrained language models and embodied reasoning.
- Object Scene Representation Transformer (OSRT): OSRT discovers objects without ground-truth segmentations and learns 3D-centric neural scene representations from multi-view images.Its object slots are projected into multiple language-space embeddings for each object.
- Entity referrals: Entity referrals help PaLM-E reference objects that cannot be uniquely identified using only a few natural-language properties.This is especially relevant when multiple objects share properties such as color but differ in location.
5. Training Recipes
PaLM-E trains a decoder-only language model on multimodal prefixes and text prediction targets, injecting continuous observations through learned encoders. Training recipes vary which components are updated and whether diverse embodied and internet-scale data are co-trained.
- Training objective: Each training example contains continuous observations, a multimodal text prefix, and a text-only prediction target.The objective is cross-entropy over the non-prefix text tokens.
- Multimodal input construction: Special text tokens are replaced by encoder embeddings, allowing continuous observations to be injected into pretrained PaLM decoder-only language models.The resulting model sizes include PaLM-E-12B, PaLM-E-84B, and PaLM-E-562B.
- Variation with Model freezing: Training can update the encoder, projector, and language model jointly, or freeze the language model while training input encoders.With a frozen language model, encoders must ground observations and communicate embodiment capabilities through their embeddings.
- Co-training across tasks: The full mixture combines diverse internet-scale vision-and-language data with embodied data across several tasks and embodiments.Only 8.9% of the full mixture is embodied data.
6. Experiments
PaLM-E is evaluated across diverse robotic embodiments, planning settings, and visual-language tasks, with experiments testing representations, co-training, transfer, and scale. Results show that joint training and multimodal grounding improve data efficiency, transfer across tasks and embodiments, real-robot generalization, and retention of language and visual-language performance.
- Experimental scope: Experiments span three robotic environments, simulation, two real robots, visual-language tasks, and established language-modeling tasks.The evaluation covers mobile manipulation, tabletop pushing, TAMP planning, VQA, image captioning, and language benchmarks.
- TAMP Environment: With only 1% of TAMP data, co-training on robotic and general vision-language datasets more than doubles ViT-4B planning performance.The 1% regime contains 320 examples for each planning task, and the full-mixture model transfers across robot embodiments and tasks.
- Language-Table Environment: Joint training on internet-scale vision and language improves Language-Table planning, particularly in the few-shot regime with 10 demonstrations per task.Scaling from 12B to 84B improves performance on two of three tasks, while SayCan and zero-shot PaLI cannot solve the easiest tested task.
- Mobile Manipulation Environment: PaLM-E guides real robots through long-horizon manipulation and generalizes one-shot or zero-shot to novel object combinations and unseen objects.A single model directs low-level policies for two real robots, including a kitchen mobile-manipulation task and tabletop manipulation.
- Mobile Manipulation Environment: PaLM-E outperforms PaLI and specialized comparison methods on affordance prediction and failure detection in the mobile manipulation environment.Failure detection is compared with zero-shot PaLI, finetuned CLIP, and a method using two CLIP models with hindsight-relabeled data.
- General Visual-Language and Language Tasks: PaLM-E-562B achieves the highest reported OK-VQA result, while language-capability degradation falls from 87.3% for PaLM-E-12B to 3.9% for PaLM-E-562B.The OK-VQA result is achieved without task-specific finetuning, and the language results average 21 NLU and NLG benchmarks.
7. Summary of Experiments & Discussion
PaLM-E benefits from joint training across tasks, datasets, and embodiments, improving transfer and data efficiency while retaining language capabilities as model scale increases. A promising future direction is combining geometric representations with large-scale visual data.
- Generalist vs specialist models – transfer: Co-training on the full mixture achieves more than double the performance of separately trained models.
- Generalist vs specialist models – transfer: Training with LLM/ViT pre-training and the full mixture significantly improves performance over mobile-manipulation-only training.
- Data efficiency: PaLM-E solves robotics tasks from few training examples, including 10–80 examples for Language Table and 320 for TAMP.
- Data efficiency: Combining geometric input representations with methods that benefit from large-scale visual data is identified as a promising future opportunity.
- Retaining language capabilities: Freezing the LLM preserves language capabilities but can occasionally hinder robotics tasks, whereas end-to-end training retains more language performance at larger scales.
8. Conclusion
PaLM-E injects multimodal information into a pretrained LLM to support embodied reasoning across robots and general vision-language tasks. Diverse training enables data-efficient robot planning while preserving visual-language competence.
- PaLM-E injects multimodal information such as images into the embedding space of a pretrained LLM.
- The single model controls different robots in simulation and the real world while remaining competent at VQA and captioning.
- Diverse training across robot embodiments and vision-language tasks enables transfer into embodied decision making and data-efficient robot planning.
A. Data Mixture
The full training mixture is dominated by general vision-language data, with robot data comprising less than 10% of the distribution.
- Less than 10% of the full-mixture data is robot data.
- General vision-language tasks make up the majority of the full-mixture data distribution.
B.1. Task and Motion Planning (TAMP)
The TAMP setup evaluates object reasoning and planning through visual-language prompts involving object relations, feasibility, grasping, and stacking. Training and evaluation use distinct object-count conditions and generated planning data.
- Scene setup: TAMP training scenes contain 3–5 cube-shaped objects, while one test example contains 6 objects.
- Planning tasks: The planning tasks require generating multistep grasping and stacking plans from visual prompts.
- VQA tasks: The TAMP VQA tasks cover object color, object-table relations, object-object relations, and plan feasibility.
- Data generation: A planner and low-level policies from Driess et al. (2020) generate the planning-task dataset and execution policies.
- Evaluation: Table 7 reports TAMP success rates across input representations, training-distribution scenes, and out-of-distribution color-referenced tasks.
- Related setup: The Language-Table setup uses a real-world tabletop environment and a simulated environment, with short-horizon commands issued every four seconds.
- Evaluation: Simulation evaluation for Tasks 2 and 3 uses an automated reward and 80 rollouts per task.
C. Natural Language Generation and Understanding Results
The language evaluation reports NLU and NLG results for original PaLM models and associated unfrozen PaLM-E models. Frozen PaLM-E models match the performance of their underlying PaLM models.
- The evaluation covers both natural-language understanding and natural-language generation tasks.
- Results compare original PaLM models with associated unfrozen PaLM-E models.
- Frozen PaLM-E models have the same performance as their corresponding underlying PaLM models.
D. Additional Data for Affordance and Success Detection
The additional mobile-manipulation results report precision and recall for failure detection and affordance prediction. The supplied passages also identify image sources used in the paper.
- Failure detection: Failure detection is evaluated using individual precision and recall scores.
- Affordance prediction: Affordance prediction is evaluated using individual precision and recall scores.
- Image sources: The paper includes credited New York Knicks–Boston Celtics images and egocentric video images.