Source-linked AI summary
EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought
Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, Ping Luo
TL;DR
EmbodiedGPT addresses the difficulty of collecting structured real-world embodied data and connecting language-model planning to robotic control. It combines the EgoCOT and EgoVQA datasets with prefix-tuned plan generation and a closed-loop feature-extraction paradigm. Across embodied and multimodal tasks, it achieves state-of-the-art or comparable performance, with reported control success-rate increases over a BLIP-2 baseline on two benchmarks.
Problem
Embodied robotics needs large-scale egocentric data and structured language instructions, while open questions remain about domain gaps, chain-of-thought planning, and end-to-end use of language plans for manipulation.
Method
EmbodiedGPT combines machine-generated, semantically filtered, and human-verified EgoCOT and EgoVQA data with prefix tuning and a closed loop from high-level plans to task-relevant low-level control features.
Results
EmbodiedGPT performs effectively across embodied planning, control, visual captioning, and visual question answering, achieving state-of-the-art or comparable performance.
Takeaways & Limitations
The model supports step-by-step planning and low-level command execution while connecting high-level embodied plans with downstream control.
Abstract
from arXiv · showhide
Embodied AI is a crucial frontier in robotics, capable of planning and executing action sequences for robots to accomplish long-horizon tasks in physical environments. In this work, we introduce EmbodiedGPT, an end-to-end multi-modal foundation model for embodied AI, empowering embodied agents with multi-modal understanding and execution capabilities. To achieve this, we have made the following efforts: (i) We craft a large-scale embodied planning dataset, termed EgoCOT. The dataset consists of carefully selected videos from the Ego4D dataset, along with corresponding high-quality language instructions. Specifically, we generate a sequence of sub-goals with the "Chain of Thoughts" mode for effective embodied planning. (ii) We introduce an efficient training approach to EmbodiedGPT for high-quality plan generation, by adapting a 7B large language model (LLM) to the EgoCOT dataset via prefix tuning. (iii) We introduce a paradigm for extracting task-related features from LLM-generated planning queries to form a closed loop between high-level planning and low-level control. Extensive experiments show the effectiveness of EmbodiedGPT on embodied tasks, including embodied planning, embodied control, visual captioning, and visual question answering. Notably, EmbodiedGPT significantly enhances the success rate of the embodied control task by extracting more effective features. It has achieved a remarkable 1.6 times increase in success rate on the Franka Kitchen benchmark and a 1.3 times increase on the Meta-World benchmark, compared to the BLIP-2 baseline fine-tuned with the Ego4D dataset.
1 Introduction
EmbodiedGPT addresses open challenges in applying language models to robotics by combining structured embodied planning, multimodal learning, and a closed loop from planning to control. It uses EgoCOT and related model designs to support planning, question answering, captioning, and low-level action execution.
- Embodied AI requires agents to perceive environments, reason about tasks, and execute autonomous actions over long horizons.
- Structured language instructions for embodied planning are costly to collect, while existing simulated datasets leave a gap between simulation and the real world.
- EgoCOT provides egocentric videos with high-quality step-by-step, chain-of-thought planning instructions generated, filtered, and human-verified for quality.
- EmbodiedGPT is an end-to-end multimodal foundation model supporting embodied planning, embodied VQA, and embodied control through integrated vision and language processing.
- Its closed-loop design extracts task-relevant features from language plans to connect high-level planning with low-level control.
- The model produces executable, object-part-level sub-goals and achieves state-of-the-art or comparable performance across embodied control, planning, captioning, and video QA.
2 Related Work
Related work spans modular and end-to-end vision-language systems, egocentric video datasets, and embodied multimodal models. EmbodiedGPT builds on these directions while targeting more precise embodied planning and control.
- Vision-Language Pre-training: Vision-language pre-training links visual observations with natural language, increasingly using frozen pretrained components to reduce end-to-end computational expense.
- Embodied Planning: Prior embodied planning work lacked open-source multimodal data for detailed task decomposition and precise executable plans.
- Egocentric Video Datasets: Egocentric video datasets capture natural daily activities but are expensive to collect and historically tend to be small-scale and domain-specific.
- EmbodiedGPT Framework: Figure 2 depicts EmbodiedGPT’s vision-language planning process and the use of queried language plans for low-level policy learning.
- Large Foundation Model Assistant System: Modular systems coordinate vision tools and language models but can accumulate errors from the accuracy and capacity limits of fixed components.
- Large Foundation Model Assistant System: End-to-end models such as Flamingo and BLIP-2 align visual features with language models, while later systems extend multimodal interaction to video.
3 Method
EmbodiedGPT combines visual and language models with an embodied-former to generate chain-of-thought plans and task-relevant control features. Its training and data pipeline uses staged pre-training, EgoCOT planning annotations, video segmentation, multi-sample generation, and CLIP-based filtering.
- Training: Training proceeds in three stages, culminating in chain-of-thought vision-language pre-training on EgoCOT with eight video keyframes, task descriptions, plans, and structured verb-noun actions.Conv3D transfers the stage-2 vision model to video encoding before the third stage.
- Model Architecture: The embodied-former bridges visual inputs and a frozen language model through learned query embeddings, cross-attention, self-attention, and dimensionality-reducing projections.It produces soft visual prompts for language-model planning.
- Model Architecture: For low-level control, the embodied-former extracts instance-level features from visual observations conditioned on the embodied plan, which a policy network combines with global context to generate executable actions.Global context comes from a pretrained ResNet50, while the policy network is an MLP.
- EgoCOT and EgoVQA Dataset: EgoCOT pairs timestamped Ego4D narrations with video clips, then uses ChatGPT to generate detailed embodied plans and verb-noun action sequences.Generation uses temperature 0.9, top-p 0.95, and five sampling iterations per prompt.
- EgoCOT and EgoVQA Dataset: CLIP-based cleaning compares videos with candidate plans and filters video-caption-planning pairs using text-image similarity thresholds; EgoVQA adds five question-answer pairs per Ego4D caption.Video-level similarity ensembles scores across keyframes to reduce individual-frame variability.
4 Experiments
EmbodiedGPT is evaluated across visual captioning, embodied planning, question answering, and embodied control tasks. Results and ablations assess its planning quality, low-level control performance, and the contributions of EgoCOT and closed-loop design.
- The experiments evaluate multimodal models across visual captioning, embodied planning, and embodied control tasks.
- Image input tasks: A user study with 30 participants evaluates image-based captions and planning across MS-COCO, simulated embodied scenarios, and real-world scenes.Participants rate generated captions on five dimensions using a 1-to-10 scoring system.
- Video input embodied AI tasks: EmbodiedGPT interprets demonstration videos and produces step-by-step plans for manipulation tasks on Franka Kitchen and Meta-World.The benchmarks include tasks such as opening appliances, moving blocks, assembling rings, and hammering nails.
- Embodied control tasks: EmbodiedGPT is compared with R3M and BLIP-2[Ego4D] under 10- and 25-demonstration settings using visual observations.BLIP-2[Ego4D] has the same structure and parameter count but is fine-tuned only on Ego4D video captioning.
- Ablation study: Ablations show that EgoCOT training improves success rate and that closed-loop planning-to-control design is necessary because plans contain control-relevant sub-goals.
5 Conclusion
The paper presents EmbodiedGPT as an end-to-end multimodal foundation model for step-by-step planning and low-level action execution. It combines EgoCOT, prefix tuning, and a closed-loop planning-control paradigm, while noting frozen vision and language parameters as a limitation.
- EmbodiedGPT enables embodied agents to perform step-by-step planning and execute low-level commands.
- The model uses the EgoCOT dataset, prefix tuning, and coordinated high-level planning with low-level control.
- Experiments report state-of-the-art or comparable performance across various embodied tasks.
- Limitations and future work: The vision and language model parameters are frozen because of limited computational resources.
A.1 Hyper-parameters
Downstream policy learning uses imitation learning with behavioral cloning on expert demonstrations. Training uses fixed optimization and data-processing settings, with demonstration lengths differing between benchmarks.
- Hyper-parameters: Vision-language pre-training uses AdamW, cosine learning-rate decay, linear warm-up, 224×224 images, and maximum sequence length 256.
- The policy is learned through imitation learning from expert observation-action demonstrations.
- The loss compares predicted action probabilities πθ(a|s) with ground-truth action probabilities P ∗(a|s).
- Franka Kitchen demonstrations contain 50 state-action pairs, whereas Meta-World demonstrations contain 500.
B.1 Visual Captioning
EmbodiedGPT performs visual captioning across embodied AI benchmarks and demonstrates strong zero-shot performance without task-specific fine-tuning.
- EmbodiedGPT accurately describes objects, characters, and spatial relationships relevant to embodied AI tasks.
- The model shows robust zero-shot learning across multiple benchmarks without task-specific fine-tuning.
B.2 Embodied Planning with image input
EmbodiedGPT uses image input to generate multi-step plans for concrete and abstract tasks. It identifies relevant objects and relationships, then translates abstract goals into actionable sub-tasks.
- Concrete tasks: For concrete tasks such as making coffee, EmbodiedGPT identifies required objects and their positional relationships from visual input.It produces coherent multi-step plans that account for task requirements and environmental constraints.
- Abstract tasks: For abstract tasks, EmbodiedGPT combines visual observations with abstract concepts to generate concrete sub-task descriptions.For example, when given the goal of feeling hot, it identifies a fan and suggests turning it on.
- Abstract tasks: The model then generates detailed plans for the identified abstract sub-tasks, demonstrating adaptability across problem-solving contexts.
C Evaluation metric and scoring criteria for user study
The user study evaluates multimodal question answering with five metrics, while additional prompts direct models toward visual information. The supplied examples also illustrate embodied planning prompts for real-world and abstract tasks.
- Evaluation metric and scoring criteria for user study: Table 3 outlines the scoring criteria for a user study incorporating five evaluation metrics.
- Evaluation metric and scoring criteria for user study: Additional prompts explicitly direct models to use visual information when answering questions about images or videos.Examples include referencing the depicted scene or asking the model to describe the visual content.
- Embodied planning examples: The examples include planning tasks about meeting with someone, cooling oneself when hot, making coffee, and hanging clothes.The meeting example selects a room and identifies a meeting point from the image; another example gives step-by-step instructions for hanging clothes.
- Embodied planning examples: Figures 8–10 present embodied-planning scenarios and performance comparisons using different prompt types.
- Evaluation metric and scoring criteria for user study: The experiments report that additional prompts significantly improve models’ use of visual cues and answer accuracy.