Source-linked AI summary
3D-VLA: A 3D Vision-Language-Action Generative World Model
Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, Chuang Gan
TL;DR
Existing embodied models rely on 2D inputs and lack multimodal world-modeling abilities for reasoning, imagining future states, and planning actions in 3D environments. 3D-VLA addresses this with a 3D language-model backbone, interaction tokens, aligned diffusion models, and a 3D embodied dataset, improving reasoning, generation, and planning tasks.
Problem
Existing foundation and embodied models lack 3D understanding, multimodal future-state imagination, and 3D annotations needed for reasoning and planning in physical environments.
Method
3D-VLA combines a 3D large language model, specialized interaction and action tokens, aligned RGBD and point-cloud diffusion models, and a 3D-language-action dataset.
Results
3D-VLA outperforms baseline models across embodied goal generation, goal-based planning, action prediction, language reasoning, and localization tasks.
Takeaways & Limitations
3D-VLA demonstrates stronger embodied reasoning, multimodal generation, and planning capabilities through a generative world-model framework.
Takeaways & Limitations
Real-world deployment is conducted under human supervision after simulator training to reduce potential collisions with objects and humans.
Abstract
from arXiv · showhide
Recent vision-language-action (VLA) models rely on 2D inputs, lacking integration with the broader realm of the 3D physical world. Furthermore, they perform action prediction by learning a direct mapping from perception to action, neglecting the vast dynamics of the world and the relations between actions and dynamics. In contrast, human beings are endowed with world models that depict imagination about future scenarios to plan actions accordingly. To this end, we propose 3D-VLA by introducing a new family of embodied foundation models that seamlessly link 3D perception, reasoning, and action through a generative world model. Specifically, 3D-VLA is built on top of a 3D-based large language model (LLM), and a set of interaction tokens is introduced to engage with the embodied environment. Furthermore, to inject generation abilities into the model, we train a series of embodied diffusion models and align them into the LLM for predicting the goal images and point clouds. To train our 3D-VLA, we curate a large-scale 3D embodied instruction dataset by extracting vast 3D-related information from existing robotics datasets. Our experiments on held-in datasets demonstrate that 3D-VLA significantly improves the reasoning, multimodal generation, and planning capabilities in embodied environments, showcasing its potential in real-world applications.
1. Introduction
Existing embodied models largely rely on 2D perception or direct perception-to-action mappings, while human-like agents require 3D understanding and future-state imagination. 3D-VLA addresses these gaps with a 3D generative world model, interaction tokens, multimodal goal generation, and a large 3D-language-action dataset.
- Motivation: 2D vision-language and embodied models lack the richer 3D understanding needed for reasoning, planning, and acting in physical environments.The paper contrasts image-based models with human interaction in a 3D world.
- Motivation: Existing 3D embodied models mainly map perception directly to action without modeling world dynamics or action–dynamics relations.The proposed motivation is to model future states for planning toward predicted goals.
- Motivation: Building 3D world models is constrained by language-focused foundation models and embodied datasets dominated by 2D images or videos without 3D annotations.These limitations affect multimodal future-state simulation and 3D reasoning or planning data.
- Approach: 3D-VLA links 3D perception, reasoning, and action through a generative world model built on a 3D large language model.Interaction tokens represent scenes, objects, and actions for embodied interaction.
- Approach: The model aligns embodied diffusion models with the LLM to generate multimodal goals, including images, depths, and point clouds.A projector bridges multimodal diffusion decoders and the LLM embedding space.
- Approach: 3D-VLA is trained on a curated 2M 3D-language-action dataset covering captioning, action prediction, localization, and multimodal goal generation.The dataset is assembled by enriching existing embodied data with 3D information and language annotations.
- Results: 3D-VLA performs goal generation, goal-based planning, embodiment action prediction, and traditional language tasks, outperforming baselines by a large margin on novel embodied tasks.The reported evaluations cover both embodied and language-based capabilities.
2. Related Works
Prior multimodal and 3D foundation models established image-text understanding and 3D scene reasoning, but embodied action models largely rely on 2D features and current-state analysis. 3D-VLA extends this direction toward 3D-guided action planning and future-feature prediction.
- Multimodal Language Models: Recent multimodal language models support vision-language understanding and increasingly unified combinations of text, images, and videos.The related work includes interleaved image-text understanding and generation.
- Vision-Language-Action Models: Previous vision-language-action models predominantly use 2D features, limiting their 3D spatial understanding for action generation.3D-VLA instead aligns 3D features, including point clouds, with goal objectives for action-token generation.
- 3D Foundation Models: 3D foundation models incorporate 3D features into multimodal language models but primarily reason about the currently observable scene.The paper positions future-feature prediction as the contrasting capability of its general world model.
3. 3D Embodied Instruction Tuning Dataset
The paper constructs a large-scale 3D embodied instruction dataset by enriching existing robot and human-object interaction data with estimated or extracted 3D annotations and language prompts. The resulting pairs cover spatial structure, goals, actions, and textual instructions.
- 3. 3D Embodied Instruction Tuning Dataset: Existing embodied datasets often lack depth, 3D annotations, and precise control information needed for commands requiring 3D spatial reasoning.Examples include placing the farthest cup into the middle drawer.
- 3. 3D Embodied Instruction Tuning Dataset: The dataset bridges this gap by extracting 3D-language-action pairs from existing embodied datasets.The extracted annotations include point clouds, depth maps, 3D bounding boxes, 7D robot actions, and text.
- 3.1. Dataset Collection: The source collection combines diverse real and synthetic robot-manipulation data with human-object interaction datasets.The sources include 12 robot datasets selected from Open-X Embodiment and datasets such as Epic-Kitchens and HOI4D.
- 3.2. Visual Annotations: Because over 95% of embodied video datasets lack 3D information, the pipeline estimates framewise depth with ZoeDepth and optical flow with RAFT.Optical flow is used to refine generated data when camera pose does not change.
- 3.2. Visual Annotations: The pipeline generates 3D bounding boxes, goal images, depths, point clouds, and robot actions in 3D space.These annotations support attention to manipulated objects and future-state imagination.
- 3.3. Language Annotations: Dense language annotations wrap generated 3D annotations in modality-specific tokens such as <image></image> and <pcd></pcd>.The tokens encompass bounding boxes, goal modalities, and actions.
- 3.3. Language Annotations: Predefined templates and ChatGPT-based prompting diversify prompts using annotated objects, bounding boxes, and few-shot human-written demonstrations.ChatGPT summarizes and rewrites template-generated descriptions.
4. Methods
3D-VLA combines a 3D-LLM backbone with interaction tokens and aligned embodied diffusion models to reason over scenes, represent actions, and generate future goals. Its pipeline feeds generated goal states back into control, while evaluations report advantages over 2D baselines across reasoning and goal generation.
- 4. Methods: The method builds a 3D world model on 3D-LLM, adds interaction tokens, and aligns diffusion models with the LLM through a projector.The architecture targets 3D reasoning, goal generation, and embodied decision-making.
- 4.1. 3D-VLA Base Model: The base model leverages multi-view features to generate 3D scene features and integrate them into a pretrained vision-language model.This avoids training a multimodal LLM from scratch at billion-level scale.
- 4.2. Interaction Tokens: Interaction tokens represent referenced objects, spatial locations, scenes, and robotic actions within interleaved 3D-text inputs.Object tokens enclose object nouns, while scene tokens compose dynamic-scene inputs.
- 4.2.2. INTERACTION TOKENS: Seven-degree-of-freedom robot actions are discretized into location, rotation, and gripper tokens separated by <ACT SEP>.The tokens encode intended absolute arm location, rotation, and gripper openness.
- 4.3. Injecting Goal Generation Ability into 3D-VLA: The model targets goal generation because pre-visualized final states support action prediction and decision making.Preliminary experiments also found that providing ground-truth final states enhanced reasoning and planning.
- 4.3.1. Embodied Diffusion Models: The method trains conditional RGBD-to-RGBD and point-cloud-to-point-cloud diffusion models to edit initial states into instruction-conditioned final states.These embodied diffusion models address shortcomings of generic video diffusion for robotics.
- 4.3.2. BRIDGING LLM AND GOAL GENERATION: A transformer-based projector maps LLM features and embeddings into diffusion-model space and uses modality signals to select generated content.The alignment stage connects pretrained decoders for different output modalities to the LLM.
- 5. Experiments: The paper evaluates reasoning ability using held-in data, with asterisks marking zero-shot transfer results without training on the pretraining datasets.This evaluation convention is specified in the Table 1 caption.
5. Experiments
Experiments evaluate 3D-VLA across 3D reasoning and localization, multimodal goal generation, and embodied action planning. The model generally outperforms or matches baselines, including in unseen environments and selected generalization settings.
- Evaluation Setup: 3D-VLA compares against 2D vision-language, 3D-LLM, grounding, image-generation, and point-cloud-generation baselines across the evaluated tasks.Evaluations include zero-shot and held-in comparisons, with 4000 unseen Open-X test episodes for goal generation.
- 3D Reasoning and Localization: 3D-VLA outperforms 2D vision-language baselines on language reasoning and localization tasks in held-in robotics datasets.The authors attribute this to 3D spatial information and learned 3D localization annotations.
- Multi-modal Goal Generation: 3D-VLA outperforms directly transferred generation methods and Instruct-P2P* on most image-goal metrics, while predicted bounding boxes improve performance.The point-cloud model also performs best with intermediate predicted bounding boxes.
- Multi-modal Goal Generation: 3D-VLA consistently maintains backgrounds, identifies target objects, and modifies their states correctly when generating RGB-D goal images.Generated goals closely align with ground-truth goals in visual appearance and semantic content, including Internet and daily-life scenes.
- Embodied Action Planning: 3D-VLA surpasses or matches the baseline on most RLBench action-prediction tasks and achieves promising CALVIN results.The evaluation includes an unseen pick-up-cup task and long-horizon CALVIN control with five sequential tasks.
6. Conclusion
The paper introduces 3D-VLA as a generative world model for embodied reasoning, generation, and planning. It combines 3D-language-action data, a 3D-LLM with interaction tokens, and aligned diffusion models, and reports stronger embodied-task capabilities than 2D baselines.
- Conclusion: 3D-VLA combines a 2M 3D-language-action dataset, a 3D-LLM backbone with interaction tokens, and aligned image-to-image and point-to-point diffusion models.The components support task captioning, localization, goal generation, and action prediction.
- Conclusion: Experiments show that 3D-VLA has stronger embodied-task capabilities than the 2D baseline.The reported capabilities include reasoning, understanding, generation, and planning in embodied environments.
- Conclusion: 3D-VLA is presented as a generative world model that reasons, understands, generates, and plans in embodied environments.
Impact Statement
The paper addresses robot manipulation while recognizing that physical-world operation can create collision risks involving objects and humans. It therefore emphasizes simulator pretraining followed by supervised real-world deployment to reduce adverse impacts.
- Collision risks involving objects and humans arise when robot systems are inadequately configured.
- The approach first trains in a simulator, then deploys in the real world under human supervision.This sequence is intended to minimize adverse impacts during deployment.
A. Model Implementation Details
The implementation uses pretrained BLIP-2 FlanT5 and separate pretraining and alignment stages with specified training durations, hardware, batch sizes, and learning-rate schedules.
- Pretrained BLIP-2 FlanT5 serves as the model backbone.
- Pretraining runs for 30 epochs on 6 × 32 V100s with batch size 4 per node.Validation occurs every epoch.
- The pretraining learning rate warms up from 10−8 to 10−5 over 1K steps, then follows cosine decay to 10−6.
- Alignment training runs for up to 20 epochs on 6 × 64 V100s with batch size 2 per node.
B. Datasets Details
The dataset details include six task-specific question templates and ChatGPT-based generation covering seven tasks with textual execution information and manually written guidance examples.
- Question templates are designed for six tasks by replacing instruction, object, location, and action fields with processed sample information.
- Table 7 presents the detailed question templates used for data generation.
- ChatGPT-based generation covers seven tasks using robot instructions, execution time, object locations, and other scene information as text.Each prompt includes two manually written samples to guide more natural generation.
B.3. Details on Dataset Construction
The dataset construction combines robotics and human-object-interaction datasets, filtering complex robotics scenes and adding HOI data to diversify backgrounds for goal generation.
- The construction uses robotics datasets and human object interaction datasets as its two main dataset categories.
- Complex-scene robotics datasets are filtered to reduce incorrect object-location detections by Grounded-SAM.
- HOI datasets are included for goal generation because backgrounds within individual robotics datasets are largely similar.
C. More Visualization Results about Goal Generation
This section presents additional qualitative visualizations for goal generation and identifies the datasets used in the paper. The examples include generated RGB-D goal images and goal point clouds across held-in and unseen environments.
- Additional qualitative examples of goal generation are shown in Figures 5 and 6.
- The paper’s datasets are categorized into Robotics, HOI, and Room datasets.
- Figure 5 visualizes generated RGB-D goal images from held-in test data and unseen environments gathered from daily life.
- Figure 6 visualizes generated RGB-D goal images and goal point clouds for RLBench.