Source-linked AI summary
RoboDreamer: Learning Compositional World Models for Robot Imagination
Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit-Yan Yeung, Chuang Gan
TL;DR
RoboDreamer addresses limited generalization in robot video world models, which struggle with unseen combinations of actions, objects, and spatial relations. It factorizes video generation over parsed language primitives and multimodal goals, achieving strong instruction alignment and promising robot-planning deployment. The approach remains limited by single-camera inputs and poor generalization to many real-world images.
Problem
Existing robot video models struggle with precise spatial rearrangements and language instructions that differ from training data, where datasets are scarce and biased.
Method
RoboDreamer parses language into action and relation primitives and conditions compositional diffusion models on these components alongside multimodal goal inputs.
Results
RoboDreamer achieves strong alignment with multimodal tasks and promising robot-manipulation deployment, including a 15% success rate on challenging long-term RLBench tasks.
Takeaways & Limitations
Factorized generation supports zero-shot language-combination generalization, multimodal-conditioned video generation, and potential robotic manipulation.
Takeaways & Limitations
RoboDreamer is limited to single-camera views and generalizes poorly to many real-world images tested.
Abstract
from arXiv · showhide
Text-to-video models have demonstrated substantial potential in robotic decision-making, enabling the imagination of realistic plans of future actions as well as accurate environment simulation. However, one major issue in such models is generalization -- models are limited to synthesizing videos subject to language instructions similar to those seen at training time. This is heavily limiting in decision-making, where we seek a powerful world model to synthesize plans of unseen combinations of objects and actions in order to solve previously unseen tasks in new environments. To resolve this issue, we introduce RoboDreamer, an innovative approach for learning a compositional world model by factorizing the video generation. We leverage the natural compositionality of language to parse instructions into a set of lower-level primitives, which we condition a set of models on to generate videos. We illustrate how this factorization naturally enables compositional generalization, by allowing us to formulate a new natural language instruction as a combination of previously seen components. We further show how such a factorization enables us to add additional multimodal goals, allowing us to specify a video we wish to generate given both natural language instructions and a goal image. Our approach can successfully synthesize video plans on unseen goals in the RT-X, enables successful robot execution in simulation, and substantially outperforms monolithic baseline approaches to video generation.
1. Introduction
RoboDreamer addresses the difficulty of generating accurate robot-action videos for precise spatial relations and unseen instruction combinations by factorizing generation through language components and multimodal conditions.
- Motivation: Robotics commands require precise object rearrangements and spatial relations, but existing models struggle especially with scarce, biased data and unfamiliar instructions.For example, existing methods misplace the specified object when asked to move a Pepsi can near a plastic bottle.
- Method: RoboDreamer parses instructions into action and relation primitives, then conditions separate diffusion models on these components to generate videos compositionally.The factorization targets spatial-relationship nuances while retaining compatibility with conventional text-to-video systems.
- Compositional Generalization: Composing previously seen parsed components enables generalization to new language instructions when each component remains in distribution.This addresses the lack of systematic data covering all possible robot actions and combinations.
- Multimodal Specification: RoboDreamer also composes language with goal images and sketches, allowing richer multimodal task specifications at inference time.These visual modalities provide detailed spatial information about desired final configurations.
- Results: The paper reports strong alignment under multimodal instructions and promising deployment results on robot manipulation tasks.The contributions specifically identify multimodal task alignment and robot-execution performance as empirical outcomes.
2. Background
The paper formulates robotic planning as text-conditioned video generation: a world model predicts future image states, and an inverse-dynamics policy converts predicted videos into executable actions.
- Planning Formulation: UPDP represents robotic planning with observation space X, textual task space C, horizon H, and a conditional video generator ρ.The generator synthesizes future image-state videos from an initial observation and textual task description.
- Planning Formulation: Decision-making becomes learning a generator that produces future image states conditioned on a natural-language instruction.The implementation uses a video diffusion model based on prior source code.
- Plan Execution: An inverse-dynamics policy takes adjacent frames from a synthesized video plan and predicts actions sequentially to realize that plan.Actions are inferred from each pair of neighboring observations in the predicted trajectory.
- Plan Execution: Closed-loop planning periodically regenerates video plans to reduce the effect of intermediate action-estimation errors.Execution is repeatedly updated using newly predicted plans rather than relying on one fixed video.
3. RoboDreamer
RoboDreamer factorizes video generation using parsed language components, enabling compositional generalization to unseen instructions and multimodal conditioning. Experiments report improved task alignment and robot-planning performance over baseline approaches.
- 3.1. Text Parser: RoboDreamer parses instructions into action and spatial-relation components, then conditions separate diffusion models on these components.The parser separates verb phrases from prepositional phrases so models can capture object relationships more precisely.
- 3.2. Compositional Generation: The model represents video generation as a product of distributions conditioned on parsed language components, allowing novel combinations at sampling time.Learned score functions can be combined for unseen compositions when the individual components remain in distribution.
- 3.2. Compositional Generation: A hybrid training objective encourages both the composed model and individual component-conditioned score functions to denoise videos accurately.The objective randomly selects subsets of language components to train the component-level score functions.
- 3.3. Multi-modal Composition: RoboDreamer extends the factorized formulation to combine language with variable numbers of multimodal conditions, including goal images and sketches.These conditions can be recombined at inference time even when paired language-modality combinations were absent during training.
- 4.1. Evaluation on Video Generation: Human evaluation shows RoboDreamer outperforms baselines, especially on unseen tasks, by more accurately inferring object relationships and task completion.Figure 4 reports substantially more accurate synthesis than a single monolithic text-to-video model for novel language combinations.
- 4.2. Evaluation on Robotic Planning: In RLBench, RoboDreamer achieves a 15% success rate on long-term tasks with predicted future observations, while baseline models struggle on these tasks.The reported results associate synthesized videos with feasible robot planning and execution.
5. Related Work
RoboDreamer builds on diffusion-based decision-making and compositional-generation research while targeting compositional video world models for robotics. Its factorization is intended to support unseen combinations of detailed language and image specifications.
- Diffusion models have been used to generate robot-planning trajectories from low-level state and action spaces, but these approaches are difficult to generalize to high-dimensional settings.
- VideoAdapter transfers pretrained general text-to-video models to domain-specific tasks with small datasets but fails to generalize to unseen tasks or text descriptions.
- HIP addresses long-horizon tasks by composing several expert foundation models.
- RoboDreamer decomposes the learned video probability distribution into compositional components, enabling combinations of detailed language and image specifications unseen during training.
6. Conclusion
RoboDreamer is presented as a compositional video-generation approach for robotics that improves generalization and supports multimodal conditioning. The paper reports zero-shot generalization, multimodal video generation, and potential use in robotic manipulation, while identifying several practical limitations.
- RoboDreamer generalizes significantly better than prior robotics video-generation methods.
- Experimental results verify zero-shot generalization, multimodal-conditioned video generation, and potential application to robotic manipulation tasks.
- RoboDreamer is limited to single-camera views, restricting tasks that require detailed 3D information from multiple cameras.
- The model generalizes poorly to many tested real-world images, and video-generation capabilities remain constrained in moving-camera settings.
A. Experimental Details
The implementation uses cascaded diffusion models built on AVDC and Imagen, with specified U-Net components, temporal attention, channel scaling, training resources, and progressive video upsampling.
- RoboDreamer is built on AVDC and Imagen and uses a three-stage cascaded diffusion model for super-resolution.
- The video diffusion models use four ResNetBlocks within a U-Net, combining spatial-temporal convolutions with cross-attention conditioned on instructions.
- Temporal-attention layers are added to the last encoder block and first decoder block of the U-Net.
- The models use base channel 128 and channel multiplier [1, 2, 4, 8].
- Training uses batch size 256, learning rate 5e-5, and approximately 100 V100 GPUs.
- Base video diffusion models are trained on 8 × 64 × 64 videos before upsampling to 8 × 128 × 128 and 8 × 256 × 256.
A.2. Other Details
The additional experimental details describe the RT-1 and RLBench settings, the inverse-dynamics model, and the human evaluation protocol.
- RT-1 contains about 70k demonstrations averaging 44 frames, covering about 500 tasks across categories including pick, place, open, close, knock, and pull.
- The inverse-dynamics model predicts actions from two adjacent frames and the current state using a ResNet18 backbone followed by an MLP.
- The inverse-dynamics model is trained with Adam at learning rate 1e-4 for 10K steps.
- RLBench experiments use a Franka Panda arm and gripper with seven degrees of freedom and an 8-dimensional action space plus gripper state.
- Human evaluation rates each sample with at least three raters across approximately 128 samples and more than 20 text prompts.
B.1. Visualization on RLBench
This section visualizes the RLBench tasks in Figure 6.
- RLBench tasks are visualized in Figure 6.
B.2. More Results on Video Generation
Additional video-generation results are available on the RoboDreamer website.
- More video-generation results are shown on the RoboDreamer website.The paper directs readers to https://robovideo.github.io/ for these results.
B.3. IMO Metrics
The paper evaluates multimodal generation using object-bounding-box overlap and human evaluation, reporting strong human-evaluation performance and good video quality for both multimodal variants.
- IMO evaluates target-object bounding-box overlap using pretrained GroundingDino detections.IMO is defined as the IoU of the detected target-object bounding boxes.
- The IMO results are consistent with human evaluation, supporting better alignment through multimodal inputs.
- RoboDreamer (t+s) and RoboDreamer (t+i) achieve strong human-evaluation performance and good video quality.