Source-linked AI summary
Ctrl-World: A Controllable Generative World Model for Robot Manipulation
Yanjiang Guo, Lucy Xiaoyang Shi, Jianyu Chen, Chelsea Finn
TL;DR
Generalist robot policies are difficult to evaluate and improve on unfamiliar objects and instructions because real-world rollouts and expert corrective data are costly and slow. Ctrl-World provides a controllable multi-view world model for policy-in-the-loop imagination, and synthetic post-training raises novel-instruction success from 38.7% to 83.4%.
Problem
Evaluating and improving brittle generalist robot policies requires costly real-world rollouts and additional expert corrective data, while prior world models lack compatible multi-view, fine-grained, long-horizon interaction capabilities.
Method
Ctrl-World adapts a pretrained video diffusion model into a controllable multi-view world model using frame-level action conditioning and pose-conditioned memory retrieval for policy-in-the-loop imagination rollouts.
Results
44.7% improvement: post-training raises π0.5’s success rate from 38.7% to 83.4% on downstream tasks with novel instructions and unfamiliar objects.
Takeaways & Limitations
Ctrl-World supports policy evaluation aligned with real-world instruction-following behavior and policy improvement using successful trajectories generated entirely in imagination.
Takeaways & Limitations
The model can fail on precise interactions or long-horizon reasoning, is sensitive to initial observations, and may not improve aspects beyond instruction following.
Abstract
from arXiv · showhide
Generalist robot policies can now perform a wide range of manipulation skills, but evaluating and improving their ability with unfamiliar objects and instructions remains a significant challenge. Rigorous evaluation requires a large number of real-world rollouts, while systematic improvement demands additional corrective data with expert labels. Both of these processes are slow, costly, and difficult to scale. World models offer a promising, scalable alternative by enabling policies to rollout within imagination space. However, a key challenge is building a controllable world model that can handle multi-step interactions with generalist robot policies. This requires a world model compatible with modern generalist policies by supporting multi-view prediction, fine-grained action control, and consistent long-horizon interactions, which is not achieved by previous works. In this paper, we make a step forward by introducing a controllable multi-view world model that can be used to evaluate and improve the instruction-following ability of generalist robot policies. Our model maintains long-horizon consistency with a pose-conditioned memory retrieval mechanism and achieves precise action control through frame-level action conditioning. Trained on the DROID dataset (95k trajectories, 564 scenes), our model generates spatially and temporally consistent trajectories under novel scenarios and new camera placements for over 20 seconds. We show that our method can accurately rank policy performance without real-world robot rollouts. Moreover, by synthesizing successful trajectories in imagination and using them for supervised fine-tuning, our approach can improve policy success by 44.7\%.
1 INTRODUCTION
Ctrl-World addresses the costly, slow evaluation and limited improvement of brittle generalist robot policies with a controllable world model for policy-in-the-loop imagination rollouts.
- Motivation: Generalist policies remain brittle on unfamiliar objects and instructions despite competence across many manipulation tasks.Evaluation requires many repeated real-world rollouts, while improvement often requires additional expert corrective data.
- Motivation: Prior action-conditioned world models inadequately support policy-in-the-loop interaction because they use single views, lack fine-grained action control, and lose long-horizon consistency.Single-view prediction can cause partial observability and hallucinated interactions, while high-frequency action effects and long-horizon dynamics remain difficult to model.
- Approach: Ctrl-World combines joint multi-view prediction, frame-level action conditioning, and pose-conditioned memory retrieval for multi-step imagination rollouts.The design includes wrist-camera prediction, action-aligned visual dynamics, and history-based temporal stabilization.
- Results: Trained on DROID, Ctrl-World generalizes to novel scenes and camera placements while sustaining coherent rollouts for over 20 seconds.The model is used to evaluate instruction-following alignment with real-world rollouts and to synthesize trajectories for policy improvement.
- Results: Ctrl-World supports an imagination-based workflow that ranks policy performance and improves policies using targeted synthetic trajectories.The paper reports both real-world ranking alignment and downstream policy improvement on unseen objects and novel instructions.
2 RELATED WORKS
Prior robotics work spans realistic video generation and action-conditioned predictive models, but many approaches use fake action labels, task-specific models, or passive prediction settings.
- Video Generation Models for Robotics: Recent video-generation methods synthesize realistic, temporally consistent robotic content and sometimes generate trajectories with fake action labels for policy learning.Other methods use video models directly as policy backbones.
- Action-Conditioned World Models: Ctrl-World builds on these directions by targeting controllable multi-view prediction for interactions with generalist robot policies.Its intended setting differs from passive video prediction and task-specific modeling.
- Action-Conditioned World Models: Action-conditioned world models have been studied in low-dimensional state spaces and image-observation settings, including task-specific predictive models.Ctrl-World instead focuses on generalist, multi-task world modeling.
3 PROBLEM FORMULATION
The formulation models autoregressive interaction between a generalist policy and a world model: policy actions are executed in imagination, predicted observations are returned, and the cycle continues.
- Policy and observations: A generalist policy maps multi-view observations and a language instruction to an H-step action chunk.The observation includes multiple camera images and the robot pose.
- World-model prediction: The world model predicts future multi-view observations resulting from each action in the policy’s action chunk.This establishes the model as a predictive simulator for policy-proposed actions.
- Autoregressive interaction: The predicted final observation is fed back to the policy to produce the next action chunk, creating autoregressive long-horizon rollouts in imagination space.Policy and world model repeatedly exchange actions and observations.
4 CONTROLLABLE WORLD MODEL FOR ROBOT MANIPULATION
Ctrl-World adapts a pretrained video diffusion model into a policy-compatible simulator through multi-view prediction, pose-conditioned memory, frame-level action conditioning, and imagination-based evaluation and improvement.
- Design requirements: Ctrl-World must support multi-view observations, reliable action control, and long-horizon temporal consistency despite occlusions.The model is initialized from a pretrained video diffusion backbone.
- Multi-View Joint Predictions: Joint multi-view prediction combines third-person and wrist-camera views to provide global context and precise interaction information.The wrist view is particularly important for contact-rich object interactions.
- Pose-conditioned Memory Retrieval Mechanism: Pose-conditioned memory retrieval samples sparse history frames and embeds their robot poses to anchor future predictions to relevant past states.This mechanism reduces long-rollout drift while limiting context length.
- Frame-level Action Conditioning: Frame-level action conditioning associates each future frame with the policy’s action-derived Cartesian arm pose through frame-wise cross-attention.The conditioning is designed to align high-frequency actions with visual dynamics.
- Training Objective: The model is initialized from Stable-Video-Diffusion and fine-tuned with diffusion loss after adding an action-projection MLP.The training target is the future observation sequence conditioned on history, poses, actions, and current observations.
- Policy Evaluation and Improvement: Policy evaluation uses closed-loop synthetic trajectories judged for success, while policy improvement searches for successful synthetic trajectories and fine-tunes on them.Rollout diversity is encouraged through instruction rephrasing and new initial arm states.
5 EXPERIMENTS
Experiments assess Ctrl-World’s rollout quality, controllability, consistency, policy evaluation, and policy improvement across DROID-based tasks. The model supports long-horizon multi-view interaction, tracks policy instruction-following behavior, and generates synthetic data that improves downstream success.
- World Model Quality: Ctrl-World jointly predicts third-view and wrist-view observations, producing precise trajectories where single-view baselines hallucinate or miss robot–object interactions.Examples include failing to move a green towel or grasp a red bowl; wrist-view prediction supplies fine-grained contact and object-state information.
- World Model Quality: Ctrl-World generates distinct future rollouts for actions differing by only a few centimeters, with controllability supported by multi-view prediction, frame-level conditioning, and memory retrieval.Attention for a predicted t = 4 s frame strongly focuses on a t = 0 s frame with a similar pose.
- World Model Quality: Multi-view information and historical-frame retrieval enable consistent wrist-view predictions despite dramatic field-of-view changes, while removing key components reduces performance.Ablations removing memory, frame-level action conditioning, or multi-view joint prediction all lead to performance drops.
- Policy Evaluation: World-model rollouts closely correlate with real-world instruction-following behavior but underestimate execution success and do not fully model complex physics or policy retries.Observed gaps include collisions, sliding objects, rotations, and failure-retry behavior outside the training distribution.
6 CONCLUSION
Ctrl-World supports closed-loop policy evaluation and improvement within imagination, but remains limited on precise interactions, long-horizon reasoning, and broader performance aspects.
- Ctrl-World supports closed-loop policy evaluation and improvement entirely within the model’s imagination.
- The experiments focus on improving instruction following and may not extend to low-level success rates on previously seen instructions.
A MORE DETAILS FOR WORLD MODEL LEARNING
The world model follows a Stable Video Diffusion architecture, encodes history and action inputs, and predicts future frames from DROID trajectories.
- The model initializes from a Stable Video Diffusion checkpoint, adding a 3-layer MLP for 7-dimensional Cartesian actions.
- Each input uses 7 history frames, while a one-second action-conditioning window contains 15 action steps.
- The 15 action steps are temporally downsampled to 5 steps before entering the model.
- Each frame contains three 192 × 320 images encoded into latent features of shape 24 × 40.
- Training uses all 95k DROID trajectories, sampling a current frame, backward history frames, and subsequent future-frame targets.
- Training runs for 100k steps on 2×8 H100 GPUs with batch size 64 and takes approximately 2–3 days.
B MORE DETAILS FOR POLICY EVALUATION
Ctrl-World enables autoregressive interaction between generalist policies and a world model entirely in imagination space. Policy outputs are converted from joint velocities to Cartesian poses before the world model predicts future multi-view frames, supporting evaluation under human-defined criteria.
- Policy–world-model interaction: The official π0-DROID, π0-FAST-DROID, and π0.5-DROID policies interact with Ctrl-World despite using joint angles and camera views while the world model uses end-effector poses.This interface mismatch is handled through the learned adapter and forward kinematics conversion.
- Policy–world-model interaction: The policy outputs H-step joint velocities, which an adapter converts into future joint configurations before forward kinematics computes Cartesian-space poses.The adapter is a two-layer MLP trained to bridge the policy’s joint-angle interface and Ctrl-World’s Cartesian-pose conditioning.
- Policy–world-model interaction: Ctrl-World predicts the next H frames from the current multi-view observation, calculated Cartesian poses, and historical Cartesian poses.The predicted observation is returned to the policy, enabling fully autoregressive rollouts in imagination space.
- Evaluation criteria: Table 3 compares instruction-following and success rates across methods and tasks.The supplied table passage identifies the comparison dimensions but does not provide the individual values.
- Evaluation criteria: Policy evaluation distinguishes instruction following from full task success using human-annotated trajectory outcomes.Criteria cover pick-place, towel folding, drawer placement, table wiping, laptop closing, tissue pulling, and stacking.
C MORE DETAILS FOR POLICY IMPROVEMENT
The policy-improvement process fine-tunes π0.5-DROID on synthetic trajectories across spatial, shape, directional towel-folding, and novel-object tasks. Detailed success rates are reported by category in Tables 4–7.
- Finetuning process: π0.5-DROID is fine-tuned on a synthetic dataset for 2k steps using 4 H100 GPUs.The pretrained checkpoint and official open-source codebase are used for post-training.
- Improvement tasks: Spatial-understanding tasks require selecting objects by specified table locations and placing them in a box.Scenes contain 2–6 random objects, with instructions such as placing the far-left or top-right object into the box.
- Improvement tasks: Shape-understanding tasks require distinguishing objects by size when objects share other attributes.Scenes contain 2–3 random objects, including instructions to pick the larger red block.
- Improvement tasks: Directional towel-folding tasks specify a folding direction, such as folding the towel from left to right.The towel is presented with possible distractor objects on the table.
- Improvement tasks: Novel-object tasks introduce unseen objects such as a glove and stapler that the pretrained policy cannot identify well.This category tests improvement on objects outside the pretrained policy’s stronger recognition setting.
- Results: Detailed task success rates are reported for spatial, shape, directional towel-folding, and novel-object categories.Tables 4–7 separately cover these four policy-improvement categories.