Source-linked AI summary
DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control
Teli Ma, Jia Zheng, Zifan Wang, Chunli Jiang, Andy Cui, Junwei Liang, Shuo Yang
TL;DR
Existing VLA models largely inherit static image-text representations, while integrating video generation into policy learning remains an open question. DiT4DiT jointly models video dynamics and actions using intermediate denoising features, achieving state-of-the-art performance across simulation and real-world evaluations.
Problem
Existing VLA systems leave spatiotemporal structure and physical dynamics to downstream action-data-limited training, while principled end-to-end integration of video generative models remains open.
Method
DiT4DiT couples video and action Diffusion Transformers in an end-to-end dual flow-matching framework, conditioning action prediction on intermediate video denoising features.
Results
DiT4DiT consistently outperforms strong VLA baselines across simulation and real-world experiments, achieving a 98.6% average success rate on LIBERO.
Takeaways & Limitations
Video generation provides an effective unsupervised pre-training signal and can serve as a scaling proxy for robot policy learning.
Takeaways & Limitations
Physical deployment currently relies on a single egocentric camera, which is susceptible to severe visual occlusions in complex bimanual tasks.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models have emerged as a promising paradigm for robot learning, but their representations are still largely inherited from static image-text pretraining, leaving physical dynamics to be learned from comparatively limited action data. Generative video models, by contrast, encode rich spatiotemporal structure and implicit physics, making them a compelling foundation for robotic manipulation. But their potentials are not fully explored in the literature. To bridge the gap, we introduce DiT4DiT, an end-to-end Video-Action Model that couples a video Diffusion Transformer with an action Diffusion Transformer in a unified cascaded framework. Instead of relying on reconstructed future frames, DiT4DiT extracts intermediate denoising features from the video generation process and uses them as temporally grounded conditions for action prediction. We further propose a dual flow-matching objective with decoupled timesteps and noise scales for video prediction, hidden-state extraction, and action inference, enabling coherent joint training of both modules. Across simulation and real-world benchmarks, DiT4DiT achieves state-of-the-art results, reaching average success rates of 98.6% on LIBERO and 50.8% on RoboCasa GR1 while using substantially less training data. On the Unitree G1 robot, it also delivers superior real-world performance and strong zero-shot generalization. Importantly, DiT4DiT improves sample efficiency by over 10x and speeds up convergence by up to 7x, demonstrating that video generation can serve as an effective scaling proxy for robot policy learning. We release code and models at https://dit4dit.github.io/.
1 INTRODUCTION
Existing VLA models largely inherit static image-text representations, while video generation offers spatiotemporal and physical-dynamics priors. DiT4DiT couples these video representations with action generation and reports stronger efficiency and manipulation performance.
- Static image-text pretraining leaves VLA models to learn spatiotemporal structure and physical dynamics during downstream policy training.
- Multi-stage video-model approaches make control indirect and leave open how generative video should serve as a policy-learning backbone.
- Video generation outperforms Grounding and FLARE-style objectives in convergence speed and final success rates as a proxy objective for policy learning.
- DiT4DiT extracts intermediate denoising features from future-frame generation to condition action learning within a unified dual-DiT, dual-flow-matching framework.Separate timesteps and noise scales support video and action generation in joint training.
- 98.6% and 50.8% average success rates establish state-of-the-art results on LIBERO and RoboCasa-GR1, respectively.The method also maintains advantages in real-world Unitree G1 deployments and uses a single egocentric camera.
2 RELATED WORKS
VLA research transfers semantic priors from vision-language models into robot control but remains constrained by static representations. Video-generation research instead supplies spatiotemporal and physical-dynamics priors, with recent work exploring shared representations and unified systems.
- VLA policies transfer semantic and linguistic priors from vision-language backbones to embodied control across generalist robot-learning tasks.
- Static image-text representations leave physical interactions and temporal state transitions to downstream fine-tuning, requiring thousands of hours of training data.
- Generative video models provide spatiotemporal priors and complex physical dynamics that address the physical limitations of static VLM representations.
- Recent models project visual dynamics and control signals into shared latent spaces to combine capabilities such as forward simulation and inverse dynamics.
- Cosmos Policy further simplifies adaptation by fine-tuning a pretrained video diffusion model for policy learning.
3 VALIDATION OF VIDEO GENERATION AS A SCALING PROXY
The paper validates video generation as a scaling proxy by comparing it with grounding and FLARE-style latent modeling on 24 RoboCasa tasks. Video generation converges faster and scales more efficiently with data.
- The comparison evaluates object-level grounding and FLARE-style latent modeling alongside video generation as proxy objectives for robot control.The study keeps trainable parameter scales consistent across the VLM and video backbones.
- 24 tabletop manipulation tasks involving the GR1 humanoid robot form the RoboCasa simulation validation setting.Backbones are self-supervised pretrained on the target dataset and frozen during action-expert fine-tuning.
- Up to 7× faster convergence enables high-performance policies to emerge earlier during training.The authors report that the video proxy captures essential manipulation cues early in training.
- Up to 10× higher data efficiency and consistent improvement as data volume increases demonstrate stronger scaling behavior than semantic-centric methods.
4 DIT4DIT: UNLEASHING THE POTENTIAL OF VIDEO MODEL
DiT4DiT jointly models future video dynamics and robot actions with coupled Video and Action Diffusion Transformers. It uses flow matching and asymmetric timesteps to extract stable visual features for action prediction while training both modules together.
- Flow matching: Flow matching learns a velocity field transporting samples from Gaussian noise at τ = 1 to clean data at τ = 0.Inference integrates the learned velocity field backward from noise toward the data distribution, using Euler updates.
- Dual-DiT architecture: The video pathway predicts future-frame dynamics from encoded observations, while hidden states from its denoising process provide visual conditions for the Action DiT.The action head combines these visual features with proprioceptive state embeddings and noisy action trajectories to predict an action velocity field.
- Dual-DiT architecture: DiT4DiT unifies video generation and inverse-dynamics action inference through jointly optimized Video and Action DiTs.The dual flow-matching mechanism trains both modules within one learning paradigm.
- Tri-timestep scheme: A fixed feature-extraction timestep τf yields deterministic visual representations, unlike the video module’s uniformly sampled τv used across the denoising trajectory.Fixing τf selects a consistent backbone operating point for action prediction during training and inference.
- Tri-timestep scheme: The action timestep τa is independently sampled from a Beta distribution to allocate more training capacity to critical flow-trajectory stages.This decoupling lets the action decoder learn inverse dynamics while remaining conditioned on stable visual features.
5 EXPERIMENTS
DiT4DiT is evaluated across simulation and real-world settings, including benchmark comparisons, household manipulation, zero-shot generalization, and ablations of representation extraction and training. It achieves strong performance across these evaluations, with gains over state-of-the-art and parameter-matched baselines.
- LIBERO benchmark results: 98.6% average success rate establishes DiT4DiT as state-of-the-art on the LIBERO simulation benchmark.The model is trained from scratch without action data outside the benchmark.
- LIBERO benchmark results: 97.6% success on LIBERO-Long demonstrates strong performance on extended-horizon behaviors, while LIBERO-Object reaches 99.6% and LIBERO-Goal reaches 98.6%.DiT4DiT also improves consistently over the Qwen3DiT baseline’s 96.6% average across all four suites.
- RoboCasa-GR1 tabletop benchmark results: 50.8% average success rate sets a new state-of-the-art on 24 RoboCasa-GR1 tabletop tasks.This exceeds GR00T-N1.5 and GR00T-N1.6 by 9.0 and 10.0 absolute percentage points, respectively.
- RoboCasa-GR1 tabletop benchmark results: 14.6% absolute improvement over Qwen3DiT’s 36.2% average supports the benefit of generative video conditioning for complex inverse dynamics.DiT4DiT achieves the highest success rate on 16 of 24 evaluated tasks, including tasks requiring precise spatial coordination and complex physical interaction.
- Generalization capability: 70% success on zero-shot Arrange Flower and 50% on Stack Cup (Number) show transfer under unseen objects and changed object counts.Qwen3DiT scores 0% on Arrange Flower (Category) and on three of four evaluated zero-shot tasks overall.
- Ablations: A single denoising step yields the highest hidden-feature performance, whereas additional steps produce monotonically degrading success rates.The ablation varies extraction denoising steps from 1 to 32 across five selected RoboCasa tasks.
- Ablations: Joint training increases the silhouette score from 0.09 to 0.17 and produces clearer Early-to-Middle-to-Late temporal feature separation.The comparison is against a decoupled scheme that optimizes the video model and action policy independently.
6 CONCLUSION
DiT4DiT unifies video and action diffusion through dual flow matching, using intermediate video denoising features for physics-aware control. It achieves strong simulation, real-world transfer, and robustness under distribution shifts.
- DiT4DiT unifies video and action Diffusion Transformers through a dual flow-matching objective.
- 98.6% and 50.8% average success rates are achieved on LIBERO and RoboCasa-GR1, respectively.
- Intermediate denoising features condition action prediction instead of fully reconstructed future frames.
- DiT4DiT maintains strong transfer performance on the Unitree G1 humanoid robot under real-world dynamics.
- The method improves robustness under distribution shifts involving unseen object categories and object or scene variations.
A.1 MODEL & TRAINING CONFIGURATIONS
Table 4 summarizes the model and training configurations used for DiT4DiT.
- Table 4 reports the model and training configurations.
- The table organizes configuration information for the DiT4DiT framework.
- These configurations are presented as part of the paper’s experimental setup.
A.2 DATASET CONFIGURATION
The datasets are divided into simulation and real-world pipelines to assess learning capabilities and physical deployment. They include benchmark episodes, large-scale pretraining data, and robot-specific teleoperated demonstrations.
- The dataset usage is partitioned into separate simulated-benchmark and real-world-deployment pipelines.
- Real-world system setup: Figure 9 depicts the Unitree G1 platform with dual ALOHA 2 grippers and an Intel RealSense D435i camera for data collection.
- Simulated benchmark data: RoboCasa-GR1 training uses 24,000 episodes from the Fourier GR1 Unified 1K dataset, collected with a 29-DoF GR1 humanoid.
- Simulated benchmark data: LIBERO training uses its official dataset containing 1,693 episodes based on a 7-DoF Franka Emika Panda arm.
- Pre-training data for real-world tasks: Real-world pretraining uses 241,450 episodes from the scaled Fourier GR1 Pretrain 10K dataset.
- Real-world fine-tuning data: Real-world fine-tuning uses 1,400 teleoperated demonstrations, with 200 episodes per task, for the Unitree G1’s continuous 16-DoF action space.
A.3 REAL-WORLD EXPERIMENT SETTING
The real-world system uses a Unitree G1 humanoid with bimanual manipulation hardware and egocentric RGB sensing. Demonstrations are collected through VR teleoperation managed by XRoboToolkit.
- The Unitree G1 has a 16-DoF action space driven by dual 7-DoF arms.
- Each arm uses an ALOHA 2 gripper to support high-precision bimanual manipulation.
- An Intel RealSense D435i camera mounted on the robot’s head captures egocentric RGB observations at 640x480 resolution.
- A PICO VR headset and handheld controllers enable human operators to provide natural demonstrations by mapping motion to robot joints.
- XRoboToolkit manages the software stack and synchronizes multimodal sensor data for recording demonstration trajectories.
A.4 LIMITATIONS AND DISCUSSION
DiT4DiT’s physical deployment has scope boundaries around visual sensing and pre-training scale. The authors identify occlusion robustness and broader cross-embodiment data as key directions for future work.
- Sensing limitations: Single egocentric-camera deployment is vulnerable to severe visual occlusions that can disrupt temporal visual features during complex bimanual tasks.The robot’s arms or larger objects may temporarily block the camera’s line of sight.
- Future directions: Auxiliary inputs such as wrist-mounted cameras or tactile feedback are proposed to maintain robust state estimation during severe occlusions.These modalities would be fused with the video DiT backbone.
- Data scale: The real-world zero-shot generalization results use a pre-training corpus representing 15% of the data volume used by contemporary large-scale models like GR00T.The authors frame this data efficiency as motivation for scaling across diverse robotic embodiments.
- Future directions: Future work could scale pre-training across embodiments with varied kinematics, grippers, and camera parameters.The proposed direction targets more generalized robotic foundation models.