Source-linked AI summary
Chain of World: World Model Thinking in Latent Motion
Fuxiang Yang, Donglin Di, Lulu Tang, Xuancheng Zhang, Lei Fan, Hao Li, Chen Wei, Tonghua Su, Baorui Ma
TL;DR
VLA models need predictive temporal structure, but world-model approaches reconstruct redundant backgrounds while latent-action methods omit continuous dynamics and world knowledge. CoWVLA combines a disentangled video-VAE motion space with terminal-frame prediction and action-aligned autoregressive modeling, outperforming both approach families on robotic benchmarks.
Problem
VLA models lack future prediction, while world-model and latent-action approaches each omit complementary aspects of efficient temporal world modeling.
Method
CoWVLA uses a disentangled latent motion representation, predicts continuous dynamics and a terminal frame, and aligns them with discrete actions through unified autoregressive modeling.
Results
CoWVLA outperforms both world-model and latent-action approaches on LIBERO and SimplerEnv benchmarks.
Takeaways & Limitations
CoWVLA provides a more efficient pretraining route by capturing temporal evolution without reconstructing intermediate pixels.
Takeaways & Limitations
The latent motion space depends on the pretrained video VAE’s quality and domain coverage, while the model requires substantial computational resources.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models are a promising path toward embodied intelligence, yet they often overlook the predictive and temporal-causal structure underlying visual dynamics. World-model VLAs address this by predicting future frames, but waste capacity reconstructing redundant backgrounds. Latent-action VLAs encode frame-to-frame transitions compactly, but lack temporally continuous dynamic modeling and world knowledge. To overcome these limitations, we introduce CoWVLA (Chain-of-World VLA), a new "Chain of World" paradigm that unifies world-model temporal reasoning with a disentangled latent motion representation. First, a pretrained video VAE serves as a latent motion extractor, explicitly factorizing video segments into structure and motion latents. Then, during pre-training, the VLA learns from an instruction and an initial frame to infer a continuous latent motion chain and predict the segment's terminal frame. Finally, during co-fine-tuning, this latent dynamic is aligned with discrete action prediction by jointly modeling sparse keyframes and action sequences in a unified autoregressive decoder. This design preserves the world-model benefits of temporal reasoning and world knowledge while retaining the compactness and interpretability of latent actions, enabling efficient visuomotor learning. Extensive experiments on robotic simulation benchmarks show that CoWVLA outperforms existing world-model and latent-action approaches and achieves moderate computational efficiency, highlighting its potential as a more effective VLA pretraining paradigm. The project website can be found at https://fx-hit.github.io/cowvla-io.
1. Introduction
CoWVLA addresses complementary limitations in world-model and latent-action VLAs by combining compact, disentangled motion representations with continuous temporal reasoning and terminal-frame prediction.
- World-model VLAs improve action learning through future-frame prediction but spend capacity reconstructing static, redundant background pixels.
- Latent-action methods compactly encode frame-to-frame transitions, yet lack temporally continuous dynamics and world knowledge about scene evolution.
- CoWVLA unifies world modeling and latent-action learning through continuous latent-motion sequences and terminal keyframe prediction.
- A pretrained video VAE disentangles video segments into structure and motion representations for compact, interpretable supervision.
- CoWVLA achieves state-of-the-art performance across multiple benchmarks, surpassing existing world-model and latent-action approaches.
2. Related Work
Related work spans unified vision-language-action models, future-state prediction for robotic world modeling, compact latent actions, and disentangled video representations.
- Vision-Language-Action Models: VLA models generate actions from visual and language inputs within a unified framework for robotic control.
- World Models for Robotics: World-model methods commonly capture environment states and future evolution, with VLA systems using future visual states to provide implicit world knowledge.
- Latent Actions for Robotics: Latent-action methods learn compact transitions between two frames and use pseudo-action supervision to improve robotic control.
- Latent Actions for Robotics: Existing latent actions can encode static appearance and context, motivating explicit separation of structure from motion for cleaner representations.
- Video Compression and Decoupling: Video representation methods compress visual information into disentangled latent spaces that separately encode spatial structure and temporal motion.
3. Method
CoWVLA builds a unified VLA around disentangled latent motion, using a video VAE and a motion-query decoder to connect continuous dynamics with action prediction. Its two-stage training predicts future observations during pretraining, then jointly models sparse keyframes and discrete actions for multi-step control.
- Overall Framework: The framework partitions action sequences into fixed-length chunks, extracts their first observations as keyframes, and quantizes visual and action sequences into discrete tokens.A learnable motion query summarizes past context and conditions subsequent vision or action generation.
- Latent Motion Extractor: A pretrained video VAE encodes video subsequences into structure and directional motion features, which are combined into a unified latent motion representation.The decoder reconstructs video using structure and motion latents, with reconstruction, perceptual, adversarial, and KL losses.
- Co-Fine-Tuning: The co-fine-tuning objective jointly enforces discrete-action accuracy, continuous latent-motion consistency, and consistency at sparse visual checkpoints.This alignment supports stable multi-step control driven by predicted dynamics.
- Pre-training to Think in Latent Motion: During pretraining, the VLA infers continuous latent dynamics from an instruction and initial frame while predicting the segment’s terminal frame.Causal masking prevents the motion query from accessing the future terminal frame, and training combines latent-motion supervision with terminal-frame consistency.
- Co-Fine-Tuning: During co-fine-tuning, a single motion query aggregates the temporal window while the decoder autoregressively predicts alternating sparse keyframes and action tokens.The query is causally masked from future keyframes and actions, preserving explicit dynamics reasoning under sparse observations.
4. Experiments
Experiments evaluate CoWVLA’s disentangled motion representation, downstream performance, ablations, and computational trade-offs across robotic benchmarks. CoWVLA improves cross-domain success and balances performance with moderate pre-training efficiency.
- Comparison with SOTA Methods: CoWVLA achieves 0.956 on LIBERO and 0.760 on SimplerEnv, outperforming UniVLA’s 0.950 and 0.698 on the respective benchmarks.The authors report higher absolute performance and greater cross-domain stability than UniVLA.
- Disentangled Latent Analysis: The structure latent preserves scene layout and appearance, while the motion latent captures robot-arm trajectories and fine-grained temporal dynamics.Cross-reconstruction further shows that combining structure and motion latents highlights motion-affected regions.
- Disentangled Latent Analysis: Robot-data fine-tuning raises SimplerEnv-WidowX average task success from 0.729 to 0.760 while improving reconstruction quality.The result supports adapting motion latents to robot-domain dynamics for downstream policy learning.
- Ablation Analysis: CoWVLA’s motion and cot configuration reaches 0.947 success, exceeding the motion-only configuration’s 0.936 after adding terminal-frame supervision.The ablation attributes gains to latent motion guidance during action inference and terminal-frame supervision during pre-training.
- Ablation Analysis: With λ2 = 0, increasing λ1 from 0.1 to 1.0 raises success from 0.936 to 0.945, while λ1 = 0.1 and λ2 = 0.01 reaches 0.955.The study finds that low-weight visual-token prediction combined with latent-motion supervision most effectively guides real-action inference.
- Efficiency Analysis: The motion & cot configuration surpasses UniVLA while maintaining a better efficiency-performance balance; the motion configuration is the second-fastest.Efficiency is assessed using training speed, GPU memory usage, and task success rate.
5. Conclusion
CoWVLA integrates world-model temporal reasoning with disentangled latent motion, predicting continuous motion chains and terminal keyframes without reconstructing intermediate pixels. Experiments show stronger benchmark performance, but the method remains dependent on pretrained VAE quality and substantial computational resources.
- CoWVLA combines temporal reasoning with disentangled structure–motion latents for world modeling in latent space.
- The method predicts a continuous latent motion chain and terminal keyframe from an instruction and initial observation.
- CoWVLA outperforms world-model and latent-action approaches on LIBERO and SimplerEnv while improving dynamic consistency and visuomotor grounding.
- The latent motion space depends on pretrained video VAE quality and domain coverage, while the model requires substantial computational resources.
1. Implementation Details
The implementation uses a large pretrained VLA backbone, staged training, video-derived latent motion, and benchmark-specific co-fine-tuning. The world-model formulation predicts environment evolution before incorporating action-conditioned modeling.
- Training configuration: Training uses 32 A800 GPUs for VLA pretraining and 16 A800 GPUs for co-fine-tuning, with sequence lengths of 2500 and 3200 tokens.
- World-model formulation: The world-model component first predicts future environment evolution from language and an initial state, then introduces action-conditioned modeling during co-fine-tuning.
- Latent dynamics: Latent motion encodes continuous, decoupled dynamics over a temporal window rather than explicitly performing multi-step rollouts.
2. Additional Results
Additional analyses show that CoWVLA benefits from intermediate temporal granularity, robot-domain motion-latent adaptation, and structure–motion disentanglement. Across CALVIN and SimplerEnv-Google Robot, it achieves strong task performance and improved future-frame guidance.
- Sensitivity analysis: N = 2 keyframes and l_a = 10 achieve the best LIBERO performance, with both hyperparameters showing an inverted-U trend.
- Sensitivity analysis: Sparse keyframes constrain inference without replacing motion reasoning, whereas dense observations encourage short-term visual matching.
- Sensitivity analysis: l_a = 10 balances temporal abstraction and predictability, while smaller chunks approach step-wise imitation and larger chunks increase future uncertainty.
- Latent representation comparison: The Wan 2.1 VAE variant reaches 0.920 average success rate on LIBERO, below the proposed latent motion design at 0.947.
- Benchmark results: CoWVLA achieves average success lengths of 4.473 on ABCD→D and 4.211 on ABC→D in CALVIN, outperforming the reproduced UniVLA under matched training settings.
- Benchmark results: CoWVLA achieves 0.609 average success rate on SimplerEnv-Google Robot and surpasses UniVLA on all four tasks.
- Latent-motion analysis: Cross-reconstruction highlights robot-arm motion regions, while clustered trajectories exhibit high-level motion semantics such as reversals and multi-phase movements.
- Future-frame visualization: Compared with pixel-oriented world models, motion latents provide stronger guidance for future-frame prediction and better alignment with task instructions.
3. Real-Robot Experiments
The real-robot study uses a Realman RM75B with an Intel RealSense camera and a cup-grasping dataset. Despite lighting differences between collection and deployment, the model correctly executed the task.
- Setup: The experiment uses a 7-DoF Realman RM75B robot with a single gripper and an Intel RealSense RGB camera.
- Dataset: The cup-grasping dataset contains 127 episodes and 65,382 frames, averaging approximately 515 frames per episode.
- Dataset: The data cover four cup colors: red, blue, yellow, and purple.
- Deployment: The model correctly executed the task despite lighting differences between data collection and real-world deployment.