Source-linked AI summary
Causal World Modeling for Robot Control
Lin Li, Qihang Zhang, Yiming Luo, Shuai Yang, Ruilin Wang, Fei Han, Mingrui Yu, Zelin Gao, Nan Xue, Xing Zhu, Yujun Shen, Yinghao Xu
TL;DR
Robot policies often entangle scene understanding, dynamics, and control, while existing world models may lack feedback and persistent memory for precise closed-loop manipulation. LingBot-VA addresses this with an autoregressive diffusion model that interleaves video and action tokens, integrates real observations, and overlaps prediction with execution. It achieves strong simulation and real-world results, including 92.0% on RoboTwin 2.0, 98.5% on LIBERO, and over 20% improvement on challenging tasks over π0.5 using 50 demonstrations for adaptation.
Problem
Existing robot policies entangle visual understanding, physical dynamics, and motor control, while prior world models can lack real-time feedback and persistent memory for closed-loop manipulation.
Method
LingBot-VA is an autoregressive diffusion world model that interleaves video and action tokens in a Mixture-of-Transformers architecture, using closed-loop observations and asynchronous inference.
Results
92.0% on RoboTwin 2.0, 98.5% on LIBERO, and over 20% improvement on challenging tasks compared to π0.5 with only 50 demonstrations for adaptation.
Takeaways & Limitations
The results support autoregressive video-action world modeling as a foundation for generalizable robotic manipulation across simulation and real-world deployment.
Takeaways & Limitations
Future work identifies computational overhead from video modeling and the need for additional tactile, force, and audio inputs for complex contact dynamics.
Abstract
from arXiv · showhide
This work highlights that video world modeling, alongside vision-language pre-training, establishes a fresh and independent foundation for robot learning. Intuitively, video world models provide the ability to imagine the near future by understanding the causality between actions and visual dynamics. Inspired by this, we introduce LingBot-VA, an autoregressive diffusion framework that learns frame prediction and policy execution simultaneously. Our model features three carefully crafted designs: (1) a shared latent space, integrating vision and action tokens, driven by a Mixture-of-Transformers (MoT) architecture, (2) a closed-loop rollout mechanism, allowing for ongoing acquisition of environmental feedback with ground-truth observations, (3) an asynchronous inference pipeline, parallelizing action prediction and motor execution to support efficient control. We evaluate our model on both simulation benchmarks and real-world scenarios, where it shows significant promise in long-horizon manipulation, data efficiency in post-training, and strong generalizability to novel configurations. The code and model are made publicly available to facilitate the community.
1 Introduction
LingBot-VA addresses limitations of feedforward and open-loop robot policies by unifying video dynamics prediction with action inference in an autoregressive world model. Its design combines shared video-action processing, closed-loop feedback, persistent context, and asynchronous execution, supporting strong performance in challenging manipulation settings.
- Motivation: Existing VLA policies entangle visual understanding, physical dynamics, and motor control under a single supervision signal, limiting sample efficiency and generalization.This coupling makes it difficult to disentangle visual reasoning from action prediction without explicit dynamics modeling.
- Motivation: Prior chunk-based and open-loop world models struggle with real-time feedback and persistent long-term memory, causing difficulty adapting to disturbances and temporal drift.Bidirectional attention within independently generated chunks also conflicts with causal closed-loop control.
- Approach: LingBot-VA unifies visual dynamics prediction and action inference in one autoregressive diffusion framework while preserving their conceptual distinction.Video and action representations are generated in a continuous latent space through iterative denoising.
- Approach: A dual-stream Mixture-of-Transformers architecture interleaves video and action tokens, using shared attention and asymmetric capacity for efficient control.The design includes partial denoising and asynchronous coordination between prediction and execution.
- Approach: Closed-loop rollouts incorporate current real-world observations, while KV-cache context preserves interleaved trajectories and causal masking maintains temporal consistency.These mechanisms allow the model to recalibrate predicted futures and motor commands using newly acquired environmental feedback.
- Results: Extensive simulation and real-world experiments show state-of-the-art performance, especially on long-horizon and high-precision manipulation, with improved sample efficiency and generalization.The evaluation covers long-horizon, deformable-object, and precision-manipulation tasks across simulation and real-world environments.
2 Preliminary
Flow matching learns a continuous-time vector field that transforms noise into data, and generates samples by integrating the learned flow. In conditional video generation, this process operates in a pretrained video autoencoder’s latent space and can be incorporated into LingBot-VA’s unified video-action modeling.
- 2.1 Flow Matching: Flow matching learns a time-dependent vector field that continuously transforms a simple source distribution, such as Gaussian noise, into a target data distribution.The flow evolves samples along a continuous-time trajectory from noise to data.
- 2.1 Flow Matching: At inference, generated samples are obtained by solving the learned ordinary differential equation from flow time s = 0 to s = 1.The flow time parameter satisfies s ∈ [0, 1].
- 2.1 Flow Matching: The model is trained to predict the true velocity along an interpolation path between a noise sample and a data sample.For the stated linear interpolation, the velocity is x1 − ϵ.
- 2.2 Video Generation with Conditional Flow Matching: Conditional video flow matching generates latent video sequences from noise using a conditioning signal such as text or an initial image.The resulting latent video is decoded to pixel space after integrating the learned vector field.
- 2.2 Video Generation with Conditional Flow Matching: LingBot-VA interleaves video and action tokens in a dual-stream Mixture-of-Transformers sequence, predicting future visual latents before decoding corresponding actions.The action stream uses inverse dynamics conditioned on predicted visual transitions.
3 Method
LingBot-VA frames robot manipulation as autoregressive video-action world modeling, predicting visual evolution and decoding actions while retaining causal context. Its deployment combines partial denoising, asynchronous execution, and feedback-grounded prediction for closed-loop control.
- Problem Statement & Approach Overview: Existing VLA policies directly map observations to actions, coupling scene understanding, physical dynamics, and motor control under one supervision signal.This coupling is associated with poor sample efficiency and limited generalization without explicit dynamics modeling.
- Problem Statement & Approach Overview: LingBot-VA predicts future visual observations before inferring actions, separating visual-dynamics learning from action grounding through an inverse dynamics model.The decomposition allows visual prediction to use video data while action decoding remains grounded in robot demonstrations.
- Autoregressive Video-Action World Modeling: The model interleaves latent video and action tokens in one autoregressive process, jointly modeling visual dynamics and robot actions across causal temporal context.A causal video VAE compresses observations into latent tokens, while action vectors are projected into compatible token embeddings.
- Autoregressive Video-Action World Modeling: Chunk-wise autoregressive prediction generates K future video frames with causal structure across chunks and parallel attention within each chunk.This balances generation efficiency with the ability to incorporate feedback for closed-loop correction.
- Unified Architecture & Training: Partial denoising reduces video-generation steps by half while maintaining action prediction quality.Noisy-history training teaches the action decoder to extract action-relevant information from partially noisy video states.
- Real-time Deployment & Asynchronous Inference: Asynchronous inference overlaps action prediction with motor execution, while a forward dynamics pass refreshes stale forecasts using recent real-world observations.The feedback-grounded step re-aligns prediction before the next forecast, strengthening closed-loop reaction to environmental changes.
4 Experiments
The training data combines multiple public and internal robot-manipulation sources spanning embodiments, environments, and task categories. A unified action representation supports cross-embodiment training, while per-dataset validation monitors training dynamics.
- Dataset Curation and Preprocessing: Each dataset is preprocessed for consistent data format and annotation quality, then split into 90% training and 10% validation.The validation split is used to monitor training dynamics.
- Dataset Curation and Preprocessing: The universal action interface represents each dual-arm system with 30 dimensions covering EEF pose, joint angles, and gripper actions.Each arm contributes 7 EEF dimensions, 7 joint dimensions, and 1 gripper dimension; missing joint dimensions are zero-padded.
- Dataset Curation and Preprocessing: The corpus aggregates six data sources covering diverse embodiments, environments, and task categories.The listed sources include Agibot, RoboMind, InternData-A1, OXE, UMI Data, and RoboCOIN.
- Dataset Curation and Preprocessing: Approximately 16K hours of robot manipulation data comprise the training corpus, including internally collected demonstrations.The corpus spans diverse tasks and environments.
4.2 Implementation & Training Details
LingBot-VA combines a large video backbone with a smaller action stream connected through MoT, and uses diffusion-specific inference and training settings. Post-training can adapt the model to novel robot platforms with as few as 50 demonstrations.
- Architecture: The video stream uses Wan2.2-5B, while the action stream shares its depth but uses a 4× smaller hidden dimension, adding approximately 350M parameters to a 5.3B-parameter model.Both streams use RoPE positional encoding and connect through the MoT architecture.
- Inference and training: Inference uses three Euler steps for video tokens and ten for action tokens, with classifier-free guidance scales of 5.0 and 1.0, respectively.Training applies noise augmentation with probability p = 0.5 and samples saug from Uniform[0.5, 1.0].
- Pretraining: The model is pretrained on 1.4T tokens using AdamW, cosine annealing with linear warmup, bfloat16 mixed precision, and gradient clipping at 2.0.The inverse-dynamics loss weight λ is 1, and data sources are sampled uniformly.
- Post-training: 50 demonstrations are sufficient for effective deployment on novel robot platforms after post-training.A learning rate of 1 × 10^-5 for 3K steps yields robust performance; 1 × 10^-4 for 1K steps is faster but slightly inferior.
4.3 Main Results
LingBot-VA performs strongly in real-world manipulation and simulation, spanning long-horizon, precision, deformable-object, bimanual, and multi-suite benchmark settings. Its reported results include state-of-the-art performance across six real-world tasks and high average success rates on RoboTwin and LIBERO.
- Real-world deployment: The six real-world tasks consist of manipulation-primitives sequences whose scoring criteria are detailed in Tables S2 through S4.Figure 6 presents the detailed task progressions and key execution steps.
- Real-world deployment: LingBot-VA achieves state-of-the-art performance across all six real-world tasks and both success-rate and progress-score metrics, substantially outperforming π0.5.The tasks cover long-horizon, precision, and deformable-object manipulation.
- Simulation evaluation: 92.9% Easy and 91.6% Hard average success rates are reported on RoboTwin 2.0, with gains of +8.2% and +9.1% at Horizon = 3 over the second-best approach.Hard uses varied object poses and scene layouts, while Easy uses fixed initial configurations.
- Simulation evaluation: 98.5% average success rate is reported on LIBERO, including 98.5% on LIBERO-Long.The benchmark covers Spatial, Object, Goal, and Long task suites.
4.4 Ablation
Ablations show that asynchronous inference preserves task performance while reducing completion time, and that joint video-action pretraining and curated action initialization improve adaptation and training dynamics.
- Deployment mode: Asynchronous generation achieves comparable success rates to synchronous generation while completing tasks 2× faster.It predicts future video and action sequences while executing current actions.
- Pretraining: LingBot-VA reaches 92.10% Easy and 91.12% Hard average success rates, substantially outperforming WAN fine-tuning under identical post-training procedures.Both models use 50 task-specific demonstrations, a learning rate of 1 × 10^-5, and 3K steps.
- Pretraining: Joint video-action pretraining provides visual-motor priors that facilitate fast adaptation to complex bimanual manipulation tasks.The ablation compares the pretrained LingBot-VA model with WAN using the same RoboTwin post-training setup.
- Initialization: Random action-stream initialization causes volatile dynamics and slower convergence because action-token outputs initially diverge from the video distribution.Curated initialization interpolates pretrained video weights with scaling factor α = dv/da, producing smooth convergence and lower loss.
4.5 Analysis
Analysis experiments examine data efficiency, temporal memory, and generalization. LingBot-VA outperforms π0.5 with limited data and on memory tasks, while generalizing to novel objects and out-of-distribution positions.
- Data efficiency: 15.6% higher progress score on Make Breakfast and 10.3% higher on RoboTwin 2.0 Easy are achieved with 10 demonstrations versus π0.5.LingBot-VA consistently outperforms π0.5 across all evaluated data regimes and both tasks.
- Data efficiency: The authors attribute data efficiency to jointly pretrained video representations that encode visual priors about physical dynamics and object interactions.These priors serve as implicit regularization during post-training and reduce the data needed for adapting to new tasks.
- Temporal memory: LingBot-VA substantially outperforms π0.5 on Wipe Plate and Search Box, which require counting repeated actions and retaining search history.The authors attribute this to autoregressive prediction and KV-cache preservation of historical information.
- Generalization: The model shows stronger generalization to novel objects with varying shapes and textures and to out-of-distribution object positions.The evaluation compares novel-object and spatial generalization after training on restricted object and position distributions.
5 Related Work
Related work covers VLA policies and robotic world models, while LingBot-VA combines autoregressive video prediction with action decoding under causal temporal structure.
- Vision-Language-Action Policies: VLA policies use web-scale knowledge and robot demonstrations to generalize across manipulation tasks without hand-crafted rules or restricted action abstractions.
- Vision-Language-Action Policies: LingBot-VA unifies autoregressive video prediction with action decoding under a strict causal temporal structure.Each prediction conditions exclusively on past observations and actions.
- World Models for Robotic Control: World-model approaches for robotic control are categorized by their state representations, including latent-space methods.
6 Conclusion
LingBot-VA unifies video dynamics prediction and action inference for robotic manipulation and integrates real-world observations for closed-loop control. The reported evaluation shows strong simulation and real-world performance, while future work targets computational efficiency and richer sensory inputs.
- LingBot-VA unifies video dynamics prediction and action inference for robotic manipulation through an autoregressive diffusion framework.
- 92.0% on RoboTwin 2.0 and 98.5% on LIBERO, with over 20% improvement on challenging real-world tasks compared to π0.5 using 50 demonstrations for adaptation.
- Future work includes more efficient video compression and multimodal sensory inputs such as tactile, force, and audio signals.These directions target reduced computational overhead and more robust manipulation in tasks with complex contact dynamics.
A Real-world Evaluation Details
The real-world evaluation compares LingBot-VA with π0.5 across 20 alternating trials per method, using step-level progress and all-steps success criteria across six manipulation tasks.
- Each real-world task uses 20 trials for LingBot-VA and π0.5 under an alternating evaluation protocol.Trials alternate between π0.5 and the proposed method to support fair comparison.
- Intermediate steps receive scores of 1, 0.5, or 0 for first-attempt success, retry success, or failure, while trial success requires completing every step.
- Progress Score averages trial scores relative to the maximum possible score, expressed as a percentage.
- Success Rate is the percentage of trials in which all required steps are completed successfully.
- The evaluation covers Make Breakfast, Pick Screws, Fold Clothes, Unpack Delivery, Insert Tubes, and Fold Pants.The tasks contain between 2 categories and 10 sequential steps, depending on the task.
- RoboTwin 2.0 evaluation compares Easy fixed configurations with Hard randomized object poses and scene layouts in a 50-task bimanual benchmark.