Source-linked AI summary
Do World Action Models Generalize Better than VLAs? A Robustness Study
Zhanguang Zhang, Zhiyuan Li, Behnam Rahmati, Rui Heng Yang, Yintao Ma, Amir Rasouli, Sajjad Pakdamansavoji, Yangzheng Wu, Lingfeng Zhang, Tongtong Cao, Feng Wen, Xinyu Wang, Xingyue Quan, Yingxue Zhang
TL;DR
Robot policies must predict how environments evolve, yet VLAs show limited generalization and robustness under contextual perturbations. The paper compares VLAs and WAMs across LIBERO-Plus and RoboTwin 2.0-Plus, finding strong WAM robustness but persistent inference-speed and geometric-perturbation challenges.
Problem
Robot action planning requires predicting environment changes, while VLAs remain limited in generalization and vulnerable to contextual perturbations.
Method
The paper comparatively evaluates state-of-the-art VLAs and WAMs under visual and language perturbations on LIBERO-Plus and RoboTwin 2.0-Plus.
Results
WAMs generally show strong robustness across both single-arm and bimanual settings, while VLAs can be competitive or superior on some tasks.
Takeaways & Limitations
Spatiotemporal video priors support WAM robustness, but VLAs can match it with diverse data or explicit dynamic-prediction objectives.
Takeaways & Limitations
WAM inference is at least 4.8 times slower than π0.5, limiting deployment in real-world robotic systems requiring rapid responses.
Abstract
from arXiv · showhide
Robot action planning in the real world is challenging as it requires not only understanding the current state of the environment but also predicting how it will evolve in response to actions. Vision-language-action (VLA), which repurpose large-scale vision-language models for robot action generation using action experts, have achieved notable success across a variety of robotic tasks. Nevertheless, their performance remains constrained by the scope of their training data, exhibiting limited generalization to unseen scenarios and vulnerability to diverse contextual perturbations. More recently, world models have been revisited as an alternative to VLAs. These models, referred to as world action models (WAMs), are built upon world models that are trained on large corpora of video data to predict future states. With minor adaptations, their latent representation can be decoded into robot actions. It has been suggested that their explicit dynamic prediction capacity, combined with spatiotemporal priors acquired from web-scale video pretraining, enables WAMs to generalize more effectively than VLAs. In this paper, we conduct a comparative study of prominent state-of-the-art VLA policies and recently released WAMs. We evaluate their performance on the LIBERO-Plus and RoboTwin 2.0-Plus benchmarks under various visual and language perturbations. Our results show that WAMs achieve strong robustness, with LingBot-VA reaching 74.2% success rate on RoboTwin 2.0-Plus and Cosmos-Policy achieving 82.2% on LIBERO-Plus. While VLAs such as $π_{0.5}$ can achieve comparable robustness on certain tasks, they typically require extensive training with diverse robotic datasets and varied learning objectives. The evaluation code for the RoboTwin2.0-Plus benchmark is available at: https://robot-robustness.github.io/RoboTwin2.0-Plus/.
1 Introduction
Robot action planning must handle uncertain environments and predict action consequences, while VLAs remain vulnerable to limited generalization and contextual perturbations. This study compares VLAs and WAMs on two perturbed manipulation benchmarks, finding strong WAM robustness alongside substantial deployment costs.
- Real-world robot planning is difficult because diverse, uncertain settings make action consequences hard to anticipate.
- VLAs achieve strong performance across robotic tasks but show limited generalization and reduced robustness to distractions and clutter.
- The study compares state-of-the-art VLAs and WAMs using LIBERO-Plus and RoboTwin 2.0-Plus under contextual perturbations.
- WAMs generally show strong robustness to noise, lighting, and layout perturbations in both single-arm and bimanual settings.
- WAM robustness is associated with spatiotemporal priors, whereas comparable VLA robustness often requires curated diverse datasets or explicit dynamic-prediction objectives.
- WAM inference remains costly, with each evaluated step at least 4.8 times slower than π0.5, although GigaWorld-Policy and Fast-WAM target this bottleneck.
2 Related Works
Related work develops unified VLA policies, learned world-model simulators, predictive policy augmentations, and WAMs that directly decode world-model representations into actions. The central distinction concerns whether models acquire dynamics implicitly through VLM-based training or explicitly through video-generation backbones and future-state prediction.
- Vision-Language-Action Models: VLA research unifies visual perception, language understanding, and robot control, with scaling efforts emphasizing multimodal pretraining, heterogeneous datasets, and cross-embodiment generalization.
- World Models: World models predict future visual states conditioned on actions and support robotics as simulators, planning modules, or adapted policies.
- World Models as Learned Simulators: Learned simulators enable explicit planning by generating action-conditioned counterfactual futures for test-time decision-making.
- World Models in Policy Learning: Other approaches augment policies with auxiliary future-state or motion prediction objectives, or co-train policies with predictive models.
- Efficient WAMs: GigaWorld-Policy and Fast-WAM reduce WAM inference latency by making visual-state generation optional at test time.
- World Action Models: WAMs use pretrained video-generation backbones for robot action generation, while VLAs typically use VLM backbones pretrained with next-token prediction.
- Differences between WAMs and VLAs: Video backbones learn future visual dynamics from diverse internet-scale videos, allowing WAM embodied pretraining to focus primarily on action prediction.
3 Experiments
The experiments compare VLAs and WAMs on enhanced LIBERO and RoboTwin benchmarks under diverse perturbations, examining robustness, perturbation-specific behavior, explanatory mechanisms, and runtime. WAMs generally show strong robustness, but performance depends on training-data diversity and carries a runtime cost.
- RQ 1. Are WAM-based policies robust to perturbations?: 74.2% overall success rate places LingBot-VA above π0.5 at 58.6% and Fast-WAM at 72.7% on RoboTwin 2.0-Plus.LingBot-VA ranks first in five of seven perturbation categories.
- RQ 3. How to explain the performance differences between VLAs and WAMs?: 72.7% overall success makes Fast-WAM second on RoboTwin 2.0-Plus without embodied pre-training, indicating video-backbone priors can support robustness when task-specific data is diverse.Its released checkpoint uses clean and domain-randomized demonstrations.
- RQ 2. Is the performance advantage of WAMs consistent across different perturbation types?: WAM robustness varies by perturbation: LingBot-VA is strong under light, noise, and layout changes but struggles with camera viewpoint and robot initial-state variations.Cosmos-Policy predictions remain accurate under noise and light changes but show spatial distortions and inconsistent colors for some background perturbations.
- RQ 3. How to explain the performance differences between VLAs and WAMs?: Training-data diversity remains critical: identical Fast-WAM architectures drop about 18 points on RoboTwin but about 46 points on LIBERO when trained without perturbation-diverse demonstrations.The comparison indicates that video spatiotemporal priors are necessary but not sufficient for robustness.
- RQ 4. What are the runtime characteristics of WAMs and how do they compare to those of VLAs?: At least 4.8× higher inference runtime than π0.5 characterizes MOTUS and WAMs, driven largely by future-state diffusion and denoising steps.LingBot-VA reaches 5.2 seconds per inference in the RoboTwin 2.0 configuration with 25 state and 50 action denoising steps.
4 Conclusion
The study finds that WAMs are generally robust to several visual perturbations across both single-arm and bimanual settings, but camera-viewpoint and initial-state changes remain difficult. WAMs also face a substantial inference-speed disadvantage, while Fast-WAM shows competitive robustness but sensitivity to training-data conditions.
- WAMs consistently withstand noise, lighting, and layout perturbations across LIBERO-Plus and RoboTwin 2.0-Plus.This pattern holds in both single-arm and bimanual settings.
- Camera-viewpoint and robot initial-state perturbations remain challenging for WAMs.The conclusion attributes this boundary to limited benefit from video priors when scene geometry changes.
- π0.5 achieves competitive or superior robustness by incorporating diverse robotic and web data during training.The comparison suggests data diversity can compensate for the absence of explicit world modeling.
- WAM inference is at least 4.8 times slower per inference than π0.5, primarily because of visual state denoising.LingBot-VA uses up to 25 denoising steps on RoboTwin, while GE-Act uses one.
- Fast-WAM reaches 72.7% overall robustness on RoboTwin 2.0-Plus but falls to 51.5% on LIBERO with clean training demonstrations.Its LIBERO result contrasts with 97.6% on the original benchmark, indicating sensitivity to the training setting.
A RoboTwin 2.0-Plus Benchmark
RoboTwin 2.0-Plus extends the RoboTwin benchmark with controlled LIBERO-Plus-style perturbations to evaluate world-action-model robustness.
- RoboTwin 2.0-Plus covers 7 perturbation dimensions and 21 sub-dimensions across all 50 RoboTwin 2.0 dual-arm tasks.
A.1 Perturbation Taxonomy
The benchmark’s 21 perturbation sub-dimensions are individually configurable, supporting combined and ablation-style evaluations.
- Each of the 21 perturbation sub-dimensions can be configured independently through YAML flags.The configuration supports both combined evaluations and ablation-style evaluations.
A.2 Evaluation Protocol
Each RoboTwin 2.0-Plus task uses one clean baseline and seven isolated perturbation branches, with 50 episodes per configuration.
- A full evaluation requires 8 configurations per task: 1 clean baseline and 7 perturbation branches.Each branch activates one perturbation dimension while all other dimensions remain at clean defaults.
- Each configuration runs 50 episodes per task.
A.3 Sub-dimension Details
The benchmark applies structured perturbations across sensing, lighting, viewpoint, robot state, background, object layout, and language. These dimensions are organized into 21 sub-dimensions across seven LIBERO-Plus dimensions.
- Sensor Noise: Five sensor-noise types are cycled deterministically by episode, with severity sampled uniformly from [2, 3].The types are motion, Gaussian, zoom, fog, and glass blur.
- Lighting: Lighting perturbations vary RGB tint, directional-light orientation, shadows, specular strength, and shininess, with some effects jointly activated.Dramatic side-lighting has a 35% chance, while shadow casting is toggled with 50% probability per episode.
- Camera Viewpoints: Camera perturbations modify viewing distance, spherical position, and orientation, while spherical-position changes are disabled by default unless enabled for ablation.Orientation perturbations independently vary yaw, pitch, and roll within 0° to 5° with random sign.
- Robot Initial States: Robot initial states receive joint-angle noise with std = 0.1 rad, clipped to ±0.225 rad, and gripper extremes occur with probability 0.25.The perturbations affect both arms and set each gripper to 0.05 or 0.95.
- Background, Objects Layout, and Language Instructions: Background, object layout, and language are randomized through material and color changes, distractor counts from [3, 15], target-pose perturbations, and 2,500 instruction variants.The language variants include distraction, common-sense rewording, and reasoning-chain forms sampled at approximately 30%, 50%, and 20%.
- Taxonomy: The perturbation taxonomy covers all 21 sub-dimensions across seven LIBERO-Plus dimensions.Implementation-level parameter details are provided for the RoboTwin 2.0-Plus taxonomy.
B Benchmark comparison
The evaluation uses eight required configurations for complete robustness testing and compares the LIBERO-Plus and RoboTwin 2.0-Plus benchmarks.
- Evaluation Configurations: Eight required configurations are specified for a complete RoboTwin-Plus robustness evaluation.The configurations are presented in the benchmark evaluation setup.
- Benchmark Comparison: LIBERO-Plus and RoboTwin 2.0-Plus are compared as benchmark settings.The supplied benchmark comparison identifies the two evaluation suites without reporting additional comparison metrics.
C Comparison of Evaluated Models
The study compares VLA and WAM policies by their action representations, dimensionality, chunk sizes, and control frequencies. The evaluated models use different action-generation designs across LIBERO and RoboTwin settings.
- Model Setup: The model comparison records action representation, dimensionality, chunk size, and control frequency for each evaluated policy.Unless noted, LIBERO models use the native axis-angle rotation convention.
- VLA Models: Pi0 and Pi0-FAST use delta end-effector control with axis-angle rotation and generate 50-step action chunks.Pi0 uses flow matching, whereas Pi0-FAST uses FAST tokenization.
- VLA Models: Pi0.5 predicts absolute target poses with 50-step chunks at 50 Hz and supports end-effector or joint-space control depending on the setup.This extends the Pi0 model family with absolute target-pose prediction.
- Training Setup: The Fast-WAM LIBERO checkpoint was trained only on clean demonstrations, whereas its RoboTwin checkpoint used clean and domain-randomized demonstrations.The distinction is reported in the detailed LIBERO-Plus results table.
- VLA Models: xVLA uses a unified absolute end-effector representation with 6D rotation, while VLA-JEPA predicts delta end-effector actions with axis-angle rotation.xVLA outputs 10 dimensions per arm, padded to 20 dimensions for dual-arm control, with chunk size 32.
- WAM Models: Cosmos-Policy encodes actions as latent frames in a video-diffusion process, while GE-Act predicts absolute end-effector poses and MOTUS operates in a learned latent action space.Cosmos-Policy uses the native 7-dimensional LIBERO action space with chunk size 16; GE-Act generates 54-step trajectories at 30 Hz.