Source-linked AI summary
WorldVLA: Towards Autoregressive Action World Model
Jun Cen, Chaohui Yu, Hangjie Yuan, Yuming Jiang, Siteng Huang, Jiayan Guo, Xin Li, Yibing Song, Hao Luo, Fan Wang, Deli Zhao, Hao Chen
TL;DR
WorldVLA addresses the challenge of jointly predicting robot actions and future environmental states while unifying image and action understanding and generation. It integrates an action model with an action-conditioned world model and introduces attention masking for autoregressive action chunks. The resulting unified framework outperforms standalone action and world models, while masking improves action chunk generation.
Problem
Existing action and world models provide separate capabilities, while autoregressive action chunk generation suffers from error propagation caused by limited action generalization.
Method
WorldVLA integrates an action model and an action-conditioned world model using shared multimodal tokens, and masks prior actions during current-action generation.
Results
WorldVLA outperforms standalone action and world models, and attention masking improves performance in autoregressive action chunk generation.
Takeaways & Limitations
Mutual enhancement between world modeling and action modeling supports a unified framework for image and action comprehension and generation.
Abstract
from arXiv · showhide
We present WorldVLA, an autoregressive action world model that unifies action and image understanding and generation. Our WorldVLA intergrates Vision-Language-Action (VLA) model and world model in one single framework. The world model predicts future images by leveraging both action and image understanding, with the purpose of learning the underlying physics of the environment to improve action generation. Meanwhile, the action model generates the subsequent actions based on image observations, aiding in visual understanding and in turn helps visual generation of the world model. We demonstrate that WorldVLA outperforms standalone action and world models, highlighting the mutual enhancement between the world model and the action model. In addition, we find that the performance of the action model deteriorates when generating sequences of actions in an autoregressive manner. This phenomenon can be attributed to the model's limited generalization capability for action prediction, leading to the propagation of errors from earlier actions to subsequent ones. To address this issue, we propose an attention mask strategy that selectively masks prior actions during the generation of the current action, which shows significant performance improvement in the action chunk generation task.
1 Introduction
WorldVLA unifies action and image understanding and generation in one autoregressive action world model. It combines complementary world- and action-model capabilities while addressing error accumulation in autoregressive action chunk generation.
- WorldVLA: WorldVLA unifies action and image understanding and generation within a single autoregressive action world model.It uses shared multimodal tokens across image, text, and action modalities within one LLM architecture.
- Mutual enhancement: The world model predicts visual representations from input actions to capture environmental physics, while the action model improves visual interpretation for image generation.This bidirectional interaction is intended to strengthen both action and image capabilities.
- Action chunk generation: Autoregressively generating multiple actions can cause performance drops because limited action generalization allows earlier errors to propagate to later actions.The paper attributes this limitation partly to multimodal pretraining dominated by images and text rather than actions.
- Action chunk generation: The proposed action attention masking strategy selectively masks prior actions during action chunk generation to address action error accumulation.The strategy is introduced specifically for autoregressive models generating multiple actions in sequence.
- Results: 4% grasping success rate improvement over an action model with the same backbone was achieved on LIBERO, while FVD decreased by 10% versus a vanilla world model.The reported comparisons support benefits from integrating world and action models.
- Results: With conventional autoregressive action chunking, grasping success decreased by 10% to 50%; attention masking improved it by 4% to 23%.These results quantify both the degradation addressed and the improvement obtained by the proposed strategy.
2 Related Works
Related work spans vision-language-action models, video generation, and unified multimodal understanding and generation. WorldVLA is presented within this landscape as a model combining action prediction with action-conditioned world-state forecasting.
- Vision-Language-Action Model: Vision-language-action models augment pretrained multimodal language models with discrete action decoders or continuous diffusion policy heads.Their internet-scale prior knowledge supports generalization to unseen scenarios and tasks.
- Video Generation: Video generation has been used in robotics for visual imagination and planning, including generating future video before corresponding actions.Large-scale video data can support pretraining of future-video generation components.
- Unified Understanding and Generation Model: Recent multimodal language-model research increasingly seeks to unify visual understanding and visual generation within a single framework.Most earlier multimodal language models primarily generate text from image and language inputs.
- WorldVLA: WorldVLA integrates an action model conditioned on textual and visual data with a world model that predicts the next environmental state from text, image, and action.The two components are described as distinct but complementary.
3 Methods
WorldVLA unifies action prediction and future-state forecasting in one autoregressive model, using shared representations and multimodal tokenization. It combines action and world-model data while masking prior actions to improve autoregressive action-chunk generation.
- Problem formulation: The action model generates actions conditioned on historical image observations and a language instruction.Its input contains image history and task language, formalizing policy prediction within the unified model.
- Problem formulation: The world model predicts the next image frame from historical observations and corresponding actions, learning environmental dynamics for action generation.The paper motivates world-model data because predicted outcomes can help evaluate candidate actions and avoid actions leading to undesirable states.
- Problem formulation: WorldVLA jointly performs action prediction and world-state forecasting through an integrated action-world model Mψ.The action component predicts actions from image history and language, while the world component forecasts the next frame from observation and action histories.
- Architecture: Three tokenizers encode images, text, and actions into a shared vocabulary for unified autoregressive understanding and generation.The image tokenizer produces discrete image tokens, the action tokenizer discretizes each action dimension into 256 bins, and actions use 7 tokens.
- Training strategy: WorldVLA mixes action-model and world-model data and balances their cross-entropy losses with L = Laction + αLworld.The coefficient α compensates for the larger number of image tokens relative to action tokens.
- Training strategy: The attention-mask strategy selectively limits prior-action access during action-chunk generation to address autoregressive action error accumulation.The standard causal mask is described as inadequate for generating multiple consecutive actions, motivating the proposed masking design.
4 Experiments
Experiments evaluate WorldVLA on LIBERO and compare its unified action-world modeling against standalone, video-prediction, and ablated alternatives. Results show mutual gains between action and world modeling, while attention masking mitigates autoregressive action-chunk degradation.
- World Model Helps Action Model: Integrating a world model enhances action-model performance by supporting physical-dynamics understanding and prospective simulation of action consequences.The world model predicts subsequent environmental states conditioned on the current state and an action.
- Action Model Helps World Model: Integrating an action model improves world-model generation quality, especially for longer videos, producing more coherent and physically plausible future states than a pure world model.The action model contributes visual interpretation and behavioral-pattern understanding.
- Action Chunking Generation with Proposed Attention Mask: Action success decreases as naive autoregressive action chunks become longer because later actions depend on preceding generated actions and accumulate errors.The paper attributes this partly to limited action generalization in pretrained multimodal language models.
- Action Chunking Generation with Proposed Attention Mask: The proposed attention mask generates each action independently from visual input, improving performance over naive masking particularly for longer chunks.The mechanism is designed to mitigate error propagation within generated action sequences.
- World Model versus Video Prediction Model: Unlike video prediction, the world model is conditioned on actions; across evaluated tasks, this integration improves action-model performance, whereas video prediction helps two tasks and hurts one.The paper associates the difference with ambiguity when action inputs are absent from video prediction.
5 Conclusion and Future Work
The conclusion presents WorldVLA as a unified autoregressive framework for action and visual understanding and generation. It identifies scaling, improved visual tokenization, and an auxiliary action head as future development directions.
- Conclusion: WorldVLA unifies action and visual understanding with generation capabilities in an autoregressive framework.The framework integrates world modeling and action modeling within one architecture.
- Conclusion: The study reports mutual performance enhancement from integrating world modeling and action modeling.The conclusion presents this integration as a central supported finding.
- Future Work: Future work includes scaling data and model size, designing a unified high-quality visual tokenizer, and adding an auxiliary action head.The current discrete image tokenizer is described as limited in perceptual expressiveness.