Source-linked AI summary
RynnVLA-002: A Unified Vision-Language-Action and World Model
Jun Cen, Siteng Huang, Yuqian Yuan, Kehan Li, Hangjie Yuan, Chaohui Yu, Bohan Hou, Yuming Jiang, Jiayan Guo, Xin Li, Hao Luo, Fan Wang, Deli Zhao, Hao Chen
TL;DR
VLA models lack explicit action dynamics and foresight, while world models cannot directly produce actions. RynnVLA-002 unifies both capabilities through shared multimodal modeling, outperforming standalone models and achieving 97.4% LIBERO success without robot-data pretraining; its real-world evaluation remains limited in scope.
Problem
VLA models lack explicit action dynamics, foresight, and physics understanding, while world models cannot directly generate action outputs for explicit planning.
Method
RynnVLA-002 jointly models action prediction and action-conditioned visual prediction using shared multimodal representations, attention masking, and a continuous Action Transformer head.
Results
RynnVLA-002 outperforms standalone VLA and world models, achieving 97.4% success rate on LIBERO without robot-data pretraining.
Takeaways & Limitations
The results support mutual enhancement between VLA and world-model capabilities within a unified framework.
Takeaways & Limitations
Real-world evaluation currently covers SO100 pick-and-place manipulation and does not yet cover broader platforms, long-horizon mobile manipulation, or deformable-object tasks.
Abstract
from arXiv · showhide
We introduce RynnVLA-002, a unified Vision-Language-Action (VLA) and world model. The world model leverages action and visual inputs to predict future image states, learning the underlying physics of the environment to refine action generation. Conversely, the VLA model produces subsequent actions from image observations, enhancing visual understanding and supporting the world model's image generation. The unified framework of RynnVLA-002 enables joint learning of environmental dynamics and action planning. Our experiments show that RynnVLA-002 surpasses individual VLA and world models, demonstrating their mutual enhancement. We evaluate RynnVLA-002 in both simulation and real-world robot tasks. RynnVLA-002 achieves 97.4% success rate on the LIBERO simulation benchmark without pretraining, while in real-world LeRobot experiments, its integrated world model boosts the overall success rate by 50%.
1 Introduction
RynnVLA-002 unifies VLA action prediction with action-conditioned visual prediction to jointly model action planning and environmental dynamics. It addresses limitations of standalone VLA and world models through shared multimodal modeling and complementary enhancements.
- Motivation: World models add action-aware internal states, imagination, and physics-informed representations that standard VLA architectures lack.Standard VLA models cannot explicitly represent action dynamics, forecast world evolution under candidate actions, or model physical interactions.
- Mutual enhancement: The VLA component improves visual understanding, while the world model improves action decision making through environmental physics learning.The two capabilities share a modeling space and improve each other inside the unified framework.
- Action generation: RynnVLA-002 explores discrete and continuous action generation, adding an Action Transformer head after discrete generation showed limited real-world generalization and slow inference.The hybrid design retains discrete joint modeling while incorporating continuous action generation for real-world control.
- Contribution: RynnVLA-002 unifies VLA and world modeling for image and action understanding and generation.The framework uses shared multimodal modeling rather than treating the world model only as an external reward model, simulator, or pretraining signal.
- Action generation: An action attention masking strategy addresses error accumulation during autoregressive discrete action-chunk generation.The contribution targets dependencies among sequentially generated actions within a chunk.
- Results: 97.4% success rate is achieved on LIBERO without robot-data pretraining, while real-world LeRobot experiments show a 50% overall success-rate boost from the integrated world model.The model also achieves 93.3% with discrete actions and 97.4% with continuous actions in the reported benchmark results.
2 Related Work
Related work includes VLM-based policies, visual-generation-based action learning, and systems that use world models to assist VLA training. RynnVLA-002 combines these directions through a shared model for action prediction and action-conditioned visual prediction.
- VLM-based VLA: VLM-based VLA models map visual-language inputs to actions, often using discrete action tokens and later methods targeting generalization or representation efficiency.RT-2 co-trained VLMs on robotic trajectories and web-scale vision-language data, while subsequent work extended the architecture.
- Visual Generation-based VLA: Visual-generation-based VLA methods predict future visual states or jointly generate future frames and actions to guide policy learning.These approaches model dynamics through future visual observations and can improve temporal consistency or provide pretraining objectives.
- Positioning: RynnVLA-002 is built on Chameleon to combine image understanding and generation within an action world model.Its shared backbone supports action prediction and action-conditioned visual prediction, with a continuous Action Transformer for real-world control.
- World-model-assisted VLA: World-model-assisted VLA systems use predicted rollouts, rewards, or refinement environments to optimize policy models after training.Examples include world models serving as reward models, virtual simulators, or closed-loop refinement environments.
- World Models: World models provide internal representations and predictive dynamics that support physics-consistent interaction in dynamic environments.Recent implementations include transformer-based world-model architectures.
3 Methods
RynnVLA-002 unifies VLA action prediction and world-model future-image prediction through shared parameters, tokenization, and jointly trained data. Its masking strategy addresses discrete action error accumulation, while a continuous Action Transformer supports smoother real-world control.
- 3.1 Overview: The VLA policy generates actions from language goals, proprioceptive state, and observation history, while the world model predicts the next observation from past observations and actions.The model performs only one queried task per inference call; world-model prediction is not rolled out inside the real-time action-generation loop.
- 3.1 Overview: RynnVLA-002 shares parameters between VLA action prediction and world-model future-image prediction, enabling either capability through user queries.VLA data and world-model data are mixed for training, with a 1:1 sampling ratio unless otherwise specified.
- 3.2 Data Tokenization: Four tokenizers encode images, text, states, and actions, placing all modalities in a shared vocabulary within one language-model architecture.The image and text tokenizers are inherited from Chameleon, while state and action dimensions are discretized into 256 bins.
- 3.2 Data Tokenization: The training objective combines discrete action-token and image-token cross-entropy losses, allowing the model to learn action prediction and world modeling together.The two task types are trained in a single finetuning stage without additional large-scale robot manipulation pretraining.
- 3.3 Action Generation: The proposed action attention mask lets multiple discrete actions be generated in isolation, reducing error accumulation from autoregressive action dependence.The world-model component retains the conventional attention mask, while excessively long action chunks can still constrain timely policy adaptation.
- 3.3 Action Generation: A continuous Action Transformer processes language, image, and state context with learnable action queries to generate complete action chunks in parallel.Its compact architecture is intended to reduce overfitting on limited real-world data and produce fluid, stable trajectories.
4 Experiments
Experiments evaluate RynnVLA-002 on LIBERO simulation and real-world SO100 manipulation, showing strong performance and mutual benefits between its VLA and world-model components. Ablations further examine action generation, auxiliary inputs, efficiency, and the current real-world scope.
- LIBERO Simulation: 97.4% success rate is achieved with continuous actions on LIBERO, compared with 93.3% using discrete actions, without robot-data pretraining.RynnVLA-002 remains competitive with pretrained policies and outperforms most non-pretrained methods.
- Real-World Robot Results: 90% success is reached on multi-target block placement and 80% with distractors, giving RynnVLA-002 the best overall performance across six real-world settings.The multi-target result improves over the strongest pretrained baseline by 20%, while distractor robustness matches it.
- Ablation Study: World-model training raises continuous-action LIBERO success from 91.6% to 94.6% and real-world performance from below 30% to over 80%.Controlled comparisons under the same backbone, data, and training recipe isolate the contribution of the world-modeling objective.
- Ablation Study: World-model supervision improves target-oriented behavior, including retrying grasp attempts after failures, because moving objects receive stronger prediction-loss supervision.Without world-model data, the policy may move toward the placement region without grasping the target or recovering after an unsuccessful attempt.
- Ablation Study: The action attention mask improves discrete action-chunk performance, particularly for longer chunks, by reducing autoregressive error accumulation.Each action is generated independently from visual input, while excessively long chunks still limit timely policy adaptation.
- Efficiency and Scope: Continuous action generation converges faster and is substantially faster at inference, while additional camera or historical-frame inputs improve performance but reduce speed.The real-world evaluation remains limited to SO100 pick-and-place tasks, and training one real-world task takes roughly four days.
5 Conclusion
RynnVLA-002 integrates VLA and world-model capabilities in a unified framework, with each enhancing the other. The work presents this as a methodology for synergistic embodied-AI modeling.
- RynnVLA-002 integrates the VLA and world model within one unified framework.
- The framework demonstrates mutual enhancement between action prediction and world modeling.
- The contribution offers a concrete methodology for synergistic interplay between VLA and world-model capabilities.