Source-linked AI summary
Unified Vision-Language-Action Model
Yuqi Wang, Xinghang Li, Wenxuan Wang, Junbo Zhang, Yingyan Li, Yuntao Chen, Xinlong Wang, Zhaoxiang Zhang
TL;DR
Existing VLAs often use static, language-centric mappings that overlook temporal and causal structure in visual observations. UniVLA instead jointly autoregresses discrete vision, language, and action tokens, using world-model learning from videos to improve downstream policy learning; it achieves strong results across simulation benchmarks, including 94.0% on LIBERO long-horizon tasks.
Problem
Existing VLA models often follow static, language-centric pipelines that overlook the temporal and causal structure of real-world interactions and limit video-based training.
Method
UniVLA converts vision, language, and action into discrete tokens in a shared vocabulary and models their interleaved sequences autoregressively, with world-model learning during post-training.
Results
UniVLA achieves clear improvements across CALVIN, LIBERO, and SimplerEnv benchmarks, including improving LIBERO long-horizon performance from 69.0% to 94.0%.
Takeaways & Limitations
The unified design supports flexible multimodal tasks and efficient downstream policy learning, particularly for long-horizon and out-of-distribution scenarios.
Takeaways & Limitations
Post-training scalability remains at an early stage because of limited computational resources, and fuller integration with reinforcement learning requires further research.
Abstract
from arXiv · showhide
Vision-language-action models (VLAs) have garnered significant attention for their potential in advancing robotic manipulation. However, previous approaches predominantly rely on the general comprehension capabilities of vision-language models (VLMs) to generate action signals, often overlooking the rich temporal and causal structure embedded in visual observations. In this paper, we present UniVLA, a unified and native multimodal VLA model that autoregressively models vision, language, and action signals as discrete token sequences. This formulation enables flexible multimodal tasks learning, particularly from large-scale video data. By incorporating world modeling during post-training, UniVLA captures causal dynamics from videos, facilitating effective transfer to downstream policy learning--especially for long-horizon tasks. Our approach sets new state-of-the-art results across several widely used simulation benchmarks, including CALVIN, LIBERO, and Simplenv-Bridge, significantly surpassing previous methods. For example, UniVLA achieves 95.5% average success rate on LIBERO benchmark, surpassing pi0-FAST's 85.5%. We further demonstrate its broad applicability on real-world ALOHA manipulation and autonomous driving.
1 Introduction
UniVLA addresses the limits of language-centric VLA models by jointly modeling discrete vision, language, and action tokens with temporal and causal structure. World-model training on videos supports downstream policy learning, with strong results across simulation benchmarks and broader multimodal applications.
- Existing VLA models commonly derive actions from static, semantically projected visual representations, overlooking temporal and causal structure.
- UniVLA encodes vision, language, and action as discrete tokens in a shared vocabulary and models them through autoregressive sequence learning.
- Interleaved multimodal modeling supports joint learning across modalities and large-scale video-based training.
- World-model post-training learns temporal dynamics from videos and enhances downstream policy learning, data efficiency, and training efficiency, especially for long-horizon and out-of-distribution tasks.
- UniVLA achieves state-of-the-art performance across CALVIN, LIBERO, and SimplerEnv while also supporting spatial reasoning, video prediction, and driving scenarios.
2 Related Works
Prior VLA work follows action-prediction or visual-guided action-prediction paradigms, each with distinct limitations. UniVLA combines video-generation pretraining with VLM-based modeling in a native multimodal framework, while world-model research supplies relevant foundations for robotics and driving.
- VLA models are broadly categorized into pure action prediction and visual-guided action prediction according to their output spaces.
- Pure action prediction: Pure action-prediction methods integrate with VLMs but lack spatial understanding and visual prediction capabilities.
- Visual-guided action prediction: Visual-guided methods predict future visual signals before decoding actions, but typically separate generative and action-prediction models.
- UniVLA unifies video-generation pretraining and VLM strengths in a native multimodal model.
- World models for robotics: World models are used across interactive video generation, autonomous driving, and robotics to capture and reason about physical-world dynamics.
- The framework overview represents modalities as discrete interleaved tokens and trains them with an autoregressive Transformer.
3 Unified Vision-Language-Action Model
UniVLA converts language, vision, and action into discrete tokens in a shared sequence and models them autoregressively across tasks. Its training separates video-based world modeling from action-focused policy fine-tuning while preserving temporal and causal structure.
- UniVLA uses an encoder-free architecture that converts all modalities into discrete tokens and learns them autoregressively.
- The framework shares a representation space across modalities and interleaves their tokens to support sequence modeling across video generation, visual grounding, and action learning.
- Unified Multimodal Model: Language, vision, and action sequences are concatenated into one sequence, with temporal-step markers preserving alignment across modalities.
- Action Modeling: The FAST tokenizer applies the Discrete Cosine Transform to continuous action windows, producing variable-length discrete action-token sequences.
- Training reduces to next-token prediction with cross-entropy loss, while task-specific token selection provides format flexibility.
- Interleaved observations and actions represent task execution as a causal temporal process in which actions alter the environment and affect later observations.
- Training: World-model post-training predicts future visual content from instructions and observed states, while policy fine-tuning computes loss solely from action tokens.
- Training: The two-stage paradigm initializes from a vision-language-aligned checkpoint, learns video dynamics with a world-model objective, then refines task-specific behaviors through action learning.
4 Experiments
Experiments evaluate UniVLA across simulation benchmarks, post-training strategies, efficiency, multimodal capabilities, and autonomous driving. The model achieves strong results across manipulation settings, with world-model post-training improving transfer, efficiency, and long-horizon performance.
- Main Results: UniVLA achieves state-of-the-art performance across CALVIN, LIBERO, and SimplerEnv simulation benchmarks.The evaluations cover long-horizon tasks, diverse generalization, and real-to-sim manipulation transfer.
- Main Results: UniVLA records the highest performance on both CALVIN ABC→D and ABCD→D tasks, demonstrating strong multi-task learning and long-horizon planning.
- Main Results: 94.0% LIBERO long-horizon success improves the previous state of the art from 69.0%.UniVLA achieves the best overall performance across the Spatial, Object, Goal, and Long suites.
- Main Results: 69.8% average success on SimplerEnv improves over the prior 42.7%.Gains are especially marked on stack block, put carrot, and put spoon tasks.
- In-Depth Analysis: World-model post-training provides the most substantial gains, enhancing generalization and long-horizon planning without requiring action annotations.The comparison emphasizes temporal dynamics in video data and supports scalable learning from large-scale videos.
- In-Depth Analysis: Post-training enables higher success rates with only 10% of CALVIN fine-tuning data and faster convergence with fewer iterations.The efficiency gains extend to SimplerEnv and support policy adaptation across diverse robotic setups.
5 Conclusion
UniVLA unifies vision, language, and action modeling through a shared token space and autoregressive learning, while using video-trained world models to improve downstream policy learning. The paper reports strong generalization and efficiency across diverse domains, but identifies scalability and reinforcement-learning integration as open challenges.
- UniVLA bridges heterogeneous modalities through a shared token space and autoregressive modeling, supporting flexible multimodal tasks.The framework jointly models vision, language, and action signals and is designed for cross-modal integration.
- World-model training captures video dynamics and causality, improving downstream policy-learning performance and efficiency.The paper connects this strategy to strong generalization and efficient policy learning across tasks.
- The method demonstrates applicability across diverse domains through extensive simulation experiments and downstream policy learning.The conclusion highlights generalization ability, efficient policy learning, and broad applicability.
- Post-training scalability remains at an early stage because of limited computational resources, and reinforcement-learning integration requires further research.The authors identify larger video datasets and more robust, adaptive policy learning as future directions.
A Implementation Details
The implementation uses curated video datasets, task-specific training objectives, and resource-intensive full-parameter optimization across simulation and real-robot settings. Real-robot finetuning uses three camera views, end-effector control, and world-model pretraining.
- Post-training Stage: Post-training uses high-quality robotics datasets with dataset-specific frame sampling, filtering short videos and sequences without text instructions.Sampling intervals target approximately one-second gaps between keyframes, and sequences with fewer than six frames are removed.
- Post-training Stage: The post-training comparisons vary only the strategy while keeping the dataset fixed, with supervision applied to task-specific action or vision tokens.The action, text-to-image, video prediction, and world-model tasks use distinct input organizations and loss targets.
- Simulation Finetuning: Training uses six input frames, 256×256 observations, a maximum sequence length of 6400, and 50k full-parameter steps on 32 A100 GPUs.The reported run takes approximately 4–5 days.
- Real-robot Finetuning: Real-robot finetuning uses three camera views, 128×128 images, end-effector pose control, 14-dimensional actions, and action chunks of 20.The setup is evaluated on the ALOHA platform.
- Real-robot Finetuning: World-model pretraining on a collected real-ALOHA video dataset provides substantial benefits when transferring to real-robot execution.Each task is trained for 8k steps with batch size 256 and learning rate 5 × 10^-5.
B Real-Robot Experiments
The AgileX Cobot Magic dual-arm robot uses three RGB cameras for visual observation, combining wrist-mounted views with an overhead high-angle view.
- The real-world setup uses an AgileX Cobot Magic dual-arm robot.
- Three RGB cameras provide visual observations from the left wrist, right wrist, and an overhead high-angle position.
- The camera arrangement combines local wrist views with a high-angle scene view.
B.1 ALOHA Experimental Setup
The real-world experiments use the AgileX Cobot Magic V2.0 dual-arm platform and collect diverse manipulation trajectories from three camera perspectives. The dataset covers eight tasks with approximately 500 trajectories per task.
- ALOHA Experimental Setup: The experimental platform is a dual-arm AgileX Cobot Magic V2.0 robot with three camera views.The setup supports manipulation tasks collected from real-world scenarios.
- ALOHA Experimental Setup: Real-world examples include wiping a whiteboard, organizing tableware, making a burger, and plugging in a connector.
- ALOHA Experimental Setup: The real-world dataset contains eight tasks recorded at 30 Hz, with approximately 500 trajectories per task on average.Static frames at the beginning and end of each trajectory were filtered during preprocessing.
- ALOHA Experimental Setup: Table 9 summarizes the collected real-world task trajectories, including Fold Clothes.
C Autonomous Driving Experiments
The autonomous-driving setup uses NAVSIM, a benchmark emphasizing challenging scenarios, with images and ego status predicting trajectories for the next eight frames.
- NAVSIM Setup: NAVSIM contains 1,192 training-and-validation scenarios and 136 test scenarios.The dataset is resampled from OpenScene to emphasize challenging autonomous-driving scenarios.
- Training Setup: Input images are resized to 512×288 before model training.
- Training Setup: The current image frame and ego status are used to predict trajectories for the next 8 frames.Both action and ego status are encoded with the fast tokenizer.