Source-linked AI summary
UP-VLA: A Unified Understanding and Prediction Model for Embodied Agent
Jianke Zhang, Yanjiang Guo, Yucheng Hu, Xiaoyu Chen, Xiang Zhu, Jianyu Chen
TL;DR
VLM-based VLA models provide semantic knowledge but have limited low-level spatial and physical-dynamics understanding important for embodied control. UP-VLA unifies multimodal understanding and future prediction during VLA training, reporting improved simulated and real-world manipulation performance, including a 33% gain on Calvin ABC→D. The paper also identifies a visual-generation generalization limitation in the Calvin D environment.
Problem
Pretrained VLMs often neglect low-level visual details, spatial information, and physical dynamics, leaving an important gap for embodied control.
Method
UP-VLA co-trains an autoregressive VLA model with multimodal understanding and future prediction objectives across three dataset types.
Results
33% improvement on the Calvin ABC→D generalization benchmark was reported, alongside improved success rates across simulated and real-world manipulation tasks.
Takeaways & Limitations
Future image prediction improves policy precision and visual generalization, while multimodal understanding strengthens semantic grounding and spatial understanding.
Takeaways & Limitations
In Calvin D, predicted frames can inherit background colors from the ABC training datasets, limiting generalization in visual generation tasks.
Abstract
from arXiv · showhide
Recent advancements in Vision-Language-Action (VLA) models have leveraged pre-trained Vision-Language Models (VLMs) to improve the generalization capabilities. VLMs, typically pre-trained on vision-language understanding tasks, provide rich semantic knowledge and reasoning abilities. However, prior research has shown that VLMs often focus on high-level semantic content and neglect low-level features, limiting their ability to capture detailed spatial information and understand physical dynamics. These aspects, which are crucial for embodied control tasks, remain underexplored in existing pre-training paradigms. In this paper, we investigate the training paradigm for VLAs, and introduce \textbf{UP-VLA}, a \textbf{U}nified VLA model training with both multi-modal \textbf{U}nderstanding and future \textbf{P}rediction objectives, enhancing both high-level semantic comprehension and low-level spatial understanding. Experimental results show that UP-VLA achieves a 33% improvement on the Calvin ABC-D benchmark compared to the previous state-of-the-art method. Additionally, UP-VLA demonstrates improved success rates in real-world manipulation tasks, particularly those requiring precise spatial information.
1. Introduction
UP-VLA addresses weaknesses in VLM-based VLA pretraining by jointly emphasizing semantic understanding and low-level visual information. Its experiments report gains across simulated and real-world manipulation settings.
- Motivation: VLM-based VLA models inherit semantic knowledge and reasoning capabilities, but often miss low-level visual details, spatial relationships, and physical dynamics.These limitations motivate a training pipeline that retains semantic understanding while emphasizing features critical for control.
- Approach: UP-VLA co-trains an autoregressive model on three dataset types using a flexible attention mask, combining multi-modal understanding with future predictive generation.The design aligns representations with high-level features from understanding data and low-level features from future prediction.
- Results: 33% improvement on the Calvin ABC→D generalization benchmark demonstrates UP-VLA’s reported advantage over the previous state-of-the-art method.The study evaluates task clusters spanning simulation and real-world settings.
- Results: UP-VLA improves success rates across simulated and real-world manipulation tasks, including settings requiring semantic generalization, adaptation, and precise control.The comparison includes VLM-based VLA and visual prediction-based methods selected from each method type.
2. Related Works
Related work develops VLA policies from pretrained vision-language and vision models, increasingly using visual prediction to improve robotic generalization. UP-VLA builds on these directions by combining future-image prediction with multimodal understanding.
- VLA Models for Generalist Robot Policies: VLM-based VLA models leverage pretrained language and visual scene understanding to generate robot actions and generalize across tasks.RT-2 is cited as an example that autoregressively generates discrete action tokens from a VLM.
- Visual Pretraining Methods for Robotics: Robotics visual pretraining has progressed from pretrained vision encoders toward predicting keyframes, future images, and latent visual changes.Examples include SuSIE, GR-1, PAD, and IGOR.
- Visual Pretraining Methods for Robotics: Visual prediction tasks are reported to benefit visual generalization to unseen scenes, while UP-VLA uses autoregressive VLMs to predict future images and capture physical dynamics.This direction complements VLM-based semantic understanding with predictive visual information.
3. Preliminaries
Language-conditioned robot control models decision sequences from instructions and visual observations to actions. VLA pretraining extends this setup with multimodal understanding and image-generation objectives using distinct token representations.
- VLA for Language Conditioned Robot Control: A language-conditioned manipulation problem specifies a task through a free-form instruction and an initial visual observation, with demonstrations containing observation-action trajectories.The demonstrations are represented as sequences of paired visual observations and actions.
- VLA for Language Conditioned Robot Control: VLA policies minimize action prediction error from visual observations and language instructions, using VLM capabilities to improve task and semantic generalization.The cited benefits include understanding unseen objects and reasoning about complex natural-language instructions.
- Unified Vision-Language Pretraining via Autoregressive Modelling: Unified vision-language pretraining adds image-generation tasks to multimodal language models, using discrete image tokens alongside continuous visual representations.Language tokens are denoted by l, while u and v represent continuous and discrete image tokens for different tasks.
4. Methodology
UP-VLA uses a unified VLM-based pipeline to combine multimodal understanding, future visual prediction, and action learning. Its prompting, attention, and joint training design connects semantic scene understanding with low-level visual information for embodied control.
- Unified architecture: The model encodes visual inputs into language-space or discrete image tokens, then processes them with a shared language-model backbone.Continuous visual features support multimodal understanding, while discrete tokens represent current observations for future prediction.
- Visual prediction: For future visual prediction, image tokens follow language tokens so they attend to the instruction and all preceding information before reconstructing the predicted observation.A special PRE token identifies the prediction task, and a discrete decoder reconstructs the future image.
- Action learning: Action learning jointly predicts future observations and action sequences, while action tokens use features from the prediction-and-understanding pathway.A policy head maps action-token features to low-level actions, and generated scene descriptions extend the instruction prompt.
- Unified prompting: The unified prompting mechanism assigns task-specific token arrangements and attention patterns so action learning can use both high-level and low-level visual information.MMU places image tokens before language tokens, prediction places image tokens after language tokens, and action learning concatenates both task representations.
- Unified architecture: UP-VLA combines multimodal understanding, future image prediction, and action modeling within a unified VLA training pipeline.The three targets are language modeling, image modeling, and action modeling.
- Training objective: The training objective combines multimodal-understanding, future-prediction, and action losses, using staged pretraining and downstream action tuning.The first stage mixes robotic demonstrations for future prediction with image-text pairs for understanding; the second stage fine-tunes on embodied tasks while continuing image-text co-training.
5. Experiments
UP-VLA is evaluated in simulation and real-world manipulation settings against VLM-based and future-prediction baselines. Results show stronger multitask generalization, semantic grounding, spatial understanding, and precise control, while ablations support both visual prediction and multimodal understanding.
- Simulation Evaluation: UP-VLA achieves the highest performance on both ABC→D and ABCD→D simulation tasks, with higher completion lengths in both scenarios.These results indicate stronger multitask learning and generalization in simulation tasks.
- Ablation Studies: Visual prediction raises UP-VLA’s Calvin completion length from 1.44 without prediction to 4.08 with prediction.RoboFlamingo reaches 2.47, compared with GR-1’s 3.06, while UP-VLA-RT-2 reaches 1.44.
- Ablation Studies: UP-VLA outperforms prediction-based methods, while removing multimodal understanding makes the model perform worse than full UP-VLA.The comparison isolates complementary contributions from visual prediction and multimodal understanding.
- Real Robot Evaluation: UP-VLA-based methods outperform RT-1 and Diffusion Policy on seen real-world tasks, and UP-VLA shows better visual-semantic generalization on unseen objects.The real-world evaluation covers tabletop picking, placing, cable routing, button pressing, and drawer opening with over 2k demonstrations across 6 skills.
- Real Robot Evaluation: UP-VLA performs best on precise operations such as cable routing, grasping small objects, and picking up unseen pens.The authors associate this advantage with integrating future visual prediction for improved understanding of physical space and visual details.
- Ablation Studies: Removing visual prediction reduces Calvin performance from 4.08 to 1.44, while removing multimodal understanding reduces performance on unseen objects but can preserve or improve seen-task performance.The ablations support distinct roles for visual prediction in spatial generalization and multimodal understanding in semantic generalization.
- Quantitative Results: UP-VLA identifies embodied-scene objects and estimates approximate relative positions, but predicted frames can inherit incorrect background colors from training scenes.The authors attribute object-identification errors to data and backbone scale, and background-color errors to insufficient visual-generation pretraining.
6. Conclusion
UP-VLA unifies policy learning with visual prediction and multimodal understanding for embodied action. The reported results associate future image prediction with improved precision and visual generalization, while multimodal knowledge strengthens semantic grounding and spatial understanding.
- UP-VLA unifies policy learning with visual prediction and multimodal understanding to understand scenes, predict future images, and plan actions.
Impact Statement
Accurate robot control requires both high-level semantic content and low-level visual and spatial details. The paper addresses this need with a unified training paradigm that captures semantics and physical-world dynamics.
- Accurate robot control requires high-level semantic information together with low-level visual and spatial details.
A. Implementation Details
UP-VLA uses a pretrained Showo-512x512 backbone with CLIP-VIT and MagVIT components. Pretraining jointly covers future prediction and vision-language understanding before action learning.
- UP-VLA uses pretrained Showo-512x512 (1.3B), CLIP-VIT, and MagVIT components as its backbone.
- The pretraining stage runs for 20k steps with batch size 64 on future prediction and vision-language understanding tasks.A linear warmup is applied during the first 1k steps.
B. Manipulation Dataset Details
The manipulation data combines CALVIN-based simulation setups with manually collected and scripted real-world demonstrations. Real-world evaluation emphasizes semantic generalization in unseen settings with varied objects, backgrounds, distractors, and object positions.
- Simulation data follows the CALVIN benchmark’s training and evaluation-set setups.
- Real-world demonstrations use manual joystick teleoperation for toy-fruit grasping, drawer opening, and cable-routing tasks, while scripted policies collect colored-block grasping data.Manual demonstrations roughly evenly distribute target objects in the field of view.
- Real-world tests primarily use unseen settings, varying distractors, positional variation, backgrounds, and objects to assess semantic generalization.Examples include differently shaped vegetables, arrow-shaped paper, toy pizza, and blocks with unseen colors.