Source-linked AI summary
InternVLA-A1: Unifying Understanding, Generation and Action for Robotic Manipulation
Junhao Cai, Zetao Cai, Jiafei Cao, Yilun Chen, Zeyu He, Lei Jiang, Hang Li, Hengjie Li, Yang Li, Yufei Liu, Yanan Lu, Qi Lv, Haoxiang Ma, Jiangmiao Pang, Yu Qiao, Zherui Qiu, Yanqing Shen, Xu Shi, Yang Tian, Bolun Wang, Hanqing Wang, Jiaheng Wang, Tai Wang, Xueyuan Wei, Chao Wu, Yiman Xie, Boyang Xing, Yuqiang Yang, Yuyin Yang, Qiaojun Yu, Feng Yuan, Jia Zeng, Jingjing Zhang, Shenghan Zhang, Shi Zhang, Zhuoma Zhaxi, Bowen Zhou, Yuanzhen Zhou, Yunsong Zhou, Hongrui Zhu, Yangkun Zhu, Yuchen Zhu
TL;DR
Existing VLA models provide semantic understanding but lack physical-dynamics reasoning, while video-prediction approaches often lack semantic grounding and are sensitive to prediction errors. InternVLA-A1 unifies scene understanding, visual foresight generation, and action execution, training these capabilities on heterogeneous real-robot, simulated, and human-video data. It achieves consistent gains across static, dynamic, and simulation evaluations, including +4.4% over pi0.5 on static manipulation and +26.7% on dynamic manipulation tasks.
Problem
VLA models lack physical-dynamics reasoning, while video-prediction approaches often lack semantic grounding and are brittle to video prediction errors.
Method
InternVLA-A1 uses a unified Mixture-of-Transformers framework integrating scene understanding, visual foresight generation, and action execution with heterogeneous training data.
Results
InternVLA-A1 consistently outperforms prior models across static manipulation, dynamic manipulation, and simulation benchmarks, including +4.4% over pi0.5 on static tasks and +26.7% on dynamic tasks.
Takeaways & Limitations
Jointly coupling semantic reasoning with dynamics prediction and heterogeneous data yields particularly strong robustness in highly dynamic manipulation scenarios.
Takeaways & Limitations
The understanding expert lacks joint training with large-scale multimodal VQA data, and efficient foresight generation requires lower-fidelity future-frame prediction.
Abstract
from arXiv · showhide
Prevalent Vision-Language-Action (VLA) models are typically built upon Multimodal Large Language Models (MLLMs) and demonstrate exceptional proficiency in semantic understanding, but they inherently lack the capability to deduce physical world dynamics. Consequently, recent approaches have shifted toward World Models, typically formulated via video prediction; however, these methods often suffer from a lack of semantic grounding and exhibit brittleness in the presence of video prediction errors. To synergize semantic understanding with dynamic predictive capabilities, we present InternVLA-A1. This model employs a unified Mixture-of-Transformers architecture, coordinating three experts for scene understanding, visual foresight generation, and action execution. These components interact seamlessly through a unified masked self attention mechanism. Building upon InternVL3 and Qwen3-VL, we instantiate InternVLA-A1 at 2B and 3B parameter scales. We pre-train these models on heterogeneous data sources over real-world robot data, synthetic simulation data, and human videos, covering over 692M frames. This hybrid training strategy effectively harnesses the diversity of synthetic simulation data while minimizing the sim-to-real gap. We evaluated InternVLA-A1 on 12 real-world robotic tasks and a simulation benchmark. The results show that InternVLA-A1 consistently outperforms prior leading models: compared with pi0.5, it achieves +4.4\% on static manipulation tasks and +2.6\% on the RoboTwin 2.0 simulation benchmark, and delivers a +26.7\% boost on dynamic manipulation tasks.
1. Introduction
Existing VLA policies provide strong semantic understanding but remain limited in physical-dynamics reasoning and adaptation to dynamic scene variations. InternVLA-A1 addresses these challenges by combining foresight-aware architecture with heterogeneous training data, and evaluates the resulting system across real-world manipulation settings.
- Generalization bottlenecks arise from deficient physical-world cognition and limited adaptive manipulation capabilities.
- Existing VLA models built on MLLMs excel at semantic understanding but lack physical-dynamics reasoning.
- Scaling real-robot data improves adaptability but remains costly and does not fully address scene variation in highly dynamic environments.The cited examples include 10,000 hours of demonstrations and continued difficulty adapting to dynamic scene variations.
- Simulation offers diverse, efficient data, while real-world data provides physical fidelity; combining them is proposed to mitigate their respective limitations.
- InternVLA-A1 integrates understanding, visual foresight generation, and action execution while training on simulated trajectories, real-robot demonstrations, and egocentric human videos.The joint pipeline is designed to support scalable variation, physical grounding, and richer manipulation representations.
- The model is evaluated on 12 real-world robotic tasks and a simulation benchmark, including real-world dynamic manipulation tasks.
2. Related Works
Related work combines foundation-model capabilities, video prediction, and increasingly diverse robotic data for vision-language-action learning. InternVLA-A1 is positioned as integrating semantic reasoning with future visual-state prediction while jointly using real, simulated, and human-video data.
- VLA methods combine multimodal foundation models with robotic control, using either vocabulary replacement or mixture-based action modeling.
- World-model approaches use future video prediction, sometimes pairing video generators with inverse dynamics or action-execution guidance.
- Video-guided policies can be sensitive to generation quality and lack the semantic reasoning capabilities of MLLMs.
- Real-robot datasets capture physical dynamics but become prohibitively expensive and operationally inefficient to expand by another order of magnitude.
- Simulation data provides low-cost diversity and long-tail scene variation, complementing the physical realism of real-world robot data.
3. InternVLA-A1: Unified Understanding-Generation-Action Framework
InternVLA-A1 unifies scene understanding, visual foresight generation, and action execution through a Mixture-of-Transformers pipeline with masked information flow. Its generation expert compresses multi-view temporal inputs for efficient predictive guidance, while joint foresight and action objectives train the model to forecast dynamics and synthesize control commands.
- Architecture Overview: InternVLA-A1 adopts a Mixture-of-Transformers architecture to integrate scene understanding, visual foresight, and action execution.The unified framework is designed to combine multimodal semantic processing with predictive dynamics modeling.
- Architecture Overview: The understanding expert processes multimodal inputs, the generation expert predicts future visual states, and the action expert uses both to produce control commands via Flow Matching.Information flows sequentially from understanding to generation to action.
- Generation Expert: The generation expert uses six 256 × 256 images from three views and two timestamps, compressing them to 96 tokens before transformer processing.The inputs include head and two wrist views at the current and historical timesteps.
- Generation Expert: Temporal average pooling reduces the processed sequence to 48 tokens, which are projected and upsampled into visual guidance for action execution.The non-autoregressive decoding design is computationally efficient and provides effective guidance for subsequent control.
- Architecture Overview: A blockwise attention mask enforces understanding →generation →action information flow while allowing bidirectional attention within understanding and generation blocks.Later expert blocks attend to earlier blocks, but earlier blocks cannot attend forward.
- Optimization Objectives: The model jointly optimizes latent visual foresight generation and Flow Matching-based action prediction across pre-training and post-training.The training tuples include observations, language, proprioception, and action annotations where available.
4. Data Corpus
InternVLA-A1 is pretrained on a heterogeneous corpus combining synthetic simulation, real-world robot demonstrations, and human videos to balance scalability, physical fidelity, and manipulation diversity.
- 4. Data Corpus: The pretraining corpus interleaves synthetic simulation data, real-world robot data, and human videos using configurable sampling weights.The recipe is summarized in Table 3.
- 4.2. Simulated synthetic data: InternData-A1 contributes over 630k trajectories and 7,433 hours across 4 embodiments, 18 skills, 70 tasks, and 227 scenes.It covers rigid, articulated, deformable, and fluid-object manipulation through an autonomous compositional simulation pipeline.
- 4. Data Corpus: Synthetic data provides scalability and diversity, while real-world demonstrations remain essential for nuanced physical dynamics and reducing the sim-to-real gap.The real-world datasets are selected for broad trajectory coverage, diverse task distributions, and high-quality teleoperation.
- 4.3. Real-world robot data: The real-world component incorporates AgiBot-World and RoboMind to expand trajectory and task coverage with high-quality teleoperation data.These datasets complement the simulated sources in the pretraining corpus.
- 4.4. Human video data: EgoDex contributes 829 hours spanning over 200 tasks, supplying egocentric human manipulation videos without human action labels.These videos support the generation expert’s modeling of real-world interaction dynamics and manipulation skills.
5. Experiments
InternVLA-A1 is evaluated across real-world static and dynamic manipulation, RoboTwin 2.0 simulation, and ablations of pre-training and its generation expert. It consistently outperforms prior models, with especially strong gains in dynamic settings.
- Experimental Setup: Evaluations cover 12 real-world tasks across three robot embodiments, including static, dynamic, long-horizon, articulated-object, and contact-rich manipulation.Performance is averaged over 30 rollouts per task under predefined object placements and scene initializations.
- Static Manipulation: 75.1% average success rate is achieved by InternVLA-A1 (3B) on 10 real-world static tasks, exceeding π0.5 by +4.4%.InternVLA-A1 (2B) reaches 64.7%, surpassing the larger π0 (3.3B) baseline at 60.6%.
- Dynamic Manipulation: 86.7% average success rate is achieved by InternVLA-A1 (3B) across Express Sorting and Ingredient Picking, including a +26.7% gain over π0.5 on Express Sorting.On Ingredient Picking, it reaches 93.3% versus 66.7% for π0.5, a +26.6% improvement.
- Simulation Benchmark: InternVLA-A1 (3B) surpasses π0.5 by 2.6% in both Easy and Hard settings of the RoboTwin 2.0 benchmark.The benchmark covers 50 bimanual tasks with clean and domain-randomized settings.
- Ablation Studies: Removing pre-training reduces average success from 77.0% to 25.4%, an overall performance drop of 51.6%.The ablation compares InternVLA-A1 with a version trained from scratch.
- Ablation Studies: Jointly pre-training on human videos, synthetic data, and real-world demonstrations achieves the best overall performance and improves real-world manipulation results.Place Flower reaches 60.0% and Sort Parts reaches 53.3% with heterogeneous joint pre-training.
- Ablation Studies: Adding the generation expert improves performance in 11 of 12 real-world tasks and raises average success from 57.6% to 77.0%.The generation-expert ablation produces gains ranging from 6.7% to 53.3%.
- Visualization: Future predictions sacrifice high-frequency visual detail to prioritize inference efficiency while preserving essential motion trends and dynamics.The visualization argues that instructive latent features matter more than pixel-level clarity for guiding action execution.
6. Conclusion and Limitations
InternVLA-A1 unifies scene understanding, visual-foresight generation, and action execution through a Mixture-of-Transformers design, coupling semantic reasoning with dynamics prediction. It achieves robustness across static, dynamic, and simulation benchmarks, while remaining limited by semantic-training coverage and visual-foresight fidelity.
- InternVLA-A1 integrates scene understanding, visual-foresight generation, and action execution through a unified Mixture-of-Transformers architecture.
- The architecture couples semantic reasoning with dynamics prediction and supports joint training on human videos, synthetic data, and real-world demonstrations.
- InternVLA-A1 achieves consistent robustness across static manipulation, dynamic manipulation, and simulation benchmarks, with particularly strong gains in highly dynamic scenarios.
- The understanding expert lacks joint training with large-scale multimodal VQA data, weakening general semantic reasoning and complex instruction following.
- Inference-efficiency constraints reduce visual prediction fidelity and limit the granularity of generated future frames.
A. Contributors
The contributors are listed in alphabetical order by their last names.
- All contributors are listed in alphabetical order by their last names.
Core Contributors
The listed core contributors are Junhao Cai, Yang Li, Haoxiang Ma, Jiangmiao Pang, Zherui Qiu, Yang Tian, Jia Zeng, and Hongrui Zhu.
- The listed contributors are Junhao Cai, Yang Li, Haoxiang Ma, Jiangmiao Pang, Zherui Qiu, Yang Tian, Jia Zeng, and Hongrui Zhu.