PaperlayerView exact revision on arXiv

Source-linked AI summary

InternVLA-A1: Unifying Understanding, Generation and Action for Robotic Manipulation

Junhao Cai, Zetao Cai, Jiafei Cao, Yilun Chen, Zeyu He, Lei Jiang, Hang Li, Hengjie Li, Yang Li, Yufei Liu, Yanan Lu, Qi Lv, Haoxiang Ma, Jiangmiao Pang, Yu Qiao, Zherui Qiu, Yanqing Shen, Xu Shi, Yang Tian, Bolun Wang, Hanqing Wang, Jiaheng Wang, Tai Wang, Xueyuan Wei, Chao Wu, Yiman Xie, Boyang Xing, Yuqiang Yang, Yuyin Yang, Qiaojun Yu, Feng Yuan, Jia Zeng, Jingjing Zhang, Shenghan Zhang, Shi Zhang, Zhuoma Zhaxi, Bowen Zhou, Yuanzhen Zhou, Yunsong Zhou, Hongrui Zhu, Yangkun Zhu, Yuchen Zhu

arXiv:2601.02456v2cs.RO

TL;DR

VLA policies have strong semantic understanding but lack physical-dynamics reasoning. InternVLA-A1 unifies scene understanding, visual-foresight generation, and action execution in a Mixture-of-Transformers architecture. It consistently outperforms prior leading models; versus π0.5, it gains +4.4% on real-world static tasks, +2.6% on RoboTwin 2.0, and +26.7% on real-world dynamic tasks.

  • Problem

    VLA policies built on MLLMs have strong semantic understanding but lack physical-dynamics reasoning about state evolution under motion and contact.

    Source: §1 · p. 2

  • Method

    InternVLA-A1 unifies scene understanding, visual-foresight generation, and action execution in a Mixture-of-Transformers architecture, jointly trained on human videos, synthetic data, and real-world demonstrations.

    Source: §6 · p. 16

  • Results

    InternVLA-A1 consistently outperforms prior leading models; versus π0.5, it gains +4.4% on real-world static tasks, +2.6% on RoboTwin 2.0, and +26.7% on real-world dynamic tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    InternVLA-A1 achieves consistent robustness across static manipulation, dynamic manipulation, and simulation benchmarks, with particularly strong gains in highly dynamic scenarios.

    Source: §6 · p. 16

    The understanding expert is not jointly trained with large-scale multimodal VQA data, weakening general semantic reasoning and complex instruction following.

    Source: §6 · p. 16

Abstract

from arXiv · show

Prevalent Vision-Language-Action (VLA) models are typically built upon Multimodal Large Language Models (MLLMs) and demonstrate exceptional proficiency in semantic understanding, but they inherently lack the capability to deduce physical world dynamics. Consequently, recent approaches have shifted toward World Models, typically formulated via video prediction; however, these methods often suffer from a lack of semantic grounding and exhibit brittleness in the presence of video prediction errors. To synergize semantic understanding with dynamic predictive capabilities, we present InternVLA-A1. This model employs a unified Mixture-of-Transformers architecture, coordinating three experts for scene understanding, visual foresight generation, and action execution. These components interact seamlessly through a unified masked self attention mechanism. Building upon InternVL3 and Qwen3-VL, we instantiate InternVLA-A1 at 2B and 3B parameter scales. We pre-train these models on heterogeneous data sources over real-world robot data, synthetic simulation data, and human videos, covering over 692M frames. This hybrid training strategy effectively harnesses the diversity of synthetic simulation data while minimizing the sim-to-real gap. We evaluated InternVLA-A1 on 12 real-world robotic tasks and a simulation benchmark. The results show that InternVLA-A1 consistently outperforms prior leading models: compared with pi0.5, it achieves +4.4\% on static manipulation tasks and +2.6\% on the RoboTwin 2.0 simulation benchmark, and delivers a +26.7\% boost on dynamic manipulation tasks.

1. Introduction

InternVLA-A1 unifies semantic understanding, visual foresight, and action, and combines simulated trajectories, real-robot demonstrations, and human videos to address generalization challenges. It outperforms 𝜋0.5 across the reported real-world and simulation results.

1. Introduction

  • Existing VLA policies struggle to generalize to scene variations, especially dynamic settings, because they lack physical-world cognition and adaptive manipulation capabilities.The introduction frames progress as requiring both architectural advances and large-scale, diverse robot action data.Source: §1 · p. 2
  • +4.4% on real-world static tasks, +2.6% on the RoboTwin 2.0 simulation benchmark, and +26.7% on real-world dynamic tasks versus 𝜋0.5.The evaluation covers 12 real-world tasks and a simulation benchmark.Source: §1 · p. 3
  • InternVLA-A1 integrates understanding, visual foresight generation, and action to couple MLLM semantic reasoning with World Model-style prediction for foresight-aware action.This unified architecture targets the weak semantic grounding and prediction-error sensitivity of video-prediction approaches, while addressing the physical-dynamics reasoning limitations of text-based visual features.Source: §1 · pp. 2–3
  • Training on simulated robot trajectories, real-robot demonstrations, and egocentric human videos combines simulation diversity with real-interaction fidelity and richer representations of complex manipulation.The joint recipe uses real-robot interactions to reduce the sim-to-real gap and human videos to enrich visual representations.Source: §1 · pp. 2–3

2. Related Works

Related work frames InternVLA-A1 against VLA architectures and video-prediction methods, and motivates combining semantic understanding with dynamics prediction. It also situates the model’s training recipe among real-robot, simulated, and human-video data sources.

Model architecture

  • InternVLA-A1 unifies MLLM semantic understanding with dynamics prediction, unlike prior VLA designs that use discrete action tokens, action experts, or separate reasoning and action systems.RT-2 (Brohan et al., 2023) and OpenVLA (Kim et al., 2024) map text tokens to discrete actions; 𝜋0 (Black et al., 2024) uses an action expert, while GR00T N1/N1.5 (Bjorck et al., 2025) pairs a VLM with a DiT. 𝜋0.5 (Black et al., 2025) combines sub-task and action prediction for long-horizon tasks and cross-scene generalization.Source: §2 · p. 3

Video prediction and world models

  • InternVLA-A1 integrates MLLMs with future visual-state prediction to provide semantically grounded action guidance and greater adaptability to video-prediction errors.Prior methods use future-video generation, image prediction, or pretrained video models to guide robotic control, but can be sensitive to generation quality and lack MLLM-level semantic reasoning; examples include UniPi (Du et al., 2023), UniSim (Yang et al., 2024), and VPP (Hu et al., 2025).Source: §2 · pp. 3–4

Training data

  • InternVLA-A1’s data recipe mixes real-world robot data, synthetic simulation data, and human videos to increase diversity and help close the sim-to-real gap.Real-robot data captures physical dynamics but is costly to scale; simulation offers diverse, low-cost data, while human video provides visual priors without costly teleoperation. InternData-A1 (Tian et al., 2025b) is cited as a simulated-data pipeline spanning multiple object types and comprising over 630k trajectories and 7,433 hours.Source: §2 · p. 4

3. InternVLA-A1: Unified Understanding-Generation-Action Framework

InternVLA-A1 unifies scene understanding, visual foresight, and action execution in a Mixture-of-Transformers framework. Its experts share a masked-attention pipeline and are jointly trained for visual prediction and action prediction.

3.1. Architecture Overview

  • The Mixture-of-Transformers architecture coordinates understanding, generation, and action experts: semantic context informs future visual-state prediction, which guides control-command generation.The action expert uses Flow Matching to produce control commands.Source: §3.1 · pp. 4–5, Figure 2 · p. 5

3.2. Core Components

  • The generation expert encodes six 256 × 256 images from three views at t and t−15 into 96 compressed tokens, then parallel-decodes future frames at t+15 without autoregression.The inputs use a COSMOS VAE tokenizer; temporal average pooling and upsampling precede reconstruction of the predicted frames.Source: §3.2 · pp. 6–7
  • The action expert conditions on semantic and predictive features plus proprioception to predict action chunks; blockwise masking enforces understanding → generation → action information flow.Understanding and generation tokens attend bidirectionally within their respective blocks, while action-token attention is restricted by the blockwise mask.Source: Figure 2 · p. 5, §3.2 · p. 7

3.3. Optimization Objectives

  • Pre-training and post-training share a joint objective for visual foresight and action prediction, using foresight data D1 and its action-annotated subset D2.The foresight expert predicts future COSMOS latents from current and historical observations plus the understanding prefix; the objective is intended to help the model internalize physical dynamics.Source: §3.3 · pp. 7–8
  • Flow matching learns continuous pathways from Gaussian noise to demonstrated action chunks, conditioned on proprioception and understanding and generation features, with inference using Euler ODE updates.The training procedure samples time steps from Beta(1.5, 1.0) and interpolates Gaussian noise with target actions.Source: §3.3 · pp. 7–8

3.4. Implementation Details

  • InternVLA-A1 is instantiated at 2B and 3B scales using InternVL3-1B and Qwen3-VL-2B understanding backbones, respectively; both run at approximately 13 Hz on one RTX 4090.The 2B model uses 448 × 448 inputs while the 3B model uses 224 × 224, offsetting the 2B model’s parameter advantage in latency.Source: §3.4 · p. 8
  • Training uses large-scale pre-training for 700K AdamW steps with a constant learning rate, followed by task-specific post-training with a lower learning rate and warmup and decay.The loss-balancing hyperparameter λ is set to 0.01, and the historical-to-current and current-to-future frame interval m is 15.Source: §3.4 · pp. 8–9
  • Load-balanced Parallel Training assigns datasets across workers to balance loads and reduce memory pressure and implicit reweighting; controlled replication avoids idle workers when datasets are fewer than workers.The greedy assignment orders datasets by size and places each on the least-loaded worker; replicated datasets use independent random seeds and load-aware placement.Source: §3.4 · p. 9

4. Data Corpus

The pretraining corpus combines synthetic simulation, real-robot demonstrations, and human video, with trajectories from different sources interleaved using configurable sampling weights.

4.1. Pre-training data recipe

  • Pretraining mixes synthetic simulation data, real-world robot data, and human videos by interleaving trajectories with configurable sampling weights.Table 3 presents the data mixture.Source: §4.1 · pp. 9–10, Table 3 · p. 10

4.2. Simulated synthetic data

  • InternData-A1 (Tian et al., 2025b) anchors synthetic pretraining with over 630k trajectories and 7,433 hours across 4 embodiments, 18 skills, 70 tasks, and 227 scenes.Its autonomous, compositional simulation pipeline supports long-horizon skill composition and flexible task assembly; the corpus also incorporates RoboTwin simulation data (Chen et al., 2025c).Source: §4.2 · p. 10

4.3. Real-world robot data

  • AgiBot-World (Bu et al., 2025) and RoboMind (Wu et al., 2024b) provide high-quality, diverse robot demonstrations to capture physical dynamics and help bridge the sim-to-real gap.The data are selected for broad trajectory coverage, diverse tasks, and high-quality teleoperation.Source: §4.3 · p. 10

4.4. Egocentric human video

  • EgoDex (Hoque et al., 2025) contributes 829 hours of egocentric dexterous-manipulation video across over 200 tasks, with human action labels excluded during pretraining.Human-hand-object interactions help the generation expert capture nuanced dynamics and diverse manipulation skills.Source: §4.4 · pp. 10–11

5. Experiments

InternVLA-A1 is evaluated on real-world and simulated manipulation tasks, with results showing strong performance across static, dynamic, and simulation settings. Ablations examine pre-training data and the generation expert, while visualizations show how future predictions prioritize motion trends over fine detail.

5.1. Evaluation Configuration

  • Across three robot embodiments, evaluation covers ten static tasks and two dynamic task families, reporting average success over 30 rollouts per task.The suite includes articulated-object, contact-rich, long-horizon bimanual, and moving-target tasks; the dynamic tasks require reasoning about near-future scene changes.Source: §5.1 · p. 11

5.2. Evaluation on the Static Manipulation Tasks

  • On static tasks, InternVLA-A1 (3B) averages 75.1% success, +4.4% over 𝜋0.5 (70.7%) and +14.5% over 𝜋0 (60.6%).InternVLA-A1 (2B) averages 64.7%, exceeding the larger 𝜋0 (3.3B) baseline's 60.6%.Source: §5.2 · pp. 11–12, Table 4 · p. 12

5.3. Evaluation on Dynamic Manipulation Tasks

  • InternVLA-A1 (3B) averages 86.7% success across two dynamic tasks, scoring 80.0% on Express Sorting (+26.7% over π0.5 at 53.3%) and 93.3% on Ingredient Picking (+26.6% over π0.5 at 66.7%).Ingredient Picking is described as a highly dynamic and timing-sensitive setting.Source: §5.3 · p. 13

5.4. Evaluation on Simulation Benchmark

  • On RoboTwin 2.0, InternVLA-A1 (3B) surpasses 𝜋0.5 by 2.6% in both Easy and Hard settings and 𝜋0 by 9.4% and 10.1%, respectively.The benchmark covers 50 bimanual tasks; models are fine-tuned on 27,500 demonstrations and evaluated over 100 trials.Source: §5.4 · p. 13, Figure 7 · p. 14

5.5. Ablation Studies

  • Removing pre-training drops average success from 77.0% to 25.4%; joint pre-training on human videos, synthetic data, and real-world demonstrations achieves 60.0% on Place flower and 53.3% on Sort parts.Simulation-only pre-training yields 88.3%/88.5% on RoboTwin Easy/Hard but 53.3% on Place flower and 33.3% on Sort parts; adding human videos raises RoboTwin results to 89.4%/89.3% and Sort parts to 40.0%.Source: §5.5 · p. 14, Figure 8 · p. 14, Table 5 · p. 14
  • Without the generation expert, InternVLA-A1 (3B) loses in 11 of 12 real-world tasks, with gains from 6.7% to 53.3% for the full model and average success falling from 77.0% to 57.6%.The largest declines occur in Express Sorting and In-motion Ingredient Picking.Source: §5.5 · p. 15, Figure 9 · p. 15

5.6. Visualization

  • Future predictions sacrifice high-frequency visual details for inference efficiency while capturing essential motion trends and dynamics to guide action execution.The authors prioritize instructive latent features over pixel-level granularity or visual clarity.Source: §5.6 · p. 15, Figure 10 · p. 15

6. Conclusion and Limitations

InternVLA-A1 unifies scene understanding, visual-foresight generation, and action execution, with robustness across manipulation and simulation benchmarks. Its understanding and foresight components retain limitations in semantic reasoning and future-frame fidelity.

Conclusion

  • InternVLA-A1 integrates scene understanding, visual-foresight generation, and action execution through a Mixture-of-Transformers design, enabling joint training on human videos, synthetic data, and real-world demonstrations.The framework couples semantic reasoning with dynamics prediction and shows robustness across static manipulation, dynamic manipulation, and simulation benchmarks, with particularly strong gains in highly dynamic scenarios.Source: §6 · p. 16

Limitations

  • The understanding expert lacks joint training with large-scale multimodal VQA data, weakening general semantic reasoning and complex instruction following.Source: §6 · p. 16
  • The visual-foresight module compromises image-prediction fidelity for efficient inference, limiting the granularity of generated future frames.Source: §6 · p. 16

Related papers

Loading 2601.02456v2…