Source-linked AI summary
InternVLA-A1.5: Unifying Understanding, Latent Foresight, and Action for Compositional Generalization
Haoxiang Ma, Junhao Cai, Xiaoxu Xu, Hao Li, Yuyin Yang, Yang Tian, Jiafei Cao, Hongrui Zhu, Zherui Qiu, Zhaxizhuoma, Yuqiang Yang, Jiaqi Peng, Xueyuan Wei, Yangkun Zhu, Jiahao Jiang, Xing Gao, Hanqing Wang, Feng Yuan, Kailin Li, Xueyue Zhu, Tai Wang, Yan Ding, Jiangmiao Pang, Jia Zeng, Jingjing Zhang, Bowen Zhou, Yao Mu, Chunhua Shen, Weinan Zhang
TL;DR
Unified robot policies must combine VLM semantics with physical-dynamics knowledge without weakening pretrained understanding or learning future prediction from scratch. InternVLA-A1.5 preserves native VLM training and uses latent foresight tokens supervised by a frozen video generator, achieving the best or highly competitive results across six simulation benchmarks and strong real-world generalization and long-horizon execution. Its remaining scope is limited by short-horizon foresight supervision and generic frozen video-model priors.
Problem
Existing unified models can weaken pretrained VLM semantics, suffer interference among heterogeneous objectives, and learn future prediction from scratch in pixel space.
Method
InternVLA-A1.5 retains native VLM training and uses learnable foresight tokens to query task-relevant future latents under supervision from a frozen pretrained video generator.
Results
InternVLA-A1.5 achieves the best or highly competitive performance across six simulation benchmarks and four real-world tasks, with clearest gains in zero-shot generalization and long-horizon execution.
Takeaways & Limitations
The policy inherits pretrained video-model dynamics priors without pixel-level generation, while discarding the video branch at deployment to retain real-time inference.
Takeaways & Limitations
Foresight supervision covers only one action chunk, and the frozen generic video generator limits inherited priors according to its embodied-scene coverage.
Abstract
from arXiv · showhide
Unified models for robot manipulation aim to equip one policy with both the semantic priors of pretrained VLMs and the physical dynamics learned through future prediction. In practice, existing designs tend to erode the semantics of the pretrained backbone, suffer interference among heterogeneous objectives, and learn future prediction from scratch in pixel space, leaving the dynamics priors of pretrained video generators unexploited. We present InternVLA-A1.5, which builds the policy on a native VLM backbone that keeps training on VQA and subtask prediction, and attaches a lightweight unified expert for continuous action generation. Future prediction is recast as a latent-querying problem, where a small set of learnable foresight tokens condenses the task-relevant future into a compact latent code under the supervision of a frozen pretrained video generation model, so the policy inherits world-model dynamics priors without ever learning pixel-level generation. The video branch is discarded at inference, keeping real-time control. Pretrained on 1.2M robot episodes and 3M multimodal samples, InternVLA-A1.5 achieves the best overall results on all six simulation benchmarks. In the real world, the preserved semantics deliver the strongest compositional generalization on held-out instruction bindings, and the two designs together sustain long-horizon execution.
1. Introduction
InternVLA-A1.5 addresses unified robot-policy limitations by preserving native VLM understanding while using latent foresight supervision from a pretrained video generator. It combines these designs in evaluations spanning six simulation benchmarks and real-world tasks.
- Motivation: Unified models risk semantic drift, weakened instruction following, objective interference, and pixel-space future prediction learned from scratch.These limitations arise when VQA or language training is reduced and heterogeneous objectives are combined.
- Approach: InternVLA-A1.5 keeps training its native VLM with VQA, subtask prediction, and discrete action objectives during policy learning.The design targets preservation and strengthening of semantic and instruction-following abilities.
- Approach: Latent foresight queries condense task-relevant futures into codes supervised through a frozen pretrained video generation model rather than pixel-space generation.Video prediction supervises the foresight queries, while the generator supplies pretrained dynamics priors.
- Architecture: The Mixture-of-Transformers framework combines a pretrained VLM with a lightweight unified expert using foresight and action query tokens.The expert shares full attention layers with the VLM while maintaining separate linear attention layers, and action queries use flow matching.
- Evaluation: InternVLA-A1.5 is evaluated across six simulation benchmarks and real-world instruction-following and long-horizon tasks.The simulation suite spans single-arm, bimanual, mobile, real-to-sim, and zero-shot generalization settings.
2. InternVLA-A1.5 Model Design
InternVLA-A1.5 uses a pretrained VLM for multimodal perception and reasoning alongside a smaller unified expert for foresight and continuous action prediction. The components share full attention while retaining separate linear-attention processing, and video supervision is training-only.
- Architecture: The Mixture-of-Transformers architecture combines a pretrained VLM backbone with a lightweight unified expert for action prediction.The expert follows the VLM architectural blueprint with a smaller hidden dimension.
- Expert Interaction: The unified expert interacts with the VLM through shared full attention while maintaining separate modality-specific linear-attention layers.This provides multimodal context to the smaller expert while preserving separate processing paths.
- Multimodal Inputs: Robot observations, language, discretized proprioceptive state, and a control-mode token are encoded through the native VLM processing pipeline.This input formulation preserves the pretrained VLM’s multimodal representation.
- VLM Outputs: The VLM can produce visual-question answers, subtask descriptions, and discrete action chunks encoded with the FAST tokenizer.These outputs support multimodal reasoning and action-aware policy learning.
- Visual Foresight: Learnable foresight tokens query shared multimodal context and condition a frozen pretrained video generator on future frames matching the action-chunk horizon.The video branch supervises foresight representations only during training and is discarded at inference.
3. Training Recipe
InternVLA-A1.5 uses a unified training recipe that preserves VLM understanding while adding foresight supervision and continuous action generation. Stage 1 trains shared tokenized robot and VQA inputs; Stage 2 adds latent future prediction through a frozen video model and flow-matching actions.
- Stage 1: VLM Transferring: Stage 1 jointly trains the VLM on VQA answers, subtasks, and discrete action chunks using a shared tokenized format.Robot prompts include image blocks, instructions, control mode, and discretized state; VQA prompts omit state and use an answer span.
- Stage 1: VLM Transferring: Only label tokens receive next-token supervision, with robot labels containing a subtask description followed by FAST action tokens and VQA labels containing an answer span.The same cross-entropy objective covers robot and VQA samples without auxiliary heads or separate loss weighting.
- Stage 2: Foresight and Action Generation: Stage 2 adds a lightweight unified expert that combines retained VLM supervision with foresight and continuous-action objectives.The expert uses flow matching for continuous action chunks, targeting low-latency closed-loop control rather than discrete FAST-token generation.
- Stage 2: Foresight and Action Generation: Learnable foresight tokens query current visual-language context and produce conditioning embeddings for a frozen pretrained video-generation model.Video supervision trains the tokens to encode future-relevant information, which is reused to condition continuous action generation.
- Stage 2: Foresight and Action Generation: Gradients from video supervision update the foresight tokens and upstream unified-expert layers while the WAN video-generation parameters remain frozen.This trains the conditioning pathway without updating the frozen denoising transformer.
4. Data Recipe
InternVLA-A1.5 trains on complementary robot and multimodal streams: robot episodes supervise action and foresight, while multimodal samples preserve semantic and spatial-grounding abilities.
- Complementary data streams: The training recipe combines a robot corpus for action and future-observation supervision with a multimodal corpus for semantic and spatial-grounding preservation.The two streams mirror the model’s action/foresight and VLM-maintenance objectives.
- Robot corpus: 1.2M episodes and 861M frames come from six robot sources, including one synthetic source and five real-world sources unified into one action space.The real-world sources add embodiment and viewpoint diversity and help bridge the sim-to-real gap.
- Episode supervision: Each robot episode supplies continuous action chunks, future frames for latent foresight, and FAST-tokenized actions for discrete VLM targets.The same episode therefore contributes supervision to all three training signals.
- Multimodal corpus: About 3M multimodal samples span general QA, Box QA, Point QA, and Trajectory QA for broad vision-language ability and robotics-oriented grounding.General QA covers captioning, VQA, OCR, and knowledge grounding; the other categories target detection and localization.
- Sampling strategy: Two-level grouped sampling counters source imbalance by sampling within groups using (#frames)^γ and manually refining inter-group weights.The scheme up-weights smaller real-world sources and down-weights the dominant synthetic source.
- Sampling strategy: A fixed 0.15:0.85 robot-to-multimodal sampling ratio makes multimodal data the bulk of each batch while retaining robot-stream learning for action and foresight.This ratio reinforces semantic and grounding ability alongside embodied objectives.
5. Experiments
InternVLA-A1.5 is evaluated across six simulation benchmarks and four real-world tasks, including held-out instruction bindings and a long-horizon chemistry procedure. It achieves strong benchmark performance, compositional generalization, faster convergence, and evidence that foresight supervision transfers pretrained dynamics knowledge.
- Evaluation setup: Six simulation benchmarks span single-arm, bimanual, mobile, real-to-sim, and zero-shot generalization settings.LIBERO, RoboTwin, EBench, and SimplerEnv cover varied manipulation settings, while LIBERO-Plus and DOMINO test zero-shot distribution shifts.
- Evaluation setup: Four real-world tasks test tube instruction following under held-out bindings and long-horizon metal-organic framework preparation.Sort Tubes, Insert Tubes, and Move Tubes vary the target type; MOF evaluates a multi-step chemistry procedure.
- Real-world results: 76.4% success on MOF exceeds π0.5’s 29.3%, while InternVLA-A1.5 leads on Insert Tubes and Move Tubes and remains slightly behind π0.5 on Sort Tubes, 75.9 vs. 77.8.The reported gap is attributed to explicit subtask prediction and learned dynamics priors supporting extended sequential execution.
- Compositional generalization: InternVLA-A1.5 achieves the best success rate on all three held-out instruction-binding tasks, showing robustness to compositional shifts beyond replaying demonstrated bindings.The authors caution that seen-to-held-out differences partly reflect target-specific physical difficulty.
- Simulation results: 80.8% average success on SimplerEnv, 98.9% on LIBERO, 84.8% on LIBERO-Plus, and 93.2% on RoboTwin establish leading or highly competitive simulation results.InternVLA-A1.5 also achieves 27.7% zero-shot SR on DOMINO, improving to 29.3% after fine-tuning, and leads all EBench splits.
- Foresight analysis: Removing video supervision or learnable foresight tokens degrades performance across benchmarks, with larger video-supervision drops in zero-shot settings.The ablation attributes this to improved visual invariance, future-state awareness, motion consistency, and transfer of structured dynamics knowledge.
- Training efficiency: InternVLA-A1.5 converges fastest and reaches the lowest final training loss, entering a low-loss regime substantially earlier than π0.5 and InternVLA-A1.The reported optimization trajectory is also more stable throughout training.
- World-model exploitation: Foresight-conditioned rollouts remain temporally coherent and physically plausible in zero-shot settings, modeling articulated motion and downstream effects such as liquid-level changes.These qualitative results indicate that foresight embeddings capture control-relevant dynamics and causal scene evolution beyond pure kinematics.
6. Conclusion and Limitations
InternVLA-A1.5 unifies vision-language understanding, visual foresight, and action through a native VLM backbone and lightweight expert, using latent foresight tokens supervised by a frozen video generator during training. Experiments show strong generalization and long-horizon performance while retaining real-time inference, but foresight remains short-horizon and depends on generic frozen video priors.
- InternVLA-A1.5 integrates understanding, visual foresight, and action on a native VLM backbone with a lightweight unified expert.The design avoids synthesizing future frames by reading task-relevant future information through latent foresight tokens.
- Experiments across six simulation benchmarks and four real-world tasks show best or highly competitive performance, clearest gains in zero-shot generalization and long-horizon execution, and real-time inference.
- Latent foresight tokens encode task-relevant future information under frozen video-model supervision, and the video generator is discarded at deployment.This lets the policy query pretrained world-evolution knowledge without learning pixel-level future-video generation.
- Limitations: Foresight supervision covers only one action chunk, limiting the policy to local dynamics priors rather than long-horizon imagination or explicit world-model planning.
- Limitations: The frozen generic video generator bounds inherited dynamics priors by how well its pretraining covers embodied scenes.
A. Contributors
The paper lists its authors, including core contributors marked with an asterisk and the corresponding author marked with a dagger in the author line.
- The author list begins with Haoxiang Ma, Junhao Cai, Xiaoxu Xu, and Hao Li.
- The author list includes Hongrui Zhu, Zherui Qiu, Zhaxizhuoma, Yuqiang Yang, and Jiaqi Peng among its contributors.
- The author list concludes with Chunhua Shen, Weinan Zhang, and other named contributors, with no role annotations included in this passage.
A.1. Real-world Task Details
The real-world evaluation tests compositional instruction following through held-out object bindings and long-horizon execution through a 13-subtask MOF procedure.
- Held-out instruction bindings: The test-tube sorting task evaluates reversed arm-to-color bindings that are held out until testing.
- Held-out instruction bindings: Insert Tubes withholds blue-at-hole-2 and orange-at-hole-4 demonstrations, requiring instruction grounding rather than memorized insertion.The task uses 15 trials per binding.
- Held-out instruction bindings: The Move Tubes task instructs moving a specified tube from the left rack into a specified hole on the right rack.
- Long-horizon execution: The MOF task targets long-horizon execution across 13 sequential subtasks, from inserting a funnel and pouring liquid to stoppering the flask and switching on the stirrer.
- Long-horizon execution: MOF progress must be tracked across stages with subtle observational cues, and evaluation uses 20 trials with randomized initial object placements.
A.2. Simulation Benchmark Details
The simulation evaluation spans diverse embodiments, simulators, task types, perturbations, and generalization settings across LIBERO, LIBERO-Plus, RoboTwin, DOMINO, EBench, and SimplerEnv.
- Bimanual manipulation: RoboTwin 2.0 evaluates 50 bimanual tasks under clean and domain-randomized settings.
- Dynamic manipulation: DOMINO evaluates dynamic, constrained manipulation with both binary Success Rate and continuous Manipulation Score metrics.
- Dynamic manipulation: DOMINO uses 35 clean Level-1 suites with 100 rollouts per task in zero-shot and fine-tuned settings.
- Mobile manipulation: EBench covers 26 mobile-manipulation tasks across capability and generalization dimensions.
- Cross-embodiment evaluation: SimplerEnv evaluates cross-embodiment generalization and simulation-to-real correlation on four manipulation tasks.