PaperlayerView exact revision on arXiv

Source-linked AI summary

InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulation

Shuai Yang, Hao Li, Bin Wang, Yilun Chen, Yang Tian, Tai Wang, Hanqing Wang, Feng Zhao, Yiyi Liao, Jiangmiao Pang

arXiv:2507.17520v2cs.ROcs.CV

TL;DR

Existing VLAs struggle to preserve multimodal reasoning while acquiring generalizable manipulation skills. InstructVLA addresses this gap with VLA-IT, staged training, and a benchmark for instruction generalization. It reports leading performance across manipulation, multimodal understanding, and real-world deployment, including improvements over SpatialVLA and OpenVLA-based baselines.

  • Problem

    Existing VLA models can sacrifice multimodal reasoning or remain limited in reasoning-guided manipulation, motivating methods that preserve VLM capabilities while learning manipulation.

  • Method

    InstructVLA uses VLA-IT, MoE adaptation, a 650K-sample annotated dataset, and two-stage training to combine multimodal reasoning with latent action generation.

  • Results

    InstructVLA achieves leading performance across manipulation tasks, multimodal benchmarks, and real-world deployments, including 33% improvement over SpatialVLA and 96% over fine-tuned OpenVLA.

  • Takeaways & Limitations

    The results support using embodied multimodal reasoning and instruction tuning to improve generalist and steerable robot manipulation.

  • Takeaways & Limitations

    Current tasks are largely limited to basic primitives such as open/close and pick/place, making extension to more dexterous skills necessary.

Abstract

from arXiv · show

To operate effectively in the real world, robots should integrate multimodal reasoning with precise action generation. However, existing vision-language-action (VLA) models often sacrifice one for the other, narrow their abilities to task-specific manipulation data, and suffer catastrophic forgetting of pre-trained vision-language capabilities. To bridge this gap, we introduce InstructVLA, an end-to-end VLA model that preserves the flexible reasoning of large vision-language models (VLMs) while delivering leading manipulation performance with the help of embodied reasoning. InstructVLA introduces a novel training paradigm, Vision-Language-Action Instruction Tuning (VLA-IT), which employs multimodal training with mixture-of-experts adaptation to jointly optimize embodied reasoning and action generation on both standard VLM corpora and a curated 650K-sample VLA-IT dataset. On in-domain SimplerEnv tasks, InstructVLA achieves 33% improvement over SpatialVLA. To evaluate generalization, we introduce SimplerEnv-Instruct, an 80-task benchmark requiring closed-loop control and high-level instruction understanding, where it outperforms a fine-tuned OpenVLA by 96% and an action expert aided by GPT-4o by 29%. Additionally, InstructVLA surpasses baseline VLMs on multimodal tasks and exhibits inference-time scaling by leveraging textual reasoning to boost manipulation performance in both simulated and real-world settings. These results demonstrate InstructVLA's potential for bridging intuitive and steerable human-robot interaction with efficient policy learning.

1 INTRODUCTION

InstructVLA addresses the challenge of preserving multimodal reasoning while learning precise manipulation. It introduces VLA-IT, a unified training and evaluation framework for instruction-following VLAs.

  • Existing VLAs adapt pretrained VLMs for manipulation, but reasoning-guided manipulation beyond initialization remains underexplored.
  • InstructVLA extends pretrained VLMs for accurate action generation while preserving strong multimodal understanding.
  • VLA-IT treats language-conditioned action generation as an integral component of instruction following and is supported by 650K annotated human-robot interactions.
  • SimplerEnv-Instruct provides 80 zero-shot tasks testing closed-loop manipulation, high-level instruction reasoning, and task decomposition.
  • 33% improvement over SpatialVLA is reported for closed-loop manipulation, while SimplerEnv-Instruct performance exceeds fine-tuned OpenVLA by 96%.
  • The model is reported to achieve leading performance across manipulation tasks, multimodal benchmarks, and real-world deployments.

2 RELATED WORKS

Prior work combines vision, language, and action through unified modeling or embodied reasoning, but faces limitations in reasoning complexity, scalability, and control efficiency. InstructVLA combines multimodal reasoning, latent action generation, and action decoding through MoE adaptation and staged training.

  • VLA research integrates perception, language, and manipulation using autoregressive action-token models and other unified architectures.
  • Existing reasoning approaches include intermediate representations, future-video prediction, and unified generative formulations, but can depend on rigid heuristics or incur computational overhead.
  • InstructVLA: InstructVLA jointly generates language and actions while dynamically switching between reasoning and execution modes.
  • Architecture: The VLM produces textual reasoning and latent action representations, while an action expert decodes final actions from image features and VLM-derived intentions.
  • Architecture: MoE adaptation uses LoRA experts to integrate multimodal reasoning with language-steered latent action prediction while preserving pretrained capabilities.
  • Training: Training separates action pretraining from VLA instruction tuning to align latent action embeddings before integrating complex instruction reasoning with manipulation.

4 VLA DATASET AND BENCHMARK

The paper develops annotated data and SimplerEnv-Instruct to evaluate instruction generalization and contextual reasoning in VLAs. The benchmark combines linguistic and visual diversity with task decomposition in a practical zero-shot evaluation suite.

  • Dataset: The VLA-IT data includes hierarchical language annotations, language motion, and instruction-tuning examples for reasoning transfer.
  • Dataset: GPT-4o annotates episodes using three frames and corresponding ground-truth instructions, which support annotation accuracy in embodied tasks.
  • Benchmark: SimplerEnv-Instruct evaluates zero-shot VLA instruction following and reasoning beyond atomic actions or low-level control.
  • Benchmark: The benchmark contains 80 tasks and 1.1K trials with novel objects and instructions held out from the original data.
  • Benchmark: Its task aggregation suite tests varied verbs, multilingual expressions, object references, rephrasings, and out-of-distribution objects.
  • Benchmark: Its situated-reasoning suite tests contextual interpretation and decomposition of indirect commands into actionable sub-goals.
  • Benchmark: The benchmark is designed to provide affordable, diagnostic, human-interpretable, and standardized evaluation of task generalization and reasoning.

5 EXPERIMENT

Experiments evaluate InstructVLA across multimodal benchmarks, simulated manipulation, real-world robots, and targeted ablations. Results show strong manipulation and instruction-following performance, with gains linked to multimodal training, language motion, richer visual perception, and data diversity.

  • Main Results: InstructVLA (expert) improves over SpatialVLA by 33.3% on SimplerEnv, while its generalist version improves over OpenVLA with GPT-4o by 31.7% on SimplerEnv-Instruct.The generalist model also maintains strong performance on SimplerEnv atomic instructions.
  • Real-World Experiments: In real-world experiments, InstructVLA improves over OpenVLA by 23.3% on direct instructions and by 41.7% few-shot and 46.7% zero-shot on reasoning tasks.Reasoning tasks include celebrity recognition, OCR, and tool-use inference.
  • Real-World Experiments: InstructVLA achieves a 2.5× improvement over π0 on reasoning and math tasks, while π0 performs close to random guessing.The experiments include few-shot Franka Research 3 tasks and zero-shot WidowX-250 Arm tasks.
  • Multimodal and Manipulation Co-Training: The proposed architecture and two-stage training achieve a 12.5% improvement over Magma on SimplerEnv, outperforming alternative co-training and adaptation strategies.Removing MoE preserves multimodal performance but reduces manipulation capability; full finetuning performs suboptimally on both objectives.
  • Ablation Studies: Language motion supervision improves overall success rate by 9.3%, while removing the DINOv2-based action-expert encoder causes a 50.0% drop and FiLM adds 15.3%.These ablations test language supervision and richer action-expert visual representations.
  • Multimodal Ability Transfers to Manipulation: Adding QA and captioning improves InstructVLA generalization by 10.8%, whereas OpenVLA benefits mainly from instruction diversity and shows limited situated-reasoning gains.The findings connect multimodal data diversity with reasoning-guided manipulation performance.

6 CONCLUSION

The paper concludes that InstructVLA unifies multimodal reasoning and action generation through a data and training pipeline designed for generalizable manipulation. Additional analyses examine scaling, reasoning gains, inference acceleration, and dataset differences, while broader impacts and limitations are discussed separately.

  • Conclusion: InstructVLA integrates multimodal reasoning and action generation in a unified VLA model.The conclusion frames the model as supporting more generalizable, interpretable, and interactive robots.
  • Conclusion: The proposed data and training pipeline achieves leading performance across manipulation tasks, multimodal benchmarks, and real-world deployments.The conclusion presents this as the paper’s overall validation result.
  • Limitations and Future Directions: Section I discusses InstructVLA’s broader impacts, limitations, and future directions.The supplied conclusion material points to a dedicated discussion rather than specifying a particular limitation.
  • Further Analysis: Further analysis covers MoE and latent-action scaling, reasoning gains in manipulation, zero-shot dual-frequency generation, and dataset-scale comparisons.These analyses are used to examine model design, manipulation reasoning, inference speed, and study-scale differences.

A.1.1 EXTRA MODEL DESIGN ANALYSIS

The extra model-design analysis examines how MoE routing and latent-action tokens coordinate multimodal reasoning with manipulation. It reports expert-specific activation patterns, a trade-off in latent-token quantity, and generally improved situated reasoning when reasoning is enabled.

  • Design Components: MoE and latent action are identified as key design components for interleaving multimodal reasoning and manipulation decision-making.The analysis uses activation visualizations and scaling curves to study these components.
  • MoE Gating: Visual information activates both language and action adapters, indicating roles for semantic understanding and manipulation decision-making.System prompts primarily use the language adapter, reflecting its connection to pretraining.
  • MoE Gating: During language generation, the action expert activates for manipulation planning and attends more strongly to nouns and verbs in generated tokens.This pattern is presented as evidence of the action expert’s role in instruction following.
  • MoE Gating: During latent-action generation, stronger action-expert activation accompanies more concentrated action generation, while the language expert plays a less prominent role.The activation comparison is described for multimodal reasoning settings.
  • Latent Action: A latent-action setting of 64 tokens balances behavioral diversity and training efficiency; fewer tokens limit diversity, while more tokens reduce efficiency.The analysis varies token counts from 16 to 128.
  • Reasoning Evaluation: Activating reasoning generally improves performance across 30 situated reasoning tasks spanning subtasks, commonsense reasoning, and tool use.The tasks are grouped by subtask identification, broad world knowledge, and tool-related reasoning.

A.1.2 EXTRA REASONING-MANIPULATION ANALYSIS

The analysis examines how VLA-IT and multimodal reasoning affect instruction following, manipulation, cross-embodiment transfer, and generalization. Results show reasoning benefits complex tasks, while data and training choices shape both capability retention and performance.

  • Scaling and reasoning: InstructVLA-expert shows scaling ability for complex instruction understanding and test-time thinking after stage-2 VLA-IT training.The model does not outperform OpenVLA(OXE) on Situated Reasoning, which benefits from direct full fine-tuning of the VLM backbone.
  • Embodied reasoning: Test-time textual reasoning improves manipulation, particularly on situated reasoning tasks, while robot state provides limited gains when instruction following is required.State information helps more when no instruction response is required.
  • Fine-grained reasoning gains: Reasoning produces the clearest gains for commonsense tool use and articulated-object interaction, where instructions leave actions and objects implicit.The robot may need to infer whether a drawer is open and identify the appropriate cleaning tool.
  • Cross-embodiment understanding: Adding Bridge data supports cross-embodiment evaluation alongside Fractal and Bridge VLA-IT settings, testing whether instruction tuning transfers across embodiments.The ablation compares training without VLA-IT data, Bridge-only data, and data from both environments.
  • Inference efficiency: Performance remains stable at a 1:2 VLM:expert execution ratio but degrades at higher ratios, indicating that latent actions can guide the expert without frequent VLM queries.Dual-frequency inference decouples VLM and action-expert execution frequencies.
  • Capability retention and generalization: Multimodal training preserves or improves capabilities beyond manipulation, including OCR-related generalization, while action-only fine-tuning is associated with unchanged situated reasoning in OpenVLA.InstructVLA improves Written-out Numbers, Matchstick Numbers, and Hard Case 1; OpenVLA’s performance remains unchanged on situated reasoning, likely from catastrophic forgetting.

B.2 MULTI-STAGE TRAINING

The paper combines multimodal instruction understanding with manipulation training through language motion representations and diverse instruction categories. It also documents failure modes involving visual depth estimation, real-to-sim mismatch, and imperfect instruction parsing.

  • Reasoning Cases: InstructVLA uses multimodal reasoning to interpret visual concepts that OpenVLA mishandles, including shapes, fruit properties, and negation.The reported examples link these interpretation errors to failed or unsafe grasp attempts.
  • Failure Cases: Third-person-only views and real-to-sim differences limit depth estimation and can cause inaccurate grasps or out-of-distribution robot states.The authors suggest depth and robot-state inputs as future additions, while observing that language responses transfer more readily than action execution.
  • GPT4o as the Auxiliary System 2: GPT-4o is used as an initial instruction parser, but it frequently fails at physical grounding, scene coherence, and instruction interpretation across 80 cases.Its prompt asks for a concise next-step instruction from the first image and the original command.
  • Language Motion Pre-training Data: Language motion distills low-level end-effector trajectories into discrete, natural-language motion primitives for training.Overlapping state windows are summarized by position, rotation, and gripper displacements, then quantized into six-dimensional motion codes and verbalized.
  • Prompting: The instruction data covers command rewriting, spatial identification, tool-use understanding, quantitative identification, and other task types.The categories include complex object references, novel action references, negative specifications, subtasks, situated reasoning, and direct instructions.

D.4 GROUND TRUTH INSTRUCTION FOR DATA ANNOTATION

The annotation study evaluates GPT-4o without ground-truth instructions and finds substantial errors in interpreting robot behavior. The authors associate the gap with weak temporal grounding and low-quality manipulation images.

  • Error Analysis: GPT-4o produces hallucinated object identities and incorrect instructions when interpreting robot trajectories from images.One representative case mistakes the robotic arm for a bread roll, leading to an incorrect caption and instruction.
  • Error Analysis: GPT-4o can reverse the temporal order of actions, producing inaccurate annotations.The error is illustrated in the second representative case and attributed to missing temporal grounding.
  • Discussion: The authors attribute the performance gap to insufficient temporal grounding and low visual quality in manipulation datasets.Human instructions encode temporal links by connecting context, target objects, and robot actions.
  • Annotation Evaluation: GPT-4o’s annotation performance drops significantly when ground-truth instructions are withheld.The study samples 100 examples and manually scores generated annotations for correctness.

E BENCHMARK DETAILS

The benchmark evaluates instruction generalization across embodied task types, while the model combines VLM responses, latent action planning, MoE adaptation, and a flow-based action expert. Its evaluation also measures inference speed and data settings.

  • Qualitative Results: The benchmark examples show zero-shot interpretation of multilingual instructions, novel objects, OCR, and situated changes in the environment.The paper visualizes task aggregation and situated reasoning responses across selected scenes.
  • Embodied VLM: The unified VLM produces textual responses and latent action representations for language-steered manipulation.Learnable action queries attend to VLM hidden states to extract task-relevant latent actions.
  • MoE Adaptation: MoE adaptation reweights two LoRA experts with a scalar gating coefficient while preserving the frozen VLM backbone.The action adapter is pretrained first, then the language adapter and scalar head are jointly trained in the second stage.
  • Action Expert: The flow-based action expert generates action chunks from visual features, latent actions, noisy action embeddings, and optional proprioception.Training uses flow matching, while inference uses forward Euler integration from Gaussian noise with 10 denoising steps.
  • Inference Evaluation: Inference is compared across Thinking, Action Only, and Latent Action Caching settings without action chunking.Thinking enables autoregressive VLM generation every 20 action-expert steps, while cached latent actions are generated every two expert steps.

F.6 EXPERIMENTS COMPUTE RESOURCES

The paper reports substantial GPU requirements for action pretraining, VLA instruction tuning, and evaluation, alongside reduced-resource reproduction. Qualitative examples illustrate multimodal and embodied reasoning capabilities.

  • Training and Evaluation Compute: Action pretraining takes approximately 27 hours on 64 A100 GPUs, while VLA-IT takes about 12 hours on the same configuration.Simulation evaluation uses 8 A100 GPUs; real-world training uses 32 A100 GPUs for four hours and deployment uses one A100 GPU.
  • Reduced-Resource Training: Pretraining results are reproduced with 8 A800 GPUs in 2.5 days.This experiment assesses the minimum training resources reported by the paper.
  • Qualitative Reasoning: The model demonstrates visual inference, scene-text recognition, object-color grounding, high-level goal interpretation, action prediction, and task-completion verification.The examples span multimodal and embodied commonsense reasoning scenarios.

H REAL-WORLD EXPERIMENTS SETUP AND ANALYSIS

The real-world evaluation uses few-shot clustered classification and spatial-action settings, alongside zero-shot tests of semantic grounding, lighting robustness, and novel environments.

  • The few-shot data include clustered-object classification and spatial-action tasks, with 70 complete episodes and 677 pick-and-place actions reported for the first setting.
  • Zero-shot celebrity grounding tests whether the model can use object and celebrity recognition to place the instructed blue cube correctly.
  • Extreme-lighting evaluation examines robustness supported by stable visual features from DINO and SigLIP.
  • Zero-shot Bridge kitchen evaluation uses augmented backgrounds and unfamiliar objects to test novel-instruction following.
  • Real-world demonstrations use a third-person Intel D435i camera for Franka few-shot and WidowX zero-shot settings.

I.1 LIMITATION

The paper identifies limitations in current manipulation-task coverage and highlights broader concerns around dataset bias and real-world safety. It proposes richer sensory inputs, synthetic data, and wider evaluation as directions for improving reliability and generalization.

  • Current manipulation tasks are limited to basic primitives such as open/close and pick/place.
  • Dataset bias and safety require careful attention when deploying large pretrained models in the real world.
  • Additional depth and tactile modalities are planned to enhance safety and reliability during physical interactions.
  • Synthetic datasets and broader environmental evaluation are proposed to reduce reliance on real-world data and assess generalization.
Loading 2507.17520v2…