PaperlayerView exact revision on arXiv

Source-linked AI summary

Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodiments

Qiuyue Wang, Mingsheng Li, Jian Guan, Jinhui Ye, Sicheng Xie, Yitao Liu, Junhao Chen, Zhixuan Liang, Jie Zhang, Xintong Hu, Xuhong Huang, Pei Lin, Junyang Lin, Dayiheng Liu, Shuai Bai, Jingren Zhou, Jiazhao Zhang, Haoqi Yuan, Gengze Zhou, Hang Yin, Ye Wang, Yiyang Huang, Zixing Lei, Wujian Peng, Delin Chen, Yingming Zheng, Jingyang Fan, Xianwei Zhuang, Xin Zhou, Haoyang Li, Anzhe Chen, Tong Zhang, Xuejing Liu, Yuchong Sun, Ruizhe Chen, Zhaohai Li, Chenxu Lü, Zhibo Yang, Tao Yu, Xionghui Chen

arXiv:2605.30280v2cs.ROcs.AIcs.CL

TL;DR

Embodied models often specialize in narrow task families, limiting evidence for one model spanning tasks, environments, and robot embodiments. Qwen-VLA unifies these settings through shared action-and-trajectory prediction and achieves strong multi-task performance, including 76.9% average OOD success in real-world ALOHA experiments.

  • Problem

    Existing embodied systems remain specialized to narrow task families despite heterogeneous problems sharing a common computational structure.

  • Method

    Qwen-VLA unifies manipulation, navigation, egocentric action modeling, and trajectory prediction through shared action-and-trajectory prediction, embodiment-aware prompts, and a DiT-based action decoder.

  • Results

    76.9% average OOD success was achieved in real-world ALOHA experiments, alongside consistent multi-task performance across manipulation, navigation, and trajectory-centric benchmarks.

  • Takeaways & Limitations

    The results support treating manipulation, navigation, and trajectory-centric embodied tasks as different manifestations of a shared action-and-trajectory prediction problem.

  • Takeaways & Limitations

    Current evaluations are largely short-horizon and benchmark-driven, leaving long-duration, failure-prone real-world deployment an open challenge.

Abstract

from arXiv · show

Embodied intelligence is often studied through specialized models for individual tasks such as manipulation or navigation, resulting in fragmented capabilities and limited generalization across tasks, environments, and robot embodiments. In this work, we study whether heterogeneous embodied decision-making problems can be unified within a single vision-language-action model. We present Qwen-VLA, a unified embodied foundation model that extends Qwen's vision-language modeling stack from perception, understanding, and reasoning to continuous action and trajectory generation through a DiT-based action decoder. Qwen-VLA is trained with a large-scale joint pretraining recipe over diverse data sources, including robotics manipulation trajectories, human egocentric demonstrations, synthetic simulation data, vision-and-language navigation data, trajectory-centric supervision, and auxiliary vision-language data. To support multiple robot platforms, we introduce embodiment-aware prompt conditioning, where robot-specific textual descriptions specify the current embodiment and control convention. We further cast manipulation, navigation, and trajectory prediction into a unified action-and-trajectory prediction framework, enabling transferable visual grounding, spatial reasoning, and continuous action generation across robot morphologies, task families, and environments. Experiments on manipulation, navigation, and trajectory-centric benchmarks show consistent multi-task performance and out-of-distribution generalization under variations in scene layout, background, lighting, object configuration, and robot embodiment. Qwen-VLA-Instruct achieves 97.9% on LIBERO, 73.7% on Simpler-WidowX, 86.1%/87.2% on RoboTwin-Easy/Hard, 69.0% OSR on R2R, 59.6% SR on RxR, 76.9% average OOD success in real-world ALOHA experiments, and 26.6% zero-shot success on DOMINO dynamic manipulation.

1 Introduction

Qwen-VLA addresses heterogeneous embodied decision-making by unifying manipulation, navigation, and egocentric action modeling within a shared action-and-trajectory framework. It combines a Qwen3.5-4B multimodal backbone, a DiT-based policy head, heterogeneous joint pretraining, embodiment-aware conditioning, and staged training for cross-task and cross-embodiment generalization.

  • Motivation: Embodied decision-making spans diverse outputs, including end-effector poses, joint positions, gripper states, waypoints, and discrete movement decisions.These heterogeneous requirements motivate a unified modeling approach across manipulation and navigation.
  • Model: Qwen-VLA unifies manipulation, navigation, and egocentric action modeling in a shared action-and-trajectory space across robot platforms and task families.It is built on the Qwen3.5-4B vision-language backbone with a DiT-based flow-matching policy head.
  • Training: Joint pretraining combines manipulation, egocentric, simulation, navigation, spatial-grounding, autonomous-driving, action-caption, and general vision-language data.The mixture is designed to cover low-level motor priors and high-level semantic reasoning.
  • Cross-embodiment learning: Embodiment-aware prompts specify the robot platform, arm configuration, control convention, frequency, and prediction horizon, providing the interface for embodiment-specific control semantics.Combined with a unified action representation, this enables one action decoder to handle multiple embodiments.
  • Evaluation: Evaluation covers manipulation, vision-and-language navigation, out-of-distribution generalization, and cross-embodiment transfer under changes in layout, lighting, background, instructions, sensor noise, and embodiment.The evaluation measures both in-domain task success and generalization under environmental and robotic variation.

2 Unified Embodied Model · 2.1 Problem Formulation · 2.2 Model Architecture

Qwen-VLA unifies manipulation, navigation, trajectory prediction, and egocentric action modeling through a conditional action-and-trajectory prediction framework. Its architecture combines a vision-language backbone for reasoning with a DiT-style flow-matching action expert for precise embodied generation.

  • 2.1 Problem Formulation: The unified model covers robot manipulation, vision-and-language navigation, trajectory prediction, and human egocentric action modeling.These tasks share language grounding in visual observations, spatial-temporal reasoning, and future-action prediction.
  • 2.1 Problem Formulation: At time step t, prediction conditions on visual context, a language instruction, an embodiment description, and an optional task identifier.Visual context can include image frames, videos, or history windows; embodiment prompts specify the robot platform and control convention.
  • 2.1 Problem Formulation: All tasks represent their target sequence in a unified action-and-trajectory space.The target sequence is task-dependent while retaining a common representation.
  • 2.1 Problem Formulation: Manipulation uses future robot actions, navigation uses decisions or waypoints, and trajectory-centric tasks use continuous future spatial coordinates.Egocentric data represents human body or hand motion in structured pose spaces such as MANO.
  • 2.2 Model Architecture: The architecture combines a vision-language backbone for high-level understanding and reasoning with a flow-matching action expert for fine-grained generation.This division separates semantic reasoning from precise action synthesis.
  • 2.2 Model Architecture: The vision-language backbone adopts Qwen3.5, which directly interleaves ViT visual tokens with text tokens for unified image, video, and language processing.Its hybrid attention design combines gated linear attention with periodically inserted grouped-query softmax attention.
  • 2.2 Model Architecture: A single-stream DiT-style flow-matching policy predicts precise actions across robot and human embodied data.It concatenates VLM hidden states with a noisy action chunk and applies joint self-attention with AdaLN timestep conditioning and multi-section RoPE.

2.3 Embodiment-aware Prompt Conditioning · 2.4 Unified Action and Trajectory Representation

Qwen-VLA supports multiple robot embodiments through robot-specific textual prompts that specify platform and control conventions. It unifies manipulation and navigation outputs with a shared tensor and masking interface while preserving native action semantics and conditioning generation on embodiment-aware prompts.

  • 2.3 Embodiment-aware Prompt Conditioning: Each training example is prefixed with a robot-specific prompt describing the platform, arm configuration, and control convention.The prompt conditions one shared model across multiple robot embodiments.
  • 2.3 Embodiment-aware Prompt Conditioning: The prompt specifies the robot tag, single- or dual-arm configuration, optional waist or mobile base, control frequency, action chunk size, and task instruction.Robot-specific modifiers are set per embodiment, while FPS and chunk_size follow the dataset’s control frequency and prediction horizon.
  • 2.3 Embodiment-aware Prompt Conditioning: Representative robot platforms in the pretraining corpus are summarized with their arm configurations and action types.The prompt fields and platform metadata reflect the heterogeneous embodiments covered during pretraining.
  • 2.4 Unified Action and Trajectory Representation: The unified interface shares tensor dimensions and masking but preserves each embodiment’s native control convention through prompts and dataset-specific normalization.Embodiments are not forced into a single physical action semantic space.
  • 2.4 Unified Action and Trajectory Representation: Each sample produces a target tensor Y ∈R^H×K with fixed prediction horizon H and shared channel dimension K.Manipulation signals include end-effector, joint, gripper, and dexterous-hand controls, while navigation uses (∆x, ∆y, ∆θ) per waypoint.
  • 2.4 Unified Action and Trajectory Representation: Control modes occupy their leading c ≤ K channels, zero-pad the remainder, and use a binary mask to mark valid channels and task time steps.A single DiT parameter set handles all control modes without embodiment-specific output heads.
  • 2.4 Unified Action and Trajectory Representation: Embodiment-aware and navigation prompts are processed by the VLM backbone and concatenated with noisy action chunks before DiT decoding.This conditions the action expert on the robot platform, control frequency, prediction horizon, navigation convention, or waypoint horizon.

2.5 Training Objectives

The model is trained end-to-end with a weighted combination of continuous action-generation and vision-language objectives. Flow matching supervises action trajectories, while next-token prediction preserves multimodal grounding and reasoning capabilities.

  • End-to-end training combines continuous action generation and vision-language understanding through a weighted sum of two objectives.
  • Flow matching supervises continuous control targets from manipulation, VLN waypoints, and action-aligned human egocentric data.The action expert predicts a conditional velocity field along linear interpolants between clean targets and Gaussian noise.
  • Per-channel, per-step masked averaging excludes padding and gives each active control dimension equal gradient contribution across embodiments.At inference, action chunks are generated with a few Euler integration steps from τ = 1 to τ = 0.
  • Next-token prediction on diverse vision-language data stabilizes language grounding and prevents catastrophic forgetting during embodied co-training.Sources include auxiliary vision-language data, embodied action captions, autonomous-driving VQA, and general vision-language pretraining corpora.

(I) T2A · (IV) RL · (II) CPT & (III) SFT

Qwen-VLA uses a four-stage training recipe that progresses from text-only action-prior learning to visual grounding, multi-task and real-robot supervised fine-tuning, and reward-based closed-loop optimization. Joint batches combine manipulation, VLN trajectory, and vision-language signals while balancing action and vision-language objectives.

  • (I) T2A: Stage I (T2A) trains the DiT action decoder from text alone to reconstruct actions and build a structured action prior without visual input.
  • (II) CPT & (III) SFT: Stage II (CPT) unfreezes both modules to ground the text-derived action prior in visual observations.
  • (II) CPT & (III) SFT: Stage III (SFT) branches into separate multi-task and real-robot training tracks.
  • (IV) RL: Stage IV (RL) optimizes closed-loop task success using environment rewards.
  • (II) CPT & (III) SFT: The joint training objective weights action and vision-language losses to balance their gradient magnitudes.The weights are denoted λact and λvl.
  • (II) CPT & (III) SFT: Each mini-batch mixes all task families according to a fixed sampling ratio, jointly updating the backbone and action expert.The mixed signals include manipulation, VLN trajectory, and vision-language data.

3 Large-Scale Joint Pretraining

Qwen-VLA uses a four-stage training recipe that progressively equips a pretrained VLM and DiT action decoder with language-conditioned action priors, visual grounding, task coverage, and closed-loop success optimization. Its heterogeneous corpus combines real and simulated robotics data with synthetic, navigation, trajectory, and auxiliary vision-language supervision.

  • Stage I: Text-to-action DiT pretraining (T2A): T2A freezes the VLM and trains the DiT on text plus embodiment prompts without images, installing a language-indexed prior over platform-specific action distributions.The decoder learns to reconstruct high-dimensional actions from compact linguistic descriptions without relying on visual shortcuts.
  • Four-stage training recipe: The recipe comprises T2A, CPT, SFT, and RL, with each stage addressing a limitation left by the preceding stage.It is built on a pretrained Qwen3.5 VLM backbone, with SFT branching into parallel tracks and RL producing Qwen-VLA-Instruct.
  • Stage II: Continued pretraining (CPT): CPT unfreezes both modules and mixes simulation with real-robot trajectories to ground actions in visual observations while adapting the backbone to embodied perception.This stage targets the visual-grounding problem that T2A cannot address and provides cross-domain exposure before post-training.
  • Pretraining data mixture: Real and simulated robot manipulation trajectories account for approximately 74.2% of the pretraining mixture across tabletop, mobile, bimanual, and dexterous embodiments.The corpus also includes over 1,000 hours of in-house real-robot data and over 8M synthetic simulation trajectories.
  • Synthetic simulation data: 359,848 full successful synthetic trajectories provide diverse, controllable, automatically verified VLA supervision that complements real-world demonstrations.The data improves generalization across task types, visual conditions, and execution variations.
  • Auxiliary vision-language supervision: Auxiliary supervision includes 2.4% autonomous-driving VQA and 2.5% 2D bounding-box grounding data to reinforce temporal scene understanding and object-level spatial grounding.Spatial grounding supports language-conditioned manipulation by helping localize instruction-relevant objects.

4 Post-Training

Qwen-VLA uses two-stage post-training to convert broad pretrained knowledge into precise closed-loop control: multi-task supervised fine-tuning followed by task-success-driven reinforcement learning. The resulting Qwen-VLA-Instruct jointly supports heterogeneous tasks and transfers through embodiment prompting without platform-specific adaptation.

  • Post-training overview: The two-stage procedure first jointly fine-tunes the VLM backbone and action expert, then applies on-policy reinforcement learning to produce Qwen-VLA-Instruct.SFT covers visual question answering, spatial grounding, manipulation, and navigation with embodiment- and task-balanced sampling; RL optimizes task-success-driven rewards in simulation.
  • Supervised fine-tuning: SFT combines vision-language next-token prediction with flow matching on action tokens, weighting language loss 0.1 and manipulation and navigation action losses 1.0.The weighting preserves language and visual understanding while emphasizing action generation.
  • Reinforcement learning: RL uses PPO with GAE to optimize a clipped policy objective over action chunks, combining the policy surrogate with value-function regression.The state includes visual observation, task instruction, and embodiment prompt; the method uses γ=0.99, λ=0.95, and clipping threshold ϵ=0.2.
  • Reinforcement learning: For the flow-matching decoder, PPO log-probabilities are estimated by converting the deterministic probability-flow ODE into a corresponding stochastic process.This addresses the absence of directly available softmax log-probabilities in flow-matching policies.
  • Reinforcement learning: The RL reward is sparse and simulator-grounded: R=1 when the episode-end goal is achieved and R=0 otherwise, with GAE propagating credit across action chunks.No learned reward model is used; task-completion semantics come directly from the simulator.
  • Out-of-domain generalization: The RL-refined policy transfers across out-of-domain environments, tasks, and embodiments by replacing only the embodiment prompt, without an adaptation head or domain-specific fine-tuning.The prompt describes the physical platform’s arm configuration, control frequency, and action space while the VLM backbone and DiT action decoder remain unchanged.

5 Experiments

The experiments evaluate Qwen-VLA in simulation and real-world settings across robotic manipulation and visual navigation. They compare a broadly pretrained base model with an instruction-fine-tuned variant designed for fine-grained manipulation.

  • Experimental scope: Experiments cover simulation and real-world settings in robotic manipulation and visual navigation.These are identified as the two core embodied AI domains evaluated.
  • Model variants: Qwen-VLA-Base is trained through large-scale pretraining on diverse embodied data.
  • Model variants: Qwen-VLA-Instruct is further fine-tuned with instruction-following data from diverse simulated environments and tasks for fine-grained manipulation.

5.1 Main Results

Qwen-VLA-Instruct delivers strong unified performance across manipulation, real-world manipulation, navigation, trajectory-centric, and dynamic manipulation benchmarks. Its results also show substantial benefits from instruction tuning and large-scale pretraining, including improved out-of-distribution generalization.

  • Manipulation: 97.9% on LIBERO, 73.7% on Simpler-WidowX, and 86.1%/87.2% on RoboTwin-Easy/Hard demonstrate strong manipulation performance from one generalist model.Qwen-VLA-Instruct surpasses most specialist baselines, including StarVLA-OFT at 64.6% on Simpler-WidowX, and reaches 56.7% on RoboCasa-GR1.
  • Manipulation: +7.1% on LIBERO, +16.3% on RoboCasa-GR1, +9.4% on Simpler-WidowX, and +21.8%/+20.8% on RoboTwin-Easy/Hard show consistent gains from instruction tuning.QwenVLA-Base already reaches 90.8% on LIBERO and 64.3% on Simpler-WidowX after large-scale pretraining.
  • Real-world manipulation: 76.9% average OOD success exceeds π0.5 by 35.4 percentage points and Qwen-VLA-alohaw/o pretrain by 40.7 percentage points across five generalization settings.The settings include color, instance, position, background, and instruction generalization.
  • Navigation: 69.0 Oracle Success and 57.5 Success Rate on R2R Val-Unseen, plus 59.6 SR and 47.8 SPL on RxR Val-Unseen, establish leading navigation results.On R2R, Qwen-VLA-Instruct surpasses StreamVLN by 4.8 and 0.6 points in Oracle Success and Success Rate, respectively.
  • Dynamic manipulation: 26.6% SR and 39.5 MS on DOMINO are the benchmark’s best overall results, with zero-shot SR exceeding OpenVLA-OFT and π0.5 by over 19 percentage points.Qwen-VLA-Instruct also surpasses both standard vision-language-action baselines and edges out LingBot-VA in the reported comparison.

5.2 Ablation Studies

Ablations show that T2A benefits from mixed synthetic-real data, full-trajectory prediction, vision-free training, Sigmoid-Normal timestep sampling, and moderate training duration. These choices together establish a stronger language-action prior before visual grounding.

  • Data composition: 71.1% is achieved by mixing ∼20% synthetic with 80% real data, outperforming purely synthetic data at 64.1% and purely real data at 51.0%.The mixed setting also gives a +10.2 pp gain over the no-T2A baseline (60.9%).
  • Sequence prediction mode: 65.4% vs. 60.4% at 10% synthetic data shows full-sequence prediction outperforming chunk prediction by +4.9 pp.At 0% synthetic data, the corresponding gap is +2.9 pp (51.0% vs. 48.2%).
  • Flow-matching timestep distribution: 71.1% results from Sigmoid-Normal p(τ) at T2A combined with Beta p(τ) at SFT, while Beta at both stages reaches 59.4%.Using Beta at T2A drops performance to 65.4% (−5.7 pp), and using Sigmoid-Normal at SFT yields 62.8% (−8.3 pp).
  • Training duration: 71.1% at 2,000 steps peaks over 67.5% at 4,000 steps, 67.2% at 10,000 steps, and 60.4% at 40,000 steps.The results indicate rapid convergence followed by degradation with prolonged training.

6 Conclusion

Qwen-VLA unifies perception, reasoning, and embodied action generation in a single vision-language-action model. It uses a shared action-and-trajectory prediction framework to learn from heterogeneous embodied data across tasks, environments, and robot embodiments.

  • Qwen-VLA extends the Qwen vision-language backbone from perception and reasoning to embodied action generation.
  • A shared action-and-trajectory prediction framework covers manipulation, navigation, egocentric action modeling, and trajectory prediction.
  • The unified model learns from heterogeneous embodied data across tasks, environments, and robot embodiments.

7 Limitations and Future Work

Qwen-VLA remains limited by the scale and diversity of embodied action data and by optimization trade-offs in joint multimodal training. Future work should expand interaction data and physical, temporal, and planning capabilities to improve robust control.

  • Limitations: Embodied action data is smaller and less diverse than vision-language pretraining data, limiting robustness to long-tail objects, environments, embodiments, and contact-rich interactions.The stated data gap constrains generalization across these long-tail conditions.
  • Limitations: Joint training across vision-language understanding, navigation, and action generation introduces optimization trade-offs, with action-oriented training modestly regressing some capabilities.The passage identifies a tension between improved policy learning and regressions in some vision-language capabilities.
  • Future Work: Future work should scale real-world interaction data through autonomous collection, simulation, and sim-to-real transfer, while using egocentric and third-person human video to provide physical priors and temporal abstractions.These data sources are proposed to improve robustness beyond robot demonstrations.
  • Future Work: Future models should incorporate long-horizon planning, episodic memory, world models, richer physical feedback, and large-scale reinforcement learning in simulation and the real world.Force, tactile, and proprioceptive signals are identified as feedback modalities that may support more reliable control.

8 Contributions and Acknowledgments

The paper lists its authors, with Qiuyue Wang, Mingsheng Li, and Jian Guan marked by an asterisk, and Shuai Bai marked by a dagger.

  • The author list begins with Qiuyue Wang, Mingsheng Li, Jian Guan, Jinhui Ye, and Sicheng Xie.
  • Qiuyue Wang, Mingsheng Li, and Jian Guan carry asterisks in the author listing.
  • Shuai Bai carries a dagger symbol in the author listing.
Loading 2605.30280v2…