Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodiments

Introduction

• Existing embodied systems often specialize in narrow task families, embodiments, or settings, while differing action formats and observation conditions complicate transfer.Despite these differences, manipulation, navigation, and egocentric demonstrations share the structure of conditioning on visual observations, language, and embodiment constraints to predict actions or trajectories.Source: §1 · p. 2
• Qwen-VLA uses a Qwen3.5-4B backbone and DiT-based flow-matching decoder to predict manipulation actions, navigation waypoints, and human motions in a shared action-and-trajectory space.This design provides one inference interface rather than separate task-specific output heads or architectures.Source: §1 · pp. 2–3
• Joint pretraining combines robot and human trajectories, synthetic simulation, navigation, spatial grounding, driving VQA, action captions, and general vision-language data.The mixture is intended to provide motor priors and long-horizon instruction following while retaining perception, grounding, and reasoning capabilities.Source: §1 · p. 2
• Embodiment-aware prompts describe each platform, arm configuration, control convention, frequency, and prediction horizon, letting one decoder handle differing action dimensions and temporal horizons.The prompt is the sole interface for embodiment-specific control semantics; the model architecture remains unchanged across embodiments.Source: §1 · p. 2
• A staged recipe first trains the decoder to map text to actions, then adds visual grounding, task-specific fine-tuning, and reinforcement learning for closed-loop success.Text-to-action DiT pretraining teaches the randomly initialized decoder to decompress compact language-and-embodiment prompts into high-dimensional trajectories before visual input is introduced.Source: §1 · p. 3
• Evaluations span manipulation, vision-and-language navigation, out-of-distribution robustness, and cross-embodiment transfer; results show strong multi-task performance and robustness under scene and embodiment shifts.Tested variations include object layout, lighting, background, instructions, sensor noise, and robot embodiment.Source: §1 · p. 3

2.1 Problem Formulation

• The formulation unifies manipulation, vision-and-language navigation, trajectory prediction, and human egocentric action modeling through shared grounding, spatial-temporal reasoning, and future prediction.The tasks differ in output format and evaluation protocol but share this computational structure.Source: §2.1 · p. 3
• At time t, the model conditions on visual context o_t, language instruction x, embodiment description e, and optional task identifier z to predict a target sequence over horizon H.Visual context may comprise image frames, video observations, or history windows; e describes the robot platform and control convention.Source: §2.1 · p. 3
• The target sequence is task-dependent but represented in a unified action-and-trajectory space.The model predicts y_{t:t+H−1} over prediction horizon H.Source: §2.1 · p. 3
• Manipulation targets future robot actions, such as end-effector positions, while navigation targets decisions or waypoints.These task-specific targets are instances of the unified prediction formulation.Source: §2.1 · pp. 3–4
• Trajectory-centric tasks predict continuous spatial trajectories, while egocentric data predicts human body or hand motion in structured pose spaces such as MANO or skeletal joint sequences.The passage cites MANO as Romero et al., 2017.Source: §2.1 · p. 4
• Joint optimization across heterogeneous embodied datasets is intended to facilitate transferable visual grounding, spatial reasoning, and action generation across task families.This transfer is enabled by the unified formulation of the task-dependent targets.Source: §2.1 · p. 4
• The formulation could extend its inputs with episodic memory or persistent state for long-horizon planning and failure recovery, and its outputs with future visual states for action-conditioned world modeling.Co-predicting future visual states would allow the agent to anticipate action consequences.Source: §2.1 · p. 4

2.2 Model Architecture

• The model combines a vision-language backbone for high-level understanding and reasoning with a flow-matching action expert for fine-grained action generation.Source: §2.2 · p. 4
• Qwen3.5 (Team, 2026) provides early vision-language fusion, interleaving spatially merged ViT visual tokens with text for unified image, video, and language processing.Its hybrid attention uses gated linear attention in most layers and periodically interleaved grouped-query softmax attention.Source: §2.2 · p. 4
• The single-stream DiT-style expert concatenates VLM hidden states with noisy action chunks and processes them using joint self-attention, AdaLN timestep conditioning, and multi-section RoPE.The design is intended to specialize in fine-grained action generation while preserving the backbone’s pretrained capabilities.Source: §2.2 · p. 4
• The action expert uses a flow-matching objective and generates action sequences through a few Euler integration steps at inference for low-latency real-time control.Source: §2.2 · p. 4
• 1.15B parameters make up the action expert, including 16 DiT blocks with 70.8M parameters each and action projection MLPs, VLM projection, timestep embedding, and output AdaLN modulation.The 16 blocks total 1.13B parameters; the remaining parameters are distributed across the listed components.Source: §2.2 · p. 4

2.3 Embodiment-aware Prompt Conditioning

• Each training example receives a robot-specific textual prompt describing the platform, arm configuration, and control convention to support multiple embodiments in one shared model.Source: §2.3 · p. 4
• The prompt template specifies the robot tag, arm and optional waist or mobile-base configuration, control frequency, chunk size, and original task instruction.The robot tag and optional modifiers vary by embodiment, while FPS and chunk size reflect the dataset’s control frequency and prediction horizon; Table 2 summarizes representative pretraining platforms, arm configurations, and action types.Source: §2.3 · p. 4

2.4 Unified Action and Trajectory Representation

• Each sample uses a target tensor Y ∈ R^{H×K}, with fixed prediction horizon H and shared channel dimension K, while preserving its native control convention through embodiment prompts and dataset-specific normalization.The framework unifies the tensor interface and masking scheme without forcing embodiments into one physical action semantic space.Source: §2.4 · pp. 4–5
• Manipulation signals include end-effector, joint, gripper, and dexterous-hand controls, while navigation signals encode each waypoint as (∆x, ∆y, ∆θ); both are treated as real-valued sequences over a horizon.The action expert handles both families identically despite their different physical semantics.Source: §2.4 · p. 5
• For a control mode with c ≤ K valid channels, the representation places values in the leading c dimensions, zero-pads the rest, and masks invalid channels and timesteps so one DiT handles all modes.The mask prevents padded entries from influencing the gradient, eliminating the need for embodiment-specific output heads.Source: §2.4 · p. 5
• Prompts specify the robot platform, arm configuration, control frequency, and prediction horizon—or the navigation convention and waypoint horizon—and condition the DiT through VLM hidden states without architectural changes.The hidden states are concatenated with the noisy action chunk as DiT input.Source: §2.4 · p. 5

2.5 Training Objectives

• The entire model is trained end-to-end with a weighted sum of action-generation and vision-language-understanding objectives.Source: §2.5 · p. 5
• Continuous-control samples use conditional flow matching (Lipman et al., 2023), interpolating clean targets with Gaussian noise and training the action expert to predict conditional velocity.This covers manipulation, VLN trajectory waypoints, and action-aligned human egocentric data.Source: §2.5 · p. 5
• Two-level averaging gives each active control channel equal gradient contribution and fully excludes padded positions.At inference, action chunks are generated with a few Euler integration steps from τ = 1 to τ = 0.Source: §2.5 · p. 5
• A standard next-token prediction loss on vision-language and embodied data stabilizes language grounding and prevents catastrophic forgetting of perception and reasoning.Training sources include auxiliary vision-language data, fine-grained embodied action captions, autonomous-driving VQA, and general VL pretraining corpora.Source: §2.5 · p. 5

(I) T2A

• Stage I (T2A) trains the DiT action decoder to reconstruct actions from text alone, building a structured action prior without visual input.This establishes the action prior before visual grounding.Source: Figure 2 · p. 6

(II) CPT & (III) SFT

• Stage II (CPT) unfreezes both modules to ground the action prior in visual observations.Source: Figure 2 · p. 6
• Stage III (SFT) branches into multi-task and real-robot training tracks.Source: Figure 2 · p. 6

(IV) RL

• Stage IV (RL) optimizes closed-loop task success via environment rewards.Source: Figure 2 · p. 6

(II) CPT & (III) SFT

• The overall training loss is a weighted combination, with λact and λvl tuned to balance the gradient magnitudes of the two objectives.Source: (II) CPT & (III) SFT · p. 6
• Each mini-batch mixes task-family samples at a fixed ratio, jointly updating the backbone and action expert with manipulation, VLN trajectory, and vision-language signals.Source: (II) CPT & (III) SFT · p. 6

3.1 Training Recipe

• The four-stage recipe trains a text-to-action prior, continues multimodal pretraining, branches into two supervised fine-tuning tracks, and applies simulation-only reinforcement learning to produce Qwen-VLA-Instruct.T2A freezes the VLM and withholds images; CPT unfreezes both modules on mixed simulated and real-robot data; SFT branches into multi-task and real-world teleoperation tracks; RL uses sparse binary success rewards in SimplerEnv.Source: §3.1 · pp. 6–7

3.2 Pretraining Data

• The pretraining mixture spans five families: robot manipulation, human egocentric demonstrations, synthetic simulation, navigation and trajectory-centric data, and auxiliary vision-language data.The authors describe the mixture as equipping the model with broad embodied perception, spatial reasoning, and action generation capabilities.Source: §3.2 · p. 7

3.2.1 Robotics Manipulation Trajectories

• Robot manipulation trajectories account for approximately 74.2% of pretraining and cover diverse embodiments, while public datasets contribute over 10,000 hours of heterogeneous interaction data.The corpus also includes over 1,000 hours of in-house real-robot data (approximately 20% of the mixture) and over 8M synthetic simulation trajectories (3.7%); embodiment prompts specify platform and control convention, and datasets retain their original action formats.Source: §3.2.1 · p. 8

3.2.2 Egocentric Human Data

• Egocentric human demonstrations make up 6.0% of pretraining and complement robot data with broad manipulation priors from open-world scenes and tasks.Sources include Ego4D and EPIC-KITCHENS subsets, EgoDex, EgoVerse, and Xperience; human action targets combine wrist motion with hand articulation encoded as 10 eigengrasp coefficients per hand, for 32 action dimensions per time step.Source: §3.2.2 · pp. 9–10

3.2.3 Synthetic Simulation Data

• Synthetic simulation supervision combines image-conditioned VLA data with text-only language-action data, providing visual grounding and language-action regularities across tasks and embodiments.The VLA corpus contains 359,848 successful trajectories including subtask segments, with randomized appearance, viewpoints, layouts, and execution factors; text-only data supplies roughly 7.2M planned trajectories across six task templates and six robot configurations for Stage I T2A pretraining.Source: §3.2.3 · pp. 10–11

3.2.4 Navigation data.

• Navigation data makes up 7.5% of pretraining, covering instruction following (4.3%), object searching (2.3%), and target tracking (1.0%) through visual histories, instructions, and trajectories.The tasks support long-horizon mobile behaviors, including landmark-guided movement, exploration to find described objects, and following moving targets while maintaining a safe distance.Source: §3.2.4 · p. 12

3.2.5 Vision-language data.

• Auxiliary vision-language supervision accounts for 8.5% combined and is included to strengthen semantic grounding, fine-grained instruction following, and general visual reasoning.Source: §3.2.5 · p. 12

3.2.5 Vision-language data

• About 48,000 fine-grained video–caption pairs (0.2% of the pretraining mixture) provide episode-level action descriptions reviewed and corrected by human annotators.Captions cover 13 dimensions and are produced in two stages: coarse action extraction followed by frame-dense refinement, with additional camera-view refinement for multi-view datasets.Source: §3.2.5 · pp. 12–13
• A dedicated embodied action-captioning VLM is fine-tuned on the annotations, with a companion QA benchmark and rubric designed to evaluate caption quality.Source: §3.2.5 · p. 13
• Autonomous-driving VQA contributes 2.4% of the mixture, supplying supervision for temporal scene understanding, surround-view spatial reasoning, language-grounded localization, and planning-aware reasoning.Before pretraining, data are standardized into conversational format, including freeform QA, 1000-scale bounding boxes, and frame and view tags.Source: §3.2.5 · p. 13
• 2D bounding-box grounding data (2.5%) reinforces object-level spatial understanding needed to localize instruction-relevant objects for language-conditioned manipulation.Source: §3.2.5 · p. 13
• General vision-language data comprises 3.4% of total tokens and is intended to mitigate catastrophic forgetting while retaining world knowledge and robust language-conditioned embodied perception.The mixture includes captioning, knowledge-oriented VQA and reasoning, OCR and text grounding, instruction following, and common grounding tasks, with video-centric, spatial-relation, and 3D-aware supervision up-weighted.Source: §3.2.5 · p. 13

4 Post-Training

• Post-training specializes Qwen-VLA through multi-task SFT of the VLM backbone and action expert, followed by RL refinement into Qwen-VLA-Instruct.RL optimizes task-success-driven rewards from on-policy simulation rollouts.Source: §4 · p. 14

4.1 Multi-Task Supervised Fine-Tuning

• SFT combines vision-language samples, navigation episodes, and manipulation demonstrations, jointly training language-token prediction and action flow matching.Samples use embodiment-aware prompts; the loss weights are 0.1 for vision-language prediction and 1.0 for both manipulation and navigation action prediction.Source: §4.1 · p. 14

4.2 Reinforcement Learning

• PPO with GAE optimizes policy rollouts against task success, using a clipped objective, γ=0.99, λ=0.95, and clipping threshold ϵ=0.2.A lightweight value head on the VLM backbone estimates values; stop-gradient prevents value-function gradients from flowing into the pretrained backbone.Source: §4.2 · pp. 14–15
• For PPO’s flow-matching decoder, the method converts the probability-flow ODE into an SDE with Gaussian transitions to compute log-probabilities analytically.It stores intermediate denoising states and estimates each action chunk’s log-probability and advantage at chunk level, using one randomly selected denoising step by default.Source: §4.2 · p. 15
• RL uses sparse simulator rewards of 1 for episode-end goal achievement and 0 otherwise, with GAE assigning credit across chunks.Rollouts use 128 parallel environments and collect 8,192 transition chunks per iteration; rollout and evaluation temperatures are 1.0 and 0.6, respectively.Source: §4.2 · p. 15
• The RL-refined policy is designed to transfer zero-shot to new environments, tasks, and embodiments by replacing the embodiment prompt while keeping the VLM backbone and DiT decoder unchanged.The paper attributes this to prior exposure to simulated and real-world visual distributions and RL rewards based on task success rather than visual domain.Source: §4.2 · p. 15

5 Experiments

• Experiments cover simulation and real-world robotic manipulation and visual navigation, evaluating Qwen-VLA-Base and Qwen-VLA-Instruct.Base is pretrained on diverse embodied data; Instruct is further fine-tuned with instruction-following data from diverse simulated environments and tasks.Source: §5 · p. 15

5.1.1 Manipulation Results in Simulation

• Qwen-VLA-Instruct scores 97.9% on LIBERO (on par with the best specialists), 56.7% on RoboCasa-GR1, 73.7% on Simpler-WidowX, and 86.1%/87.2% on RoboTwin-Easy/Hard, surpassing most specialist baselines.Instruction tuning improves performance by +7.1% on LIBERO, +16.3% on RoboCasa-GR1, +9.4% on Simpler-WidowX, +21.8% on RoboTwin-Easy, and +20.8% on RoboTwin-Hard.Source: Table 4 · p. 16, §5.1.1 · p. 16

5.1.2 Manipulation Results in the Real World

• 83.6% average in-domain success: Qwen-VLA-alohaw/ pretrain outperforms strong baselines on most task categories, versus 48.5% for the from-scratch variant.Both variants share the same architecture; the evaluation spans six real-world task categories, from pick-and-place to fine-grained manipulation.Source: §5.1.2 · pp. 17–18, Table 5 · p. 18
• 76.9% average OOD success: Qwen-VLA-alohaw/ pretrain leads across all five generalization settings, exceeding π0.5 by 35.4 percentage points and the from-scratch variant by 40.7 points.It reaches 80.8% under background generalization and 84.6% under instruction generalization; the results indicate pretraining supports robustness to unseen visual and instruction conditions.Source: §5.1.2 · p. 18, Table 6 · p. 19

5.1.3 Navigation Results

• On R2R Val-Unseen, Qwen-VLA-Instruct achieves 69.0 Oracle Success rate and 57.5 Success Rate; on RxR Val-Unseen, it leads with 59.6 SR and 47.8 SPL.The evaluation uses the default VLN-CE setting and a sliding-window waypoint action compatible with Qwen-VLA's predicted trajectory.Source: §5.1.3 · p. 18, Table 7 · p. 19

5.1.4 Out-of-Distribution Evaluation on Static Manipulation

• 32.0% average success on SimplerEnv-OOD: Qwen-VLA-Instruct outperforms π0.5 (12.6%) across most tasks after both models are fine-tuned only on simple pick-and-place demonstrations.Evaluation covers six unseen tasks; Qwen-VLA-Instruct reaches 33.3% on MoveRight and 39.6% on PlaceNear, where π0.5 fails completely, and scores 22.9% on StackYellow versus 4.2% for π0.5.Source: §5.1.4 · p. 19, Table 8 · p. 20

5.1.5 Out-of-Distribution Evaluation on Dynamic Manipulation

• In zero-shot DOMINO evaluation, Qwen-VLA-Instruct achieves the highest SR (26.6%) and MS (39.5), surpassing PUMA by 9.4 percentage points in SR and 4.5 points in MS.Across all 35 suites, it uses current-frame observations without dynamic-manipulation fine-tuning; the paper attributes its OOD performance to coherent action chunks and broad embodied pretraining.Source: Table 9 · p. 20, §5.1.5 · p. 20

5.1.6 Out-of-Distribution Generalization in Real World

• On ALOHA, Qwen-VLA-Base demonstrates color-conditioned grasping, sequential cleanup with novel objects, and interaction with unseen object categories and backgrounds.The qualitative trials include successful grasps of broccoli and a toy duck (though its shape prevents a stable grasp), sequential cleanup of three everyday objects, and approaching unseen objects such as sunglasses, a plush doll, and a toy duck.Source: Figure 5 · p. 21, §5.1.6 · pp. 21–22

5.2.1 Text-to-Action Pretraining Ablations

• 71.1% success comes from mixing ∼20% synthetic with 80% real T2A data, a +10.2 pp gain over the 60.9% no-T2A baseline.Purely real data reaches 51.0% and purely synthetic data 64.1%. Performance peaks at 2,000 T2A steps (71.1%) and falls to 60.4% at 40,000 steps; the paper adopts 2,000 steps.Source: §5.2.1 · pp. 22–23
• Full-sequence prediction beats chunk prediction by 4.9 pp at 10% synthetic data (65.4% vs. 60.4%) and 2.9 pp on real-only data (51.0% vs. 48.2%).Full trajectories expose the decoder to long-horizon structure and coherent language-to-action mappings, whereas fixed-length chunks truncate dependencies. T2A and downstream stages use the same action representation: delta end-effector displacements relative to each chunk’s first frame, with embodiment prompts specifying platform, normalization, and prediction horizon.Source: §5.2.1 · p. 23
• At 10% synthetic data, adding images to chunk prediction lowers success from 60.4% to 57.6%, a −2.9 pp penalty.The ablation supports suppressing image tokens during T2A and deferring visual grounding to CPT, so the decoder learns a language–action prior rather than relying on visual correlations.Source: §5.2.1 · p. 23
• 71.1% success is achieved with Sigmoid-Normal timesteps in T2A and Beta timesteps at SFT; using Beta at both stages yields 59.4%.Switching T2A to Beta drops performance to 65.4% (−5.7 pp), while using Sigmoid-Normal at SFT yields 62.8% (−8.3 pp); the final pipeline uses Sigmoid-Normal for T2A and Beta for CPT/SFT.Source: §5.2.1 · p. 23

5.2.2 Multi-Embodiment Co-training

• VL+VLA improves RoboCasa-GR1 by +4.9 pp (51.1% →56.0%) and RoboTwin-2.0 by +4.6 pp (81.8% →86.4%) over action-only training.On Libero and Simpler-WidowX the configurations perform comparably; a pretrained DiT also converges faster and reaches a higher peak than a randomly initialized decoder paired with a fresh Qwen3.5-4B VLM.Source: §5.2.2 · pp. 23–24, Figure 7 · p. 24
• Co-training projection designs stay within <1.2% absolute performance of one another, while matching single-training results at about 63% on Bridge and 52–53% on RoboCasa.Zero-Padding uses 2h dmax projection parameters rather than 2h ∑i di for Multi-MLP and Concatenation, so the paper adopts it as the default.Source: §5.2.2 · pp. 24–25, Table 10 · p. 25

5.2.3 Effect of RL Post-Training

• RL adds +2.9 pp on SimplerEnv (70.8% to 73.7%), where rollouts are collected, while held-out DOMINO improves in SR (25.7% →26.6%) and MS (39.1 →39.5).Other held-out changes are RoboCasa +0.7 pp, RoboTwin-Hard +0.1 pp, LIBERO +0.1 pp, and SimplerEnv-OOD +0.4 pp; RoboTwin-Easy changes <0.6 pp.Source: Table 11 · p. 25, §5.2.3 · p. 25

5.2.4 State Conditioning

• Proprioceptive state yields at most +0.7 pp on RoboTwin-2.0 Easy and +1.3 pp on Hard, with the two injection strategies differing by ≤0.4 pp.The paper attributes the small gains to sufficient visual information and relative-displacement action prediction, and excludes state conditioning from the default framework to avoid embodiment-specific interfaces.Source: §5.2.4 · pp. 25–26, Table 12 · p. 26

6 Conclusion

• Qwen-VLA places manipulation, navigation, egocentric action modeling, and trajectory prediction in a shared action-and-trajectory prediction framework.This lets one model learn from heterogeneous embodied data across tasks, environments, and robot embodiments.Source: §6 · p. 26
• Its design combines embodiment-aware prompts, a DiT-based flow-matching action decoder, large-scale joint pretraining, and SFT/RL post-training.The approach retains flexibility across different control conventions while achieving strong generalist performance.Source: §6 · p. 26
• Qwen-VLA emphasizes grounding multimodal understanding in executable actions, positioning VLA models as an interface between multimodal foundation models and embodied agents.The conclusion contrasts this emphasis with visual prediction-centric world models and describes a bridge from high-level language reasoning to low-level physical control.Source: §6 · p. 26

• Embodied-action data remains much smaller and less diverse than vision-language pretraining data, limiting robustness to long-tail objects, environments, embodiments, and contact-rich interactions.Source: §7 · p. 26
• Action-oriented joint training improves policy learning but can modestly regress some pure vision-language and navigation evaluations, motivating better objective balancing, data curricula, and modular specialization.Source: §7 · p. 26
• Current evaluations remain largely short-horizon and benchmark-driven, leaving long-duration, failure-prone real-world deployment an open challenge.Source: §7 · p. 26
• Future work calls for scaling real-world interaction data through autonomous collection, simulation, and sim-to-real transfer, alongside large-scale egocentric and third-person human video for physical priors and temporal abstractions.Source: §7 · p. 26
• Future models should add long-horizon planning, episodic memory, and world modeling to track state, decompose tasks, recover from failures, and anticipate action consequences.Source: §7 · p. 26
• Richer force, tactile, and proprioceptive feedback combined with large-scale reinforcement learning in simulation and the real world may bridge language-grounded embodied understanding and reliable control.Source: §7 · pp. 26–27