PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications

Kento Kawaharazuka, Jihoon Oh, Jun Yamada, Ingmar Posner, Yuke Zhu

arXiv:2510.07077v1cs.ROcs.AIcs.CVcs.LG

TL;DR

Earlier robot policies struggled to generalize beyond predefined tasks, while VLA research seeks broader action-grounded generalization across tasks and embodiments. This survey systematically reviews VLA architectures, learning, data, hardware, evaluation, and deployment, concluding that foundation models and richer training practices are advancing the field despite persistent real-world limitations.

  • Problem

    Earlier systems typically used fixed motion primitives or imitation-learned policies, limiting generalization to unseen tasks.

  • Method

    The survey systematically reviews VLA strategies, architectures, modalities, learning paradigms, robot platforms, data practices, benchmarks, and real-world applications.

  • Results

    The analysis identifies large-scale datasets and pretrained foundation models, hierarchical architectures, and multimodal inputs as key developments, while sim-to-real transfer and embodiment generalization remain challenging.

  • Takeaways & Limitations

    The review offers practical guidance for applying VLA models to real-world robotic systems.

  • Takeaways & Limitations

    Most existing VLA systems remain restricted to predefined environments and do not yet achieve the robustness and adaptability required for practical deployment.

Abstract

from arXiv · show

Amid growing efforts to leverage advances in large language models (LLMs) and vision-language models (VLMs) for robotics, Vision-Language-Action (VLA) models have recently gained significant attention. By unifying vision, language, and action data at scale, which have traditionally been studied separately, VLA models aim to learn policies that generalise across diverse tasks, objects, embodiments, and environments. This generalisation capability is expected to enable robots to solve novel downstream tasks with minimal or no additional task-specific data, facilitating more flexible and scalable real-world deployment. Unlike previous surveys that focus narrowly on action representations or high-level model architectures, this work offers a comprehensive, full-stack review, integrating both software and hardware components of VLA systems. In particular, this paper provides a systematic review of VLAs, covering their strategy and architectural transition, architectures and building blocks, modality-specific processing techniques, and learning paradigms. In addition, to support the deployment of VLAs in real-world robotic applications, we also review commonly used robot platforms, data collection strategies, publicly available datasets, data augmentation methods, and evaluation benchmarks. Throughout this comprehensive survey, this paper aims to offer practical guidance for the robotics community in applying VLAs to real-world robotic systems. All references categorized by training approach, evaluation method, modality, and dataset are available in the table on our project website: https://vla-survey.github.io .

I. INTRODUCTION

VLA research addresses the limited generalization of earlier robot-policy systems by jointly learning vision, language, and action. This survey reviews the field's challenges, development, full-stack components, and practical deployment scope.

  • Earlier systems relied on fixed motion primitives or imitation-learned policies, limiting generalization beyond predefined tasks.
  • VLAs jointly learn visual, linguistic, and action modalities to support broader task performance and transfer across robotic embodiments.The intended benefit is reduced task-specific data collection and training for deployment.
  • The survey provides a systematic, full-stack overview spanning VLA evolution, architectures, modality integration, learning paradigms, hardware, data, augmentation, and evaluation.
  • The review defines a VLA as taking visual observations and natural-language instructions as core inputs and directly generating robot control commands.Additional sensory modalities may be included, but high-level skill selection without direct action grounding is excluded.
  • Robust deployment remains constrained by limited multimodal data, embodiment mismatch, computational demands, and restricted operating environments.These constraints create design trade-offs across architecture, training, and deployment feasibility.

C. COMPUTATIONAL AND TRAINING COST

VLA development and deployment face substantial computational and data demands, motivating architectures that progressively combine pretrained multimodal backbones with diffusion, flow matching, latent actions, and hierarchical control.

  • C. COMPUTATIONAL AND TRAINING COST: Adapting and fine-tuning pretrained VLMs remains computationally intensive for long sequences, high-resolution images, and additional modalities.Transformer-based architectures also scale poorly with sequence length and input dimensionality.
  • III. VLA DESIGN STRATEGY AND TRANSITION: VLA architectures evolved from CNN-based end-to-end models toward transformer sequence models, pretrained-VLM policies, diffusion policies, and hierarchical frameworks.
  • III. VLA DESIGN STRATEGY AND TRANSITION: VIMA performs diverse robotic tasks but was evaluated only in simulation environments.
  • III. VLA DESIGN STRATEGY AND TRANSITION: OpenVLA uses full fine-tuning on the Open-X Embodiment dataset and outperforms RT-2 and Octo.
  • III. VLA DESIGN STRATEGY AND TRANSITION: Recent approaches use unlabeled video for latent-action pretraining and hierarchical policies to connect high-level language plans with low-level motor execution.

IV. ARCHITECTURES AND BUILDING BLOCKS

VLA architectures differ in how they align perception, language, and control, with sensorimotor models forming the main class and world-model and affordance-based alternatives extending the design space.

  • A. SENSORIMOTOR MODEL: Sensorimotor models jointly represent visual, linguistic, and action information and directly map images and language to robot actions.They may use flat or hierarchical structures and varying backbone architectures.
  • A. SENSORIMOTOR MODEL: Transformer-plus-discrete-token models tokenize modalities and predict actions, while RT-1 predicts all action tokens non-autoregressively after input compression.RT-1 extracts 11 action outputs from a 48-token transformer input.
  • A. SENSORIMOTOR MODEL: Diffusion action heads and diffusion transformers produce continuous actions by applying diffusion after, or directly within, the transformer.
  • A. SENSORIMOTOR MODEL: VLM-based architectures use pretrained knowledge for generalization, optionally pairing VLMs with discrete tokens or diffusion action heads.
  • A. SENSORIMOTOR MODEL: The survey categorizes seven sensorimotor architectures by combining transformers or VLMs with discrete tokens, diffusion, or flow-matching action generation.
  • A. SENSORIMOTOR MODEL: Flow-matching action heads target responsive, smooth continuous control, with π0 reaching rates of up to 50 Hz.
  • A. SENSORIMOTOR MODEL: VLM-plus-diffusion-transformer designs separate high-level reasoning from low-level control, using the VLM as system 2 and the diffusion transformer as system 1.GR00T N1 generates continuous actions via flow matching with cross-attention to VLM tokens.

B. WORLD MODEL

World models in VLAs are organized around predicting future observations, learning latent actions, or jointly predicting actions and observations. Affordance-based models provide complementary routes for conditioning or structuring action generation.

  • World-model taxonomy: World-model-based VLAs are grouped into action generation, latent action generation, and sensorimotor models with implicit world models.These categories respectively use predicted observations, latent actions, or joint action-and-observation prediction.
  • Action generation in world models: Action-generation methods predict future images, videos, or visual representations and use them to derive low-level actions.UniPi combines video prediction with an inverse dynamics model, while other methods predict subgoals, trajectories, optical flow, or feature points.
  • Latent action generation: Latent-action methods learn action representations from human demonstrations or web-scale videos when explicit robot actions are unavailable.LAPA, UniVLA, and UniSkill use latent actions as training signals or conditions for downstream action policies.
  • Implicit world models: Sensorimotor models jointly predict actions and future observations within VLA architectures to support planning and improve task performance.GR-1 predicts future images before being fine-tuned for joint action and frame prediction, and reports improved task success.
  • Affordance-based models: Affordance-based VLAs predict actionable properties, extract them from human videos, or integrate affordance prediction directly into the architecture.The resulting affordances can condition action generation, provide latent guidance, or identify action points for control.

D. DATA MODALITIES

VLA systems convert heterogeneous visual, linguistic, sensor, and action information into tokens or continuous representations. Their processing choices balance multimodal integration, action expressivity, embodiment diversity, and inference efficiency.

  • Vision: Visual processing commonly uses ResNet or ViT encoders, often pretrained on ImageNet or LAION, with outputs converted into tokens.Some systems process full images directly, while others first divide images into patches.
  • Vision: VLAs also use CLIP, SigLIP, DINOv2, discrete visual tokenizers, and compression modules to align or shorten visual representations.Perceiver Resampler, Q-Former, QT-Former, and TokenLearner reduce or integrate visual token sequences for multimodal models.
  • Vision: Object-centric representations provide bounding boxes or cropped-region embeddings extracted by detection, segmentation, or tracking models.Examples include Mask R-CNN, OWL-ViT, SAM, GroundingDINO, Detic, and Cutie.
  • Action: Action representations include discretized tokens, continuous decoding through MLPs, and diffusion or flow-matching generation.Standard binning commonly uses 256 bins per action dimension, whereas FAST compresses temporal action tokens using DCT and BPE.
  • Action: Latent action representations learned from web-scale video provide training signals when explicit actions are unavailable.This approach uses world modeling to support larger datasets and more generalizable VLA training.
  • Embodiment: Cross-embodiment approaches standardize heterogeneous robot data or tokenize modality-specific observations into shared sequences.The Open X-Embodiment Project uses a unified format with a single camera, language instructions, and 7-DoF actions; CrossFormer masks missing modalities.

4) Miscellaneous Modalities

Beyond vision, language, and action, VLAs incorporate audio, tactile sensing, and 3D information. These modalities are encoded through specialized tokenizers or visual backbones to support perception, spatial reasoning, and manipulation.

  • Audio: Audio inputs are typically represented from spectrograms or mel-spectrogram images using visual or speech encoders.Examples include ResNet, ViT-VQGAN, SpeechTokenizer, AST, and Whisper.
  • Tactile sensing: Tactile sensors provide image-based inputs that are encoded with ViTs or Touch-Vision-Language models.These representations support integration of visual and tactile information for fine-grained manipulation.
  • 3D information: 3D information is incorporated through depth images, multi-view images, voxel representations, and point clouds.These representations are intended to improve environmental perception and action planning.
  • Depth images: Depth processing can combine estimated 3D coordinates with 2D visual features or use RGB-D inputs in 3D action heads.SpatialVLA uses Ego3D Position Encoding, while other systems apply diffusion-based action generation in 3D space.
  • 3D information: Multi-view images, voxel grids, and point clouds are encoded using geometry-aware transformers, VQ-VAE, 3D U-Nets, or point-based backbones.The selected representation determines whether 3D structure is modeled implicitly, as occupancy, or directly as points.
  • Additional modalities: Additional motion-related inputs include 3D tracking data and discretized SMPL-X joint angles.These modalities extend VLA perception and interaction beyond the primary sensory channels.

E. EMERGING TECHNIQUES

Emerging VLA techniques introduce intermediate structure through hierarchical policies and Chain-of-Thought reasoning, while training combines supervised, self-supervised, reinforcement, and in-context strategies.

  • Hierarchical architectures: Hierarchical architectures decompose task instructions into subtasks before a low-level VLA executes them.Cleaner subtask descriptions can make low-level execution more reliable than processing complex instructions directly.
  • Chain-of-Thought reasoning: Chain-of-Thought VLA methods predict intermediate task descriptions, subtasks, or object positions before generating final actions.ECoT uses this autoregressive intermediate process to enhance planning and inference capabilities.
  • Supervised learning: Supervised learning is the dominant training paradigm, usually framing VLA training as next-token prediction over image, language, and action pairs.The action loss is selected according to the action head, including MLP, diffusion, or flow-matching architectures.
  • Training stages: VLA training commonly uses pre-training followed by post-training on smaller, task- or robot-specific datasets.Pre-training often starts from a web-scale pretrained LLM or VLM, while post-training prioritizes data quality and may update the full model or only the action head.
  • Reinforcement learning: Reinforcement learning is integrated either to fine-tune VLA models or to train low-level policies beneath high-level VLA policies.The survey identifies PPO and SAC among the reinforcement-learning approaches used for these roles.
  • In-context learning: In-context VLA learning conditions action generation on 1–3 teleoperated demonstrations supplied as prompts at test time.ICRT uses this setup to generate corresponding robot actions without explicit fine-tuning on the demonstrations.
  • Self-supervised learning: Self-supervised learning supports modality alignment, visual representation learning, and latent action representation learning.These objectives include contrastive alignment, masked autoencoding, self-distillation, and reconstructing goal images from latent actions.

C. REINFORCEMENT LEARNING

Reinforcement learning is used either to improve VLA models directly or to provide low-level control beneath high-level VLA policies. Despite diverse combinations with imitation learning, most RL-based VLA work remains constrained by simulation or simplified real-world settings.

  • Motivation: Imitation learning remains common, but it struggles with novel behaviors and requires sufficiently large, high-quality expert demonstrations.
  • RL strategies: RL-based VLA methods follow two broad strategies: improving the VLA directly or using it as a high-level policy with RL-based low-level control.Examples include VLA fine-tuning with success or failure rewards and separate RL controllers for humanoid or mobile-manipulator systems.
  • Hybrid learning: Representative systems combine supervised fine-tuning, online or offline RL, successful-trajectory collection, and subsequent imitation learning.iRe-VLA repeatedly alternates supervised fine-tuning and online RL, while other systems use RL-trained teachers, student VLAs, or VLA-guided exploration.
  • Limitations: Most prior RL-based VLA methods remain limited to simulation or simplified real-world setups because of sample inefficiency, unsafe exploration, and computational inefficiency.
  • Training design: VLA training increasingly combines action, grounding, and reasoning objectives with large-scale, heterogeneous data and varied VLM backbones.Examples include real-world trajectories, vision-language datasets, bounding-box supervision, human videos, and synthetic trajectories.
  • Training design: Gradient insulation is used to protect pre-trained vision-language representations from destabilizing gradients originating in a randomly initialized action head.The survey reports improvements in training stability and efficiency from this design.

2) Post-training

Post-training adapts pre-trained VLA models using robot- and task-specific data while balancing performance against computational cost and preservation of general representations. Action-head fine-tuning, LoRA, and quantization provide more efficient alternatives to full fine-tuning.

  • Post-training: Post-training requires high-quality robot- and task-specific data, unlike pre-training, which uses large-scale and diverse datasets.
  • Efficient adaptation: BitVLA compresses the vision encoder to 1.58 bits through distillation, achieving substantial memory savings with minimal performance degradation.
  • Adaptation choices: Freezing the backbone reduces GPU memory and training time, whereas full fine-tuning enables end-to-end adaptation to robot-specific visual patterns and domain shifts.
  • Adaptation choices: Full fine-tuning often produces the highest task-specific performance when sufficient data and compute are available, while LoRA can achieve competitive performance with lower resource requirements.
  • Adaptation choices: Frozen backbones preserve web-scale visual and linguistic representations, while full fine-tuning risks degrading pre-trained knowledge through catastrophic forgetting.

E. INFERENCE

Real-world VLA systems require efficient inference and scalable data collection. The survey covers asynchronous and cache-based acceleration alongside teleoperation, proxy-device, and human-data collection pipelines.

  • Inference: Real-Time Chunking reduces execution latency by fixing previously executed actions while generating later actions asynchronously.Soft masking preserves temporal consistency while updated sensory inputs support dynamic replanning.
  • Inference: DeeR-VLA and VLA-Cache accelerate inference by skipping stable transformer layers and reusing features for static tokens.
  • Data collection: Teleoperation records high-quality trajectories by letting human operators control robots directly, but hardware-intensive leader-follower systems motivate lower-cost alternatives.
  • Data collection: Direct-control devices such as 3D mice and game controllers simplify collection but may reproduce natural human motions with lower fidelity.
  • Data collection: Proxy devices decouple human motion from physical robot control, enabling more intuitive and scalable collection while preserving transferable motion information.UMI uses a handheld gripper and camera-based trajectory estimation; DexUMI extends the approach to dexterous hands with tactile sensing.
  • Annotation: Foundation models can automate language and perceptual annotation more cheaply and scalably than human post-hoc annotation, but fine-grained understanding and hallucinations remain challenges.Visual or multimodal grounding is described as helping address these issues.

B. DATASETS FOR VLA

VLA pre-training draws on human egocentric, simulation, and real-robot datasets, each offering different scalability, control, embodiment, and realism properties. The survey emphasizes that datasets aligning vision, language, and action remain limited in scale and diversity.

  • Human datasets: Human data collection is more scalable than robotic collection because it avoids physical robots, precise calibration, and safety-critical execution environments.First-person data is especially relevant because it approximates perceptual inputs from head-mounted or human-like robot sensors.
  • Dataset organization: Table 1 distinguishes atomic skills such as pick, place, and reach from instruction-level tasks when reporting real-world robot dataset statistics.
  • Human datasets: Egocentric datasets such as Aria Everyday Activities, Ego-Exo4D, HOT3D, and HD-EPIC support VLA pre-training through naturalistic visual data and latent action prediction.
  • Simulation datasets: Simulation provides scalable, safe, controllable, and cost-effective data for imitation learning and large-scale pre-training.It remains useful when real-world collection is impractical or cost-prohibitive, despite declining relative dominance.
  • Real robot datasets: Real-robot datasets provide realistic interactions, diverse embodiments, and rich sensory inputs needed for real-world generalization and evaluation.Examples include standardized DROID data, multimodal FuSe trajectories, diverse-embodiment RoboMIND data, and million-trajectory AgiBot World data.

C. DATA AUGMENTATION FOR VLA

VLA data augmentation expands limited datasets across vision, language, and action while respecting robotics-specific embodiment constraints. The survey also situates these methods within the robot platforms and data resources used in VLA research.

  • Data augmentation expands existing VLA datasets across vision, language, and action modalities because collecting datasets is costly.
  • Vision augmentation: Standard image transformations can distort the robot’s embodiment and camera relationship, so generation models and inpainting address visual variability differently.BYOVLA inpaints task-irrelevant image regions during runtime to improve robustness against visual distractions.
  • Language augmentation: DIAL combines manually labeled trajectory-instruction pairs, VLM similarity matching, and LLM-generated paraphrases to label additional trajectories.
  • Action augmentation: Action augmentation commonly expands data interactively, with DAgger collecting expert actions in policy-visited states and CCIL generating corrective data for out-of-distribution states.
  • Robot platforms: VLA research uses diverse platforms, including manipulators, hands and grippers, mobile robots, quadrupeds, and humanoids.These platforms cover manipulation, navigation, uneven-terrain locomotion, and human-compatible environments.

B. EVALUATION FOR VLA

VLA evaluation is dominated by simulation because physical-robot evaluation faces embodiment differences, safety concerns, and limited reproducibility. The survey catalogs benchmarks spanning navigation, manipulation, whole-body control, modalities, and environment variation.

  • Most VLA evaluations occur in simulation because physical-robot generalization is difficult to assess reproducibly across embodiments and safety constraints.
  • MuJoCo: MuJoCo-based benchmarks support manipulation learning, from robosuite’s 11 tasks and robomimic’s 8 Panda tasks to RoboCasa’s 100 photorealistic tasks.
  • MuJoCo: LIBERO is a widely used language-conditioned manipulation benchmark with 4 task suites and 130 tasks executed using a Franka Emika Panda.
  • Other simulation engines: Other benchmark families cover humanoid control, articulated and deformable-object manipulation, dual-arm tasks, long-horizon planning, and environment perturbations.LeVERB-Bench includes 154 vision-language and 460 language-only tasks; RoboTwin offers 50 tasks, 731 objects, and 5 embodiments.
  • Evaluation dimensions: Benchmark tables organize environments by task type, observation modality, and available scenes or objects, while VLATest varies confounding objects, lighting, camera poses, unseen objects, and instructions.
  • Realistic and scalable evaluation: SIMPLER reduces visual and control domain gaps when evaluating real-world-trained policies in simulation, achieving high correlation between simulated and real-world performance.

VIII. RECOMMENDATIONS FOR PRACTITIONERS

The survey recommends practical choices for deploying VLAs: diverse multimodal data, continuous generative control, protected pretrained representations, lightweight adaptation, and scalable learning strategies. It also identifies multimodal data standardization and reasoning as unresolved needs.

  • Data and deployment: Practitioners should prioritize large, high-quality datasets spanning vision, language, action, task coverage, environmental variability, and embodiment diversity.
  • Action generation: Continuous action generation through diffusion or flow matching is recommended for smooth, precise, high-fidelity real-world control.
  • Pretraining: Freezing the VLM backbone or insulating its gradients from action heads can preserve semantic representations while improving training efficiency and final performance.
  • Model adaptation: Action-head tuning or LoRA provides lightweight alternatives to full finetuning when computational resources are limited.
  • Scalability: Latent action learning can exploit human video without explicit action labels, while world models support planning through future-observation prediction.
  • Representation learning: Auxiliary tasks such as affordance estimation, keypoint detection, future-state prediction, and segmentation can align VLM representations with action generation.
  • Multimodal data: Large-scale additional-modality datasets remain difficult to collect because tactile sensors vary in design, formats, and hardware configurations.The survey identifies sensor unification as critical for scalable multimodal VLA systems.
  • Reasoning: Long-horizon VLA reasoning requires retaining and selectively retrieving relevant information over time, making memory a key future direction.

C. CONTINUAL LEARNING

The survey identifies continual learning, safety, failure recovery, rigorous evaluation, and embodiment transfer as central barriers to reliable VLA deployment. It concludes that current systems remain constrained despite broad application potential, while hybrid and adaptive methods define future directions.

  • Continual learning: Offline-trained VLA systems typically remain frozen, limiting adaptation to novel or out-of-distribution situations and motivating online or continual learning.Continual updates introduce catastrophic forgetting and safety concerns from untested real-world changes.
  • Continual learning: Reinforcement-learning fine-tuning has mainly been evaluated in simulation because real-world exploration requires many samples and risks unsafe behavior.Learned world models are proposed as a safer, more sample-efficient alternative.
  • Safety and recovery: Controlled manipulation performance does not resolve safety challenges in unstructured environments, where systems may fail to detect unexpected humans or recover from failures.
  • Safety and recovery: Hybrid architectures combining learned-policy generalization with model-based controller reliability are proposed to improve VLA safety.
  • Evaluation: Statistically rigorous evaluations with controlled conditions, sufficient trials, and confidence intervals are needed to determine which VLA approaches perform best.
  • Applications: VLA models have potential across healthcare, assistive technology, industrial automation, and autonomous driving but remain below the robustness and adaptability needed for deployment.
  • Conclusion: The survey’s synthesis highlights large-scale data, foundation models, hierarchical architectures, broader modalities, and persistent sim-to-real and embodiment-generalization challenges.It anticipates roles for world models, affordance reasoning, and reinforcement learning in future systems.
Loading 2510.07077v1…