PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challenges

Ranjan Sapkota, Yang Cao, Konstantinos I. Roumeliotis, Manoj Karkee

arXiv:2505.04769v2cs.CV

TL;DR

VLA research addresses the difficulty of integrating perception, language understanding, and embodied action in systems that must operate in real-world environments. This review synthesizes VLA foundations, architectures, training and inference advances, applications, challenges, and future directions across the recent literature. It identifies efficiency gains and architectural developments while emphasizing unresolved robustness, supervision, safety, generalization, and deployment constraints.

  • Problem

    Separate vision, language, and action systems struggled to integrate modalities and generalize or adapt to complex, unpredictable real-world challenges.

  • Method

    The review systematically analyzes VLA foundations, architectural progress, training and inference strategies, applications, challenges, and future research directions.

  • Results

    VLA architectures unify visual perception, language comprehension, and motor control, while dual-system frameworks achieve up to 2.5× faster policy inference with per-step latency below 5 ms.

  • Takeaways & Limitations

    The review frames convergence among VLMs, VLA architectures, and agentic AI as a direction toward intelligent, embodied, and human-aligned agents.

  • Takeaways & Limitations

    VLA deployment remains constrained by environmental variability, scarce and noisy supervision, expert-intervention requirements, ambiguous instructions, and limited dataset breadth.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models mark a transformative advancement in artificial intelligence, aiming to unify perception, natural language understanding, and embodied action within a single computational framework. This foundational review presents a comprehensive synthesis of recent advancements in Vision-Language-Action models, systematically organized across five thematic pillars that structure the landscape of this rapidly evolving field. We begin by establishing the conceptual foundations of VLA systems, tracing their evolution from cross-modal learning architectures to generalist agents that tightly integrate vision-language models (VLMs), action planners, and hierarchical controllers. Our methodology adopts a rigorous literature review framework, covering over 80 VLA models published in the past three years. Key progress areas include architectural innovations, efficient training strategies, and real-time inference accelerations. We explore diverse application domains such as autonomous vehicles, medical and industrial robotics, precision agriculture, humanoid robotics, and augmented reality. We analyzed challenges and propose solutions including agentic adaptation and cross-embodiment planning. Furthermore, we outline a forward-looking roadmap where VLA models, VLMs, and agentic AI converge to strengthen socially aligned, adaptive, and general-purpose embodied agents. This work, is expected to serve as a foundational reference for advancing intelligent, real-world robotics and artificial general intelligence. The project repository is available on GitHub as https://github.com/Applied-AI-Research-Lab/Vision-Language-Action-Models-Concepts-Progress-Applications-and-Challenges. [Index Terms: Vision Language Action, VLA, Vision Language Models, VLMs, Action Tokenization, NLP]

1. Introduction

VLA models emerged to bridge separately developed vision, language, and action systems that struggled to integrate modalities and generalize in real-world settings. This review organizes VLA foundations, progress, applications, limitations, and future directions through a structured literature analysis.

  • Motivation: Separate vision, language, and action systems worked independently but struggled to cooperate, generalize to novel scenarios, and adapt to unpredictable real-world challenges.
  • VLA emergence: VLA models unify visual perception, language comprehension, and motor control so embodied agents can interpret instructions and execute actions dynamically.
  • VLA emergence: Action tokens extend vision-language models with numerical or symbolic representations of robot motor commands learned from paired vision, language, and trajectory data.
  • Review scope: The review synthesizes VLA concepts, developmental progress, technical challenges, applications, and future directions, including multimodal integration and unified tokenization strategies.
  • Progress: Architectural, data-efficient, parameter-efficient, and acceleration strategies are presented as critical for scaling VLA systems to real-world applications.
  • Challenges and outlook: The review discusses inference bottlenecks, safety concerns, computational demands, limited generalization, and ethical implications, alongside potential solutions.

2. Concepts of Vision-Language-Action Models

VLA models unify visual perception, language understanding, and physical action through multimodal architectures, token-based representations, hybrid training, and adaptive control. Their development spans foundational integration, specialization, and embodied reasoning, with the framework designed for generalization across tasks and robot morphologies.

  • Core Concept: VLAs jointly process visual inputs, natural-language instructions, and executable action representations for task-conditioned control in dynamic robotic environments.They combine vision encoders, language models, and policy or planning modules.
  • Evolution and Timeline: The field evolved through foundational integration in 2022–2023 and specialization and embodied reasoning in 2024.Early systems established visuomotor coordination, while later systems added retrieval, scene-graph, memory, and partial-observability capabilities.
  • Multimodal Integration: Traditional robotic pipelines separated perception, language, and control, requiring manually defined interfaces and domain-specific engineering that limited adaptability and generalization.Modern VLAs instead fuse modalities end-to-end using pretrained encoders and transformer architectures.
  • Tokenization and Representation: Action-token representations let transformers encode vision, language, state, and action in a shared space and autoregressively generate physical motion sequences.Prefix tokens describe goals and scenes, state tokens encode robot posture, and action tokens specify motor behavior.
  • Generalization and Applications: The tokenization pipeline is presented as extensible across tasks and robot morphologies, supporting deployment in applications such as apple picking, household tasks, and mobile navigation.The architecture separates tokenization steps while maintaining a coherent representation of perception, instruction, and embodiment.
  • Learning Paradigms: Hybrid training aligns web-scale semantic knowledge with robotics data so models map visual and linguistic inputs to actions while learning affordances and supporting novel scenarios.RT-2 treats action generation as text generation and uses web-scale multimodal data with thousands of robot demonstrations for zero-shot generalization.
  • Adaptive Control and Real-Time Execution: Real-time feedback enables VLAs to reinterpret changing scenes and adjust trajectories during execution, including when target objects move or new objects enter view.This adaptive control is presented as important for unstructured environments such as orchards, homes, and hospitals.

3. Progress in Vision-Language-Action Models

VLA development was catalyzed by transformer-based language and multimodal models, then accelerated by large-scale robotic datasets. RT-1 supplied action-grounding demonstrations, while RT-2 unified vision, language, and action tokens for robotic control.

  • Progress Drivers: Transformer-based LLMs and GPT-4’s multimodal capabilities helped motivate extensions of language models toward perception and action in robotics.ChatGPT’s semantic reasoning and GPT-4’s text-and-image processing are identified as pivotal developments.
  • Robotic Data: RT-1 contributed 130,000 robotic demonstrations that supplied diverse action-grounding data for co-training vision, language, and action components.The demonstrations covered diverse tasks and environments to support generalizable behaviors.
  • Architectural Progress: RT-2 unified vision, language, and action tokens, marking a landmark architectural development in VLA models.Its development followed the creation of large-scale robotic datasets such as RT-1’s demonstrations.

3.1. Architectural Innovations in VLA Models

VLA architectures have evolved from unified and hierarchical designs toward dual-system and self-correcting frameworks that balance planning, control, efficiency, and robustness. These innovations support long-horizon reasoning, real-time execution, spatial precision, and deployment on constrained hardware.

  • Architectural paradigms: Three major VLA paradigms are early fusion models, dual-system architectures, and self-correcting frameworks.They address grounding, generalization, and action reliability through different architectural choices.
  • Dual-system architectures: Dual-system architectures separate fast reactive control from slower language-model planning for multi-timescale task execution.GR00T N1 uses a diffusion-based System 1 for low-level control and an LLM-based System 2 for task decomposition and sequencing.
  • Dual-system architectures: 17% higher success rate and 28% fewer collision failures were reported for GR00T N1 than RT-1, RT-2, and OpenVLA on multi-stage household manipulation.The comparison concerns benchmark tests involving multi-stage household tasks.
  • Self-correcting frameworks: Self-correcting VLAs add a selectively activated recovery path that diagnoses execution failures and generates corrective actions.SC-VLA retains a fast default inference path while invoking slower reasoning after failures such as unsuccessful grasps or obstacle collisions.
  • Architectural design space: Architectural design choices span end-to-end versus modular pipelines, hierarchical versus flat policies, and low-level control versus high-level planning.These dimensions provide a framework for comparing scalability, adaptability, and task performance across VLA systems.

3.2. Training

VLA training advances combine large-scale vision-language data, robotic trajectories, synthetic edge cases, and self-supervised alignment. These strategies improve generalization, robustness, convergence, and real-time inference while reducing dependence on task-specific labels and computation.

  • Data-Efficient Learning: Co-fine-tuning vision-language corpora with robotic trajectories aligns semantic understanding with motor skills.OpenVLA achieves a 16.5 % higher success rate than a 55 B-parameter RT-2 variant despite using 7 B parameters.
  • Data-Efficient Learning: Synthetic data generation with UniSim improves robustness in cluttered environments by over 20 % through photorealistic edge-case scenes.The generated scenes include occlusions and dynamic lighting to augment rare scenarios.
  • Data-Efficient Learning: Self-supervised visual-text alignment reduces reliance on task-specific labels and accelerates downstream grasp-and-place convergence by 12 %.Qwen2-VL uses contrastive objectives before action fine-tuning.
  • Inference Acceleration: Compressed action tokens and parallel decoding achieve up to 2.5× faster policy inference with per-step latency below 5 ms.The acceleration incurs slightly higher action discretization error and reduced fine-grained motion continuity under high-frequency control.
  • Overall progress: Together, these training and inference methods produce practical agents for language-conditioned, vision-guided tasks in dynamic real-world settings.The stated advances target multimodal reconciliation, compute reduction, and real-time control.

3.3. Parameter-E

Parameter-efficient adaptation and runtime acceleration reduce the computational and memory barriers to deploying large VLA policies on real robotic platforms. The resulting methods support faster control, consumer-grade hardware, and high-frequency manipulation, with limited trade-offs in trajectory granularity.

  • Inference acceleration: 15× faster inference is achieved by compressing 1000 ms action windows into 16 discrete tokens, enabling control rates up to 200 Hz.The method trades minimal trajectory granularity for speed in high-frequency reactive manipulation.
  • Deployment efficiency: LoRA and quantization enable smaller labs to fine-tune and operate billion-parameter VLAs on consumer-grade hardware.These methods reduce the effective adaptation or runtime resource footprint while preserving semantic capabilities.
  • Model and action compression: Pruning and FAST tokenization enable sub-4 GB, sub-5 ms control loops without sacrificing precision in dexterous tasks.The methods compress both model and action representations.
  • Inference acceleration: Parallel decoding and action chunking support 100–200 Hz decision rates by reducing sequential autoregressive bottlenecks.These rates target agile manipulation and legged locomotion.
  • Deployment impact: Collectively, these techniques make VLA deployment practical across industrial manipulators, assistive drones, and other robotic platforms.The section frames them as a bridge from research prototypes to real-world autonomy.

3.4. Applications of Vision-Language-Action Models

VLA models are being applied across humanoid robotics, autonomous driving, aerial systems, industrial manipulation, and healthcare. Across these domains, their combination of multimodal grounding, planning, and control supports adaptable behavior, interpretable outputs, and transfer across tasks or embodiments.

  • Humanoid robotics: Humanoid VLAs support high-frequency full-body manipulation and assistive tasks in households, hospitals, retail, logistics, and manufacturing.Examples include voice-commanded domestic tasks, surgical instrument handovers, shelf restocking, and warehouse picking or sorting.
  • Autonomous vehicle systems: CoVLA pairs over 80 hours of real-world driving videos with synchronized sensors, language annotations, and high-resolution trajectories for autonomous-driving VLA training.Its components include CLIP visual grounding, LLaMA-2 instruction embedding, and trajectory decoders.
  • Autonomous vehicle systems: OpenDriveVLA and ORION combine multimodal perception, language reasoning, and trajectory generation for interpretable autonomous-driving decisions.OpenDriveVLA generates action plans and trajectory visualizations, while ORION retains long-horizon context and aligns discrete reasoning with continuous vehicle control.
  • Autonomous vehicle systems: Integrated VLA architectures dynamically replan trajectories, communicate safety-relevant intentions, and adjust control policies in real time.The reported comparison emphasizes semantic feedback across language intent, visual context, and embodied state relative to loosely coupled pipelines.
  • Industrial robotics: CogACT outperforms OpenVLA by over 28% in real-world task success rates, especially for multi-step assembly, screw fastening, and part sorting.It also adapts across 6-DoF arms and bimanual systems through efficient fine-tuning.
  • Healthcare and medical robotics: Medical VLAs support patient-assistive tasks while offering scene grounding, trajectory visualization, post-hoc review, and low-data hospital adaptation.These capabilities are presented as relevant to clinical trust, auditability, and procedural workflows.

4. Challenges and Limitations of Vision-Language-Action Models

VLA models face intertwined challenges in real-time inference, data-efficient adaptation, safety, generalization, multimodal action representation, and system integration. The paper maps these limitations to mitigation strategies aimed at reliable real-world deployment.

  • Data-Efficient Adaptation: VLA adaptation under scarce, noisy data remains difficult because successful approaches may depend on expert interventions, reward shaping, complex module coordination, or narrow datasets.ConRFT reaches 96.3% success over eight contact-rich tasks, while tactile-language-action models exceed 85% success on unseen peg-in-hole tasks, but both retain broader generalization constraints.
  • Safety and Generalization: Safety and generalization remain unresolved in dynamic environments, where richer scene dynamics, limited visual–physical transfer, added complexity, and incomplete safety rules constrain reliability.SafeVLA cuts unsafe behavior by over 80%, while ReVLA improves oriented object detection OOD grasp success by up to 77%, but both approaches retain deployment trade-offs.
  • Mitigation Strategies: Mitigation strategies include accelerated decoding and hardware pipelines, hybrid diffusion–autoregressive policies, dynamic risk assessment, adaptive planning, curated debiased corpora, and contrastive fine-tuning.The proposed mapping connects specific limitations to technical remedies and anticipated benefits for real-world VLA deployment.
  • Real-Time Inference Constraints: Real-time inference remains constrained by autoregressive decoding, hardware bandwidth demands, and accuracy losses from aggressive efficiency techniques.Parallel decoding can accelerate inference, but may reduce trajectory smoothness; high-dimensional visual embeddings can require approximately 1.2 GB/s memory bandwidth, while quantization may degrade precision.
  • Multimodal Action Representation: Multimodal action representation presents a precision–efficiency trade-off: discrete tokenization lacks fine-grained control, whereas diffusion policies capture richer possibilities at roughly three times conventional decoder cost.These limitations hinder complex dynamic tasks and make richer diffusion-based policies impractical for real-time deployment.
  • Dataset Bias, Grounding, and Generalization: Dataset bias and grounding deficiencies impair semantic alignment and compositional generalization, with web-crawled data associating approximately 17% of standard-dataset relationships with stereotypical interpretations.OpenVLA has been documented to overlook approximately 23% of object references in novel settings, while performance on entirely novel tasks can degrade by as much as 40%.

5. Discussion

The discussion identifies computational, safety, generalization, integration, and ethical barriers that limit practical VLA deployment, then outlines architectural, training, and governance directions to address them. The roadmap emphasizes efficient deployment, reliable intelligence, unified systems, and continual adaptation toward more capable embodied agents.

  • Challenges: Real-time VLA inference remains difficult because autoregressive decoding, high-dimensional multimodal inputs, and large compute demands constrain deployment on embedded hardware.The discussion highlights latency, throughput, memory, energy, and resource constraints as central deployment concerns.
  • Potential Solutions: Model compression, distillation, quantization, adaptive inference, and specialized accelerators are proposed to reduce inference cost while preserving benchmark performance.LoRA and distillation can reduce parameter counts by up to 90% while retaining over 95% of original benchmark performance; mixed precision can reduce computation by 2–4×.
  • Potential Solutions: Safe multimodal action requires integrated perception, reasoning, and control, with hybrid policies, preference optimization, and formal verification of planner outputs.The proposed combination uses diffusion for low-level motion primitives, autoregressive planning for high-level actions, and symbolic checks for safety invariants.
  • Potential Solutions: Generalization to unseen tasks depends on diverse debiased datasets, 3D spatial transfer, and sim2real fine-tuning with randomized environmental conditions.The review connects web-scale image–text data, robot trajectory archives, 3D perception, domain randomization, and real-world calibration to robustness under distribution shift.
  • Future Roadmap: Future VLA systems are framed around multimodal foundation-model “cortices,” agentic lifelong learning, hierarchical planning, and world models for adaptive embodied reasoning.These directions target persistent scene structure, continual skill growth, scalable planning, counterfactual evaluation, and corrective action in changing environments.
  • Future Roadmap: Continual adaptation can mitigate distribution shift and long-horizon brittleness but requires replay, safety-aware updates, modular adapters, and verification-informed policy revision.Repeated policy updates may cause catastrophic forgetting, behavioral regressions, or corruption from noisy and adversarial feedback.

6. Conclusion

The conclusion synthesizes three years of VLA research by reviewing foundations, representations, learning paradigms, adaptive control, architectures, applications, and unresolved challenges. It proposes a structured path involving compression, grounding, adaptation, and agentic learning as VLMs, VLAs, and agentic AI converge toward embodied AGI.

  • Conclusion: The review defines VLAs as multimodal systems that unify visual perception, natural-language understanding, and action generation in physical or simulated environments.It also traces their evolution and major historical milestones.
  • Conclusion: The analysis covers tokenization and representation techniques for visual, linguistic, action, and spatial information.These techniques include action primitives and spatial semantics.
  • Conclusion: The review examines supervised, imitation, reinforcement, and multimodal pretraining, alongside adaptive control, real-time execution, and more than 50 recent VLA architectures.It also surveys datasets and training strategies shaping VLA performance.
  • Conclusion: The proposed path forward addresses inference, safety, bias, generalization, integration, compute, and ethical deployment through compression, grounding, domain adaptation, and agentic learning.The roadmap connects these solutions with the convergence of VLMs, VLA architectures, and agentic AI toward intelligent, embodied, human-aligned agents.

Funding Declaration

The work received support from the NSF and USDA-NIFA AI Institute for Agriculture program, plus an additional USDA/NIFA grant supporting agricultural robotics research.

  • Funding Declaration: The project was supported by NSF and USDA-NIFA through the AI Institute for Agriculture program under awards AWD003473 and AWD004595.The declaration also lists USDA-NIFA Accession Number 1029004 for robotic blossom thinning with soft manipulators.
  • Funding Declaration: Additional support came from USDA/NIFA Grant Number 2024-67022-41788 for expanding UCF AI research into novel agricultural engineering applications.The declaration identifies Accession Number 1031712 and the PARTNER project.
Loading 2505.04769v2…