PaperlayerView exact revision on arXiv

Source-linked AI summary

An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challenges

Chao Xu, Suyu Zhang, Yang Liu, Baigui Sun, Weihong Chen, Bo Xu, Qi Liu, Juncheng Wang, Shujun Wang, Shan Luo, Jan Peters, Athanasios V. Vasilakos, Stefanos Zafeiriou, Jiankang Deng

arXiv:2512.11362v3cs.RO

TL;DR

VLA research faces open-world generalization, long-horizon execution, and safety challenges even as systems advance in closed settings. This survey organizes the field from basic modules and milestones to a review of five challenge areas, current approaches, and future opportunities. It offers a roadmap for navigating and advancing VLA research.

  • Problem

    VLA models can generalize poorly in open, dynamic environments, while long-horizon operations remain unresolved and hazardous behavior is a concern in human-shared settings.

    Source: §4.2.2 · p. 8, §4.3.1 · p. 10, §4.4.1 · p. 13

  • Method

    The survey traces VLA modules and milestones, then reviews approaches and future directions across five challenge areas.

    Source: §1 · p. 2

  • Results

    The survey presents a structured analysis of representation, execution, generalization, safety, and dataset evaluation, reviewing current solutions and future opportunities.

    Source: §5 · p. 16

  • Takeaways & Limitations

    The work aims to serve as a roadmap for newcomers and experienced researchers to navigate and advance embodied intelligence.

    Source: §1 · p. 2, §5 · p. 16

Abstract

from arXiv · show

Vision-Language-Action (VLA) models are driving a revolution in robotics, enabling machines to understand instructions and interact with the physical world. This field is exploding with new models and datasets, making it both exciting and challenging to keep pace with. This survey offers a clear and structured guide to the VLA landscape. We design it to follow the natural learning path of a researcher: we start with the basic Modules of any VLA model, trace the history through key Milestones, and then dive deep into the core Challenges that define recent research frontier. Our main contribution is a detailed breakdown of the five biggest challenges in: (1) Representation, (2) Execution, (3) Generalization, (4) Safety, and (5) Dataset and Evaluation. This structure mirrors the developmental roadmap of a generalist agent: establishing the fundamental perception-action loop, scaling capabilities across diverse embodiments and environments, and finally ensuring trustworthy deployment-all supported by the essential data infrastructure. For each of them, we review existing approaches and highlight future opportunities. We position this paper as both a foundational guide for newcomers and a strategic roadmap for experienced researchers, with the dual aim of accelerating learning and inspiring new ideas in embodied intelligence. A live version of this survey, with continuous updates, is maintained on our \href{https://suyuz1.github.io/VLA-Survey-Anatomy/}{project page}.

1 INTRODUCTION

The survey addresses gaps in how VLA research is organized: challenges are often secondary, and methods are presented without a clear learning trajectory. It centers a structured analysis of five challenges within a progression from VLA modules, through milestones, to challenges.

1 INTRODUCTION

  • Existing surveys often append challenges and group methods separately, leaving researchers without a unified, progressive account of the field.The survey identifies a need to compare solution paths and chart future directions, rather than merely listing problems.Source: §1 · p. 2
  • The survey makes its analysis of five VLA challenges the central pillar, reviewing competing solutions and outlining future research avenues.The challenges span multimodal alignment and physical-world modeling; execution; generalization; safety; and data and benchmarking.Source: §1 · p. 2
  • Its learning-oriented structure moves from foundational Modules to historical Milestones and then core Challenges, serving both newcomers and experienced researchers.The figure describes this progression as a pyramid, with applications depicted in Appendix A.1.Source: Figure 1 · p. 1, §1 · p. 2

2 BASIC MODULES

A VLA system links perception, multimodal reasoning and planning, and action. The survey outlines common choices for each module, from visual and language encoders to action representations and decoding strategies.

2.1 Overall and Architectural Trend

  • A VLA system comprises perception, a brain for multimodal fusion and planning, and an action module; current designs increasingly use language-aligned perception, pretrained VLMs, and continuous generative action.The overview is intentionally streamlined to prioritize the survey’s challenge analysis.Source: §2.1 · p. 2

2.2.1 Vision Encoders in VLA

  • Visual encoders include CNNs and ViTs, language-supervised and self-supervised features, hybrid encoders, and pretrained VLMs.Language-supervised encoders provide semantic alignment, while self-supervised features capture geometry; hybrid designs combine these strengths.Source: §2.2.1 · pp. 2–3

2.2.2 Language Encoders in VLA

  • Language encoders range from text-only Transformers and LLMs to native VLMs jointly pretrained with vision.The VLM approach makes language encoding part of multimodal understanding rather than a standalone component.Source: §2.2.2 · p. 3

2.2.3 Proprioceptive Encoders in VLA

  • Low-dimensional proprioceptive states are typically encoded with efficient MLPs and fused with vision and language.Inputs can include joint and end-effector states and gripper status.Source: §2.2.3 · p. 3

2.3 Robot Brain

  • The robot brain fuses input representations, reasons and plans, and generates action intentions using Transformer, diffusion, hybrid, or VLM-centered architectures.Hybrid systems combine Transformer-based semantic reasoning with generative heads for smooth, high-frequency control.Source: §2.3 · pp. 3–4

2.4 Robot Action

  • The action module translates the brain’s abstract decisions into low-level commands, shaping precision, smoothness, real-time performance, and generalization.Source: §2.4 · p. 4

2.4.1 Action Representation

  • Action representations may be discrete, continuous, or hybrid, trading off learnability against control precision and combining encodings across control facets.Hierarchical systems may represent high-level skills discretely and low-level execution continuously.Source: §2.4.1 · p. 4

2.4.2 Action Decoding

  • Action decoding may be autoregressive, non-autoregressive, or chunked, balancing temporal conditioning with lower latency and efficient local rollout.Chunked decoding is autoregressive across coarse time but parallel within each action chunk.Source: §2.4.2 · p. 4

3 EVOLUTION & MILESTONES

VLA research has progressed from early language-grounded embodied tasks toward unified, larger-scale models and increasingly diverse approaches. The survey traces milestones across models, datasets, and evaluation benchmarks from 2022 to 2025.

3 EVOLUTION & MILESTONES

  • Early work from 2017 to 2019 established language-grounded navigation and closed perception–action-loop tasks, with subsequent work adding language-to-action learning and 3D reasoning.Source: §3 · p. 4
  • From 2020 to 2021, benchmarks emphasized long-horizon, interactive tasks, while CLIPort showed zero-shot generalization in robotic manipulation using pretrained visual representations.Source: §3 · p. 4
  • Since 2022, the field introduced hierarchical planning, continuous multimodal feedback, and end-to-end vision-and-language-to-action Transformer models.The survey presents this period as the beginning of large models and generalized learning.Source: §3 · pp. 4–5
  • In 2023, milestones included unified multimodal inputs, diffusion-based action generation, and large, diverse cross-robot datasets.Source: §3 · p. 5
  • In 2024, work broadened across open-source scaling, generalist policies, flow-matching action generation, web-scale video pretraining, and 3D world modeling.Source: §3 · p. 5
  • By 2025, models explored full-body humanoid control, open-world reasoning, visual chain-of-thought, unified hierarchy and control, online reinforcement learning, and scaling interaction data.The survey characterizes this as pluralistic evolution across embodiments, modalities, and learning paradigms.Source: §3 · p. 5

4 CHALLENGES & SOLUTIONS & FUTURE DIREC-

This section surveys VLA challenges and approaches spanning physical grounding, planning and execution, open-world generalization, continual learning, sim-to-real transfer, and online reinforcement learning. It highlights directions toward unified, adaptive agents that transfer across embodiments, explore, and evolve through deployment.

4.1.1 The GAP between Semantics, Perception, and Physical Interaction

  • VLA systems must bridge gaps between vision and language, semantics and action, and additional sensory modalities needed for precise physical interaction.Methods align modalities with language-conditioned features or symbolic reasoning, map language to action tokens or intermediate plans, and fuse tactile, force, or audio inputs through specialized encoders and fusion strategies.Source: §4.1.1 · p. 6

4.1.2 From 2D Images to Spatial-Temporal Representations

  • Because 2D-pretrained VLMs lack sufficient spatial reasoning, VLA systems add depth, point clouds, semantic 3D structures, or temporal representations and integrate them while preserving pretrained alignment.Architectures inject or project geometry, use external modules, or reproject 3D data into 2D; these strategies aim to retain 2D pretraining strengths alongside spatial awareness.Source: §4.1.2 · pp. 6–7

4.1.3 Dynamic and Predictive World Models

  • Predictive world models can support counterfactual reasoning and long-horizon planning, but future systems should unite multimodal inputs with geometry, dynamics, semantics, affordances, and physics-aware simulation.Approaches predict future states in pixel or latent space and use those predictions to condition policies or evaluate candidate plans; the proposed hybrid model would simulate outcomes such as motion, contact, and stability.Source: §4.1.3 · p. 7, §4.1.4 · p. 8

4.2.1 Parsing Complex Instructions

  • Mixed-modality and underspecified instructions can cause errors to propagate through perception, planning, and control, motivating interleaved image-text inputs and ambiguity handling.Methods parse images and text in a shared sequence, infer or verify intended targets, fill missing spatial cues, or ask users for clarification before acting.Source: §4.2.1 · p. 8

4.2.2 Hierarchical Planning and Task Decomposition

  • Long-horizon tasks are addressed through language-driven plans, multimodal intermediates, and reusable skill libraries that decompose high-level intent into executable steps.Examples include textual subgoals, visual goals or affordances, and explicit or learned atomic skills; Agentic Robot [165] produces 2–5 verifiable steps before execution.Source: §4.2.2 · pp. 8–9

4.2.3 Error Detection and Autonomous Recovery

  • Long-horizon execution is vulnerable to interruptions, perception drift, and actuation failures, so recovery methods use human feedback or autonomous error detection and correction.Examples include reactive language corrections, proactive clarification, self-generated corrective actions, and plan–act–verify loops that trigger recovery when a subgoal fails.Source: §4.2.3 · p. 9

4.2.5 Future Directions

  • Future VLA architectures should adapt reasoning depth to task complexity and combine rapid action with deeper planning in a unified decision stream.The proposed direction also shifts from open-loop behavior toward self-aware, closed-loop autonomy that can replan or adjust reflexively and recover without intervention.Source: §4.2.5 · p. 10

4.3.1 Open-World Generalization

  • Open-world generalization remains difficult because imitation datasets miss long-tail real scenes; approaches transfer knowledge from multi-robot, human-video, and internet-scale data.Octo [49] pretrains on about 800k robot trajectories, while Dita [199] adapts with as few as 10 real demonstrations; in-context learning also enables retrain-free adaptation from prompted examples.Source: §4.3.1 · pp. 10–11
  • Open-world generalization uses compositional and semantic representations, lightweight output steering, and runtime verification to combine skills, manipulate unseen objects, and adapt to new environments.TRA [204] uses temporal contrastive loss; ObjectVLA [205] jointly trains on robot trajectories and box-labeled language-vision data; LERF combines CLIP with 3D NeRFs. Align-Then-Steer [207] adapts frozen outputs with a latent-space adapter, while RUM supports runtime verification and retries.Source: §4.3.1 · p. 11
  • Data augmentation and architectural design improve generalization through diverse scenes, semantic task variants, hierarchical planning, multimodal fusion, and varied plausible action structures.CACTI [209] and GenAug [210] expand visual diversity without additional robot rollouts; ROSIE [211] distills VLM knowledge. Hierarchical designs separate planning from execution [212], BAKU dynamically fuses sensor inputs, and StructDiffusion [214] generates language-guided action structures robust to unseen object sizes and shapes.Source: §4.3.1 · pp. 11–12

4.3.2 Continual Learning and Incremental Skill Acquisition

  • Continual learning mitigates forgetting through parameter isolation or expansion and replay of selected past examples while acquiring new skills.Prompt- and codebook-based methods add entries without modifying existing components [14],; InstructVLA [62] routes between reasoning and action modules, and iManip adds skill-specific weights while freezing old ones. ExpReS-VLA [217] compresses replay, while iManip replays critical frames.Source: §4.3.2 · p. 12

4.3.3 Sim-to-Real Gap in Deployment

  • Sim-to-real transfer is hindered by differences in dynamics and perception; approaches improve simulator fidelity, learn robust intermediate representations, or use real-world data and learned world models.ManiSkill3 [219] combines GPU-parallel rendering, domain randomization, and background composition for zero-shot transfer, while SLIM [218] uses segmentation and depth maps. GenAug [210] synthesizes scenes from real demonstrations; DreamGen [220] learns dynamics from real data, and RynnVLA-001 [221] combines video-generation pretraining with human-centric trajectory perception.Source: §4.3.3 · p. 12

4.3.4 Online Interaction and Reinforcement Learning

  • Online RL addresses imitation learning’s distributional-shift and performance limits, but VLA-scale exploration is sample-inefficient and reward design is difficult; methods leverage VLA priors and automate reward generation.RLDG [223] distills specialist RL trajectories, Refined Policy Distillation [225] constrains RL with VLA action distributions, iRe-VLA [222] alternates lightweight-head RL with supervised fine-tuning, and CO-RFT [224] uses chunked temporal-difference learning. Automated rewards include visual or video-language similarity [228],, affordance guidance [230], VLM-based trajectory preferences [231], [226], and LLM-generated rewards [232], [233], [36].Source: §4.3.4 · pp. 12–13

4.3.5 Future Directions

  • Future VLAs should disentangle semantic planning from proprioceptive control to enable zero-shot transfer across embodiments and evolve through self-supervised exploration and online reinforcement learning.The proposed closed loop, “Deployment →Discovery →Evolution,” would let agents identify knowledge gaps, generate data in the wild, and continuously refine their world models and capabilities.Source: §4.3.5 · p. 13

4.4 Safety, Interpretability and Reliable Interaction

The survey treats trustworthy VLA interaction as requiring both safety assurance and interpretable, predictable behavior. It also examines how heterogeneous data and limited evaluation standards constrain progress, alongside future directions for both challenges.

4.4.1 Reliability and Safety Assurance

  • Safety work ranges from explicit constraints (AutoRT [234], SafeVLA) to learned alignment and uncertainty-aware pausing or replanning (Gemini Robotics [114], GPI [154]).The survey describes risk from perception errors, generalization failures, and instruction misinterpretation in human-shared environments.Source: §4.4.1 · p. 13

4.4.2 Interpretability and Trustworthy Interaction

  • Interpretability efforts expose reasoning through editable text, visual subgoals, or hierarchical plans, while structured prompts and task rollback target predictable interaction.The approaches aim to make decision processes understandable and robot behavior easier to anticipate during collaboration.Source: §4.4.2 · p. 14

4.4.3 Future Directions

  • The survey calls for intrinsic uncertainty awareness and actionable, pre-action transparency so agents can pause, seek clarification, replan, and accept human intervention.It argues that reactive guardrails separated from the policy can fail to prevent hallucinations or confident but wrong actions in real time.Source: §4.4.3 · p. 14

4.5 Data Construction and Benchmarking Standards

  • The data and benchmarking challenge spans unifying diverse training resources and standardizing increasingly complex assessment protocols.Data heterogeneity includes simulation versus real-world sources and differing robot embodiments and control interfaces; benchmarks also lack unified standards and frontier-generalization tests.Source: Figure 8 · p. 14, §4.5.1 · p. 14, §4.5.2 · p. 15

4.5.1 Multi-Source Heterogeneous Data

  • Data unification approaches align heterogeneous behaviors and modalities in shared representations, while augmentation and mixture optimization improve the use of raw datasets.Examples include latent action representations, generative augmentation, and performance-feedback adjustments to dataset sampling weights.Source: §4.5.1 · p. 15
  • Data standardization spans synchronized collection protocols, standardized formats and simulation environments, and cross-dataset benchmarks such as Open X-Embodiment (OXE).RoboMM [132] adds three-level semantic alignment for joint training across datasets.Source: §4.5.1 · p. 15

4.5.2 Evaluation Benchmarks

  • Evaluation work targets unified protocols, longer-horizon and lifelong-learning tasks, and tests of frontier generalization and reasoning.The survey notes that current benchmarks often use simple short-horizon tasks and lack systematic probes of frontier generalization.Source: §4.5.2 · p. 15

4.5.3 Future Directions

  • Future directions pair simulation-first data generation with failure-based learning and diagnostic stress tests measuring task completion, safety margins, efficiency, and perturbation resilience.The proposed role for real-world data is to calibrate simulator physics and rendering fidelity rather than serve as the primary training source.Source: §4.5.3 · p. 16

5 CONCLUSION

The conclusion presents the survey as a roadmap from VLA fundamentals and historical milestones to five research challenges. It reviews current solutions and future opportunities to help readers navigate and advance embodied intelligence.

5 CONCLUSION

  • The survey organizes VLA research from foundational modules and milestones through five challenges: representation, execution, generalization, safety, and dataset evaluation.It aims to serve as a foundational roadmap for newcomers and experienced researchers.Source: §5 · p. 16

A.1 Applications

The paper frames household robotics and industrial automation as two domains where VLA models’ real-world problem-solving ability can be assessed.

A.1 Applications

  • The survey examines VLA impact in household robotics and industrial automation, where models translate abstract human intent into grounded, executable actions through perception, reasoning, and control.It presents large-scale pretraining as supporting generalization and adaptation in these domains.Source: §A.1 · p. 22

A.1.1 Embodied Manipulation and Household Robotics

Households test VLA systems with natural-language requests, diverse unseen objects, and complex long-horizon tasks. The survey identifies personalization, collaboration, and low-power on-device operation as future directions.

A.1.1 Embodied Manipulation and Household Robotics

  • Unstructured, dynamic, human-centric homes require robots to understand natural language, handle diverse unseen objects, and perform complex long-horizon tasks.Source: §A.1.1 · p. 22
  • Internet-scale knowledge supports interaction with varied household items without task-specific training, while hierarchical reasoning can decompose vague commands into executable subtasks.The survey cites benchmark progression from ALFRED to real-world validations such as ChatVLA as evidence of sequential-reasoning robustness.Source: §A.1.1 · p. 22
  • Future household systems are envisioned as personalized, collaborative assistants that learn user preferences and operate on low-power, on-device hardware.The proposed interactive learning includes feedback, clarification questions, and proactive suggestions based on learned routines.Source: §A.1.1 · p. 22

A.1.2 Industrial and Field Robotics

Industrial VLA work addresses production environments’ stricter demands for precision, reliability, and safety. The survey describes current development directions and future priorities for bridging flexible intelligence with those demands.

Industrial setting

  • Industrial environments demand greater precision, reliability, and safety than household scenarios, prompting a focus on physical intelligence and robustness.Source: §A.1.2 · p. 22

Current directions

  • Industrial VLA development targets physical perception with tactile and force sensors, industrial-grade reasoning, and safety mechanisms such as safe reinforcement learning.Examples include Tactile-VLA [121] and VTLA [43] for sensing, ForceVLA [25] and CogACT [263] for reasoning, and SafeVLA for safety.Source: §A.1.2 · p. 22

Future directions

  • Certification-ready safety and formal verification are proposed to move industrial VLA beyond empirical safety toward provable guarantees on robot behavior.The survey also identifies zero-shot adaptation to new tasks and parts, and multi-agent coordination, as future directions.Source: §A.1.2 · p. 22
  • Customized manufacturing may require learning new tasks and parts from CAD files, technical manuals, and human-worker video demonstrations rather than reprogramming for each product.Source: §A.1.2 · p. 22

A.2.1 Training Strategy

The survey groups VLA training into behavioral cloning, predictive modeling, and reinforcement learning, which are often combined in practice. It describes BC as dominant and notes RL approaches that build on BC-pretrained models.

Behavioral Cloning (BC)

  • Behavioral cloning is the dominant VLA training paradigm, using supervised imitation to map multimodal observations to expert actions.It appears across diffusion controllers and Transformer generalists, and also serves as a pre- or post-training stage in hierarchical systems.Source: §A.2.1 · p. 22

Predictive Modeling

  • Predictive modeling learns future observations or latent dynamics instead of imitating actions, providing self-supervised signals for learning physics and causality.WorldVLA [80], LUMOS, and UVA [264] use predictive objectives; LAPA [265] learns discrete latent actions from unlabeled video.Source: §A.2.1 · p. 22

Reinforcement Learning (RL)

  • Reinforcement learning optimizes policies through interaction and reward feedback, often building on BC-pretrained backbones to improve robustness and long-horizon performance.On-policy methods update from fresh rollouts, while off-policy methods use replay for sample efficiency; HIL-SERL adds human demonstrations and online corrections.Source: §A.2.1 · pp. 22–23
  • π∗0.6 introduces RE-CAP, which uses advantage-conditioned policy extraction with flow-matching or diffusion VLAs for scalable training without complex objectives such as PPO.Source: §A.2.1 · p. 23

A.2.2 Dateset

The survey presents embodied datasets as complementary sources of supervision whose scale, diversity, and quality matter for VLA generalization and robustness. It groups them by setting and research role, from simulation and real-robot data to human-centered and question-answering data.

Simulation-Centered Datasets

  • Simulation-centered datasets support large-scale, safe, reproducible data generation with full state access, making them suited to reasoning and long-horizon planning.ALFRED provides 25,000 language-grounded household tasks; LIBERO targets lifelong learning, while VLA-3D [273] pairs language instructions with 3D scenes.Source: §A.2.2 · p. 23

Real-World Robotic Manipulation Datasets

  • Real-world robotic manipulation datasets capture sensing, dynamics, and environmental variability, supporting policies intended to remain robust under uncertainty and generalize to unstructured settings.Examples range from DROID’s [274] more than 350K trajectories across 50+ environments to OXE’s aggregation of over 60 datasets across 22 embodiments.Source: §A.2.2 · p. 23

Human-Centric and Egocentric Datasets

  • Human-centric and egocentric datasets typically lack robotic action labels but provide first-person data for grounding perception in human experience and inferring intent.Ego4D supports visual-representation pretraining for human-object interaction, while TEACH focuses on dialogue-driven task execution.Source: §A.2.2 · p. 23

Embodied Visual Question Answering Datasets

  • Embodied visual question-answering datasets pair scenes with language-based QA supervision for semantic alignment and environment-level reasoning.MT-EQA requires active exploration, EgoTaskQA covers several reasoning types, and EmbodiedEval [280] includes 328 tasks across 125 scenes.Source: §A.2.2 · p. 23

A.2.3 Evaluation

The survey organizes embodied benchmarks by the capabilities they evaluate, ranging from language-conditioned manipulation to long-horizon interaction and advanced cognition. It also describes benchmarks for assessing holistic capabilities of embodied foundation models.

Language-Conditioned Manipulation

  • Language-conditioned manipulation benchmarks test instruction following and precise actions, using settings such as RLBench, the ManiSkill series [282] [283] [284], and RoboMimic [285].Their environments and protocols support evaluation of imitation, reinforcement learning, multi-task manipulation, generalization, and offline learning.Source: §A.2.3 · p. 23

Long-Horizon and Interactive Task Completion

  • Long-horizon and interactive benchmarks assess sequential reasoning, memory, and sustained interaction through compositional tasks, unseen environments, and dialogue-based clarification or recovery.Examples include ALFRED, CALVIN, and TEACH.Source: §A.2.3 · p. 24

Advanced Cognitive Capabilities

  • Advanced-cognition benchmarks probe lifelong learning and physical reasoning, including skill transfer and retention in LIBERO and interaction in unstable, cluttered scenes in RoboCAS [286].Source: §A.2.3 · p. 24

Evaluation of Embodied Foundation Models

  • Embodied foundation-model benchmarks evaluate broader capabilities, including end-to-end competence in EmbodiedBench [287], world-model physical realism in EWMBench [288], and cross-robot transfer in RoboTwin [289].The benchmark overview table presents task type, evaluation metrics, and environment or platform as comparison facets.Source: §A.2.3 · p. 24, Table S2 · p. 26
Loading 2512.11362v3…