PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Vision-Language-Action Models for Embodied AI

Yueen Ma, Zixing Song, Yuzheng Zhuang, Jianye Hao, Irwin King

arXiv:2405.14093v8cs.ROcs.CLcs.CV

TL;DR

VLAs address language-conditioned robotic tasks by processing vision and language to generate robot actions, but their rapidly expanding literature requires synthesis. This survey defines and classifies VLAs, reviews their components, control policies, task planners, resources, challenges, and future directions.

  • Problem

    Language-conditioned robotic tasks require models to understand instructions, perceive environments, and generate appropriate actions, motivating multimodal VLA models.

  • Method

    The survey proposes a generalized VLA definition and taxonomy covering components, low-level control policies, and high-level task planners, alongside datasets, simulators, and benchmarks.

  • Results

    The survey reviews LVLAs alongside generalized VLAs and compares their architectures, training strategies, modules, and embodied-AI resources.

  • Takeaways & Limitations

    The taxonomy and resource summary organize the rapidly evolving VLA landscape and identify challenges and future research directions in embodied AI.

  • Takeaways & Limitations

    Action representations involve trade-offs: simple SE(2) actions may be insufficient for complex tasks requiring additional degrees of freedom, such as pouring water.

Abstract

from arXiv · show

Embodied AI is widely recognized as a cornerstone of artificial general intelligence (AGI) because it involves controlling embodied agents to perform tasks in the physical world. Building on the success of large language models (LLMs) and vision-language models (VLMs), a new category of multimodal models -- referred to as vision-language-action (VLA) models -- has emerged to address language-conditioned robotic tasks in embodied AI by leveraging their distinct ability to generate actions. The recent proliferation of VLAs necessitates a comprehensive survey to capture the rapidly evolving landscape. To this end, we present the first survey on VLAs for embodied AI. This work provides a detailed taxonomy of VLAs, organized into three major lines of research. The first line focuses on individual components of VLAs. The second line is dedicated to developing VLA-based control policies adept at predicting low-level actions. The third line comprises high-level task planners capable of decomposing long-horizon tasks into a sequence of subtasks, thereby guiding VLAs to follow more general user instructions. Furthermore, we provide an extensive summary of relevant resources, including datasets, simulators, and benchmarks. Finally, we discuss the challenges facing VLAs and outline promising future directions in embodied AI. A curated repository associated with this survey is available at: https://github.com/yueen-ma/Awesome-VLA.

I. INTRODUCTION

VLAs extend multimodal embodied AI by processing vision and language to generate robot actions for language-conditioned tasks. This survey organizes the field around components, low-level control policies, and high-level task planners, while reviewing resources and future challenges.

  • VLA models: VLAs process vision, language, and action to control physical embodiments in language-conditioned robotic tasks.They use visual representations, language embeddings, and robot-data finetuning to predict actions.
  • Motivation: Conventional reinforcement-learning robot policies addressed limited tasks in controlled environments, motivating more versatile multitask and language-conditioned policies.Diverse skills, environments, and intuitive language interfaces make multitask policy development challenging.
  • Taxonomy: The survey taxonomy covers individual VLA components, low-level control policies, and high-level task planners.Control policies generate actions such as translation and rotation, while task planners decompose long-horizon tasks into subtasks.
  • Related work: The survey is presented as the first comprehensive review focused specifically on emerging VLA models in embodied AI.It complements earlier surveys on robotics foundation models, LLMs in robotics, and real-world robot applications.
  • Resources and future directions: The article summarizes datasets, simulators, and benchmarks and discusses challenges including data scarcity, safety, foundation models, and real-world deployment.Its stated future directions include methods addressing data scarcity and inconsistency.

2) Pretrained Visual Representations:

Pretrained visual representations provide VLAs with information about environmental states, while dynamics-learning objectives model state-action transitions. The survey distinguishes representation strategies and summarizes their use in downstream robotic learning.

  • Pretrained Visual Representations: Vision encoders provide information about object categories, positions, and affordances that directly influences VLA performance.
  • Pretrained Visual Representations: CLIP learns vision-language relationships from 400 million image-text pairs, whereas MAE-style methods reconstruct masked visual patches through self-supervision.
  • Pretrained Visual Representations: Language-conditioned masked reconstruction and language generation improve alignment between visual and language modalities in Voltron.
  • Pretrained Visual Representations: DINOv2 uses self-distillation across different image views, while I-JEPA predicts masked-patch embeddings through a nongenerative approach.
  • Dynamics Learning: Forward dynamics predicts subsequent states, inverse dynamics predicts actions connecting states, and forward prediction is generally more challenging.
  • Dynamics Learning: Vi-PRoM combines contrastive video learning, shuffled-frame recovery, and pseudo-label image classification, demonstrating effectiveness for behavior cloning and PPO.
  • Dynamics Learning: MaskDP reconstructs masked state and action tokens and can be applied directly to downstream tasks in a zero-shot manner.

6) LLM-Induced World Models:

LLM-induced world models use language models to represent world states, generate plans, or guide search, while reasoning methods refine embodied decision-making. The survey contrasts these textual approaches with visual world models and notes distinct roles for world models and chain-of-thought reasoning.

  • LLM-Induced World Models: LLMs provide commonsense knowledge that motivates their use in world modeling and embodied reasoning.
  • LLM-Induced World Models: DECKARD alternates between sampling subgoals from an abstract world model and updating it through game interaction during execution.
  • LLM-Induced World Models: LLM-DM generates PDDL world models, mediates corrective feedback, and uses the resulting symbolic simulator for plan generation.
  • LLM-Induced World Models: RAP combines an LLM policy and world model with Monte Carlo tree search to construct reasoning trees balancing exploration and exploitation.
  • Visual World Models: Visual world models generate future images, videos, or 3D scenes and can provide new trajectories or goal states for embodied agents.
  • Visual World Models: Genie combines a video tokenizer, autoregressive dynamics model, and latent action model to create an interactive environment from unlabeled videos.
  • World Models & Reasoning: World models mainly support low-level control through immediate next-state generation, whereas chain-of-thought methods mainly support text-based task planning.

2) Transformer-Based Control Policies:

Transformer-based control policies organize multimodal observations and instructions into action generation, with extensions for multitask learning, multimodal prompts, 3D inputs, and diffusion-based policies. The surveyed architectures differ in conditioning, representation, action modeling, and training data.

  • Transformer-Based Control Policies: Since Transformers were introduced, language-conditioned robot control policies have converged toward similar Transformer-based architectures.
  • Transformer-Based Control Policies: Hiveformer uses multiview observations and full observation history, extending policies that rely only on the current observation.
  • Transformer-Based Control Policies: Gato uses unified tokenization to train one model across Atari games, image captioning, and block stacking.
  • Transformer-Based Control Policies: RoboCat iteratively fine-tunes and self-generates data to adapt to new tasks with as few as 100 demonstrations.
  • Transformer-Based Control Policies: Q-Transformer uses autoregressive Q-functions and a conservative regularizer, allowing learning from successful demonstrations and failed trajectories.
  • Multimodal Prompts: Multimodal prompts support tasks such as novel-concept grounding, visual goal reaching, and visual constraint satisfaction that can be difficult to express with text alone.
  • 3D Visual Inputs: 3D policies use point clouds, voxels, continuous feature fields, or virtual-view renderings to represent spatial inputs.
  • Diffusion-Based Action Generation: Diffusion Policy models robot control with a DDPM and supports multimodal action distributions, high-dimensional action spaces, and training stability.

6) Diffusion-Based Control Policies with 3D Vision:

This section surveys control-policy designs that use 3D vision, varied action representations, and efficiency-oriented VLA architectures. It also highlights trade-offs between embodiment generalization, dexterity, inference speed, and scaling.

  • Point-Based Actions: Point-based action prediction offers a cost-effective alternative to constructing full VLAs by selecting spatial affordances or keypoints.PIVOT iteratively refines visual proposals, while RoboPoint projects predicted 2D affordance points into 3D using depth.
  • Constraint-Based Control: Constraint-based methods convert 3D keypoints into costs and obtain robot actions by solving constrained optimization problems.ReKep represents manipulation tasks as sequences of constraint functions generated by large vision models and VLMs.
  • VLA Design Directions: VLA research also explores efficient architectures, unsupervised video pretraining, and action experts to improve generalization or reduce deployment costs.Examples include π0’s action expert, RoboMamba’s linear-complexity Mamba backbone, and LAPA’s latent-action pretraining.
  • Action Types and Their Training Objectives: End-effector pose abstraction improves generalization across embodiments but limits dexterity, especially for tasks requiring richer motion.Most low-level policies predict end-effector poses while leaving fine-grained motion to a separate planner.

IV. TASK PLANNERS

Task planners decompose complex instructions into subtasks that guide low-level control policies, enabling VLAs to address longer-horizon embodied tasks. The survey covers monolithic, grounded, and 3D-aware planning approaches.

  • Task Planner Formulation: A high-level planner decomposes a complex task into an ordered sequence of subtasks, each serving as an instruction to a low-level policy.This task or subgoal decomposition is closely related to task and motion planning and embodied decision making.
  • Monolithic Task Planners: Monolithic planners use a single LLM or multimodal LLM to generate task plans, leveraging pretrained knowledge for embodied reasoning.PaLM-E generates text plans from images and language, maps them to executable instructions, and can replan as the environment changes.
  • Task Planners with 3D Vision: 3D-aware planners incorporate point clouds or other spatial features to support embodied reasoning, manipulation, navigation, and task planning.LEO aligns 3D vision and language before vision-language-action instruction tuning, while 3D-LLM and ShapeLLM inject 3D representations into language models.
  • Programmatic Planning: Planning systems may use executable programs, object detectors, APIs, and control policies to connect high-level task descriptions with actions.The illustrated room-cleaning example invokes detection and control components through programmatic steps.
  • Grounded Task Planning: Grounded planning selects subtasks by combining an LLM’s proposed skills with low-level policies’ estimates of whether those skills are executable.SayCan names these components task-grounding and world-grounding, respectively.

B. Modular Task Planners

Modular task planners assemble pretrained LLMs, VLMs, and control components rather than finetuning a single end-to-end model. Language-based designs ease multimodal communication, while code-based designs improve debugging and controllability but require API preparation.

  • Overview: Modular planners reduce the expense of end-to-end embodied-data finetuning by assembling off-the-shelf LLMs and VLMs.They follow a tool-use architecture in which components exchange information through language or callable interfaces.
  • Language-Based Task Planners: Language-based planners exchange multimodal information through natural-language descriptions and can update instructions using environmental or human feedback.Inner monologue dynamically revises low-level instructions in a closed-loop process without requiring additional LLM training.
  • Code-Based Task Planners: Code-based planners generate executable programs that invoke object detectors, VLMs, and control policies through APIs.ProgPrompt uses program-like specifications and assertions, while ChatGPT for robotics iteratively generates code using simulation and user feedback.
  • Strengths and Limitations: Modular planners are more readily deployable than monolithic end-to-end planners, whose substantial training costs can limit practical use.The comparison concerns deployment convenience versus the cost of finetuning large models on specialized embodied data.
  • Strengths and Limitations: Language-based planners may require extra alignment between generated plans and low-level admissible instructions, whereas code-based planners require manually prepared APIs and documentation.Code-based systems offer debugging and greater controllability, but their performance remains constrained by the underlying models’ programming capabilities.

V. DATASETS AND BENCHMARKS

The survey reviews datasets, simulators, and benchmarks for VLAs while emphasizing data scarcity, evaluation gaps, safety, generalization, multimodality, and long-horizon planning challenges.

  • Datasets: Real-world robot data are scarce because collection requires costly equipment, environments, and expert demonstrations across diverse robot configurations.These differences create inconsistencies in sensing and control data.
  • Datasets: Simulation scales data collection but introduces a sim-to-real gap caused by rendering, physics, object, and motion-planning discrepancies.Deformable objects and liquids are especially difficult to simulate, while adding new objects can require 3D scanning and mesh editing.
  • Benchmarks: Benchmarks cover task planning, embodied question answering, spatial reasoning, physics understanding, and world knowledge, but EQA does not directly evaluate manipulation or navigation.Active exploration in EQA requires a simulator, limiting the use of real-world data.
  • Challenges: Comprehensive benchmarks spanning skills, objects, embodiments, and environments remain undeveloped, and metrics beyond success rate are needed for fine-grained diagnosis.The survey also identifies safety guardrails, risk assessment, human-robot interaction protocols, and interpretability as important safety components.
  • Challenges: VLA foundation models still lack generalization comparable to LLMs, while multimodal alignment and additional modalities introduce unresolved design challenges.Audio, haptics, and gaze can benefit embodied applications but increase model complexity.
  • Challenges: Hierarchical long-horizon frameworks increase complexity and failure points, while replanning after execution failures can cause significant latency.Using separate large task-planning and control models may be redundant and hinder scalability.
  • Challenges: Robotics also raises privacy, job-displacement, bias, legal, and social-norm concerns.These issues accompany robots’ direct interaction with the physical world.
  • Scope: The survey organizes VLA research into components, control policies, and task planners, and compiles datasets, simulators, and benchmarks.It presents itself as the first survey covering both LVLAs and generalized VLAs.

APPENDIX

The appendix extends background coverage of vision, language, and reinforcement-learning foundations, including representative architectures, inputs, timelines, and model specifications relevant to VLAs.

  • Unimodal foundations: VLAs commonly rely on existing unimodal models to process vision, language, and action-related inputs.The appendix therefore summarizes representative developments in these component areas.
  • Computer Vision: Vision coverage spans image classification, object detection, segmentation, visual Transformers, and 3D inputs such as depth maps, point clouds, and volumetric data.Detection and segmentation help policies identify and localize objects for robotic learning.
  • Computer Vision: The appendix traces vision-model development from CNNs such as AlexNet and VGG to detection and segmentation systems including R-CNN variants, YOLO, DETR, Segmenter, and SAM.DETR uses encoder-decoder processing, cross-attention, and learnable object queries.
  • Model development: The appendix includes timelines of pivotal unimodal models and their growing scale, noting that GPT-4’s size is estimated because it is undisclosed.It also compares representative VLM specifications and surveys BERT-based, contrastive-pretrained, and large multimodal models.
  • Natural Language Processing: Natural-language coverage describes the transition from early computational-linguistics approaches and recurrent models such as LSTM and GRU toward Transformer-based language modeling.Transformers support contemporary NLP components used for understanding instructions and generating text.

• OSRT [409]

The appendix collects additional VLA-related work, abbreviations, and emerging applications beyond the main text, spanning components, world models, embodiments, and data collection.

  • Components of VLA: Additional component research covers pretraining, imitation learning, self-supervised representations, world models, and multimodal inputs such as audio and tactile data.Examples include VIP, SpawnNet, Holo-Dex, AuRL, T-Dex, MWM, SWIM, Iso-Dream, and TWM.
  • World Models: World-model approaches improve latent dynamics, transfer knowledge from human videos through affordance maps, decouple controllable dynamics, or generate trajectories in latent imagination.These methods target representation learning and long-horizon decision-making.
  • Additional methods: Supplementary work includes structured scene-language representations and action-discretization methods for noisy, multimodal human demonstrations.SceneScript represents layouts and objects autoregressively, while BeT, C-BeT, and VQ-BeT model action trajectories.
  • Recent directions: Recent extensions include automatic language relabeling of robot videos, dexterous action tokenization, in-the-wild co-training, hierarchical navigation, and VLA deployment on humanoid and quadruped robots.These works broaden data sources, skills, navigation, and embodiments.
  • Survey resources: The survey’s supplementary retrieval pipeline combined automated scripting and manual searching for VLA papers published from January 2020 through December 2025.Keyword filtering produced approximately 400 VLA-related papers, with automated errors acknowledged.

• CNRS: CNRS IRL 2010 CROSSING

The appendix provides abbreviation lists for institutions and organizations represented in the survey’s VLA landscape, together with a list of VLAs appearing in Figure 9.

  • Institution abbreviations: Institution abbreviations in Figure 9 span universities, research institutes, companies, laboratories, and collaborative teams.Examples include AI2, Google, MIT, Stanford, Tsinghua, and the Open X-Embodiment Collaboration.
  • Institution abbreviations: The appendix identifies additional organizations including USTC, University of Washington, Physical Intelligence, and Zhejiang University.These entries support interpretation of the affiliations shown in Figure 9.

• ACT [100]

The listed works include Control Your Robot, Counterfactual VLA, and Don’t Blind Your VLA.

  • Control Your Robot is listed as a VLA-related work.
  • Counterfactual VLA is listed as a VLA-related work.
  • Don’t Blind Your VLA is listed as a VLA-related work.

• EyeVLA [560]

The listed works include ImaginationPolicy, Interactive Language, Mind to Hand, and Robotic Assistant.

  • ImaginationPolicy is listed as a VLA-related work.
  • Interactive Language is listed as a VLA-related work.
  • Mind to Hand is listed as a VLA-related work.
  • Robotic Assistant is listed as a VLA-related work.

• RS-CL [699]

The survey supplements standard VLA architectures with world-model integration, additional perception modalities, and video-action modeling, while visualizing development from 2020 to 2025.

  • Recent VLA research expands beyond standard architectures in several key directions.
  • VLA + World Model: WorldVLA, UniVLA, NORA-1.5, RynnVLA-002, and Motus unify VLAs with world models.
  • PLA, MLA: PLA and MLA incorporate perception modalities beyond vision.
  • VAM: Video-action models jointly capture semantics and dynamics during video pretraining for more efficient post-training in low-level robot control.
  • The survey visualizes VLA development from 2020 to 2025 using Figures 9–12.
  • The paper organizes its visual analysis around publication counts, citations, collaboration, and institutional output and impact.
Loading 2405.14093v8…