PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Vision-Language-Action Models for Embodied AI

Yueen Ma, Zixing Song, Yuzheng Zhuang, Jianye Hao, Irwin King

arXiv:2405.14093v8cs.ROcs.CLcs.CV

TL;DR

The rapidly evolving VLA landscape calls for a comprehensive survey in embodied AI. This paper provides a taxonomy of VLA research and related resources, offering a high-level overview of LVLAs and generalized VLAs.

  • Problem

    The rapidly evolving VLA landscape creates a need for a comprehensive survey of research in embodied AI.

    Source: §VII · p. 17

  • Method

    The survey organizes VLA research into components, low-level control policies, and high-level task planners, and compares technical details while summarizing datasets, simulators, and benchmarks.

    Source: §VII · p. 17

  • Results

    The survey provides a high-level overview of LVLAs and generalized VLAs, alongside resources for training and evaluating them.

    Source: §VII · p. 17

  • Takeaways & Limitations

    The survey aims to capture embodied AI's evolving landscape and inspire future research.

    Source: §VII · p. 17

    Action representations involve task-dependent tradeoffs: SE(2) actions suffice for many tabletop tasks, while more complex tasks may require SE(3).

    Source: 10) Strengths and Limitations · p. 12

Abstract

from arXiv · show

Embodied AI is widely recognized as a cornerstone of artificial general intelligence (AGI) because it involves controlling embodied agents to perform tasks in the physical world. Building on the success of large language models (LLMs) and vision-language models (VLMs), a new category of multimodal models -- referred to as vision-language-action (VLA) models -- has emerged to address language-conditioned robotic tasks in embodied AI by leveraging their distinct ability to generate actions. The recent proliferation of VLAs necessitates a comprehensive survey to capture the rapidly evolving landscape. To this end, we present the first survey on VLAs for embodied AI. This work provides a detailed taxonomy of VLAs, organized into three major lines of research. The first line focuses on individual components of VLAs. The second line is dedicated to developing VLA-based control policies adept at predicting low-level actions. The third line comprises high-level task planners capable of decomposing long-horizon tasks into a sequence of subtasks, thereby guiding VLAs to follow more general user instructions. Furthermore, we provide an extensive summary of relevant resources, including datasets, simulators, and benchmarks. Finally, we discuss the challenges facing VLAs and outline promising future directions in embodied AI. A curated repository associated with this survey is available at: https://github.com/yueen-ma/Awesome-VLA.

I. INTRODUCTION … 1) Reinforcement Learning:

This survey defines VLAs as models that process vision and language to produce robot actions, and organizes the field across components, low-level control, and high-level planning. It reviews prior work, resources, challenges, and future directions for embodied AI.

I. INTRODUCTION

  • VLAs are presented as versatile policies for complex environments, addressing the need for multitask, language-conditioned robots beyond conventional RL policies focused on limited tasks in controlled settings.The survey identifies factories and household tasks as settings for VLA versatility.Source: §I · p. 1

A. Components of VLA

  • VLA components draw on computer vision, NLP, and RL: pretrained vision encoders produce visual representations, while vision-language alignment and robot-data finetuning enable action decoding.The cited alignment approaches include Flamingo, BLIP-2, and LLaVA.Source: §I · p. 2, A. Components of VLA · p. 3

I. INTRODUCTION

  • VLAs process vision and language to produce robot actions for embodied tasks; the survey distinguishes these broadly defined models from the original LLM- or large-VLM-based “Large VLAs.”Language-conditioned robotic tasks require understanding instructions, perceiving the environment, and generating appropriate actions.Source: §I · pp. 1–2

A. Related Work

  • To the authors’ knowledge, this is the first survey focused on emerging VLA models, complementing broader surveys of foundation models, LLMs, and real-world robotics.The survey distinguishes its emphasis on VLAs from earlier embodied-AI reviews.Source: A. Related Work · p. 2

B. Contributions

  • The survey reviews VLA components, architectures, training objectives, and tasks, presents a control-policy/task-planner taxonomy, summarizes datasets and benchmarks, and outlines challenges and future directions.Highlighted challenges and opportunities include data scarcity, safety, foundation models, and real-world deployment.Source: B. Contributions · p. 2

II. BACKGROUND

  • Embodied AI centers on active physical interaction; robot learning is often framed as RL over state-action-reward trajectories, with imitation learning used when rewards are difficult to define.Language-conditioned policies use instructions to select tasks or skills.Source: §II · pp. 2–3

1) Reinforcement Learning:

  • RL’s sequence-modeling lineage includes DT and TT, extends to multimodal, multitask Gato, and includes π*0.6, which uses RL to learn from experience.DQN demonstrated policy learning directly from high-dimensional pixel inputs, while RL trajectories naturally fit sequence modeling.Source: 1) Reinforcement Learning · p. 3
  • RL and LLMs support embodied AI in both directions: RLHF and skill-based RL address preference alignment, sparse rewards, and safety, while LLMs provide verbal feedback and design reward functions.SEED uses RLHF with skill-based RL; Reflexion uses linguistic feedback, and Eureka generates reward functions reported to outperform expert human-engineered ones.Source: 1) Reinforcement Learning · p. 3

2) Pretrained Visual Representations: … 5) World Models:

These sections survey visual and video representations, dynamics learning, and world models as complementary approaches for equipping VLAs with useful perceptual and predictive capabilities. They describe methods spanning pretrained visual features, action-conditioned dynamics objectives, and models that support planning through imagined experience.

2) Pretrained Visual Representations:

  • Visual encoders provide state information such as object categories, positions, and affordances, while CLIP learns image-text alignment from 400 million WIT pairs.CLIP identifies the correct text-image pair among candidates in a batch.Source: 2) Pretrained Visual Representations · p. 3
  • R3M learns temporal proximity and video-language alignment, while VIP also uses temporal relationships to enrich pretrained visual representations.R3M’s time-contrastive objective brings temporally proximal frames together and separates distant frames; its alignment objective checks whether a video matches an instruction.Source: 2) Pretrained Visual Representations · p. 3
  • MVP uses masked-patch reconstruction, RPT also reconstructs actions and proprioceptive states, and Voltron combines language-conditioned reconstruction with language generation.Voltron alternates between language-conditioned masked-image reconstruction and generating language from masked images to align vision and language.Source: 2) Pretrained Visual Representations · p. 4
  • VC-1 explores ViT configurations, DINOv2 exceeds MAE, I-JEPA predicts masked-patch embeddings, and Theia distills vision models into a compact representation.Theia fuses segmentation, depth, and semantic information, outperforming prior PVRs with less data and a smaller model; I-JEPA is nongenerative and uses masked rather than cropped patches.Source: 2) Pretrained Visual Representations · p. 4

3) Video Representations:

  • Video representations can concatenate per-frame PVRs, while multiview video enables temporal-contrastive and masked-autoencoding methods, 3D scene representations, and audio cues for robot policies.NeRF-derived 3D information is used in F3RM and 3D-LLM; 3D Gaussian Splatting, surpasses NeRF in visual quality and rendering speed.Source: 3) Video Representations · p. 4

4) Dynamics Learning:

  • Dynamics learning trains models to predict next states, infer actions between states, or reorder shuffled state sequences; Vi-PRoM combines contrastive and supervised objectives.Vi-PRoM’s supervised tasks recover shuffled video frames and classify images using pseudo labels; comparisons with prior pretraining methods show effectiveness for BC and PPO. MaskDP masks state and action tokens for reconstruction and can be applied to downstream tasks zero-shot.Source: 4) Dynamics Learning · pp. 4–5
  • MIDAS, SMART, and PACT model action-state transitions, while VPT bootstraps labeled internet video and GR-1 predicts future frames.SMART combines next-latent-state and previous-action prediction with masked hindsight control to capture short- and long-term dependencies; PACT autoregressively predicts state and action tokens and can be fine-tuned for downstream tasks.Source: 4) Dynamics Learning · p. 5

5) World Models:

  • World models predict future states and enable planning in imagined space; Dreamer-family models learn latent dynamics, while IRIS trains policies on imagined trajectories and TWM explores Transformer world models.Dreamer uses representation, transition, and reward models with actor-critic learning through imagination; DayDreamer applies the approach to real-world physical-robot tasks.Source: 5) World Models · p. 5

6) LLM-Induced World Models: … 1) Non-Transformer Control Policies:

The surveyed approaches use language or visual models to represent and reason about the world, steer decisions, and build low-level robotic control policies. The sections also describe how these approaches differ in their representations, mechanisms, and intended roles.

6) LLM-Induced World Models:

  • LLM-induced world models support crafting, symbolic planning, and search: DECKARD, LLM-DM, RAP, and LLM-MCTS use distinct model-and-policy mechanisms.DECKARD alternates between sampling subgoals from abstract world models and updating them through Minecraft interactions; LLM-DM generates PDDL models that serve as symbolic simulators. RAP uses MCTS with an LLM as policy and transition model, while LLM-MCTS uses it to initialize POMDP beliefs and guide action selection; Tree-Planner generates diverse action-tree paths with one LLM prompt.Source: 6) LLM-Induced World Models · p. 6

7) Visual World Models:

  • Visual world models generate future images, videos, or 3D scenes and can provide trajectories, simulated experience, or robot-control goals.Genie learns an interactive environment from unlabeled video; 3D-VLA generates image or point-cloud goal states from visual inputs; UniSim simulates outcomes of high- and low-level actions for agent training.Source: 7) Visual World Models · p. 6

8) Reasoning:

  • CoT supports high-level planning and low-level control, while ECoT adds embodied reasoning before action prediction to improve generalization without additional robot data.ThinkBot recovers missing action descriptions, ReAct interleaves reasoning and actions, RAT uses retrieval to mitigate hallucinations, and Tree-Planner uses tree-of-thoughts. ECoT trains OpenVLA to reason about plans, subtasks, motions, and visual features; CoT-VLA introduces visual CoT reasoning.Source: 8) Reasoning · p. 6

9) Policy Steering:

  • Test-time policy steering can improve VLA performance without retraining by reranking actions with learned values or selecting among samples with a VLM verifier.V-GPS uses a learned value function; RoboMonkey uses a VLM-based verifier.Source: 9) Policy Steering · p. 6

10) Strengths and Limitations:

  • The survey contrasts visual-representation and dynamics tradeoffs, and notes that world models mainly support control while text-based CoT suits task planning.PVRs provide pixel-level detail useful for precise manipulation; DINOv2 combines pixel- and image-level features, I-JEPA captures low-level features, and Theia distills vision models. Forward dynamics are harder than inverse dynamics, and their difficulty can yield greater performance gains, as in SMART; inverse dynamics can label state-only datasets. World models generate immediate next states from low-level actions, whereas CoT refines text-based plans.Source: 10) Strengths and Limitations · pp. 6–7

B. Low-Level Control Policies:

  • A low-level control policy combines an action decoder with vision and language perception modules to execute language instructions.The section also calls these policies low-level controllers or action primitives and surveys their modules and architectures.Source: B. Low-Level Control Policies · p. 7

1) Non-Transformer Control Policies:

  • The cited approaches demonstrate language-conditioned pick-and-place (CLIPort), zero-shot generalization to unseen tasks (BC-Z), unstructured-demonstration learning (MCIL), hierarchical plans (HULC), and text-conditioned video generation (UniPi).HULC++ adds a self-supervised affordance model to guide HULC toward actionable regions specified by language instructions; UniPi extracts actions from generated video frames through inverse dynamics.Source: 1) Non-Transformer Control Policies · p. 7

2) Transformer-Based Control Policies: … (a) FiLM

The sections survey Transformer-based, multimodal, 3D-vision, and diffusion control policies, highlighting varied approaches to task conditioning, observation representations, and action generation.

2) Transformer-Based Control Policies:

  • Gato uses unified tokenization to train one parameter set across Atari, image captioning, and block stacking, while Astra adds trajectory attention.The survey notes that the shared tokenization harmonizes inputs and outputs across tasks and domains.Source: 2) Transformer-Based Control Policies · p. 9
  • With as few as 100 demonstrations, RoboCat self-improves through iterative finetuning and generated data, while Q-Transformer learns from successful and failed trajectories.Q-Transformer adds a conservative regularizer to its Q-learning objective to keep maximum-value actions in-distribution.Source: 2) Transformer-Based Control Policies · p. 9
  • RT-Trajectory conditions on transferable end-effector trajectory sketches, whereas ACT predicts action chunks and RoboAgent extends chunking for temporal consistency.Trajectory sketches can be manually drawn, extracted from demonstration videos, or generated by foundation models; RoboAgent also adds inpainting-based semantic augmentation.Source: 2) Transformer-Based Control Policies · p. 9

3) Control Policies for Multimodal Instructions:

  • VIMA uses multimodal prompts for tasks difficult to specify in text alone, while MOO extends RT-1 with object localization and pointing or clicking prompts.VIMA-Bench evaluates placement, novel combinatorial, novel object, and novel task generalization; MOO uses OWL-ViT and expanded training data.Source: 3) Control Policies for Multimodal… · p. 9

4) Control Policies with 3D Vision:

  • 3D control policies use point clouds, voxels, or learned 3D features: PerAct maps voxel observations to gripper target voxels, while Act3D uses adaptive-resolution feature fields.DP3 and 3D Diffuser Actor use point clouds; VER voxelizes multiview images, RoboUniView adds 3D occupancy-pretrained features, and RVT/RVT-2, re-render point clouds from virtual views.Source: 4) Control Policies with 3D Vision · p. 9

5) Diffusion-Based Control Policies:

  • Diffusion Policy formulates robot control as a DDPM with receding-horizon control, visual conditioning, and a time-series diffusion Transformer, supporting multimodal actions and high-dimensional spaces.The survey also identifies training stability as a strength of this visuomotor policy.Source: 5) Diffusion-Based Control Policies · p. 9, (a) FiLM · p. 10

(a) FiLM

  • SUDD uses an LLM and robot utilities to generate and filter data, then distills it into a language-conditioned diffusion policy for multitask learning.Its utilities include grasp samplers and motion planners.Source: (a) FiLM · p. 10
  • Octo offers a modular Transformer-based diffusion policy; MDT adds masked foresight and latent alignment objectives, while RDT-1B [113] scales bimanual pretraining across heterogeneous robot data.Octo reports positive transfer across robots and tasks; MDT outperforms prior U-Net-based diffusion models. RDT-1B uses over 1M trajectories, scales to 1.2B parameters, and demonstrates zero-shot generalization.Source: (a) FiLM · p. 10

6) Diffusion-Based Control Policies with 3D Vision: … 10) Strengths and Limitations:

These sections survey control policies ranging from 3D diffusion and motion-planning methods to point-based actions and large VLAs. They also describe tradeoffs in action representations, model efficiency, and scaling.

6) Diffusion-Based Control Policies with 3D Vision: / 7) Control Policies for Motion Planning:

  • 3D diffusion policies use point-cloud inputs or combine Act3D with diffusion, while motion-planning approaches use language costs, voxel affordance maps, or tracked points to guide robot movement.Language costs support human corrections through predicted cost maps; VoxPoser composes affordance and constraint maps for model predictive control, and RoboTAP chains gripper-marked stages for few-shot visual imitation.Source: 6) Diffusion-Based Control Policies… · p. 10, 7) Control Policies for Motion Planning · p. 10

8) Control Policies with Point-Based Actions:

  • Point-based approaches use VLMs to select visual keypoint proposals, predict 2D affordance points projected into 3D, or generate constraints for a 3D-keypoint optimization problem.PIVOT iteratively refines candidate keypoints, RoboPoint fine-tunes a VLM for affordance prediction, and ReKep obtains actions by solving generated constraints.Source: 8) Control Policies with Point-Based… · pp. 10–11

9) Large VLA:

  • RT-2 co-fine-tunes on Internet-scale VQA and robot data, RT-X models retrained on the larger open X-embodiment dataset outperform their originals, and OpenVLA provides an open-source counterpart to RT-2-X.OpenVLA also explores LoRA and quantization; OpenVLA-OFT applies an optimized fine-tuning recipe, and TraceVLA adds visual trace prompting.Source: 9) Large VLA · p. 11
  • Large VLA methods add intermediate language motions for task sharing and correction, flow-matching action experts, linear-complexity Mamba inference, 3D position encoding, and dual-system humanoid control.RT-H adds the language-motion hierarchy; π0 adds a mixture-of-experts action component, SpatialVLA uses adaptive action grids, and GR00T N1 pairs 10 Hz perception and language processing with 120 Hz diffusion motor actions.Source: 9) Large VLA · p. 11
  • LAPA learns latent actions from unlabeled videos before robot-data fine-tuning, while other work combines VLAs with diffusion or world models and explores autoregressive generation over shared quantized multimodal tokens.The diffusion approaches include TinyVLA, CogACT, DexVLA, and HybridVLA; WorldVLA and UniVLA model action, text, and image generation autoregressively.Source: 9) Large VLA · p. 11

10) Strengths and Limitations:

  • End-effector pose prediction can generalize across embodiments but limits dexterity; discrete actions can cause early grasping, and complex tasks may require SE(3) rather than SE(2) actions.Point-based actions may be coarse-grained, but VLMs can obtain them more easily in a zero-shot manner.Source: 10) Strengths and Limitations · pp. 11–12
  • LVLAs may improve instruction following but can incur training costs and slow inference, motivating smaller diffusion-based models and early-exit inference; robotics scaling laws implicate model and dataset size and environmental diversity.TinyVLA targets inference speed and data efficiency, while DeeR-VLA partially activates the model; scaling-law research may guide robust in-the-wild generalization.Source: 10) Strengths and Limitations · p. 12

IV. TASK PLANNERS … 2) End-to-End Task Planners with 3D Vision:

Task planners decompose complex instructions into subtasks for low-level policies, enabling VLAs to address longer-horizon tasks. The survey describes monolithic and end-to-end planners, including models that incorporate 3D vision.

IV. TASK PLANNERS

  • High-level planners decompose complex tasks into subtasks that instruct low-level control policies, supporting more complex, long-horizon tasks.This decomposition is related to task and motion planning and embodied decision making; plans ideally also schedule subtasks optimally.Source: §IV · pp. 12–13

A. Monolithic Task Planners

  • A single LLM or multimodal LLM can generate task plans through a tailored framework or embodied-dataset finetuning; the survey calls these monolithic models.Source: A. Monolithic Task Planners · p. 13

1) End-to-End Task Planners:

  • End-to-end MLLMs leverage Internet-scale knowledge for task planning, and can be used as embodied task planners.Source: 1) End-to-End Task Planners · p. 13
  • PaLM-E generates plans from perceived images and language instructions, which SayCan maps to executable low-level instructions, and it can replan as the environment changes.Built by integrating ViT and PaLM, it also handles standard and embodied VQA tasks.Source: 1) End-to-End Task Planners · p. 13
  • EmbodiedGPT uses an embodied-former to produce instance-level features from vision embeddings and LLM planning information, guiding the low-level policy's next action.Source: 1) End-to-End Task Planners · p. 13

2) End-to-End Task Planners with 3D Vision:

  • Because most current MLLMs process images, 3D-vision task planners require architectural changes and are usually end-to-end models; LEO trains with 3D alignment followed by action instruction tuning.LEO performs well on 3D question answering, manipulation, navigation, and task planning.Source: 2) End-to-End Task Planners with 3D… · p. 13
  • 3D-LLM uses point-cloud, gradSLAM, and neural-voxel-field features for 3D tasks, while MultiPLY adds audio, tactile, and thermal modalities.Source: 2) End-to-End Task Planners with 3D… · p. 13
  • ShapeLLM combines ReCon++—which distills multiview image and text teacher knowledge—with a point-cloud MAE; integrating ReCon++ with LLaMA improves embodied interaction on its 3D MM-Vet benchmark.Source: 2) End-to-End Task Planners with 3D… · p. 13

3) Grounded Task Planners: … C. Strengths and Limitations

Task planners connect high-level instructions to executable actions through grounded, language-based, or code-based methods. Modular designs avoid costly embodied-data finetuning but bring alignment, API-preparation, or underlying-model constraints.

3) Grounded Task Planners:

  • Grounded planners check whether high-level plans can be executed: SayCan combines the LLM’s proposed skill with the low-level policy’s affordance, while other approaches translate or learn actions iteratively.Translated ⟨LM⟩ maps generated phrases to admissible VirtualHome actions and uses re-prompting after precondition errors; (SL)3 alternates segmentation, labeling, and parameter updates to discover reusable skills from sparse language annotations.Source: 3) Grounded Task Planners · p. 13

B. Modular Task Planners

  • Modular planners assemble off-the-shelf LLMs and VLMs as a tool-use architecture to avoid the expense of finetuning end-to-end models on embodied data.These designs are described as more deployable than end-to-end models.Source: B. Modular Task Planners · p. 13

1) Language-Based Task Planners:

  • Language-based planners exchange multimodal information in text; Inner monologue and ReAct update low-level instructions from feedback, while LLM-Planner replans to get unstuck.Inner monologue uses success, object-and-scene, and human feedback communicated textually, without additional LLM training.Source: 1) Language-Based Task Planners · p. 14
  • LID uses active data gathering and hindsight relabeling to reuse unsuccessful trajectories and improve combinatorial generalization; Socratic models compose pretrained models through multimodal-informed language prompts.Socratic models unify non-language inputs as language descriptions and support robot perception and planning, with task plans also representable as pseudocode.Source: 1) Language-Based Task Planners · p. 14

2) Code-Based Task Planners:

  • ProgPrompt uses program-like action and object specifications with assertions for few-shot household plans; ChatGPT for robotics iteratively writes API code using simulation and user feedback.The ChatGPT procedure defines APIs, specifies the environment and task, evaluates code quality and safety, and executes the generated code as high-level planning with low-level function calls.Source: 2) Code-Based Task Planners · p. 14
  • CaP generates policy code that invokes perception and control APIs; COME-robot uses GPT-4V for closed-loop adaptive planning, failure recovery, and free-form instructions.CaP is described as proficient in spatial-geometric reasoning, generalizing to new instructions, and parameterizing low-level control primitives; COME-robot removes the perception APIs CaP requires.Source: 2) Code-Based Task Planners · p. 14
  • DEPS explains failures from environmental feedback to support replanning and selects achievable parallel subgoals; ConceptGraphs gives LLMs semantic and spatial scene relationships for planning.ConceptGraphs builds open-vocabulary 3D scene graphs from observations and translates them into JSON text descriptions for LLMs.Source: 2) Code-Based Task Planners · p. 14

C. Strengths and Limitations

  • End-to-end grounded planners can benefit from embodied-data finetuning but are costly; modular planners use off-the-shelf models, with language-alignment or API-preparation needs and model-coding limits.Language-based planners must align plans with skills admissible to low-level policies; code-based planners require manually wrapped APIs and documentation, and their performance depends on the models’ programming capabilities.Source: C. Strengths and Limitations · pp. 14–15

V. DATASETS AND BENCHMARKS … VII. CONCLUSION

The survey reviews datasets and benchmarks for embodied AI, the obstacles and open challenges shaping VLA development, and a taxonomy spanning components, control policies, and task planners. It identifies needs for broader evaluation and more capable, safe, generalizable systems.

A. Real-World Robot Datasets & Benchmarks; B. Simulators, Simulated Robot Datasets & Benchmarks

D. Human Datasets

  • Human data offer diverse guidance for robot policies, but motion transfer, viewpoint inconsistency, and labor-intensive filtering complicate their use; UMI proposes hand-held grippers.The survey points to for a broader comparison of human datasets.Source: D. Human Datasets · pp. 15–16

E. Task Planning Benchmarks

  • Task-planning benchmarks assess annotated real-world planning, plan quality and replanning, or simulator-executed success; EAI adds fine-grained metrics to diagnose LLM modules.The survey notes that success rates alone may not pinpoint issues in LLMs.Source: E. Task Planning Benchmarks · p. 16

F. Embodied Question Answering (EQA) Benchmarks

  • EQA benchmarks test spatial reasoning, physics understanding, and world knowledge through exploration rather than direct manipulation or navigation; active exploration requires a simulator.They also assess multimodal expression and spatial, temporal, and causal reasoning.Source: F. Embodied Question Answering (EQA)… · p. 16

VI. CHALLENGES AND FUTURE DIRECTIONS

  • VLA challenges include safety and interpretability, broader benchmarks and metrics, limited foundation-model generalization, and unresolved multimodal alignment and embedding design.Additional modalities such as gaze can help policies attend to salient locations, but increase model complexity; RLHF and evaluation without execution can reduce safety risks.Source: §VI · p. 16
  • Long-horizon hierarchies add failure points and replanning latency; planners can be redundant or hard to integrate, while real-time response, multiagent coordination, ethics, and broader applications remain challenges.The survey proposes exploring end-to-end translation from long-horizon tasks to low-level control signals; healthcare may require stronger safety and privacy protections.Source: §VI · pp. 16–17

VII. CONCLUSION

  • The survey organizes VLA research into components, control policies, and task planners, and reviews datasets, simulators, benchmarks, and future research directions.It aims to capture the rapidly evolving landscape of embodied AI.Source: §VII · p. 17

APPENDIX … • MAE [404]

This appendix gathers related work relevant to VLAs, covering unimodal foundations and the development of vision-language models, as well as works omitted from the main text due to page limitations. It includes contributions from vision, language, reinforcement learning, and graph methods, along with an HLSM object detector and ViT variants.

B. Background (Extended Version): Unimodal Models

  • VLA components draw on vision, language, and reinforcement learning models, so the appendix summarizes relevant model developments and depicts their timeline and growth in scale.Vision models support image classification, detection, and segmentation; language models enable instruction understanding and responses; reinforcement learning supports policy learning and action generation.Source: B. Background (Extended Version):… · p. 37, 1) Unimodal Models · p. 37

1) Computer Vision:

  • Computer vision progressed from CNN classifiers and region-, grid-, and encoder-decoder methods to Transformers such as ViT, DETR, Segmenter, and SAM; robotics also uses depth, point clouds, and volumetric inputs.The appendix notes classification, detection, and segmentation as particularly relevant to robotics, while irregular 3D meshes pose challenges for neural networks.Source: 1) Computer Vision · pp. 37–38

2) Natural Language Processing:

  • NLP moved from word-, syntax-, and semantics-based processing through RNNs and CNNs to Transformer encoders, decoders, and LLMs, with instruction tuning, prompting, and DPO shaping newer language models.Transformers enable self-attention; BERT targets understanding, GPT-family models target generation, and DPO trains on human preferences without explicit reward modeling or reinforcement learning.Source: 2) Natural Language Processing · pp. 38–40

3) Reinforcement Learning:

  • Reinforcement learning learns policies from environment observations and informs VLA action generation; the appendix reviews value-based, policy-search, imitation, hierarchical, and robotic-control methods.Examples include DQN, AlphaGo, PPO, GAIL, and robotic systems progressing from image-to-torque control to closed-loop QT-Opt and long-horizon Dreamer.Source: 3) Reinforcement Learning · p. 40

4) Graph:

  • Graph methods process structured data across vision and language, spanning spectral and spatial convolutional GNNs, graph Transformers, scene relationships, and syntax, semantics, dialogue, and knowledge graphs.Spatial approaches are described as more efficient and flexible; GIN addresses MPNN’s lack of embedded graph-structure information, while GraphSage improves efficiency through neighbor sampling.Source: 4) Graph · pp. 40–41

C. Background (Extended Version): Vision-Language Models

2) Contrastive Pretraining:

  • BERT-based VLMs use single-stream or multi-stream Transformers, while contrastive approaches such as CLIP learn image-text alignment and support zero-shot transfer to downstream tasks.Other approaches include word-region alignment in UNITER, patch-based image embeddings in ViLT, token-wise contrastive learning in FILIP, noisy-data training in ALIGN, and training text while freezing image models in LiT.Source: 1) Self-Supervised Pretraining · p. 41, 2) Contrastive Pretraining · p. 41

3) Large Multimodal Model:

  • Large multimodal models connect vision encoders to LLMs through mechanisms including gated cross-attention, Q-Former, or linear layers, and multimodal instruction tuning equips models to follow visual instructions.Examples include Flamingo, BLIP-2, LLaVA, and MiniGPT-4; LLaVA uses two-stage instruction tuning, whereas MiniGPT-4 freezes its vision encoder and LLM while training a linear layer.Source: 3) Large Multimodal Model · p. 43

D. Supplementary Related Work

• MDETR [402]

• OWL-ViT [405] … • JARVIS-VLA [477]

The merged sections extend the survey’s VLA landscape with additional work on model components, learning methods, later developments, and related surveys. They also list selected model terminology and describe how the survey’s broader VLA-paper charts were compiled.

• OSRT [409]; • PaLI-X [365]; • DD-PPO [432]

1) Components of VLA: Pretraining:

  • VIP explores video-based temporal representation learning, while SpawnNet, Holo-Dex, Dobb·E, AuRL, and T-Dex broaden pretraining across visual, audio, and tactile inputs.VIP attracts initial and target frames and repels successive frames; later comprehensive evaluations of VIP report conflicting findings. SpawnNet fuses pretrained and newly learned features without training the pretrained encoder, and the other methods explore self-supervised, home-collected, audio, or tactile inputs.Source: 1) Components of VLA: Pretraining · p. 44

2) Components of VLA: World Models:

  • World-model work spans reward-augmented latent dynamics, human-video transfer grounded in affordances, decoupled dynamics for long-horizon decisions, Transformer-XL agents, and tokenized scene descriptions.MWM adds an auxiliary reward-prediction loss; SWIM infers target poses from visual affordance maps; Iso-Dream separates controllable and noncontrollable dynamics. TWM learns in latent imagination, while SceneScript represents scenes with autoregressive language commands.Source: 2) Components of VLA: World Models · p. 44

3) Components of VLA: Imitation Learning:

  • Imitation-learning variants address multimodal demonstrations through discretized and corrected actions, then extend goal specification and long-range action modeling.BeT uses k-means action discretization with continuous correction, C-BeT adds a target frame or demonstration as the goal, and VQ-BeT replaces k-means with vector quantization.Source: 3) Components of VLA: Imitation Learning · pp. 44–45

4) Subsequent Surveys on VLA; • DDIM [437]

5) Latest Developments of VLA:

  • Later developments include language relabeling for robot demonstrations, dexterity-focused action tokenization, diverse-data co-training, and VLM-guided topological navigation.LLaRA automatically relabels existing demonstration videos; FAST and π0.5 extend π0, respectively emphasizing dexterous skills and in-the-wild generalization. Mobility VLA combines VLM commonsense reasoning with a topological-graph-based low-level navigation policy.Source: 5) Latest Developments of VLA · p. 45

• QUAR-VLA [467]; • EMMA [475]

• JARVIS-VLA [477]

  • The survey charts VLA developments from 2020 to 2025 using a hybrid search and scripting pipeline that identified approximately 400 related papers.The search covered January 2020 through December 2025 and filtered results for relevance to embodied AI and robotics; the authors acknowledge possible automation errors and invite corrections.Source: • JARVIS-VLA [477] · p. 45

• CNRS: CNRS IRL 2010 CROSSING · • NVIDIA: NVIDIA Research

This section identifies organizations and VLA-related names, expanding organization names across alphabetic ranges and mapping Xiaomi-related names to their organizations.

• CNRS: CNRS IRL 2010 CROSSING

  • The C–J entries expand abbreviations for organizations including Central South University, the Chinese University of Hong Kong, Case Western Reserve University, Caltech, and Google groups.The list also includes companies, laboratories, and other institutions.Source: • CNRS: CNRS IRL 2010 CROSSING · pp. 45–46

• NVIDIA: NVIDIA Research

• ACG [483] … • FPC-VLA [570]

The supplied excerpts list one work under each of three subsections: ConRFT, CoT-VLA, and DLR.

• ConRFT [516]

• CoT-VLA [79]

• DLR [534]

• FTM, FLA [572] … • RT-H [702]

The supplied excerpts list four named works across the HULC++, INSIGHT, MemER, and RoboOS-NeXT subsections.

• HULC++ [85]

• INSIGHT [600]

• MemER [623]

• RoboOS-NeXT [692]

• RT-RAS [703] … 6) Beyond VLA:

The supplied passages identify TA-VLA and UP-VLA entries and describe research directions extending beyond standard VLA architectures, including world-model integration, additional perception modalities, and video-action models.

• TA-VLA [736]

• UP-VLA [750]

6) Beyond VLA:

  • Beyond standard VLA architectures, one direction unifies VLAs with world models, including WorldVLA, UniVLA, NORA-1.5, RynnVLA-002, and Motus.Source: 6) Beyond VLA · p. 50
  • Other directions incorporate perception modalities beyond vision through PLA and MLA.Source: 6) Beyond VLA · p. 50
  • Video-action models such as mimic-video (VAM) capture semantics and dynamics during video pretraining to enable more efficient post-training for low-level robot control.Source: 6) Beyond VLA · p. 50

Related papers

Loading 2405.14093v8…