PaperlayerView exact revision on arXiv

Source-linked AI summary

Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey

Rui Shao, Wei Li, Lingsen Zhang, Renshan Zhang, Zhiyang Liu, Ran Chen, Liqiang Nie

arXiv:2508.13073v3cs.ROcs.CV

TL;DR

Robotic manipulation needs precise control and multimodal understanding, but traditional methods struggle in novel, unstructured environments. This survey defines large VLM-based VLAs, organizes them into monolithic and hierarchical architectures, and synthesizes their methods, evidence, and future directions. Across reported examples, these models show broad generalization, including over 90% OOD success for π0.5 on new home tasks and a 20% boost for HAMSTER over OpenVLA across seven axes, while efficiency and modular separation retain important trade-offs.

  • Problem

    Traditional robotic manipulation methods struggle to scale and generalize in unstructured settings, while existing reviews lack a comprehensive synthesis of large VLM-based VLA models.

  • Method

    The survey defines large VLM-based VLAs and develops a taxonomy separating monolithic models from hierarchical models, while synthesizing architectures, learning paradigms, datasets, benchmarks, and research directions.

  • Results

    Reported models demonstrate broad generalization: π0.5 exceeds 90% OOD success on new home tasks, and HAMSTER improves success by 20% over OpenVLA across seven generalization axes.

  • Takeaways & Limitations

    The survey provides a structured landscape for understanding large VLM-based VLA development and emphasizes cross-embodiment adaptation, scalable deployment, and tighter reasoning-execution coupling as future priorities.

  • Takeaways & Limitations

    Inference-efficiency gains can reduce reasoning depth and task accuracy, while cascade designs lack real-time action feedback and can propagate planning errors during dynamic tasks.

Abstract

from arXiv · show

Robotic manipulation, a key frontier in robotics and embodied AI, requires precise motor control and multimodal understanding, yet traditional rule-based methods fail to scale or generalize in unstructured, novel environments. In recent years, Vision-Language-Action (VLA) models, built upon Large Vision-Language Models (VLMs) pretrained on vast image-text datasets, have emerged as a transformative paradigm. This survey provides the first systematic, taxonomy-oriented review of large VLM-based VLA models for robotic manipulation. We begin by clearly defining large VLM-based VLA models and delineating two principal architectural paradigms: (1) monolithic models, encompassing single-system and dual-system designs with differing levels of integration; and (2) hierarchical models, which explicitly decouple planning from execution via interpretable intermediate representations. Building on this foundation, we present an in-depth examination of large VLM-based VLA models: (1) integration with advanced domains, including reinforcement learning, training-free optimization, learning from human videos, and world model integration; (2) synthesis of distinctive characteristics, consolidating architectural traits, operational strengths, and the datasets and benchmarks that support their development; (3) identification of promising directions, including memory mechanisms, 4D perception, efficient adaptation, multi-agent cooperation, and other emerging capabilities. This survey consolidates recent advances to resolve inconsistencies in existing taxonomies, mitigate research fragmentation, and fill a critical gap through the systematic integration of studies at the intersection of large VLMs and robotic manipulation. We provide a regularly updated project page to document ongoing progress: https://github.com/JiuTian-VL/Large-VLM-based-VLA-for-Robotic-Manipulation

1 INTRODUCTION

The survey motivates large VLM-based VLA models as a response to the limited scalability and generalization of traditional robotic manipulation, then defines and organizes them into monolithic and hierarchical paradigms.

  • Traditional manipulation relies on engineered control policies and predefined task specifications that falter with novel objects and unstructured environments.
  • Large VLM-based VLAs combine visual-language understanding with robotic action generation to interpret instructions, generalize across scenarios, and perform manipulation in dynamic settings.
  • The survey defines these models as using a large VLM for visual observations and language instructions while reasoning processes directly or indirectly support action generation.
  • Monolithic Models: Monolithic models integrate perception, language understanding, and action generation in single systems or use a VLM backbone with a separate action expert.
  • Hierarchical Models: Hierarchical models decouple planning from policy execution through interpretable intermediate outputs and independently optimized modules.
  • The taxonomy and longitudinal, cross-cutting synthesis organize fragmented research, compare structural and functional practices, and identify datasets, challenges, and future directions.

2 BACKGROUND

Large VLMs provide multimodal perception and reasoning that have enabled unified VLA systems for robotic control. The background traces the shift from modular language-conditioned robots toward pretrained-VLM-based models and surveys the remaining gap in focused taxonomies.

  • Large VLMs: Modern VLMs typically combine a vision encoder, projector, and language model to align visual inputs with textual embeddings for multimodal reasoning.
  • Large VLMs: Visual instruction tuning gives VLMs flexibility with abstract instructions and unseen multimodal scenarios, supporting their use in real-world applications.
  • Large VLMs: Recent VLMs add unified image-video processing, dynamic input resolution, and reinforcement-learning-based reasoning improvements that support action-generation integration.
  • From VLMs to VLAs: Large VLM-based systems extend VLM capabilities from passive observation and high-level planning to embodied physical interaction through robotic manipulation.
  • Evolution of Robotic Manipulation: Earlier systems used separate vision and language modules, whereas RT-2 casts robot actions as text tokens and co-trains internet-scale vision-language data with robot trajectories.
  • Relevant Surveys and Taxonomies: Existing surveys often address VLMs, manipulation, or broad VLA architectures separately, leaving limited systematic analysis of large pretrained VLMs specifically integrated into robotic control.

3 MONOLITHIC MODELS

Monolithic VLA models unify perception, language, robot-state understanding, and action generation either within a single system or across cooperating dual systems. Their development expands modalities and reasoning, improves generalization and efficiency, and balances cognitive capability against real-time control constraints.

  • Monolithic Models: Monolithic VLA models use either single-system architectures or dual-system architectures that divide reasoning between a VLM backbone and an action expert.Single-system models jointly process visual observations, language instructions, and robot states; dual-system models separate these functions across cooperating modules.
  • Single-system Models: Autoregressive single-system models discretize continuous robot actions into token sequences and decode executable actions from visual, linguistic, and optional state inputs.A downstream de-tokenizer converts the predicted action tokens into robot actions.
  • Model Performance Enhancement: Additional modalities and reasoning improve capability but require task-specific fine-tuning data and increase computational burden under real-time robotic control constraints.The survey identifies a trade-off between performance enhancement and real-time deployment requirements.
  • Inference Efficiency Optimization: Inference optimization compresses parameters, prunes computation, and parallelizes decoding, but gains can reduce reasoning depth and task accuracy while standard methods overlook embodied spatial-temporal control.The survey points toward embodiment-native designs and dual-system separation of deliberate reasoning from high-frequency motor control.
  • Dual-system Models: Dual-system models separate high-level semantic reasoning from low-level action generation, using cascade or parallel interaction patterns between the VLM backbone and action expert.Cascade designs forward representations in a serial manner, whereas parallel designs allow tighter interaction during inference.

4 HIERARCHICAL MODELS

Hierarchical VLA models decouple high-level planning from low-level policy execution through interpretable intermediate representations. The survey organizes them by planner constitution and by subtask-, keypoint-, and program-based representations, each offering distinct trade-offs for complex manipulation.

  • Hierarchical Models: Hierarchical models transform instructions and observations into interpretable intermediate representations that policies convert into executable actions.The planner and policy can operate independently, supporting modular combinations with existing execution policies.
  • Taxonomy: The taxonomy distinguishes Planner-Only and Planner+Policy models, with further representation categories for subtasks, keypoints, and programs.The figure also marks methods that combine affordances for auxiliary purposes.
  • Program-based Methods: Program-based methods encode complex logic, conditional control flow, and parameterization as structured text or robot-executable programs.They suit procedurally structured tasks but are less focused on continuous geometric reasoning.
  • Keypoint-based Methods: Keypoint-based methods ground planning spatially through waypoints, trajectories, sketches, or coordinate sequences consumed by low-level controllers.Their interpretable spatial representations bridge VLM reasoning and continuous action generation for precision-demanding tasks.
  • Subtask-based Methods: Subtask-based methods decompose open-ended instructions into textual step-by-step commands, providing semantic flexibility and open-world generalization.Their real-world performance remains limited by the capability of low-level control policies.
  • Comparison: Monolithic and hierarchical architectures provide complementary strategies: unified learning streamlines generalization, while explicit cognition-control layering improves transparency and modular flexibility.The survey presents comparative experiments to analyze their performance and trade-offs.

5 OTHER ADVANCED FIELD

The survey reviews four advanced directions for improving VLA robustness, efficiency, and long-horizon planning: reinforcement learning, training-free methods, human-video learning, and world-model integration. These approaches address reward sparsity, sample efficiency, computational cost, embodiment gaps, and predictive planning.

  • Reinforcement Learning: RL improves VLA generalization and task completion, but long-horizon trajectories make rule-based rewards sparse and unstable.Learned dense rewards, including success-likelihood and progress-based signals, address this challenge.
  • Reinforcement Learning: Hybrid offline-online RL combines pre-collected trajectories with real-time optimization to accelerate learning and improve deployment safety.Human-in-the-loop interventions are used in related training schemes.
  • Reinforcement Learning: RL can serve as a data engine: expert policies are trained through intervention, distilled into foundation models, or iteratively expanded through successful trajectories and SFT.RLDG and iRe-VLA exemplify these strategies.
  • Training-free Methods: Training-free methods improve existing VLA efficiency without retraining or architectural changes by selectively skipping decoding, pruning tokens or layers, and caching features.These designs support rapid prototyping and targeted enhancements while preserving original capabilities.
  • Learning from Human Videos: Human-video learning transfers task-relevant visual and temporal knowledge through cross-domain alignment, latent actions, video tokens, 3D reasoning, and motion trajectories.The reviewed methods use human videos for perception pretraining or policy adaptation.
  • World Models: World-model-based VLA systems simulate future states and action consequences, enabling predictive planning rather than direct action generation from current observations.WorldVLA jointly predicts visual outcomes and actions in a token-based architecture.

6 CHARACTERISTICS OF VLA MODELS

Large VLM-based VLA models combine semantically aligned multimodal representations with instruction grounding, reasoning, and broad generalization. Their capabilities extend from unified perception-action processing to cross-task, cross-domain, and cross-embodiment transfer.

  • Multimodal Fusion: A shared embedding space aligns visual observations and linguistic instructions, establishing semantic correspondence for action generation.This alignment provides a unified representational basis for subsequent control.
  • Multimodal Fusion: VLA models increasingly integrate vision, language, proprioception, actions, 3D perception, tactile sensing, and audio within extensible multimodal frameworks.Token-level integration allows modalities to condition one another during the perception-action cycle.
  • Instruction Following: Semantic instruction grounding uses pretrained VLM world knowledge to map natural-language commands to context-sensitive actions beyond fixed templates.ChatVLA-2 is reported to interpret mathematical problems written on whiteboards without explicit mathematics training.
  • Instruction Following: Hierarchical decomposition and latent visual goals connect language instructions to subgoals, anticipated outcomes, and action sequences for complex multi-step tasks.This reasoning supports iterative plan refinement and more reliable execution.
  • Cross-Task Generalization: DexVLA generalizes across embodiments, unseen tasks, objects, and scenes without task-specific tuning while outperforming OpenVLA and π0 on reported workflows.It combines a large VLM, a billion-parameter diffusion action expert, and an embodied curriculum.
  • Cross-Domain Generalization: π0.5 achieves out-of-the-box deployment in new home environments with OOD success rates exceeding 90% on multi-stage tasks such as dishwashing.Its training combines web text, simulation videos, and cross-robot embodiments.
  • Cross-Embodiment Generalization: HAMSTER achieves a 20% success boost over OpenVLA across seven axes of cross-embodiment and sim-to-real generalization.Hierarchical planners can leverage off-domain data while domain-specific decoders handle low-level control.

7 CONCLUSION

The survey organizes large VLM-based VLA research through a taxonomy of monolithic and hierarchical architectures, alongside datasets, benchmarks, and advanced learning paradigms. It evaluates these architectures using standardized success-rate comparisons across simulation and real-world deployments while identifying cross-embodiment adaptation and scalable deployment as priorities.

  • Conclusion: The survey consolidates monolithic and hierarchical architectures with datasets, benchmarks, and advanced learning paradigms into a structured view of the field.It identifies cross-embodiment adaptation, scalable real-world deployment, and tighter reasoning-execution coupling as future priorities.
  • Comparative Evaluation: The comparative analysis synthesizes results from LIBERO, LIBERO-Plus, CALVIN, and diverse real-world deployments to assess lifelong learning, long-horizon capability, and generalization.These benchmarks target complementary aspects of VLA performance.
  • Comparative Evaluation: Success Rate measures the percentage of episodes that complete the target task within a predefined maximum step limit after a natural-language instruction.For multi-task suites, the reported metric is the average success rate across tasks.

A.1.1 Evaluation on Diverse Tasks

LIBERO results show a sharp capability gap between early language-conditioned systems and large-VLM monolithic VLAs, especially on long-horizon tasks. Architectural mechanisms beyond scaling are important for sustained temporal execution.

  • The evaluation covers Spatial, Object, Goal, and Long-horizon task categories on LIBERO.
  • Large-VLM monolithic models exceed 95% overall success on LIBERO, compared with roughly 65%–75% for early language-conditioned systems.OpenVLA-OFT, UniVLA, and UnifiedVLA demonstrate the reported monolithic performance leap.
  • Earlier architectures achieve roughly 20%–35% on LIBERO long-horizon tasks despite stronger spatial and object performance.The decline is associated with context saturation and error accumulation during naive autoregressive decoding.
  • Recent single-system and dual-system models push long-horizon success rates above 90% using parallel decoding, flow-matching, or continuous latent feature propagation.The cited mechanisms address temporal execution rather than only basic spatial-semantic grounding.

A.1.2 Evaluation on Generalization Performance

Generalization evaluations expose substantial perceptual and semantic fragility under visual, kinematic, linguistic, and cross-domain shifts. Robust action-generation mechanisms help advanced architectures retain performance across difficult long-horizon settings.

  • OpenVLA’s overall LIBERO-Plus performance falls to 16.1%, including 1.1% under camera viewpoint shifts and 4.1% under robot initial-state perturbations.The analysis attributes these drops to visual-kinematic entanglement and overfitting to rendering and photometric properties.
  • Instruction rewriting, conversational distractors, synonyms, complex reasoning chains, and object-layout changes reveal semantic and cognitive fragility.The passage characterizes integrated VLMs as often relying on shallow linguistic pattern matching.
  • In CALVIN’s ABC→D setting, RT-1 declines from 53.3% at 1/5 tasks to 1.3% at 5/5 tasks.This setting tests unseen-scene transfer, sequential execution, and resistance to compounding covariate shift.
  • TriVLA, UnifiedVLA, and π0 maintain high final-stage success in ABC→D, with diffusion and flow-matching action generation supporting dynamic correction.

A.1.3 Evaluation on Real-World Custom Tasks

Real-world comparisons require relative gains because hardware, baselines, and task complexity vary across experiments. Monolithic models excel at reactive multimodal control, while hierarchical models show advantages in structurally complex and zero-shot settings.

  • Absolute real-world success rates are often infeasible to compare directly because experimental hardware, baselines, and task complexity differ.The survey instead examines relative improvements over established baselines and task-specific structural advantages.
  • VILA achieves a 97.5% SR in contact-rich peg-in-hole assembly, illustrating monolithic strength in high-frequency reactive control.Unified latent-space mappings preserve subtle physical and tactile nuances that may be lost during intermediate feature extraction.
  • OpenVLA and TraceVLA improve SR over standard baselines by 16.8% and 57.5%, respectively, on tasks with visual distractors and randomized initial positions.
  • Hierarchical models demonstrate advantages in extreme zero-shot generalization, long-horizon multi-stage planning, and cross-embodiment transfer.Structured intermediate representations support 3D-aware spatial reasoning and affordance-guided manipulation.

HIERARCHICAL

Hierarchical models use structured intermediate representations and modular planning to address complex reasoning and execution demands. Their reported gains are strongest in zero-shot recombination, long-horizon tasks, and spatial composition.

  • HIERARCHICAL: Hierarchical modularity lets high-level planners adapt to novel tasks and instructions through interpretable sub-goals.
  • HIERARCHICAL: RoboMatrix achieves a 100% relative SR improvement over OpenVLA in zero-shot skill recombination and long-horizon tasks.
  • HIERARCHICAL: DexVLA outperforms π0 by 13.5% on tasks evaluating implicit reasoning and cross-embodiment dexterity.
  • HIERARCHICAL: RT-H and VoxPoser surpass RT-2 by 15% and 64%, respectively, on contextual sequential execution and spatial composition tasks.

B.1 Real-world Robot Datasets

Large VLM-based VLA models rely on diverse datasets spanning real-world robot demonstrations, simulation, human behavior videos, and reasoning benchmarks. These resources support language grounding, semantic generalization, task decomposition, and embodied planning.

  • Real-world robot datasets: Real-world datasets capture environmental complexity and align language instructions, perceptual inputs, and precise action responses for VLA training and evaluation.Examples include BC-Z, which provides expert demonstrations with language commands across 100 tasks, and RT-1.
  • Simulated datasets: Physics-based simulators provide scalable, safe, and reproducible interaction data for instruction following, multi-stage planning, and automatic evaluation.BEHAVIOR targets cluttered household settings, while ALFRED focuses on long-horizon tasks.
  • Human behavior datasets: Egocentric human behavior datasets support object recognition, action sequencing, and task decomposition through daily activities and fine-grained cooking demonstrations.Resources include Ego4D, Ego-Exo4D, EgoPlan-Bench, EgoVid-5, EPIC-Kitchens, and COM-Kitchens.
  • Benchmarks: Reasoning-oriented embodied benchmarks evaluate planning and reasoning abilities by integrating vision, language, and embodied-agent decision-making.EmbodiedQA and IQUAD are identified as early efforts in this direction.

C.1 Future Directions

Future VLA research must address gaps in evaluation realism, long-horizon memory, dynamic perception, collaborative autonomy, lifelong learning, and deployment efficiency. The survey proposes richer datasets, temporal and contextual reasoning, and resource-aware adaptation as directions forward.

  • Datasets and Benchmarking: Current benchmarks emphasize short-horizon pick-and-place tasks and simple success rates, leaving long-term planning, mobile manipulation, and multi-agent collaboration underexplored.Synthetic data also lacks real-world visual complexity, while real-world collection limits diversity and scale.
  • Memory Mechanisms and Long-Term Planning: Most current VLAs reason frame by frame, limiting historical context and producing short-sighted behavior in extended manipulation tasks.Memory mechanisms and episodic awareness are proposed to support coherent, goal-driven action sequences.
  • 3D and 4D Perception: Existing VLA models primarily use static 2D inputs, limiting reasoning about depth, affordances, object movement, and human actions.4D perception would integrate depth or point clouds, multimodal fusion, temporal context, and online replanning.
  • Multi-Agent Cooperation: Multi-agent manipulation requires interaction-aware representations for intention negotiation, teammate adaptation, role assignment, and joint multi-step reasoning.Shared world models and emergent dialogue protocols are proposed to support coordination.
  • Lifelong Learning in Open-World: Static-dataset training leaves VLA models struggling with unfamiliar objects, novel interaction modes, and incorporation of new experiences without catastrophic forgetting.Incremental knowledge accumulation through exploration, feedback, and growing memory structures is proposed.
  • Model Efficiency: VLA deployment faces prohibitive computational and memory costs, creating a trade-off between model capacity, real-time inference, latency, and multimodal accuracy.Compression must preserve multimodal alignment on resource-constrained robotic platforms.

C.2 Discussion

The survey finds broad agreement that internet-scale visual-semantic priors benefit embodied systems, while major disputes concern explicit world models and persistent limitations in physical grounding, temporal control, and data scaling. These bottlenecks constrain progress toward reliable real-world autonomy.

  • Established and Emerging Consensus: Large pretrained VLMs are widely viewed as useful for transferring visual-semantic priors to robots and enabling zero-shot generalization potential.The claimed benefit centers on addressing the semantic grounding bottleneck for novel objects and unconstrained instructions.
  • Key Controversies and Opposing Views: Researchers disagree over implicit end-to-end architectures versus explicit world models for representing physics and environmental dynamics.Explicit models offer predictive simulation and geometric constraints but may impose prohibitive computational overhead.
  • Critical Insights: Successes and Persistent Bottlenecks: Current VLA systems remain structurally limited in physical grounding, data scaling, temporal control, and explicit environmental modeling.The survey identifies these limitations as bottlenecks to real-world autonomy.
  • Illusory Physical Reasoning and Statistical Shortcuts: Large VLMs can rely on language priors and visual co-occurrence rather than causal understanding of 3D physical dynamics.The survey characterizes apparent success in tasks such as object placement as trajectory imitation rather than genuine embodied cognition.
  • Embodiment Heterogeneity and the Breakdown of Scaling: Robot-data heterogeneity across kinematics, sensing, and embodiment creates distributional conflicts that can cause performance collapse under naive aggregation.Teleoperation cost further limits scalability beyond imitation learning.
  • Pseudo Closed-Loop Control and Missing Temporal Dynamics: Open-loop action sequences delegated to low-level controllers cause temporal misalignment and error accumulation over long horizons in dynamic, contact-rich settings.The survey links this weakness to the absence of continuous-time dynamics modeling.
  • Implicit Memorization vs. Explicit World Modeling: Implicit physical memorization is vulnerable to hallucination under distribution shift, whereas explicit world models improve physical fidelity at prohibitive real-time computational cost.The resulting scalability–physical-fidelity trade-off remains a barrier to safe deployment.

APPENDIX D PRACTICAL MODEL-SELECTION GUIDELINES

The appendix frames model selection as a task-dependent choice among single-system, dual-system, and hierarchical VLA designs. Selection should balance control demands, horizon, interpretability, data and sensing requirements, latency, and integration complexity rather than compare reported numbers globally.

  • Selection principles: Reported numerical results come from different platforms, datasets, and protocols, so they should not be compared across table rows or treated as a global ranking.The appendix presents the tables as model-selection guides rather than unified benchmarks.
  • Dual-System Models: Dual-system models suit high-frequency or precise continuous control by separating VLM scene interpretation from specialized action generation.The additional action module or more complex training and data recipe is the principal trade-off.
  • Hierarchical Models: Hierarchical models are preferable for long-horizon decomposition, interpretable intermediates, human intervention, or verifiable geometric constraints.Planner-only systems can use existing low-level skills, while keypoint- or path-based methods bridge visual reasoning and control.
  • Single-System Models: Single-system models offer a simple, reproducible starting point, with variants targeting reasoning, visual shift, temporal cues, spatial structure, or deployment speed.OpenVLA is positioned as a default when openness and reproducibility matter, while other methods add annotations, sensing assumptions, or computation.
  • Selection principles: Model selection should balance task requirements against control latency, data budgets, sensing assumptions, interpretability, and integration complexity.Tables 10–12 instantiate these rules with named baselines and explicit trade-offs.
Loading 2508.13073v3…