Source-linked AI summary
10 Open Challenges Steering the Future of Vision-Language-Action Models
Soujanya Poria, Navonil Majumder, Chia-Yu Hung, Amir Ali Bagherzadeh, Chuan Li, Kenneth Kwok, Ziwei Wang, Cheston Tan, Jiajun Wu, David Hsu
TL;DR
Robotic policies often struggle to generalize beyond narrow training scenarios, and VLA evaluations cover only a few robots and settings. This paper revisits key VLA challenges and proposes an exploratory framework, highlighting multimodal sensing and cross-robot action generalization as future research directions.
Problem
Robotic policies often struggle to generalize beyond their training scenarios, while VLA evaluations cover only a few robots and predefined settings.
Source: §1 · p. 1, §2.4 · p. 3Method
The paper revisits key VLA challenges, including multimodal sensing and cross-robot action generalization, and proposes an exploratory framework aimed at addressing them.
Source: §4 · p. 8Results
The authors identify multimodal sensing and cross-robot action generalization as challenges that will shape future VLA research directions.
Source: §4 · p. 8Takeaways & Limitations
Agentic VLA frameworks could address agent-level resource constraints by delegating compute or gathering observations through other robots.
Source: §2.9 · p. 4Simulation environments often poorly correlate with real-life performance, and still need greater environmental and embodiment diversity and fidelity.
Source: §2.4 · p. 3
Abstract
from arXiv · showhide
Due to their ability of follow natural language instructions, vision-language-action (VLA) models are increasingly prevalent in the embodied AI arena, following the widespread success of their precursors -- LLMs and VLMs. In this paper, we discuss 10 principal milestones in the ongoing development of VLA models -- multimodality, reasoning, data, evaluation, cross-robot action generalization, efficiency, whole-body coordination, safety, agents, and coordination with humans. Furthermore, we discuss the emerging trends of using spatial understanding, modeling world dynamics, post training, and data synthesis -- all aiming to reach these milestones. Through these discussions, we hope to bring attention to the research avenues that may accelerate the development of VLA models into wider acceptability.
1 Introduction to VLA Models
VLA models combine visual observations and language instructions to produce robotic actions, addressing the gap between visual-language understanding and robot-executable policies. Their action modeling trades off discrete-token efficiency and compatibility with language models against continuous-action fidelity and compute demands.
1 Introduction to VLA Models
- VLA models use imitation learning to map visual observations, language instructions, and prior context to robot actions.The objective is to maximize the likelihood of demonstrated action sequences.Source: §1 · pp. 1–2
1.1 Discrete Action Models
- Discrete-action models represent robot actions as tokens, enabling next-token prediction but risking quantization errors and slow inference.OpenVLA incorporates action tokens into its language-model vocabulary, while FAST+ uses DCT and BPE to shorten action sequences.Source: §1 · pp. 1–2, §1.1 · p. 2
1.2 Continuous Action Models
- Continuous-action models predict expressive action distributions, preserving movement fidelity and better suiting high-frequency control.Diffusion-based models jointly predict action chunks but require substantially more compute to converge.Source: §1 · p. 2, §1.2 · p. 2
- Hybrid approaches combine autoregressive discrete-action pretraining with a specialized continuous-action expert, with reported superior performance and faster convergence.Knowledge insulation is used to preserve the VLM’s semantic knowledge while training the expert.Source: §1.2 · p. 2
2 The 10 Open Challenges
The paper frames multimodal sensing and perception as an open challenge for VLA models, including explicit depth, robustness to visual artifacts, and modalities beyond vision and language.
2.1 Multimodal Sensing and Perception
- Current VLA models often lack explicit depth perception, while inferring depth from visual frames can miss object size, distance, and fine movements.MolmoAct and SpatialVLA learn to impute depth from RGB frames during training, avoiding a specialized depth camera.Source: §2.1 · p. 2
- Current evaluations leave reflections, lens flares, and noisy camera feeds from water, dust, or debris unaccounted for.SimplerEval varies backgrounds, lighting, distractors, texture, and camera pose.Source: §2.1 · p. 2
- Adding audio and touch could support rescue sensing and delicate force-sensitive tasks, and a touch-capable VLA improved performance on pick, place, and wipe tasks over vision-only models.Audio could capture victims’ calls or hazardous sounds; touch could support handling fragile objects and applying force carefully.Source: §2.1 · pp. 2–3
2.2 Robust Reasoning
Robust reasoning remains a VLA challenge: reasoning traces do not yet yield consistently reliable results, while tool selection and use remain open problems.
2.2 Robust Reasoning
- VLA models trained with linguistic and action-level reasoning traces still produce imperfect results on simple real-world and simulated tasks.Performance generally drops as task horizons increase, despite reported trends linking robust chain-of-thought reasoning with out-of-distribution performance.Source: §2.2 · p. 3
- The paper argues that error rates on simple tasks must approach near perfection for deployment in more complex and sensitive environments.The cited evaluations include picking and placing items, operating drawers, and pointing to items.Source: §2.2 · p. 3
- Equipping VLA models to understand, select, and use the right tools for everyday goals remains an open problem.The passage illustrates this with making coffee, which requires a cup and spoon.Source: §2.2 · p. 3
2.3 Quality Training Data
Training data remains challenging despite large multi-robot datasets: models can remain brittle beyond their training environments and setups, while simulated data and collected demonstrations have limitations.
2.3 Quality Training Data
- Open-X-Embodiment unifies around 70 datasets containing more than 1M episodes, yet VLA models remain brittle to out-of-distribution environments and robot setups.This brittleness can require collecting additional episodes for fine-tuning.Source: §2.3 · p. 3
- Creating realistic simulations remains a challenge for both data collection and evaluation.Simulation-to-real approaches train on simulation data and evaluate on real robots.Source: §2.3 · p. 3
- Collected data varies with robot embodiment, camera position and angle, and inconsistencies in human data collectors’ cognitive behaviors.Source: §2.3 · p. 3
2.4 Evaluation of VLA Models
VLA evaluation is constrained by scarce real-world resources and imperfect simulations, while generalizing actions across different robots remains difficult because their action spaces and control interfaces vary.
2.4 Evaluation of VLA Models
- Real-robot evaluations cover only a few robots and predefined environments, while simulations broaden environmental variation but may correlate poorly with real-world performance.Simulation fidelity, embodiment diversity, and inclusion of modalities such as audio and touch remain areas for improvement.Source: §2.4 · p. 3
2.5 Cross-Robot Action Generalization
- Action heterogeneity—including differences in robot structure, degrees of freedom, and control interfaces—makes generalization from fixed training embodiments to others difficult.Addressing this heterogeneity is described as a first step toward zero-shot generalization across robots.Source: §2.5 · p. 3
2.6 Resource Efficiency
VLA deployment must balance model capacity against limited onboard resources and network reliability; whole-body control must coordinate locomotion and manipulation under uncertainty.
2.6 Resource Efficiency
- Embodiments often rely on smaller onboard VLA models because remote inference is vulnerable to network latency and disruption, but these models are generally less performant than larger counterparts.The challenge is balancing model capacity and resource efficiency.Source: §2.6 · pp. 3–4
2.7 Whole-Body Coordination
- Whole-body coordination couples locomotion and manipulation, but model-based and learning-based control each face limitations in generalization and execution.Model-based control offers interpretable coordination and safety guarantees in structured settings but relies on accurate models and can face real-time bottlenecks; learning-based methods adapt to unstructured settings but struggle with exploration, credit assignment, reward shaping, and cross-embodiment generalization. Proposed directions include hybrid frameworks and objectives that explicitly couple base motion to stable, precise end-effector control.Source: §2.7 · p. 4
2.8 Safety Assurances
Safety concerns arise because embodied systems can harm people through unsupervised actions; multi-agent VLA systems could share resources and observations, but require appropriate autonomy and trustworthy workflows.
2.8 Safety Assurances
- Embodied AI actions can directly harm people, motivating guardrails and failsafes; Zhang et al. (2025) proposed RL-based VLA safety alignment that constrains actions while maintaining general performance.Source: §2.8 · p. 4
2.9 VLA Models in Agentic Frameworks
- Multi-agent VLA systems could delegate computation or share sensor observations, but require choices about agent autonomy and trustworthy, safe, verifiable workflows.An early heterogeneous framework uses an LLM planner and VLM verifier to assist a VLA model (Yang et al. 2025b).Source: §2.9 · p. 4
2.10 Human-Robot Coordination
Current VLA communication is one-way, from human instructions to robot; future systems could return visual and language outputs that support interpretation and user interaction. Related approaches generate intended-state visuals or natural-language rationales, with reported benchmark and action-generation benefits.
2.10 Human-Robot Coordination
- Current human-to-robot natural-language communication could be extended with model outputs containing reasoning traces and questions that seek missing information.Source: §2.10 · p. 4
- A model trained to produce a visual output of the intended state before action decoding improved overall performance across standard benchmarks; Emma-X (Sun et al. 2024) generated high-level natural-language rationale and outperformed direct action generation.Source: §2.10 · p. 5
3 Emerging Trends to Face the Challenges
The section sketches emerging directions for addressing VLA challenges. Its proposals span hierarchical planning, spatial perception, action representations, world models, data synthesis, and post-training.
3.1 Hierarchical Planning
- A hierarchical planner decomposes goals into subtasks, assigns them to expert VLAs, and can insert reasoning before actions to ground motor commands in interpretable guidance.The proposed workflow also uses world-model simulation and evaluators for safety checks, with replanning if risks or inconsistencies are found.Source: §3.1 · pp. 5–6
3.2 Improving Perception by Spatial Understanding
- Depth-aware training with real and synthesized RGB-D frames is proposed to address VLM backbones’ limited depth-aware spatial understanding.Locate 3D could add objects to RGB frames to create depth-dependent questions, while generated scenes and video could provide additional training examples.Source: §3.2 · p. 6
3.3 Universal Action Representation
- Universal atomic action representations may reduce the data and effort needed to adapt to new robots, but do not yet provide zero-shot generalization.The section proposes prompting as a possible way to teach a VLA a new robot’s action space, requiring a fundamental shift in pre-training.Source: §3.3 · p. 6
3.4 Modeling World Dynamics
- World models can predict next states from state-action pairs or learn future-state embeddings, with embedding prediction avoiding pixel-level reconstruction for long-horizon prediction.The section notes that high-level state changes may matter more for robotics than exact pixel reproduction.Source: §3.4 · pp. 6–7
3.5 Data Synthesis with Visual Generative Models
- Video and simulated data could expand embodied-AI training, but generated videos lack explicit robot actions, motivating latent-action learning and alignment with real control signals.The proposal infers latent actions from consecutive states, then uses datasets with ground-truth actions to align those latents with robot control; evaluators can also filter simulated states for subgoal satisfaction.Source: §3.5 · p. 7
3.6 Post-Training
- A simulator-based reinforcement-learning approach would execute multiple action rollouts and score their outcomes, but depends on highly accurate, efficient robot-specific simulators that can be difficult and costly to obtain.The section frames defining reward signals for VLA models as a fundamental challenge.Source: §3.6 · p. 7
- World and video generative models could estimate rewards from predicted action outcomes, offering a route to VLA post-training without fully engineered simulators.Predicted states can be compared with subgoal states or assessed by an LLM; the section also proposes evaluators as virtual guardrails for safety and task compliance.Source: §3.6 · pp. 7–8
4 Conclusion
The paper revisits key VLA challenges, including multimodal sensing and perception and cross-robot action generalization, and proposes an exploratory framework aimed at addressing them.
4 Conclusion
- The conclusion presents an exploratory framework alongside a review of challenges the authors believe will shape future VLA research.The named challenges include multimodal sensing and perception and cross-robot action generalization.Source: §4 · p. 8