Source-linked AI summary
Rethinking the Practicality of Vision-language-action Model: A Comprehensive Benchmark and An Improved Baseline
Wenxuan Song, Jiayi Chen, Xiaoquan Sun, Huashuo Lei, Yikai Qin, Wei Zhao, Pengxiang Ding, Han Zhao, Tongxin Wang, Pengxu Hou, Zhide Zhong, Haodong Yan, Donglin Wang, Jun Ma, Haoang Li
TL;DR
Existing VLAs face barriers from large parameter counts, costly pre-training, and limited embodiment applicability. The paper introduces CEBench and develops LLaVA-VLA from studies of practical VLA design choices. The model matches or surpasses much larger models, with real-world evaluation establishing it as an end-to-end VLA for mobile manipulation.
Problem
Existing VLAs are hindered by excessive parameter scales, costly pre-training, and limited applicability across embodiments.
Source: §I · p. 2Method
The paper introduces CEBench across embodiments and settings, studies practical VLA design choices, and builds LLaVA-VLA with a compact backbone and unified navigation-manipulation action space.
Source: §I · p. 2Results
LLaVA-VLA matches or surpasses models over 10× larger, especially under domain-randomized settings, and real-world evaluations establish it as the first end-to-end VLA for mobile manipulation.
Source: §I · p. 2, §VI · p. 8Takeaways & Limitations
The results demonstrate cross-embodiment versatility and visual generalization for a lightweight VLA designed for practical deployment.
Source: §I · p. 2, §VI · p. 8
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models have emerged as a generalist robotic agent. However, existing VLAs are hindered by excessive parameter scales, prohibitive pre-training requirements, and limited applicability to diverse embodiments. To improve the practicality of VLAs, we propose a comprehensive benchmark and an improved baseline. First, we propose CEBench, a new benchmark spanning diverse embodiments in both simulation and the real world with consideration of domain randomization. We collect 14.4k simulated trajectories and 1.6k real-world expert-curated trajectories to support training on CEBench. Second, using CEBench as our testbed, we study three critical aspects of VLAs' practicality and offer several key findings. Informed by these findings, we introduce LLaVA-VLA, a lightweight yet powerful VLA designed for practical deployment on consumer-grade GPUs. Architecturally, it integrates a compact VLM backbone with multi-view perception, proprioceptive tokenization, and action chunking. To eliminate reliance on costly pre-training, LLaVA-VLA adopts a two-stage training paradigm including post-training and fine-tuning. Furthermore, LLaVA-VLA extends the action space to unify navigation and manipulation. Experiments across embodiments demonstrate the capabilities of generalization and versatility of LLaVA-VLA , while real-world mobile manipulation experiments establish it as the first end-to-end VLA model for mobile manipulation. We will open-source all datasets, codes, and checkpoints upon acceptance to foster reproducibility and future research.
I. INTRODUCTION
The paper targets barriers to practical VLA deployment with CEBench and LLaVA-VLA. CEBench supports study of lightweight designs, training curricula, and unified action spaces across embodiments.
I. INTRODUCTION
- Existing VLAs face deployment barriers from large parameter counts, costly pre-training, and limited applicability to mobile manipulation.Prior lightweight approaches had not systematically examined practicality and remained unable to perform mobile manipulation tasks.Source: §I · p. 2
- CEBench spans single-arm, bimanual, and mobile-bimanual embodiments in simulation and the real world, with 14.4k simulated and 1.6k real-world demonstrations.The benchmark considers domain randomization.Source: §I · p. 2
- The authors study lightweight designs, training curricula, and unified action spaces, then use those findings to inform LLaVA-VLA.The model combines a compact VLM, multi-view input, proprioception tokenization, action chunking, and post-training followed by task-specific fine-tuning.Source: §I · p. 2
- LLaVA-VLA matches or surpasses models over 10× larger, especially under domain-randomized settings, and is reported as the first end-to-end VLA for mobile manipulation.Cross-embodiment experiments also demonstrate versatility and the effectiveness of its hybrid action space.Source: §I · p. 2
- The authors plan to release the datasets, code, and checkpoints upon acceptance.The stated aim is to provide a reference for future open-source VLA research.Source: §I · p. 2
II. RELATED WORKS
The related work traces the development of vision-language models and vision-language-action models. It notes that current VLA approaches remain demanding to deploy and limited in mobile manipulation.
A. Vision-Language Models
- Vision-language models extend language models to multimodal tasks, using visual encoders and multistage alignment training to support cross-modal alignment and visual reasoning.The passage names CLIP and SigLIP as representative vision encoders.Source: A. Vision-Language Models · p. 2
B. VLA Models
- Early VLA work trained on web-scale vision-language data and robot trajectories, while later systems adapted pre-trained VLMs or released open-source models.Examples include RoboFlamingo and OpenVLA.Source: B. VLA Models · p. 2
- Recent VLA work explores faster inference, added low-level perception, and lightweight bridges, but models remain computationally demanding for resource-constrained and mobile settings.The passage also describes work on humanoid control and benchmarking VLM choices.Source: B. VLA Models · p. 3
III. CEBENCH
CEBench is designed for practical VLA evaluation across embodiments, simulation, and the real world. Its system setup and datasets include both bimanual and mobile-manipulation tasks.
III. CEBENCH
- CEBench addresses gaps in practical evaluation across embodiments, domain randomization, and mobile manipulation, treating CALVIN as a subset and adding RoboTwin and real-world data.The benchmark is intended for systematic evaluation of VLA practicality.Source: §III · p. 3
A. System Setup
- The evaluation spans CALVIN single-arm manipulation, RoboTwin bimanual simulation, and real-world bimanual mobile manipulation using the Cobot-Magic robot.The mobile robot has four Piper arms—two master and two puppet arms—and captures 480 × 640 RGB images at 30 Hz.Source: A. System Setup · p. 3
B. Datasets
- The dataset includes 14.4k simulated trajectories across 36 RoboTwin tasks and 200 real-world trajectories for each of eight tasks.Two real-world tasks involve moving to fetch bottles and moving to open a drawer; the simulated collection used simplified tabletop conditions.Source: B. Datasets · p. 3
C. Evaluation and Metrics
Evaluation uses benchmark-specific task metrics and tests generalization in unseen or varied settings.
C. Evaluation and Metrics
- CALVIN reports subtask success rates and average success length; RoboTwin tests seen and unseen domain-randomized settings, while real-world tests vary tabletop appearance and distractors.RoboTwin randomizes clutter, lighting, textures, and table heights.Source: C. Evaluation and Metrics · p. 3
D. Baselines
The baselines cover policies with fewer than 1B parameters evaluated in RoboTwin and the real world, alongside approaches spanning imitation learning, diffusion, and VLA models.
- Comparisons in RoboTwin and the real world include policies with fewer than 1B parameters.Source: D. Baselines · p. 3
- ACT uses CVAE imitation learning with action chunking and temporal ensembling, while Diffusion Policy predicts actions through conditional denoising.Source: D. Baselines · pp. 3–4
- TinyVLA is a compact multimodal VLA baseline, whereas RDT targets bimanual manipulation with diffusion, unified actions, and multimodal inputs.Source: D. Baselines · p. 4
IV. LLAVA-VLA AND ITS ASSOCIATED FINDINGS
The paper frames its VLA design around model scale, pre-training, and a unified action space. LLaVA-VLA combines a compact VLM backbone with multi-view images, proprioception, action chunks, and hybrid navigation-manipulation actions.
IV. LLAVA-VLA AND ITS ASSOCIATED FINDINGS
- The study asks whether compact models can match larger counterparts and which techniques make that possible.Source: §IV · p. 4
- It examines whether small models need pre-training to perform tasks in specific scenarios.Source: §IV · p. 4
- It investigates a unified action space for manipulation across fixed-base and mobile embodiments.Source: §IV · p. 4
A. LLaVA-VLA
- LLaVA-VLA uses a pre-trained LLaVA-OneVision-0.5B backbone, concatenated multi-view images, proprioceptive signals, and an action tokenizer to produce action chunks.Its hybrid action space uses direction and value tokens to switch between navigation and manipulation.Source: A. LLaVA-VLA · p. 4
B. Study on Lightweight Designs (Q1)
The lightweight-design study reports that a small VLA can rival larger models and examines visual input, proprioception, and action chunking as design choices.
B. Study on Lightweight Designs (Q1)
- 50.6% on the last task exceeds RoboFlamingo (3B) at 23.5% and OpenVLA (7B) at 43.5%; on the first task, LLaVA-VLA-0.5B reaches 96.2% versus its 7B counterpart at 97.4%.The paper reports performance on par with 7B models despite using less than 10% of their parameters.Source: B. Study on Lightweight Designs (Q1) · p. 4
- Vertically concatenating first- and third-person views into one image reduces token count while preserving multi-view information; the perspectives provide global context and precise gripper cues.The approach also captures disparity information needed for a three-dimensional understanding of the scene.Source: B. Study on Lightweight Designs (Q1) · p. 4
- Tokenizing proprioception works better than encoding it with linear layers and helps the VLM generate coherent actions from physical-state information.Source: B. Study on Lightweight Designs (Q1) · p. 5
- Predicting action chunks of size five is reported to strengthen planning and improve temporal coherence in manipulation.Source: B. Study on Lightweight Designs (Q1) · p. 5
C. Study on Training Curriculum (Q2)
The training-curriculum study favors in-domain post-training followed by fine-tuning, while the action-space study examines discrete actions, token granularity, and a navigation-manipulation interface.
C. Study on Training Curriculum (Q2)
- Post-training on in-domain multi-task data followed by single-task fine-tuning achieves the best overall performance in simulated and real-world dual-arm experiments.The authors report that domain-specific data can establish vision-to-action mappings when tasks provide sufficient diversity in goals and scenes.Source: C. Study on Training Curriculum (Q2) · p. 5
D. Study on Action Space (Q3)
- With action chunking, discrete actions achieve competitive performance without a diffusion head, while preserving the model’s autoregressive property.Source: D. Study on Action Space (Q3) · p. 5
- Using more action bins yields lower success rates than coarser discretization, with the paper associating overly fine tokenization with reduced efficiency and generalization.Source: D. Study on Action Space (Q3) · p. 5, Table IX · p. 5
- The hybrid action space places a navigation direction token before a value token and, after “stop,” appends manipulation values to support switching between navigation and manipulation.The authors report that tokenizing navigation like manipulation was unstable and could restart robot movement during manipulation.Source: D. Study on Action Space (Q3) · p. 6
V. EVALUATION OF LLAVA-VLA
The final LLaVA-VLA architecture is evaluated on CEBench across manipulation, visual generalization, cross-embodiment versatility, and mobile manipulation.
A. Training Setup
The section reports CALVIN and RoboTwin results for LLaVA-VLA and describes the training setup used for these evaluations.
A. Training Setup
- Training used 8 NVIDIA H100 GPUs for post-training and 1 NVIDIA 4090 GPU for fine-tuning, with CALVIN ABC→D taking about six hours for one post-training epoch.RoboTwin and real-world bimanual manipulation used 2 post-training epochs followed by 8 fine-tuning epochs.Source: A. Training Setup · p. 6
B. Single-arm Manipulation on CALVIN
- 3.68 was the best average completed trajectory length against other Large VLA Models on CALVIN; LLaVA-VLA also topped all subtask success rates.Against 3D-aware baselines, its average completed trajectory length was higher by 0.33.Source: B. Single-arm Manipulation on CALVIN · p. 6
C. Bimanual Manipulation on RoboTwin
- Across 100 evaluations in each setting, small, non-pretrained LLaVA-VLA achieved 40.3% success on seen tasks and 28.6% on domain-randomization tasks, outperforming diffusion- and VAE-based baselines.Source: C. Bimanual Manipulation on RoboTwin · p. 7
D. Bimanual Manipulation in the Real World
Real-world evaluations report LLaVA-VLA’s performance on fixed-base bimanual and mobile manipulation. The conclusion presents its cross-embodiment versatility and visual generalization as supported findings.
D. Bimanual Manipulation in the Real World
- Across six fixed-base real-world tasks, LLaVA-VLA outperformed ACT and TinyVLA and was less affected by unseen background variations and distractor objects.Source: D. Bimanual Manipulation in the Real… · p. 7
- ACT achieved only 10% success on mobile manipulation; LLaVA-VLA learned from diverse multitask trajectories for instruction following and precise mobile manipulation.Source: D. Bimanual Manipulation in the Real… · p. 7
VI. CONCLUSION
- The conclusion identifies cross-embodiment versatility, visual generalization, and real-world mobile manipulation as demonstrated capabilities of LLaVA-VLA.Source: §VI · p. 8