Source-linked AI summary
VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks
Shiduo Zhang, Zhe Xu, Peiju Liu, Xiaopeng Yu, Yuan Li, Qinghui Gao, Zhaoye Fei, Zhangyue Yin, Zuxuan Wu, Yu-Gang Jiang, Xipeng Qiu
TL;DR
Existing benchmarks inadequately evaluate foundation-model-based language-conditioned manipulation involving implicit intent, common sense, diverse scenes, and long-horizon reasoning. VLABench introduces a 100-category benchmark with automated data construction and multidimensional evaluation, and experiments show that current VLAs and VLM-based workflows still face substantial challenges.
Problem
Existing benchmarks insufficiently cover semantic intent, common-sense transfer, diverse visual scenes, and sophisticated multi-step reasoning needed by foundation-model-based manipulation.
Method
VLABench provides 100 task categories spanning visual, linguistic, spatial, physical, knowledge, and reasoning abilities, supported by standardized automated data construction and evaluation.
Results
Current pretrained VLAs perform poorly on VLABench, while VLM evaluations show mixed effects from chain-of-thought and few-shot prompting across task types.
Takeaways & Limitations
VLABench provides a standardized setting for testing whether embodied models generalize across skills, knowledge, language, vision, planning, and action.
Takeaways & Limitations
Current VLA pretraining data are much smaller than language-model data, leaving VLA scaling behavior an open topic.
Abstract
from arXiv · showhide
General-purposed embodied agents are designed to understand the users' natural instructions or intentions and act precisely to complete universal tasks. Recently, methods based on foundation models especially Vision-Language-Action models (VLAs) have shown a substantial potential to solve language-conditioned manipulation (LCM) tasks well. However, existing benchmarks do not adequately meet the needs of VLAs and relative algorithms. To better define such general-purpose tasks in the context of LLMs and advance the research in VLAs, we present VLABench, an open-source benchmark for evaluating universal LCM task learning. VLABench provides 100 carefully designed categories of tasks, with strong randomization in each category of task and a total of 2000+ objects. VLABench stands out from previous benchmarks in four key aspects: 1) tasks requiring world knowledge and common sense transfer, 2) natural language instructions with implicit human intentions rather than templates, 3) long-horizon tasks demanding multi-step reasoning, and 4) evaluation of both action policies and language model capabilities. The benchmark assesses multiple competencies including understanding of mesh\&texture, spatial relationship, semantic instruction, physical laws, knowledge transfer and reasoning, etc. To support the downstream finetuning, we provide high-quality training data collected via an automated framework incorporating heuristic skills and prior information. The experimental results indicate that both the current state-of-the-art pretrained VLAs and the workflow based on VLMs face challenges in our tasks.
1. Introduction
Existing benchmarks do not fully test foundation-model-based manipulation, especially semantic intent, common-sense knowledge, visual diversity, and multi-step reasoning. VLABench addresses these gaps with 100 task categories, over 2,000 objects and scenes, automated data construction, and evaluations spanning language, vision, planning, knowledge transfer, and action.
- Existing simulation benchmarks provide diverse tasks but insufficiently address nuanced user intent, common sense, visual-scene interpretation, and sophisticated multi-step reasoning.
- VLABench evaluates common sense, mesh and texture, semantic instructions, spatial understanding, physical rules, and reasoning across 100 task categories.Its collection contains over 2,000 3D objects and scenes for varied visual contexts.
- The benchmark supports generalization assessment across visual, linguistic, planning, knowledge-transfer, and action dimensions through learning across multiple skills.
- An automated data-collection framework constructs standardized datasets for training and fine-tuning, while experiments compare pretrained VLAs, foundation-model workflows, and VLMs.
- Current pretrained VLAs lack the strong generalization observed in LLMs, and state-of-the-art VLMs remain limited in embodied scenarios.
2. Related Works
Prior benchmarks often emphasize skill learning, interface-based interaction, or template instructions, limiting their coverage of long-horizon physical manipulation and implicit language-guided goals. VLABench instead combines diverse real-world-aligned tasks with natural interaction, common sense, and scalable automated simulation data.
- Many manipulation benchmarks focus on skill learning and insufficiently address long-horizon planning.
- Room-scale benchmarks may require long-term memory or reasoning but typically use interfaces rather than direct physical manipulation.
- Existing benchmarks often rely on template instructions that explicitly specify robot actions, overlooking language's guiding role.
- VLABench introduces natural human interaction, implicit goal-oriented semantics, and common-sense requirements into robot manipulation.
- VLABench expands task and scenario diversity through automated simulated-data generation, addressing scalability and limited diversity in prior datasets.
3. VLABench
VLABench combines primitive and composite manipulation tasks spanning visual, linguistic, spatial, semantic, physical, and world-knowledge requirements. It evaluates generalization and progress using diverse randomized simulation, automated trajectories, and metrics beyond binary task success.
- VLABench contains 60 primitive and 40 composite tasks covering picking, doors, drawers, hanging, tools, buttons, insertion, pouring, twisting, and exploration.
- Its task dimensions include spatial relations, common-sense and world knowledge, and semantically implicit instructions requiring interpretation of natural conversation.
- Composite tasks combine multiple skills, long-term planning, and multi-step reasoning from instructions, scenes, and game rules.Their average episode length exceeds 500 timesteps, compared with 120 timesteps for primitive tasks.
- Generalization evaluations contrast seen and unseen objects while testing visual recognition, common-sense knowledge, and unfamiliar instruction tokens.
- Progress Score supplements binary success with a graduated measure based on correct object or receptacle decisions and completed sub-steps.The benchmark also reports Skill Recall Rate, Parameter Recall Rate, Skill&Parameter Recall Rate, and Precise Matching Rate for VLMs.
- The Mujoco-based framework uses modular entities, domain randomization, and scalable automated trajectory generation to produce diverse simulated tasks.
4. Experiments
The experiments evaluate pretrained VLAs, foundation-model workflows, and VLMs on generalization, manipulation, and reasoning tasks. Results show persistent weaknesses in semantic transfer, long-horizon planning, physical reasoning, and complex embodied scenarios, while visual perception improves spatial reasoning but can reduce semantic and reasoning scores.
- Generalization Ability of VLAs: Pretrained VLAs were tested on unseen objects, language instructions, similar tasks, and composite tasks to assess generalization and long-horizon reasoning.The evaluation includes visual generalization and knowledge transfer, language instruction generalization, unseen-task generalization, and composite-task evaluation.
- Generalization Ability of VLAs: Current VLA architectures and pretraining approaches remain insufficient for semantic understanding, skill transfer, and long-horizon planning.The conclusion is based on evaluations involving out-of-domain semantically rich language, unseen similar tasks, and composite tasks.
- Performance of Workflow Utilizing Foundation Model: Voxposer achieved Progress Scores of 30–40 on basic tasks, but LLM-driven motion planning often caused grasping failures when information for grasp planning was limited.The comparison evaluates Voxposer with and without visual perception alongside CoPA.
- Performance of Workflow Utilizing Foundation Model: Adding visual perception slightly reduced semantic-understanding and reasoning performance while significantly improving spatial reasoning.Without visual input, the foundational LLM maintained relatively stable semantic and reasoning scores but struggled with spatial accuracy.
- Performance of Workflow Utilizing Foundation Model: The evaluated workflows lacked closed-loop feedback for dynamic physical reasoning and struggled with high-complexity tasks and long-horizon task decomposition.Their modular design was reported to limit the upper bound of performance.
- Comprehensive Ability of VLMs: VLM performance fell short on complex scenarios, abstract instructions, and long-term decomposition, although Qwen2-VL-7B-Instruct surpassed GPT-4-turbo-2024-04-09 on some dimensions.Only GPT-4o achieved reasoning scores comparable to its other dimensions; the other models scored around 20 points in reasoning.
5. Conclusion
VLABench is presented as a large-scale benchmark for long-horizon, multidimensional reasoning, combining standardized tasks with scalable automated data collection. Experiments show that current VLAs and VLMs still face significant challenges, while research on robotics scaling remains uncertain.
- Benchmark and scope: VLABench provides 100 standardized task settings for evaluating long-horizon and multidimensional reasoning across vision, language, knowledge, planning, and decision-making.The benchmark is intended to define capabilities associated with intelligent embodied agents.
- Data and evaluation: The benchmark includes a scalable automated data-collection framework and standardized datasets for fair comparison and future VLA development.The framework is described as supporting larger-scale training and benchmarking.
- Findings: Experiments indicate that current VLAs and VLMs face significant challenges on VLABench tasks, with substantial uncertainty remaining about robotics scaling.The conclusion frames these findings as an open research challenge.
Supplementary Material
The supplementary material illustrates a Select Toy data instance combining multiple observation modalities with an expert trajectory.
- Task observations: A Select Toy instance contains multi-camera positions, multi-view RGB images, 3D point clouds, and an expert trajectory.These components provide visual and trajectory information for the example task instance.
6. Benchmark Implementation
VLABench combines primitive and composite manipulation tasks with long-horizon, multistep reasoning requirements, broad asset coverage, and substantial visual and domain randomization. Its design emphasizes diverse task instances and competencies spanning spatial, semantic, commonsense, physical, and logical understanding.
- Task composition: VLABench includes 60 primitive and 40 composite tasks covering manipulation skills and higher-level evaluation dimensions.
- Long-horizon reasoning: 27.0% and 35.1% are the reported episode-length margins over RoboCasa Atomic and RoboCasa Composite, respectively.VLABench has the longest horizon among the compared primitive and composite task sets.
- Long-horizon reasoning: Multistep reasoning includes hidden-semantic inference, visual and commonsense integration, spatial reasoning, subtask planning, and logical reasoning.
- Task diversity: Task instances vary target objects, receptacles, objectives, distractors, spatial configurations, and poses while remaining grouped under broad task categories.
- Data modalities: The benchmark provides multi-view RGB-D images, semantic segmentation, point clouds, language instructions, actions, rewards, and episode termination information.
- Domain randomization: Domain randomization varies object meshes and textures, positions and orientations, scales, and visual scene conditions.The documented defaults include position offsets in [-0.05, 0.05], yaw in [-π/10, π/10], and mesh scaling in [0.95, 1.05].
7. Simulation and Framework
VLABench supports varied simulation contexts through diverse scenes, multiple robot embodiments, and an asset library with annotated grasping and placement information. These components broaden the environments and physical platforms available for task generation and evaluation.
- Scene diversity: The benchmark includes over 20 scene types spanning household settings and dynamic contexts such as shopping malls, laboratories, and medical rooms.
- Embodiment diversity: VLABench supports single-arm and dual-arm robots, humanoids, quadrupeds with end-effectors, and mobile robots.
- Asset representation: Its asset library separates graspable objects from receptacles and annotates grasp points, bounding boxes, and placement points.
8. Dataset Building
VLABench builds datasets through a domain-specific language that represents manipulation skills, parameters, and execution sequences, then automates task-specific data collection. Instruction generation adds natural, implicit, commonsense, semantic, and composite-task scenarios.
- Task representation: The DSL represents atomic skills, their parameters, and sequential or hierarchical task-execution sequences.
- Automated collection: Automated collection instantiates predefined task sequences while dynamically generating grasp poses and target positions from the simulation environment.
- Instruction generation: The instruction-generation framework uses object lists, target entities, everyday needs, and task-specific descriptions to create linguistically rich commands.
- Implicit semantics: Semantic prompts require implied requests that avoid naming the target directly and reflect casual real-life scenarios.
- Composite tasks: Composite tasks combine primitive abilities within scenario-specific prompts, including instructions for realistic book-classification contexts.
9. Experiment Implementation
Experiments fine-tune and evaluate pretrained VLAs, foundation-model workflows, and VLMs using standardized procedures. VLM outputs are assessed by matching generated skill graphs against ground-truth skills, parameters, combinations, and logical dependencies, with a weighted aggregate score.
- Model training: OpenVLA, Octo, and RDT-1B are fine-tuned on VLABench data using their original code and authors’ default hyperparameters.
- Evaluation setup: VLM evaluation simplifies outputs to DSL generation and computes scores through direct graph matching.
- Evaluation metrics: Skill Recall Rate measures whether the model identifies and invokes the correct skills against manually labeled skill lists.
- Evaluation metrics: Skill-and-parameter recall requires jointly identifying correct skills and their exact parameters, providing a stricter measure of scene understanding and task planning.
- Evaluation metrics: Precise Matching Rate evaluates skill-parameter matches together with logical dependencies encoded in directed acyclic graphs.A match requires the same skill and parameters plus satisfied incoming and outgoing dependency relationships.
- Aggregate scoring: Score = w1 · SR + w2 · PR + w3 · SPR + w4 · PM, with weights constrained by w1 + w2 + w3 + w4 = 1.
10. Detailed Analysis and Case Study
Experiments expose limitations in current VLAs and foundation-model workflows across perception, planning, modular integration, prompting, and few-shot learning. These analyses show both persistent failure modes and context-dependent improvements on VLABench tasks.
- VLA Analysis: Current open-source VLAs perform poorly on VLABench tasks, reflecting limitations in their architectures and pretraining for diverse downstream data.The benchmark results contrast with the stronger downstream adaptation reported for LLMs under minimal fine-tuning.
- VLA Analysis: Single-image inputs, low resolution, and missing temporal memory hinder precise spatial perception and cause repeated behavior in long-horizon tasks.OpenVLA and Octo process 224×224 single images, while current models lack representations for historical actions and temporal sequences.
- Workflow Analysis: Perception and planning failures cause incorrect pose adjustment, invalid grasps, and task failures for occluded, irregularly shaped, or interaction-heavy objects.The reported cases include failures to rotate the arm, recognize object interactions, and generate valid grasp points.
- Workflow Analysis: Modular workflows lose robustness at component interfaces when language-model outputs or generated constraints fail to become correct waypoints.The resulting errors reduce adaptation across diverse operational scenarios.
- VLM Analysis: Chain-of-Thought improves several VLMs on challenging reasoning and physics tasks, but common-sense performance may stagnate or decline and MiniCPM scores drop to 0.0.InternVL2 improves overall, while LLaVA-NeXT and Qwen2-VL benefit especially on difficult reasoning scenarios.
- VLM Analysis: Increasing Qwen2-VL few-shot examples from 0 to 7 improves multimodal reasoning, while diverse semantic and spatial tasks show diminishing returns beyond two or three shots.The benefit is strongest under Chain-of-Thought prompting and depends on task context.