Source-linked AI summary
LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization
Xueyang Zhou, Yangming Xu, Guiyao Tie, Yongchao Chen, Guowen Zhang, Duanfeng Chu, Pan Zhou, Lichao Sun
TL;DR
The paper argues that LIBERO’s near-identical training and evaluation setup can overestimate VLA competence. It introduces LIBERO-PRO, which applies structured perturbations across four dimensions, and finds that reported performance above 90% largely reflects memorization rather than robust understanding.
Problem
LIBERO reuses nearly identical training and evaluation tasks, limiting assessment of generalization beyond narrow variations in object configurations.
Method
LIBERO-PRO extends LIBERO with coherent, physically executable perturbations across manipulated objects, initial states, task instructions, and environments.
Results
Reported accuracies above 90% on standard LIBERO largely reflect rote memorization, while models nearly collapse under object-position changes or minor task modifications.
Takeaways & Limitations
Robust VLA evaluation should incorporate structured perturbations to assess genuine generalization and understanding capabilities.
Abstract
from arXiv · showhide
LIBERO has emerged as a widely adopted benchmark for evaluating Vision-Language-Action (VLA) models; however, its current training and evaluation settings are problematic, often leading to inflated performance estimates and preventing fair model comparison. To address these issues, we introduce LIBERO-PRO, an extended LIBERO benchmark that systematically evaluates model performance under reasonable perturbations across four dimensions: manipulated objects, initial states, task instructions, and environments. Experimental results reveal that, although existing models achieve over 90% accuracy under the standard LIBERO evaluation, their performance collapses to 0.0% under our generalized setting. Crucially, this discrepancy exposes the models' reliance on rote memorization of action sequences and environment layouts from the training set, rather than genuine task understanding or environmental perception. For instance, models persist in executing grasping actions when the target object is replaced with irrelevant items, and their outputs remain unchanged even when given corrupted instructions or even messy tokens. These findings expose the severe flaws in current evaluation practices, and we call on the community to abandon misleading methodologies in favor of robust assessments of model generalization and comprehension. Our code is available at: https://github.com/Zxy-MLlab/LIBERO-PRO.
1 INTRODUCTION
LIBERO is widely used for VLA evaluation, but its near-identical training and evaluation tasks can inflate scores and obscure genuine comprehension. LIBERO-PRO addresses this with structured perturbations and reveals substantial degradation in current models.
- Motivation: LIBERO functions as the field’s widely adopted benchmark, making its methodological rigor important for VLA research.Both large-scale pre-trained and task-specific systems are evaluated on LIBERO, which serves as a common evaluation reference.
- Motivation: Identical training and evaluation tasks with barely perceptible state changes allow overfit models to achieve near-perfect scores.Accuracy collapses under minor changes in object position or task phrasing, indicating that standard evaluation does not adequately test comprehension or environmental reasoning.
- Findings: Over 90% success on standard LIBERO contrasts with nearly unchanged trajectories after object substitutions, relocation, removal, or nonsensical instructions.The observed behavior indicates reliance on rote recall rather than robust task understanding.
- LIBERO-PRO: LIBERO-PRO evaluates VLA models through coherent, executable perturbations across manipulated objects, initial states, task instructions, and environments.The suite is designed to provide a more comprehensive and reliable measure of performance under realistic variation.
- Findings: Leading models including OpenVLA, Pi0, and Pi0.5 show significant degradation on LIBERO-PRO, motivating a fairer and more reproducible evaluation standard.LIBERO-PRO is presented as a plug-and-play extension supporting systematic perturbations and randomized combinations.
2 Observing Overfitting Issues
Qualitative and perturbation experiments reveal that VLA policies often overfit training configurations, showing brittle perception, weak instruction grounding, poor spatial generalization, and limited compositional reasoning.
- Over 95% LIBERO success coexists with systematic failures indicating severe overfitting to the training distribution.Observed failures include brittle visual perception, instruction-semantic insensitivity, spatial sensitivity, and compositional weakness.
- 2.2 Instruction understand hallucinations: Replacing the target salad dressing with alphabet soup still triggers grasping, showing unreliable binding between instructions and visual entities.The model reaches toward and grasps the irrelevant object while attempting the instructed placement.
- 2.2 Instruction understand hallucinations: Removing the target object still elicits a grasp toward its usual location, indicating no online verification of object existence.The model performs the grasping motion despite the salad dressing being absent.
- 2.2 Instruction understand hallucinations: Corrupted instructions produce nearly unchanged trajectories, indicating that linguistic semantics contribute little to action selection.The policy still grasps the salad dressing and attempts placement after receiving nonsensical tokens, suggesting retrieval of a memorized visuomotor pattern.
- 2.3 Location perception hallucinations: Task success for OpenVLA and pi0 collapses to zero beyond 0.2 displacement units, while pi0.5 remains non-trivial only to roughly 0.4 units.The models move toward memorized coordinates rather than the displaced object's actual position.
- 2.4 Lack of task understanding: The model executes two independent instructions but consistently fails when they are combined into one multi-step instruction.The failure indicates dependence on memorized action fragments rather than coherent temporal or causal composition.
3 Problem Definition
The paper formalizes VLA tasks as language-instruction and environment instances evaluated by a success predicate, then identifies limited distributional shift in LIBERO’s protocol. Because test tasks largely preserve training instructions, objects, environments, and goals, high scores may overestimate broader generalization.
- Formal VLA Task Definition: A VLA task instance consists of an instruction, an environment instance, and a binary success predicate over states and instructions.The environment includes visual context, interactive objects, and their initial spatial configuration.
- Formal VLA Task Definition: A policy generates trajectories from agent observations and language instructions under environment dynamics, and evaluation measures success within a finite horizon.The objective captures language grounding, perception, and action jointly.
- LIBERO Evaluation Protocol: In LIBERO, test tasks typically retain the training instruction, visual context, objects, and success goal, with only minor initial-configuration perturbations.This construction keeps test tasks close to their corresponding training tasks.
- LIBERO Evaluation Protocol: Consequently, LIBERO primarily tests robustness to small object-configuration variations rather than generalization to novel task compositions or environments.Strong performance can therefore reflect interpolation within a narrow neighborhood of the training distribution.
4 Methodology
LIBERO-PRO evaluates VLA models through controlled perturbations of objects, configurations, instructions, tasks, and environments while preserving task coherence and executability. It supports varied single- and multi-dimensional distribution shifts, with task perturbations excluded from cross-type compositions for feasibility.
- Perturbation-Based Evaluation Framework: LIBERO-PRO generates perturbed task variants across object attributes, initial configurations, instructions, and environments from a formal task representation.The framework represents each variant as T^(k) = ϕ_k(T), with k covering the perturbation dimensions.
- Perturbation Types: Object perturbations change non-essential color, texture, or size while preserving semantic equivalence.These changes target visual attributes without changing the object’s intended meaning.
- Perturbation Types: Configuration perturbations alter absolute and relative object positions while maintaining physical plausibility.The goal is to vary initial spatial arrangements without making tasks infeasible.
- Perturbation Types: Instruction perturbations rephrase task descriptions while preserving task intent, whereas task perturbations modify target objects or actions within the training distribution.The task-level formulation is represented as T^(T) = (l, e, G^(T)).
- Perturbation Types: Environment perturbations vary visual context such as background, lighting, or texture without altering task feasibility.This tests behavior under changed environmental appearance.
- Perturbation Constraints: Perturbations are bounded by dimension-specific magnitudes and constrained to induce meaningful behavioral differences.Varying δ_k produces multiple variants and characterizes policy sensitivity to perturbation intensity.
- LIBERO-PRO Benchmark: Task perturbations are excluded from cross-type compositions, while object, configuration, instruction, and environment perturbations can be freely combined.LIBERO-PRO is designed as a plug-and-play benchmark supporting systematic single- and multi-dimensional evaluation.
5 Experiments
Experiments evaluate three representative VLA models with LIBERO-PRO’s controlled perturbations across objects, positions, instructions, and environments. Models that exceed 90% on standard LIBERO nearly collapse under perturbations, although sensitivity differs across models and tasks.
- Robustness is limited across object, semantic-instruction, and environment changes and varies across models and task suites.
- Three representative models—OpenVLA, pi0, and pi0.5—are evaluated with LIBERO-PRO perturbations across objects, positions, instructions, and environments.
- Above 90% success on standard LIBERO falls to near-collapse under object-position changes or minor task modifications.
- Environmental perturbations affect models differently according to environmental divergence and reliance on memorized contexts.
- Pi0.5 reaches 0.38 success on libero-goal under position changes, versus 0 for OpenVLA and Pi0.
6 Related Work
Related work positions LIBERO as the dominant VLA benchmark among several alternatives, while prior studies question whether its high scores reflect reliable performance. The field still lacks a systematic and standardized evaluation of these concerns.
- VLA models aim to unify perception, natural-language understanding, and action execution for embodied AI.
- LIBERO has become the most widely adopted protocol for evaluating VLA models, whereas RLBench, CALVIN, RoboCasa, and BridgeData have more limited broader adoption.
- Several studies report that high LIBERO scores do not reliably translate to performance on tasks with minor variations.
- Researchers have proposed stronger algorithms and stress tests, but systematic and standardized evaluation remains lacking.
7 Conclusion
The conclusion finds that standard LIBERO evaluation reuses training tasks with only imperceptible perturbations, so reported accuracies above 90% largely reflect memorization. It advocates LIBERO-PRO for assessing generalization and understanding.
- Standard LIBERO reuses identical training and evaluation tasks with only imperceptible perturbations, failing to measure genuine capability.
- Above 90% reported accuracy largely reflects rote memorization of fixed training-set mappings rather than instruction comprehension or robust action strategies.
- LIBERO-PRO is presented as a principled framework for assessing real generalization and understanding capabilities.