PaperlayerView exact revision on arXiv

Source-linked AI summary

LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization

Xueyang Zhou, Yangming Xu, Guiyao Tie, Yongchao Chen, Guowen Zhang, Duanfeng Chu, Pan Zhou, Lichao Sun

arXiv:2510.03827v2cs.CVcs.RO

TL;DR

LIBERO’s evaluation tasks closely match its training tasks, leaving generalization beyond small variations insufficiently assessed. LIBERO-PRO tests VLA models across systematic perturbations, under which models with above 90% standard-LIBERO success nearly collapse.

  • Problem

    Because LIBERO’s evaluation tasks match training tasks apart from marginal initial-state changes, it provides limited evidence about generalization to novel task compositions or environments.

    Source: §1 · p. 1, §3.2 · p. 5

  • Method

    LIBERO-PRO evaluates VLA models using coherent, executable perturbations across manipulated objects, initial states, task instructions, and environments.

    Source: §1 · p. 2

  • Results

    Models with above 90% success on standard LIBERO nearly collapse under object-position changes or minor task modifications.

    Source: §1 · p. 2, §5.2 · p. 8

  • Takeaways & Limitations

    The authors conclude that original LIBERO scores largely reflect rote memorization rather than comprehension, motivating evaluation of VLA generalization beyond fixed training mappings.

    Source: §7 · p. 8

Abstract

from arXiv · show

LIBERO has emerged as a widely adopted benchmark for evaluating Vision-Language-Action (VLA) models; however, its current training and evaluation settings are problematic, often leading to inflated performance estimates and preventing fair model comparison. To address these issues, we introduce LIBERO-PRO, an extended LIBERO benchmark that systematically evaluates model performance under reasonable perturbations across four dimensions: manipulated objects, initial states, task instructions, and environments. Experimental results reveal that, although existing models achieve over 90% accuracy under the standard LIBERO evaluation, their performance collapses to 0.0% under our generalized setting. Crucially, this discrepancy exposes the models' reliance on rote memorization of action sequences and environment layouts from the training set, rather than genuine task understanding or environmental perception. For instance, models persist in executing grasping actions when the target object is replaced with irrelevant items, and their outputs remain unchanged even when given corrupted instructions or even messy tokens. These findings expose the severe flaws in current evaluation practices, and we call on the community to abandon misleading methodologies in favor of robust assessments of model generalization and comprehension. Our code is available at: https://github.com/Zxy-MLlab/LIBERO-PRO.

1 INTRODUCTION

LIBERO [13] is the de facto VLA benchmark, but its training–evaluation setup can reward memorization over task understanding. LIBERO-PRO introduces coherent perturbations across four dimensions to test robustness and generalization.

Introduction

  • LIBERO [13] has become the de facto standard and common currency for VLA performance reporting, making its evaluation rigor consequential for research direction.The benchmark supports reproducible, fair, quantitative comparisons across methods.Source: §1 · p. 1
  • LIBERO evaluation tasks match training tasks apart from visually imperceptible initial-state changes, allowing overfit models to score near-perfectly before failing under minor shifts or phrasing changes.The authors argue that this inflates reported performance and fails to measure genuine task comprehension or environmental reasoning.Source: §1 · p. 1
  • Over 90% success on standard LIBERO [13] coexists with nearly unchanged trajectories under four modifications; all five VLA families exceed 70% on LIBERO but drop substantially on LIBERO-PRO.The modifications replace, relocate, or remove the target object, or corrupt the instruction with nonsensical tokens.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • LIBERO-PRO is a plug-and-play suite with logically coherent, physically executable perturbations across manipulated objects, initial states, task instructions, and environments.It supports single- and multi-dimensional randomized combinations.Source: §1 · p. 2
  • OpenVLA [8] and Pi0/Pi0.5 show near-complete collapse under several perturbation regimes, including altered object positions and minimal instruction paraphrases.The paraphrases remain semantically equivalent to expressions encountered during training.Source: §1 · p. 2

2 Observing Overfitting Issues

Across object, instruction, position, and task-composition perturbations, the models often reproduce familiar actions instead of grounding them in the actual observation and instruction. The results suggest reliance on memorized patterns rather than robust visuomotor understanding.

2.1 Visual perceptual hallucinations

  • The model grasps alphabet soup instead of the named salad dressing and reaches toward the usual location when the dressing is absent, indicating weak visual grounding and no existence check.Its actions appear guided by scene context and familiar trajectories rather than verifying the target object.Source: §2.1 · p. 3

2.2 Instruction understand hallucinations

  • With meaningless tokens, the model still performs nearly the same grasp-and-place actions, indicating that linguistic semantics contribute little and the policy retrieves a memorized visuomotor pattern.The tested tokens include “fdsgfdsgsd” and “xxx.”Source: §2.2 · p. 3

2.3 Location perception hallucinations

  • OpenVLA and pi0 success drops to zero beyond 0.2 units of object displacement; pi0.5 retains non-trivial performance only to roughly 0.4 units before also dropping to zero.The models reach toward memorized object locations instead of localizing displaced objects.Source: Figure 4 · p. 4, §2.3 · p. 4

2.4 Lack of task understanding

  • The model succeeds at bowl placement and stove activation separately but fails when they are combined, replaying action fragments instead of composing a coherent plan.This suggests reliance on shortcut feature-action correlations rather than robust, compositional visuomotor understanding.Source: §2.4 · p. 4

3 Problem Definition

The section formalizes VLA task success and argues that LIBERO’s test tasks remain too close to training tasks to assess broad generalization.

3.1 Formal VLA Task Definition

  • A VLA task instance is specified by an instruction, an environment instance, and a binary predicate indicating whether a state satisfies the task.The environment instance comprises visual context, interactive objects, and their initial spatial configuration.Source: §3.1 · p. 5
  • A policy acts from observations under environment dynamics, and its finite-horizon success probability captures the joint requirements of language grounding, perception, and action.Success is defined as reaching a state satisfying the instruction’s predicate within horizon H.Source: §3.1 · p. 5

3.2 Current LIBERO Evaluation Protocol

  • LIBERO test tasks typically preserve the training instruction, visual context, objects, and success predicate, varying only the initial object configuration slightly.Evaluation therefore primarily reflects robustness to small configuration changes; strong performance may overestimate generalization to novel task compositions or environments.Source: §3.2 · p. 5

4 Methodology

LIBERO-PRO evaluates VLA models under controlled perturbations to task components, with constraints intended to keep variants coherent and executable. The benchmark combines five perturbation types while excluding task changes from cross-type combinations.

4.1 Proposed Perturbation-Based Evaluation Framework

  • The framework perturbs four orthogonal dimensions—object attributes, object positions, instructions, and environment—and also defines task changes to target objects or actions.Variations include changes to object color, texture, or size; plausible object placements; paraphrased instructions; and changes to background, lighting, or texture.Source: §4.1 · pp. 5–6

4.2 Perturbation Constraints

  • Perturbations are bounded within dimension-specific neighborhoods and must create non-trivial task-level separation, with magnitude varied to characterize policy sensitivity.The constraints aim to keep variants semantically coherent and executable; task-level separation is measured using total variation distance.Source: §4.2 · p. 6

4.3 LIBERO-PRO Evaluation Benchmark

  • LIBERO-PRO is a plug-and-play benchmark spanning object, configuration, instruction, task, and environment variations, including combinations of perturbation types.Task perturbations introduce new object sets and goals while remaining executable; they are excluded from cross-type compositions, while the other types may be freely combined.Source: §4.3 · p. 7

5 Experiments

LIBERO-PRO evaluates VLA models under controlled perturbations and finds that high standard-LIBERO success does not translate to robust performance. Models nearly collapse under simple shifts, with some model-specific differences.

5.2 Main Results

  • Under position changes, Pi0.5 achieves a 0.38 success rate on the libero-goal task, versus 0 for OpenVLA and Pi0, despite standard-LIBERO success rates above 90%.Models nearly collapse under changes to object positions or minor task modifications, even when the changes use training components.Source: Figure 7 · p. 7, §5.2 · p. 8

5.1 Settings

  • LIBERO-PRO evaluates OpenVLA, pi0, and pi0.5 under controlled perturbations to objects, initial positions, task instructions, and environments.OpenVLA uses four official checkpoints trained on different LIBERO suites; pi0 and pi0.5 use one official checkpoint each, with 50 evaluation episodes per task.Source: §5.1 · p. 8

5.2 Main Results

  • Apparent robustness to object and instruction perturbations largely reflects overfitting rather than genuine task understanding.The evaluation includes object, semantic-instruction, and structural-instruction variations.Source: §5.2 · p. 8, §5.1 · p. 8
  • Environmental perturbations affect models differently, depending on environmental divergence and their reliance on memorized contexts.Robustness to environment changes is limited and varies across models and task suites.Source: Figure 7 · p. 7, §5.2 · p. 8

6 Related Work

VLA research spans large-scale pretrained and task-specific approaches, but fair comparison depends on standardized benchmarks. LIBERO has become the dominant evaluation protocol, while concerns about its reliability motivate LIBERO-PRO as a systematic robustness extension.

Vision-Language-Action Models

  • VLA models unite perception, language understanding, and action, with approaches ranging from large-scale multimodal pretraining to task-specific architectures [14, 10, 2].The growth of these approaches underscores the need for standardized benchmarks to enable fair, reproducible comparison [13, 15, 7].Source: §6 · p. 8

Benchmarks for VLA Evaluation

  • LIBERO [13] is the de facto VLA benchmark, with virtually all recent studies reporting results on its standardized task suite and unified metrics.Other testbeds include RLBench [7], CALVIN, RoboCasa [16], and BridgeData [19], but their adoption has been more limited.Source: §6 · p. 8

Limitations of Current Evaluation Practices

  • LIBERO scores may not reflect reliable performance under even minor task variations, and the field lacks a standardized framework for robust evaluation [21, 3, 11].Although some work has explored stronger algorithms or stress tests [4] [5] [18] [20], most studies continue to use the original setup; LIBERO-PRO is proposed as a plug-and-play extension for evaluating reasonable perturbations.Source: §6 · p. 8

7 Conclusion

The authors argue that standard LIBERO evaluation does not measure genuine VLA capability because it reuses training tasks with only imperceptible perturbations. They present LIBERO-PRO as a framework for assessing generalization and understanding.

7 Conclusion

  • Standard LIBERO evaluation reuses identical training and evaluation tasks with only imperceptible perturbations, so it fails to measure genuine capability.Source: §7 · p. 8
  • Accuracies above 90% largely reflect rote memorization of fixed training-set mappings rather than instruction comprehension or robust action strategies.Source: §7 · p. 8
  • The authors call on the community to adopt LIBERO-PRO as a principled framework for assessing VLA models’ generalization and understanding.Source: §7 · p. 8

Related papers

Loading 2510.03827v2…