PaperlayerView exact revision on arXiv

Source-linked AI summary

LIBERO-X: Robustness Litmus for Vision-Language-Action Models

Guodong Wang, Chenkai Zhang, Qingjie Liu, Jinjin Zhang, Jiancheng Cai, Junjie Liu, Xinmin Liu

arXiv:2602.06556v1cs.CVcs.AIcs.RO

TL;DR

Existing VLA benchmarks provide limited evidence under realistic distribution shifts, motivating a benchmark that jointly improves evaluation protocols and training-data diversity. LIBERO-X introduces hierarchical multi-level evaluation and a high-diversity teleoperated dataset, and experiments show performance degrades as task and scene complexity increase. The benchmark is intended to provide a more faithful basis for assessing VLA generalization and robustness.

  • Problem

    Existing benchmarks can provide overly optimistic or misleading assessments because their perturbations, train–test scenarios, and task distributions insufficiently capture realistic multi-source shifts.

    Source: §I · p. 1

  • Method

    LIBERO-X combines a progressively challenging multi-level evaluation protocol with a high-diversity human-teleoperation dataset containing multiple fine-grained tasks per scene.

    Source: §I · pp. 1–2

  • Results

    Performance degrades notably as task and scene complexity increase, revealing limitations in scene comprehension and instruction grounding across representative VLA models.

    Source: §I · p. 3, §V · p. 8

  • Takeaways & Limitations

    LIBERO-X provides a more faithful and rigorous framework for assessing VLA generalization and robustness under multi-dimensional distribution shifts.

    Source: §I · p. 2, §V · p. 8

Abstract

from arXiv · show

Reliable benchmarking is critical for advancing Vision-Language-Action (VLA) models, as it reveals their generalization, robustness, and alignment of perception with language-driven manipulation tasks. However, existing benchmarks often provide limited or misleading assessments due to insufficient evaluation protocols that inadequately capture real-world distribution shifts. This work systematically rethinks VLA benchmarking from both evaluation and data perspectives, introducing LIBERO-X, a benchmark featuring: 1) A hierarchical evaluation protocol with progressive difficulty levels targeting three core capabilities: spatial generalization, object recognition, and task instruction understanding. This design enables fine-grained analysis of performance degradation under increasing environmental and task complexity; 2) A high-diversity training dataset collected via human teleoperation, where each scene supports multiple fine-grained manipulation objectives to bridge the train-evaluation distribution gap. Experiments with representative VLA models reveal significant performance drops under cumulative perturbations, exposing persistent limitations in scene comprehension and instruction grounding. By integrating hierarchical evaluation with diverse training data, LIBERO-X offers a more reliable foundation for assessing and advancing VLA development.

I. INTRODUCTION

Existing VLA benchmarks can give overly optimistic robustness assessments because they use limited perturbations, narrow train–test gaps, and insufficiently diverse data. LIBERO-X addresses these issues with hierarchical multi-dimensional evaluation and diverse teleoperated training data, revealing degradation as complexity increases.

  • Existing evaluations can produce overly optimistic or misleading conclusions because protocols and data distributions inadequately reflect real-world shifts.Source: §I · p. 1
  • Prior extensions model perturbation dimensions independently and lack a systematic easy-to-hard progression for measuring degradation under compound shifts.Source: §I · p. 1
  • LIBERO-X evaluates spatial generalization, object recognition, and instruction understanding through a progressively challenging multi-level test suite.Source: §I · pp. 1–2
  • Its training dataset uses multi-task scenes with diverse trajectories and variations in object attributes, spatial relations, and language instructions.Source: §I · p. 2
  • 2,520 demonstrations across 600 tasks and 100 scenes increase scene diversity and task granularity through human teleoperation.Source: §I · p. 2
  • Experiments reveal notable performance degradation as task and scene complexity increase, exposing limitations in scene comprehension and instruction grounding.Source: §I · p. 3

II. RELATED WORK

Prior robotic-manipulation benchmarks established reproducible evaluation but often test isolated shifts, use limited training diversity, or rely on off-the-shelf checkpoints. LIBERO-X combines diverse teleoperated training data with progressively harder, multi-dimensional evaluation.

  • Simulation benchmarks provide experimental convenience and reproducibility for assessing robotic manipulation generalization and robustness.Source: §II · p. 3
  • High nominal success rates can decline sharply under minor perturbations, suggesting reliance on memorized training data rather than generalized skills.Source: §II · p. 3
  • Existing evaluations often test isolated visual or positional shifts rather than coupled multi-factor shifts encountered in real environments.Source: §II · p. 3
  • Training datasets often lack variation in scenes, tasks, and trajectories, constraining the generalization capacity of resulting policies.Source: §II · p. 3
  • LIBERO-X integrates teleoperation-collected training data with evaluation that increases difficulty through spatial perturbations, object substitutions, and instruction rewrites.Source: §II · p. 3
  • The dataset contains 2,520 demonstration trajectories spanning 600 tasks and 100 scenes, with finer-grained task extensions and multiple manipulation objectives.Source: LIBERO-X · p. 4

1) Training Trajectory Collection:

LIBERO-X constructs its training data from human-operated demonstrations in diverse simulated scenes containing varied object types, tasks, and attribute-conditioned interactions.

  • Trajectories are collected through VR teleoperation in MuJoCo, recording 6D end-effector pose, gripper status, Cartesian increments, and gripper commands.Source: 1) Training Trajectory Collection · p. 5
  • The dataset comprises 100 scenarios and 600 distinct tasks spanning rigid and articulated objects, multi-object interactions, and attribute-conditioned manipulation.Source: 2) Example Training Demonstrations · p. 5
  • The test framework combines progressively difficult levels with fine-grained multi-label task annotations for quantitative analysis across perturbations and environments.Source: B. Evaluation Framework · p. 5

1) Multi-level Evaluation:

LIBERO-X uses a two-dimensional evaluation design: progressively harder scene perturbations form the vertical axis, while hierarchical multi-label annotations provide fine-grained task analysis.

  • Each evaluation level superimposes additional perturbations across spatial reasoning, object recognition, and instruction understanding to assess cumulative robustness.Source: 1) Multi-level Evaluation · p. 5
  • Levels 1 and 2 progress from minor positional changes to larger-scale spatial randomization.Source: 1) Multi-level Evaluation · p. 5
  • Level 3 reconstructs scene topology by altering object relationships, swapping targets with distractors, and adding irrelevant objects.Source: 1) Multi-level Evaluation · p. 5
  • Level 4 varies texture, color, and size while introducing unseen categories and visually similar confounders.Source: 1) Multi-level Evaluation · p. 5
  • Level 5 reformulates instructions through synonym replacement, compression, word-order changes, voice conversion, and redundant descriptions while preserving semantics.Source: 1) Multi-level Evaluation · pp. 5–6
  • Multi-label categories cover interaction type, subtask count, spatial relation, and object attributes for detailed capability analysis.Source: 2) Multi-label Evaluation · p. 6

IV. EXPERIMENTS

LIBERO-X evaluates five VLA models under progressively compounded spatial, object, and semantic changes, exposing substantial degradation where standard LIBERO results often saturate. The evaluation highlights weaknesses in spatial extrapolation, topology changes, novel-object grounding, and instruction variation.

  • Five representative VLA models were evaluated under the multi-level framework using their official configurations.Source: A. Experimental setup · p. 6
  • 39.4% average success at Level 1 and 65.2% for π0.5 contrast with often ∼90% performance reported on LIBERO.Source: B. Multi-level Evaluation · p. 7
  • 9.8% average success decline from Level 1 to Level 2 indicates difficulty with larger spatial shifts beyond training configurations.Source: B. Multi-level Evaluation · p. 7
  • 12.7% degradation at Level 3 is the largest decline, revealing limited structural invariance under topological rearrangements.Source: B. Multi-level Evaluation · p. 7
  • Diversified training preserves non-zero performance on out-of-distribution tasks, but unseen-object success remains below confounding-object success.Source: B. Multi-level Evaluation · p. 7
  • Instruction rephrasing causes a 3.26% Level 4-to-Level 5 drop, while voice conversion is the least disruptive variation.Source: B. Multi-level Evaluation · p. 7

C. Multi-label Evaluation

The multi-label evaluation reveals synchronized performance degradation across models as task complexity increases, with long-horizon tasks producing especially severe failures. Time-limit analysis shows that modest temporal slack can buffer execution imperfections, while additional slack eventually saturates.

APPENDIX

The appendix details LIBERO-X’s fine-grained, multi-label evaluation system and its supporting analyses. Labels cover interaction type, subtask count, spatial relations, and object attributes, enabling performance analysis across task dimensions.

  • The supplementary materials include detailed multi-label results, time-limit analysis, model details, scene design, visual-attribute cases, and semantic-reformulation cases.Source: APPENDIX · p. 11
  • The fine-grained system assigns labels across Interaction Type, Subtask Count, Spatial Relation Type, and Object Attribute Recognition.Source: A. Detailed Results of Multi-label… · p. 11

Interaction Type

Subtask Count

Spatial Relation Type

Object Attribute Recognition

B. Further Analysis of the Impact of Time Limits

Time-limit analysis finds that a modest temporal buffer improves performance across all evaluation levels, but gains diminish after the main buffer is added. Beyond 1.5× human reference time, performance plateaus because perception and reasoning remain limiting factors.

C. Model Details

The study fine-tunes diverse VLA architectures on LIBERO-X and evaluates them with fine-grained tests across progressively difficult levels and time limits.

  • The evaluation compares diverse VLA architectures initialized from official pre-trained checkpoints and fine-tuned on LIBERO-X.Source: C. Model Details · p. 12
  • OpenVLA-OFT combines parallel decoding, action chunking, continuous actions, and FiLM-based language grounding.Source: C. Model Details · p. 12
  • π0 generates continuous actions with flow matching, while π0.5 adds broader semantic generalization through internet-scale data and hierarchical subtask prediction.Source: C. Model Details · pp. 12–13
  • The benchmark reports fine-grained evaluation results and time-limit thresholds across Levels 1–5.Source: Table V · p. 13, Table VI · p. 13, Table VII · p. 13
  • π0 and π0.5 are fine-tuned on LIBERO-X using resized 224×224 wrist and third-person images, with distinct adaptation strategies and action-chunk configurations.Source: C. Model Details · p. 13
  • X-VLA uses learnable soft prompts, multimodal Transformer fusion, and flow matching for cross-embodiment adaptation.Source: C. Model Details · p. 14
  • GR00T N1.5 combines a frozen VLM, an alignment adapter, and a Diffusion Transformer, with FLARE aligning predicted future states to ground truth.Source: C. Model Details · p. 14

D. Scene Design Details

LIBERO-X expands scene assets and evaluation predicates to make containment, posture, and object variation more precise and diverse.

  • The scene framework expands the original LIBERO system by optimizing core features and enlarging the 3D asset library.Source: D. Scene Design Details · p. 15
  • ExactIn replaces imprecise In detection with a vertical-displacement threshold that confirms an object is fully inserted and resting at the container bottom.Source: D. Scene Design Details · p. 15
  • Upright and side-on predicates classify book posture using angles between the object up-vector and surface normal.Upright placement requires an angle below approximately 37°, whereas side-on placement is within 17° of perpendicular.Source: D. Scene Design Details · pp. 15–16
  • These posture predicates support more precise recognition and reliable execution in placement tasks requiring posture differentiation.Source: D. Scene Design Details · p. 16
  • Novel objects increase color, texture, and size diversity, while functional zones such as shelves and drawers provide additional scene and task variation.Source: D. Scene Design Details · p. 16
  • Training and test datasets track Predicate, Object, and Container frequencies across scenes and tasks.Source: D. Scene Design Details · p. 16, Table XI · p. 16

E. Visual Attribute Variation Cases Analysis

Level 4 tests visual attribute variation using confounding and unseen objects, revealing failures in target selection, instruction following, and multi-step execution.

F. Semantic-Equivalent Reformulation Cases

Level 5 evaluates semantic-equivalent task reformulations that preserve meaning while changing wording and sentence structure.

Related papers

Loading 2602.06556v1…