PaperlayerView exact revision on arXiv

Source-linked AI summary

LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models

Senyu Fei, Siyin Wang, Junhao Shi, Zihao Dai, Jikun Cai, Pengfang Qian, Li Ji, Xinzhe He, Shiduo Zhang, Zhaoye Fei, Jinlan Fu, Jingjing Gong, Xipeng Qiu

arXiv:2510.13626v3cs.ROcs.CLcs.CV

TL;DR

High success rates on controlled robotic manipulation benchmarks leave robustness to environmental and task variation an open concern. LIBERO-Plus systematically perturbs seven dimensions and finds broad brittleness, including vulnerability to camera and robot-state changes and weak use of language instructions.

  • Problem

    Evidence remains limited on how robust high-performing VLA models are to minor environmental and task variations beyond controlled benchmark conditions.

    Source: §1 · p. 1

  • Method

    Using LIBERO as a diagnostic, the authors systematically vary seven factors, from camera viewpoints and robot states to language, lighting, backgrounds, sensor noise, and object layout.

    Source: §1 · p. 2

  • Results

    Contemporary VLA models are brittle, particularly under camera and robot-state changes, while almost all models ignore language instructions.

    Source: §1 · p. 2, §8 · p. 10

  • Takeaways & Limitations

    The authors urge evaluation that prioritizes diverse embodied tasks and architectures capable of robust generalization beyond limited benchmark environments.

    Source: §8 · p. 10

    The authors frame weak language-perturbation effects as potentially reflecting either strong language generalization or reliance on limited instruction keywords.

    Source: §4 · p. 5

Abstract

from arXiv · show

Visual-Language-Action (VLA) models report impressive success rates on robotic manipulation benchmarks, yet these results may mask fundamental weaknesses in robustness. We perform a systematic vulnerability analysis by introducing controlled perturbations across seven dimensions: objects layout, camera viewpoints, robot initial states, language instructions, light conditions, background textures and sensor noise. We comprehensively analyzed multiple state-of-the-art models and revealed consistent brittleness beneath apparent competence. Our analysis exposes critical weaknesses: models exhibit extreme sensitivity to perturbation factors, including camera viewpoints and robot initial states, with performance dropping from 95% to below 30% under modest perturbations. Surprisingly, models are largely insensitive to language variations, with further experiments revealing that models tend to ignore language instructions completely. Our findings challenge the assumption that high benchmark scores equate to true competency and highlight the need for evaluation practices that assess reliability under realistic variation.

1 Introduction

High benchmark success in controlled settings can conceal brittle VLA behavior, while static aggregate evaluations miss reliability under realistic variation. The paper systematically perturbs seven factors in LIBERO to expose this gap and motivate robustness-focused evaluation.

Introduction

  • Static evaluations based on aggregate success rates under ideal conditions fail to measure policy stability under realistic variations, including viewpoint and robot-initial-state shifts.This limitation applies even when tasks remain within simulation and is especially relevant to models integrating multiple modalities.Source: §1 · p. 1
  • Nominal perturbations across task and environmental factors expose brittle VLA performance, suggesting models overfit narrowly defined training cues rather than achieve robust multimodal understanding.The analysis varies camera viewpoints, robot initial states, language instructions, lighting, background textures, sensor noise, and object layout.Source: §1 · p. 2
  • VLA models show visual-shift vulnerability, limited generalization across initial robot configurations, and superficial language interaction, with instruction variations having minimal impact.The paper reports reliance on fixed visual features, inadequate kinematic reasoning, and language inputs that may be underutilized or ignored.Source: §1 · p. 2
  • The work contributes a systematic vulnerability analysis, a diagnostic framework for quantifying perturbation impacts, and insights into the gap between apparent multimodal competence and robust understanding.These contributions use LIBERO as a diagnostic benchmark.Source: §1 · p. 2
  • The findings challenge treating high benchmark scores as proof of true competency and call for evaluation practices and models that prioritize robustness and reliability under inherent variability.The paper frames its work as a step toward VLA systems that are high-performing, reliable, and adaptable.Source: §1 · p. 2

2 How Do Single-Dimension Perturbations Affect VLA Models?

The section evaluates seven single-dimension perturbations across representative VLA models and finds brittle generalization, particularly under camera-viewpoint and robot-initial-state changes. Models’ small performance drop under language perturbations may reflect reduced reliance on instructions rather than superior linguistic generalization.

2.1 Perturbation Factors

  • The evaluation tests seven single-dimension perturbations: object layout, camera viewpoint, robot initial state, language, lighting, background texture, and sensor noise.Examples include shifting target-object position, changing the third-person camera pose or field of view, and injecting image blur.Source: §2.1 · p. 2

2.2 Models

  • Ten representative open-checkpoint models span autoregressive and diffusion architectures and varied training paradigms, including web-data co-training, world modeling, and reinforcement learning.The evaluated set includes OpenVLA and variants, π0, π0-fast, Nora, WorldVLA, UniVLA, and RIPT-VLA.Source: §2.2 · p. 2

2.3 Results

  • Current VLAs show brittle generalization under input perturbations, with camera-viewpoint and robot-initial-state changes especially damaging.The paper presents its main results in Table 1 and Figure 1; the supplied passages do not provide specific table values.Source: §2.3 · pp. 2–3, Table 1 · p. 3
  • Models are most vulnerable to camera-viewpoint and robot-initial-state changes, but relatively resilient to lighting and background variations.The former changes demand spatial-geometry and proprioceptive understanding, whereas the latter are described as more superficial, low-level visual changes.Source: §2.3 · p. 3
  • Language perturbations produce the second-smallest average performance drop (-25.3), but experiments suggest this reflects reduced reliance on instructions rather than superior linguistic generalization.Models may instead use task cues from visual context.Source: §2.3 · p. 3
  • Models with a first-person wrist camera generalize better, especially to viewpoint changes, while diversity-focused co-training strategies yield greater robustness across multiple perturbation types.The passage contrasts OpenVLA-OFT with the third-view-only OpenVLA-OFT_w and identifies π0 and π0-fast as examples of robust models.Source: §2.3 · p. 3

3 Do contemporary VLA Models truly pay attention to visual inputs?

VLA models tolerate distracting objects and moderate lighting changes, but target displacement exposes positional bias; the wrist camera helps explain lighting robustness by supplying stable geometric cues.

Do Models Genuinely Attend to Task-Relevant Objects?

  • Confounding objects cause only marginal declines, but target displacement sharply reduces success, indicating positional bias rather than invariant object-semantic understanding.π0, π0-Fast, RIPT-VLA, UniVLA, and WorldVLA show only marginal decreases with confounding objects, but performance drops significantly when target placement changes.Source: Figure 1 · p. 4, §3 · p. 4

How Do Models Maintain Performance Under Illumination Changes?

  • Several models lose only around 10 points under light perturbations, whereas third-person-only OpenVLA, Nora, and WorldVLA often lose over 60 points.The wrist camera’s close-range view provides stable geometric and contact cues, helping explain the limited lighting degradation; models without wrist-camera inputs are more vulnerable.Source: §3 · pp. 4–5
  • Masking both camera views collapses performance nearly to zero, while preserving the wrist view yields accuracies of 43.6, 43.0, and 67.3.The wrist view alone provides critical, stable close-range geometric and contact cues.Source: Figure 2 · p. 4, §3 · pp. 4–5

4 Do Contemporary VLA Models Truly Follow Language Instructions?

OpenVLA-OFT’s performance was barely affected by language perturbations, but blank-instruction and goal-replacement tests indicate that this apparent robustness often reflects language underuse, with long-horizon tasks an exception.

4 Do Contemporary VLA Models Truly Follow Language Instructions?

  • OpenVLA-OFT’s performance remained close to baseline under language perturbations, motivating hypotheses of strong language generalization, keyword matching, or reliance on nonlinguistic signals.The study tested these explanations with additional analyses.Source: §4 · p. 5
  • Commonsense keyword rewrites also caused nearly negligible performance drops, making limited keyword extraction an unlikely explanation for the observed robustness.The authors instead identify language underuse as the explanation supported by their further tests.Source: §4 · p. 5, §4.2 · p. 6

4.1 What If We Remove Language, Does Performance Drop?

  • With blank instructions across four LIBERO suites, OpenVLA-OFT’s object-suite performance remained largely unchanged, while significant degradation appeared only on the long suite.The authors attribute the long-suite drop to greater reliance on instruction guidance in long-horizon tasks.Source: §4.1 · p. 5

4.2 What If We Replace Goals with OOD Objects, Do Models Fail?

  • Replacing target objects in layout-suite instructions tested cross-object following, with alternatives drawn from objects in the same scene.For example, “pick up the alphabet soup” was changed to “pick up the tomato sauce.”Source: §4.2 · p. 6
  • In layout tasks with replaced target objects, success dropped nearly to zero, showing that VLA models lack strong cross-object instruction-following generalization.The degradation was particularly severe for OpenVLA-OFT.Source: §4.2 · p. 6
  • Rollouts showed that after the instructed target changed, the model still tended to execute the original target action, indicating reliance on fixed vision–action mappings over language signals.Further rollout details are reported in Appendix E.Source: §4.2 · p. 6

5 Does There Exist Compositional Generalization Gap Across Multi-Dimensional Perturbations?

The paper tests whether robustness to isolated perturbations generalizes to combinations, finding that perturbations interact and undermine performance under joint shifts.

5.1 Statistical Definition of the Compositional Generalization Gap

  • The compositionality gap is the covariance between two perturbation indicators among successful trials; positive, negative, and zero values indicate joint handling, added difficulty, and no interaction, respectively.The indicators record whether each perturbation is applied, and whether the task succeeds.Source: §5.1 · pp. 6–7

5.2 Experimental Setup and Results Analysis

  • The analysis uses OpenVLA-OFT across seven selected perturbation dimensions, with 2,000 repeated trials for single and pairwise perturbations, excluding language rewrites because language-following limitations may make their scores deceptive.The selected dimensions are objects spanning, environment sampling, illumination variations, camera-sphere shifts, robot initialization, and sensor noise.Source: §5.2 · p. 7
  • Consistent negative compositionality gaps show that perturbation combinations introduce interactions and pronounced robustness degradation, exposing entangled representations and a lack of mechanisms for higher-order dependencies.Figure 4 contrasts independence-based products of single-dimension probabilities with actual joint outcomes.Source: Figure 4 · p. 7, §5.2 · p. 7

6 LIBERO-Plus

LIBERO-Plus organizes tasks across seven perturbation factors and five difficulty levels to evaluate generalization. Training with an expanded dataset improves overall robustness, especially to camera-view changes.

6.1 Benchmark Construction

  • 10,030 tasks span seven perturbation factors and twenty-one low-level components, organized into five difficulty levels based on accuracy across four representative models.Construction expands and enriches LIBERO, then filters and balances task categories; Figure 5 reports model accuracy by level under four representative perturbation factors.Source: §6.1 · p. 8, Figure 6 · p. 8

6.2 Does Training on Generalized Sets Improve Generalization?

  • Over 20,000 successful trajectories form an expanded training dataset with increased trajectory count and scene diversity, used for mixed fine-tuning from official OpenVLA-OFT weights.The fine-tuned model is evaluated on the LIBERO-Plus benchmark.Source: §6.2 · p. 8
  • 79.6% overall success: our generalized-data post-training method outperforms all baselines across nearly all perturbations, with 92.8% camera-view robustness—37.2 percentage points above the next best model.It also achieves 89.3% under noise and 77.6% under layout perturbations.Source: §6.2 · pp. 8–9, Table 2 · p. 9

7 Related Work

The section traces VLA research from several policy paradigms to a gap in robustness evaluation: existing benchmarks lack broad, fine-grained assessment, while LIBERO-Plus combines seven dimensions, automated task generation, and difficulty levels.

7.1 Vision-Language-Action Models

  • VLA research spans autoregressive token policies, diffusion trajectory generation, and reinforcement-learning methods, but familiar-setting “zero-shot” success may reflect interpolation rather than true generalization.The section notes that existing benchmarks lack comprehensive insight into performance under distribution shifts.Source: §7.1 · p. 9

7.2 Generalization Robotic Manipulation Evaluations

  • Earlier manipulation benchmarks had narrow task scope; manual perturbation studies often tested fewer than 100 scenarios, while automated benchmarks broadened coverage but lacked fine-grained analysis.LIBERO-Plus addresses these gaps with evaluation across seven robustness dimensions and 21 sub-dimensions, automated task generation, and difficulty levels L1–L5.Source: §7.2 · pp. 9–10

8 Conclusion … D.3 Trajectory Collection

The paper concludes that high benchmark performance can mask VLA generalization failures and motivates evaluation across diverse perturbations and architectures. Its appendices describe perturbation dimensions and variants, evaluated models, benchmark-data and trajectory collection, and model success rates across perturbation difficulty levels.

8 Conclusion

  • VLA models remain brittle under camera and robot-state changes; nearly all ignore language, some memorize trajectories, and combined perturbations reveal positional bias and negative generalization gaps.The authors urge evaluation across diverse embodied tasks and development of architectures that generalize beyond limited benchmark environments.Source: §8 · p. 10

A Perturbation Dimensions

  • The evaluation defines seven perturbation dimensions—object layout, background, lighting, camera views, robot initialization, language rewrites, and image noise—with quantifiable sub-dimensions.The dimensions target robustness and generalization across vision, state, and language inputs.Source: A Perturbation Dimensions · p. 13

A.1 Objects Layout; A.2 Background Textures

  • Object-layout tests add unseen distractors or perturb target poses while preserving semantic relations; background tests alter scene themes or work-surface textures.Distractors are drawn from 416 object categories, while scene-theme textures come from a curated collection of 950 textures.Source: §A.1 · p. 13, §A.2 · p. 13

A.3 Light Conditions; A.4 Camera Viewpoints; A.5 Robot Initial States

  • Lighting tests vary diffuse color, parallel-light direction, specular-highlight intensity, and shadow casting; camera tests move along its optical axis, while robot-state tests perturb initial joint positions (qpos).Camera distance ranges from 1.01× to 2.00× the original value; joint perturbations range from 0.1 to 0.5.Source: §A.3 · p. 13, §A.4 · p. 14, §A.5 · p. 14

B.1 Model Overview; B.5 WorldVLA (Cen et al., 2025); B.6 UniVLA (Li et al., 2025)

  • The evaluated models span varied VLA designs; WorldVLA jointly learns action generation and future-state prediction, while UniVLA predicts task-centric latent actions for robot-specific decoding.WorldVLA uses a customized action-generation attention mask to limit error propagation; UniVLA uses a discrete latent-action space for cross-embodiment generalization.Source: §B.1 · p. 15, §B.5 · p. 17, §B.6 · p. 17

B.2 OpenVLA and OpenVLA-OFTs

  • OpenVLA-OFT uses parallel action decoding, continuous action representations, and FiLM for language grounding; π0-fast compresses action sequences using DCT followed by BPE.OpenVLA-OFT_w removes the wrist camera, while OpenVLA-OFT_m trains jointly across four LIBERO suites.Source: §B.2 · p. 16

C.1 Data Generation and Filtering

  • The benchmark began with 14,000 candidate tasks from 40 LIBERO tasks, then removed tasks solved by all or most models and balanced sub-dimensions, yielding 10,030 test tasks.Candidates covered four generalization sub-tasks—Spatial, Object, Goal, and Long—and seven generalization dimensions.Source: §C.1 · p. 18

D.1 Dataset Overview; D.2 Data Generation Process

  • The generalized training dataset contains over 20,000 successful trajectories; its variants include object, environment, lighting, camera, language, and sensor changes, generated with parameters adjusted for diverse scenarios.Compounding objects and their poses were varied without test-set overlap; additional table, wall, and floor textures were sampled to avoid overlap with test environments.Source: §D.1 · p. 19, §D.2 · p. 19

D.2 Data Generation Process

  • Data variants changed lighting parameters, shifted camera angles by 5°, generated language rewrites, and used image-noise parameters different from Table 5.Source: §D.2 · pp. 19–20

D.3 Trajectory Collection

  • 2,400 trajectories were collected for the compounding-object variant and 4,000 for each other variant, totaling 22,400, with over 20,000 retained after filtering.Original LIBERO (state, action) pairs were executed in the new environments; only successful trajectories were kept and no-op actions removed.Source: §D.3 · p. 20
  • Figure 8 plots each model’s success rate as perturbation intensity increases across all seven dimensions.Source: Figure 8 · p. 20

D.4 Training Configuration · D.5 Storage Format

The appendix specifies the OpenVLA-OFT mixed-fine-tuning setup and states that trajectories are stored in RLDS format.

D.4 Training Configuration

  • Mixed fine-tuning from official OpenVLA-OFT weights used 8 × A100 GPUs for 100,000 steps.The learning rate was 5 × 10−4, with batch size 2 per GPU (effective batch size 16), AdamW with weight decay 0.1, and cosine scheduling with warmup.Source: §D.4 · p. 20

D.5 Storage Format

  • Trajectories are stored in RLDS format for compatibility with existing training pipelines.The paper describes RLDS as consistent with standard robotics-dataset practices.Source: §D.5 · p. 20

E Goal Replacement Rollout Cases Analysis · F Details of the Compositional Generalization Experiments · F.1 Success Rate Record

Replacing the instructed target with another object from the same scene drove success rates nearly to zero, and rollout cases showed models repeating actions for the original targets. Table 8 reports pairwise success rates across isolated and joint perturbations, enabling analysis of their interaction effects.

E Goal Replacement Rollout Cases Analysis

  • The evaluation replaced the target object named in both instruction and task goal with another object from the same scene while leaving the environment otherwise unchanged.It was conducted on the object suite, where action–instruction misalignment was most pronounced.Source: E Goal Replacement Rollout Cases… · pp. 20–21
  • Success rates on replaced-target tasks fell nearly to zero across tested instances, with OpenVLA-OFT dropping from high baseline accuracy to almost complete failure.The authors interpret this as evidence that prior robustness to language perturbations did not reflect genuine linguistic comprehension; models appear to rely on fixed perception–action associations.Source: E Goal Replacement Rollout Cases… · p. 21
  • In nearly all rollout cases, models performed the original target action despite an explicitly changed instruction, consistent with shallow language processing and memorized visual–action associations.Figure 10 presents original-to-new instruction pairs alongside the behavior actually executed.Source: E Goal Replacement Rollout Cases… · p. 21, Figure 10 · p. 22
  • Examples show models choosing the old target: one picked up alphabet soup instead of instructed butter, while another performed the butter action when instructed to pick up tomato sauce.Source: E Goal Replacement Rollout Cases… · p. 21
  • The authors characterize the models as visual pattern matchers mapping scene configurations to predetermined action sequences rather than integrating task-relevant instructions.Source: E Goal Replacement Rollout Cases… · p. 21

F.1 Success Rate Record

  • Table 8 reports pairwise success rates: diagonal entries show single perturbation dimensions, while off-diagonal entries show joint perturbations.The analysis is intended to examine both isolated robustness and interaction effects relevant to compositional generalization.Source: Table 8 · p. 23, §F.1 · p. 23

F.2 Significance Experiments for Compositional Generalization · G Failure Cases Study

Most perturbation pairs show significant interaction effects rather than independent outcomes. The failure-case study examines how perturbations produce distinct errors in localization, task understanding, and action execution.

F.2 Significance Experiments for Compositional Generalization

  • The significance experiments use a chi-square test for independence to distinguish systematic interaction effects from finite-trial sampling noise.The test assesses whether negative compositional gaps between expected product-based and actual joint success rates reflect interactions.Source: §F.2 · p. 23
  • For each perturbation pair, the test uses a 2 × 2 contingency table of success counts under neither, either, or both perturbations.The counts are n00, n01, n10, and n11, corresponding to success under neither, perturbation 2, perturbation 1, and both, respectively.Source: §F.2 · p. 23
  • For a 2 × 2 table, the test has 1 degree of freedom, and a p-value below 0.05 indicates evidence against independence.The p-value is the probability under the independence null hypothesis of observing a test statistic at least as extreme as χ2.Source: §F.2 · p. 23
  • A large χ2 with a small p-value signals that the joint success/failure distribution deviates from independence; a small χ2 offers no such evidence.The paper interprets significant deviations as interaction effects between perturbations.Source: §F.2 · p. 24
  • Most perturbation pairs have large χ2 values and p-values below 0.05, indicating significant interactions that challenge independence.The paper concludes that perturbation interactions cannot be ignored when evaluating compositional generalization.Source: Table 9 · p. 24, §F.2 · p. 24

G Failure Cases Study

  • The failure-case study qualitatively analyzes characteristic errors and reports distinct failure modes in object localization, task understanding, and action execution across perturbation types.It provides explanatory context for the quantitative results and points to Figures 17–19 for typical cases.Source: G Failure Cases Study · p. 24
  • Background-texture rendering figures compare the original image with transformed versions labeled by rotation and distance parameters.The captions define hr and vr as horizontal and vertical rotation angles, dis as distance pulled away, and chr and cvr as in-place horizontal and vertical rotation angles.Source: Figure 11 · p. 25, Figure 12 · p. 25

H Detailed results of LIBERO-Plus

This section reports LIBERO-Plus generalization results across seven perturbation categories and four task suites, finding that methods show different robustness patterns across perturbations.

H Detailed results of LIBERO-Plus

  • The comparison reveals differences in robustness patterns across methods and perturbation types.These results provide insights into model generalization capabilities.Source: H Detailed results of LIBERO-Plus · p. 26
  • Figure 17 depicts distinct failure patterns, including viewpoint localization errors, shadow artifacts, feature corruption, trajectory-planning effects, and recognition confusion.Figure 19 further describes camera-shift localization errors and object-distractor confusion that can lead to collision-prone trajectories when arm-motion flexibility is insufficient.Source: Figure 17 · p. 28, Figure 19 · p. 30
  • Table 10 reports VLA-method success rates across seven perturbation categories, their average, and four task suites.The categories are Camera, Robot Initialization, Language Instruction, Lighting, Background, Sensor Noise, and Scene Layout; the suites are Spatial, Object, Goal, and Long.Source: H Detailed results of LIBERO-Plus · p. 25, Table 10 · p. 31

Related papers

Loading 2510.13626v3…