Source-linked AI summary
LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models
Senyu Fei, Siyin Wang, Junhao Shi, Zihao Dai, Jikun Cai, Pengfang Qian, Li Ji, Xinzhe He, Shiduo Zhang, Zhaoye Fei, Jinlan Fu, Jingjing Gong, Xipeng Qiu
TL;DR
VLA benchmark success can conceal fragile robustness under modest environmental and task variations. This study systematically perturbs multiple factors across representative VLA models and finds brittleness, including minimal sensitivity to language changes that may reflect underuse of language instructions.
Problem
VLA models achieve near-perfect success rates in controlled benchmark environments, but their robustness under environmental and task variations remains limited.
Method
The study systematically varies perturbation factors on the LIBERO benchmark and analyzes representative VLA models spanning diverse architectures and training paradigms.
Results
Models show vulnerability to visual and robot-state changes, while language perturbations produce the second smallest average performance drop (-25.3) and are often ignored.
Takeaways & Limitations
Evaluation should prioritize diverse embodied tasks and architectures that generalize robustly beyond limited benchmark environments.
Takeaways & Limitations
The analysis considers that models may not fully utilize language, instead relying primarily on visual or other non-language cues.
Abstract
from arXiv · showhide
Visual-Language-Action (VLA) models report impressive success rates on robotic manipulation benchmarks, yet these results may mask fundamental weaknesses in robustness. We perform a systematic vulnerability analysis by introducing controlled perturbations across seven dimensions: objects layout, camera viewpoints, robot initial states, language instructions, light conditions, background textures and sensor noise. We comprehensively analyzed multiple state-of-the-art models and revealed consistent brittleness beneath apparent competence. Our analysis exposes critical weaknesses: models exhibit extreme sensitivity to perturbation factors, including camera viewpoints and robot initial states, with performance dropping from 95% to below 30% under modest perturbations. Surprisingly, models are largely insensitive to language variations, with further experiments revealing that models tend to ignore language instructions completely. Our findings challenge the assumption that high benchmark scores equate to true competency and highlight the need for evaluation practices that assess reliability under realistic variation.
1 Introduction
VLA models achieve near-perfect benchmark performance in controlled settings, but aggregate success rates can conceal fragile robustness under realistic variation. LIBERO-Plus systematically probes these weaknesses and identifies a mismatch between apparent multimodal competence and robust understanding.
- Motivation: Near-perfect benchmark success rates can conceal critical deficiencies when environmental conditions or task parameters vary.Standardized evaluations often use static, ideal conditions that do not capture policy stability under realistic variation.
- Approach: LIBERO-Plus systematically varies object layout, camera viewpoints, robot initial states, language instructions, lighting, background textures, and sensor noise.The study uses LIBERO as a diagnostic benchmark for controlled vulnerability analysis.
- Findings: The analysis exposes vulnerabilities in visual shifts, kinematic generalization, and language use across contemporary VLA models.The reported weaknesses include sensitivity to camera angle and illumination, limited generalization across robot configurations, and underutilized language inputs.
- Contributions: The work contributes a systematic vulnerability analysis, a diagnostic framework for quantifying perturbation impacts, and insights into the gap between apparent competence and robust understanding.These contributions target evaluation and diagnosis rather than benchmark success alone.
- Implications: The findings challenge equating high benchmark scores with true competency and support evaluating VLA reliability under inherent variability.The paper frames robustness and adaptability as necessary complements to high performance.
2 How Do Single-Dimension Perturbations Affect VLA Models?
Across seven single-dimension perturbations, contemporary VLAs show brittle generalization, with camera viewpoints and robot initial states producing the greatest vulnerability. Robustness also varies with architecture and training paradigm, while small language-related drops may reflect underuse of language rather than strong linguistic generalization.
- Evaluation Setup: Seven perturbation dimensions are applied to evaluation episodes, including object layout, camera viewpoints, robot initial states, language, lighting, backgrounds, and sensor noise.The study evaluates controlled changes to both visual and embodied task conditions.
- Models: The evaluation covers representative open-checkpoint VLAs spanning autoregressive and diffusion architectures and diverse training paradigms.Models include OpenVLA variants, π0 variants, Nora, WorldVLA, UniVLA, and RIPT-VLA.
- Overall Results: Even minor perturbations can cause drastic performance degradation, revealing significant fragility in current VLA generalization.Table 1 reports success rates and corresponding absolute drops for each model and perturbation dimension.
- Robustness Patterns: Camera viewpoint and robot initial-state changes are the strongest vulnerabilities, whereas lighting and background variations are comparatively less disruptive.The more damaging changes require spatial-geometric or proprioceptive generalization, unlike superficial visual shifts.
- Language Perturbations: -25.3 average performance drop makes language perturbation the second-smallest drop across most models, likely because models rely less on language than expected.The paper later tests whether this apparent robustness reflects language generalization or language underuse.
- Architecture and Training: Wrist-camera inputs and diverse co-training are associated with stronger robustness across perturbation types.OpenVLA-OFT is more robust to viewpoint changes than its third-person-only variant, while π0 and π0-fast generalize more consistently.
3 Do contemporary VLA Models truly pay attention to visual inputs?
Visual-input analyses reveal that VLAs can tolerate distractors and some illumination changes while failing when task-relevant spatial arrangements change. This pattern indicates positional bias and shows that wrist-camera inputs provide important illumination robustness.
- Visual Robustness: Models show surprising resilience to background changes and limited sensitivity to some light variations, motivating finer-grained visual analyses.These patterns raise whether models learn task-relevant semantics or rely on superficial visual cues.
- Object Layout: Object-layout testing separates confounding-object additions from changes to target-object placement and pose.This decomposition distinguishes distraction resistance from sensitivity to task-relevant spatial changes.
- Illumination: Light perturbations cause around 10-point drops for several models, but third-person-only models often lose more than 60 points under illumination changes.The extreme ablations compare light perturbations with masking the third-person view or all camera inputs.
- Object Layout: Target displacement causes significant performance drops, suggesting reliance on memorized positional information rather than high-level task semantics.The models can ignore distracting objects but fail to generalize when the interactive target moves.
- Illumination: Wrist cameras provide critical illumination robustness by supplying close-range, illumination-invariant geometric cues.Models without wrist-camera inputs are substantially more vulnerable to lighting variation.
4 Do Contemporary VLA Models Truly Follow Language Instructions?
Language perturbations initially leave performance nearly unchanged, but removing or replacing instructions reveals that many VLAs underuse language and follow fixed vision–action mappings. The effect is weaker on some shorter tasks and more pronounced when targets or horizons require instruction guidance.
- Hypotheses: OpenVLA-OFT remains close to baseline under language perturbations, prompting hypotheses about linguistic generalization, keyword matching, or language underuse.The analysis tests whether stable performance reflects genuine instruction understanding.
- Hypotheses: Commonsense keyword rewriting produces nearly negligible performance drops, making simple keyword matching an unlikely explanation for the observed robustness.The paper treats this as evidence against relying only on limited instruction keywords.
- 4.1 What If We Remove Language, Does Performance Drop?: The blank-instruction experiment replaces language input with an empty value and evaluates all four LIBERO suites.This directly tests whether removing linguistic information substantially degrades performance.
- 4.1 What If We Remove Language, Does Performance Drop?: Object-suite performance remains largely unchanged without valid language input, while significant degradation appears only on the long suite.The paper attributes the long-suite difference to greater reliance on instruction guidance in long-horizon tasks.
- 4.2 What If We Replace Goals with OOD Objects, Do Models Fail?: Goal replacement changes both the instructed target object and task goal to alternatives within the same scene.The layout-suite experiment directly tests cross-object instruction following.
- 4.2 What If We Replace Goals with OOD Objects, Do Models Fail?: Replaced-target success rates drop nearly to zero, especially for OpenVLA-OFT, showing weak cross-object instruction-following generalization.The result indicates that prior language-perturbation robustness did not reflect deep language modeling.
- 4.2 What If We Replace Goals with OOD Objects, Do Models Fail?: Rollouts show models often execute the original target action even after the instruction target is explicitly changed.This supports reliance on fixed vision–action mappings rather than fully exploiting language signals.
5 Does There Exist Compositional Generalization Gap Across Multi-Dimensional Perturbations?
The study defines compositional generalization as performance under potentially dependent perturbations and measures whether joint effects depart from independent single-dimension effects. Across repeated pairwise experiments, negative compositionality gaps reveal degraded robustness under combined perturbations.
- Statistical Definition: Compositional generalization concerns performance under perturbation dimensions that may exhibit complex dependencies rather than acting independently.
- Statistical Definition: The perturbation indicator D_i equals 1 when the i-th perturbation is applied and 0 otherwise, while Y indicates successful task execution.
- Statistical Definition: The conditional success rate s(D_i = d_i, D_j = d_j) is defined as P(Y = 1 | D_i = d_i, D_j = d_j).
- Statistical Definition: Conditional probabilities measure how frequently individual or joint perturbations occur among successful cases, with lower values indicating greater sensitivity.
- Experimental Setup and Results Analysis: The compositionality gap compares actual joint success probability with the product of the corresponding marginal probabilities.
- Experimental Setup and Results Analysis: The analysis uses independent single- and pairwise-perturbation tests over 2000 repeated trials and visualizes expected versus actual joint outcomes in a heatmap.
- Results: Negative compositionality gaps consistently indicate interaction effects and pronounced robustness degradation under complex perturbation combinations.
6 LIBERO-Plus
LIBERO-Plus expands LIBERO into a large benchmark covering seven perturbation factors and multiple difficulty levels. Post-training on an expanded trajectory dataset produces the strongest reported robustness, especially for camera-view changes, noise, and layout perturbations.
- Benchmark Construction: LIBERO-Plus is constructed by expanding LIBERO with seven perturbation factors, filtering and balancing task categories, and stratifying tasks into five difficulty levels.
- Benchmark Construction: 10,030 tasks span seven perturbation factors and twenty-one low-level components.
- Training and Evaluation: The authors construct a training dataset containing over 20,000 successful trajectories to evaluate generalization-oriented training.
- Training and Evaluation: Mixed fine-tuning starts from the official OpenVLA-OFT weights and evaluates the resulting model on LIBERO-Plus.
- Training and Evaluation: 79.6% overall success rate is achieved by the post-training method, outperforming baseline models across nearly all perturbation types.
- Training and Evaluation: 92.8% camera-view robustness exceeds the next-best model by 37.2 percentage points, while noise and layout robustness reach 89.3% and 77.6%.
7 Related Work
Prior VLA benchmarks established reproducible evaluation settings but generally covered limited task and perturbation scopes. Later automated approaches increased task counts, while existing manually designed evaluations often remained small.
- VLA Models: VLA research includes autoregressive and diffusion-based approaches that extend foundation-model paradigms into robotic action generation.
- Robustness Benchmarks: Early robotic manipulation benchmarks created reproducible testing environments but focused on specific tasks with limited robustness coverage.
- Robustness Benchmarks: Manually designed perturbation benchmarks typically used fewer than 100 test scenarios, limiting large-scale systematic robustness evaluation.
- Robustness Benchmarks: Automated task-generation frameworks increased the number of evaluation tasks and addressed part of the scale limitation.
8 Conclusion
The study concludes that contemporary VLA models remain brittle despite strong benchmark performance, with weaknesses spanning visual robustness, kinematic generalization, language use, and compositional reliability. It evaluates these weaknesses through systematic perturbations and diverse model architectures and training strategies.
- The analysis exposes a significant generalization problem beneath nearly saturated LIBERO benchmark performance.
- The evaluation varies object layout, background textures, lighting, camera viewpoints, robot initial states, language instructions, and sensor noise.The perturbations include confounding objects, target-object poses, scene and surface textures, four lighting factors, three camera factors, joint angles, rewritten instructions, and five noise types.
- The study analyzes representative VLA models spanning autoregressive and diffusion-based architectures and diverse training paradigms.The evaluated models include OpenVLA variants, π0 variants, NORA, WorldVLA, UniVLA, and RIPT-VLA.
- WorldVLA unifies policy action generation and future-state prediction within an autoregressive action-world model.It discretizes text, images, and actions into tokens and uses a customized attention mask for action generation.
- RIPT-VLA adds reinforcement interactive post-training using dynamic sampling, RLOO advantages, rejection filtering, and PPO optimization.The procedure targets failure modes encountered through environment interaction while constraining policy updates with probability ratios.
D.4 Training Configuration
The mixed fine-tuning configuration uses official OpenVLA-OFT weights and a fixed multi-GPU optimization schedule, with results reported on LIBERO-plus.
- Mixed fine-tuning uses official OpenVLA-OFT weights on 8 × A100 GPUs for 100,000 steps.
- The training uses a 5 × 10−4 learning rate, effective batch size 16, AdamW with weight decay 0.1, and cosine decay with warmup.
- All trajectories are stored in RLDS format for compatibility with existing robotics training pipelines.
E Goal Replacement Rollout Cases Analysis
Goal replacement tests whether VLA models follow changed target instructions rather than replaying actions associated with the original scene. Models generally continue executing the original target behavior, revealing shallow language use and fixed vision–action mappings.
- Goal replacement substitutes the instructed target object with an alternative from the same scene while leaving the rest of the environment unchanged.The manipulation is applied to the object suite, where instruction-action misalignment is most pronounced.
- Success rates in replaced-target tasks drop nearly to zero across tested instances.OpenVLA-OFT falls from high baseline accuracy to almost complete failure in the modified-target setting.
- Models often execute the original target action even after the instruction explicitly changes the target.Rollout cases show persistence with original targets such as alphabet soup, butter, tomato sauce, chocolate pudding, and cream cheese.
- The rollout behavior indicates reliance on fixed vision–action mappings rather than dynamic instruction-based planning.The models function more like visual pattern matchers that map scene configurations to predetermined action sequences.
- Pairwise evaluation compares isolated perturbations on the diagonal with joint perturbations off the diagonal.This design examines interaction effects relevant to compositional generalization.
F.2 Significance Experiments for Compositional Generalization
The study uses chi-square tests of independence to determine whether joint perturbation outcomes depart from product-based expectations. Most perturbation pairs show significant interaction effects, indicating that perturbation combinations matter for compositional generalization.
- The significance analysis tests whether deviations between expected product-based and observed joint success rates reflect systematic interaction effects rather than sampling noise.
- The test uses a 2 × 2 contingency table of success counts under two perturbations, with n00, n01, n10, and n11 representing the four outcome combinations.
- The expected count is computed from the corresponding row and column totals under the independence hypothesis.
- The p-value gives the probability of observing a chi-square statistic at least as extreme under independence, using dof = 1.A p-value below 0.05 indicates strong evidence against independence.
- Most perturbation pairs yield large χ2 values with tiny p-values, indicating significant departures from independence.The results show that perturbation interactions cannot be ignored when evaluating compositional generalization.
H Detailed results of LIBERO-Plus
LIBERO-Plus reports success rates across seven perturbation categories and task suites, revealing method-dependent robustness patterns and characteristic failures in localization and recognition.
- Generalization performance: Success rates are reported across seven perturbation categories and four task suites for multiple VLA methods.The categories are Camera, Robot Initialization, Language Instruction, Lighting, Background, Sensor Noise, and Scene Layout; the suites are Spatial, Object, Goal, and Long.
- Failure patterns: Camera-position changes cause inaccurate target-object localization.
- Failure patterns: Added distractor objects cause target-plate mislocalization when a nearby object is mistaken for the plate.
- Failure patterns: Failure visualizations associate camera shifts with localization errors, language modifications with semantic misinterpretations, and object distractors with recognition confusion.
- Failure patterns: Modified language descriptions can cause the model to localize cookies instead of the instructed dark-colored dish.
- Failure patterns: Perturbing object position causes the model to fail at correctly localizing the object.