PaperlayerView exact revision on arXiv

Source-linked AI summary

LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models

Chanyoung Kim, Minwoo Kim, Minseok Kang, Hyunwoo Kim, Dahuin Jung

arXiv:2603.28301v1cs.LG

TL;DR

VLA models are often fine-tuned with limited environment-specific data, leaving their robustness to paraphrased instructions underexplored. The paper introduces LIBERO-Para and PRIDE to evaluate controlled action and object variation and quantify paraphrase difficulty. Across configurations, paraphrases expose persistent, primarily object-level fragility and frequent planning-level trajectory divergence.

  • Problem

    Limited-data fine-tuning may overfit VLA models to specific instruction formulations, while existing evaluations insufficiently test robustness to paraphrased instructions.

  • Method

    LIBERO-Para independently varies action expressions and object references, while PRIDE combines lexical and syntactic similarity with task success for fine-grained robustness analysis.

  • Results

    Paraphrase fragility persists across architectures, scales, and fine-tuning strategies, with object-level lexical variation dominant and 80–96% of failures arising from planning-level trajectory divergence.

  • Takeaways & Limitations

    Improving paraphrase robustness requires prioritizing instruction-to-task identification over low-level control refinement, with object grounding as a key direction.

  • Takeaways & Limitations

    The study evaluates simulated LIBERO environments and isolates one paraphrase variation type per axis, leaving physical-robot validation and compound variations for future work.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models achieve strong performance in robotic manipulation by leveraging pre-trained vision-language backbones. However, in downstream robotic settings, they are typically fine-tuned with limited data, leading to overfitting to specific instruction formulations and leaving robustness to paraphrased instructions underexplored. To study this gap, we introduce LIBERO-Para, a controlled benchmark that independently varies action expressions and object references for fine-grained analysis of linguistic generalization. Across seven VLA configurations (0.6B-7.5B), we observe consistent performance degradation of 22-52 pp under paraphrasing. This degradation is primarily driven by object-level lexical variation: even simple synonym substitutions cause large drops, indicating reliance on surface-level matching rather than semantic grounding. Moreover, 80-96% of failures arise from planning-level trajectory divergence rather than execution errors, showing that paraphrasing disrupts task identification. Binary success rate treats all paraphrases equally, obscuring whether models perform consistently across difficulty levels or rely on easier cases. To address this, we propose PRIDE, a metric that quantifies paraphrase difficulty using semantic and syntactic factors. Our benchmark and corresponding code are available at: https://github.com/cau-hai-lab/LIBERO-Para

1 Introduction

VLA models are vulnerable to paraphrased instructions because data-scarce fine-tuning can overfit to seen formulations, while existing benchmarks provide limited linguistic robustness evaluation. LIBERO-Para addresses this gap with controlled analysis of action and object variation, revealing persistent and object-driven fragility.

  • Motivation: Data-scarce fine-tuning can make VLA models perform well on seen instruction phrasings but fail on paraphrased variants.This creates a deployment concern because environment-specific demonstration data is costly and limited.
  • Motivation: Existing LIBERO evaluations use identical instructions for training and evaluation, leaving linguistic robustness largely unexamined.LIBERO primarily tests visual generalization to new object configurations or scene layouts.
  • LIBERO-Para: LIBERO-Para independently varies action expressions and object references to enable interpretable paraphrase-robustness analysis.The benchmark targets meaning-preserving linguistic variation rather than conflating paraphrases with broader multimodal or task-semantic changes.
  • Key findings: Paraphrase fragility persists across architectures, model scales, and fine-tuning strategies.The result indicates that robustness degradation is not confined to a particular VLA configuration.
  • Key findings: Object-level lexical variation is the dominant source of degradation, indicating reliance on surface-level matching rather than semantic grounding.The benchmark isolates object and action expressions for fine-grained diagnosis.
  • Key findings: 80–96% of failures arise from planning-level trajectory divergence rather than action-execution errors.These failures suggest that paraphrasing disrupts task identification.

2 Related Work

VLA research spans diverse architectures and benchmarks, but existing evaluations provide limited control over paraphrase variation. LIBERO-Para isolates linguistic factors through action–object variation axes and evaluates instruction-conditioned task identification.

  • Vision-Language-Action Models: VLA architectures include autoregressive action-token decoders, action-chunking models, VLMs with flow-matching experts, bridge-based adapters, and soft-prompted cross-embodiment designs.These families range from 0.6–0.9B to multi-billion-parameter models and all require environment-specific fine-tuning with limited demonstrations.
  • LIBERO-Para: LIBERO-Para independently varies action expressions and object references, the two core linguistic components of robotic manipulation instructions.It keeps other factors fixed and holds paraphrases out for evaluation.
  • Benchmarks: Existing benchmarks assess rephrased instructions or broader multimodal perturbations but offer limited linguistic categorization and paraphrase control.Paraphrasing is often treated as an unstructured or secondary evaluation axis.

3 LIBERO-Para: A Controlled VLA Benchmark for Paraphrase Robustness

LIBERO-Para is a controlled benchmark that independently varies action expressions and object references while holding other task factors fixed, enabling fine-grained analysis of paraphrase robustness.

  • Benchmark Design: The benchmark separates action expressions and object references to analyze their distinct effects on VLA performance.This two-axis design also supports analysis of whether action and object variations have independent or interaction effects.
  • Benchmark Design: LIBERO-Para evaluates paraphrase robustness by changing only instructions while holding other factors fixed and reserving paraphrases for evaluation.The benchmark is built on LIBERO-Goal, where the instruction is the sole cue for task identification.
  • Action Variation: Action variation includes lexical, structural, and pragmatic paraphrases grounded in established paraphrase taxonomies.Examples include synonym substitution, adverb insertion, coordination, subordination, and indirect speech acts.
  • Object Variation: Object variation focuses on lexical changes to noun-phrase references, including addition and contextual or habitual same-polarity substitution.For example, “the stove” may become “the cooktop.”
  • Benchmark Scale: The benchmark contains approximately 100 samples per variation type and 4,092 paraphrased instructions.Additional taxonomy and generation details, including excluded variation types, are provided in Appendix A.

4 PRIDE: Paraphrase Robustness Index in Robotic Instructional DEviation

PRIDE evaluates paraphrase robustness by combining task-critical keyword preservation with syntactic similarity, then relating deviation to task success. It complements binary success rates by distinguishing paraphrases according to their semantic and structural deviation.

  • Motivation: Binary success rate does not distinguish easy from difficult paraphrases, obscuring whether performance relies on simpler instruction variants.
  • PRIDE Score: PRIDE quantifies linguistic deviation between original and paraphrased robotic instructions using keyword and structural variation.
  • 4.1 Keyword Similarity SK: Keyword similarity focuses on task-critical action and object content words rather than function words or whole-sentence form.Because robotic instructions often follow an “[ACT] the [OBJ]” structure, a small set of content tokens can determine intended behavior.
  • 4.1 Keyword Similarity SK: SK matches each original content word to its most similar paraphrase word using Sentence-BERT embeddings and cosine similarity, then averages the matches.Higher SK indicates better preservation of the original instruction’s key content words.
  • 4.2 Structural Similarity ST: ST measures syntactic change with dependency-tree edit distance using POS and dependency-relation labels rather than surface words.This reduces sensitivity to lexical substitutions while capturing structural differences such as reordering and relation changes.
  • 4.2 Structural Similarity ST: ST normalizes tree edit distance by the combined tree size, with lower values indicating greater structural divergence.The metric uses the node count of the original and paraphrased trees in its normalization.
  • 4.3 PRIDE Score: Paraphrase Distance combines SK and ST with α controlling their relative contribution, where higher PD indicates greater semantic and structural deviation.The default setting is α = 0.5.
  • 4.3 PRIDE Score: PRIDE complements binary success metrics by distinguishing successful paraphrased instructions according to their semantic and structural deviation.

5 Experiment

The experiment evaluates seven VLA configurations across architectures and training choices, then compares uniform success rate with PRIDE to assess paraphrase robustness and difficulty.

  • Experimental Setup: Seven VLA configurations spanning four architecture families are evaluated, with controlled comparisons of fine-tuning data scope and VLM training strategy.The configurations range from 0.6B to 7.5B parameters.
  • Success Rate Comparison: 22.8–51.9 pp of success-rate degradation occurs under paraphrasing across all models, showing that the effect is pervasive rather than architecture-specific.Xiaomi-Robotics-0 and VLA-Adapter drop by 22.8 pp and 51.9 pp, respectively.
  • PRIDE Reveals Hidden Severity: PRIDE exposes robustness overestimation when uniform success rate rewards easy and difficult paraphrases equally.Overestimation reaches 22.0% for VLA-Adapter and 18.2% for π0.5 expert-only, while π0.5 and Xiaomi-Robotics-0 show 8.4% and 8.9%.
  • Cell-Level Analysis: At the cell level, degradation intensifies along both object and action axes, with indirect actions combined with object paraphrasing producing the hardest cases.The SP-habitual × Question cell reaches a PRIDE score of 0.42.

6 Analysis

Analysis across architectures, training choices, linguistic axes, and trajectories shows that object variation is the dominant vulnerability and that most failures begin with incorrect task planning.

  • 6.1 Robustness Across Models and Training Choices: Paraphrase fragility persists across architectures, data scopes, and fine-tuning strategies.Across seven configurations, success-rate drops range from 22.8 pp to 51.9 pp; increasing task diversity fourfold yields similar drops of 32.4 pp and 33.2 pp.
  • 6.2 Object-Level Bottleneck: Object-paraphrased instructions cause larger success-rate gaps than action variation within the same object condition, identifying object variation as the stronger failure driver.Object-preserved versus object-paraphrased rows show the larger gap, especially when combined with indirect actions.
  • 6.2 Object-Level Bottleneck: 19.8–51.0 pp of performance loss follows object paraphrasing, including simple synonym substitutions such as stove to range.The nearly unchanged gaps for OpenVLA-OFTgoal and OFTmixed indicate that task diversity and object-paraphrase robustness are decoupled.
  • 6.3 Planning-Level Failures: 79.5–95.5% of failures are Far-GT, meaning trajectories diverge from successful-task trajectories at the planning level rather than during execution.Near-GT cases account for less than 5% in most models, with π0.5 expert-only as the exception at 12.5%.

7 Conclusion

LIBERO-Para and PRIDE reveal persistent paraphrase fragility in VLA models, dominated by object-level lexical variation and planning-level trajectory divergence.

  • Contributions: LIBERO-Para independently varies action and object expressions, while PRIDE provides fine-grained paraphrase-robustness assessment.The benchmark and metric target linguistic variation in robotic task interpretation.
  • Main Findings: 80–96% of failures arise from planning-level trajectory divergence, and object-level lexical variation is the dominant source of degradation.The findings indicate difficulty mapping diverse instructions to correct task identification.

Limitations

The paper limits evaluation to controlled, isolated paraphrase variations in the LIBERO simulation setting, while documenting the dataset-generation and taxonomy choices that define this scope.

  • Scope Limitations: Physical-robot validation remains necessary because the study evaluates VLA models only in the LIBERO simulation environment.The paper identifies differences in rendering fidelity, physics modeling, and sensor noise between simulation and real-world settings.
  • Scope Limitations: Compound paraphrase variations are deferred because the benchmark isolates one variation type along each axis at a time.Natural language may combine synonym substitution, adverb insertion, structural reorganization, and indirect speech acts.
  • Paraphrase Taxonomy: The taxonomy excludes several paraphrase categories when they risk semantic drift, unnatural commands, ambiguity, or disruption of imperative structure.Excluded examples include morphology, lexical-syntactic substitutions, discourse changes, and selected EPT categories.
  • Paraphrase Taxonomy: The benchmark retains selected lexical, syntactic, and directive types while treating canonical imperatives as the baseline rather than paraphrase variants.The action taxonomy includes coordination, subordination, and five directive types; the imperative condition corresponds to Action axis: None.
  • Dataset Construction: LIBERO-Para is generated through axis-wise paraphrase generation, verification, merging, and final verification.Independent action and object paraphrases can yield up to n × m merged paraphrases after validation.
  • Dataset Construction: The dataset contains 4,092 paraphrases from 10 LIBERO-Goal instructions selected across four task types.The selected task types are Spatial, Object, Goal, and Long.
  • Metric Design: PRIDE separates keyword distance from structural distance, with object substitutions driving lexical variation and coordination or subordination driving structural variation.Object substitutions yield keyword distances of 0.41–0.45, while coordination and subordination exceed 0.28 structurally.

A.5 Human Evaluation

The human evaluation supports the semantic validity of LIBERO-Para, with strong annotator agreement and high consensus that sampled paraphrases preserve successful behavior. Disagreement was concentrated in indirect speech-act transformations, showing that the benchmark includes pragmatic variation beyond simple lexical substitution.

  • Protocol: 205 samples were independently judged by 15 annotators for whether each paraphrase would elicit the same successful behavior in the given scene.The evaluation used a binary Yes/No decision on randomly sampled original–paraphrase pairs.
  • Agreement: Gwet’s AC1 was 0.854, indicating strong inter-annotator agreement despite labels being skewed toward the positive class.AC1 was selected because κ statistics can be deflated under positive-class imbalance.
  • Consensus: 99.51% of samples passed majority vote, while 89.27% passed the stricter threshold requiring at least 12 of 15 Yes judgments.Annotators selected Yes at an average rate of 14.13/15, or 94.18%, across all samples.
  • Error analysis: Disagreement among the 22 samples failing the stricter criterion concentrated in paraphrases changing imperatives into suggestive, declarative, or indirect speech acts.The pattern was attributed to differences in interpreting speech-act form rather than semantic distortion.
  • Benchmark scope: LIBERO-Goal uses one fixed keyword per object, whereas LIBERO-Para evaluates paraphrases that introduce lexical and pragmatic variation.The benchmark therefore tests variation absent from the fixed-reference instruction format.

B PRIDE: Paraphrase Robustness Index in Robotic Instructional DEviation

PRIDE is a task-grounded paraphrase distance metric designed to reflect how linguistic changes affect robotic task success. It combines keyword and structural variation, and its scores track increasing paraphrase difficulty more faithfully than general-purpose NLP metrics.

  • Metric design: PRIDE combines keyword similarity (SK) and structural similarity (ST) with task success to quantify paraphrase deviation in robotic instructions.The metric targets task-critical action and object expressions rather than text similarity alone.
  • Difficulty sensitivity: PRIDE rises monotonically as success rate falls, increasing from 0.03 to 0.35 to 0.56 while success drops from 90.8% to 70.3% to 31.5%.SK responds sharply to replaced task-critical keywords, while ST captures structural divergence when keywords remain preserved.
  • Comparison with NLP metrics: 1−BERTScore stays within 0.10–0.28 across paraphrase types, failing to distinguish benign additions from disruptive compound paraphrases.The comparison illustrates why general-purpose text distances can miss task-relevant difficulty.
  • Comparison with NLP metrics: 1−BLEU can assign the same distance to a trivial addition and a coordination paraphrase despite a 10.6 pp success-rate gap.METEOR likewise provides one scalar and cannot decompose whether distance comes from keyword replacement or structural transformation.
  • Validation: Across seven models, higher PRIDE scores correlate negatively with success, with Pearson r ranging from −0.671 to −0.877 and p < .0001.This relationship supports PRIDE as a meaningful paraphrase-difficulty metric.

C Experiment

The experiments evaluate seven VLA configurations using public or documented model setups, repeated across five random seeds in LIBERO. Analyses include PRIDE weighting and action-paraphrase breakdowns to characterize robustness trends.

  • PRIDE analysis: PRIDE scores decrease consistently as α increases toward keyword-centric weighting, indicating that keyword-based evaluation gives higher credit to samples models already solve easily.The analysis also examines per-model slopes of the PRIDE–α relationship.
  • Implementation: Experiments use publicly released checkpoints or official codebases, with π0.5 (expert-only) fine-tuned by freezing the VLM and updating only the action expert.The total evaluation cost is approximately 194 GPU hours.
  • Paraphrase analysis: Action paraphrases are grouped into Lexical, Structural, and Pragmatic categories for success-rate comparison.The reported averages decline from 82.7% for original instructions through 66–70% lexical and 57–63% structural variants to 48.1% for Questions and 48.4% for Hints.
  • Evaluation protocol: Each model is evaluated with five random seeds, and reported success rates are means over those independent runs.The protocol focuses on aggregate robustness trends rather than per-configuration variance.

D.1 Finding 1: Paraphrase Fragility Persists Across Architectures, Data Scales, and Fine-tuning Strategies

Paraphrase fragility appears across architectures, scales, and fine-tuning strategies, with object-level lexical changes producing the clearest degradation. Failure analysis indicates that most errors arise during task identification and planning rather than low-level execution.

  • D.1 Finding 1: OpenVLA-OFTgoal, OpenVLA-OFTmixed, and VLA-Adapter show preserved-versus-paraphrased gaps of 48.3 pp, 51.0 pp, and 37.1 pp, respectively.Their heatmaps show a visual boundary between object-preserved and object-paraphrased rows.
  • D.1 Finding 1: Every evaluated model falls below 50% success in the most challenging compound paraphrase cells, indicating universal paraphrase fragility across architectures.Object-paraphrased conditions produce especially sharp declines for several OpenVLA-OFT and VLA-Adapter variants.
  • D.1 Finding 1: Object-level keyword changes, rather than syntactic reformulations, are the dominant factor driving success-rate degradation across current VLA architectures.OpenVLA-OFTgoal and OpenVLA-OFTmixed also have the steepest PRIDE–α slopes, −17.3 and −18.7, respectively.
  • D.1 Finding 1: Action success declines from 66–70% for lexical changes to around 57–63% for structural reorganizations and approximately 48% for Question and Hint forms.The sharper action-axis decline for indirect speech acts reflects the need to recover an underlying imperative.
  • D.1 Finding 1: Fixed object references in LIBERO-Goal leave models unexposed to lexical variation during fine-tuning, plausibly reinforcing surface-level keyword matching.LIBERO-Para replaces these references while keeping the task setting grounded in the same manipulation environment.
  • D.1 Finding 1: Far-GT failures remain dominant under progressively stricter DTW thresholds, so the failure pattern is robust to the classification boundary.Near-GT failures are a small fraction across categories, except for elevated ratios in π0.5 (expert-only).
  • D.1 Finding 1: π0.5 (expert-only) can track the ground-truth path yet fail at execution because its unadapted action expert lacks sufficient precision.This contrasts with the broader pattern in which paraphrase failures originate in instruction-to-task identification.
Loading 2603.28301v1…