PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Uncertainty and Quality of Vision-Language-Action-enabled Robots

Pablo Valle, Chengjie Lu, Shaukat Ali, Aitor Arrieta

arXiv:2507.17049v4cs.SEcs.RO

TL;DR

Binary task-completion oracles can overlook quality differences among VLA robot executions. The paper adapts quality and uncertainty metrics and compares them with domain-expert labels. Several metrics correlate moderately to strongly with expert judgments, while the analyses also show that success rates alone are inadequate for assessing execution quality.

  • Problem

    Task-completion oracles assess whether a task succeeds but do not capture execution quality dimensions such as safety, smoothness, or efficiency.

    Source: §I · p. 1

  • Method

    The authors adapt eight uncertainty and five quality metrics and evaluate them against expert labels for 908 successful executions across three VLA models.

    Source: §I · p. 2

  • Results

    Several adapted metrics showed moderate to high correlation with expert evaluations, and π0 executions displayed poor quality despite generally optimistic self-assessments.

    Source: §I · p. 2

  • Takeaways & Limitations

    The findings challenge reliance on binary success rates alone and suggest potential for improved real-time monitoring and adaptive enhancement of VLA robots.

    Source: §I · pp. 1–2

    Metric settings pose potential internal-validity threats: EV used four inferences due to hardware limits, and temporal metrics used an eight-step window.

    Source: §VII · p. 16

Abstract

from arXiv · show

Vision-Language-Action (VLA)-enabled robots integrate visual perception, natural language understanding, and action planning to interpret their environment, comprehend instructions, and perform embodied tasks autonomously. Such robots are typically evaluated through task success rates, i.e., whether a robot performs its intended task, which are commonly used as test oracles for evaluating such robots. Such an evaluation fails to capture the quality of task execution and the robot's confidence in its decisions. In this paper, we adapt eight uncertainty metrics and five quality metrics specifically designed for VLA-enabled robotic manipulation tasks. We assess their effectiveness through a large-scale empirical study involving 908 successful task executions from three state-of-the-art VLA models across four representative robotic manipulation tasks and two robot embodiments. Human domain experts manually labeled task quality, enabling us to analyze the correlation between our proposed metrics and expert judgments, serving as a human oracle for testing such robots. The results reveal that several metrics show moderate to strong correlation with human assessments, highlighting their utility for evaluating task quality and model confidence. Furthermore, we found that some metrics can discriminate between high-, medium-, and low-quality executions from unsuccessful tasks, which is useful when test oracles are absent. Our findings challenge the adequacy of current evaluation practices that rely solely on binary success rates and pave the way for improved real-time monitoring and adaptive enhancement of VLA-enabled robots.

I. INTRODUCTION

Binary task-completion oracles can miss important differences in VLA execution quality, motivating complementary quality and uncertainty metrics. The paper evaluates these metrics against expert labels and describes how VLA systems turn observations and instructions into robot actions.

I. INTRODUCTION

  • Symbolic oracles check task completion but can miss safety, smoothness, and efficiency; successful executions may still involve collisions, unstable paths, or repeated drops.The paper argues that binary success alone provides limited assurance about overall robot performance.Source: §I · p. 1
  • The proposed evaluation framework integrates five quality metrics with eight uncertainty metrics for VLA-enabled robots.The metrics are intended to assess task quality and quantify robot confidence during execution.Source: §I · pp. 1–2
  • The study combines expert quality labels for 908 successful executions with correlation analyses of proposed quality and uncertainty metrics.Three domain experts labeled executions from three VLA models across four tasks.Source: §I · p. 2
  • The adapted metrics showed moderate to high correlation with expert evaluations, while π0 executions displayed poor quality despite generally optimistic self-assessments.The study analyzed 908 successful executions across three VLA models.Source: §I · p. 2

II. BACKGROUND ON VISION-LANGUAGE-ACTION-ENABLED ROBOTS

  • VLA systems combine visual observations, instructions, and robot state to generate action sequences that low-level controllers translate into actuator commands.The described pipeline includes visual, language, and proprioceptive encoders and an action decoder.Source: §I · p. 1, §II · p. 2

III. UNCERTAINTY METRICS

The paper measures uncertainty through token probabilities as well as generated actions. Its token-based metrics characterize prediction confidence and probability-distribution ambiguity at the model-output level.

III. UNCERTAINTY METRICS

  • Four token-based metrics complement four action-based metrics, covering probabilistic action generation and uncertainty reflected in embodied behavior.The token metrics are computed from discrete action-token probabilities, including for diffusion-based models using their pre-diffusion representation.Source: §III · p. 3, A. Token-based Uncertainty Metrics · p. 4

1) Token Probability (TB-TP):

  • TB-TP uses each action token's maximum predicted probability to assess how strongly the model favors its top prediction.Lower maximum probabilities indicate that probability is distributed across alternatives.Source: 1) Token Probability (TB-TP) · p. 4

2) Prediction Confidence Score (TB-PCS):

  • TB-PCS measures the probability margin between the two most likely predictions for each token, with a smaller gap indicating greater ambiguity.The margin reflects uncertainty in distinguishing between competing actions.Source: 2) Prediction Confidence Score (TB-PCS) · p. 4

3) DeepGini (TB-D):

  • TB-D uses the full token-probability distribution: a flatter distribution indicates more uncertainty than one dominated by a single token.It complements measures focused on the top predictions by capturing global distribution dispersion.Source: 3) DeepGini (TB-D) · p. 4

4) Entropy (TB-E):

The paper pairs distribution-based token uncertainty with behavioral measures of action consistency. Entropy summarizes token-distribution uncertainty, while finite differences capture changes in generated control sequences.

4) Entropy (TB-E):

  • TB-E measures randomness across the predicted token distribution, with higher entropy indicating that probability is spread across more possible tokens.Source: 4) Entropy (TB-E) · p. 4

B. Action-based Uncertainty Metrics

  • Action-based metrics apply successive finite differences to continuous actions, measuring temporal consistency at first-, second-, and third-order scales.They are intended to detect oscillatory or inconsistent behavior that token-based measures may not reveal.Source: B. Action-based Uncertainty Metrics · p. 4

1) Action Position Instability (A-PI):

  • A-PI measures changes between consecutive actions; large differences may indicate abrupt behavioral changes, although consistent high-speed motion can inflate the metric.Lower values correspond to more temporally coherent action sequences.Source: 1) Action Position Instability (A-PI) · p. 5

2) Action Velocity Instability (A-VI):

  • A-VI measures second-order action differences to capture changes in velocity, making it sensitive to oscillations and repeated corrections.It distinguishes variations in motion dynamics from the amount of motion itself.Source: 2) Action Velocity Instability (A-VI) · p. 5

3) Action Acceleration Instability (A-AI):

The section distinguishes uncertainty in generated actions from quality in executed trajectories. It describes action-instability and repeatability measures alongside trajectory-based measures of motion quality.

3) Action Acceleration Instability (A-AI):

  • A-AI detects persistent oscillations and high-frequency fluctuations by measuring action acceleration changes independently across action dimensions.It is sensitive to repeated micro-corrections that positional and velocity instability measures may miss.Source: 3) Action Acceleration Instability… · p. 5

4) Execution Variability (EV):

  • EV measures action dispersion across repeated inferences with identical inputs, serving as a proxy for epistemic uncertainty in the learned policy.Lower EV indicates more consistent actions and higher confidence; higher EV indicates greater uncertainty.Source: 4) Execution Variability (EV) · p. 5

IV. QUALITY METRICS

  • Trajectory-based quality metrics assess executed behavior and task progression, complementing uncertainty measures of action confidence and consistency.They target smoothness, stability, efficiency, and human-perceived motion quality not captured by task success alone.Source: §IV · p. 5

A. Trajectory Position Instability (TCP-PI)

  • TCP-PI measures consecutive Tool Center Point position changes to capture trajectory smoothness; larger changes may reflect abrupt or inconsistent motion.Positions are computed in Cartesian task space and trajectories are resampled to fixed temporal resolution before their values are averaged.Source: A. Trajectory Position Instability… · pp. 5–6

B. Trajectory Velocity Instability (TCP-VI)

These metrics characterize executed motion and goal-directed progress at different temporal and task scales. They measure velocity variation, acceleration change, overall trajectory jerk, and distance-to-goal behavior.

B. Trajectory Velocity Instability (TCP-VI)

  • TCP-VI measures second-order trajectory differences to capture motion-dynamics variation, with higher values indicating irregular acceleration patterns.Uniformly resampled task-space trajectories provide fixed timestep resolution across durations and sampling rates.Source: B. Trajectory Velocity Instability… · p. 6

C. Trajectory Acceleration Instability (TCP-AI)

  • TCP-AI measures acceleration-change rate, making it sensitive to high-frequency oscillations and abrupt transitions associated with jittery or unstable motion.Trajectories are normalized to a fixed timestep because the discrete jerk approximation is sensitive to sampling frequency.Source: C. Trajectory Acceleration Instability… · p. 6

D. Trajectory Instability (TI)

  • TI uses root mean square jerk to summarize motion quality across a trajectory; lower values indicate smoother, more consistent motion.The metric assesses overall end-effector motion rather than step-level behavior.Source: D. Trajectory Instability (TI) · p. 6

E. Optimal Trajectory Difference (OT)

  • OT evaluates goal-directed behavior through distance changes, shifting its reference from object approach to final placement after grasping.Distance is normalized for task and trajectory comparability; negative distance change indicates progress, while positive change suggests divergence or inefficiency.Source: E. Optimal Trajectory Difference (OT) · pp. 6–7

V. EMPIRICAL STUDY

The empirical study evaluates proposed uncertainty and quality metrics across three VLA models and four tasks. Its research questions address alignment with expert judgments, performance correlation, success discrimination, and inference overhead.

A. Research Questions

  • The study replicates VLATest quality assessment by examining successful tasks with human experts and tests whether the metrics correlate with expert-labeled quality levels.The correlation analysis asks whether the metrics can indicate robot performance degradation.Source: A. Research Questions · p. 7
  • The study examines whether metric distributions and effect sizes distinguish successful from failing tasks, and measures the inference-time overhead of metric integration.Overhead matters because robotic control operates in real time.Source: A. Research Questions · p. 7

B. VLA Models

The evaluation uses three VLA models on two robot embodiments and four manipulation tasks. It specifies the selected checkpoints, benchmark environments, and settings used to support reproducibility.

B. VLA Models

  • The study uses OpenVLA, SpatialVLA, and π0, with fine-tuned models mapped to Google robot tasks or WidowX robot tasks.SpatialVLA and π0 add model results not previously included in VLATest.Source: B. VLA Models · p. 7

C. Environments

  • SimplerEnv evaluation covers pickup and move-near tasks on the Google robot, and stacking and placing-inside tasks on the WidowX robot.The study uses the first 500 generated scenes per task, with randomly selected target objects and randomized object positions and poses.Source: C. Environments · pp. 7–8

D. Configurations

  • A fixed seed keeps initial scenes consistent across models, while EV uses four inference samples and temporal-difference metrics use an eight-step window.The authors selected eight steps because shorter windows produced less stable estimates, especially for A-AI and TCP-AI.Source: D. Configurations · p. 8

E. Execution Platform and Runs

The experiments used separate compute platforms by model, while task-quality judgments came from expert labeling and metrics were statistically compared with those labels and unsuccessful tasks.

E. Execution Platform and Runs

  • π0 experiments ran on an AMD EPYC 7773X CPU with an NVIDIA RTX A6000 GPU, while OpenVLA and SpatialVLA ran on an AMD EPYC 7763 CPU with two NVIDIA A100-SXM4-80GB GPUs.Both platforms used 64-bit Ubuntu 20.04 LTS, Python 3.10, and CUDA 12.8.Source: E. Execution Platform and Runs · p. 8
  • Average scene execution took approximately 270 seconds for SpatialVLA, 235 seconds for π0, and 210 seconds for OpenVLA; the 4-task, 500-scenes-per-task study required 397 GPU hours.The reported total was 1,430,000 seconds of GPU time.Source: E. Execution Platform and Runs · p. 8

F. Human Evaluation Procedure

  • Two domain-expert engineers labeled 908 successful executions as high, medium, or low quality, with a third expert resolving disagreements.The first two labelers had 85% agreement by Cohen’s kappa; labeling was distributed over 15 days, with at most 160 cases per session.Source: F. Human Evaluation Procedure · p. 8
  • Quality levels were defined with robotics experts before execution, and task-specific definitions were presented in Table I.Labelers received clear, illustrative examples.Source: F. Human Evaluation Procedure · pp. 8–9, Table I · p. 9

G. Statistical tests

  • Metric values were averaged over each execution and correlated with its human quality label using Spearman’s rank correlation; p < 0.05 indicated significance.Correlation strength was classified from no correlation through weak, moderate, and strong using Cohen’s thresholds.Source: G. Statistical tests · p. 9
  • For successful quality levels versus failed tasks, the study used Mann–Whitney U tests after finding non-normal distributions with Shapiro–Wilk tests.P-values below 0.05 indicated significance, and Vargha–Delaney’s A12 quantified effect size.Source: G. Statistical tests · p. 9

VI. ANALYSIS OF THE RESULTS AND DISCUSSION

The replication shows that binary success rates can obscure substantial differences in execution quality. Correlation analyses find that metric performance varies by model and task, with final object orientation remaining a limitation for Move near.

A. RQ1 - Replication

  • SpatialVLA led success rates on the two Google-robot tasks, while π0 led on the two WidowX-robot tasks; OpenVLA’s success rates ranged from 1.2% to 13.8%.OpenVLA quality also varied by task, and its last two task success rates were too low for reliable conclusions.Source: A. RQ1 - Replication · p. 10
  • For Pick up, π0 had 51 high-quality executions versus SpatialVLA’s 132 despite 161 versus 191 successful executions, showing success counts can conceal quality differences.Across tasks, SpatialVLA had the highest overall execution quality, while many successful π0 executions were labeled low-quality.Source: A. RQ1 - Replication · p. 10

B. RQ2 - Correlation

  • Uncertainty-metric correlations varied by model and task: nearly all were significantly positive for SpatialVLA, but none were positive for π0 on Put in or OpenVLA on Move near.The OpenVLA sample was too small for meaningful statistical assessment on Put in and Put on.Source: B. RQ2 - Correlation · pp. 10–11
  • A-VI and A-AI showed moderate-to-strong correlations in 8 of 12 task-model combinations, while orientation-based action variants often matched or exceeded positional variants.Neither positional nor orientation components consistently dominated across tasks, suggesting they provide complementary uncertainty information.Source: B. RQ2 - Correlation · pp. 10–11
  • Quality-metric correlations were task- and model-dependent: TCP-VI, TCP-AI, and TI were strong for Pick up across all three models, while component rankings differed between Pick up and Put in.For Put in, positional TCP variants often outperformed orientation variants, particularly for SpatialVLA; combined variants generally remained among the best performers.Source: B. RQ2 - Correlation · pp. 11–12
  • TCP-VI showed the most consistent statistically significant positive correlation, but Move near metrics did not adequately capture whether the object ended in the correct orientation.After removing low-quality Move near executions attributed to incorrect orientation, most metrics showed strong correlation for π0 and SpatialVLA.Source: B. RQ2 - Correlation · p. 12, Table IV · p. 12

C. RQ3 - Discrimination

The metrics often distinguish high-quality successes from failures, with OT the most consistent discriminator, but performance varies by model and task. Discrimination weakens for lower-quality successes, and static failures can misleadingly appear favorable on motion-based metrics.

C. RQ3 - Discrimination

  • OT was the strongest discriminator across models and tasks, with statistically significant, large effect sizes except for low-quality OpenVLA “Pick up” executions.Successful trajectories converge near the target, while failed ones end at non-zero target distances.Source: C. RQ3 - Discrimination · p. 13
  • The authors propose using quality metrics for design-time assessment and uncertainty metrics for runtime monitoring, particularly where high-quality execution is critical.The metrics may help identify unsuccessful or degraded behavior without relying exclusively on symbolic task oracles.Source: C. RQ3 - Discrimination · pp. 13, 15
  • Metric discriminability depended on the model: near-deterministic policies made EV uninformative, while more accurate policies with structured failure modes improved separability.OpenVLA’s overlapping distributions were associated with few high-quality successes and behavioral similarity between successes and failures.Source: C. RQ3 - Discrimination · p. 13
  • For medium-quality successes, discrimination varied by model and task; most metrics, except OT, generally failed to separate low-quality successes from failures.Low-quality successes often resembled unsuccessful tasks under the evaluated metrics.Source: C. RQ3 - Discrimination · pp. 13, 15
  • Some failures appeared favorable on motion-based metrics because the robot remained static; a minimal-motion threshold could flag these cases.Near-zero motion can indicate failure to localize or detect a target, rather than stable task execution.Source: C. RQ3 - Discrimination · p. 15
  • Most metrics distinguished high-quality successes from failures, but OpenVLA showed weaker or absent discrimination in several task-specific comparisons.The broader finding reports promising results for most metrics apart from OT, except for OpenVLA.Source: C. RQ3 - Discrimination · pp. 13, 15

D. RQ4 - Overhead

Most metrics add low computational overhead, but EV’s multiple model executions make it impractical for frequent control, and OT is mainly suited to offline simulation-based evaluation.

D. RQ4 - Overhead

  • EV had the highest overhead because it requires multiple model executions, making it impractical for frequent real-world control.The authors also report no positive correlation for EV in most cases.Source: D. RQ4 - Overhead · p. 16
  • The remaining metrics add low overhead: token-based metrics have the highest of these, while AI-, TCP-based metrics, TI, and OT add negligible cost.The reported TB overhead mainly comes from extracting token probabilities.Source: D. RQ4 - Overhead · p. 16
  • OT is primarily suited to offline simulation because detours, obstacle avoidance, and dynamic adjustments challenge its assumption that effective executions continually reduce goal distance.Measuring optimal trajectory distance can be infeasible in uncertain environments without complete environmental knowledge.Source: D. RQ4 - Overhead · p. 16

VII. THREATS TO VALIDITY

The evaluation’s validity is bounded by parameter choices, the diversity of tested platforms and settings, and the use of expert-assigned quality labels. The authors describe controls for these risks, including consistent hardware per model and procedures to resolve labeling disagreements.

VII. THREATS TO VALIDITY

  • Parameter choices may threaten internal validity: EV used four inferences, and temporal-difference metrics used an empirically selected eight-step window.The authors chose eight steps as a trade-off between temporal context and computational cost.Source: §VII · p. 16
  • Findings are primarily in-domain for three models, four manipulation tasks, and two simulated embodiments, and should not be directly extrapolated to substantially different settings without validation.The authors note that broader generalizability is constrained by limited robotic-platform diversity.Source: §VII · p. 16
  • Expert labeling may affect conclusion validity, although three experienced experts, disagreement resolution, and an 85% Cohen’s kappa agreement between the first two labelers helped address this risk.Labeling was distributed across 15 days, with a maximum of 160 tasks per labeler.Source: §VII · pp. 16–17

A. Uncertainty in DL Models and Foundation Models

Prior uncertainty and quality-assessment methods leave gaps for VLA-enabled robotics: uncertainty quantification is underexplored, and existing evaluation often centers on task completion. The paper adapts uncertainty and quality metrics, reports their empirical utility, and outlines future applications and limitations.

A. Uncertainty in DL Models and Foundation Models

  • Uncertainty methods span Bayesian, dropout, ensemble, and task-specific metrics, but uncertainty quantification remains largely unexplored for VLA models.The paper adapts token-probability metrics and introduces output-based measures of variability across stochastic inferences.Source: A. Uncertainty in DL Models and… · pp. 17–18

B. Quality Assessment in VLA-Enabled robots

  • VLA benchmarks and end-to-end evaluations commonly emphasize task completion, while the paper’s approach assesses the broader performance of the integrated robot.The proposed evaluation encompasses perception, language, reasoning, and execution quality.Source: B. Quality Assessment in VLA-Enabled… · p. 18

IX. CONCLUSIONS AND FUTURE WORK

  • Across 908 successful executions, several of the eight uncertainty and five quality metrics correlated moderately to highly with expert evaluations; OT best distinguished successful from failing tasks.The conclusion highlights A-VI, A-AI, and TCP-VI as particularly adequate indicators, and proposes future work on test oracles, test generation, and runtime monitoring.Source: §IX · pp. 18–19

APPENDIX A IMPLEMENTATION GUIDELINES

The appendix describes the evaluated models and their model-specific implementations of token-based uncertainty metrics, then visualizes metric distributions across human quality labels. The plots complement the Spearman correlations reported in Table IV.

APPENDIX A IMPLEMENTATION GUIDELINES

  • The appendix outlines model architectures and ad-hoc token-metric implementations, with implementation scripts available in the GitHub repository.Source: Appendix A · p. 21

A. Models Architecture and Details

  • OpenVLA combines a visual encoder, projector, and Llama 2 language-model backbone, with image features from SigLIP and DINOv2 concatenated.It is a 7B-parameter model pretrained on 970k Open X-Embodiment demonstrations.Source: A. Models Architecture and Details · p. 21, Figure 4 · p. 21
  • π0 pairs a SigLIP–Gemma-2B vision-language backbone with a 300M-parameter action-expert transformer for continuous actions.It is a 3.3B-parameter model trained with conditional flow matching on a mixture of datasets including Open X-Embodiment, Bridge, and DROID.Source: A. Models Architecture and Details · p. 21, Figure 5 · p. 22
  • SpatialVLA uses SigLIP visual features, ZoeDepth-derived 3D position features, adaptive spatial action grids, and a PaLIGemma-2 backbone.Its adaptive action grids align cross-robot actions with 3D spatial structure, eliminating the need for robot-camera extrinsic calibration and making the model agnostic to specific robot setups.Source: A. Models Architecture and Details · p. 22, Figure 6 · p. 22

B. Implementation of the token-based uncertainty metrics

APPENDIX B IN DEPTH ANALYSIS OF THE RESULTS

  • Figures 7–9 show metric-value distributions by human evaluation label, while Table IV provides the corresponding Spearman correlations; Figure 7 independently scales each task-and-metric y-axis.The figures visualize full distributions across individual test cases, complementing the numerical correlations.Source: Appendix B · pp. 22–23, Figure 7 · p. 24, Figure 8 · p. 25
Loading 2507.17049v4…