PaperlayerView exact revision on arXiv

Source-linked AI summary

LIBERO-RIG: A Hardware-Setup Robustness Benchmark for Vision-Language-Action Models

Andrea Yaoyun Cui, Hanyu Wang, Xinyi Wang, Bangzheng Li, Lin Zhang, Zhenggang Tang, Shyamal Buch, Dejia Xu, Matthew Tancik

arXiv:2610.18843v1cs.ROcs.AI

TL;DR

Robot policies must generalize across hardware and setup changes, but existing robustness benchmarks cover few rig faults. LIBERO-RIG varies the rig while holding LIBERO tasks fixed, and pooled success declines as perturbation severity increases. The benchmark’s per-axis profiles show which rig changes each model tolerates.

  • Problem

    Existing robustness benchmarks perturb tasks, observations, or scenes but omit several rig faults, leaving limited evidence about policies’ robustness to changes such as calibration, control rate, and end effectors.

    Source: §1 · p. 2

  • Method

    LIBERO-RIG perturbs the robot rig across 11 axes and five severity levels while keeping LIBERO tasks fixed, evaluating nine VLA models across 210 conditions.

    Source: §1 · p. 2, §2 · pp. 2–3

  • Results

    Pooled success across the eight models tested on all four suites falls from 0.932 nominal to 0.767 at middle severity and 0.509 at the highest severity.

    Source: §3 · p. 3, §4 · p. 4

  • Takeaways & Limitations

    Per-axis robustness profiles show practitioners which rig changes each model tolerates.

    Source: §4 · p. 4

    Measurements use simulation, one embodiment, and one perturbed axis per episode; uncertainty intervals are based on ten tasks per suite.

    Source: §4 · p. 4

Abstract

from arXiv · show

Robot policies are expected to generalize across deployment environments, including variations in the robot itself: cameras are bumped and remounted, calibrations drift, controllers run at different rates, and end-effector pads wear down. We present LIBERO-RIG, a benchmark that perturbs only the robot's hardware and setup while holding every LIBERO task fixed. The benchmark spans eleven perturbation axes, including camera mounting and optics, camera-to-arm calibration, actuation gain, control rate, proprioceptive noise, base pose, table height, and gripper pads. Each axis has five severity levels, yielding 210 conditions across the four LIBERO suites. We evaluate nine recent vision-language-action models using 500 episodes per condition and substantiate each finding with rigorous statistical tests. Pooled success falls from 0.932 to 0.767 at the middle severity and 0.509 at the highest severity. Performance degradation varies substantially across perturbation axes and action interfaces, and training on visual perturbations does not mitigate faults that leave camera images unchanged. We release the benchmark together with manifests specifying the fixed perturbation parameters for each episode.

1 Introduction

Robot policies must handle changes to the robot and its setup even when the task stays the same. LIBERO-RIG addresses gaps in existing robustness benchmarks with a benchmark focused on rig changes.

1 Introduction

  • Robot policies are expected to generalize beyond their training conditions because deployment robots may differ in cameras, calibration, control rates, gripper pads, or mounting.The task remains fixed while the hardware and setup change.Source: §1 · p. 2
  • Existing LIBERO-based robustness benchmarks perturb tasks, observations, or scenes but omit several rig faults, leaving policies’ dependence on specific setups untested.The omitted faults include camera-to-arm calibration, actuation gain, control rate, the wrist camera, proprioception, and the end effector.Source: §1 · p. 2
  • LIBERO-RIG perturbs the robot rig across 11 axes and five severity levels, yielding 210 conditions across four LIBERO suites.The levels range from everyday drift to a markedly different setup.Source: §1 · p. 2

2 The LIBERO-RIG Benchmark

LIBERO-RIG changes the robot setup while keeping each LIBERO task fixed, and scores performance across perturbation conditions. The benchmark also checks task solvability and goal visibility.

2 The LIBERO-RIG Benchmark

  • LIBERO-RIG keeps each task’s instruction, objects, success predicate, and nominal initial states fixed while changing only the rig.The simulated platform is a Franka Panda arm in MuJoCo using robosuite.Source: §2 · p. 2
  • The 11 axes span cameras, calibration, controller gain and rate, reported arm-state noise, base pose, table height, and end-effector pads.Faults are classified by their direct effect on camera images at a fixed arm and object configuration: observable faults change an image, while latent faults do not.Source: §2 · p. 2, Table 1 · p. 2
  • Each condition combines a suite, axis, and severity level; each evaluates 500 episodes, and scores are raw success rates compared with unperturbed performance.Table height applies to only two suites, giving 210 perturbed conditions and four nominal conditions; policies receive no fault metadata.Source: §2 · p. 3
  • The benchmark checks that every condition is solvable and audits goal visibility for camera-axis perturbations.Solvability is verified using either a controller with access to fault parameters or at least one evaluated model.Source: §2 · p. 3
  • Nine VLA models are evaluated across four action-decoding families, while LingBot-VA has a checkpoint only for LIBERO-10.The model set includes two world-model policies, Cosmos Policy and LingBot-VA.Source: §2 · p. 3

3 Findings

Across models and perturbation axes, success declines as severity rises, and the size of degradation differs across axes and action interfaces. Results also vary by suite, with LIBERO-10 presenting a consistent challenge.

3 Findings

  • At severity 3, mean success across axes ranges from 0.448 to 0.931, with camera remounting among the most damaging faults and reported arm-pose noise causing no observed degradation.Reorienting the scene camera in place and shifting the robot’s base produce the next-largest drops; the ordering is similar at severities 4 and 5.Source: §3 · p. 3, Figure 2 · p. 3
  • Pooled success falls from 0.932 nominal to 0.767 at severity 3 and 0.509 at severity 5.Severity 1 is the only level within a 5 percent non-inferiority margin of nominal; success at levels 1, 2, and 4 is 0.914, 0.845, and 0.591, respectively.Source: §3 · p. 3
  • π0.5 ranks first on LIBERO-Spatial, LIBERO-Object, and LIBERO-Goal, but its narrow LIBERO-10 lead is not statistically distinguishable from five rivals.It retains top rank in 65 percent of LIBERO-10 task resamples.Source: §3 · p. 3
  • Every model tested on all four suites has lower success on LIBERO-10 than its average across the other three suites, with severity-3 gaps from 0.049 for X-VLA to 0.184 for NORA.The pattern also holds for every model at severities 4 and 5.Source: §3 · pp. 3–4
  • Corrupting the policy’s reported arm pose leaves pooled success at 0.931 versus 0.932 nominal, whereas controller-state noise that causes motion errors reduces OpenVLA-OFT success from 0.960 to 0.470 at 2 cm.The results are consistent with reliance on visual localization of the arm.Source: §3 · p. 4
  • Visual-perturbation training improves OFT-Plus only on camera axes represented in its training, while X-VLA’s absolute-pose interface shows little degradation under calibration faults.OFT-Plus is within 0.02 of OpenVLA-OFT on action-frame rotation and every other axis; X-VLA’s calibration-fault success is 0.968 versus 0.976 nominal.Source: §3 · p. 4

4 Limitations and Conclusion

LIBERO-RIG’s conclusion reports substantial performance losses under rig changes and releases replayable episode manifests. The appendix adds figures, tables, and analyses documenting results and methods.

4 Limitations and Conclusion

  • Measurements use simulation and one embodiment, perturb one axis per episode, and base uncertainty intervals on ten tasks per suite.Raw success rates reflect both baseline task proficiency and robustness.Source: §4 · p. 4
  • Pooled success falls to 0.767 at middle severity and 0.509 at the highest; camera placement is most damaging, and calibration and timing faults resist visual-robustness training.The benchmark provides per-axis profiles and manifests specifying fixed perturbation parameters for each episode.Source: §4 · p. 4

A Additional Figures and Tables

B The Axes in Detail

The benchmark changes hardware and setup while keeping LIBERO tasks fixed, with perturbation axes defined by physical changes and severity ladders. Its axes span visible camera changes and faults that affect motion or reported state instead.

B The Axes in Detail

  • Task instructions, objects, success predicates, and nominal starts stay fixed; frozen manifests specify episode perturbations, and fairness checks assess feasibility without filtering scores.The benchmark includes 11 axes and five-level ladders; the appendix describes 107,000 manifest entries.Source: B The Axes in Detail · p. 7
  • Table 3 pools raw success and nominal ratios across models and axes, with only severity s1 within its non-inferiority margin; Table 4 describes severity-3 rank stability across task resamples.Source: Table 3 · p. 8, Table 4 · p. 8
  • Scene-camera orientation, position, and field of view change the images; audits check goal visibility, and camera-position collapses can occur without blind episodes.The camera-position ladder moves and re-aims the camera around the workspace; the orientation ladder twists it in place, while the field-of-view ladder changes the lens view.Source: B The Axes in Detail · pp. 8–9
  • Wrist-camera mounting changes are not shown in scene-camera renders, while action-frame rotation leaves the first image identical and changes later images through arm motion.The wrist-camera axis twists and slides the bracket; the action-frame fault rotates commanded motion by 2 to 25 degrees.Source: Figure 5 · p. 9, B The Axes in Detail · pp. 9–10
  • Actuation gain scales motion commands, while control-frequency mismatch changes the loop rate from trained 20 Hz and deliberately uses a non-monotone severity ordering.At level 5, gain compensation can saturate; frequency conditions use a time-constant episode budget and leave the per-step motion cap unchanged.Source: B The Axes in Detail · p. 10
  • Proprioception noise corrupts the pose reported to the policy, not the controller’s state; this report-only setup understates a real encoder bias that affects both paths.Noise sequences are fixed per episode, and the gripper state remains clean.Source: B The Axes in Detail · pp. 10–11
  • Base shifts and table-height changes alter robot–object geometry; oversized gripper pads can destabilize a mug during transport, while worn pads remain replayable.Table height is measured only on two suites, so its suite mix differs from all-suite axes.Source: B The Axes in Detail · p. 11

C Full Results and the Severity 4 and 5 Analyses

The severity-4 and -5 analyses preserve the benchmark’s broad finding of axis-dependent degradation, while changing which axes are worst at the harshest levels. The accompanying results also report long-horizon gaps and model-specific tolerance thresholds.

Severity 4 and 5 analyses

  • At severity 4, axis spread is 0.775 and control-frequency mismatch is worst at 0.153; at severity 5, action-frame rotation is lowest at 0.279, ahead of camera position, orientation, and base pose.At level 5, floor effects compress the axes to five significance-letter groups.Source: C Full Results and the Severity 4 and… · p. 11
  • All eight models have negative LIBERO-10 gaps at severities 3, 4, and 5, with exact Wilcoxon p = 0.008 at each level.Source: C Full Results and the Severity 4 and… · p. 12
  • At severity 3, every model passes wrist-camera mount, actuation gain, and proprioception noise under the absolute 0.80 threshold, while seven of eight pass proprioception at severity 5.The tolerance grid also reports the harshest passing level under a threshold of 0.90 of each model’s nominal rate.Source: C Full Results and the Severity 4 and… · p. 12

D Additional Findings from the Profiles

Rig robustness depends on the fault and the action interface: camera-perturbation training gains concentrate on camera axes, while other faults expose distinct sensitivities.

D Additional Findings from the Profiles

  • Control-frequency mismatch damages policies through overshoot, and the two video-predictive models have the lowest rates on this axis.The proposed fixed-frame-rate training explanation is a hypothesis based on two model rows.Source: D Additional Findings from the Profiles · p. 12
  • X-VLA’s absolute-pose interface is associated with less sensitivity to calibration faults but does not protect against scene-camera perturbations or control-frequency mismatch.Its calibration scores are 0.968 against a 0.976 nominal, while scene-camera scores are 0.329 and 0.295 and control-frequency score is 0.557; training data and chunk length also differ, so this is an association, not a controlled comparison.Source: D Additional Findings from the Profiles · p. 12
  • OFT-Plus improves over OpenVLA-OFT on camera orientation (0.867 against 0.701), camera position (0.787 against 0.590), and field of view (0.922 against 0.874).Overall, the models score 0.796 and 0.761, respectively, from near-identical nominal rates.Source: D Additional Findings from the Profiles · p. 12
  • On every other axis, including the wrist camera absent from OFT-Plus’s training mixture, the models differ by at most 0.02.A camera-only robustness benchmark would therefore characterize OFT-Plus as robust while missing these unrepresented axes.Source: D Additional Findings from the Profiles · p. 12

E Statistical Methods

The analyses account for clustered, paired task outcomes, use resampling and permutation tests for comparisons, and distinguish evidence of harmlessness from a failed significance test.

E Statistical Methods

  • Tasks, not episodes, are the independent units: treating episodes as independent narrows the severity-3 interval about twelvefold, while paired evaluations use the same tasks and starts across models and axes.Pooled rates weight episodes equally; equal weighting of suites changes pooled levels by at most 0.001.Source: E Statistical Methods · p. 12
  • Table 5 reports each model’s harshest passed severity using either a 0.80 success threshold or 0.90 of its own nominal rate.The table marks models that pass the harshest level with ≥ and those that fail the mildest with <.Source: Table 5 · p. 13
  • Confidence intervals resample each suite’s 10 tasks together across models, axes, and severities, but their finite-sample coverage is not established with so few clusters.Independent draws per model inflated rank variation; on the goal suite, π0.5’s rank-1 frequency was 0.83 rather than 0.99 with shared draws.Source: E Statistical Methods · p. 13
  • Within-block permutation found the 11 axes’ spread in success rates unlikely under exchangeable axis labels (p < 0.001), with the result interpreted as descriptive of this model roster.The test shuffles axis labels within model-suite blocks measured on the same tasks; these bounded proportions include many values at 0 or 1.Source: E Statistical Methods · pp. 13–14
  • Pairwise sign-flip tests with Holm correction separated 35 axis pairs, while overlapping compact-letter groups are not rankings or tiers.Camera orientation and control-frequency mismatch differ by 0.190 in their means but are not separated (adjusted p = 0.078); actuation gain error and proprioception noise differ by 0.023 and are separated.Source: E Statistical Methods · pp. 14–15
  • Severity 1 passes the 5 percent non-inferiority criterion (ratio 0.981, interval [0.978, 0.984]), but severity 2 fails it (0.907, [0.896, 0.916]).Passing bounds the loss rather than proving its absence, and the margin is a convention.Source: E Statistical Methods · p. 15
  • Among the 292 variable blocks, 287 have negative severity-success correlations; 193 of all 304 blocks decrease strictly across levels.Levels are ordered but not on a common scale across axes, so the analysis uses ranks; the 10,000-permutation test reports p = 1/10,001.Source: E Statistical Methods · pp. 15–17
  • All eight models have a negative LIBERO-10 gap at every checked level (exact Wilcoxon p = 0.008), describing this chosen roster rather than a sampled population.The gap mixes task length, nominal rates, and task content, which raw success rates do not separate.Source: E Statistical Methods · p. 17

F Control-Frequency Decomposition

Diagnostic interventions show that control-rate effects depend on how often a policy receives new observations, not simply on its simulated-time budget.

F Control-Frequency Decomposition

  • Across both models, looking more often helps more than having more time, although the two intervention-based shares are not an additive decomposition.The diagnostic rules freeze either the step count or the re-query interval on the same 20 conditions and paired starting states.Source: F Control-Frequency Decomposition · pp. 18–19
  • At 5 Hz, the time limit accounts for about 12 percent of OpenVLA-OFT’s deficit and blind execution for about 78 percent; for π0.5, the corresponding shares are about 61 percent for the time limit.π0.5 natively re-queries every 5 steps, so it was never as blind.Source: F Control-Frequency Decomposition · p. 19
  • π0.5 scores 0.929 at 100 Hz officially, but forcing full-chunk execution drops it to 0.490.The passage attributes its high-rate strength to frequent replanning, not tolerance of the rate.Source: F Control-Frequency Decomposition · p. 19
  • The overshoot probe replays one OpenVLA-OFT chunk from one episode and state, so it illustrates a mechanism rather than estimating a population.Net travel was 4.77 cm at 20 Hz and 17.36 cm at 5 Hz; trajectory deviation was the least stable measure across identical invocations.Source: F Control-Frequency Decomposition · p. 19
  • At 5 Hz, achievable speed remains about 94 percent of nominal, and the runs rule out a speed-budget account and a physical-deadline account in LIBERO’s quasi-static setting.The passage also notes that slower control compresses learned grasp-phase coordination.Source: F Control-Frequency Decomposition · p. 20
  • At 5 Hz, each command executes for 200 ms and the network re-queries OpenVLA-OFT only once per eight-step chunk, or every 1.6 seconds.The arm re-anchors commands on its measured pose each step, but new visual information arrives only when the network re-queries.Source: F Control-Frequency Decomposition · p. 20

G Proprioception: Per-Model Reading and Follow-Ups

Harsh proprioception noise has different effects depending on whether a model uses the reported state, and small no-op discrepancies remain unexplained.

G Proprioception: Per-Model Reading and Follow-Ups

  • At the harshest dose, NORA and MolmoAct match nominal success because their inference code does not pass state into the network, while X-VLA alone shows a large gap (−0.199).X-VLA’s first action changes by 0.436 under the corruption; its absolute hand-position goals must be anchored on the sensed state.Source: G Proprioception: Per-Model Reading… · p. 21
  • For π0.5 and OpenVLA-OFT, first-action changes under corrupted state are small, while camera images at the first step remain identical to the unperturbed run.OpenVLA-OFT’s first action changes by 0.006 at severity 3 and 0.023 at severity 5, compared with a typical action norm of 0.33.Source: G Proprioception: Per-Model Reading… · p. 21
  • The OpenVLA-OFT follow-up found no degradation below 5 mm of in-loop noise and halved success at 2 cm, but a general noise budget needs evaluation across more models and suites.No-op discrepancies also occur for models that do not read the perturbed input; their cause was not isolated, so they are not interpreted as policy responses.Source: G Proprioception: Per-Model Reading… · pp. 21–22

H Fairness Checks and the Scoring Decision

The authors retain all benchmark conditions after audits found most witness-rejected units solvable, avoiding a scoring gate that inflated scores and removed discriminative cases. They also exclude actuation latency as an executor-level fault and document model-specific setup and parity checks.

H Fairness Checks and the Scoring Decision

  • 29 of 30 witness-rejected units were solvable by at least one evaluated model, and retaining all 210 conditions avoids gate-driven score inflation and deleted discriminative cases.The remaining unit had successes across four models and was deemed hard, not proven unsolvable; the protocol therefore scores every episode.Source: H Fairness Checks and the Scoring… · pp. 25–26
  • The audit flagged 315 camera-direction combinations, mapping to 2,017 of 40,000 camera-axis episodes; the blind-episode list is released but does not filter scores.The wrist-camera axis had no blind direction at any severity.Source: H Fairness Checks and the Scoring… · pp. 26–27

I The Excluded Axis: Actuation Latency

  • A pilot latency fault cut OpenVLA-OFT from 97 percent to 1 percent at 400 ms, but was excluded because a software executor can remove it without modifying the policy.The authors treat such a fault as measuring the executor rather than the policy; level 5 remains a parity-gate condition.Source: I The Excluded Axis: Actuation Latency · p. 27

J Model Setup and Parity

  • Model-specific serving and preprocessing settings were documented, and nominal preflights served as parity and integration checks rather than isolating causes of performance gaps.The preflight used 400 episodes per model, or 100 for the LIBERO-10-only arm; some models entered below the 0.85 guideline after integration checks.Source: J Model Setup and Parity · pp. 27–28

K Sources for the Benchmark Comparison

The benchmark comparison is limited to five checked benchmarks and distinguishes reported evaluations from simulator capabilities and training-time randomization.

K Sources for the Benchmark Comparison

L Compute and Provenance

The confirm evaluation covered 881,500 episodes at a reported cost of roughly 6,800 node-hours. The rig-augmentation pilot did not beat its same-era baseline overall, with camera-axis gains accompanied by losses elsewhere.

L Compute and Provenance

  • The confirm tier evaluated 881,500 episodes across nine arms and cost roughly 6,800 node-hours, or 54,000 GPU-hours.The reported total includes idle tails and reruns; four MolmoAct units totaling 200 episodes recorded success counts only.Source: L Compute and Provenance · p. 28

M Rig-Augmentation Pilot

  • The benchmark comparison describes existing evaluations as changing tasks or scenes, while LIBERO-RIG varies rig properties; its table sources distinguish reported evaluation factors from implemented but excluded ones.The compared set is limited to five benchmarks, and a documented dash is not proof that software cannot express a factor.Source: Table 17 · p. 29, Table 18 · p. 30
  • Neither rig-augmentation arm beat the same-era baseline overall: T1 scored 0.76 and T1b about 0.71 against 0.79.Camera-axis gains appeared on the two short suites represented in replayed demonstrations, while latent-axis scores did not move; T1’s LIBERO-10 nominal fell to 0.69 and T1b recovered to 0.85 while sacrificing most camera gains.Source: M Rig-Augmentation Pilot · pp. 28–29

Related papers

Loading 2610.18843v1…