PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboCasa-Shift: Stage-Aware Robustness Evaluation of Vision-Language-Action Policies in Long-Horizon Household Manipulation

Haocheng Wang, Youquan Liu, Weiyu Guo, Aokai Zhang, Donghang Wu, Hongyu Liu, Sha Zhang, Weidong Yang, Ben Fei

arXiv:2610.27861v1cs.RO

TL;DR

Robustness benchmarks increasingly cover controlled distribution shifts, yet robustness is still commonly summarized at the task level. RoboCasa-Shift evaluates controlled single-factor shifts using matched clean-perturbed rollouts and simulator-grounded diagnostics of semantic execution progress. Robustness responses vary across perturbation factors and tasks, severity does not always yield monotonic degradation, and similar end-to-end degradation can correspond to different semantic failure depths.

  • Problem

    Robustness benchmarks increasingly cover controlled distribution shifts, yet robustness is still commonly summarized at the task level.

    Source: §I · p. 1

  • Method

    RoboCasa-Shift evaluates controlled single-factor shifts using matched clean-perturbed rollouts and simulator-grounded diagnostics of semantic execution progress.

    Source: §I · pp. 1–2

  • Results

    Robustness responses vary across perturbation factors and tasks, severity does not always yield monotonic degradation, and similar end-to-end degradation can correspond to different semantic failure depths.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    The findings motivate robustness evaluation that jointly captures outcome degradation, episode-level sensitivity, and failure progression rather than relying only on endpoint outcomes.

    Source: §V · p. 8

    The formal results cover one policy on four clean-qualified tasks in simulation, and transfer to real-robot deployment remains to be validated.

    Source: E. Scope and Limitations · p. 8

Abstract

from arXiv · show

Vision-language-action (VLA) robustness is commonly summarized by aggregate task success, which obscures how individual executions change under distribution shifts and where failures emerge in long-horizon tasks. We introduce \textbf{RoboCasa-Shift}, a diagnostic robustness benchmark built on RoboCasa365 that combines controlled single-factor shifts, matched clean-perturbed rollouts in which each pair shares the same underlying settings and differs only in the target shift, and simulator-grounded stage-aware diagnostics. We evaluate GR00T N1.5 on four long-horizon tasks that pass a predefined clean-performance gate under four perturbation factors and three calibrated severity levels, including conditions that drive the evaluated policy to complete task failure, totaling 1,040 formal rollouts. Robustness responses vary substantially across factors and tasks, and increasing severity does not always produce monotonic degradation. For example, one camera condition yields $\Delta\mathrm{SR}=0$ despite 8 of 80 matched episodes changing outcome. Stage-aware analysis further shows that the same end-to-end degradation can correspond to different execution-depth profiles. Camera $L_2$ and Gaussian blur $L_1$ both yield $\Delta\mathrm{SR}=0.188$, yet their first-stage failure rates are 19.0\% and 36.4\%, respectively. Our findings suggest that VLA robustness should be viewed as a multi-dimensional property of execution rather than a single end-to-end performance measure, motivating evaluation protocols that jointly capture outcome degradation, paired outcome sensitivity, and failure progression. The code is publicly available at https://github.com/HaochengWang23/RoboCasa-Shift.

I. INTRODUCTION

Aggregate success rates do not show how individual executions change under distribution shifts or where failures emerge in long-horizon tasks. RoboCasa-Shift pairs clean and perturbed rollouts and adds simulator-grounded stage diagnostics; its evaluation finds varied robustness responses across factors, tasks, and execution depths.

I. INTRODUCTION

  • Aggregate success-rate summaries do not reveal how individual executions change under distribution shifts or where failures emerge during long-horizon execution.Comparisons across different task instances or initial conditions may confound episode difficulty with the target shift.Source: §I · p. 1
  • Long-horizon tasks unfold through semantically meaningful subgoals, so failed executions can terminate after different amounts of progress.Stage-aware evaluation tracks progress and the first incomplete subgoal alongside final success or failure.Source: §I · pp. 1–2
  • RoboCasa-Shift combines controlled single-factor shifts at frozen severity levels, matched clean-perturbed rollouts, and simulator-grounded stage-aware diagnostics for long-horizon manipulation.The benchmark characterizes end-to-end outcomes, episode-level outcome changes, and semantic execution behavior.Source: §I · pp. 1–2
  • Robustness responses vary across perturbation factors and tasks, and increasing calibrated severity does not always produce monotonic end-to-end degradation.Similar end-to-end degradation can also correspond to failures emerging at different semantic depths.Source: §I · pp. 1–2

II. RELATED WORK

Prior work develops VLA policies, broadens robustness benchmarks, and structures long-horizon manipulation evaluation. RoboCasa-Shift builds on these directions by using controlled shifts, paired rollouts, and semantic subgoals to diagnose robustness during execution.

A. Vision-Language-Action Models for Robot Manipulation

B. Robustness and Generalization Benchmarks for Robot Manipulation

  • Recent robustness benchmarks evaluate generalist policies across diverse controlled shifts, but predominantly report task- or condition-level performance –[5].RoboCasa-Shift complements them with episode-level counterfactual comparison and semantic subgoal-aware diagnosis.Source: B. Robustness and Generalization… · p. 3

C. Long-Horizon and Compositional Manipulation Evaluation

  • Long-horizon benchmarks expose temporal or subtask structure for capabilities such as planning and sequential reasoning [6],, while its use for shift robustness is less explored.RoboCasa-Shift uses semantic subgoals to track progress and execution depth under shifts.Source: C. Long-Horizon and Compositional… · p. 3

III. ROBOCASA-SHIFT BENCHMARK

  • RoboCasa-Shift combines clean-competence screening, controlled single-factor shifts with frozen severity levels, matched counterfactual rollouts, and simulator-state-based semantic diagnostics.Official task success remains the end-to-end criterion, complemented by diagnostics of execution progress.Source: §III · p. 3, Figure 2 · p. 3

A. Task Suite

The formal suite is selected through a predefined clean-performance screen, then evaluated under controlled shifts and matched clean-perturbed rollouts. This design aims to isolate shift-associated changes while retaining task and episode conditions for comparison.

A. Task Suite

  • Four of 16 screened RoboCasa365 Composite-Seen tasks met the predefined threshold of at least 12/20 clean successes for the formal suite.Candidates with at most three successes in the initial 10 clean rollouts were excluded as clean-floor cases; others were extended to 20.Source: A. Task Suite · p. 3

B. Controlled Distribution Shifts

  • The benchmark evaluates illumination, camera orientation, robot base initialization, and Gaussian blur at three calibrated, frozen severity levels.Equal level indices across factors do not imply physically equivalent perturbation magnitudes.Source: B. Controlled Distribution Shifts · p. 4

C. Counterfactual Paired Evaluation Protocol

  • Each clean-perturbed pair shares its task, instruction, scene and object configuration, initial simulator state, horizon, and policy, with only the target perturbation changed.For stochastic policies, the episode-level policy seed is shared; validation reports that sharing it restores consistent action sequences.Source: C. Counterfactual Paired Evaluation… · p. 4, Table II · p. 4

D. Subgoal-Aware Failure Tracking

StageTracker represents long-horizon tasks as ordered semantic subgoals with simulator-state predicates, then tracks completion and the first incomplete subgoal. Stage-aware measures summarize execution depth and localize the first unreached milestone, complementing official task success.

D. Subgoal-Aware Failure Tracking

  • StageTracker evaluates ordered subgoal predicates during rollout, records first completion, latches completed subgoals, and identifies the deepest milestone and first incomplete subgoal.This provides simulator-grounded failure localization without relying on terminal-state conditions alone.Source: D. Subgoal-Aware Failure Tracking · pp. 4–5, Figure 3 · p. 4

E. Stage-Aware Robustness Measures

  • Official task success remains the primary end-to-end outcome, while normalized subgoal progress measures the deepest semantic milestone reached during execution.For a task with K ordered subgoals, m_i is the number sequentially completed in episode i.Source: E. Stage-Aware Robustness Measures · p. 5
  • Reach rate reports the fraction of rollouts that reach at least milestone k, while the first incomplete subgoal localizes the first unreached milestone.The first-incomplete-subgoal distribution is operational localization, not the causal instant of failure; episodes completing all subgoals are not assigned one.Source: E. Stage-Aware Robustness Measures · p. 5

IV. EXPERIMENTS

In four clean-qualified tasks, robustness responses differed across perturbation factors and tasks, and increasing severity did not always produce monotonic degradation. The evaluation used matched clean-perturbed rollouts and found that aggregate outcomes can conceal varied task responses.

A. Experimental Setup

  • The formal evaluation used GR00T N1.5 on four clean-qualified tasks, with 20 base episodes per task and 1,040 total rollouts across clean and shifted conditions.Each episode had one clean evaluation and twelve shifted conditions; paired degradation used equal-weight task-level macro averages and pair-level bootstrap confidence intervals.Source: A. Experimental Setup · p. 5
  • Two additional VLA checkpoints did not meet the predefined competence gate, so formal perturbation results are restricted to GR00T N1.5 on the clean-qualified task suite.SmolVLA RoboCasa had 1/40 clean successes and π0 RoboCasa365 Human300 had 2/40.Source: A. Experimental Setup · p. 5

B. End-to-End Robustness under Controlled Distribution Shifts

  • Gaussian blur degradation rose from ΔSR 0.188 at L1 to 0.513 at L2 and 0.738 at L3, while camera orientation peaked at L2 and illumination showed small net changes.Figure 4 summarizes macro degradation and Figure 5 shows task-level variation; severity levels are ordinal within each factor, not equivalent across factors.Source: Figure 4 · p. 5, B. End-to-End Robustness under… · p. 6, Figure 5 · p. 6

C. Counterfactual Sensitivity Beyond Aggregate Success

Matched outcomes can change in opposite directions and cancel in aggregate success-rate comparisons. Stage-aware analysis shows that similar aggregate degradation can still reflect different failure depths, with patterns varying by perturbation and severity.

C. Counterfactual Sensitivity Beyond Aggregate Success

  • For camera orientation at L1, ΔSR = 0.000 despite 8 of 80 matched episodes changing outcome; Gaussian blur at L3 yields ΔSR = 0.738 from 59 S →F transitions and no F →S transitions.Across the conditions, opposing S →F and F →S transitions can partially or completely cancel in the aggregate.Source: C. Counterfactual Sensitivity Beyond… · pp. 6–7, Table III · p. 7

D. Semantic Depth of Perturbation-Induced Failures

  • Camera orientation at L2 and Gaussian blur at L1 both yield ΔSR = 0.188, but first-stage failure rates are 19.0% and 36.4%, respectively.Mean normalized progress is 0.437 for camera orientation and 0.356 for Gaussian blur.Source: D. Semantic Depth of… · p. 7, Figure 6 · p. 7
  • For Gaussian blur, S →F events rise from 22 at L1 to 59 at L3 as mean normalized progress falls from 0.356 to 0.000 and pre-milestone failures rise from 36.4% to 100%.At L3, all 59 perturbation-induced failures terminate before the first tracked semantic milestone.Source: D. Semantic Depth of… · p. 7, Figure 6 · p. 7
  • Base initialization shows a nonuniform tendency toward earlier failures, camera failures span multiple depths, and illumination has comparatively few S →F events without a stable localization pattern.The localization response remains perturbation dependent rather than following a universal severity trend.Source: D. Semantic Depth of… · p. 7, Figure 6 · p. 7, Figure 7 · p. 8

E. Scope and Limitations

The evaluation is limited to a clean-qualified four-task RoboCasa suite with GR00T N1.5, simulator-grounded diagnostics, and simulation-only experiments. The authors conclude that aggregate success can mask episode sensitivity and failure depth, motivating execution-level diagnostics.

E. Scope and Limitations

  • Formal perturbation results cover GR00T N1.5 on four long-horizon RoboCasa tasks selected by a predefined clean-performance gate, so they do not establish cross-model generalization or broader task and mobility coverage.The simulator-state predicates are execution diagnostics rather than ground-truth stage labels, and transfer to real-robot deployment remains unvalidated.Source: E. Scope and Limitations · p. 8
  • The experiments are conducted in simulation, and whether the observed robustness patterns transfer to real-robot deployment remains to be validated.The authors identify broader task coverage and real-robot settings as directions for future work.Source: E. Scope and Limitations · p. 8

V. CONCLUSION

  • RoboCasa-Shift combines matched clean-perturbed evaluation with simulator-grounded stage-aware diagnostics; results show that aggregate success can mask episode sensitivity and failures at different semantic depths.The authors motivate moving beyond endpoint-only robustness reporting toward protocols that diagnose how shifts affect task execution.Source: §V · p. 8

VI. APPENDIX

The appendix describes how the formal task suite, perturbations, and severity levels were selected and checked. Screening produced four clean-qualified tasks, while audits checked perturbation validity independently of policy success.

A. Task Selection and Clean-Competence Screening

  • The preregistered 12+4 screening schedule produced the final four-task suite after only three of the first 12 candidates passed the clean gate and WashLettuce passed among the remaining four.The candidate pool contained 16 tasks and was frozen in advance.Source: A. Task Selection and Clean-Competence… · p. 9, Table IV · p. 9

1) Perturbation Implementation:

  • Each perturbation modifies only its targeted factor while preserving task semantics, object placement, and language instructions.The factors alter lighting, external camera pose, robot base initialization, or visual sharpness; wrist-mounted views remain unchanged for camera-orientation shifts.Source: 1) Perturbation Implementation · p. 9

2) Severity Calibration:

  • All 156 reset/render records passed the model-free severity validity audit, which checked initialization, observation integrity, pre-perturbation consistency, and intended factor-specific changes.The audit used three fixed base episodes per task across the clean condition and twelve perturbation conditions; it established audited reset and rendering validity, not task completion under perturbation.Source: 2) Severity Calibration · pp. 9–10, Figure 8 · p. 10

1) Pair Construction:

Paired rollouts preserve task conditions while changing only the target perturbation, and audits check both simulator-state and policy-side consistency. The appendix also specifies and validates the task-specific semantic milestones used by StageTracker.

1) Pair Construction:

  • Each perturbed rollout starts from its corresponding clean episode configuration, preserving matched task conditions and policy-side stochasticity while applying the target perturbation before execution.The task objective and evaluation protocol remain unchanged, enabling analysis of perturbation-associated outcome changes.Source: 1) Pair Construction · p. 10

2) Pairing Integrity Audit:

  • Pairing integrity checks compare pre-injection simulator-state fingerprints and restore the same policy checkpoint and random-number-generator state before paired execution.The final audit verifies the required consistency conditions for all paired executions.Source: 2) Pairing Integrity Audit · p. 10, Table V · p. 10

1) Task-Specific Subgoal Specifications:

  • The task-specific semantic milestones and their simulator-state completion criteria are listed for the four formal evaluation tasks, with criteria frozen before formal perturbation evaluation.For WashLettuce, thresholds refer to the simulator washed time counter.Source: 1) Task-Specific Subgoal Specifications · p. 11, Table VI · p. 11

2) StageTracker Validation:

  • StageTracker validation replayed five successful target demonstrations per task; all 20 reached the specified milestones in order without predicate-evaluation errors.This establishes predicate consistency on the evaluated demonstrations, not agreement with ground-truth per-frame stage annotations.Source: 2) StageTracker Validation · p. 11

E. Implementation Details

The supplementary sections report macro-level robustness results, paired outcome transitions, and stage-aware failure patterns, alongside implementation details and validation of the formal evaluation.

E. Implementation Details

  • The formal evaluation passed all task-level validators across 1,040 rollouts, with no reported state-hash or seed mismatches, runtime or tracker errors, or unexpected truncations.The runtime used SDPA rather than the original FlashAttention path for the Blackwell GPU without modifying the policy checkpoint.Source: E. Implementation Details · p. 11

F. Additional End-to-End Robustness Results

G. Complete Counterfactual Transition Statistics

H. Additional Stage-Aware Failure Analysis

I. Cross-Model Clean-Competence Screening

The cross-model extension screened additional policies for clean competence, while qualitative matched rollouts illustrate opposing outcome changes under one camera condition.

I. Cross-Model Clean-Competence Screening

  • Neither SmolVLA nor π0 reached the clean-screen extension threshold on any of the four tasks, so no additional policy entered perturbation evaluation or enabled a cross-model robustness comparison.Source: Table XII · p. 12, §I · pp. 12–13
  • On a separate 24-task SmolVLA pool, only OpenStandMixerHead and WashLettuce achieved one success each of ten episodes; no task qualified for perturbation evaluation.Source: §I · p. 13

J. Additional Qualitative Counterfactual Examples

Related papers

Loading 2610.27861v1…