Source-linked AI summary
RoboRecover: Benchmarking Robot Policy Recovery under Execution Deviations
Yang Li, Chen Zhao, Zhuoran Wang, Jiankang Wang, Chao Shao, Yihan Lin, Haitao Shen, Jing Zhang
TL;DR
Robot-policy benchmarks often evaluate from predefined initial states, leaving recovery from interaction-induced deviations less examined. RoboRecover reconstructs intermediate states by replaying action prefixes and evaluates policies on the original task. Across RoboTwin and LIBERO, initial-state performance does not determine recovery performance, and recovery strength varies across scenarios.
Problem
Many robot-policy evaluations start from predefined initial states and report final success or failure, providing less coverage of the dynamic interaction process and recovery from its deviations.
Source: §1 · p. 1Method
RoboRecover evaluates policies from interaction-induced recovery states reconstructed by replaying action prefixes, with the original task instruction and goal retained.
Source: §1 · pp. 1–2, §3 · p. 3Results
Takeaways & Limitations
Recovery from execution-induced intermediate states is a distinct dimension of robot-policy evaluation, and a single overall score cannot fully describe policy behavior.
Source: §1 · p. 2, §6 · p. 7The benchmark currently covers two simulation platforms, and the studied recovery mechanisms are an initial demonstration rather than an exhaustive exploration of strategies.
Source: §6 · p. 7
Abstract
from arXiv · showhide
Robot-policy benchmarks increasingly cover diverse tasks and preset out-of-distribution conditions, but typically evaluate complete trajectories from predefined initial states. These evaluations often focus on the initialized scene and the final outcome, while paying less attention to the dynamic interaction process. During closed-loop execution, actions and contacts can alter object relations and task progress, producing off-nominal intermediate states that need recovery. Recovery requires a policy to infer how task progress has changed, correct the relevant relations, and continue the original goal. We introduce RoboRecover, a benchmark for robot policy recovery under execution deviations. RoboRecover selects deviation states from trajectories, reconstructs them by replaying action prefixes, and evaluates policies on the original task. RoboRecover contains 2,000 scenarios across RoboTwin and LIBERO, with 1,000 scenarios and a fixed 800/200 train/test split on each platform. Results show that initial-state performance does not determine recovery performance and policies exhibit different recovery strengths across scenarios. Using its training split, RoboRecover further supports study on recovery interventions. RoboRecover establishes recovery from execution-induced intermediate states as a distinct dimension of robot policy evaluation.
1 Introduction
Conventional evaluations start from predefined states and summarize final outcomes, while execution can create off-nominal intermediate states that require policies to reassess progress, repair relations, and continue the task. RoboRecover evaluates recovery from replayed interaction states and examines policy performance and interventions.
Motivation
- Unlike predefined-start evaluations, recovery evaluation begins at a selected point in interaction, where deviations can disrupt task progress or invalidate a completed subtask.Policies must reassess the state, restore relevant relations or contact, and continue the original task.Source: Figure 1 · p. 1, §1 · p. 1
Research questions
- The introduction frames three questions: how well policies recover, how recovery varies across stages, deviation types, and trajectory sources, and whether interventions improve recovery.These questions target interaction-induced states that predefined initial-state evaluations cannot answer.Source: §1 · p. 1
Recovery performance analysis
- On LIBERO, UnifoLM pairs 98.83% initial-state success with 48.00% recovery, while π0.5 pairs 94.17% initial-state success with the best recovery rate, 64.40%.Across RoboTwin and LIBERO, initial-state success does not determine recovery performance, and recovery varies by stage and deviation type.Source: §1 · p. 2
Benchmark and protocol
- RoboRecover replays an action prefix to reconstruct a recovery state, then tests whether a policy corrects the deviation and continues the original task.The benchmark includes 2,000 scenarios across RoboTwin and LIBERO, with 1,000 per platform and fixed 800/200 train/test splits.Source: §1 · pp. 1–2
Recovery mechanisms
- Reversion returns execution to an earlier, less-deviated state, while correction briefly uses a learned intervention before returning control to the unchanged base policy.Correction learns from successful trajectories generated by alternative policies on scenarios where the base policy performs poorly.Source: §1 · p. 2
2 Related Work
Existing manipulation benchmarks broaden task and preset variation, while recovery-focused work studies failures and predefined errors. RoboRecover focuses on intermediate states arising from dynamic interaction and evaluates multiple policies from the same recovery points.
2.1 General Manipulation Benchmarks
- General manipulation benchmarks vary tasks, scenes, embodiments, and preset conditions, but still begin evaluation from conditions specified before execution.RoboRecover instead tests whether a policy can continue the original task from off-nominal intermediate states formed after actions and contacts change the scene or task progress.Source: §2.1 · p. 2
2.2 Failure-Aware Recovery Benchmarks
- RoboRecover converts naturally occurring deviations into recovery scenarios, adds action-perturbed and human-constructed cases, and compares multiple policies across shared recovery points.Earlier failure and recovery work analyzes complete trajectories or predefined physical errors; RoboRecover centers errors emerging during dynamic interaction.Source: §2.2 · p. 2
3 Problem Formulation
The formulation defines recovery as continuing the original task from a recoverable off-nominal state produced during policy–environment interaction. Each scenario fixes the task, replay prefix, source, recovery stage, and deviation type.
Execution Deviation and Recoverability
- A recoverable off-nominal state preserves the original instruction and success criterion, with task completion still possible by another policy.An execution deviation occurs when prior actions or contacts disrupt nominal progress and produce an off-nominal intermediate state.Source: §3 · p. 2
Recovery Stage and Deviation Type
- Recovery Stage identifies the task phase to resume—Approach, Contact, Move, or Place—while Deviation Type identifies the error to correct.The four deviation types are Wrong Target, Wrong Pose, Stage Rollback, and Scene Change.Source: §3 · p. 3
Recovery Scenario and Evaluation
- A recovery scenario fixes the task, action prefix, construction source, stage, and deviation type; evaluation replays the prefix before the candidate policy resumes the original instruction.The policy receives up to the task’s maximum number of new interactions; replay actions do not count toward that limit.Source: §3 · p. 3
4 RoboRecover
RoboRecover builds replayable recovery scenarios on RoboTwin and LIBERO from dynamic interactions, annotates stable deviations by stage and type, and filters for difficulty and replay consistency.
4.1 Benchmark Overview
- RoboRecover contains 2,000 scenarios across RoboTwin and LIBERO, with 1,000 per platform and fixed 800/200 train/test splits; both test sets include nine Stage–Type combinations.The benchmark includes task and source-policy coverage alongside representative recovery points for the occupied groups.Source: §4.1 · p. 3, Figure 2 · p. 4
4.2 Scenario Sources and Construction
- Scenarios come from natural policy trajectories, action-perturbed trajectories, and human construction, with deviations produced through robot–environment interaction rather than direct simulator state editing.Action perturbations affect translational coordinates during the first 150 interaction steps; LIBERO human-constructed cases supplement rare but recoverable states.Source: §4.2 · p. 3
4.3 Stage–Type and Recovery Point Annotation
- Annotators select stable deviations that persist into sustained failure, label recovery points and Stage–Type categories together, and remove cases without majority agreement.Temporary deviations followed by restored nominal progress are discarded for natural and action-perturbed trajectories.Source: §4.3 · pp. 3–4
4.4 Scenario Filtering
- Candidates are screened for intermediate recovery difficulty and then replay-validated against simulator states and rendered observations.Difficulty filtering retains 2–18 successes among 20 trials; replay checks require robot and object position deviations below 2 mm and average SSIM above 0.99, with manual inspection as a final sanity check.Source: §4.4 · pp. 4–5
5 Experiments
The experiments compare initial-state and recovery performance across six policies per platform, then examine recovery by task stage, deviation type, and source policy. They also test whether reversion, cross-policy correction, and monitoring improve recovery.
5.1 Experimental Setup
- Six policies per platform are evaluated under matched initial-state and reconstructed recovery conditions, using the original task instruction and success criterion.The selected policies span different architectures, pretraining, and action-generation designs.Source: §5.2 · p. 5, §5.1 · p. 5
5.2 Initial-State and Recovery Performance
- 64.40% is the best LIBERO recovery rate, achieved by π0.5, while all six policies score at least 85.17% from predefined initial states.UnifoLM drops from 98.83% initial-state success to 48.00% recovery.Source: §5.2 · p. 5
5.3 Stage–Type Recovery Differences
- RoboTwin LingBot-VA reaches 86.36% on Approach–Wrong Pose but 21.74% on Place–Wrong Pose, where Fast-WAM reaches 65.22%.The two Wrong Pose groups involve different task stages and corrective behavior.Source: §5.3 · p. 5, Figure 4 · p. 6
- RoboTwin’s highest Overall RSR is LingBot-VA’s 59.50%, while Fast-WAM leads Macro-9 RSR at 55.62%; π0.5 leads both LIBERO metrics.Overall RSR weights scenarios by their frequency, whereas Macro-9 gives equal weight to the nine Stage–Type groups.Source: §5.3 · pp. 5–6
5.4 Cross-Policy Recovery Differences
- The retrospective oracle reaches 95.63% on RoboTwin and 91.80% on LIBERO, exceeding the best fixed policy on each platform.The gap reflects that evaluated policies recover different sets of scenarios.Source: §5.4 · p. 6
5.5 Benchmark-Guided Recovery Interventions
- Reversion starts 30 steps before the recovery point; Correction uses successful alternative-policy trajectories when π0.5 recovery is below 40% and an alternative exceeds 60%.Correction temporarily controls the robot before returning control to the unchanged base policy.Source: §5.5 · p. 6
5.4 Cross-Policy Recovery Differences
- On RoboTwin, source policies recover 15.24–19.82% on their own scenarios, while alternative policies reach 58.06–70.27%.Figure 5 compares source-policy retries with cross-policy recovery on Natural deviations.Source: §5.4 · p. 6, Figure 5 · p. 7
5.5 Benchmark-Guided Recovery Interventions
- 77.0% for Monitor+Reversion and 75.5% for Monitor+Correction exceed their corresponding unmonitored interventions.The monitor selectively triggers interventions on the fixed test split.Source: Table 1 · p. 7, §5.5 · p. 7
6 Conclusion
RoboRecover evaluates policies from replayable intermediate states produced during execution, making recovery a distinct evaluation dimension. Its results show that initial-state success does not guarantee recovery, and its training split supports targeted interventions; the benchmark currently covers two simulation platforms.
6 Conclusion
- RoboRecover evaluates robot policies from replayable intermediate states created during execution, separating recovery capability from initial-state task completion.Action-prefix replay reconstructs the state so policies can continue the original task.Source: §6 · p. 7
- Across RoboTwin and LIBERO, recovery performance changes substantially, and the strongest policy varies across deviation sources, types, and task stages.High initial-state success does not guarantee that a policy can interpret off-nominal states and continue the original task.Source: §6 · p. 7
- Reversion and short corrections learned from successful cross-policy trajectories demonstrate two ways to improve recovery using the benchmark’s training split.Reversion returns execution to an earlier state with less accumulated deviation.Source: §6 · p. 7
- The benchmark currently covers two simulation platforms; extending it to additional task suites, robot embodiments, and real-world systems would broaden its scope.The studied recovery mechanisms are an initial demonstration, not an exhaustive exploration.Source: §6 · p. 7
A Benchmark Construction and Composition
This appendix section adds annotation, validation, and corpus statistics to the main paper’s construction and taxonomy account.
A Benchmark Construction and Composition
- The section reports additional annotation, validation, and corpus statistics beyond the main paper’s account of construction sources, taxonomy, and recovery-point criteria.Figure 6 pairs RoboTwin and LIBERO examples for each occupied Stage–Type group, showing observations before recovery inference.Source: A Benchmark Construction and Composition · p. 10, §A.1 · p. 10
A.1 Annotation and Recovery-Point Selection
- Three annotators select recovery points and assign labels, resolving disagreements by majority vote; pre-resolution agreement was 92.17% for Recovery Stage and 82.82% for Deviation Type.Scenarios without a stable majority are removed.Source: §A.1 · p. 10
A.2 Difficulty Screening and Replay Validation
- Candidates received 20 screening trials; those with 2–18 successes were retained, replay validation passed 97.32%, and inconsistent reconstructions or invalid cases were removed.For each retained candidate, three screening inference trajectories were replayed and compared with the originals around the recovery point.Source: §A.2 · p. 10
A.3 Corpus Composition
- Each platform contains 1,000 scenarios with a fixed 800/200 train/test split; RoboTwin uses Natural scenarios, while LIBERO includes Natural, Action-Perturbed, and Human-Constructed scenarios.The splits share no sample identifier, JSON path, or exact instruction group on RoboTwin, and no Data-Path or complete episode key on LIBERO.Source: §A.3 · p. 10
B Evaluation Implementation and Statistical Protocol
This section defines complementary recovery-rate summaries and describes the statistical protocol and materials reported for evaluation.
B Evaluation Implementation and Statistical Protocol
- Macro-9 RSR gives equal weight to the nine Stage–Type group rates, while Overall RSR preserves the test set’s empirical group frequencies.Macro-9 is used as a category-balanced diagnostic because several groups contain relatively few scenarios.Source: B Evaluation Implementation and… · p. 10
- RoboTwin confidence intervals use 10,000 trajectory-cluster bootstrap draws, while LIBERO intervals use 10,000 scenario-bootstrap draws; paired contrasts jointly resample both policies’ outcomes.Pairwise intervals are not adjusted for multiple comparisons.Source: B Evaluation Implementation and… · pp. 10–11
- Table 3 summarizes corpus composition, and Figure 6 pairs representative recovery observations from RoboTwin and LIBERO across occupied Stage–Type groups.The Table 3 caption identifies Action-Perturbed scenarios as “Pert.” and uses dashes for construction sources not used on RoboTwin.Source: Table 3 · p. 10, Figure 6 · p. 11
C Complete Recovery Results
The section reports complete fixed-test-split estimates and intervals for initial-state and recovery evaluation, plus paired policy contrasts.
C Complete Recovery Results
- Table 4 reports point estimates and 95% confidence intervals for all 12 platform-policy evaluations on the fixed test splits.RoboTwin intervals use trajectory-cluster bootstrap, while LIBERO intervals use scenario bootstrap.Source: C Complete Recovery Results · p. 11
- Table 5 reports Macro-9 RSR with equal weight for the nine Stage–Type groups, while Table 4 reports scenario-weighted Overall RSR.Source: C Complete Recovery Results · p. 11
C.1 Paired Policy Contrasts
- Table 6 gives paired policy contrasts as policy A minus policy B, so positive values favor A; the intervals are not adjusted for multiple comparisons.Source: §C.1 · p. 11
D Comparison with LIBERO-Plus Preset-OOD Evaluation
LIBERO-Plus perturbs conditions before a rollout, while RoboRecover evaluates continuation from states altered by prior interaction; the evaluations expose different failure conditions.
D Comparison with LIBERO-Plus Preset-OOD Evaluation
- Fast-WAM scores 51.5% on LIBERO-Plus, the lowest among four compared policies, but 54.00% on RoboRecover, the second-highest; Cosmos-Policy shows the opposite tendency.The reported ordering remains unchanged when the π0 JAX rerun is used instead of its original LIBERO-Plus value.Source: D Comparison with LIBERO-Plus… · p. 12
- Preset-OOD scores and recovery scores are not interchangeable, and ranking differences do not identify which model components cause the observed behaviors.Checkpoints, training data, and evaluation implementations also differ.Source: D Comparison with LIBERO-Plus… · p. 12
E Stage–Type and Source-Conditioned Results
- The section reports test-set results conditioned on Recovery Stage, Deviation Type, construction source, task suite, and Natural source policy; very small Stage–Type groups are descriptive.Source: E Stage–Type and Source-Conditioned… · p. 12
E.1 Construction-Source and Task-Suite Results
- Construction-source results vary by policy: π0.5 is strongest on Human-Constructed and Action-Perturbed scenarios, while Cosmos-Policy is strongest on Natural scenarios.Task-suite results likewise show no common ordering across policies; comparisons should be read with the group sizes.Source: §E.1 · p. 13
E.2 Natural Source-Policy Results
Natural source-policy comparisons show that the policy causing a deviation is not necessarily best positioned to recover from it. The section also reports retrospective per-scenario results and exploratory monitor-triggered intervention studies.
E.2 Natural Source-Policy Results
- The policy that produced a deviation is not necessarily the policy with the highest recovery rate from it, as all policies are tested on the same reconstructed scenarios.Table 8 reports complete RoboTwin recovery rates across Stage–Type groups; its two Scene Change groups contain only four and two scenarios.Source: §E.2 · p. 13, Table 8 · p. 13
- The retrospective per-scenario oracle reaches 95.63% on RoboTwin and 91.80% on LIBERO, versus best fixed-policy rates of 59.50% and 64.40%, respectively.The differences are 36.13 and 27.40 points; this test-outcome selection is not deployable and shows that policies succeed on different scenario sets.Source: §E.2 · p. 13
F Recovery Mechanism Studies
- The intervention studies use the training split to explore recovery mechanisms and are not part of the primary direct-recovery leaderboard.The primary benchmark evaluates fixed policies without additional assistance.Source: F Recovery Mechanism Studies · p. 13
F.1 Monitor Training and Triggering
- RoboTwin monitor data split 800 scenario IDs into 720 training and 80 validation IDs; training balances pre-recovery label-0 frames against recovery-and-later label-1 frames.Validation includes balanced and natural frame-sampling views, and all frames from an ID stay in one split.Source: §F.1 · pp. 13–14
- Monitoring starts at the recovery point, checks each environment step, and triggers once on any positive prediction; episode-level trigger rates were 76.0% for X-VLA and 83.0% for LingBot-VLA.These rates count episodes with at least one intervention activation.Source: §F.1 · p. 14
F.2 Reversion Results
Reversion was evaluated in fixed and monitor-triggered settings, alongside a small exploratory LIBERO human-intervention study and a correction-training setup.
F.2 Reversion Results
- Fixed Reversion with X-VLA reaches 54.5% recovery on RoboTwin, while monitor-triggered Reversion reaches 59.0%.These results use one base policy and one Reversion configuration, so they are an exploratory demonstration rather than a general mechanism comparison.Source: §F.2 · p. 14
F.3 LIBERO Monitor and Human Intervention
- On 50 LIBERO scenarios, direct human control succeeds on 82%, while monitor-triggered human control succeeds on 88%.Because the study uses only 50 scenarios and human control, it is reported as an exploratory reference rather than a policy benchmark.Source: §F.3 · p. 14
F.4 Correction Data and Training
- Correction training retains donor trajectories succeeding in at least three of four trials, then deduplicates records; current LIBERO training initializes from official π0.5 and uses a 3,000-step schedule.The training setup uses an action horizon of 10, batch size 256, learning rate 5 × 10−5, and 200 warmup steps.Source: §F.4 · pp. 14–15