Source-linked AI summary
LIBERO-RECOVER: Beyond Task Success Towards Failure Recovery in Robotic Manipulation Models
Lin Liu, Zhicheng Bao, Lu Zhang, Ziying Song, Wu Yang, Shuai Tao, Wulong Liu, Huchuan Lu
TL;DR
Existing robotic manipulation benchmarks can show near-perfect one-shot success while leaving recovery from real execution failures largely unmeasured. LIBERO-Recover addresses this gap by collecting naturally occurring failures and evaluating recovery across structured capabilities and scenarios. Experiments reveal a substantial gap between standard task success and failure recovery, while temporal context and finer-grained execution can improve recovery and recovery training transfers only limitedly to naturally arising failures.
Problem
Existing benchmarks mainly evaluate task completion from predefined initial states, leaving recognition and recovery from real execution failures largely unexplored.
Method
LIBERO-Recover collects naturally occurring failures from embodied model executions and constructs recovery scenarios evaluating state understanding, action adaptation, interaction reasoning, and failure recovery.
Results
Experiments reveal a substantial gap between standard task success and failure recovery, with performance degrading as recovery requires deeper state reasoning.
Takeaways & Limitations
Temporal context and finer-grained action execution improve recovery, while recovery training provides limited transfer to failures encountered during standard task execution.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) or World Action (WAM) models have recently demonstrated remarkable performance in robotic manipulation. On LIBERO, SOTA method have achieved nearly 100\% success rates, seemingly suggesting that the models are ready for deployment in real world. However, near perfect performance on existing benchmarks can be misleading: success under ideal conditions does not imply real world robustness. Existing benchmarks primarily evaluate task completion from predefined initial states, while real world interactions inevitably involve failures such as failed grasps, collisions, and unintended object movements. A robot must therefore not only execute tasks successfully, but also recognize and recover from failures to continue the task. Yet this capability remains largely unmeasured, revealing a critical gap between benchmark performance and real world reliability. To address this gap, we introduce LIBERO-Recover Benchmark, a large scale benchmark for failure recovery in robotic manipulation. Built upon LIBERO, we collect real execution failures from SOTA embodied models and construct 1,000+ scenarios across four recovery levels: (1) Action Retry, (2) Action Adaptation, (3) Object State Recovery, and (4) Environmental Recovery. We evaluate four core capabilities: spatial understanding, object structure reasoning, interaction understanding, and topological reasoning. As the first large-scale benchmark for embodied failure recovery, LIBERO-Recover shifts evaluation from \emph{Can the robot succeed?''} to \emph{Can the robot recover after failure?''}, promoting robust and generalizable embodied agents. The project will be avaible in \textcolor{blue}{https://liulin815.github.io/LIBERO-Recovery/}.
1 INTRODUCTION
Existing manipulation benchmarks report near-perfect one-shot success under predefined initial states, but real-world failures and recovery remain insufficiently evaluated. Recent benchmark extensions add difficulty or robustness dimensions without fully addressing post-failure recovery.
- 1 INTRODUCTION: Near-perfect LIBERO success rates can overstate robotic manipulation robustness because evaluation assumes idealized, single-attempt execution from predefined initial states.Real interactions involve failures such as unsuccessful grasps and unintended state changes.
- 1 INTRODUCTION: Recent benchmarks introduce more complex task variations and robustness evaluations, but their extensions do not fully resolve the gap in failure recovery assessment.The cited extensions include LIBERO-Pro, LIBERO-Plus, and LIBERO-X.
A Comprehensive Failure Recovery Benchmark Systematic Evaluation of Recovery Capability
LIBERO-Recover evaluates recovery from naturally occurring embodied-model failures rather than manually perturbed states. It covers multiple recovery capabilities and reveals substantial limitations in current models.
- A Comprehensive Failure Recovery Benchmark Systematic Evaluation of Recovery Capability: LIBERO-Recover collects failure scenarios from model execution and expert recovery trajectories obtained through human teleoperation.The overview contrasts naturally collected failures with expert recovery behavior.
- A Comprehensive Failure Recovery Benchmark Systematic Evaluation of Recovery Capability: Existing benchmarks leave failure recognition, state recovery, and post-failure replanning largely unexplored, motivating a dedicated recovery benchmark.Prior evaluations mainly use predefined perturbations or challenging conditions.
- A Comprehensive Failure Recovery Benchmark Systematic Evaluation of Recovery Capability: LIBERO-Recover covers 4 task categories, 130 subtasks, and 16 evaluation dimensions across four core recovery capabilities.The benchmark assesses embodiment understanding, object structure reasoning, topological interaction reasoning, and failure recovery.
- A Comprehensive Failure Recovery Benchmark Systematic Evaluation of Recovery Capability: Leading VLA models show a substantial gap between standard task execution and failure recovery, particularly in failure-state understanding and recovery planning.The reported evaluation identifies critical limitations in both capabilities.
2 RELATED WORK
VLA and WAM methods improve general-purpose manipulation through multimodal modeling and action-conditioned prediction, but systematic failure modeling and recovery remain underexplored. LIBERO-Recover addresses this gap with naturally collected failures and progressively challenging recovery scenarios.
- 2 RELATED WORK: VLA methods combine vision, language, and action understanding with large-scale datasets to generalize across tasks, objects, and embodiments.Representative methods include RT-1, RT-2, OpenVLA, π0, π0.5, RoboMamba, and GR00T.
- 2 RELATED WORK: WAMs model actions and future states to provide foresight into consequences and improve long-horizon planning.The related work discusses UniVLA, Video Policy, UWM, FLARE, and Cosmos-Policy.
- 2 RELATED WORK: Despite these advances, VLA and WAM research largely emphasizes successful execution rather than failures, unexpected state transitions, and post-failure recovery.Systematic failure modeling remains largely unexplored in existing methods.
- 2 RELATED WORK: LIBERO-Recover collects naturally occurring failures and resulting state transitions to evaluate state understanding, action adaptation, interaction reasoning, and failure recovery.Its scenarios become progressively more challenging as recovery demands deeper reasoning about disrupted states.
3 LIBERO-RECOVER BENCHMARK
LIBERO-Recover reframes manipulation evaluation around recoverable failures, defining failure states, recovery policies, and four progressively demanding difficulty levels. It constructs scenarios from actual embodied-model failures and organizes them across task suites and evaluation dimensions.
- Problem formulation: Conventional evaluation measures completing a task from a valid initial configuration without explicitly modeling what happens after execution failure.Failure recovery instead requires adapting to an altered configuration and generating actions that restore the necessary state before continuing.
- Problem formulation: A failure state violates the original goal but remains recoverable through subsequent interaction, requiring a recovery policy rather than continuation of the original action sequence.State changes that preserve task executability are treated as normal variations, not failures.
- Failure taxonomy and difficulty levels: The four-level taxonomy progresses from retrying a failed action, through action adaptation and object-state recovery, to environmental recovery involving interaction topology.Higher levels require progressively deeper state reasoning, culminating in restoring environmental states that block task execution.
- Scenario definition: Each recovery scenario is grounded in an actual embodied-model failure trajectory and its resulting failure state, rather than a manually perturbed task state.The scenario records the instruction, original state, failure trajectory, resulting state, original goal, and required recovery behavior.
- Construction pipeline and statistics: The benchmark construction pipeline executes source tasks, localizes failures temporally, and characterizes their consequences and recovery difficulty.The benchmark contains 2178 scenarios from 130 subtasks across four task categories, four recovery levels, and 16 evaluation dimensions; higher-difficulty failures occur less frequently and are not manually rebalanced.
- Construction pipeline and statistics: Post-Task-Failure cases become more common at higher recovery difficulties, as repeated retries can produce progressively more severe consequences.This distribution emerges from embodied-model executions and is preserved rather than manually rebalanced.
4 DATASETS AND EVALUATION METRICS
LIBERO-Recover evaluates failure recovery with metrics that measure task completion after failure, degradation in execution capability, and consistency across failure states. The metrics also separate performance by four recovery difficulty levels.
- Evaluation Metrics: Recovery Success Rate (RSR) measures whether a policy completes the original task after entering a failure state.Higher RSR indicates stronger failure recovery capability.
- Evaluation Metrics: RSRL1, RSRL2, RSRL3, and RSRL4 report recovery success separately across the four recovery levels.These level-specific metrics characterize the effect of recovery difficulty.
- Evaluation Metrics: Recovery Degradation (RD) compares execution performance before and after failure across Pre-Failure, During-Failure, and Post-Failure phases.Lower RD indicates better retention of execution capability, while higher RD indicates greater degradation.
- Evaluation Metrics: Recovery Consistency (RC) measures stability across failure states within a task using the standard deviation of task-level recovery success rates.Higher RC indicates more consistent recovery across failure states.
5 EXPERIMENTS
Experiments show that standard task success poorly predicts failure recovery: performance drops sharply under real failures, especially when recovery requires state reasoning. Finer-grained action execution and temporal context help, but recovery training transfers weakly to naturally occurring failures.
- Experimental setup: Six representative VLA and WAM models are evaluated using official configurations on LIBERO and LIBERO-Recover.VLA methods encode visual observations and language instructions with VLM backbones, whereas WAM methods generate latent actions followed by an action head.
- Multi-level evaluation: All models suffer over 50% performance drops under naturally occurring failures, and standard benchmark rankings do not reliably transfer to recovery.Wan2-Policy leads GR00T-N1.5 by +14.40% on LIBERO-100 but falls -5.0% behind it under corresponding failures.
- Multi-level evaluation: Recovery success consistently decreases from L1/L2 action correction to L3/L4 state recovery, exposing a major gap in reasoning about object states, spatial relations, and interaction dependencies.The L2-to-L3 gap marks a transition from adapting immediate actions to recovering the underlying task state with sufficient scene memory.
- Temporal manifestation of failures: Recovery success drops from 15.0% to 6.7% on LIBERO-Spatial, 35.5% to 4.0% on LIBERO-Goal, 38.3% to 5.0% on LIBERO-Object, and 26.2% to 0.3% on LIBERO-100 after failures.The corresponding recovery degradation rates are 54.0%, 82.0%, 87.0%, and 100.0%, respectively; performance declines further after repeated failures.
- Recovery factors: WAM policies achieve higher Recovery Consistency, with Cosmos-Predict2-Policy at 0.8837 and Wan2-Policy at 0.8698, while smaller action chunks improve recovery through more frequent feedback.Longer chunks commit the robot to more actions before policy updates, making correction harder; smaller chunks support finer-grained closed-loop control.
- Recovery factors: Mixed recovery training improves LIBERO-Recover but transfers little or negatively to standard LIBERO, while temporal context improves recovery by providing a reference for detecting state deviations.For OpenVLA-OFT, recovery rises from 20.8% to 25.4% while average LIBERO success falls from 97.1% to 96.6%; temporal context raises average recovery from 20.8% to 26.8%.
- State distribution: LIBERO-Recover states are more concentrated around critical grasping, placing, and object-interaction regions than standard demonstrations.This distribution difference indicates that recovery data emphasizes underrepresented state-transition regions.
6 CONCLUSION
The paper introduces LIBERO-Recover to evaluate whether robotic manipulation models can understand disrupted states, adapt actions, and complete tasks after failure. Results reveal a substantial recovery gap, with finer-grained execution and temporal context helping while current recovery training transfers weakly to ordinary execution failures.
- Conclusion: LIBERO-Recover evaluates failure recovery using naturally occurring failures from embodied model executions.It tests understanding of disrupted states, action adaptation, and task completion after failure.
- Conclusion: Experiments reveal a substantial gap between standard task success and failure recovery, worsening as recovery requires deeper state reasoning.Finer-grained action execution and temporal context improve recovery, but current recovery training provides limited transfer to failures during standard execution.
A APPENDIX
No substantive appendix content is provided.
- Appendix: The appendix passage only indicates that additional sections may be included.