PaperlayerView exact revision on arXiv

Source-linked AI summary

Recova: Agent-Guided Failure Recovery for Autonomous Robotic Manipulation

Isabella Liu, An-Chieh Cheng, Johan Bjorck, Zhiding Yu, Hongxu Yin, Jan Kautz, Linxi Fan, Yuke Zhu, Sifei Liu

arXiv:2610.01178v1cs.RO

TL;DR

Manipulation failures can leave scenes that task policies cannot continue from, while learning corrective behavior on hardware is costly. Recova develops task and recovery capabilities in a reconstructed digital twin and refines them through real-robot deployment. It outperforms the strongest baselines across the evaluated simulation benchmarks, improves real-robot success with recovery skills, and reduces observed human intervention over collection rounds.

  • Problem

    Most VLA training data contain successful trajectories from standard initial states and provide little guidance for restoring failed scenes, while hardware exploration of failures is costly and risks damaging objects.

    Source: §1 · p. 2

  • Method

    Recova develops task and recovery behaviors in a reconstructed digital twin, then monitors real-robot execution, invokes and verifies recovery, and uses deployment experience to refine separate policies.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    78.8% mean success across six LIBERO-Pro settings and 64.9% across four MolmoSpaces categories exceed the strongest baselines; on real robots, recovery skills raise mean success from 77.5% to 87.5%.

    Source: §4.1 · p. 8, §4.2 · p. 9

  • Takeaways & Limitations

    The results show scene recovery as a learnable capability that complements task execution, with failures converted into reusable skills and human intervention declining over four rounds on one task.

    Source: §4.2 · p. 9, §5 · p. 10

Abstract

from arXiv · show

Manipulation failures can leave scenes in states from which a task policy cannot recover. Learning corrective behaviors requires scalable failure exploration and physical grounding. We present Recova, an agent-guided framework that jointly develops task execution and recovery in a reconstructed digital twin, then verifies and refines both through real-world experience. In the twin, the agent diagnoses failures, tests corrective programs, and collects successful task and recovery rollouts for separate policies. During deployment, it monitors progress, invokes a learned or programmatic recovery, verifies scene restoration, and resumes execution. When no suitable recovery is available, a human demonstration resolves the failure and enters the learning loop, allowing the system to expand its recovery capabilities. Physical rollouts and human demonstrations are routed to the corresponding policy for DAgger training. Across six LIBERO-Pro settings and four MolmoSpaces categories, Recova achieves 78.8% and 64.9% mean success, compared with 71.7% and 38.0% for the strongest baselines. With parallel collection across four real-robot workstations, DAgger fine-tuning raises mean success from 23.8% to 77.5%, and recovery skills further raise it to 87.5%. Over four collection rounds on one task, observed human intervention falls from 87.5% to 0%. Together, these results show how agent-guided recovery turns failures into reusable capabilities, improving robustness while progressively reducing human intervention. Project page: https://www.liuisabella.com/Recova

1. Introduction

Recova addresses the lack of recovery guidance in successful-trajectory training by developing task execution and scene recovery together in a reconstructed digital twin, then refining them with real-world experience. Its evaluations report higher benchmark success than the strongest baselines, improved real-robot success with recovery skills, and declining human intervention over repeated collection rounds.

1. Introduction

  • Most VLA training data contain successful trajectories from standard initial states, leaving little guidance for restoring failed scenes or continuing after a failed attempt.Simply retrying the task policy may leave the obstruction unresolved, motivating recovery capability that can expand as new failures arise.Source: §1 · p. 2
  • Learning recovery directly on hardware is costly and risks damaging objects, so Recova discovers corrections in simulation and verifies and refines them through physical experience.A parallel collection harness runs the deployment loop across four robot workstations.Source: §1 · p. 2
  • Recova develops task execution and scene recovery in a reconstructed digital twin, then improves both through real-world experience using separate task and recovery policies.Successful rollouts initialize the policies, while reusable recovery strategies are captured as programs that compose perception and motion primitives.Source: §1 · p. 2
  • During deployment, Recova monitors progress, invokes and verifies learned or programmatic recovery, and requests human assistance when no suitable recovery succeeds.Task successes and task demonstrations improve the task policy, while human recovery demonstrations improve the recovery policy; new failure modes guide additional skills.Source: §1 · p. 2
  • Recova exceeds the strongest baseline in the comparison by 7.1 percentage points on LIBERO-Pro and 26.9 points on MolmoSpaces; recovery skills add 10 points on real-robot tasks, reaching 87.5% mean success.Source: §1 · p. 3

2. Related Work

Prior work addresses manipulation planning, failure analysis, recovery, and learning from deployment through varied tools and policies. Recova’s related-work positioning is to coordinate learned and programmed recovery around a task policy, with demonstrations and deployment data routed to the policy each can teach.

Robot foundation models and agents

  • Robot foundation models and agents support generalist manipulation, planning, program generation, and reusable skills; Recova coordinates learned and programmed recoveries around a task policy.Examples include CaP-X, RATs, and ASPIRE, which benchmark coding agents or develop and refine reusable programs.Source: §2 · p. 3

Failure detection and recovery

  • Prior failure-recovery work analyzes or predicts failures, steers actions, or learns separate recovery controllers, while Recova combines scene reconstruction, restoration checks, learned and programmed executors, and demonstrations.The related work also notes benchmarks for perturbation robustness and recovery from failed executions.Source: §2 · p. 3

Learning from deployment and simulation

  • Deployment-learning approaches use expert labels, human interventions, or autonomously collected experience, while simulation pipelines reconstruct scenes or synthesize tasks and demonstrations.Recova combines simulation-initialized task and recovery behaviors with deployment data routed to the policy each example teaches, requesting human input at failures rather than every visited state.Source: §2 · p. 3
  • Figure 2 depicts Recova’s combined simulation and real-robot loop, including separate policy updates and skill-library expansion from new failures.The inset tracks human intervention across DAgger rounds.Source: Figure 2 · p. 4

3. Method

Recova separates task completion from scene recovery and connects them through an agent that develops skills in a digital twin, monitors and verifies recovery on real robots, and routes deployment experience to the appropriate policy.

3.1. Task Execution and Scene Recovery

  • Task execution pursues the instruction’s completion condition, while recovery targets the failure blocking progress so the task can resume or be completed through corrective action.A recovery skill is conditioned on a recovery instruction rather than the task instruction, allowing it to be reused wherever the same failure recurs.Source: §3.1 · p. 4

3.2. Simulation in a Digital Twin

  • The coding agent reconstructs a workstation digital twin from camera observations, robot trajectories, and available calibration, fitting object poses and contact parameters against recorded views.In the implementation, the twin is a MuJoCo scene used to explore task and failure modes without hardware resets or object-damage risk.Source: §3.2 · p. 5
  • In the twin, successful task attempts form task data, while diagnosed failures produce named recovery skills, recovery programs, and corrective rollouts for separate policy training.Recovery skills can be restorative, enabling, or adaptive, and the agent uses a vision-language model to assess whether the scene is ready for the task policy to resume.Source: §3.2 · p. 5, §3.1 · p. 5

3.3. Real-Robot Rollout Loop

  • The real-robot loop monitors completion, detects disturbance and names a recovery instruction, then selects task or recovery demonstrations or runs a trained recovery within a fixed time budget.After autonomous recovery, a restoration query checks that the scene is workable—not necessarily in its exact original configuration—before the next task attempt.Source: §3.3 · p. 5

3.4. Learning Across DAgger Rounds

  • Across DAgger rounds, successful task rollouts and human task demonstrations update task data, while human recovery demonstrations and verified autonomous recoveries update recovery data.New failure types can seed a new recovery instruction with a human demonstration, guide program development in the twin, and later enter the recovery policy’s registry.Source: §3.4 · pp. 6–7

4. Experiments

Recova is evaluated in simulation and on real robots for task success, recovery behavior, and human intervention. It combines task policies with programmatic or learned recovery skills across benchmark variations and four real-robot tasks.

4.1. Simulation Benchmarks and Comparisons

  • The simulation tests six LIBERO-Pro settings spanning position swaps and instruction changes, plus four MolmoSpaces categories across diverse simulated scenes.Real-robot evaluation covered four bimanual tasks, with 20 trials per configuration for the base policy, DAgger fine-tuning, and deployment with recovery skills.Source: §4.1 · p. 7, §4.2 · p. 8
  • Simulation pairs a MolmoAct2 task policy with programmatic recovery that takes over after execution-budget exhaustion or stalled progress in LIBERO-Pro.Development and evaluation use disjoint random seeds and held-out initial object configurations.Source: §4.1 · p. 8
  • 78.8% mean success across six LIBERO-Pro settings exceeded ASPIRE by 7.1 points, while 64.9% across four MolmoSpaces categories exceeded RATs by 26.9 points.Recova led in all four MolmoSpaces categories and in all four Goal and Spatial settings, although ASPIRE remained stronger on the Object suite.Source: §4.1 · p. 8

4.2. Real-Robot Experiments

  • 23.8% base task-policy mean success rose to 77.5% after DAgger and 87.5% with recovery skills, the best result on every real-robot task.Recovery added 15.0 points each on ring stacking and pencil-box packing, and 5.0 points on each mahjong task.Source: Table 3 · p. 9, §4.2 · p. 9
  • Human takeovers fell from 87.5% of episodes in the first collection round to 0% in the fourth on drawing a mahjong tile.The task policy completed 12.5% of episodes in the first round and 85.7% in the fourth.Source: §4.2 · p. 9
  • Real-robot recoveries address missed grasps, jammed or misplaced objects, and fallen tiles, with skills developed in the twin and learned policies executed on the robot.Examples include reinserting a pencil, sliding a detached tray into its sleeve, and standing a fallen tile upright.Source: Figure 5 · p. 9, §4.2 · p. 10

5. Conclusion

Recova treats task execution and scene recovery as distinct, coordinated capabilities, with an orchestration loop that monitors progress, invokes and verifies recovery, and routes unresolved failures into demonstrations. The conclusion identifies recovery as a learnable complement to task execution and points to automated skill registration and cross-task sharing as future work.

5. Conclusion

  • Recova outperforms code-as-policy agents and VLA policies on two simulation benchmarks, while recovery skills improve real-robot success beyond DAgger fine-tuning alone.The collection study also reports declining human intervention over four rounds on one task.Source: §5 · p. 10

B. Monitor Prompts and Example Verdicts

  • Monitor queries use four-camera composites, with intervention queries also receiving recent snapshots, prior verdicts, and known recovery skills.Across four-station sessions, median latency is 4.5 s for intervention, 5.0 s for completion, and 5.5 s for restoration queries.Source: B. Monitor Prompts and Example Verdicts · pp. 16–17
  • Figure 6 presents abridged monitor prompts, including each query’s attached text and images, the prompt itself, and its required JSON reply; symbols follow Alg. 1.Source: Figure 6 · p. 18

C.1. Hardware and Tasks

Real-robot collection uses four concurrently supervised workstations equipped for bimanual manipulation and multi-view observation. Episodes are time-limited, and the illustrated collection includes drawing sessions that continue until the hand holds eight tiles.

C.1. Hardware and Tasks

  • Each workstation has two six-joint YAM arms, four cameras, and leader arms for operator takeover.One operator supervises four stations arranged with different backgrounds and distractor objects.Source: §C.1 · p. 18
  • Episodes are limited to 3 minutes (5,400 control steps), and drawing sessions continue until the hand holds eight tiles.Rollouts start automatically during parallel collection.Source: §C.1 · p. 18
  • The recorded tile-drawing examples include recovery reuse, a task demonstration after a stall in an intact scene, restoration confirmation, and task-completion acceptance.Table 4 lists the four tasks and their success criteria.Source: Table 4 · p. 18, Figure 7 · p. 19

C.2. Real-Robot Monitor and Data Interface

The real-robot interface separates monitoring views and timing from the task policy’s inputs, and uses a learned-skill registry to route recovery attempts and human takeover.

C.2. Real-Robot Monitor and Data Interface

  • The monitor receives top, front, and two wrist views, while the task policy uses the top and wrist cameras.Task descriptions, completion requirements, and behavior notes are cached across collection sessions.Source: §C.2 · p. 19
  • Completion and intervention checks run every 20 s and 10 s, respectively, while an autonomous recovery attempt is limited to 50 s.Intervention checks run asynchronously; the robot holds still during completion and restoration checks, and restoration accepts arrangements equivalent to the initial scene rather than requiring exact poses.Source: §C.2 · p. 19
  • A registered learned skill gets one bounded attempt; otherwise, or after failed restoration, the operator takes control.After two consecutive interventions on a task, the harness requests a task demonstration; successful autonomous recoveries need no human demonstration.Source: §C.2 · p. 20

C.3. Policy Training

Task and recovery policies share a fine-tuning recipe and base checkpoint, but receive different language prompts and are trained in stages as data arrive.

C.3. Policy Training

  • Both task and recovery policies are fine-tuned from the π0.5 base checkpoint using the same recipe.Training proceeds in stages, each starting from the previous checkpoint as more data arrive.Source: §C.3 · p. 20, Table 5 · p. 20
  • Policies predict 14-dimensional joint-position targets in 50-step chunks from robot state and top-and-wrist images.Joint targets are offsets from the current state, while gripper commands are absolute values.Source: §C.3 · p. 20
  • The task policy receives the task instruction, whereas the recovery policy receives the recovery instruction.Both policies observe images at 224 × 224 alongside the robot state.Source: §C.3 · p. 20

C.4. Evaluation Protocol

The appendix specifies trial scoring and trajectory provenance, then documents simulated recovery skills through rollout examples.

C.4. Evaluation Protocol

  • Tab. 3 evaluates 20 trials for each task and configuration, with an operator judging each against the task-specific success criterion in Tab. 4.Source: §C.4 · p. 20

C.5. Trajectory Provenance

  • Recover trajectories retain recovery instructions, while Success and Expand trajectories retain task instructions; failed policy prefixes are excluded from demonstrations but kept in auditable session logs.Logs also record the acting controller, episode boundaries, outcomes, monitor data, prompts, responses, and timestamps.Source: §C.5 · p. 20

C.6. Qualitative Video Examples

  • Fig. 5 draws two frames from each of ten recordings: two task-policy rollouts, four recovery-policy rollouts, and four simulated recovery skills.Frames are uniformly resized and cropped without retouching.Source: §C.6 · p. 20

E. Recovery Examples on the Simulation Benchmarks

The examples pair failures with recovery-enabled successes from matching initial states, illustrating distinct recoveries across both simulation benchmarks.

E. Recovery Examples on the Simulation Benchmarks

Loading 2610.01178v1…