PaperlayerView exact revision on arXiv

Source-linked AI summary

Reconstruct, Practice, Go Real: Guided Self-Improvement for Embodied Agents

Yen-Jen Wang, Haozhe Jiang, Shuying Deng, Haoru Xue, Weirui Ye, Rocky Duan, Nika Haghtalab, S. Shankar Sastry, Pieter Abbeel, Haozhi Qi

arXiv:2610.02204v1cs.ROcs.AIcs.SYeess.SY

TL;DR

Reliable robot capabilities across diverse tasks require substantial human effort to develop skills and integrate perception with control. RPG uses autonomous simulation practice to revise reusable skills and a system prompt without updating model weights; on 22 held-out tasks, success reaches 95.0% after 15 rounds, and the frozen system succeeds in all 30 physical trials.

  • Problem

    Developing reliable robot capabilities across diverse tasks requires substantial human effort to develop and maintain skills, design rewards, and integrate perception with control.

    Source: §I · p. 2

  • Method

    RPG uses offline data and simulation practice to diagnose failures and revise reusable skills and the system prompt without updating model weights.

    Source: §I · p. 2

  • Results

    95.0% task success after 15 practice rounds, up from 28.6% after the first round on 22 held-out tasks, versus ASPIRE (75.5%) and CaP-Agent0 with GPT-6 Astra Pro (60.0%); all 30 physical trials succeeded.

    Source: §I · p. 2

  • Takeaways & Limitations

    Autonomous practice can develop and refine shared robot skills that support physical deployment.

    Source: §I · p. 2

    Deformable-object manipulation remains a limitation: standard towel folding peaks at 5/10 successes, while the hard variant stays at 0/10 across 15 rounds.

    Source: F. Limitation Case: Deformable… · p. 15

Abstract

from arXiv · show

Building reliable robot capabilities across diverse tasks requires substantial human effort to develop and maintain skills, design rewards, and integrate perception with control. We present Reconstruct, Practice, Go Real (RPG), a framework for autonomous improvement of robot execution systems without updating model weights. RPG identifies manipulation capabilities in an offline dataset and constructs related practice tasks in simulation. During practice, RPG uses execution feedback, privileged simulator state, and available dataset videos to diagnose failures. It develops new reusable symbolic skills, refines existing skills, and revises the system prompt based on these diagnoses. Cross-task evaluation tests individual candidate changes and merged revisions before they are retained for reuse. At test time, a multimodal LLM uses the resulting system prompt and skill library to coordinate perception and robot control. On held-out initializations of 22 manipulation tasks, RPG improves task success from 28.6% after the first practice round to 95.0% after 15 rounds, outperforming all evaluated baselines, including ASPIRE (75.5%) and CaP-Agent0 powered by GPT-6 Astra Pro (60.0%). After a common calibration and hardware-adaptation procedure, the frozen system succeeds in all 30 physical trials, with ten trials on each of three tasks. Project Website: https://rpg-robot.github.io/

I. INTRODUCTION

RPG autonomously develops reusable robot skills and revises the system prompt through simulation practice, without updating model weights. Across 22 held-out manipulation tasks, success reaches 95.0% after 15 practice rounds, exceeding the evaluated baselines.

I. INTRODUCTION

  • A scalable robot controller can delegate repeatable execution to code while reserving multimodal reasoning for decisions needing visual or semantic judgment.Executable programs can be efficient but depend on explicit logic and perception interfaces; frequent model calls can make low-level decisions slow.Source: §I · p. 2
  • RPG develops reusable skills and revises the system prompt through autonomous practice, without updating model weights.It uses dataset-derived simulation tasks, privileged execution, video-based failure diagnosis, and cross-task validation of individual and merged revisions.Source: §I · p. 2
  • 95.0% task success after 15 practice rounds, up from 28.6% after the first round on 22 held-out tasks, versus ASPIRE (75.5%) and CaP-Agent0 with GPT-6 Astra Pro (60.0%).The frozen system also succeeded in all 30 physical trials across three tasks after hardware calibration.Source: §I · p. 2

A. Robot Programs and Skill Improvement

  • Prior work spans program generation –, feedback-based revision, and reusable-skill accumulation –; RPG combines dataset-guided simulation practice with cross-task retention tests.RPG revises shared skill code and the system prompt, then tests candidate and merged changes across tasks before retaining them.Source: A. Robot Programs and Skill Improvement · pp. 2–3

B. Offline Data and Simulation Practice

RPG uses an offline dataset to identify manipulation capabilities and build related simulation practice tasks, then deploys the resulting execution system on a physical robot. Practice tasks and evaluators are fixed before self-improvement begins.

B. Offline Data and Simulation Practice

A. Agent-as-Policy Setup and Initialization

  • The Runtime Agent selects and composes reusable skills from a shared library, while the execution system pairs that library with a system prompt and stays fixed within each episode.The underlying model weights and agent harness remain unchanged throughout Practice.Source: A. Agent-as-Policy Setup and… · p. 3
  • The initial practice library contains 15 tools and skills: nine low-level tools and six higher-level manipulation skills constructed from them.The higher-level skills include pick, place, pick-and-place, table-supported transfer between arms, move-above, and push.Source: A. Agent-as-Policy Setup and… · pp. 3–4
  • On the bottle-placement development trials, the revised skill raises success from 1/5 to 5/5 after diagnosis of a bin-wall collision and addition of full-path checks.Revisions are eligible for merging only when mean task success increases without any individual task dropping by more than 20 percentage points; the merged system must pass the same cross-task validation.Source: Figure 3 · p. 4

B. Reconstruct: Build and Freeze the Practice Tasks

  • The Constructor reviews 3–8 trajectories per selected source task, proposes a related practice task, and builds its simulation environment; humans check the task evaluator before practice.Task definitions and evaluators are fixed before Practice begins.Source: B. Reconstruct: Build and Freeze the… · p. 4

C. Practice: Improve the System from Execution

RPG improves its shared execution system through simulated practice, failure diagnosis, and cross-task validation of proposed revisions. After 15 rounds, the system is frozen for physical deployment following hardware adaptation.

C. Practice: Improve the System from Execution

  • The Runtime Agent attempts each practice task five times, while a Privileged Agent runs from the same initial states with additional simulator-state access.Both agents use the same skills, which rely only on sensing and control available on the deployed robot.Source: C. Practice: Improve the System from… · p. 4
  • For underperforming development tasks, the Video Analyzer compares failures with privileged executions and available source videos, then recommends concrete changes for the Implementor.The Implementor can propose new or modified skills and system-prompt revisions; candidates are checked for valid syntax, interfaces, and deployment-available observations.Source: C. Practice: Improve the System from… · p. 5

D. Validate and Retain System Revisions

  • Each candidate is tested in complete Runtime Agent episodes across the practice suite and passes only if mean success rises without any task losing more than one of five successes.The test compares the candidate and current system on the same five fixed development seeds per task.Source: D. Validate and Retain System Revisions · p. 5
  • Eligible revisions are merged and retested against the current system; the merged system is retained only if it passes the same test and the Merger selects it.Candidates that pass individually may still be rejected after review of execution records.Source: D. Validate and Retain System Revisions · p. 5

E. Go Real: Adapt and Freeze for Deployment

IV. EVALUATION

The evaluation asks whether practice improves task success, how diagnostic inputs contribute, and whether the resulting system works after physical adaptation. Across 22 simulated tasks, RPG reaches 95.0% success and exceeds the evaluated baselines.

A. Tasks and Evaluation Protocol

  • The 22-task suite contains seven CaP-X tasks and fifteen ABC-derived or analogous tasks, spanning manipulation types from transport and sorting to bimanual tasks and cloth folding.Round-end systems are evaluated retrospectively on paired held-out initializations excluded from improvement and checkpoint selection.Source: A. Tasks and Evaluation Protocol · p. 5

B. Q1: Does RPG Improve with Practice?

  • RPG achieved 209/220 successes (95.0%), outperforming all baselines; among CaP-Agent0 models, GPT-6 Astra Pro was best at 132/220 (60.0%), below RPG using Gemini 3.8 Flash.Round-end systems were evaluated retrospectively on the same 22 tasks and 10 held-out seeds per task after development was frozen.Source: B. Q1: Does RPG Improve with Practice? · p. 6, Figure 4 · p. 6, Table II · p. 7
  • RPG improves from 63/220 successes (28.6%) after round 1 to 209/220 (95.0%) after round 15, exceeding ASPIRE's 75.5% endpoint by round 5.Success rises from 51.4% to 98.6% on CaP-X tasks and from 47.3% to 93.3% on ABC-derived or analogous tasks.Source: B. Q1: Does RPG Improve with Practice? · p. 6
  • Across 15 rounds, the library expands from 15 to 38 entries with 23 new skills and 66 modifications, while the system prompt is revised during rounds 1–4.The largest gain, 30.5 percentage points between rounds 3 and 4, coincides with a prompt revision and reflects its combined behavioral effect with changed observation-budget use.Source: B. Q1: Does RPG Improve with Practice? · pp. 6–7
  • The practice curve is not strictly monotonic: held-out success falls by one at rounds 6 and 14 despite revisions passing five-seed development checks.Those checks did not improve every result on the ten held-out evaluation seeds.Source: B. Q1: Does RPG Improve with Practice? · p. 7

C. Q2: Which Diagnostic Inputs Drive Improvement?

Diagnostic ablations show that both privileged executions and video-based analysis contribute to improvement. Separate library-only comparisons find better complete-task performance after skill revisions, though gains are not uniform across tasks.

C. Q2: Which Diagnostic Inputs Drive Improvement?

  • Full RPG reaches 172/220 successes (78.2%), versus 114/220 (51.8%) without the Video Analyzer and 112/220 (50.9%) without the Privileged Agent or both.These five-round ablations start from the same 15-skill initialization and are evaluated on 220 held-out episodes.Source: Table III · p. 7, C. Q2: Which Diagnostic Inputs Drive… · p. 7

D. Q3: Do Skill-Library Revisions Improve Task Success?

  • 37/40 successes after revision versus 26/40 before it is a 27.5-percentage-point gain in the controlled library comparison.The runtime, prompt, perception implementation, and episode budgets are fixed; the revised library also preserves 10/10 bottle placement success.Source: Table IV · p. 8, D. Q3: Do Skill-Library Revisions… · p. 8

C. Q2: Which Diagnostic Inputs Drive Improvement?

  • Removing the Video Analyzer lowers success by 26.4 percentage points and removing the Privileged Agent by 27.3 points; either single-component variant is within 0.9 points of removing both.The Privileged Agent supplies privileged-state reference executions, while the Video Analyzer combines them with Runtime Agent traces and available dataset videos to diagnose failures.Source: C. Q2: Which Diagnostic Inputs Drive… · p. 8

D. Q3: Do Skill-Library Revisions Improve Task Success?

  • The revised lift skill completes the approach, uses measured post-grasp fingertip position, and verifies displacement, grasp retention, and object levelness; observed lift invocations succeed 11/11 after revision versus 1/13 before.The task-level comparison reflects this repair together with other library revisions.Source: D. Q3: Do Skill-Library Revisions… · p. 8
  • With a privileged scripted caller, changing only the library raises success from 19/40 to 39/40, but pooled ABC-sorting results fall from 6/6 to 5/6.Both libraries pass 12 of 14 auxiliary runtime checks, underscoring that complete task-level behavior can regress locally.Source: D. Q3: Do Skill-Library Revisions… · p. 8

E. Q4: Does RPG Transfer to a Physical Robot?

On the YAM robot, RPG transfers to three quantitatively evaluated tasks and five additional qualitative tasks. The conclusion reports all 30 quantitative trials successful and identifies rollout rate limits and revision integration as bottlenecks.

E. Q4: Does RPG Transfer to a Physical Robot?

  • The physical evaluation uses ten trials per task, randomized object positions, a common calibration procedure, and no human assistance, manual reset, or system revision during trials.Trials stop after 30 VLM calls or 20 minutes; autonomous retries and recovery are allowed.Source: E. Q4: Does RPG Transfer to a Physical… · p. 8
  • RPG completes all 10 trials on each of three physical tasks; GPT-6 Astra Pro completes drawer closure in 4/10 and bowl transfer in 1/10, and folds the towel in 9/10.All methods use the same calibration and hardware-adaptation procedure.Source: E. Q4: Does RPG Transfer to a Physical… · pp. 8–9, Table V · p. 11
  • RPG’s advantage over GPT-6 Astra Pro is largest on the two multi-stage tasks, while it exceeds that baseline on towel folding by only one success.Reported outcomes include autonomous retries or recovery within the episode budget.Source: E. Q4: Does RPG Transfer to a Physical… · p. 9
  • Five additional physical tasks demonstrate qualitative zero-shot deployment of the frozen system without task-specific real-world tuning, demonstrations, or retry-based adaptation.These examples are excluded from quantitative baseline comparison.Source: E. Q4: Does RPG Transfer to a Physical… · p. 10, Figure 6 · p. 10

V. CONCLUSION

  • The conclusion reports success in all 30 physical trials across the three evaluated tasks and presents shared skill code and the system prompt as practical targets for persistent improvement.The trials follow the same calibration and hardware-adaptation procedure as the baselines.Source: §V · pp. 10–11
  • API rate limits constrain parallel rollouts across 22 tasks, while overlapping revisions to shared skills and dependencies can complicate integration as the task set grows.The paper suggests quota-aware scheduling and dependency-aware proposal integration as future work.Source: §V · p. 11

APPENDIX

Each practice round runs Runtime and Privileged Agents, diagnoses failures, proposes and tests revisions across tasks, then retains only a validated integrated system. Development decisions use separate initializations from the held-out evaluations.

System and practice rollouts

  • The execution system comprises a shared skill library and system prompt, both fixed within episodes and revisable only between practice rounds.Model weights, task definitions, initial-state distributions, and evaluation criteria remain fixed throughout practice.Source: A. Self-Improvement Protocol · p. 12
  • At each round’s start, the Runtime Agent attempts every task on five fixed development initializations while a matched-state Privileged Agent additionally accesses simulator state.Both agents use the same runtime model and shared skills; skills themselves do not receive privileged state.Source: A. Self-Improvement Protocol · p. 12

Failure diagnosis

  • The Video Analyzer compares agent executions and available dataset videos to identify the first observable failure, infer a likely cause, and propose an evidence-supported correction.Its inputs include execution traces, perception outputs, robot states, simulator ground truth, and evaluator metrics; it does not modify the system directly.Source: A. Self-Improvement Protocol · p. 12

Candidate generation

  • Only tasks below 5/5 Runtime Agent success can generate a revision, with at most one candidate per task per round; revisions may change shared skills or the system prompt.Tasks already at 5/5 and non-development tasks remain available to detect cross-task regressions.Source: A. Self-Improvement Protocol · p. 12

Cross-task candidate validation

  • Each candidate is tested against the round-start system on the same five development initializations across all 22 tasks, or 110 episodes per system.A candidate must improve mean task completion without reducing any task’s success count by more than one out of five.Source: A. Self-Improvement Protocol · p. 12

Greedy integration

  • Eligible candidates are ranked by mean development-success increase and greedily integrated, with at most eight integration attempts per round.Overlapping file changes use a three-way merge; unresolved conflicts go to a coding agent instructed to preserve compatible additions and avoid unrelated changes.Source: A. Self-Improvement Protocol · p. 12

Merged-system validation and retention

  • The integrated system is retested across the full development suite and retained only if it passes the same cross-task regression criterion; otherwise, the previous accepted system is preserved.Candidate-level success alone does not ensure revisions remain effective after integration.Source: A. Self-Improvement Protocol · p. 12

One round in summary

  • A round executes both agents, diagnoses failures, proposes candidates, validates them across tasks, integrates and retests revisions, then retains or rolls back the result; the experiment repeats this for 15 rounds.Round-end systems are evaluated afterward on ten held-out initializations per task, separate from candidate and integration decisions.Source: A. Self-Improvement Protocol · pp. 12–13

B. Qualitative Examples of Skill Acquisition and Refinement

The examples show RPG turning unreliable behaviors into reusable procedures with explicit checks, geometry-aware actions, and recovery steps. Reported development gains span object placement, insertion, wiping, and coordinated lifting.

Robust object-in-container placement

  • Place bottles in bin rises from 1/5 to 5/5 in Round 12 after the behavior shifts from gripper-proximity placement to verified object containment.The revision localizes the opening, checks the route, clears the rim, and verifies containment after release.Source: B. Qualitative Examples of Skill… · p. 13

Geometry-aware precision insertion

  • Nut assembly success increases from 1/5 in Round 4 to 5/5 in Round 5 while the system prompt remains fixed.The revised skill verifies grasp, aligns the nut’s square hole with the peg, inserts incrementally, and checks seating before release.Source: B. Qualitative Examples of Skill… · p. 13

Closed-loop wiping

  • Spill wipe development performance rises from 4/5 (80%) in Round 4 to 5/5 (100%) in Round 5.The refined procedure calibrates contact, uses short overlapping strokes, reobserves residual debris, and chooses corrective strokes accordingly.Source: B. Qualitative Examples of Skill… · pp. 13–14

Verified bimanual lifting

  • Two-arm lift success rises from 3/5 (60%) in Round 11 to 5/5 (100%) in later rounds.The behavior verifies both paths and grasps, performs a synchronized test lift, then monitors matched lifting motions and object rise.Source: B. Qualitative Examples of Skill… · p. 14

C. Round-by-Round Task Evolution

Held-out task performance evolves unevenly across rounds: some capabilities saturate early, while others improve abruptly or fluctuate. The reported examples distinguish round-level held-out outcomes from in-loop development measurements.

C. Round-by-Round Task Evolution

  • Nut assembly rises from 5/10 in Round 4 to 10/10 in Round 5, while Place bottles in bin stays near zero for many rounds before reaching 10/10 by Round 15.Some tasks, including Lift cube and Stack cubes, reach 10/10 early and remain there.Source: C. Round-by-Round Task Evolution · p. 14
  • Bimanual and long-horizon tasks show more non-monotonic trajectories, reflecting changes to multiple reusable behaviors rather than independent optimization of each benchmark instance.The table reports per-task successful episodes out of ten across all 15 rounds.Source: C. Round-by-Round Task Evolution · p. 14
  • For Place bottles in bin, the Round 12 in-loop evaluation improves from 1/5 to 5/5, whereas the separate held-out Round 12 result is 9/10.The two figures come from five-rollout practice evaluation and ten-seed held-out evaluation, respectively.Source: C. Round-by-Round Task Evolution · p. 14

D. Transfer without Direct Task-Specific Improvement Feedback

Reusable manipulation changes can transfer to tasks that provide no direct task-specific improvement feedback, though transfer is not universal. The learned corrections increasingly pair actions with checks and include reusable patterns that can be composed across tasks.

D. Transfer without Direct Task-Specific Improvement Feedback

E. Failure and Learned-Correction Taxonomy

  • The failure taxonomy is qualitative, and describes self-improvement as turning implicit assumptions into explicit checks before, during, and after actions.Procedures increasingly verify preconditions, monitor execution, and check task-level state before terminating.Source: E. Failure and Learned-Correction… · p. 14
  • Test-lift verification, full-route reach checking, object-centric alignment, incremental contact motion, and post-release verification are reusable patterns that can be composed in later tasks.Source: E. Failure and Learned-Correction… · p. 14

F. Limitation Case: Deformable Manipulation

Deformable manipulation remains a clear limitation: towel-folding performance is unreliable, and the harder alignment variant never succeeds across the reported rounds. Failure analysis points to changing cloth state and contact as a boundary that rigid-object procedural corrections do not resolve.

F. Limitation Case: Deformable Manipulation

  • The standard towel-folding task peaks at 5/10 successes, while Fold towel (hard) stays at 0/10 throughout all 15 rounds.The hard variant tightens corner alignment from 5 cm to 1 cm and is separate from the main benchmark.Source: F. Limitation Case: Deformable… · p. 15, Table IX · p. 16
  • Folding failures include uncertain corner acquisition, losing a corner during bimanual transport, insufficient support clearance, and incomplete landing reachability.The analysis says the limitation is not explained by a single missing waypoint; successful folding requires maintaining thin, deformable material in both grippers as cloth geometry changes.Source: F. Limitation Case: Deformable… · p. 15
  • Rigid-object corrections such as short-lift corner checks and shorter matched arm movements remain useful, but do not reliably estimate layer identity, persistent cloth contact, or post-fold geometry.Source: F. Limitation Case: Deformable… · p. 15
  • For deformable objects, contact substantially changes the underlying state, and visually similar configurations can represent different layering and attachment states.The paper suggests richer representations of deformable state and contact may be needed, rather than only composing existing rigid-object skills.Source: F. Limitation Case: Deformable… · pp. 15–16

G. Simulation Task Suite

The simulation suite evaluates 22 bimanual manipulation tasks on the YAM station, with separate development, validation, and feedback-held-out roles. Tasks vary initial conditions and use binary outcome evaluators, with additional diagnostic tasks kept outside the main benchmark.

G. Simulation Task Suite

  • The main suite contains 22 bimanual manipulation tasks on the YAM station, divided into 17 development tasks, one validation task, and four feedback-held-out tasks.Development tasks may generate revisions; the validation and feedback-held-out tasks serve as cross-task regression checks, and the held-out tasks are not analyzed by the Video Analyzer.Source: G. Simulation Task Suite · p. 16
  • Trials use top and wrist RGB images, top-camera depth, and robot proprioception, while initial conditions randomize object poses and selected physical and camera parameters.Task-specific skill-call and execution-time limits are part of each task definition.Source: G. Simulation Task Suite · p. 16
  • Nine outcome-evaluator types cover objectives including placement, lifting, closure, extraction, stacking, debris removal, and cloth folding.Guard conditions reject degenerate solutions such as moving the fixture instead of extracting an object or leaving a pre-stacked scene unchanged.Source: G. Simulation Task Suite · pp. 16–17
  • Representative checks require both pegs in their cups for fixture insertion, coordinated handle elevation for two-arm lifting, and removal of at least six of nine debris particles for spill wiping.Source: G. Simulation Task Suite · p. 17
  • Bottle placement evaluates two specified target bottles among six bottles, while towel folding checks corner alignment, layer order, footprint reduction, and cloth thickness.The two bottle targets are black and clear with a blue-and-yellow label; four other bottles are distractors.Source: G. Simulation Task Suite · p. 17
  • Fold towel (hard) tightens corner tolerance from 5 cm to 1 cm, and Grasp plate isolates rim grasping; neither diagnostic task is pooled into the main benchmark.Source: G. Simulation Task Suite · p. 17

Related papers

Loading 2610.02204v1…