PaperlayerView exact revision on arXiv

Source-linked AI summary

EmbodiRSI: Recursive Self-Improvement for Data-Efficient Robot Adaptation

Haoran Lang, Haotao Lu, Shiyu Sang, Haoyang Luo, Guo Chen, Qun Li, Jingyi Yu, Ye Shi, Jingya Wang

arXiv:2609.38905v1cs.RO

TL;DR

Robot manipulation policies face data-intensive adaptation to new environments, while the useful next training data depends on current policy weaknesses. EmbodiRSI uses feedback-driven recursive improvement in task-specific simulation before physical refinement; it reaches 83.1% real-world success with ten refinement trajectories per subtask, versus 75.0% with 200 real-world demonstrations.

  • Problem

    Adapting manipulation policies to new environments is data-intensive, and the experience needed for improvement depends on evolving policy weaknesses.

    Source: §1 · p. 2

  • Method

    EmbodiRSI iteratively uses policy feedback in task-specific simulation to guide corrective recovery, targeted expert data collection, and policy updates before limited physical refinement.

    Source: §1 · pp. 2–3

  • Results

    83.1% scene-balanced autonomous real-world success with 400 adaptive simulated trajectories and ten real-world refinement trajectories per subtask, versus 75.0% using 200 real-world demonstrations per subtask.

    Source: §1 · p. 3

  • Takeaways & Limitations

    EmbodiRSI demonstrates data-efficient real-world adaptation across diverse manipulation tasks with only a small amount of physical refinement data.

    Source: §1 · p. 3, §6 · p. 10

    The evaluation centers on the complete pipeline, with component-wise attribution not its primary focus.

    Source: §5 · p. 10

Abstract

from arXiv · show

Adapting robot manipulation policies to new tasks and environments remains highly data-intensive, while the data needed for further improvement depends on the policy's current capabilities and failure modes. We introduce EmbodiRSI, an agentic system for recursive self-improvement (RSI) in a real-to-sim-to-real setting, where task-specific simulations are constructed from target deployment scenarios and used as low-cost environments for iterative policy improvement before transfer back to the physical world. EmbodiRSI uses policy execution feedback to guide subsequent experience acquisition and policy updates. Two complementary mechanisms close this loop: Collaborative Error Correction generates agent-assisted corrective trajectories from policy-reached states, while Adaptive Data Collection directs expert demonstration generation toward the current policy's weaknesses. The task-specific simulation serves as a reusable workspace for policy warm-up, repeatable evaluation, failure diagnosis, and targeted data generation across successive RSI rounds. Across three tabletop environments and 14 subtasks, EmbodiRSI increases scene-balanced autonomous simulation success from 50.4% to 83.5% over two RSI updates. With 400 adaptive simulated trajectories and only ten real-world refinement trajectories per subtask, EmbodiRSI achieves 83.1% scene-balanced autonomous real-world success, compared with 75.0% for adaptation using 200 real-world demonstrations per subtask. These results demonstrate that feedback-driven recursive improvement in deployment-specific simulations can enable data-efficient adaptation of embodied policies to physical environments.

1 Introduction

EmbodiRSI uses policy feedback to adapt experience acquisition and policy updates in deployment-specific simulation before physical transfer. Across three environments, it improves autonomous success while using substantially less real-world refinement data than the comparison approach.

1 Introduction

  • Deployment failures can be underrepresented in original training data, and their locations shift as policies improve, motivating repeated evaluation and targeted experience acquisition.The stated research question asks whether current-policy feedback can guide both failure recovery and the next experience to acquire.Source: §1 · p. 2
  • EmbodiRSI iteratively evaluates the current policy in task-specific simulation, uses execution outcomes to acquire experience, updates the policy, and repeats before real-world transfer.The experience distribution changes with the policy rather than remaining fixed in advance.Source: §1 · p. 2
  • Collaborative Error Correction generates corrective trajectories from policy-reached failure states, while Adaptive Data Collection directs expert demonstrations toward recurring weaknesses.Correction operates within an episode; adaptive collection diagnoses failures across episodes.Source: §1 · p. 2
  • The task-specific simulation supports repeatable evaluation, failure diagnosis, error correction, and targeted data generation, followed by limited human-in-the-loop physical refinement.Generated trajectories and recovery actions are checked with geometric reasoning, collision checking, and motion-planning tools.Source: §1 · p. 3
  • 83.1% scene-balanced autonomous real-world success with 400 adaptive simulated trajectories and ten real-world refinement trajectories per subtask, versus 75.0% using 200 real-world demonstrations per subtask.Across three tabletop environments and 14 subtasks, scene-balanced autonomous simulation success also rises from 50.4% to 83.5% over two RSI updates.Source: §1 · p. 3

2 Related Work

Related work connects simulation-based data generation, real-to-sim-to-real transfer, and iterative policy improvement. EmbodiRSI positions its contribution as an agent-driven loop that uses evolving policy feedback to revise training experience.

2 Related Work

  • Simulation data-generation work creates tasks or environments, expands demonstrations, or uses agentic generation and verification; EmbodiRSI instead uses rollout feedback to repeatedly revise experience in a retained workspace.The workspace remains available after initial training for repeated policy improvement.Source: §2 · p. 3
  • Real-to-sim-to-real systems reconstruct or generate simulation from real observations, learn or expand experience there, and transfer resulting policies or behaviors back to robots.The cited examples include RialTo, RL-GSBridge, RoboGSim, Re3Sim, RoboSimGS, ReBot, and X-SIM.Source: §2 · p. 3
  • Prior recursive robot-learning systems use autonomous collection, corrective feedback, rollout verification, or imagined experience; EmbodiRSI couples within-episode recovery with across-episode recollection.Related real-to-sim systems bridge real observations and robot execution; EmbodiRSI adds an agent-driven loop unifying scene construction, trajectory generation, training, evaluation, recovery, and data refinement.Source: §2 · p. 4

3 Method: A System for Recursive Policy Self-Improvement

EmbodiRSI retains a reconstructed task environment as a workspace for recursive policy improvement. Agent-guided monitoring, recovery, and targeted collection turn rollout outcomes into training experience for successive updates and autonomous evaluation.

3 Method: A System for Recursive Policy Self-Improvement

  • EmbodiRSI couples policy optimization to the acquisition of subsequent training experience, using a scene-specific policy and configuration distribution within scheduled RSI rounds.A configuration specifies source object, target anchor, instruction, grasp and waypoint program, and admissible reset ranges.Source: §3 · p. 4

3.1 An Executable Environment for RSI

  • The reconstructed scene is retained as the practice environment, supporting repeatable resets, training-time diagnosis, and recovery from policy-failed states.The scene agent reconstructs objects and organizes a target scene from tabletop imagery and task goals.Source: §3.1 · p. 4

3.2 A Reusable Source of Training Experience

  • A trajectory agent compiles task configurations and validates demonstrations; stored waypoints are re-grounded to measured object poses and replanned after reset.Validation includes collision checking, multi-view review, motion feasibility, and execution-time task checks.Source: §3.2 · p. 4

3.3 Agent-Guided Recursive Policy Self-Improvement

  • Recovery supplies corrective supervision within episodes, while rollout diagnosis refines the collection distribution across episodes; accumulated experience updates the policy.Successful rollouts after correction and newly generated demonstrations are accumulated for the next round.Source: §3.3 · p. 5, Figure 2 · p. 5

3.3.1 Agentic Rollout Monitor

  • The rollout monitor uses a temporal image window, action history, and simulator state to choose whether to continue, recover, or mark completion.Visual and geometric checks inform completion and recovery decisions, which are gated by confidence and other safeguards.Source: §3.3.1 · p. 5

3.3.2 Collaborative Error Correction

  • Collaborative Error Correction plans from the state the policy actually reached, re-grounding targets to the current scene instead of replaying a clean initial-state demonstration.Successful continuations provide action supervision from imperfect states; evaluation uses the learned policy alone.Source: §3.3.2 · pp. 5–6

3.3.3 Adaptive Data Collection

  • Adaptive collection attributes final failures to reset subjects, identifies high-failure configuration cells, and samples subsequent demonstrations toward diagnosed weak configurations.The mixture retains access to the original range, and the refinement applies to training collection rather than test resets.Source: §3.3.3 · p. 6

3.3.4 Policy Updating and Recursion

  • Each scheduled round retains rollout-derived episodes and validated demonstrations, forms the next dataset, trains the next policy, then evaluates independently with the learned policy alone.The selected policy is refined with a small real-world human-in-the-loop dataset; final tests disable agent takeover, planner recovery, and human correction.Source: §3.3.4 · pp. 6–7

4 Experiments

Across the evaluated scenes, recursive improvement raises simulation success, feedback-driven collection outperforms static collection at a matched budget, and simulation adaptation achieves higher real-world success than a larger real-only demonstration set. The experiments evaluate 14 subtasks across three paired workspaces, with physical refinement and policy-only testing in the real world.

4 Experiments

  • The evaluation uses one policy per scene across 14 subtasks in Office Table, Kitchen Table, and Household Table workspaces, with ten physical trials per task and setting.Policies are tested autonomously without external recovery; +10 settings add ten human-in-the-loop refinement trajectories per subtask. Simulation test counts vary by task–checkpoint pair.Source: §4 · p. 7, Figure 3 · p. 8

4.1 Does the Policy Improve Across RSI Rounds?

  • Scene-balanced autonomous simulation success rises from 50.4% at R0 to 70.7% at R1 and 83.5% at R2; all 14 subtasks improve from R0 to R2.Black-pen placement in S1 rises from 14.0% to 76.5%, while carrot placement in S2 starts near ceiling. Figure 4 plots individual subtasks and scene means across rounds using 50, 200, and 400 retained simulated trajectories per subtask.Source: §4.1 · p. 7, Figure 4 · p. 8
  • At larger simulation budgets, most observed errors arise during execution of the requested manipulation, while instruction confusion appeared occasionally at the 50-demonstration initialization.In zero-shot deployment, policies generally approach the appropriate object, but grasping and placement remain residual failure points; RSI policies show more consistent grasp alignment and sometimes complete an episode after a new attempt.Source: §4.1 · p. 8, §4.3 · p. 9

4.2 Does Feedback Help at Matched Trajectory Counts?

  • At 400 simulated trajectories per subtask, adaptive collection achieves 79.9% scene-balanced simulation success versus 59.3% for Static, a gain of 20.6 percentage points.Adaptive collection has higher scene-level mean success in both evaluated scenes at both matched budgets.Source: §4.2 · p. 8, Table 1 · p. 9

4.3 Does RSI Enable Data-Efficient Real-World Adaptation?

  • 83.1% scene-balanced autonomous real-world success with 400 simulated and ten real-world refinement trajectories per subtask, versus 75.0% for Real-only adaptation with 200 real demonstrations per subtask.The reported observed mean is higher in each scene, using one-twentieth as many retained physical training trajectories.Source: Table 2 · p. 9, §4.3 · p. 9

5 Limitations

The evaluation is bounded by available time and computational budget and prioritizes scene-specific adaptation in selected workspaces. It also evaluates the complete pipeline rather than attributing outcomes to individual components.

5 Limitations

  • Evaluation prioritizes selected workspaces; held-out workspace generalization and sustained improvement over longer runs remain for future assessment.Source: §5 · p. 10
  • Because recovery, recollection, and agent reasoning operate jointly in the evaluation, component-wise attribution is not its primary focus.The authors plan targeted ablations to study component contributions and interactions.Source: §5 · p. 10

6 Conclusion

The conclusion presents EmbodiRSI as a real-to-sim-to-real workflow that makes experience acquisition responsive to the evolving policy. It highlights improved performance across evaluated rounds, gains over static collection at matched trajectory counts, and data-efficient real-world adaptation.

6 Conclusion

  • EmbodiRSI couples policy updates to evolving experience acquisition, using corrective continuations and targeted practice before limited real-world refinement.The paper concludes that refining the experience driving policy improvement is valuable alongside refining the policy itself.Source: §6 · p. 10

AI Use Statement

The statement describes AI assistance in manuscript preparation and analysis, while distinguishing illustrative materials from experimental evidence.

AI Use Statement

  • AI assisted manuscript restructuring, editing, literature checks, descriptive analysis code, and figure-layout work; no new robot trials or simulation measurements were introduced during manuscript preparation.Submitted experimental images were retained; generated diagrams and scene inputs are identified as illustrations rather than experimental evidence.Source: AI Use Statement · p. 10

A RSI implementation and agent coordination

The appendix specifies the operational interfaces and validation procedures connecting scene construction, trajectory generation, recovery, and supervisory coordination.

A.1 Task interfaces and recovery handover

  • Task configurations bind objects, targets, instructions, grasps, programs, and reset groups; recovery uses current states, refreshes observations, and rechecks completion.Event records distinguish recovery requests, corrections actually executed, and final outcomes.Source: §A.1 · p. 14

A.2 Grid attribution records and counting

  • Failure attribution counts only completed episodes and supported subject-attributed final failures; low-confidence, unassigned, and recovery-resolved episodes do not count as attributed failures.Attribution can use recorded execution evidence and existing records rather than requiring a new video replay.Source: §A.2 · p. 14

A.3 Supervisor scheduling and output validation

  • The supervisor coordinates typed skills through persistent workflow state, dispatches eligible jobs, and requires valid outputs—not merely process exit—for completion.When a check fails, it requests repair of the earliest invalid dependency rather than restarting the whole workflow.Source: §A.3 · pp. 14–15

B.1 Agentic Real2Sim Scene Construction and Refinement

  • Scene construction combines semantic relation planning with geometric checks: reconstruction is rectified, then coarse-to-fine target poses define object goals that can be re-grounded after resets.Collision and support checks precede task and trajectory generation.Source: §B.1 · pp. 15–16

B.2 Agentic Trajectory Generation

  • Trajectory generation proposes object routes, conditions them on candidate grasps, and validates, repairs, and executes robot programs before retaining demonstrations.The same trajectory tools later support recovery and recollection, making demonstration generation reusable within the learning loop.Source: §B.2 · pp. 17–18

B.3 Scene geometry

  • AACR resolves inter-group overlap with minimum horizontal moves, batches only disjoint sweeps, then uses relation-specific corrections for within-group collisions.Support-aware groups move as units during horizontal rectification.Source: §B.3 · p. 18

B.4 Sequential motion execution

  • Robot transitions are planned sequentially from the preceding terminal joint state; after resets, waypoints are re-grounded to current anchors and checks repeat after repairs.Visual review is repeated when the object route changes.Source: §B.4 · p. 18

C.1 Robot platform and evaluated tasks

This section describes the evaluated tasks, reconstruction examples, retained-data budgets, and comparison settings for simulation and real-world adaptation.

C.1 Robot platform and evaluated tasks

  • The evaluation covers 14 object-level goals across paired simulation–real environments, with initial and goal relations fixed across each pair.One policy covers a scene’s subtasks, which are evaluated separately; matched-collection comparisons use S1 and S3.Source: §C.1 · p. 18, Table C.1 · p. 19

C.2 Reconstruction and organization examples

  • Examples pair varied observations with reconstructions and show critique, editing, and validation of target layouts; the views are not pixel-aligned.Some reconstruction examples extend beyond the evaluation benchmark.Source: Figure C.1 · p. 19, Figure C.2 · p. 20, Figure C.3 · p. 21

D.1 Training data and retained-trajectory budgets

  • Budgets count retained trajectories per subtask: R0 uses 50, R1 reaches 200 cumulatively, and R2 reaches 400; only ultimately successful episodes enter policy training.The schedule separates newly generated from rollout-derived experience without inferring a numerical split between base and targeted collection.Source: §D.1 · pp. 20, 22, Table D.1 · p. 20, Table D.2 · p. 22

D.2 Real-world refinement and comparison settings

  • Real-only settings use 10, 100, or 200 demonstrations per subtask; static and adaptive simulation collection share 50 initial trajectories and matched retained counts.The compared policies use the same architecture and training recipe, but equal real-data counts do not imply identical datasets; HIL corrections and Real-only demonstrations are distinct settings.Source: §D.2 · p. 22, Table D.3 · p. 22

D.3 Autonomous evaluation protocol

The protocol evaluates autonomous policy execution separately from monitored training collection in both simulation and the physical world.

D.3 Autonomous evaluation protocol

  • Autonomous tests disable agent takeover, planner recovery, and human correction but retain policy reattempts; zero-shot means no task-specific real-world refinement.Source: §D.3 · pp. 22–23
  • Simulation success is measured over repeated task-checkpoint rollouts whose episode counts vary, separately from monitored training collection.Source: §D.3 · p. 23
  • Real-world evaluation uses 10 autonomous rollouts per subtask, or 140 per complete setting; subtasks sharing a policy are not independent policy replicates.Source: §D.3 · p. 23

D.4 Statistical analysis and numerical precision

Appendix D defines the weighting and precision conventions used for the reported comparisons, while Appendix E presents task-level simulation and real-world outcomes. Adaptive collection outperforms static collection in the matched-budget simulation comparisons overall, with task-level exceptions.

D.4 Statistical analysis and numerical precision

  • Scene-level standard deviations describe variation across subtasks sharing a policy, not independent training runs; task-matched comparisons do not imply episode-wise paired resets.Real-world rates use integer success counts, while simulation summaries use tabulated task-level percentages; reported calculations retain input precision and round final displayed values to one decimal.Source: §D.4 · pp. 23–24
  • The pooled R2+10 count is 116/140, while its scene-balanced mean is 83.1%, reflecting different weighting.Collection-method differences are formed per subtask before summarization, and equal-scene comparison means use S1 and S3 only.Source: §D.4 · p. 23

E Detailed results and behavioral observations

E.1 Simulation evolution

  • Table E.1 reports task-level simulation success, scene means and standard deviations, and an equally weighted final scene row; column headers give trajectories per subtask.Source: Table E.1 · p. 24

E.2 Real-world transfer

  • Figure E.1 presents the task-level comparison between R2+10 and Real-only 200, with outcomes that include ties and a negative difference.Source: §E.2 · p. 24
  • Table E.2 reports task-level real-world results from ten autonomous tests, with scene summaries and a final equally weighted scene mean.R1 and R2 use 200 and 400 simulated trajectories per subtask; +10 adds ten real HIL trajectories, while Real 10/100/200 denote real-only demonstration counts.Source: §E.2 · p. 24, Table E.2 · p. 25

E.3 Matched-trajectory collection comparisons

  • The scene-balanced adaptive advantage is 14.4 percentage points at 200 trajectories and 20.6 points at 400.These comparisons give equal weight to S1 and S3 and aggregate task-level percentages before rounding.Source: §E.3 · p. 25
  • Adaptive collection is higher on all six S1 subtasks and three of four S3 subtasks at both budgets, though the S3 pencil-case task favors Static.For that task, Static scores 94.8% versus 94.0% at 200 trajectories and 97.2% versus 96.0% at 400; larger Static datasets do not improve every task.Source: §E.3 · p. 25

E.4 Qualitative execution observations

The qualitative observations describe execution and transfer patterns, while the related-work discussion distinguishes EmbodiRSI's fixed-formulation recursion, configuration-level acquisition, and data-generation recovery from execution-time recovery mechanisms.

E.4 Qualitative execution observations

  • Missed grasps occurred at peripheral positions, and placement failures followed successful grasps, especially near narrow target openings; early instruction confusion was not recurrent at larger simulation budgets.Peripheral positions were not shown to be unreachable or outside the training distribution.Source: §E.4 · p. 25
  • Reattempts at R1 and R2 had mixed outcomes, while HIL operators mainly corrected final approach, acquisition, and placement.After refinement, broad approach motions resembled simulation demonstrations and near-grasp and placement adjustments resembled physical corrections, but this does not imply online inverse-kinematics control or that HIL changes only contact behavior.Source: §E.4 · p. 27
  • Static+10 benefited from local corrections but was less consistent under pose changes and showed fewer reattempts than refined evolved policies.With ten real-world demonstrations per subtask, successful real-only behavior was largely restricted to particular configurations; 100- and 200-demonstration policies still had difficult-pose vulnerabilities and occasional instruction-inconsistent task selection.Source: §E.4 · p. 27

F.1 Autonomous improvement and corrective experience

  • EmbodiRSI couples corrective continuations from policy-failed states with updates to subsequent collection configurations, while keeping the policy-learning formulation fixed.Its recursion operates through acquired experience rather than changes to the learning algorithm.Source: §F.1 · p. 28

F.2 Failure-driven acquisition and adaptive curricula

  • Failure-driven acquisition selects source-object and target-anchor reset regions, mixes targeted configurations with the base distribution, and uses failure concentration as a heuristic rather than an estimate of learning gains.The evaluated procedure maintains fixed per-subtask retained-data budgets.Source: §F.2 · pp. 28–29

F.3 Agentic workflows and execution-time recovery

  • Diagnostic and planning tools generate collection-time experience, but subsequent policy evaluation excludes those external recovery mechanisms; test-time reattempts reflect the learned controller.Source: §F.3 · p. 29

Related papers

Loading 2609.38905v1…