PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboRSI: Stable, efficient, and reusable robot self-evolution in complex real-world environments

Zimo Wen, Yijin Chen, Yuxuan Cao, Wendi Chen, Yanwen Zou, Wenye Yu, Fuhang Kuang, Han Xue, Jun Lv, Chuan Wen, Cewu Lu

arXiv:2610.12424v1cs.ROcs.AI

TL;DR

Robot agents can repair programs from execution feedback, but organizing that experience around task structure so repairs reach the responsible capability and are validated before reuse remains a challenge. RoboRSI uses Top-Down Skill Refinement and coordinated agents to make scoped, validated skill revisions. It develops household cleanup on a mobile manipulator and achieves the highest success rate on four simulation benchmarks.

  • Problem

    Robot agents can repair programs from feedback, but organizing experience so repairs are assigned to the responsible capability and validated before reuse remains challenging.

    Source: §1 · p. 4

  • Method

    RoboRSI uses TSR to decompose tasks into scoped skills, attribute outcomes to responsible branches, and coordinate planning, execution, diagnosis, and validated release.

    Source: §1 · p. 4

  • Results

    RoboRSI has the highest success rate across four benchmarks, exceeding the strongest baseline by 11.0 points on LIBERO-PRO.

    Source: §1 · p. 4

  • Takeaways & Limitations

    Stable skill sequences can be consolidated into compound skills for later executions to select and monitor.

    Source: §1 · p. 4

Abstract

from arXiv · show

A generalist robot should not only perform diverse tasks but also improve through experience, turning what it learns during execution into capabilities that later tasks can reuse. Robot agents that act through code can already repair programs from execution feedback, yet it remains a central challenge to organize this experience around the task structure that gives it meaning, so that each repair is attributed to the responsible capability, supported by execution evidence, and validated before it is reused. We introduce RoboRSI, a robot self-improvement system built on Top-Down Skill Refinement (TSR). TSR decomposes tasks into compound, atomic, and base skills with scoped responsibilities and explicit input--output contracts, attributes each execution outcome to the responsible branch, and confines revision to that branch. Building upon this structure, a Manager, Planner, Engineer, and Reviewer coordinate planning, execution, diagnosis, and the validated release of new skills, while people steer the process through objectives and corrections; stable skill sequences are further consolidated into reusable compound skills. On a mobile manipulator, RoboRSI develops multi-object household cleanup over 104 rounds. In simulation, it achieves the highest success rate on LIBERO, LIBERO-PRO, LIBERO-Plus, and RoboTwin, exceeding the strongest baseline by 2.7 to 11.0 percentage points.

1. Introduction

RoboRSI addresses the challenge of turning execution feedback into reusable capabilities while keeping repairs connected to task objectives and responsible skills. It combines human steering and agent-facing structure, then evaluates the system on physical cleanup and four simulation benchmarks.

Motivation

  • The paper frames robot self-improvement as sustained capability accumulation, beyond completing individual episodes.Execution feedback is presented as a resource for capabilities that persist across tasks.Source: §1 · p. 4
  • Execution experience becomes reusable only when repairs remain tied to their objective, responsible capability, and validation evidence.Without this structure, repeated trajectories crowd context, fixes can drift from the objective, and people must decide where each fix belongs.Source: §1 · p. 4
  • RoboRSI pairs human-friendly steering of objectives and constraints with agent-friendly interfaces and bounded responsibilities.People also provide domain knowledge and safety judgments at the service level.Source: §1 · p. 4

Contributions

  • TSR decomposes tasks into compound, atomic, and base skills, assigns outcomes to responsible branches, and confines revisions to those branches.A Manager, Planner, Engineer, and Reviewer coordinate planning, execution, diagnosis, and validated release; stable sequences become reusable compound skills.Source: §1 · p. 4

Evaluation

  • RoboRSI exceeds the strongest baseline by 5.3 points on LIBERO, 11.0 on LIBERO-PRO, 5.7 on LIBERO-Plus, and 2.7 on RoboTwin, with the highest success rate on all four.Physical multi-object cleanup skills were developed on a mobile manipulator over 104 rounds.Source: §1 · p. 4

2. Related Work

Prior work uses code, execution feedback, and reusable skills for robot behavior, but leaves open how repairs should be assigned and validated across tasks. RoboRSI places repair within a task–skill hierarchy and reuses capabilities through explicit interfaces.

Code as Policy

  • When repaired programs are retained and reused, deciding which skill owns a repair matters because changes can affect every task that calls it.RoboRSI uses TSR to assign repairs to the responsible branch and preserve composition with surrounding skills.Source: §2 · p. 5

Recursive Self-Improvement (RSI)

  • RoboRSI extends failure-detection and repair approaches by selecting which skill to change and validating the change before later tasks reuse it.It attributes failures to the earliest failing node, checks revisions against recent histories of affected tasks, and reuses capabilities through explicit interfaces.Source: §2 · p. 5

Large Language Models for Robot

  • RoboRSI complements robot planners, foundation models, and feedback-driven agents at the system level with reusable skills and execution trajectories.Collected trajectories can also train policies that return as tools in the same hierarchy.Source: §2 · p. 6

3. Method

RoboRSI makes human intent and agent work explicit to keep repeated robot execution aligned with task objectives. Its hierarchy supports scoped revisions, validation, and reuse across tasks.

3. Method

  • RoboRSI addresses repeated-run failures in which fixes are applied where errors appear, scene-specific edits displace the objective, and people must read logs to decide repair scope.It makes both human intent and agent work explicit.Source: §3 · p. 6

3.1. Overall Framework

  • People steer objectives and corrections, while agents use a task–skill hierarchy with declared responsibilities, interfaces, and execution records.Task families group shared goals and constraints; atomic tasks are achieved by skills with postconditions, and stable skill sequences become parameterized compound skills.Source: §3.1 · p. 6

3.2. Top-Down Skill Refinement

  • In the physical cleanup examples, TSR localizes a failed release to the Place Object branch and revises destination-view handling; both revisions passed offline replay and regression tests.Stable branches can become compound skills, shifting later execution toward skill selection and monitoring.Source: Figure 3 · p. 8, §3.2 · pp. 8–9
  • TSR uses execution evidence to locate the earliest failing node, limits patches to the affected scope, and validates them against recent executions before release.If an interface change is needed, the revision scope expands to include callers; validation includes review, functional tests, and replay without regression.Source: §3.2 · pp. 7, 9

3.3. Multi-Agent System

  • The Manager decomposes tasks and controls revision admission and release; the Planner plans, the Engineer executes, and the Reviewer diagnoses and proposes patches.Each agent receives only the hierarchy portion relevant to its step, keeping context short and changes traceable to diagnoses.Source: §3.3 · p. 9

4. Experiment

The experiments combine physical household cleanup with simulation benchmarks, self-iteration, case studies, and component comparisons. They examine task performance, transfer, recovery, and skill reuse across changing tasks and scenes.

4.1. Physical Robot Case Study

  • Across the 104-run development period, selected cleanup runs progressed unevenly; in a two-scene execution, all five first-scene and four second-scene objects were placed on site after rearrangement, with configuration unchanged.Source: §4.1 · p. 10

4.2. Evaluation Design

  • The evaluation compares serial skill improvement with fixed agents, uses LIBERO-Plus to test a fixed transferred library, and includes component studies across 230 tasks and four benchmarks.Methods share a backbone, interface, interaction budget, task panels, and simulator verdict; RoboRSI alone revises skills across episodes in the online benchmarks.Source: §4.2 · p. 11, §4.3 · p. 12

4.3. Main Simulation Results

  • RoboRSI achieves the highest success and coverage on every benchmark; on LIBERO-PRO it exceeds OpenETA by 11.0 points in success and 18.3 points in coverage.On LIBERO-Plus, the frozen source library exceeds OpenETA by 5.7 points and solves at least one perturbed instance of 28 of 30 tasks.Source: Table 1 · p. 12, §4.3 · pp. 12–13, Table 2 · p. 13

4.3.2. One-Day Self-Iteration

  • In one-day self-iteration, coverage grew within five rounds from 32 to 71 of 120 LIBERO tasks and from 43 to 80 of 120 LIBERO-PRO tasks.The five LIBERO rounds took 7.0 hours of active execution; these cumulative counts are separate from the main evaluation.Source: §4.3.2 · p. 14, Figure 7 · p. 14

4.4. Case Study

  • Case studies show RoboRSI recovering from rejected or unverified placements and drawer operations, while a validated grasp_object revision was invoked in 42 successful episodes on other tasks.The revision was developed on one task; Figure 8 shows reuse on three tasks that had failed in the earlier frozen evaluation.Source: §4.4 · pp. 14–15, Figure 8 · p. 16

4.5.1. Role Separation

  • On RoboTwin, separate Planner, Engineer, and Reviewer roles solved 36 of 50 tasks versus 9 for a single agent; TSR localized a failed task’s repair and released its candidate after success on both seeds.The flat-revision candidates were not released: one failed on a seed, and a further candidate again succeeded on only one seed.Source: §4.5.1 · p. 15, Table 4 · p. 16, §4.5.2 · p. 17

4.5.3. Code Consolidation

  • Across 600 matched episodes, code consolidation raised success from 21.5% (129/600) to 29.0% (174/600), a paired gain of 7.5 percentage points.On a separate 118-task efficiency sample, median token consumption, model calls, and wall time fell by 29.4%, 27.2%, and 17.0%, respectively.Source: §4.5.3 · p. 17, Figure 10 · p. 18

4.5.5. Learning Policies from Execution Data

  • With the same demonstrations and training budget, chunk-relative joint targets yielded 4/10 bell training-layout successes and 6/7 alarm-clock training-layout successes, whereas absolute targets solved neither task in evaluated layouts.For the alarm clock, chunk-relative targets also succeeded on 2/3 collection layouts without training data; training-observation joint-command error fell 57%, from 0.0086 to 0.0037 rad.Source: §4.5.5 · pp. 18–19, Figure 11 · p. 19

5. Conclusions and Future Work

RoboRSI combines Top-Down Skill Refinement and multi-agent coordination to organize execution, diagnosis, local revision, and reusable capability.

5. Conclusions and Future Work

  • RoboRSI combines Top-Down Skill Refinement with multi-agent coordination to organize execution, diagnosis, local revision, and reusable capability.Future work will transfer simulation-acquired experience to physical robots and share skills across robots and platforms.Source: §5 · p. 19

A. TSR Formulation and Analysis

TSR represents tasks and skills as a graph and limits revisions to diagnosed scopes. It validates proposed changes against relevant execution cases and bounds potential aggregate effects by the probability mass of affected tasks.

A. TSR Formulation and Analysis

  • A diagnosis identifies an affected scope, and each proposed patch changes only nodes within that scope unless an outside caller or interface is implicated.When diagnosis implicates an outside caller or interface, the scope expands before another proposal.Source: A. TSR Formulation and Analysis · p. 24
  • Proposed revisions draw on recent failures and selected successful caller cases; review and functional tests can reject a patch, leaving the released graph unchanged.Shared skills require checking relevant caller paths, and accepted revisions become available to subsequent execution.Source: A. TSR Formulation and Analysis · p. 24
  • The affected task set includes tasks whose possible call paths reach the edited scope, accounting for dependencies that may not appear in one rollout.Shared base skills can therefore affect several tasks under different observations.Source: A. TSR Formulation and Analysis · p. 24
  • Under stated dependency, interface, and evaluation assumptions, tasks outside the affected set have unchanged execution distributions and cancel from the aggregate difference.The assumptions exclude untracked changes to shared state, global prompts, or tool configurations.Source: A. TSR Formulation and Analysis · p. 24
  • The bound on aggregate failure-rate change scales with the probability mass of affected tasks and covers both improvements and regressions.Shared base skills may have a larger affected set than task-specific branches, motivating regression-case selection by dependency coverage.Source: A. TSR Formulation and Analysis · p. 24
  • Skills declare typed inputs and outputs, applicability conditions, and validation harnesses; release requires simulator success on at least two declared seeds and Manager approval.Atomic completion requires a postcondition verified on fresh observations after the last action; compound skills are proposed after at least three successful executions with the stated sequence overlap and skeleton conditions.Source: A. TSR Formulation and Analysis · p. 25

B. Physical Evaluation Setting & Protocol

The physical evaluation combines a mobile cleanup development record with selected scene demonstrations and a separately reported placement failure. Runs vary in scene configuration, and Figure 4 reports placement progress rather than fixed-task success rates.

B. Physical Evaluation Setting & Protocol

  • The mobile case used a WheelSingleArm M1 with RGB-D cameras, while the dual-arm setup used two PiPER arms and three RGB-D camera streams.Both platform descriptions report camera-to-arm calibration before experiments; mobile development runs used the same prebuilt navigation map.Source: B. Physical Evaluation Setting &… · pp. 25–26
  • The mobile cleanup development record spans 104 runs with varied objects and receptacle positions, while each run's manifest remains fixed during execution.RoboRSI may revise skills between runs, and candidates are checked against retained, reviewed records before later use.Source: B. Physical Evaluation Setting &… · p. 26
  • Figure 4 retrospectively selects ten runs to illustrate development stages, rather than applying a prespecified sampling rule.Each counted placement in those selected runs was confirmed by on-site personnel.Source: B. Physical Evaluation Setting &… · p. 26
  • Figure 4 plots confirmed placements among the first four items handled, so its values describe progress across varying runs rather than success rates on a fixed task.Runs with fewer than four manifest items are not treated as four-item trials.Source: B. Physical Evaluation Setting &… · p. 26
  • In two cleanup scenes, personnel confirmed every object reached its designated receptacle, with no manual takeover or correction.An operator rearranged objects and receptacles between scenes while skill code, prompts, and policy configuration remained unchanged.Source: B. Physical Evaluation Setting &… · p. 26
  • A separate physical attempt released a grasped bath pouf outside its target bucket, prompting a placement-geometry revision from the execution trace.The revision was checked against retained observations but not in a matched post-repair physical trial.Source: B. Physical Evaluation Setting &… · p. 26

C. Supplementary Experiments

The supplementary experiments classify failures using execution records and simulator outcomes, and separately describe exploratory LIBERO-Plus runs that allowed target-side capability updates.

C. Supplementary Experiments

C.1. Failure Classification

  • Failure categories are assigned by deterministic rules using public tool results and the simulator's terminal outcome; traces matching none of the rules remain unlabeled.The analysis covers 5,008 failed episodes; completion-after-failure uses complete traces with at least one execution failure.Source: §C.1 · p. 27

C.2. Exploratory Development

  • Early LIBERO-Plus runs allowed repeated attempts and target-side capability updates, unlike the main transfer panel, which keeps source capabilities fixed.The main panel evaluates perturbed instances without updating the library obtained on LIBERO.Source: §C.2 · p. 27

C.3. Learned Policies from Execution Data

This section describes learned-policy training from RoboRSI executions and introduces case studies of execution failure, local revision, skill publication, and review-driven code changes.

C.3. Learned Policies from Execution Data

  • Ten click_bell demonstrations yielded 2,161 training samples, while seven click_alarmclock demonstrations supplied training data for the learned policies.Both action representations use the same ACT architecture; evaluation covers ten layouts for each task, including three click_alarmclock layouts without training demonstrations.Source: §C.3 · p. 27

D. Case Studies: Skill Code and Failure-Driven Revision

  • The case studies examine an execution failure and local revision on the same development task and initial states.Separate cases document skill publication and review-driven code revision.Source: D. Case Studies: Skill Code and… · p. 27

D.1. Execution Repair: Removing a Premature Target Rejection

A bell-pressing failure arose when a fixed image-boundary check rejected a compact target; a local revision accepted bounded truncation and passed the stated development check.

D.1. Execution Repair: Removing a Premature Target Rejection

  • The original candidate succeeded in 1/3 trials, while the revised candidate succeeded in 3/3 and was published after meeting the predeclared requirement.Each version ran once on the same three declared initial states, invocation arguments, and independent simulator-success criterion; neither version crashed.Source: §D.1 · p. 27
  • The original check rejected a bell box with lower edge 240 against a fixed threshold of 237, failing before any tap was issued.The task goal and tool interface did not need to change; the responsible code was the association check following get_object_bbox.Source: §D.1 · p. 28
  • The revision permits bounded lower-edge truncation for compact objects while retaining size limits and other feature, depth, arm-state, and motion-endpoint checks.The revised image guard permits a lower edge at 240 but rejects extensive truncation.Source: §D.1 · p. 28
  • On the same recorded post-revision observation, the old code still rejected the box, while the revised code replayed all 22 public calls, including the tap, with simulator success.The revised run received a hover–close–press–retreat report and an independent simulator success verdict.Source: §D.1 · p. 28
  • The validation runs used different GPUs and slightly different initial images, including a one-pixel difference in the first trial’s box boundary.The supervisor supplied the tool arguments, outputs, and failed trace; the Manager revised the code and a separate review followed.Source: §D.1 · p. 28

D.2. Generated Compound Skill: Validation, Publication, and Invocation

The case studies show generated tap skills being reviewed, validated, and published, alongside a review-driven revision that conditions recovery checks on confirmed terminal state.

D.2. Generated Compound Skill: Validation, Publication, and Invocation

  • The exact_waypoint_feature_tap skill obtains a fresh image, localizes a functional point, recovers depth, checks motion endpoints, taps, and checks retreat and terminal state.Its structured outputs describe execution, recovery, and failure reasons; the simulator evaluates task completion.Source: §D.2 · p. 28
  • Figure 12 contrasts the original check rejecting a near-boundary box with the revised check accepting the compact target and the successful post-execution observation.The simulator supplies task outcomes.Source: Figure 12 · p. 29
  • The code preserves the tap procedure while recomputing object association and action targets on each call; xyz is the observed feature position and u,v are image coordinates.screen_tcp checks the candidate endpoint.Source: §D.2 · p. 29
  • The reviewed implementation passed its predeclared click_bell development check with 2/3 simulator successes and no crashes, then was published as a versioned skill.This later skill used its own three-trial development panel.Source: §D.2 · p. 29

D.3. Before/After Code: Review-Driven Terminal Checks

  • The earlier recovery condition could short-circuit occupancy and clearance queries after a failed movement or pose check, and clearance could assume empty arms without confirmation.The earlier candidate was rejected during Manager review.Source: §D.3 · p. 30
  • The revised recovery logic checks terminal occupancy after each attempt and checks clearance only after both arms are confirmed empty.These recorded checks determine whether recovery is established.Source: §D.3 · p. 30
  • The accepted review-driven revision passed the 2/3 simulator development check before publication; the previous candidate stopped at code review and had no simulator runs.An operator separately maintained the camera, low-level interfaces, and supervisor.Source: §D.3 · p. 30

D.4. Physical Skill Revisions in Rounds 25 and 33

Two household-cleanup rounds trace physical-robot failures to specific execution nodes, apply targeted revisions, and validate changes with tests or offline replay.

D.4. Physical Skill Revisions in Rounds 25 and 33

  • Both records came from the mobile-manipulator household-cleanup task, with each revision admitted alongside its trigger run, four recent eligible runs, and changed files.Live iteration paused until validation finished.Source: §D.4 · p. 31
  • Round 25’s failed placement began with a false-negative observation move and ended with the item still held; the revision added a pose recheck, non-releasable checkpoint, geometry requirement, and complete release sequence.Five new regression tests, 172 placement tests, 863 manager tests, and 97 real-robot interface tests passed; the next run required a fresh human scene reset.Source: §D.4 · p. 31
  • Round 33 ended after about 25 minutes with the second item held, after destination selection and incomplete navigation handoff failures.The revision preferred complete non-edge observations, made color advisory during materialization, and routed incomplete navigation to the Planner; offline replay and 28 focused, 866 manager, and 97 interface tests passed, with no unplanned human prompt during execution.Source: §D.4 · p. 31

D.5. Corrections That Required Human Knowledge

Four of 53 physical-development revision triggers required operator knowledge unavailable from the system’s records. These corrections became reusable contracts, checkpoints, or tests on the responsible skills.

D.5. Corrections That Required Human Knowledge

  • 4 of 53 physical-development revision triggers required knowledge the system could not obtain from its own records.The operator supplied corrections across scene resets, grasp speed, placement failure, and holding protection.Source: §D.5 · p. 32
  • A confirmed misplacement or a wrapper released beside the bin required a person to reset the scene, not automatic recovery.The operator specified scene contamination after the Round 23 misplacement and judged the Round 37 placement a failure.Source: §D.5 · p. 32, Figure 14 · p. 32
  • The grasp approach needed to respect the operator-confirmed default linear speed of 0.02 m/s.Source: §D.5 · p. 32, Figure 14 · p. 32
  • Holding protection had to inherit reviewed grip width and force, then freeze motion and request confirmation if the hold was lost.Each correction was retained as a reusable contract, checkpoint, or test on the responsible skill, avoiding the same prompt again.Source: §D.5 · p. 32, Figure 14 · p. 32
Loading 2610.12424v1…