PaperlayerView exact revision on arXiv

Source-linked AI summary

Evo-RL: Iterative Robot Reinforcement Learning from Selective Experience

Kai Ye, Yifan Jin, Qianhang Luo, Hongru Zhang, Jiedong Zhang, Shuyuan Hu, Song-Yuan Wu, Jihai Xu, Jie Zhao, Bo Zhao

arXiv:2610.13345v1cs.RO

TL;DR

Because experience can vary in policy-improvement value, even within successful trajectories, Evo-RL addresses selective reuse by combining human corrections with positive-advantage experience. On a real-world high-precision alignment task, first-round Evo-RL achieved a 30.46% higher attempt success rate than RECAP.

  • Problem

    Experience varies in policy-improvement value, and successful trajectories can contain inefficient behavior, motivating finer-grained assessment of which experience to reuse.

    Source: §I · p. 1

  • Method

    Evo-RL iteratively combines human corrections with selected positive-estimated-advantage experience from demonstrations and successful autonomous rollouts to fine-tune the policy.

    Source: §I · p. 1

  • Results

    30.46% higher attempt success rate (ASR) than RECAP was achieved by first-round Evo-RL on a real-world high-precision alignment task.

    Source: §I · p. 1

  • Takeaways & Limitations

    Experiments support complementary roles for interventions and advantage-guided refinement, alongside gains in attempt success rate and throughput on high-precision and deformable-object manipulation tasks.

    Source: §VI · p. 8

Abstract

from arXiv · show

Existing real-world reinforcement learning methods improve robot policies by combining human corrections with autonomous rollouts. However, accumulated experience varies in its value for policy improvement, and even successful trajectories can contain inefficient or redundant behavior, motivating more selective experience reuse. To this end, we propose Evo-RL, an iterative reinforcement learning method that combines human interventions with advantage-guided experience selection. Evo-RL uses human corrections and experience with positive estimated advantages from successful trajectories to iteratively refine the policy. To support learning from these different experience sources, we introduce RW-RL-Dataset, which spans over 1,000 hours across four robot embodiments. On a real-world high-precision manipulation task, Evo-RL improves attempt success rate over RECAP by 30.46\%. Our code, model checkpoints, and dataset are publicly available.

I. INTRODUCTION · 1 Initialization

Evo-RL selectively reuses experience to iteratively improve real-world robot policies, combining human corrections with positive-advantage experience. The paper also introduces a large, annotated dataset spanning four robot embodiments.

I. INTRODUCTION

  • VLA policies trained on demonstrations can encounter unfamiliar deployment states, motivating improvement through execution experience; prior approaches use human corrections, [8] and RECAP combines these with autonomous rollouts [9].VLA models condition actions on visual observations and language instructions, with advances extending them to more tasks and embodiments,, [4] and improving deployment efficiency.Source: §I · p. 1
  • Because experience value varies within and across trajectories, even successful ones may include hesitation, repeated attempts, or other inefficient behavior.Human corrections address deployment difficulties, while other policy actions in those trajectories may not be useful for updates.Source: §I · p. 1
  • Evo-RL retains human-controlled portions of successful intervention trajectories and selects positive-advantage experience from demonstrations and successful autonomous rollouts for policy updates.It also retains the original demonstrations and increases the sampling weight of selected experience.Source: §I · p. 1
  • Starting from a demonstration-trained policy, Evo-RL trains a value model on demonstrations, interventions, and autonomous experience, with rewards accounting for execution time and human takeovers.Estimated advantages guide selection, and the updated policy is redeployed for the next iteration.Source: §I · p. 1
  • RW-RL-Dataset spans over 1,000 hours across four robot embodiments and includes task-specific training experience used in the experiments.Source: §I · p. 1
  • 30.46% higher attempt success rate (ASR) than RECAP was achieved by first-round Evo-RL on a real-world high-precision alignment task.Source: §I · p. 1
  • RW-RL-Dataset pairs synchronized camera observations with robot states and actions, with annotations for task instructions, scene context, execution quality, error counts, and subtask intervals.Source: §I · p. 1

2 Experience Collection … C. Real-World RL and Iterative Policy Improvement

Evo-RL iteratively refines policies using demonstrations, human interventions, autonomous rollouts, and value-guided selection. The related work situates this approach among robot datasets, intervention aggregation, and deployment-time reinforcement learning.

2 Experience Collection

  • Evo-RL deploys the current policy to collect demonstrations, human interventions, and autonomous rollouts, then trains a value model to guide policy refinement across iterations.Complete trajectories train the value model; the updated policy is redeployed for the next iteration.Source: Figure 1 · p. 2

A. Vision-Language-Action Model and Robot Datasets

  • Large-scale robot datasets span household, office, and industrial tasks across single-arm, dual-arm, and humanoid platforms, supporting reusable VLA pretraining and fine-tuning on new tasks.Policies may still fail in situations poorly covered by datasets, where human interventions provide targeted corrections.Source: A. Vision-Language-Action Model and… · p. 2

B. Human Interventions and Data Aggregation

  • DAgger [6] aggregates expert actions queried for states visited by the current policy, while human interventions can provide corrective demonstrations in out-of-distribution situations before or after errors.These corrections can be incorporated into training data to improve responses when similar situations recur.Source: B. Human Interventions and Data… · p. 2
  • IWR [8] increases the relative weight of intervention samples and uses non-intervention data to regularize updates, but corrective labels alone do not indicate likely failures or trajectory quality.Other methods let operators choose when to intervene or trigger assistance using state novelty or failure risk.Source: B. Human Interventions and Data… · p. 2

C. Real-World RL and Iterative Policy Improvement

  • RECAP [9] uses value-function estimates from demonstrations, interventions, and autonomous rollouts to guide iterative improvement, motivating Evo-RL’s combination of human corrections and autonomous experience.Other deployment-time approaches include RLT [19], which refines action chunks with a lightweight actor–critic while keeping the VLA frozen, and LaST-R1 [20], which jointly optimizes reasoning and execution in latent space.Source: C. Real-World RL and Iterative Policy… · pp. 2–3
  • Early VLA reinforcement learning aligned robot trajectories with human intent offline, including GRAPE, which uses Direct Preference Optimization.Source: C. Real-World RL and Iterative Policy… · p. 2
  • RW-RL-Dataset spans over 1,000 hours of real-world interaction across diverse robot embodiments and tasks, with hierarchical semantic annotations and synchronized multimodal observations and control signals.Source: Figure 2 · p. 3

III. METHOD · A. Demonstration-Based Policy Initialization

Evo-RL begins with demonstration-based policy initialization, then iteratively refines the policy through human interventions and advantage-guided updates using accumulated experience.

III. METHOD

  • Evo-RL sequences demonstration-based initialization, intervention-based fine-tuning, and advantage-guided policy updates from accumulated experience.The intervention stage precedes advantage-guided refinement in each iterative data loop.Source: §III · p. 3
  • After intervention-based refinement, Evo-RL uses accumulated experience for advantage-guided policy updates.These updates form the second complementary refinement stage in the iterative loop.Source: §III · p. 3

A. Demonstration-Based Policy Initialization

  • The policy conditions action chunks on camera images, proprioceptive state, and a task instruction.The observation is defined as camera images and robot proprioceptive state; H denotes the action-chunk length.Source: A. Demonstration-Based Policy… · p. 3
  • The initial policy is fine-tuned on teleoperated demonstrations using a flow-matching objective.The policy follows the π0.5 architecture, combining a PaliGemma vision-language backbone with a flow-matching action expert.Source: A. Demonstration-Based Policy… · p. 3
  • Algorithm 1 initializes the policy with πK+1 = SFT(πK, Ddemo) and starts the automatic, demonstration, and training datasets as empty sets.The algorithm takes demonstrations, human corrections, and a number of rounds as inputs.Source: A. Demonstration-Based Policy… · p. 3
  • During intervention refinement, the deployed policy collects human corrections, updates Dinterv, and proceeds to a supervised fine-tuning update.This stage targets challenging scenarios encountered during deployment.Source: A. Demonstration-Based Policy… · p. 3, §III · p. 3

2. Advantage-Guided Policy Fine-Tuning

Evo-RL initializes a task policy by behavior cloning, then fine-tunes it using high-advantage segments selected from demonstrations and autonomous rollouts, with demonstration samples receiving elevated probability.

Advantage-Guided Policy Fine-Tuning

  • High-advantage segments from demonstrations and autonomous rollouts are selected for training after value estimation, while demonstration samples receive elevated probability.The procedure trains the value function on demonstrations, interventions, and autonomous rollouts before selecting segments from demonstrations and autonomous data.Source: §2 · p. 3
  • For demonstration samples, the action expert predicts velocity ε − a_t from observation and language after Gaussian noise and a flow time τ ∈ [0, 1] are sampled.The noisy action chunk is constructed from the sampled noise and flow time, and the objective trains the expert to predict the corresponding velocity.Source: §2 · p. 3
  • Behavior-cloning fine-tuning produces the initial task policy π0, but its capability and generalization remain bounded by demonstration quality, scale, and diversity.The initial policy is deployed to collect experience for subsequent improvement.Source: §2 · pp. 3–4

B. Policy Refinement with Human Interventions

Evo-RL gathers operator corrections when the policy struggles, then iteratively fine-tunes from successful human-controlled segments alongside demonstrations and redeploys the updated policy.

B. Policy Refinement with Human Interventions

  • Operators intervene when the policy struggles to recover or progress in poorly demonstrated states, then return control to autonomous operation.Interventions collect corrective actions during human-supervised rollouts.Source: B. Policy Refinement with Human… · p. 4
  • Each frame records the task instruction, images, proprioceptive state, executed action, and a control-source label distinguishing human corrections from policy actions.Only human-controlled data from successful intervention trajectories is retained for Dinterv.Source: B. Policy Refinement with Human… · p. 4
  • Training preserves contiguous human-controlled segments, excludes policy actions, mixes intervention data with demonstrations, and repeats correction collection after redeployment.The policy is fine-tuned with the same loss, sampling from Dinterv with probability p and Ddemo with probability 1 − p.Source: B. Policy Refinement with Human… · p. 4

C. Reinforcement Learning for Policy Refinement · 1) Reward Design with Time and Intervention Costs:

Evo-RL uses execution-time and intervention costs to estimate segment-level advantages, then selects experience for iterative policy fine-tuning. Its reward design treats takeovers as negative feedback and encourages faster, less assistance-dependent behavior.

C. Reinforcement Learning for Policy Refinement

1) Reward Design with Time and Intervention Costs:

2) Value Function Learning from Trajectory Returns: · 3) Advantage-Guided Policy Fine-Tuning:

Evo-RL learns a vision-only categorical value model from normalized trajectory returns, then uses estimated advantages to select segments for policy fine-tuning alongside human corrections.

2) Value Function Learning from Trajectory Returns:

  • The vision-only value model predicts a distribution over 201 evenly spaced bins spanning [−1, 0] from camera images.A separate model is trained for each task, independently of the policy.Source: 2) Value Function Learning from… · p. 4
  • Normalized returns closer to zero indicate lower combined costs from remaining execution time and human takeovers.Returns are divided by a fixed, task-specific positive scale so the targets lie in [−1, 0].Source: 2) Value Function Learning from… · p. 4
  • Each normalized return becomes a two-hot target, with weights linearly interpolated between adjacent bin centers, and sampled frames train the model with cross-entropy loss.Source: 2) Value Function Learning from… · p. 4
  • The scalar value estimate is the probability-weighted mean of the bin centers.Source: 2) Value Function Learning from… · p. 4

3) Advantage-Guided Policy Fine-Tuning:

  • Advantage estimates use normalized rewards and a lookahead of at most n ≤ H frames, capped by the remaining trajectory, with terminal value fixed to zero.The normalized reward is rt/Zℓ, and the terminal value is Vϕ(oT, ℓ) = 0.Source: 3) Advantage-Guided Policy Fine-Tuning · p. 4
  • Each training round uses estimated advantages to select promising segments from demonstration data and available autonomous data.Source: 3) Advantage-Guided Policy Fine-Tuning · p. 4
  • Selected segments are mixed with human corrections to fine-tune the policy using the same flow-matching loss.Selection and sampling details are deferred to Section V-E.Source: 3) Advantage-Guided Policy Fine-Tuning · p. 5

IV. RW-RL-DATASET · A. Embodiments, Modalities, and Collection Regimes

RW-RL-Dataset organizes demonstrations, human interventions, and autonomous executions for iterative robot policy improvement, spanning four embodiments, long-horizon real-world data, semantic annotations, and multiple collection regimes. Its scale and breadth cover more than 1,000 hours, nine scene domains, and over 30 task templates.

IV. RW-RL-DATASET

  • The dataset’s four dimensions are embodiment diversity, long-horizon real-world executions, frame-aligned semantic annotations, and multi-source collection regimes.Figure 2 summarizes these dimensions as support for iterative policy learning.Source: §IV · p. 5

A. Embodiments, Modalities, and Collection Regimes

  • More than 1,000 hours of real-world interaction span four embodiments: AgiBot G2, Galaxea R1Lite, AgileX PiperX, and SO-101.The platforms span industrial mobile manipulation, modular dual-arm systems, and low-cost tabletop learning.Source: A. Embodiments, Modalities, and… · p. 5
  • G2 and R1Lite provide multiple RGB views alongside robot state, pose, sensor, and action data, with their sensor configurations varying by platform.G2 has eight RGB streams; R1Lite has four. Their recorded signals include joint states, end-effector data, and platform-specific gripper, base, chassis, or inertial readings.Source: A. Embodiments, Modalities, and… · p. 5
  • PiperX and SO-101 are dual-arm systems with external and wrist RGB cameras; PiperX uses 14-dimensional state and action vectors, while SO-101 uses 12-dimensional vectors.Source: A. Embodiments, Modalities, and… · p. 5
  • The dataset spans nine scene domains and more than 30 task templates, including object sorting, precision insertion, garment folding, and box stacking.Domains include stores, industrial production, kitchens, homes, care environments, laboratories, warehouses, hotels, and education or training settings.Source: A. Embodiments, Modalities, and… · p. 5
  • Collection episodes are classified as demonstrations from human execution, autonomous policy executions, or interventions combining both.Intervention episodes include frame-level controller labels identifying human-correction segments.Source: A. Embodiments, Modalities, and… · p. 5

B. Data Organization and Hierarchical Semantic Annotations · V. EXPERIMENTS · A. Experimental Setup and Task Definitions

The paper organizes robot experience with synchronized, hierarchical semantic annotations and evaluates intervention data and policy improvements across two manipulation tasks and robot platforms. The setup specifies trial limits, task units, and the folding task’s manual recovery procedure.

B. Data Organization and Hierarchical Semantic Annotations

  • Dataset episodes link visual observations, robot states, and actions by episode identifiers and timestamps, with semantic annotations at dataset, episode, and frame levels.Dataset fields cover task, embodiment, difficulty, objects, and coordination or precision requirements; episode fields cover instructions, scenes, outcomes, execution quality, mistakes, and action source.Source: B. Data Organization and Hierarchical… · p. 5
  • Frame annotations mark subtask intervals and label human versus policy actions; interval records include instructions, manipulated objects, the active arm, and operation attributes.Annotations share a unified timestamp system for cross-reference.Source: B. Data Organization and Hierarchical… · pp. 5–6

V. EXPERIMENTS

  • Figure 4 shows chronological episode sequences for bolt insertion and garment folding, illustrating task progression in RW-RL-Dataset.Each row depicts one episode from left to right.Source: Figure 4 · p. 6, §V · p. 6
  • Experiments compare intervention data with additional demonstrations under the same data budget and assess policy improvements across successive training stages.The paper also describes its value-model choice and advantage-guided training-data mixture.Source: §V · p. 6

A. Experimental Setup and Task Definitions

  • Experiments use PiperX and SO-101, initialize policies from π0.5, and deploy them with real-time chunking (RTC).Source: A. Experimental Setup and Task… · p. 6
  • Each trial lasts at most five minutes and is a failure if the task is not completed within that limit.Source: A. Experimental Setup and Task… · p. 6
  • In PiperX insertion, the robot selects, grasps, aligns, and inserts five screws into sleeves, with either arm able to insert into a sleeve held by the other.Each unit Uj is one screw insertion.Source: A. Experimental Setup and Task… · p. 6
  • In SO-101 folding, the robot folds five T-shirts sequentially through retrieval, spreading, folding in half, folding sleeves, and folding the hem toward the collar.If a garment enters an erroneous state, an operator restores it to its pre-failure-stage state without fully smoothing it.Source: A. Experimental Setup and Task… · p. 6

B. Evaluation Metrics · C. Data Efficiency of Human Interventions · D. Intervention and Real-Robot RL

Evo-RL combines human corrections with advantage-guided experience selection, and the evaluations examine data efficiency, training contributions, and real-robot performance. Across the reported comparisons, intervention data and iterative refinement are associated with improved task metrics.

B. Evaluation Metrics

  • For PiperX and SO-101, ASR is successful attempts divided by attempts across U1–U5, while TP counts successful tasks per hour.Evaluation stops at N = 10 for PiperX and N = 20 for SO-101; each Uj reports stage completions and attempts, and zero attempts means timeout before that stage.Source: B. Evaluation Metrics · p. 6

C. Data Efficiency of Human Interventions

  • With matched four-hour budgets, SFT + Int. raises ASR from 27.03% to 51.52% and TP from 12.00 to 20.40 versus SFT.At eight hours, the mixture reaches 78.00% ASR versus 78.26% for 16-hour SFT, with TP of 53.04 versus 53.27.Source: C. Data Efficiency of Human… · p. 6, Table I · p. 7

D. Intervention and Real-Robot RL

  • On PiperX, Evo-RL achieves 88.24% ASR versus RECAP’s 57.78% and 60.61 TP versus 35.54; a second round raises ASR to 92.16% and TP to 67.14.The second-round comparison is 88.24% to 92.16% ASR and 60.61 to 67.14 TP.Source: Table II · p. 7, D. Intervention and Real-Robot RL · p. 7
  • On PiperX, combining interventions with advantage-guided refinement yields 88.24% ASR versus 78.72% for SFT + Int. and improves TP by 26.3%.Both SFT + Int. and SFT + RL improve over SFT, with corrections providing the larger gain; Evo-RL also trains on high-advantage segments from successful autonomous rollouts.Source: Table II · p. 7, D. Intervention and Real-Robot RL · p. 7

C. Data Efficiency of Human Interventions

  • Interventions target states induced by policy errors, while teleoperation may rarely cover them; corrections at critical moments provide experience relevant to current policy failures.The policy runs autonomously for most of an episode, with human supervision intervening at a few moments.Source: C. Data Efficiency of Human… · p. 7

D. Intervention and Real-Robot RL

  • Evo-RL policy fine-tuning takes approximately 4 h 52 min and 1,511 steps, compared with RECAP’s estimated 64 h and 102,111 steps.Both use eight A800 80 GB GPUs; RECAP’s estimate is for 2.5 epochs.Source: D. Intervention and Real-Robot RL · p. 7
  • The human-control time fraction falls from 15.30% for SFT to 9.56% for Evo-RL, alongside higher ASR and TP.Source: D. Intervention and Real-Robot RL · p. 7
  • On SO-101 folding, Evo-RL raises ASR from 55.56% to 76.67% and TP from 75.78 to 90.00 versus SFT.At U2, successful completions rise from 17 to 19 while attempts fall from 76 to 36.Source: D. Intervention and Real-Robot RL · p. 8

E. Value Learning and Advantage-Guided Sampling · VI. CONCLUSION

Evo-RL combines human corrections with advantage-guided experience selection to iteratively improve policies; its value-learning pipeline selects informative experience, and the paper reports benefits across manipulation tasks and introduces RW-RL-Dataset.

E. Value Learning and Advantage-Guided Sampling

  • Task-specific, vision-only value models use a SigLIP visual encoder and MLP head, trained on randomly sampled frames from demonstrations, interventions, and autonomous rollouts.Source: E. Value Learning and Advantage-Guided… · p. 8
  • Evo-RL value training takes approximately 2 h 19 min on three A100 GPUs, compared with RECAP’s 3 h 46 min on eight A800 80 GB GPUs.Source: E. Value Learning and Advantage-Guided… · p. 8
  • For demonstrations and successful autonomous rollouts, Evo-RL jointly selects nonterminal starts in the top 10% by positive advantage, merges adjacent or overlapping chunks, and weights selected data differently by source.Selected autonomous segments join demonstrations and human corrections for fine-tuning; selected demonstration segments receive increased sampling weight.Source: E. Value Learning and Advantage-Guided… · p. 8
  • Selected screw-insertion states mark grasping, insertion, and placement transitions; states near completion can have high value but low advantage, showing why value alone is insufficient for selection.Selection is determined by advantage at each start, not high state value alone.Source: E. Value Learning and Advantage-Guided… · p. 8

VI. CONCLUSION

  • Evo-RL improves attempt success rate and throughput on high-precision and deformable-object manipulation tasks; on PiperX, it outperforms RECAP and either update stage alone, with further gains in a second iteration.The supplied conclusion gives no numerical values for these comparisons.Source: §VI · p. 8
  • Under matched human-control budgets, human interventions outperform additional demonstrations.Source: §VI · p. 8
  • Evo-RL iteratively combines human corrections to address deployment difficulties with estimated advantages to guide subsequent experience selection and sampling.Source: §VI · p. 8
  • RW-RL-Dataset brings together demonstrations, interventions, and autonomous experience, including the training data used in the evaluation.Source: §VI · p. 8

Related papers

Loading 2610.13345v1…