PaperlayerView exact revision on arXiv

Source-linked AI summary

$R^2E:$ A Role-driven Reward Evolutionary Framework for Automated Reward Function Design

Shouhao Chang, Xuan Liu, Hongye Zhu, Xinning Chen, Shigeng Zhang

arXiv:2610.32439v1cs.RO

TL;DR

Reward design is important in RL, but manual engineering requires domain expertise and LLM-based methods have limited structured exploration and historical retention. R2E combines specialized roles, dynamic scheduling, and a Global Elite Pool for reward evolution. Experiments report consistent gains over baselines across robotic tasks, while the iterative search still requires substantial computation and time.

  • Problem

    Manual reward engineering depends on domain expertise, while LLM-based frameworks can lack structured exploration and persistent reuse of high-performing rewards.

    Source: §1 · pp. 1–2

  • Method

    R2E coordinates Explorer, Guardian, and Evolver roles through dynamic scheduling and a Global Elite Pool that retains strong rewards as references for subsequent search.

    Source: §1 · p. 2

  • Results

    R2E consistently outperforms baselines across 14 IsaacGym and Bi-DexHands tasks, with especially clear improvements on manipulation tasks.

    Source: §1 · p. 2, §5.3 · p. 7

  • Takeaways & Limitations

    The experiments support role-driven evolutionary reward search as an effective approach across the evaluated robotic locomotion and manipulation tasks.

    Source: §1 · p. 2, §6 · p. 10

    R2E's iterative LLM interactions and repeated RL training cycles incur substantial computational cost and time overhead.

    Source: F Limitation · p. 31

Abstract

from arXiv · show

Designing effective reward functions is a critical challenge in Reinforcement Learning (RL), which traditionally requires costly manual trial-and-error. While Large Language Model (LLM)-based methods have shown promise in reward design, undifferentiated sampling and limited use of historical feedback often lead to insufficient exploration of the reward space and optimization instability. To address these challenges, we propose $R^2E$, a Role-driven Reward Evolutionary framework for automated reward function design that structures the process through explicit role specialization. $R^2E$ employs multiple LLM roles with complementary objectives: an Explorer that promotes novelty to expand the search space, a Guardian that performs conservative refinements to improve training stability, and an Evolver that recombines high-performing rewards via crossover and mutation to integrate effective structures. A dynamic scheduling strategy coordinates these roles, progressively shifting the search from broad exploration to focused exploitation. To mitigate optimization instability and provide a medium for cross-role collaboration, $R^2E$ incorporates a Global Elite Pool that retains the best-performing rewards to guide subsequent generations, thereby enabling coordinated interaction among roles and ensuring a reliable and consistent refinement process. Extensive experiments on multiple robotic tasks demonstrate the effectiveness of our method.

1 Introduction

Reward design shapes RL behavior and training stability, but manual engineering requires domain expertise and existing LLM approaches lack structured exploration and persistent memory. R2E uses specialized roles, dynamic scheduling, and a Global Elite Pool; it is reported to outperform baselines across 14 robotic tasks.

1 Introduction

  • Reward design directly shapes agent behavior and training stability, while manual reward engineering depends heavily on domain expertise.IRL and preference-based RL can reduce expert reliance, but typically incur prohibitive physical-environment data costs and yield black-box representations.Source: §1 · p. 1
  • Existing LLM-based reward frameworks use undifferentiated sampling and immediate-iteration feedback, limiting structured exploration and retention of historically strong rewards.Stochastic generation can produce inferior candidates in later iterations, causing unstable optimization and potential performance regression.Source: §1 · p. 2
  • R2E coordinates Explorer, Guardian, and Evolver roles with dynamic scheduling that shifts search from broad exploration toward stable refinement.Source: §1 · p. 2
  • The Global Elite Pool retains top-performing rewards as persistent references for cross-role refinement and is intended to mitigate optimization instability.Source: §1 · p. 2
  • R2E consistently outperforms baselines across 14 IsaacGym and Bi-DexHands tasks, discovering high-performing rewards for locomotion and manipulation.Source: §1 · p. 2

2 Related Work

Prior work has used LLMs to automate reward design and iteratively refine rewards from environmental feedback. R2E builds on evolutionary frameworks but replaces role-agnostic sampling with role-driven specialization.

2 Related Work

  • Early LLM reward-design systems such as L2R and Text2Reward relied heavily on task-specific templates, few-shot prompting, and human feedback.Source: §2 · pp. 2–3
  • Eureka established an autonomous cycle of reward generation, training, and reflection using environmental feedback, while later methods added strategies such as ELO scoring or Monte Carlo Tree Search.Revolve still incorporates costly human feedback.Source: §2 · p. 3
  • R2E replaces the role-agnostic stochastic sampling used by evolutionary methods including Eureka with parallel role-driven specialization to balance exploration diversity and stable performance.Source: §2 · p. 3

3 Problem Setting

The paper formulates automated reward design as a bi-level optimization: an outer loop searches LLM-generated rewards, while an inner RL loop learns a policy under each fixed reward. Rewards are evaluated and evolved using task-level performance.

3 Problem Setting

  • Reward design is a bi-level optimization in which an LLM-generated reward is selected in the outer loop and an RL algorithm learns a policy under that fixed reward in the inner loop.Source: §3 · p. 3
  • The outer-loop objective J(π_r) is expected task-level performance; it is task-specific on IsaacGym and the success rate on Bi-DexHands, and is not optimized directly by inner-loop RL.Source: §3 · p. 3
  • At each iteration, specialized roles generate candidates conditioned on a fixed-capacity Global Elite Pool; candidates that outperform its worst member replace it, and the best pool reward is selected after a predefined number of iterations.Figure 2 describes an initial exploration stage, dynamically scheduled role-driven iterations, and final selection and reevaluation of the top reward across five random seeds.Source: §3 · p. 3, Figure 2 · p. 4

4 Method

R2E combines role-driven reward design with a Global Elite Pool to balance diversity and stability in search. Its process moves from broad initialization to role-guided evolution and final selection.

4.1 Overview

  • The first iteration generates unconstrained candidates to preserve structural diversity before role-driven evolution begins.After evolution, R2E selects the pool’s top-ranked reward and re-evaluates it across multiple random seeds.Source: §4.1 · p. 4

4.2 Role-driven Reward Evolutionary Design

  • Explorer seeks novel reward structures, Guardian conservatively refines rewards, and Evolver crosses over and mutates high-performing rewards.The roles query the Global Elite Pool to balance exploration, refinement, and inheritance.Source: §4.2 · pp. 4–5

4.3 Global Elite Pool

  • The Global Elite Pool retains the best-performing rewards as persistent memory and shared references for the roles.A candidate replaces the pool’s lowest-performing reward only when its task performance is higher; the fixed-capacity pool is set to K = 5.Source: §4.3 · p. 5

4.4 Dynamic Scheduling Strategy

  • With N = 16 candidates per iteration, scheduling initially favors Explorers, then shifts toward Guardians and Evolvers as the search nears its limit T.This reallocates the fixed budget from broad coverage toward refinement and inheritance.Source: §4.4 · p. 6

5 Experiments

R2E is evaluated against reward-design baselines on 14 IsaacGym and Bi-DexHands tasks. It performs best across the reported benchmarks, while ablations and analyses examine its roles, scheduling, pool, and search budget.

5.1 Experimental Setup

  • Evaluation spans 14 IsaacGym and Bi-DexHands tasks, using task-specific metrics or success rate and Qwen3-Max for R2E and all baselines.IsaacGym includes six tasks and Bi-DexHands eight; sparse and expert-crafted human rewards, Eureka, auto-REvolve, and RF-Agent are among the baselines.Source: §5.1 · p. 6, §5.2 · p. 7

5.3 Main Results

  • R2E achieves the best overall performance across six IsaacGym tasks and surpasses baselines on Bi-DexHands, with substantial margins on CatchAbreast and SwingCup.Figure 3 reports the highest mean success rate and competitive stability across Bi-DexHands environments.Source: §5.3 · p. 7
  • On FrankaCabinet, R2E converges rapidly and escapes local optima; on Pen, it maintains high late-stage success rates with lower variance.Figure 4 shows training curves for four representative locomotion and manipulation tasks, with means and standard deviations over five seeds.Source: §5.3 · p. 7, Figure 4 · p. 7

5.4 Ablation Studies

  • Removing role specialization or the Global Elite Pool degrades performance; removing any individual role also hurts, especially the Guardian on ShadowHand.Figure 5 compares performance drops relative to the full method for macro components and specific roles on Ant and ShadowHand.Source: §5.4 · p. 8, Figure 5 · p. 8
  • Dynamic scheduling improves IsaacGym metrics and Pen success rate over fixed role allocation; K = 5 performs best among tested pool sizes.A smaller pool can restrict structural diversity, while a larger pool can admit more sub-optimal parents under a fixed generation budget.Source: §5.4 · pp. 8–9, Table 2 · p. 8, Table 3 · p. 9

5.5 Qualitative Analysis & Case Studies

  • Explorer has the lowest mean and highest variance, while Guardian has the highest mean and minimal variance; in LiftUnderarm, the roles progress from discovery through stabilization to recombination.The Evolver combines elite candidates in the representative evolutionary trajectory.Source: Figure 6 · p. 8, §5.5 · p. 9
  • Across Ant, AllegroHand, and Pen, more iterations and candidates improve performance; most horizon gains arrive by T = 5, while adaptive T/N remains future work.The main comparison uses N = 16 and T = 5 to match the baselines’ evaluation budget.Source: §5.5 · pp. 9–10, Table 4 · p. 10, Table 5 · p. 10

6 Conclusion

R2E replaces undifferentiated reward sampling with collaborative role-driven search, balancing exploration and stable refinement through a persistent pool of strong candidates. Its conclusion reports consistent baseline outperformance on IsaacGym and Bi-DexHands, especially on complex manipulation tasks.

6 Conclusion

  • R2E combines Explorer, Guardian, Evolver, and a Global Elite Pool to balance broad exploration with stable refinement and consistently outperform baselines, especially on complex manipulation tasks.The pool preserves globally best-performing candidates as persistent references across iterations.Source: §6 · p. 10

A Roles Engineering & Templates

A.1 Common Interface

  • The User message stays invariant across roles and iterations, keeping task definitions and environmental context consistent while role constraints vary.This standardization is intended to isolate role-specific behavioral differences from task understanding.Source: §A.1 · p. 13

A.2 Initialization

  • The initial iteration uses a generalist prompt without role-specific priors to generate diverse candidates that seed the Global Elite Pool.This phase prioritizes broad reward-space coverage before evolutionary refinement.Source: §A.2 · p. 13

A.3 Role-Driven Evolution

  • From iteration two, Explorer and Guardian use elite rewards as exemplars, while Evolver uses a streamlined System-message prompt for crossover and mutation.All three roles receive the same User message to maintain task consistency.Source: §A.3 · p. 14

A.3.1 Explorer Role

  • Explorer uses elite rewards as negative references to generate novel reward structures and expand search breadth.Its design targets local-optimum avoidance and structural diversity.Source: §A.3.1 · p. 14

A.3.2 Guardian Role

  • Guardian conservatively refines the current best reward for stability and robustness while preserving its core behavioral logic.It focuses on smoothness, numerical scales, and component balance rather than unrelated sub-goals.Source: §A.3.2 · p. 15

A.3.3 Evolver Role

  • Evolver recombines components from elite rewards and applies controlled mutations to produce hybrid offspring with inherited behavioral traits.It treats high-performing reward functions as genotypes for structured inheritance and continuous optimization.Source: §A.3.3 · p. 16

B R2E Case Example

This section presents a LiftUnderarm case illustrating how R2E’s Explorer, Guardian, and Evolver collaborate to alter reward designs, alongside passages describing the GPU benchmarks used for evaluation.

B R2E Case Example

  • In second and third iterations, the Explorer deliberately departs from existing reward paradigms, combining a grasp incentive, progressively shaped object-to-goal distance, an early upward-motion bonus, near-goal stability bonus, and symmetry term.Source: B R2E Case Example · p. 18
  • In iteration four, the Guardian slows exponential reward attenuation, tightens boundaries and thresholds against exploitation or early triggering, and rebalances overall reward weights.The illustrated weights change to 1.2 for grasp, 2.0 for lift, 0.8 for upward motion, 1.2 for stability, and 0.6 for symmetry.Source: B R2E Case Example · p. 19
  • The Evolver selects the current best pool reward and iteratively combines it with a randomly selected pool reward using component-level crossover and mutation.One referenced candidate uses smooth grasp ramp-up, height-gated goal and stability rewards, symmetry, and an efficiency bonus discouraging excess hand movement after grasping.Source: B R2E Case Example · pp. 20–21
  • The Evolver builds a final reward by extracting functional components from two reference functions and recombining them through crossover and mutation.Source: B R2E Case Example · p. 22
  • The case example shows the Explorer, Guardian, and Evolver collaborating to optimize reward functions.The passage presents this example as demonstrating the roles’ distinct functions and the method’s design rationale.Source: B R2E Case Example · p. 23

C Environment Details

  • The evaluation uses IsaacGym for locomotion and single-arm manipulation and Bi-DexHands for high-dimensional bimanual coordination.The environment specifications cover task observations, action dimensions, and objectives.Source: C Environment Details · p. 23, Table 6 · p. 24

C.1 IsaacGym

  • 2–3 orders of magnitude faster training is attributed to IsaacGym’s GPU-resident simulation and policy training, which avoid CPU data-transfer bottlenecks.The evaluated IsaacGym suite spans locomotion and manipulation tasks, including Ant, Anymal, Quadcopter, AllegroHand, FrankaCabinet, and ShadowHand.Source: §C.1 · p. 23, Figure 7 · p. 24

C.2 Bi-DexHands

Bi-DexHands evaluates bimanual dexterous manipulation in high-dimensional, multi-agent control settings. Its tasks and environment specifications cover coordinated manipulation scenarios, including shared-hand control of objects.

C.2 Bi-DexHands

  • Bi-DexHands tests coordination between two multi-joint Shadow Hands across contact-rich bimanual manipulation tasks.The benchmark includes scenarios such as object handover, coordinated lifting, and throwing.Source: §C.2 · p. 24
  • The selected tasks include synchronized two-agent control for problems such as pouring water in Kettle and collaborative swinging in SwingCup.SwingCup requires two hands to hold and swing a dual-handle cup together.Source: Figure 8 · p. 25, §C.2 · p. 26
  • Table 7 lists each Bi-DexHands environment's observation and action dimensions alongside its task description.The table caption identifies these fields as the high-dimensional spaces and task coordination logic.Source: Table 7 · p. 25

D Implementation Details

The section describes experimental configurations for reproducibility, including simulation settings, LLM sampling strategies, and the code execution pipeline.

D Implementation Details

  • The implementation details cover simulation settings, LLM sampling strategies, and the code execution pipeline for reproducibility.Source: D Implementation Details · p. 26

D.1 RL Training Configuration

R2E’s training and search configuration pairs untuned PPO defaults with dynamically scheduled LLM roles and a persistent elite pool. Across IsaacGym and Bi-DexHands, it converges faster and achieves higher final performance than baselines.

D.1 RL Training Configuration

  • PPO uses official IsaacGym and Bi-DexHands defaults without task-specific hyperparameter tuning; parallel-environment counts are adjusted to saturate GPU memory and maximize throughput.Table 8 lists the environment counts used for the evaluated tasks.Source: §D.1 · p. 26, Table 8 · p. 26

D.2 LLM Generation & Evolutionary Settings

  • Qwen3-Max uses temperature 1.0 across roles, while parallel batch sampling reduces request overhead.The paper says this temperature is intended to retain diversity in reward structure and coefficient selection.Source: §D.2 · pp. 26–27
  • The search runs for five rounds with a top-five Global Elite Pool and 16 candidate rewards per iteration.The paper reports diminishing marginal gains after round five and says the pool keeps top rewards available for crossover.Source: §D.2 · p. 27
  • From iteration 2 onward, a linear schedule partitions 16 samples across roles, shifting from Explorer-led exploration toward Guardian and Evolver refinement; iteration 1 uses role-agnostic initialization.The allocation is described as a simulated-annealing-style transition toward exploitation.Source: §D.2 · p. 27

E.1 Evolution of the Best-So-Far Performance

  • R2E converges faster across most tasks and reaches higher final task-specific metrics on IsaacGym and success rates on Bi-DexHands than baselines.The best-so-far curves track the best performance through each iteration, averaged over five seeds; IsaacGym uses task-specific metrics and Bi-DexHands uses success rate.Source: §E.1 · pp. 27–29, Figure 9 · p. 28, Figure 10 · p. 28
  • Where baselines plateau, R2E continues improving, solving CatchAbreast and finding functional rewards on Pen.The results also report a widening late-iteration gap on PushBlock and Kettle.Source: Figure 10 · p. 28, §E.1 · p. 29

E.2 Computational Cost Analysis

The five-iteration search’s wall-clock cost varies with task complexity, chiefly because RL training scales with environment physics and action-space dimensionality. The reported searches range from under an hour for simpler tasks to at most half a day for dexterous manipulation.

E.2 Computational Cost Analysis

  • R2E’s computational efficiency is primarily determined by parallelization in IsaacGym RL training; Qwen3-Max generation is relatively constant at approximately 1–3 minutes per iteration, while RL training scales with environment physics complexity and action-space dimensionality.Source: §E.2 · p. 29
  • Five-iteration searches take approximately 45–50 minutes for Ant and Quadcopter, 1.5–2 hours for FrankaCabinet and Anymal, and 6–12 hours for Pen and AllegroHand.The timings were measured on a 4× NVIDIA RTX 4090 setup; Table 9 reports representative task times.Source: §E.2 · p. 29, Table 9 · p. 29
  • The authors report near-optimal reward discovery in under an hour for simpler tasks and completion within half a day even for the most challenging dexterous tasks.Each iteration trains 16 distinct policies, with parallelization keeping average training time per candidate efficient.Source: §E.2 · p. 29

E.3 Token Usage and Economic Cost Analysis

R2E’s reported API cost remains below 1.0 RMB per evaluated task, while token use rises with task complexity. The backbone comparison also reports better performance from stronger LLMs, with larger gains for R2E as model capability increases.

E.3 Token Usage and Economic Cost Analysis

  • The cost accounting covers prompt inputs and generated completions across Explorer, Guardian, and Evolver for the full five-iteration search, using China-region batch inference rates.The reported cost model lists separate input-processing and output-generation rates.Source: §E.3 · p. 30
  • 90k–120k tokens are typical for standard locomotion and single-arm tasks, versus 170k to over 210k for high-dimensional bimanual tasks.The authors attribute the approximate 2× increase to verbose dual-arm state descriptions and coordinated reward logic.Source: §E.3 · p. 30, Table 10 · p. 30
  • Across every evaluated task, synthesizing a high-performing reward function with R2E costs under 1.0 RMB (approximately $0.14 USD); even the most intricate task, SwingCup, costs 0.74 RMB.Source: §E.3 · p. 30

E.4 Effect of LLM Backbone

  • Stronger LLM backbones consistently improve performance, and R2E benefits more as backbone capability increases.The authors attribute the pattern to stronger models’ better role specification understanding and instruction following; Table 11 reports means and standard deviations over five seeds.Source: Table 11 · p. 31, §E.4 · p. 31

E.5 LLM Backbone Diversity and Reward Homogenization

The tested LLM backbones use different reasoning approaches that may diversify sampled rewards, but alternating models across iterations does not outperform the strongest single backbone on Ant or Pen.

E.5 LLM Backbone Diversity and Reward Homogenization

  • Qwen3.7-Max builds from available environment signals, DeepSeek-v4-pro decomposes tasks into behaviors, and GPT-5.6-sol-max reasons from likely training failures; the authors suggest this diversity may reduce reward homogeneity.The comparison examines the three models on Ant and Pen.Source: §E.5 · p. 31
  • Alternating the three backbones across iterations does not outperform the strongest single backbone on Ant’s task-specific metric or Pen’s success rate.The authors conjecture that switching models may disrupt reasoning coherence across iterations; Table 12 reports means and standard deviations.Source: Table 12 · p. 31, §E.5 · p. 31

F Limitation

This section outlines R2E’s computational and backbone-model limitations, its algorithm and reported reproducibility materials and experimental variability, and checklist disclosures on human-participant experiments and LLM use.

F Limitation

  • Each task requires multiple LLM interactions and repeated RL training cycles, leaving substantial time and compute overhead despite strong performance with relatively small LLMs.The authors aim to reduce the number of required RL cycles while preserving iterative reward refinement.Source: F Limitation · p. 31
  • Weaker backbone models degrade R2E because it depends on accurate role understanding and nuanced feedback.This reduction in model performance negatively affects the system’s overall effectiveness.Source: F Limitation · p. 31

G Algorithm

  • Algorithm 1 starts with diverse reward seeds, then dynamically allocates Explorer, Guardian, and Evolver candidates, evaluates them through RL, and retains top performers in the Global Elite Pool.The workflow shifts from exploration toward refinement; the pool guides later search and supports novelty and stability.Source: G Algorithm · p. 32

NeurIPS Paper Checklist

  • The checklist reports that Appendix D contains experimental reproduction details, supplementary materials provide code and data, and Section 5.3 reports statistical results and experimental variability.It notes that LLM randomness prevents exact numerical reproduction, while overall result ranges and comparative trends are reproducible.Source: NeurIPS Paper Checklist · pp. 34–35
  • The authors mark IRB review as not applicable because the work involved no human-participant experiments.They state that equivalent ethical review was not required.Source: NeurIPS Paper Checklist · p. 38
  • The authors declare LLM usage and direct readers to Section 4 and Appendices D.2 and E.3 for details.Source: NeurIPS Paper Checklist · p. 38
Loading 2610.32439v1…