PaperlayerView exact revision on arXiv

Source-linked AI summary

A Bird's-Eye View of Iterative Reward Design

Logan Mondal Bhamidipaty, Lauren Robson, Linda Petrini, Shengrui Lyu, Kamal Ndousse

arXiv:2610.04364v1cs.LGcs.AIcs.RO

TL;DR

Reward design in RL requires balancing efficient learning with alignment, while existing IRD methods are difficult to compare and their design choices have limited cross-setting evidence. BIRD provides a shared configurable framework and matched evaluation protocol for comparing, ablating, and combining methods. Experiments find that a few simple choices help consistently, and ERA-U and ERA-S outperform established baselines in their respective categories, though benchmark scores do not reliably correspond to intended behavior.

  • Problem

    Reward design requires expertise and trial and error, while inconsistent implementations and evaluation assumptions leave limited evidence about which IRD methods and design choices work across settings.

    Source: §1 · p. 1

  • Method

    BIRD represents IRD methods as configurations in a shared implementation and evaluates them under matched feedback conditions and policy-training budgets.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    A few simple design choices helped consistently, and ERA-U and ERA-S significantly outperformed established baselines in their respective categories.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Systematic empirical benchmarking and search can identify simple choices that transfer more reliably than many intuition-driven design choices.

    Source: §1 · p. 2, §5 · p. 9

    Higher benchmark scores did not reliably correspond to intended behavior, and model-based feedback may propagate errors that its evaluators do not detect.

    Source: §5 · p. 9

Abstract

from arXiv · show

Designing effective reward functions in RL typically requires substantial expertise and trial and error. Recent work automates this process with LLM-based systems that generate and iteratively improve reward code using policy feedback. However, these methods are often hard to compare because they differ in implementation details, feedback assumptions, and evaluation environments. To address this, we introduce a Benchmark for Iterative Reward Design (BIRD) that expresses existing methods in a unified configuration and evaluation space. This lets us compare algorithms directly, ablate individual design choices, and prototype new components under matched feedback conditions and policy-training budgets. Across MuJoCo, Meta-World, Assistax, and HumanoidBench, we identify a small set of simple design choices that consistently improve performance. Combining these choices yields significantly better performance than the evaluated methods from prior work. Our results highlight the strength of simple baselines and motivate further study of when additional algorithmic complexity improves iterative reward design. Code is available at https://github.com/safety-research/bird.

1 INTRODUCTION

Reward design in RL is laborious and requires balancing learnability against alignment, while existing IRD methods remain difficult to compare. BIRD provides a shared configurable benchmark, and its experiments find that a few simple choices help consistently while ERA-U and ERA-S outperform established baselines in their categories.

1 INTRODUCTION

  • Reward design must balance rewards that are easy to optimize but may be misaligned against aligned rewards that may be too sparse for efficient learning.Finding that balance often requires laborious, error-prone trial and error, even for experts.Source: §1 · p. 1
  • IRD methods use policy feedback in a loop of proposing, training, evaluating, and revising rewards; LLM-based systems automate reward generation and revision.Prior LLM-generated rewards have typically surpassed human-written ones.Source: §1 · p. 1
  • Existing studies offer limited evidence about which design choices transfer across algorithms and environments or remain effective when combined.Differences in feedback assumptions, implementations, model backbones, and policy training also impede fair comparison; benchmark performance alone does not establish alignment.Source: §1 · p. 1
  • BIRD represents reward generation, policy training and evaluation, and reward selection and revision as configurations, enabling comparison and recombination under matched feedback and training budgets.Shared design choices reuse the same implementation.Source: §1 · p. 2
  • ERA-U and ERA-S achieve significantly better performance than established baselines in their respective categories across the environment suites.A small human study also found that IRD rewards can yield high benchmark scores through unnatural or harmful behavior.Source: §1 · p. 2

2 PRELIMINARIES

IRD iteratively designs rewards using feedback from trained policies. Its stages include generating and screening candidate rewards, training policies, evaluating reward performance, and selecting rewards for the next iteration.

2 PRELIMINARIES

  • IRD designs rewards by trial and error based on policy feedback, often using LLMs or VLMs.The loop proposes a reward, trains a policy, evaluates its behavior, and revises the reward as needed.Source: §2 · p. 3
  • Candidate rewards are generated from a task description and environment information, with later iterations able to use feedback from earlier attempts.The environment's reference reward is withheld, so the model designs rewards from scratch; rewards are usually Python functions.Source: §2 · p. 3
  • Reward verification can screen candidates before policy training for coding errors or estimated reward quality.CARD, for example, uses earlier trajectories to check whether candidate rewards rank successful trajectories above failed ones.Source: §2 · p. 3
  • Surviving candidates receive policy training using choices such as the RL algorithm, environment-step budget, and whether to train from scratch or warm-start.One or more policies may be trained for each reward function.Source: §2 · p. 3
  • Supervised methods evaluate trained policies against benchmark scores, while unsupervised methods avoid benchmark scores and can use visual judgments of task completion.Eureka combines benchmark scores with reward-component statistics; RDA uses a VLM to rate completion from rollout frames.Source: §2 · pp. 3–4
  • BIRD depicts methods as configurations, with rows for methods and columns for design choices grouped by IRD stage.Gray cells indicate BIRD defaults, while colored cells show choices set by the algorithm.Source: Figure 3 · p. 4
  • After evaluation, methods select a winner or elite set; greedy selection, populations, search trees, and preference-based comparisons are alternatives.The selection determines which rewards seed the next generation round.Source: §2 · p. 4

3 BIRD

BIRD uses the shared structure of the IRD loop to represent methods as configurations and compare them while limiting implementation differences as confounders. Its representations aim for fidelity, though closed-source papers and deprecated model backbones make complete fidelity difficult.

3 BIRD

  • BIRD leverages the shared structure of the IRD loop to address difficulties in comparing methods with different implementations, backbones, and assumptions.Figure 4 compares methods across four environment suites; ticks show means across tasks and seeds, and bars show 95% BCa bootstrap confidence intervals.Source: §3 · p. 4, Figure 4 · p. 5
  • BIRD represents IRD methods as configurations, reusing implementations for shared choices to help prevent implementation differences from becoming confounders.For example, Eureka and RDA use different feedback sources but share ranking code for their common selection choice.Source: §3 · p. 4
  • Complete fidelity is difficult because some papers lack open-source code and some model backbones have been deprecated.BIRD represents 14 methods, and its implementations are checked against original papers and reference code.Source: §3 · p. 4

4 EXPERIMENTAL RESULTS

The experiments compare published IRD methods, test individual design choices, combine promising choices into two recipes, and examine whether benchmark scores align with human judgments of task completion and naturalness.

4.1 COMPARING IRD METHODS

  • Among six published methods, REvolve is strongest across suites, trailing only RDA on HumanoidBench; other methods lack consistent non-overlapping-CI wins.The authors describe the methods as surprisingly similar performance-wise.Source: §4.1 · p. 5
  • RDA is less consistent on Assistax and HumanoidBench, and gains diminish with iterations as weighted-sum rewards are often coefficient-tuned rather than structurally changed.The authors hypothesize that visual task-completion assessment is difficult in these domains.Source: §4.1 · p. 5

4.2 ABLATING IRD DESIGN CHOICES

  • Thirty choices were manually selected because over 300 configurable choices made exhaustive ablation infeasible; task variability may partly explain inconsistent gains.Source: §4.2 · pp. 5–6
  • Most of 30 ablated choices do not consistently improve performance; four clear helpers are peak own-reward checkpoints, summarized feedback, task-success fitness, and four-turn history.The authors conclude that added algorithmic complexity does not reliably improve IRD.Source: Figure 5 · p. 6, §4.2 · p. 6

4.3 A SIMPLE RECIPE FOR IRD

  • ERA-S and ERA-U rank first and second pooled: ERA-S beats REvolve and ERA-U beats RDA, both with non-overlapping CIs.ERA-U was hill-climbed on four MT10 tasks, while ERA-S was not tuned; the rankings persist when development tasks are excluded.Source: §4.3 · p. 6, Figure 7 · p. 7

4.4 BEYOND BENCHMARK SCORES

  • 9 of 67 policies exceeded 75% reference reward and all were rated unnatural; scripted hopping or rocking also reached comparable scores.The scratch-itch rollout applying over 3 kN of force to a patient's arm illustrates the mismatch between reference reward and human preference.Source: Figure 8 · p. 8, §4.4 · pp. 8–9
  • Completion ratings tracked benchmark scores on Meta-World (ρ = 0.78), but only weakly on Assistax (0.36) and HumanoidBench (0.26), where visual assessment can miss relevant quantities.The authors propose giving VLM judges sufficient information to assess task success.Source: §4.4 · p. 8
  • Reviewers agreed more on completion (ρ = 0.83) than naturalness (ρ = 0.54), and reference rewards tracked naturalness less than completion (0.47).HumanoidBench naturalness ratings were nearly unanimous, unlike ratings on the manipulation domains.Source: §4.4 · p. 8, Table 2 · p. 8

5 DISCUSSION

The discussion argues that empirical search over simple design choices can improve IRD methods, while better benchmark scores do not necessarily align with intended behavior. It also cautions that model-based oversight can miss relevant outcomes and points to possible future capabilities of more capable models.

Two small bitter lessons for reward design

  • A small hill-climb over simple design choices produced ERA-U and ERA-S, supporting a larger role for empirical search in IRD design.The discussion contrasts this result with choices such as tapering candidates per round, which made little difference at equal budget.Source: §5 · p. 9

Alignment

  • Higher benchmark scores did not reliably correspond to intended behavior, and task descriptions often leave how to achieve a goal unspecified.The authors hypothesize that stronger reward-seeking may widen the score–naturalness gap unless naturalness enters the objective or feedback.Source: §5 · p. 9

Scalable oversight

  • Model-based oversight can propagate errors when evaluators miss relevant outcomes; human review corrected obscured VLM judgments in this study.The authors call for systematic checks that evaluators can observe relevant outcomes and track intended goals.Source: §5 · p. 9

The future of IRD

  • The authors anticipate that more capable VLMs may emit reward values directly or control embodiments and write programs for them.Source: §5 · p. 9

6 CONCLUSION

BIRD represents iterative reward design methods as configurable components, enabling matched comparisons, ablations, and combinations. The paper’s conclusion emphasizes that a few simple choices helped consistently, while many design-choice benefits did not transfer reliably; the authors also identify limits in human feedback, method coverage, efficiency analysis, model testing, and domain scope.

6 CONCLUSION

  • In matched experiments, a few simple design choices helped consistently, while many choices—including proposed ones—did not carry over across tasks and methods.A small hill-climb starting from RDA and using components from prior methods produced the two strongest evaluated methods; the authors expect further search to improve them.Source: §6 · p. 9

A TERMINOLOGY

  • The terminology table gives examples for a cubemanipulation task, but the supplied passage does not specify its individual terms.Source: Table 3 · p. 13

B PRIOR WORK

  • Prior work progressed from optimal-reward formulations and search to LLM-guided reward generation, broader search procedures, and alternative feedback from trajectories or VLM judgments.The cited approaches also extend design to policy observations and training distributions.Source: B PRIOR WORK · pp. 13–14

C METHODS IN BIRD

  • BIRD implements 14 published methods, including exhaustive-search and model-predictive-control methods, to represent design choices beyond LLM-written rewards trained with RL.The methods table characterizes their feedback types and notes that Optimal Reward Search and Language to Rewards lack a policy-training feedback loop.Source: C METHODS IN BIRD · p. 14, Table 4 · p. 15
  • The represented methods span reward search and code generation, from exhaustive search and online reward optimization to Eureka’s iterative policy training and Text2Reward’s instruction-based reward coding.Text2Reward includes variants using execution-error feedback, retrieved expert rewards, or human feedback on rollout videos.Source: C METHODS IN BIRD · pp. 14–15
  • Other methods vary candidate evaluation and search, including CARD’s trajectory preference screening, RF-Agent’s tree search, ROSKA’s policy blending, LaRes’s shared replay, and RDA’s VLM subtask scoring.Gran Turismo ranks paired rollout clips with VLM descriptions and human feedback; R* uses trajectory critics and parameter refitting.Source: C METHODS IN BIRD · pp. 15–16
  • LIMEN jointly searches observation maps and rewards, while REvolve uses island-model evolution and human preferences over rollout videos.Source: C METHODS IN BIRD · pp. 15–16

D LIMITATIONS

  • The study’s limitations include author-only human ratings, partial method coverage, no computational-efficiency study, a deliberately small search, untested model-choice effects, possible task-information leakage, and robotic-control-only evaluation.Source: D LIMITATIONS · p. 16

E DESIGN CHOICES IN THE ABLATION STUDY

The ablation study measures the effects of enabling individual design choices across applicable baseline configurations and a selected set of manipulation and locomotion tasks. Its reported effects pool across tasks and baselines, while some interventions change multiple settings together.

E DESIGN CHOICES IN THE ABLATION STUDY

  • The study tests enabled design choices in applicable Eureka, RDA, and CARD configurations, excluding cases where a choice is inactive.Table 5 groups choices by stage of the iterative reward-design procedure, marks BIRD-introduced choices with †, and flags coupled interventions with ‡.Source: E DESIGN CHOICES IN THE ABLATION STUDY · pp. 16, 18, Table 5 · p. 17
  • The ablation covers nine tasks: six Meta-World MT10 tasks and three Gym MuJoCo tasks, with five seeds per experimental cell.The task set spans manipulation and locomotion and includes four MT10 tasks where baselines are not saturated.Source: E DESIGN CHOICES IN THE ABLATION STUDY · p. 18
  • Effects are Glass’s ∆ values pooled over applicable baselines and tasks; Figure 5 shows task-level dots and mean diamonds with 95% BCa intervals over tasks.A ‡ intervention changes two or three coupled settings, so its measured effect applies to that combined intervention.Source: E DESIGN CHOICES IN THE ABLATION STUDY · p. 18

F REWARD DISTANCE ACROSS ITERATIONS

The reward-distance analysis finds that iteration generally does not move selected rewards toward the reference reward, even though the methods’ rewards are closer to it than random rewards. Reward distance also does not reliably predict benchmark score, so policy improvement need not recover the reference reward.

F REWARD DISTANCE ACROSS ITERATIONS

  • On MT10, mean distance rises slightly over five iterations for every method, by at most 0.06; on MuJoCo, only CARD moves closer, by 0.10.Across 40 MT10 method–task combinations, distance fell significantly in one and rose in ten; MuJoCo Eureka and RDA drifted away by 0.06 and 0.08, while Gran Turismo was flat.Source: F REWARD DISTANCE ACROSS ITERATIONS · p. 18
  • On MT10, EPIC distance is about 0.2 for Eureka, RDA, and CARD and about 0.3 for Gran Turismo, versus 0.65 for random rewards; on MuJoCo, methods are 0.3–0.45 versus 0.74.Source: Figure 9 · p. 18, F REWARD DISTANCE ACROSS ITERATIONS · p. 18
  • EPIC distance does not predict benchmark-score rank reliably: pooled Spearman correlations are −0.08 on MT10 and +0.08 on MuJoCo, despite task-level values reaching ±0.4.The authors conclude that search can improve the trained policy without recovering the reference reward.Source: F REWARD DISTANCE ACROSS ITERATIONS · p. 19

G CANDIDATE ALLOCATION ISOQUANT SWEEP

The candidate-allocation sweep finds that feedback helps on some unsolved tasks, while changing candidate counts across rounds adds little at a fixed budget. Across the broader comparison, no method leads on every task; the paper also reports matched training protocols and a limited hill-climb used to develop ERA.

G CANDIDATE ALLOCATION ISOQUANT SWEEP

  • At C = 16, seven MT10 tasks are flat across K; feedback helps on drawer-open, peg-insert, and pick-place, while push trends oppositely within seed variation.The figure plots best task success among the first 16 candidates; K = 16 is a single round.Source: G CANDIDATE ALLOCATION ISOQUANT SWEEP · p. 19, Figure 10 · p. 19
  • At a fixed 80-candidate budget, Taper and Ramp differ little from constant schedules, and neither tapered schedule differs from K = 8 or K = 80 by more than 0.02 average task success.Taper and Ramp are indistinguishable from each other (−0.01, 95% CI [−0.03, +0.01]); K = 16 is the weakest constant arm on these seeds.Source: G CANDIDATE ALLOCATION ISOQUANT SWEEP · p. 19, Table 6 · p. 20

H.1 PERFORMANCE

  • ERA-S has the highest mean on 9 of 20 non-saturated tasks, while ERA-U leads on 3 and four baselines lead on the other 7.The methods tie on HumanoidBench walk; InvertedPendulum is excluded because every method reaches its maximum score.Source: §H.1 · p. 20, Figure 11 · p. 21

H.2 COMPUTE EFFICIENCY

  • CARD uses approximately 10× fewer environment steps and performs worse, though its performance gap is smaller than the compute gap would suggest.The authors interpret this as suggesting diminishing returns from widening the reward search space within iterations, while noting compute efficiency was not optimized.Source: §H.2 · p. 20, Figure 12 · p. 22

I.1 BENCHMARKS AND TASKS

  • The comparison covers 21 tasks across four benchmarks and uses eight candidates per iteration, five iterations, and 1M policy-training steps, except Assistax at 20M.The protocol trains 40 policies per search except CARD, with a common RL optimizer per benchmark rather than each paper’s original optimizer.Source: §I.2 · p. 20, §I.1 · p. 20, Table 7 · p. 23

I.3 POLICY TRAINING

  • Policy training is held fixed across methods within each benchmark; Meta-World and MuJoCo use modest-budget PPO to leave room for reward differences to matter.HumanoidBench and Assistax use their respective benchmark learners, and Assistax policies are evaluated only at the end of training.Source: §I.3 · pp. 22–23

I.4 REWARD SEARCH PROTOCOL

  • Each search returns one reward after five iterations of eight candidates, with methods differing in their settings, feedback, judging, and candidate verification.Candidates are described back to the generator after training; unsupervised arms are judged from video, using three rollouts and 20 evenly spaced frames per rollout.Source: §I.4 · p. 24

I.5 THE HILL-CLIMB BEHIND ERA

  • ERA-U was developed through a limited hill-climb on four unsaturated MT10 tasks, changing one design choice or coupled group at a time and stopping after 24 configurations.Most configurations used 24–30 policy trainings; the team retained design thought over elite crossover after tree-based configurations had collapsed seeds on two tasks, then removed the verifier and added the peak own-reward checkpoint.Source: §I.5 · pp. 25–26

I.6 COMPUTE

The paper reports model-call and token costs for its main results, then describes the human-rating sample and annotation procedure. The human raters were paper authors, familiar with the tasks, and blind to clip source but not study purpose.

I.6 COMPUTE

  • Across approximately 1,500 main-results runs, the generator and judge made about 200,000 model calls and consumed about 2.2 billion tokens.Compute totals in the accompanying table cover benchmark searches, excluding development runs, pilots, and the Section 4.3 hill-climb.Source: §I.6 · p. 26, Table 11 · p. 26

J.1 EVALUATION SETUP

  • The human study rated 257 unique policy-rollout clips across 22 tasks, using two authors who provided 300 usable annotations.Raters were blind to clip source but not the study’s purpose, and were more familiar with the tasks than a typical population member.Source: §J.1 · p. 26
  • Raters judged completion and naturalness on five-point scales and reported confidence on a three-point scale; 13 flagged records were excluded from primary analyses.Clips were shown in randomized order, and repeat showings were reserved for reliability analyses.Source: §J.1 · p. 26

J.3 ADDITIONAL RESULTS ANALYSIS

Human–benchmark agreement varies substantially by domain, and the VLM judge is less aligned with benchmark scores on high-performing policies. Confidence ratings offer only a tentative guide to rating accuracy.

J.3 ADDITIONAL RESULTS ANALYSIS

  • Human completion ratings track benchmark scores closely in MT10 but less well in Gym MuJoCo, Assistax, and HumanoidBench, where visually judging relevant task quantities is harder.On Assistax, ratings exceed benchmark scores when tool contact is hard to resolve; on HumanoidBench, the reference reward can favor movement raters do not regard as walking or crawling.Source: §J.3 · p. 27, Table 13 · p. 28
  • The VLM judge’s deviation is about 0.42 on high-scoring policies, nearly double its lower-band deviation, while human deviation stays near 0.23 across all three bands.The authors note that the unsupervised-method ranking signal may weaken as policies improve, potentially limiting performance.Source: §J.3 · p. 28, Figure 15 · p. 29
  • High-confidence ratings are closest to benchmark scores at about 0.19 mean deviation, but low-confidence ratings (0.27) are slightly more accurate than medium-confidence ratings (0.32).Differences are tentative because confidence intervals overlap substantially; the authors suggest reviewing ratings not marked high-confidence rather than only the low-confidence tail.Source: §J.3 · p. 28, Figure 16 · p. 29

J.4 STATISTICAL ANALYSIS

  • The pooled comparisons use task-level bootstrap resampling, while HumanoidBench intervals resample clips because it has only two task clusters.Primary analyses exclude records flagged by raters and repeat showings; completion and naturalness thresholds are normalized clip means of at least 0.75 and at most 0.25.Source: §J.4 · pp. 28–29

K BRADLEY–TERRY RANKINGS WITHOUT THE DEVELOPMENT TASKS

The Bradley–Terry ranking retains ERA-S and ERA-U in first and second place after development tasks are removed. The paper also discloses how coding agents contributed to the work and how authors reviewed that assistance.

K BRADLEY–TERRY RANKINGS WITHOUT THE DEVELOPMENT TASKS

L AI DISCLOSURE

  • Coding agents assisted with implementation, writing, hypotheses, methodology, experiments, analysis, and interpretation; authors reviewed the AI-assisted work and checked cited literature against originals.The authors accept responsibility for the final content, claims, and artifacts.Source: L AI DISCLOSURE · p. 30
Loading 2610.04364v1…