PaperlayerView exact revision on arXiv

Source-linked AI summary

Eureka: Human-Level Reward Design via Coding Large Language Models

Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, Anima Anandkumar

arXiv:2310.12931v2cs.ROcs.AIcs.LG

TL;DR

Complex low-level manipulation remains difficult to learn with LLMs, while reward design for RL is laborious and challenging. EUREKA uses coding LLMs with in-context evolutionary search to generate and refine executable rewards. Across 29 environments, it outperforms expert rewards on 83% of tasks with 52% average normalized improvement, and enables simulated pen spinning and gradient-free RLHF.

  • Problem

    LLMs have not established that they can learn complex low-level manipulation, while effective RL often depends on difficult manual reward design.

  • Method

    EUREKA uses environment source code and task descriptions as context for zero-shot reward generation, then applies evolutionary search and reward reflection to refine executable reward programs.

  • Results

    Across 29 open-source RL environments and 10 robot morphologies, EUREKA rewards outperform expert human rewards on 83% of tasks, with 52% average normalized improvement.

  • Takeaways & Limitations

    EUREKA enables rapid pen spinning on a simulated Shadow Hand and gradient-free RLHF that incorporates human inputs without model updating.

  • Takeaways & Limitations

    EUREKA is currently evaluated mainly on simulation-based tasks, aside from a preliminary real-robot hopping experiment using Sim2Real.

Abstract

from arXiv · show

Large Language Models (LLMs) have excelled as high-level semantic planners for sequential decision-making tasks. However, harnessing them to learn complex low-level manipulation tasks, such as dexterous pen spinning, remains an open problem. We bridge this fundamental gap and present Eureka, a human-level reward design algorithm powered by LLMs. Eureka exploits the remarkable zero-shot generation, code-writing, and in-context improvement capabilities of state-of-the-art LLMs, such as GPT-4, to perform evolutionary optimization over reward code. The resulting rewards can then be used to acquire complex skills via reinforcement learning. Without any task-specific prompting or pre-defined reward templates, Eureka generates reward functions that outperform expert human-engineered rewards. In a diverse suite of 29 open-source RL environments that include 10 distinct robot morphologies, Eureka outperforms human experts on 83% of the tasks, leading to an average normalized improvement of 52%. The generality of Eureka also enables a new gradient-free in-context learning approach to reinforcement learning from human feedback (RLHF), readily incorporating human inputs to improve the quality and the safety of the generated rewards without model updating. Finally, using Eureka rewards in a curriculum learning setting, we demonstrate for the first time, a simulated Shadow Hand capable of performing pen spinning tricks, adeptly manipulating a pen in circles at rapid speed.

1 INTRODUCTION

EUREKA uses coding LLMs to automate reward design for diverse RL tasks, addressing the gap between high-level planning and complex low-level manipulation. It outperforms expert rewards across most evaluated tasks and enables rapid simulated pen spinning.

  • LLMs excel at high-level robotic planning, but learning complex low-level manipulation such as dexterous pen spinning remains an open problem.
  • Manual reward design is difficult because RL depends on carefully shaped learning signals, especially for tasks with sparse rewards.A survey found that 92% of respondents reported manual trial-and-error reward design.
  • EUREKA is a coding-LLM reward design algorithm built around environment context, evolutionary search, and reward reflection, without task-specific prompts or templates.
  • 83% of tasks: EUREKA rewards outperform expert human rewards across 29 open-source RL environments spanning 10 robot morphologies, with 52% average normalized improvement.
  • Combining EUREKA rewards with curriculum learning enables rapid pen-spinning maneuvers on a simulated anthropomorphic Shadow Hand.
  • EUREKA also supports gradient-free in-context RLHF, incorporating human reward initialization and textual feedback without model updating.

2 PROBLEM SETTING AND DEFINITIONS

The paper formalizes reward design as selecting a shaped reward whose optimized policy achieves high ground-truth fitness, then adapts the problem to reward-program synthesis.

  • Reward design seeks a shaped reward for a ground-truth reward that may be difficult to optimize directly, such as a sparse reward.
  • A reward design problem contains a world model, reward-function space, learning algorithm, and fitness function accessed through policy queries.
  • The objective is to output reward-function code R such that the fitness F(A_M(R)) of the policy learned from R is maximized.

3 METHOD

EUREKA combines environment-as-context prompting, evolutionary search, and reward reflection to generate executable rewards and iteratively improve them using policy-training feedback.

  • EUREKA has three components: environment as context, evolutionary search, and reward reflection.
  • 3.1 ENVIRONMENT AS CONTEXT: Environment source code lets a coding LLM generate executable reward functions zero-shot using exposed state and action variables.Relevant state information can also be supplied through an API when source code is unavailable.
  • 3.1 ENVIRONMENT AS CONTEXT: EUREKA can improve rewards by changing existing-component hyperparameters, altering functional forms, or adding new reward components.
  • 3.2 EVOLUTIONARY SEARCH: Each evolutionary iteration samples K reward programs from the LLM, evaluates them with the fitness function, reflects on the best candidate, and updates the search.
  • 3.2 EVOLUTIONARY SEARCH: The method uses five independent runs per environment, five iterations per run, and K = 16 samples per iteration.
  • 3.3 REWARD REFLECTION: Reward reflection summarizes intermediate training dynamics by tracking reward-component values and task fitness at policy checkpoints.

4 EXPERIMENTS

EUREKA is evaluated across diverse robot tasks against human-engineered rewards, sparse rewards, and L2R. It produces stronger and novel rewards, supports human feedback and curriculum learning, and enables dexterous pen spinning.

  • Experimental Setup: 29 tasks across 10 robot morphologies evaluate EUREKA against expert human, sparse, and L2R reward baselines.The environments include quadrupeds, bipeds, quadrotors, cobot arms, and dexterous hands, using common RL training procedures.
  • Reward Novelty: EUREKA rewards are mostly weakly correlated with human rewards while outperforming them, and some negatively correlated rewards perform significantly better.The results suggest EUREKA can discover reward-design principles that differ from human intuition, especially on harder tasks.
  • Ablations: Removing reward reflection reduces average normalized Isaac performance by 28.6%, with larger deterioration on higher-dimensional tasks.The reflection mechanism also supports targeted reward editing during evolutionary search.
  • Human Feedback and Dexterity: EUREKA also enables gradient-free in-context RLHF that incorporates human inputs without model updating.The approach is intended to generate more performant and human-aligned reward functions.
  • Human Feedback and Dexterity: EUREKA can improve human-initialized rewards and combine with curriculum learning to achieve rapid pen spinning on a simulated Shadow Hand.Human initialization requires no modification to EUREKA, while the pen-spinning task requires continuous rotation in predefined patterns.

5 RELATED WORK

Prior approaches automate parts of reward design but commonly depend on task-specific templates, manual engineering, or non-interpretable reward outputs. EUREKA instead targets general reward programming with coding LLMs and human reward reflection.

  • Manual trial-and-error reward engineering remains difficult, while inverse reinforcement learning requires expensive demonstrations and can produce non-interpretable rewards.
  • Evolutionary reward-search methods typically optimize parameters within provided templates rather than generating unrestricted reward programs.
  • Most coding-LLM decision-making systems rely on known motion primitives and therefore do not address low-level dexterous skill learning.

6 CONCLUSION

EUREKA combines coding LLMs with in-context evolutionary search to automate reward design across robots and tasks. The paper reports broad reward-generation performance, dexterous pen spinning, and gradient-free human-feedback integration.

  • EUREKA is a universal reward design algorithm powered by coding large language models and in-context evolutionary search.
  • EUREKA achieves human-level reward generation across a wide range of robots and tasks without task-specific prompt engineering or human intervention.
  • The method solves dexterous pen spinning for the first time with a curriculum learning approach.
  • EUREKA enables gradient-free reinforcement learning from human feedback that incorporates human reward initialization and textual feedback to steer reward generation.

C.1 L2R REWARD EXAMPLES

The appendix documents baseline reward examples, environment observations, reward-history handling, and evaluation procedures. It also records score normalization used for reporting aggregate improvement.

  • L2R achieves a Human Normalized Score of 0.0 on Humanoid in the reported example.
  • L2R achieves a Success Rate of 0.07 on ShadowHandKettle in the reported example.
  • EUREKA trims its dialogue to the last reward and reward reflection, making reward improvement Markovian for the next generation.
  • Final EUREKA and baseline rewards are evaluated with 5 PPO runs, while intermediate EUREKA rewards use 1 PPO run.
  • Average normalized improvement is computed after clipping each task’s normalized score to [0, 3].

D.1 PEN SPINNING TASKS

The pen-spinning task reuses the Shadow Hand environment with a pen replacing the cube and trains toward sequential target poses. Human-written rewards can initialize EUREKA before subsequent reward refinement.

  • The pen-spinning environment changes the original Shadow Hand task by replacing the cube asset with a pen without changing physics parameters.
  • During pre-training, the policy targets uniformly sampled random SO(3) orientations; fine-tuning uses predetermined waypoint sequences for spinning patterns.
  • EUREKA can improve a human-written reward by using it as the first iteration, computing feedback, and generating later rewards from that feedback.
  • Human rewards are refactored to expose individual reward components and stripped of variables unnecessary for reward computation.

D.3 EUREKA FROM HUMAN FEEDBACK

This section evaluates EUREKA across additional environments and reports that it remains effective despite changes in simulation and observation-code syntax.

  • EUREKA remains effective on the OpenAI Gym Humanoid environment implemented with Mujoco despite different observation spaces and code syntax.The evaluation changes the tensor-formatting instruction but otherwise leaves the prompts unchanged.
  • The section provides the Mujoco Humanoid observation code and its associated environment definition, including torso, joint, and velocity variables.
  • The observation space places positional values before the corresponding velocity values for Humanoid body parts.

F ADDITIONAL RESULTS

Additional evaluations show that EUREKA improves sample efficiency and aggregate performance, with statistically significant gains over baselines, while reflection and stronger coding models improve results.

  • Training and aggregate evaluation: EUREKA reward functions show improved sample efficiency over baseline rewards on aggregate across 20 Dexterity tasks.
  • Training and aggregate evaluation: EUREKA consistently outperforms all baselines on mean, median, and interquantile-mean evaluation metrics across 100 RL runs over 20 Dexterity tasks.The metrics include associated 95% confidence intervals.
  • Training and aggregate evaluation: EUREKA’s improvement over baselines is statistically significant based on outperformance probabilities with 95% confidence intervals.
  • Ablations and model comparison: Removing reward reflection generally reduces reward performance, with larger deterioration on high-dimensional tasks.The results support targeted reward editing for tasks requiring interactions among many state components.
  • Ablations and model comparison: With GPT-3.5, EUREKA’s absolute performance decreases but remains comparable to GPT-4 on most tasks and exceeds human-engineered rewards on dexterous manipulation tasks.The authors use these results to support generality across coding-LLM capability levels.
  • Reward novelty: EUREKA reward novelty is assessed by comparing Pearson correlations with human rewards on Isaac tasks.The evaluation addresses whether syntactically different programs may be functionally identical.

G.1 REWARD REFLECTION EXAMPLES.

The examples show reward reflection using tracked reward components and policy metrics to iteratively modify reward code, including adding, scaling, or discarding components.

  • Reflection process: Reward reflection displays previous reward functions together with policy-training feedback before generating subsequent rewards.
  • Example outcomes: The ShadowHand example reports scores increasing from 9.29 in iteration 1 to 10.43 in iteration 2.
  • Reflection process: The reflection procedure tracks reward-component values, success rates, and episode lengths during reinforcement-learning training.The ShadowHand example reports these quantities at intermediate checkpoints.
  • Reflection process: The reflection prompt instructs the coding model to rewrite ineffective rewards, rescale or replace stagnant components, and balance disproportionately large values.
  • Reflection process: The generated reward output contains a total reward and a dictionary of individual reward components, using only variables from the provided environment.
  • ShadowHand example: In the ShadowHand example, reflection adds an angular-velocity penalty after identifying stable rotation reward but variable episode lengths.The new penalty applies above an angular-velocity threshold and is scaled by a temperature parameter.
  • Example outcomes: The Humanoid example reports scores increasing from 3.79 in iteration 2 to 4.37 in iteration 3.
  • Humanoid example: In the Humanoid example, reflection discards an all-zero tracking component and adds posture reward alongside velocity reward.

G.2 NEGATIVELY CORRELATED EUREKA REWARD EXAMPLES

These examples illustrate that EUREKA can discover reward functions with weak or negative correlation to human rewards while still achieving positive human-normalized scores.

  • Negative correlation and performance: EUREKA’s examples include rewards that are negatively correlated with human rewards yet superior in task performance.
  • Negative correlation and performance: A ShadowHand reward example achieves a Human Normalized Score of 1.45 with correlation −0.26.
  • ShadowHand reward: The ShadowHand reward combines quaternion-rotation alignment, fingertip proximity, and a distance penalty in its total reward.
  • FrankaCabinet reward: The FrankaCabinet example achieves a Human Normalized Score of 11.98 with correlation −0.30.
  • FrankaCabinet reward: The FrankaCabinet reward combines grasp distance, handle grasping, and drawer opening components.

G.3 EUREKA FROM HUMAN INITIALIZATION EXAMPLES

EUREKA improves human-written rewards through iterative policy evaluation and reward-code revision, with examples showing substantial gains on Kettle and CatchUnderarm.

  • Human Initialization: EUREKA compares a human-written reward at Iteration 0 with the best reward after one improvement step.The process uses policy feedback to guide reward revision.
  • Kettle: 0.91 EUREKA Success Rate versus 0.11 Human Success Rate on Kettle.
  • Reward Reflection: Reward reflection recommends rewriting ineffective components, adjusting scales or temperatures, and rescaling components with disproportionately large magnitudes.
  • CatchUnderarm: 0.64 EUREKA Success Rate versus 0.33 Human Success Rate on CatchUnderarm.
  • Reward Reflection: EUREKA analyzes reward-component values and global policy metrics, including success rates and episode lengths.These statistics are tracked after every 300 epochs, alongside maximum, mean, and minimum values.

G.4 EUREKA FROM HUMAN REWARD REFLECTION

EUREKA uses human textual feedback to revise rewards toward intended behavior, as illustrated by successive modifications for humanoid running.

  • Human Feedback: Human reward reflection identifies that the learned behavior resembles a forward squat jump and requests forward running instead.
  • First Revision: EUREKA revises rewards by encouraging alternating legs, penalizing jumping, and rewarding an upright torso.
  • Second Revision: Human feedback then reports duck walking, prompting stronger upright-torso and height rewards and a penalty for low torso position.
  • Third Revision: A later feedback step reports hopping on one foot, leading EUREKA to add balanced-leg penalties and discourage high angular velocity.
  • Reward Comparison: EUREKA-generated rewards are described as clean and interpretable compared with a human reward containing many commented-out components.

H LIMITATIONS AND DISCUSSION

The discussion identifies boundaries in EUREKA’s current evaluation and task formulation, while outlining paths toward broader real-world and simulator applicability.

  • Real-World Scope: EUREKA is currently evaluated mainly on simulation-based tasks, apart from a preliminary real-robot hopping experiment.
  • Task Specification: EUREKA requires a task fitness function F that exists and is easily definable by humans.The paper notes that behaviors such as running in a stable gait can be difficult to specify mathematically.
  • Experimental Scope: Experiments currently use a single simulator and fixed RL algorithm, except for the MuJoCo Humanoid experiment.
Loading 2310.12931v2…