PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Refined Large Language Model as Automated Reward Function Designer for Deep Reinforcement Learning in Robotics

Jiayang Song, Zhehua Zhou, Jiawei Liu, Chunrong Fang, Zhan Shu, Lei Ma

arXiv:2309.06687v2cs.ROcs.AI

TL;DR

Designing effective DRL rewards is difficult and often demands substantial manual expertise. This paper uses a self-refining LLM framework to generate rewards from natural-language instructions and evaluate them across robotic tasks. Its rewards rival or exceed hand-crafted ones across nine continuous-control tasks on three robotic systems.

  • Problem

    Effective DRL reward design remains challenging and labor-intensive, while LLM reliability for continuous robotic control is uncertain.

    Source: §1 · pp. 1–2

  • Method

    A zero-shot LLM creates a reward from natural-language instructions, then evaluation feedback guides iterative self-refinement.

    Source: §1 · pp. 2–3

  • Results

    LLM-designed rewards rival or exceed hand-crafted rewards across nine continuous-control tasks on three robotic systems.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The framework generates rewards that induce desired robotic behaviors across the evaluated tasks and systems.

    Source: §1 · pp. 2–3

    The approach struggles with nuanced behavior that is difficult to quantify automatically, such as quadruped gait, and may require human intervention.

    Source: §6 · p. 9

Abstract

from arXiv · show

Although Deep Reinforcement Learning (DRL) has achieved notable success in numerous robotic applications, designing a high-performing reward function remains a challenging task that often requires substantial manual input. Recently, Large Language Models (LLMs) have been extensively adopted to address tasks demanding in-depth common-sense knowledge, such as reasoning and planning. Recognizing that reward function design is also inherently linked to such knowledge, LLM offers a promising potential in this context. Motivated by this, we propose in this work a novel LLM framework with a self-refinement mechanism for automated reward function design. The framework commences with the LLM formulating an initial reward function based on natural language inputs. Then, the performance of the reward function is assessed, and the results are presented back to the LLM for guiding its self-refinement process. We examine the performance of our proposed framework through a variety of continuous robotic control tasks across three diverse robotic systems. The results indicate that our LLM-designed reward functions are able to rival or even surpass manually designed reward functions, highlighting the efficacy and applicability of our approach.

1 Introduction

Designing effective DRL rewards remains difficult and labor-intensive, while LLMs’ reliability for continuous robotic control is uncertain. The paper introduces a zero-shot, self-refining LLM framework and evaluates it across diverse robotic tasks.

1 Introduction

  • Effective reward design is a major challenge in training high-performing DRL agents and often requires substantial manual expertise and effort.AutoRL tunes parameters of predefined rewards but still depends on an expert-provided initial function.Source: §1 · pp. 1–2
  • LLM reliability for reward design remains uncertain, especially for continuous-action robotic control despite prior results on simpler discrete-action games.Reward design draws on common-sense knowledge, a capability that motivates exploring LLMs for this task.Source: §1 · p. 2
  • LLM-designed rewards rival or exceed hand-crafted rewards across nine continuous-control tasks on three robotic systems.The framework uses natural-language instructions and self-refinement guided by evaluation feedback.Source: §1 · pp. 2–3
  • The framework generates an initial reward from natural-language instructions, evaluates resulting behavior, and feeds the assessment back to guide iterative refinement.Evaluation and refinement continue until performance is satisfactory or a maximum iteration count is reached.Source: §1 · p. 2
  • The authors use the LLM as a zero-shot reward designer and incorporate self-refinement to improve reward-design outcomes.They assess the approach on continuous robotic control tasks across diverse systems.Source: §1 · p. 3

2 Related Work

Prior reward-design approaches either adjust existing functions or apply LLMs in adjacent settings, leaving LLM design of rewards from scratch for continuous robotic control unresolved.

2 Related Work

  • AutoRL adjusts parameters of a predefined, hand-crafted reward function, so it cannot formulate a reward entirely from scratch.The cited work applies AutoRL to navigation and multiple simulated reinforcement-learning benchmarks.Source: §2 · p. 3
  • Prior LLM studies address reward shaping, objective selection for predefined controllers, or proxy rewards for simpler discrete-action tasks, not this open continuous-control question.The authors identify independent reward design from scratch for continuous robotic control as unresolved.Source: §2 · p. 3
  • LLMs may reduce human effort by using pretrained common-sense knowledge to incorporate task requirements and user preferences into reward functions.The passage describes this as potential rather than established performance for continuous robotic control.Source: §2 · p. 3

3 Preliminary

The preliminary section frames DRL as policy optimization under a POMDP and reward design as choosing a reward that yields strong task performance after training.

3 Preliminary

  • The DRL setup is a POMDP with continuous state and action spaces, where a policy maximizes expected cumulative discounted reward under a given reward function.The policy interacts through actions, state transitions, and observations.Source: §3 · p. 3
  • Reward design seeks a function whose trained policy maximizes a human-interpretable task performance metric computed from system trajectories.For humanoid walking, the example metric is maximum distance traveled without falling.Source: §3 · p. 3

4 Self-Refined LLM for Reward Function Design

The framework combines natural-language reward generation with automated assessment and feedback-driven refinement. It uses zero-shot prompts and task success rates to guide iteration.

4 Self-Refined LLM for Reward Function Design

  • The overall framework has three stages: initial reward design, evaluation of resulting behavior, and a self-refinement loop.Evaluation and refinement repeat until performance is judged good or a preset iteration limit is reached.Source: §4 · p. 4, §4.3 · pp. 5–6

4.1 Initial Design

  • Initial prompts describe the environment, task, observable states, and rules; the LLM produces a weighted combination of reward components whose initial weights may be unreliable.The reward is restricted to observable states and excludes hard-to-quantify elements such as specific quadruped postures.Source: §4.1 · p. 4
  • The LLM is prompted zero-shot because task-specific reward design makes universally applicable examples difficult to provide.The authors rely on subsequent evaluation and refinement rather than in-context examples to help ensure performance.Source: §4.1 · p. 4

4.2 Evaluation

  • Evaluation samples trajectories from a trained policy and examines training behavior, task-specific objective metrics, and task success rate.Tasks span a manipulator, quadruped, and quadcopter, with simulations conducted in NVIDIA Isaac Sim.Source: §4.2 · pp. 4–5, Figure 2 · p. 5
  • Overall reward quality is classified using success rate alone, with a predefined threshold separating good from bad performance and bad results triggering refinement.Training summaries and objective metrics remain feedback for the refinement process because they are task-dependent.Source: §4.2 · p. 5

4.3 Self-Refinement Loop

  • Evaluation feedback, led by the overall assessment and supplemented by training details, objective metrics, and success rate, guides the LLM’s updated reward.The evaluation-refinement loop ends at satisfactory performance or the predefined maximum number of iterations, and the resulting reward is retained.Source: §4.3 · pp. 5–6

5 Experimental Results

The experiments evaluate LLM-designed rewards across nine robotic control tasks and compare initial, self-refined, and manual reward functions. The quadruped running example illustrates the refinement process and its evaluation measures.

5.1 Experimental Setup

  • The evaluation covers nine continuous robotic control tasks across a manipulator, quadruped, and quadcopter.The tasks include ball manipulation, quadruped locomotion, and quadcopter flight.Source: §5.1 · p. 6
  • For each task, the study compares the LLM’s initial reward, its final self-refined reward, and a manually designed reward.The manual reward comes from existing literature or benchmarks.Source: §5.1 · p. 6
  • PPO trains policies for each reward using the same learning parameters and neural network architecture.The evaluation samples 100 trajectories per trained policy; the maximum is five refinement iterations.Source: §5.1 · p. 6

5.2 Reward Function and Objective Metrics

  • The quadruped running example evaluates velocity, position, action, and angular-velocity metrics against a running success condition.The condition requires at least 2 m/s forward speed after an initial 0.8-second acceleration, while limiting lateral deviation and preventing falls.Source: §5.2 · pp. 6–7
  • The LLM reaches a satisfactory quadruped running reward after two self-refinement iterations.Figure 3 presents reward functions across iterations with their evaluation outcomes; Figure 4 presents corresponding behaviors alongside the manual reward behavior.Source: §5.2 · p. 7, Figure 3 · p. 7, Figure 4 · p. 7

Quadcopter

Across the examined tasks, self-refinement enables success rates that match or exceed manual designs, while complex tasks can require further refinement or human feedback. In quadruped running, feedback-driven changes improve the reward from an initially inadequate design to behavior close to the manual design.

Quadcopter

  • In quadruped running, the initial reward overweights balance, producing low forward velocity and success rate; refinement raises success to 90%.The first revision increases forward-velocity weight and adjusts lateral-deviation and action penalties; a second increases penalties for excessive actions and lateral deviation, yielding behavior close to the manual design.Source: Quadcopter · p. 8
  • Other control tasks exhibit a similar pattern to quadruped running.The passage directs readers to Appendix D and a supplementary video for details.Source: Quadcopter · p. 8

5.3 Success Rates

  • Self-refined rewards achieve success rates matching or surpassing manual designs across all examined tasks.Task complexity affects performance, and intricate reward components can require more refinement or detailed human feedback.Source: §5.3 · pp. 8–9

6 Discussion

The discussion identifies avenues for improving reward design and boundaries of the current approach, including evaluation gaps for nuanced behaviors and reliance on pretrained common-sense knowledge.

Learning Parameters and AutoRL

  • AutoRL could optimize parameters of an LLM-formulated reward, but this may greatly prolong the design process.The LLM could then focus on structural reward changes, with learning parameters optimized before each refinement iteration.Source: §6 · p. 9

Fine-tuned LLM

  • A reward-design-specific fine-tuned LLM could help with task comprehension, but fine-tuning requires substantial resources and suitable training data may be difficult to obtain.Source: §6 · p. 9

Limitations

  • Automated evaluation cannot capture some nuanced behavior goals, such as quadruped gait, and specialized tasks may require detailed human feedback or richer prompts.The approach also relies on pretrained common-sense knowledge and may struggle with tasks absent from its training data.Source: §6 · p. 9

7 Conclusion

The paper presents a self-refined LLM framework for designing DRL rewards and evaluates it across nine robotic control tasks on three systems. Its reward functions are reported as on par with or better than manually designed ones.

7 Conclusion

  • The framework uses natural-language input to generate an initial reward, evaluates its performance, and feeds results back to guide self-refinement.Source: §7 · p. 9
  • Evaluation spans nine diverse robotic control tasks across three robotic systems.Source: §7 · p. 9
  • The generated reward functions are reported as on par with or superior to manually designed rewards.Future work plans to integrate the LLM with AutoRL techniques to design learning parameters autonomously.Source: §7 · p. 9

A Natural Language Input Prompt

This section presents natural-language prompts that specify robot goals and constrain reward design to supplied variables, alongside feedback templates and task-specific RL diagnostics intended to guide reward redesign. It covers quadruped and quadcopter tasks, with reported values presented as placeholders, and then introduces experimental details.

A.1 Robotic Manipulator

  • Manipulator prompts cover ball capture and retention, tray-centered ball balancing, and pushing a ball into a table hole, using only the supplied variables.The available variables differ by task: ball/container state, ball/tray state, or gripper/ball/hole state.Source: §A.1 · pp. 12–13

A.2 Quadruped Robot

  • Quadruped prompts specify target-velocity tracking, fast straight-line running with lateral-deviation control, and target-position reaching, alongside fall-prevention and stability conditions.The target-position prompt additionally requires positive-only reward components.Source: §A.2 · pp. 13–15

A.3 Quadcopter

  • Quadcopter prompts cover reaching and hovering at a target within a height range, flying through a wind-disturbance area, and velocity tracking at a fixed height.The stated height bounds are greater than 0.8 and less than 3.0 for target hovering, and fixed at 1.0 for velocity tracking.Source: §A.3 · pp. 15–16

B Feedback Prompt Template

  • The appendix says feedback prompt templates are provided for each control task.These templates supply performance and training/evaluation information and ask the LLM to redesign the reward function.Source: B Feedback Prompt Template · p. 17

B.1 Robotic Manipulator

  • Manipulator feedback reports training convergence, episode statistics, action and ball-to-object metrics, and trial success rates before requesting reward redesign.The reported object-distance metric varies by task: ball-to-container, ball-to-tray-center, or ball-to-target-hole; success rates cover one or two goals.Source: §B.1 · pp. 17–18

B.2 Quadruped Robot

  • Quadruped feedback includes a good/bad performance label, training and episode statistics, velocity deviation, action and motion metrics, robot height, and trial success rates.The prompt requests redesign based on the feedback; the listed measures include linear and angular velocities and success rates for three goals.Source: §B.2 · pp. 18–19
  • The redesign instruction is to revise the reward function based on the given feedback.The instruction appears repeatedly across both robot subsections.Source: §B.2 · p. 19, §B.3 · p. 20
  • Quadruped prompts classify agent performance as good or bad and report training convergence, episode timestep, and average reward.The prompt also supplies task-specific motion or target-distance metrics and goal success rates, but their values are placeholders.Source: §B.2 · p. 19
  • One quadruped diagnostic set includes maximum x-axis velocity and average linear and angular velocities, while another includes normalized distance to the target.Both sets also report average normalized action value and average z-axis position; one lists three goal success rates and the other lists two.Source: §B.2 · p. 19

B.3 Quadcopter

  • Quadcopter prompts classify agent performance as good or bad and report training convergence, episode timestep, and average reward.The supplied measures include normalized distance to the target, action value, motion statistics, and goal success rates; values remain placeholders.Source: §B.3 · p. 20
  • Across the quadcopter prompt variants, the motion and target-distance measures differ, while both include action-value and goal-success diagnostics.The supplied passages provide metric names but not values, so they do not establish which variant performed better.Source: §B.3 · pp. 20–21
  • A second quadcopter diagnostic set reports linear-velocity deviation alongside average normalized action, angular velocity, z-axis position, and goal success rates.The prompt gives no usable numeric values for these measures.Source: §B.3 · p. 21

C Experimental Details

  • The following section says it explains experimental details for each control task.No experimental details are included in the supplied passage.Source: C Experimental Details · p. 21

C.1 Robotic Manipulator

This block specifies manipulator, quadruped, and quadcopter tasks, then logs iterative LLM reward designs and reports results for ball catching and balancing.

C.1 Robotic Manipulator

  • The manipulator tasks cover catching a falling ball in a container, balancing a falling ball on a tray, and pushing a ball toward a hole in a table.Source: §C.1 · pp. 21–22

C.2 Quadruped Robot

  • Quadruped tasks specify target-velocity tracking, maintaining a 2 m/s forward speed, and reaching a target point while preventing falls.Source: §C.2 · pp. 23–24

C.3 Quadcopter

  • Quadcopter tasks include reaching a target within 0.2 m, with height constraints in some variants and a wind-field condition in another.Source: §C.3 · pp. 24–25

D Experimental Log

  • The experimental log presents LLM-designed rewards across refinement iterations, their evaluation results, and manually designed reward functions.Source: D Experimental Log · p. 25

D.1.1 Ball Catching

  • In evaluation over 100 trials, Goal 1 and Goal 2 each had a 100% success rate; the reward function was qualified as the final reward function.Training reward converged after 12,000 steps. Average reward was 251 for one episode.Source: §D.1.1 · p. 26

D.1.2 Ball Balancing

  • Across 100 evaluation trials, Goal 1 and Goal 2 each achieved a 100% success rate; the reward function was identified as the final reward function.Training reward converged after 8,000 steps, and average reward was 183 per episode.Source: §D.1.2 · p. 28

D.1.3 Ball Pushing

Ball-pushing reward redesigns yield widely varying Goal 1 success, including reported outcomes from 0% to 91%; reward variants combine distance-to-hole and directional-velocity terms with other components.

D.1.3 Ball Pushing

  • 0% success is reported for a later supplied variant, despite preceding redesign evaluations reaching 91%.Both evaluations used 100 trials.Source: §D.1.3 · pp. 29, 33
  • Reward variants combine distance-to-hole and directional-velocity rewards with a gripper-to-ball distance term, whose weights change across redesigns.The instructions ask the LLM to redesign the reward using feedback.Source: §D.1.3 · pp. 29–32
  • 0%, 10%, and 82% Goal 1 success were reported across successive 100-trial redesign evaluations.A subsequent 10-trial evaluation reported 90%, followed by 91% in 100 trials.Source: §D.1.3 · pp. 30–33

D.2.1 Running

The supplied running-task reward iterations report improved success rates across three goals, with the final identified reward function reaching 98%, 99%, and 100%. Reward designs emphasize forward motion alongside balance and movement constraints.

D.2.1 Running

  • Initial testing over 100 trials reported success rates of 10%, 82%, and 90% for Goals 1–3; a later reported set gave 90%, 92%, and 95%, respectively.Source: §D.2.1 · pp. 35–36
  • Reward iterations combine forward velocity, upright posture, lateral and vertical motion, rotation, and action terms; a later variant adds a straight-line reward.One variant increases the forward-velocity weight and penalizes movement in the y and z directions.Source: §D.2.1 · pp. 34–35
  • Reported success rates were 98% for Goal 1, 99% for Goal 2, and 100% for Goal 3; the reward function is qualified as the final reward function.Source: §D.2.1 · p. 37

D.2.2 Velocity Tracking

The supplied velocity-tracking reward variants report rising success rates across three goals, with the last reported variant reaching 96%, 100%, and 100%. The designs vary target-speed reward formulations and include stability and smoothness terms.

D.2.2 Velocity Tracking

  • The reward designs use target-speed, height, smoothness, and stability terms; a later variant sets target speed to 2.0 and includes straightness.An earlier design sets target speed to 3.0, while another uses 1.0.Source: §D.2.2 · pp. 37, 38, 41
  • Across four 100-trial evaluations, success increased for all three goals: Goal 1 rose from 2% to 96%, Goal 2 from 10% to 100%, and Goal 3 from 40% to 100%.Source: §D.2.2 · pp. 38, 39, 40, 42
  • 96%, 100%, and 100% success across Goals 1–3 are reported for a variant evaluated as good.The evaluation covered 100 trials.Source: §D.2.2 · pp. 41–42

D.2.3 Walking To Target

This section documents successive reward-function redesigns and evaluations across walking-to-target, hovering, and wind-field tasks, tracking goal success rates alongside qualitative assessments of agent performance.

D.2.3 Walking To Target

  • The reported average normalized target distance fell from 8.44 in the initial evaluation to 0.69 in the latest one.Source: §D.2.3 · pp. 43, 48
  • Across successive 100-trial evaluations, Goal 1 success rose from 0% to 10%, 73%, 80%, 80%, and 85%, while Goal 2 rose from 90% to 100% and stayed there.Source: §D.2.3 · pp. 43–48
  • Later reward variants increased the distance weight to 5.0, set the action-reward weight to 0.1, and raised the stability weight to 0.2.Source: §D.2.3 · pp. 46, 48
  • Goal 1 success reached 85% in the latest 100-trial evaluation, while Goal 2 success was 100%.The evaluation also reports 38,000-step reward convergence, an average episode length of 599 timesteps, and an average normalized target distance of 0.69.Source: §D.2.3 · p. 48
  • A subsequent formulation defines stability using linear velocity, penalizes actions, and returns distance, height, and stability reward terms.Source: §D.2.3 · p. 49

D.3.1 Hovering

  • Hovering revisions raised Goal 1 success from 10% to 90% to 98% over 100 trials, while Goal 2 remained at 100%; final performance was rated good.Source: §D.3.1 · pp. 50–52

D.3.2 Wind Field

  • In Wind Field, evaluated x- and xy-distance-penalty redesigns were rated bad, with Goal 1 success at 0% and Goal 2 at 100% across 100 trials.Source: §D.3.2 · pp. 53–55
  • The final displayed Wind Field reward adds an xyz-axis distance penalty after earlier versions added x-only and xy-axis penalties.Source: §D.3.2 · pp. 54–56
  • In evaluation results from 100 trials, Goal 1 had a 20% success rate; Goal 2 had a 90% success rate.Training reward converged after 30,000 steps, and the average reward was 967 for one episode.Source: §D.3.2 · pp. 56–57
  • The redesigned reward retained closeness-to-target and hover rewards, replaced low-velocity rewards with a smaller-actions reward, and subtracted a height penalty.The initial reward instead rewarded low velocities on the x, y, and z axes.Source: §D.3.2 · pp. 56–57
  • Evaluation reported an average normalized action value of 1.38 and average target-position distances of 0.37, 0.40, and 0.28 along the x-, y-, and z-axes, respectively.Average linear velocities were 0.193, 0.123, and 0.113 along the x-, y-, and z-axes; average angular velocities were -0.066, 0.121, and 0.39 along those axes.Source: §D.3.2 · p. 57
  • After reward redesign, both goals reached 100% success across 100 trials, although the evaluation still characterized the agent’s performance as bad.The redesigned reward’s training converged after 34,000 steps, with an average episode reward of 1,011.Source: §D.3.2 · pp. 57–58

D.3.3 Velocity Tracking

Velocity-tracking reward variants produced mixed evaluations: one achieved 99% and 100% success on two goals, while another was rated bad. The section also reports earlier evaluations and reward formulations using velocity, height, and angular-velocity terms.

D.3.3 Velocity Tracking

  • Another designed reward was rated bad despite convergence after 20,000 steps, with average reward 385 and normalized linear velocity deviation 3.52.Source: §D.3.3 · p. 59
  • Earlier 100-trial evaluations reported Goal 1/Goal 2 success rates of 0%/50% and 0%/90%.Source: §D.3.3 · pp. 59–60
  • An earlier 100-trial evaluation reported Goal 1 success rate of 70% and Goal 2 success rate of 100%.Source: §D.3.3 · p. 61
  • One reward formulation used exponential rewards for velocity, height, and angular-velocity magnitude, with velocity weight 1.5 and height weight 1.0.Source: §D.3.3 · p. 61
  • Across 100 trials, Goal 1 success rate was 99% and Goal 2 success rate was 100%.The agent trained with this designed reward function was described as performing well.Source: §D.3.3 · p. 62

Related papers

Loading 2309.06687v2…