PaperlayerView exact revision on arXiv

Source-linked AI summary

Text2Reward: Reward Shaping with Language Models for Reinforcement Learning

Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, Tao Yu

arXiv:2309.11489v3cs.LGcs.AIcs.CLcs.RO

TL;DR

Reward shaping is costly because it requires expertise or domain data, motivating a data-free alternative. TEXT2REWARD uses LLMs to generate interpretable shaped dense reward code from natural-language goals, achieving expert-comparable manipulation performance, novel locomotion behaviors, real-world transfer, and feedback-based improvement.

  • Problem

    Reward shaping requires specialized knowledge or domain data, while existing IRL and preference-learning approaches require substantial human effort or data collection and produce non-interpretable models.

  • Method

    TEXT2REWARD uses LLMs to generate executable, interpretable shaped dense reward code from natural-language goals, grounded in compact environment representations and refined with human feedback.

  • Results

    On 13 of 17 manipulation tasks, generated rewards matched or exceeded expert-written rewards in success rate and convergence speed; six locomotion behaviors exceeded 94% success.

  • Takeaways & Limitations

    The framework supports generalizable, interpretable dense reward code across RL tasks, simulator-to-real deployment, and human-in-the-loop refinement.

  • Takeaways & Limitations

    The symbolic reward space may not cover all reward aspects, and improvement from human feedback depends on the initial reward-function quality.

Abstract

from arXiv · show

Designing reward functions is a longstanding challenge in reinforcement learning (RL); it requires specialized knowledge or domain data, leading to high costs for development. To address this, we introduce Text2Reward, a data-free framework that automates the generation and shaping of dense reward functions based on large language models (LLMs). Given a goal described in natural language, Text2Reward generates shaped dense reward functions as an executable program grounded in a compact representation of the environment. Unlike inverse RL and recent work that uses LLMs to write sparse reward codes or unshaped dense rewards with a constant function across timesteps, Text2Reward produces interpretable, free-form dense reward codes that cover a wide range of tasks, utilize existing packages, and allow iterative refinement with human feedback. We evaluate Text2Reward on two robotic manipulation benchmarks (ManiSkill2, MetaWorld) and two locomotion environments of MuJoCo. On 13 of the 17 manipulation tasks, policies trained with generated reward codes achieve similar or better task success rates and convergence speed than expert-written reward codes. For locomotion tasks, our method learns six novel locomotion behaviors with a success rate exceeding 94%. Furthermore, we show that the policies trained in the simulator with our method can be deployed in the real world. Finally, Text2Reward further improves the policies by refining their reward functions with human feedback. Video results are available at https://text-to-reward.github.io/ .

1 INTRODUCTION

TEXT2REWARD addresses costly, expertise- and data-intensive reward shaping by generating interpretable dense reward code from natural-language goals and environment abstractions. Experiments compare the approach across manipulation and locomotion settings, with human feedback used to refine rewards.

  • Reward shaping is difficult because manual design requires expertise and can be time-consuming or sub-optimal, while IRL and preference learning require human effort or data.
  • TEXT2REWARD generates dense reward code from natural-language goals using LLMs grounded in a compact Pythonic environment representation.
  • The framework produces data-free, interpretable, free-form shaped rewards that can use established coding packages and cover more tasks than unshaped reward code.
  • Human feedback from policy rollouts is used to refine reward code when language ambiguity or training sensitivity causes failures or unintended behavior.
  • On 13 of 17 manipulation tasks, generated rewards achieved comparable or better success rates and convergence speed than expert-written rewards, while six locomotion behaviors exceeded 94% success.

2 APPROACH

TEXT2REWARD converts natural-language instructions into shaped dense reward code using compact environment abstractions, background knowledge, examples, execution feedback, and human refinement. The interactive process targets ambiguity and practically sub-optimal or overly difficult instructions.

  • Reward code specifies the goal for RL, and this paper uses code because it reasons over abstract variables and APIs without handling perception.
  • Shaped dense rewards encourage intermediate steps and regularization, with code allowing different functional forms at each timestep.
  • Zero-shot and few-shot dense reward generation: An instruction is a natural-language sentence describing the agent’s goal, supplied by a user or planned by the LLM as a long-horizon subgoal.
  • Zero-shot and few-shot dense reward generation: The environment abstraction uses Python classes, typing, and comments to represent object states and callable functions in a compact, reusable format.
  • Zero-shot and few-shot dense reward generation: Background functions and retrieved instruction-code pairs provide domain knowledge and few-shot examples for dense reward generation.
  • Zero-shot and few-shot dense reward generation: Generated code is executed to expose syntax or runtime errors, whose feedback supports iterative refinement within the LLM.
  • Improving reward code from human feedback: Because instructions may underspecify preferences or make tasks unnecessarily difficult, users inspect rollout videos and provide feedback that improves reward functions.

3 EXPERIMENT SETUP

The evaluation covers manipulation benchmarks, MuJoCo locomotion tasks, code-generation settings, and sim-to-real transfer. Experiments use fixed RL training configurations and assess locomotion through human judgments of rollout videos.

  • TEXT2REWARD is evaluated on MetaWorld, ManiSkill2, and Gym MuJoCo, using GPT-4 and PPO or SAC with hyperparameters fixed across task comparisons.
  • Manipulation evaluation spans diverse MetaWorld and ManiSkill2 tasks, including pick-and-place, assembly, and articulated-object manipulation.
  • Six novel locomotion tasks are developed for Hopper and Ant; because expert rewards are unavailable, policies are evaluated through human judgment of rollout videos.
  • The real-robot experiment tests whether simulator-trained policies transfer to a Franka robot with minor calibration and random noise for sim-to-real transfer.
  • Interactive generation with human feedback: The Stack Cube feedback study samples three generated codes from zero-shot and few-shot methods and applies two feedback rounds.
  • Figure 2 compares L2R, Oracle, zero-shot, and few-shot reward generation on ManiSkill2 using success-rate learning curves averaged over five random seeds.
  • Figure 3 reports MetaWorld zero-shot learning curves using mean success rate and standard-deviation shading across five random seeds.
  • Figure 4 presents rollout images of novel locomotion behaviors learned with zero-shot rewards in Gym MuJoCo.

4 RESULTS AND ANALYSIS

TEXT2REWARD performs competitively across manipulation and locomotion benchmarks, transfers simulator-trained policies to a real robot, and supports reward refinement through human feedback. Results also show that prompting strategy and initial reward quality affect downstream performance.

  • Manipulation: On 13 of 17 manipulation tasks, TEXT2REWARD achieves performance comparable to expert-written rewards, while outperforming them on four tasks in success rate or convergence speed.The four gains include convergence speed on Open Cabinet Door and Handle Press, and success rate on Pick Cube and Drawer Open.
  • Manipulation: The L2R baseline is effectively applicable to only two ManiSkill2 tasks and struggles with objects whose surfaces require point-cloud representations.An additional baseline is introduced to address this limitation when evaluating shaped and staged dense rewards.
  • Prompting analysis: Few-shot prompting markedly outperforms zero-shot prompting in 2 of the 6 not-fully-solvable tasks.Few-shot code more clearly represents staged behaviors such as approach, grasp, and lift, whereas zero-shot code may combine stage rewards linearly.
  • Locomotion: Zero-shot reward functions learn six novel locomotion behaviors, with success rates evaluated over 100 rollouts per task.The table caption specifies human judgment from reviewed rollout videos as the task-success criterion.
  • Real-world deployment: Policies trained in simulation with TEXT2REWARD-generated dense rewards can be successfully deployed on a real robot for Pick Cube and Stack Cube.The real-robot evaluation uses the same 7 DoF Franka Panda arm as the ManiSkill2 simulation environment.
  • Interactive refinement: For the ambiguous instruction “control the Ant to lie down,” users can specify the desired torso orientation after observing a rollout, prompting reward-code regeneration.The example feedback requests a top-down rather than bottom-up torso orientation.
  • Interactive refinement: Human feedback improves interactive reward generation on Stack Cube, raising few-shot success from zero to one and accelerating convergence, but gains depend on the initial reward quality.Improvement is less pronounced for low-quality zero-shot codes than for few-shot codes.
  • Prompting analysis: Zero-shot prompting occasionally outperforms few-shot prompting because the quality and relevance of retrieved examples can affect which reward terms are generated.The analysis attributes specific differences to omitted terms, including door positional change in Open Door.

5 RELATED WORK

Prior work addresses reinforcement-learning reward shaping through inverse reinforcement learning, preference learning, and language-model-assisted methods. These approaches differ in their reliance on expert trajectories, human preferences, or language-based planning and control.

  • Inverse reinforcement learning recovers nonlinear reward models from expert trajectories to facilitate reinforcement-learning policy learning.It requires a large amount of high-quality trajectory data, which can be difficult to obtain for complex and rare tasks.
  • Preference learning constructs reward models from human judgments comparing pairs of actions.The required human-annotated preference data can be expensive or difficult to collect.
  • Large language models have been used to support reinforcement learning through reasoning, planning, and policy-network acquisition.This work builds on LLM capabilities as a way to transform the data-driven policy acquisition paradigm.
  • LLM-based robotics research commonly applies language models to embodied planning, reasoning, and primitive-task learning.These directions include generating plans and assisting the learning of robotic primitives.

6 CONCLUSION

TEXT2REWARD uses LLMs to automate reward shaping through interactive reward-code generation. The authors report competitive task performance, real-world deployment, and policy improvement through human-feedback-based refinement.

  • TEXT2REWARD is an interactive framework that uses LLMs to automate reward shaping for reinforcement learning.It generates reward code and incorporates feedback to refine the resulting policy behavior.
  • Policies trained with generated reward codes matched or surpassed expert-designed-code performance on the majority of evaluated tasks.The conclusion presents this as evidence of the approach’s effectiveness across the experiments.
  • A policy trained in simulation was deployed on a real robot, demonstrating real-world applicability.
  • Human feedback iteratively refines generated reward codes to address language ambiguity and improve learned-policy success rates.The process is described as improving alignment with human needs and preferences.

Appendices

The appendices document reward-generation and reinforcement-learning implementation details, including GPT-4 sampling, SAC and PPO hyperparameters, and computational training requirements.

  • Implementation details: GPT-4 generates reward functions with a sampling temperature of 0.7 in the main experiments.
  • Implementation details: SAC and PPO training use open-source algorithm implementations with task-specific hyperparameters listed in Tables 2 and 3.Table 2 covers SAC, while Table 3 covers PPO.
  • Computational requirements: Training uses AWS g5.4xlarge instances equipped with one NVIDIA A10G GPU, 16 vCPUs, and 64 GiB of memory.
  • Computational requirements: Training takes approximately 0.5 hours per MetaWorld task and 0.5 to 10 hours per ManiSkill2 task.The ManiSkill2 duration varies with task difficulty, and the default environments are used without speed optimization.

B TASK DETAILS

The paper evaluates reward generation across MetaWorld, ManiSkill2, and MuJoCo tasks using compact Pythonic environment representations and executable reward-code prompts. The task setups specify robot bodies, observations, actions, prompts, and task-specific reward components, with interactive human feedback for refinement.

  • MetaWorld: MetaWorld uses a fixed-base 7 DoF Sawyer arm for tabletop tasks with robot, object, gripper, pose, and goal observations.Actions control end-effector delta position and normalized gripper torque; initial and target positions vary across tasks.
  • ManiSkill2: ManiSkill2 uses a fixed-base Franka Panda for manipulation and mobile Sciurus17 platforms for mobile manipulation tasks.Push Chair uses a dual-arm mobile robot, while Open Cabinet Door and Open Cabinet Drawer use a single mobile arm.
  • MuJoCo: MuJoCo locomotion experiments use Ant and Hopper, whose actions are hinge-joint torques and whose observations describe the simulated agents.The environments provide predefined physics and rewards, while the study evaluates novel locomotion behaviors.
  • Prompt representation: The environment is represented through a compact Pythonic prompt using classes, attributes, typing, comments, and reusable functions.The prompt exposes object poses, velocities, joint states, point clouds, and transformation utilities for reward-code generation.
  • Reward-code prompts: Reward prompts request a compute_dense_reward function that interprets task instructions and returns a scalar reward using environment attributes.Typical components include gripper-object distance, object-goal distance, action regularization, and optional task or robot constraints.
  • Interactive refinement: Interactive generation combines previous reward code, training results, rollout videos or learning curves, and human feedback to revise rewards.This process addresses ambiguous instructions and limitations of single-turn generation for complex tasks.

D.3 INTERACTIVE RESULTS

Interactive feedback exposes concrete task failures and guides reward revisions that better encode completion conditions. In the Stack Cube example, refinement adds explicit release and stability-related reward logic after observed failures.

  • The interactive trace includes an initial reward function, human feedback, and a revised reward function for the MANISKILL2 Stack Cube task.
  • The task specification requires cube A to be stably on cube B and no longer grasped by the gripper.
  • The initial policy picked up cube A and stacked it on cube B but failed to release it afterward.
  • Human feedback requested releasing cube A after stacking it onto cube B.
  • The revised reward checks target placement, object stability, and grasp status while rewarding reaching and grasping behavior.
  • A second interaction addressed intermittent success by requesting denser reward shaping and adding a reward for releasing the grasp.

E ADDITIONAL ANALYSIS OF EFFECTIVENESS OVER PREVIOUS WORK

The additional baseline analysis shows that removing staged Python logic limits reward expressiveness on harder manipulation tasks. A modified expert baseline matches Text2Reward on simple tasks but fails on more difficult ones.

  • The modified L2R baseline adapts to only two MANISKILL2 tasks and is unsuitable for tasks involving complexly surfaced objects.
  • Oracle-L2R matches zero-shot performance on Lift Cube, Pick Cube, and Turn Faucet.
  • On the remaining three, more difficult tasks, Oracle-L2R fails to address the challenges, underscoring the need for shaped and staged dense rewards.

F EXPERIMENTS ON OPEN-SOURCE LANGUAGE MODELS

The study compares open-source and GPT-4 reward generation on MetaWorld and reports GPT-4 as the strongest performer across most tasks. Figure 8 also defines the oracle and prompting comparisons used for MANISKILL2 learning curves.

  • The experiments use instruction-tuned Llama-2 and Code-Llama as representative open-source language-model baselines.
  • Figure 8 measures MANISKILL2 task success rates and distinguishes Oracle-L2R, zero-shot, and few-shot reward-generation settings.
  • GPT-4 consistently outperforms the other models across most MetaWorld tasks, with higher success rates and more stable learning progressions.
  • Code-Llama shows moderate success with variability on Window Close and Door Unlock, while Llama-2 lags on Drawer Open and Door Close.
  • The reward-generation gap among language models is presented as a potential benchmark challenge with room for improvement.

G.1 ERROR ANALYSIS ON GENERATED FUNCTION

The error analysis finds an overall generated-reward-function error rate of around 10%, with syntax or shape mismatches forming a substantial subset. The analysis evaluates zero-shot and few-shot generations without execution feedback.

  • The analysis manually examines 100 zero-shot and 100 few-shot reward functions across 10 MANISKILL2 tasks.
  • The four error categories are class-attribute misuse, hallucinated attributes, syntax or shape errors, and wrong packages.
  • Around 10% of generated reward functions contain errors overall.
  • Syntax or shape mismatches account for 30% of the error samples, while the remaining errors arise during grounding to existing functions and attributes.
  • The error-analysis reward functions were generated without execution feedback to the language models.

H LIMITATIONS AND FUTURE WORK

The paper identifies limitations in symbolic reward representation, perception assumptions, and task scope. It also highlights opportunities to improve reward generation and extend evaluation beyond robotics.

  • The symbolic reward space may not cover all aspects of the reward.
  • The method assumes perception is already handled by off-the-shelf components.
  • Better reward-generation methods could improve success rates and enable more complex tasks.
  • Current test cases primarily cover robotics manipulation and locomotion, motivating future applications in gaming, web navigation, and household management.
  • Few-shot generation stably solves PushChair even though zero-shot generation fails on that task.
Loading 2309.11489v3…