PaperlayerView exact revision on arXiv

Source-linked AI summary

Reward Design with Language Models

Minae Kwon, Sang Michael Xie, Kalesha Bullard, Dorsa Sadigh

arXiv:2303.00001v1cs.LGcs.AIcs.CL

TL;DR

Reward design is difficult when human objectives are hard to express with reward functions or require many labeled examples. The paper prompts an LLM as a proxy reward function that evaluates RL episodes against natural-language objectives, and reports improved alignment across tasks over supervised learning. The framework is promising but currently uses only binary rewards and is supported by an initial user study.

  • Problem

    Reward functions are unintuitive for hard-to-specify preferences, while capturing user objectives with labeled examples requires costly amounts of data.

  • Method

    Users provide a natural-language objective through a few examples or a description, and an LLM evaluates textualized RL episodes to produce parsed reward signals.

  • Results

    35% average improvement in objective-aligned agent accuracy over the baseline is reported across the evaluated tasks.

  • Takeaways & Limitations

    The results indicate that LLM proxy rewards can train RL agents aligned with user objectives using few-shot or zero-shot prompting.

  • Takeaways & Limitations

    The framework currently specifies only binary rewards, and the user evaluation is an initial pilot study with 10 users.

Abstract

from arXiv · show

Reward design in reinforcement learning (RL) is challenging since specifying human notions of desired behavior may be difficult via reward functions or require many expert demonstrations. Can we instead cheaply design rewards using a natural language interface? This paper explores how to simplify reward design by prompting a large language model (LLM) such as GPT-3 as a proxy reward function, where the user provides a textual prompt containing a few examples (few-shot) or a description (zero-shot) of the desired behavior. Our approach leverages this proxy reward function in an RL framework. Specifically, users specify a prompt once at the beginning of training. During training, the LLM evaluates an RL agent's behavior against the desired behavior described by the prompt and outputs a corresponding reward signal. The RL agent then uses this reward to update its behavior. We evaluate whether our approach can train agents aligned with user objectives in the Ultimatum Game, matrix games, and the DealOrNoDeal negotiation task. In all three tasks, we show that RL agents trained with our framework are well-aligned with the user's objectives and outperform RL agents trained with reward functions learned via supervised learning

1 INTRODUCTION

The paper proposes using language prompts and LLMs as proxy reward functions to make preference specification more intuitive and data-efficient. It evaluates this framework across multiple tasks and reports improved objective alignment over a supervised-learning baseline.

  • Motivation: Users traditionally specify desired behavior through reward functions or large labeled datasets, but both approaches are difficult, costly, and poorly generalize to new objectives.Reward functions are unintuitive and vulnerable to reward hacking, while labeled examples require substantial data to capture preference nuances.
  • Approach: The proposed interface lets users communicate preferences with natural-language descriptions or a few examples rather than crafting reward functions.The approach targets objectives that are difficult to define directly, such as versatile negotiating behavior.
  • Approach: An LLM evaluates RL episode trajectories against the user’s objective and converts its output into an integer reward for agent training.The prompt may use few-shot examples or a single phrase for well-known concepts.
  • Evaluation: The framework uses few-shot prompting in the Ultimatum Game and DealOrNoDeal and zero-shot prompting in Matrix Games.The framework is presented as a general proxy-reward approach for training objective-aligned agents.
  • Results: 35% average improvement in objective-aligned agent accuracy is reported relative to the baseline.The paper also reports a pilot study in which 10 users rated the proposed agent as significantly more aligned, with p<0.001.

2 RELATED WORK

The related work positions this paper among language-guided reward shaping and foundation-model approaches. Its distinction is using in-context LLM judgments to evaluate high-level objective satisfaction without fine-tuning or many demonstrations.

  • Using Language for Reward Shaping: RLHF approaches use LLMs as rewards by fine-tuning them on large amounts of user data, whereas this work uses in-context learning from a handful of examples.The contrast concerns how the reward model is adapted to user preferences.
  • Using Language for Reward Shaping: Other language-guided methods train agents to complete intermediate subtasks, while this framework uses an LLM to judge whether a policy satisfies a higher-level task.The paper emphasizes objective evaluation rather than subtask generation.
  • RL and Foundation Models: Foundation-model work has used LLMs for planning or pretrained models for representation learning, but this paper uses an LLM to identify hard-to-specify human objective properties and design rewards.The desired task is specified with text rather than videos, and the framework targets broader objective specification than the cited robot-task setting.

3 USING LLMS AS A REWARD SIGNAL

The framework formalizes an LLM as a proxy reward function within an MDP. A prompt combines the task, user objective, and textualized episode information, and the parsed LLM output supplies the RL reward.

  • Formalization: The paper models objective-aligned training as an MDP with states, actions, transition probabilities, rewards, and a discount factor.In DealOrNoDeal, states represent negotiation utterances so far and actions are possible utterances.
  • Proxy Reward: Instead of mapping states and actions directly to real-valued rewards, the framework maps a text prompt to an LLM-generated string.The LLM is defined as a function from strings to strings, with the prompt and output represented in the same space.
  • Prompt Construction: The prompt concatenates a task description, user examples or objective description, and a textual representation of an RL episode.The episode representation can describe a final state, final action, trajectory, or another episode-level representation.
  • Reward Construction: The LLM answers whether the episode satisfies the user objective, and its output is parsed into an integer used as the agent’s reward.The resulting reward is used during RL training to update the agent through repeated episode rollouts.

4 EXPERIMENTS

Across Ultimatum Game, Matrix Games, and DEALORNODEAL, experiments test whether LLM-generated rewards can produce objective-aligned agents using few-shot or zero-shot prompts. The framework achieves strong alignment across simple and longer-horizon tasks, including human-rated negotiation styles.

  • Experimental Setup: The experiments evaluate few-shot prompting in Ultimatum Game and DEALORNODEAL, and zero-shot prompting in Matrix Games.DEALORNODEAL addresses objective-aligned reward signals in a longer-horizon negotiation domain.
  • Ultimatum Game: A single example with an explanation preserves high LLM labeling accuracy, while supervised learning trained on the same example loses accuracy.Removing explanations reduces LLM labeling accuracy by 31.67% on average and RL agent accuracy by 28.8%.
  • Matrix Games: Zero-shot LLM rewards improve matrix-game labeling accuracy over the No Objective baseline by 48% with regular ordering and 36% with scrambled ordering.Scrambling outcome associations lowers accuracy for most solution concepts, potentially because the games become more out-of-distribution for the LLM.
  • Matrix Games: RL agents trained with LLM rewards achieve perfect accuracy for Total Welfare and Equality and 75% for Rawlsian Fairness and Pareto-optimality.The No Objective baseline receives lower accuracy for all objectives.
  • DEALORNODEAL: In DEALORNODEAL, LLM labeling improves RL agent accuracy over supervised learning by 46% on average and approaches true-reward performance, underperforming by 4% on average.For Versatile, skewed negative labels prevent correct behavior despite similar model labeling performance.
  • DEALORNODEAL: Agents trained with users’ preferred negotiation styles receive higher alignment ratings than agents trained with opposite styles, averaging 3.72±1.2 versus 1.56±1.05, p < 0.001.Users selected different preferred styles, and changing prompt examples produced agents aligned with those differing objectives.

5 ANALYSIS OF DATA EFFICIENCY & PROMPT DESIGN

The paper examines data efficiency and prompt-design robustness, finding that LLM rewards can match supervised learning with fewer labels and remain robust across prompt variations.

  • 10 labeled examples suffice for supervised learning to reach comparable Ultimatum Game labeling accuracy, whereas one example does not.
  • LLM labeling remains robust to varied keywords, example negotiations, and explanations, with all tested prompts outperforming supervised learning.
  • Explanation quality is the most important tested prompt component for determining LLM labeling accuracy in DEALORNODEAL.

6 LIMITATIONS & FUTURE WORK

The paper identifies limited user-study scale and binary rewards as limitations, while proposing larger studies and multimodal foundation models as future directions.

  • The user study is a first step, and the authors plan to evaluate the approach with a larger user study.
  • Multimodal foundation models could provide more complex environment states through images or other modalities while retaining a language interface for objectives.
  • The framework currently produces only binary rewards, motivating future use of language-model word likelihoods for non-binary reward signals.

A.1 SUMMARY OF RESULTS: IS IT IS POSSIBLE TO USE LLM AS A PROXY REWARD IN RL TRAINING?

Across tasks and objectives, the framework produces objective-aligned reward signals and policies whose accuracy is close to policies trained with the true reward.

  • Average Labeling and RL Agent Accuracy are reported across different objectives for each task over 3 seeds.
  • The approach produces more objective-aligned reward signals than baselines and policies close in accuracy to those trained with the true reward.

A.2 MORE RELATED WORKS

The related-work discussion situates the framework among reward design, imitation and preference learning, and language-based reward-shaping approaches.

  • Reward design is difficult because conflicting objectives require human trade-offs, while misspecified rewards can produce reward hacking.
  • Imitation and preference-based methods may require large datasets or restrictive assumptions such as reward-function linearity.

A.3 LLM DEFINITION OF OBJECTIVES IN THE MATRIX GAME

The table reports LLM-generated definitions of the matrix-game objectives. Definitions are correct except for Rawlsian Fairness, which is only partially correct.

  • The LLM provides correct definitions for the objectives except Rawlsian Fairness, which is partially correct.

A.4 DETAILS ON RL ENVIRONMENTS AND TRAINING

The paper specifies distinct RL environments and training procedures for the Ultimatum Game, Matrix Games, and DEALORNODEAL, with LLM outputs converted into reward signals for agent learning.

  • Ultimatum Game: The Ultimatum Game uses a single-horizon accept-or-reject decision over continuous proposed splits, trained with DQN for 1e4 timesteps across 3 seeds.The DQN learning rate is 1e-4, using a Stable Baselines3 MLP policy.
  • No Objective Baseline: Without an objective, the LLM performs close to random across matrix-game objectives and often hallucinates rewards or gives incoherent reasoning.Reported random-baseline accuracies are Welfare 0.125, Equality 0.125, Rawlsian Fairness 0.078, and Pareto-optimality 0.172.
  • Matrix Games: The Matrix Game environment presents one of four joint actions without observations, and DQN agents train for 500 timesteps across 3 seeds.The learning rate is 1e-4, with a Stable Baselines3 MLP policy.
  • DEALORNODEAL: In DEALORNODEAL, agents negotiate object allocations through coarse dialogue acts, then receive utility-based points only when they agree.Dialogue acts focus learning on diverse and interpretable strategies instead of direct natural-language generation.
  • DEALORNODEAL: DEALORNODEAL agents first learn next-token prediction from human negotiations, then optimize dialogue-act rewards with REINFORCE.Training uses 250 contexts for 1 epoch and policies instantiated with four GRUs.
  • LLM reward parsing: The handcrafted parser converts Yes/No LLM responses into binary rewards, while malformed responses cause episodes or evaluation examples to be omitted.The parser searches for “Yes” and “No”; zero-shot responses may require manual parsing because LLM output is variable.

A.5 SL MODEL ARCHITECTURE AND TRAINING

This section describes supervised-learning baselines and analyzes how model choice, prompt construction, and prompt variation affect labeling and downstream RL performance.

  • SL model architecture and training: The Ultimatum Game supervised-learning baseline is a one-hidden-layer, depth-32 ReLU MLP trained on the same 10 LLM examples and evaluated on 50 held-out examples.The model uses Adam for 5 epochs and is selected by best test accuracy.
  • SL model architecture and training: The DEALORNODEAL supervised-learning model predicts binary labels from negotiation contexts, dialogue acts, and outcomes using MLP encoders.Its implementation follows prior work by Kwon et al. and Lewis et al.
  • Labeling efficiency: SL requires on the order of hundreds more labeled examples to achieve accuracy comparable to the LLM.This comparison is summarized for the supervised-learning and LLM labeling approaches.
  • Prompt variation: Varying prompts generally has little impact on LLM labeling accuracy, including changes to keywords, example negotiations, and explanations.The prompt-variation study targets the Stubborn negotiating style in DEALORNODEAL.
  • Prompt components: Removing the Ultimatum Game task description raises accuracy slightly with one explained example but lowers it slightly with 10 unexplained examples, so its influence is inconclusive.The reported changes are 8% and 9% increases in the first setting, versus 4.4% and 5.3% decreases in the second.
  • Model comparison: GPT-2 underperforms GPT-3 in labeling and RL-agent accuracy, while outperforming SL in labeling accuracy but slightly underperforming it for RL-agent accuracy.Across styles and seeds, the gaps are 15% and 49% versus GPT-3, and 24% and 2.7% relative to SL.
  • Prompt construction: Prompt construction uses multiple-choice outcomes, step-by-step reasoning, and three counterbalanced self-play negotiations with concise explanations.Figures provide examples of few-shot and zero-shot prompts, scrambled matrix-game outcomes, and DEALORNODEAL prompts and negotiations.
Loading 2303.00001v1…