Scalable agent alignment via reward modeling: a research direction

Jan Leike; David Krueger; Tom Everitt; Miljan Martic; Vishal Maini; Shane Legg

Source-linked AI summary

Scalable agent alignment via reward modeling: a research direction

Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, Shane Legg

arXiv:1811.07871v1Nov 19, 2018cs.LGcs.AIcs.NEstat.ML

TL;DR

Real-world reinforcement learning lacks suitable reward functions because task objectives are often available only through implicit user intentions. The paper proposes learning rewards from user feedback and optimizing them with reinforcement learning, while examining scalability challenges and ways to establish trust. It concludes that reward modeling is a research direction with promising examples but unresolved questions about scaling, feedback efficiency, robustness, and reward gaming.

Problem
Real-world tasks often lack readily measurable reward functions because their objectives are available only indirectly through user intentions.
Method
Reward modeling learns a reward function from user feedback and trains a reinforcement learning policy to optimize that learned reward.
Results
The paper develops a research direction around reward modeling, detailing scaling challenges, mitigation approaches, trust-building avenues, and existing successes in several task domains.
Takeaways & Limitations
Reward modeling could organize alignment research toward training agents beyond human-level performance, but its practical scalability remains an open research question.
Takeaways & Limitations
Achieving sufficient reward-model accuracy may require data that exceeds a realistic production or labeling budget, depending on generalization over the state distribution.

Abstract

from arXiv · show

One obstacle to applying reinforcement learning algorithms to real-world problems is the lack of suitable reward functions. Designing such reward functions is difficult in part because the user only has an implicit understanding of the task objective. This gives rise to the agent alignment problem: how do we create agents that behave in accordance with the user's intentions? We outline a high-level research direction to solve the agent alignment problem centered around reward modeling: learning a reward function from interaction with the user and optimizing the learned reward function with reinforcement learning. We discuss the key challenges we expect to face when scaling reward modeling to complex and general domains, concrete approaches to mitigate these challenges, and ways to establish trust in the resulting agents.

1 Introduction

Real-world tasks often lack explicit reward functions because objectives are available only indirectly through user intentions. The paper proposes reward modeling as a scalable research direction, while identifying assumptions, challenges, and trust requirements.

Real-world tasks lack readily measurable objectives because their reward functions are not specified and goals are available indirectly through user intentions.
The paper asks how to create agents that behave in accordance with user intentions while avoiding both uncreative behavior and degenerate solutions.
Reward modeling separates learning what to achieve from learning how to achieve it: user feedback trains a reward model, then reinforcement learning optimizes it.
The direction aims to scale to complex domains by improving users’ evaluation abilities, including recursively using trained agents to assist with evaluating outcomes.
Its desiderata are scalability to superhuman agents in general domains, competitive cost and performance, and pragmatic rather than perfect trust.
The approach assumes user intentions can be learned accurately and that evaluating outcomes is easier than producing correct behavior, with exceptions for low-dimensional outcomes.
Success is not guaranteed: the paper frames reward modeling as research intended to clarify whether it can scale, while trust requires design, testing, interpretability, verification, and theory.

2 The agent alignment problem

The paper frames alignment as sequential decision-making in which an agent’s policy should produce behavior consistent with user intentions. It surveys interaction forms and highlights specification risks that become harder as task scope and agent performance increase.

Agent alignment is modeled as an agent interacting sequentially with an environment while communicating with the user through an unspecified interaction protocol.
A solution is a policy that produces behavior in accordance with user intentions rather than behavior determined by the environment alone.
User interaction can provide demonstrations, scores, actions, values, advantages, or preferences over trajectories.
Specification problems: Specification problems include off-switch incentives, irreversible side effects, cheating when unsupervised, containment circumvention, and creation of potentially unaligned subagents.
Specification problems: Common metrics such as BLEU, Inception score, and Fréchet inception distance can poorly proxy intended performance and produce degenerate solutions when optimized directly.
Difficulty factors: Alignment difficulty varies with task complexity, available actuators, opportunities for unacceptable outcomes, and agent performance.

3 Scaling reward modeling

Scaling reward modeling requires handling real-world tasks without hand-engineered rewards, shifting credit assignment from users to agents and boosting human evaluation capacity. The paper proposes recursive reward modeling while identifying unresolved challenges in feedback interpretation, reward specification, generalization, and evaluation.

Motivation: Real-world tasks lack readily measurable reward functions, so deep RL must learn complex behavior as intended without hand-engineered rewards.The task objective is only indirectly available through human users’ intentions.
Reward modeling: Reward modeling separates learning what to achieve from learning how to achieve it by training a reward model from user feedback and optimizing it with reinforcement learning.The user and reinforcement-learning agent remain in the loop concurrently.
Reward modeling: Reward modeling shifts credit assignment from the user to the agent, because users evaluate outcomes while reinforcement learning searches for actions producing favorable results.This can make teaching a reward function easier than performing the task itself under the stated assumption.
Advantages of reward modeling: Separating reward learning from policy learning can reduce feedback demands, distinguish reward-model from policy alignment, and allow more capable RL agents to be plugged into the setup.Users need not provide feedback on every environment interaction, which is impractical for sample-inefficient deep RL.
Design specification: Reward modeling aims to address design-specification problems by learning a sufficiently accurate reward function that excludes undesired incentives, but this depends on capacity, data, generalization, and behavioral fidelity.The existence argument shows such a reward function can be defined if aligned histories are nonempty, while practical learnability remains unresolved.
Recursive reward modeling: Scaling introduces open challenges in interpreting varied user feedback and evaluating technical, complex, high-dimensional, delayed, or unfamiliar outcomes.Recursive reward modeling boosts user feedback by using agents trained on simpler, narrower tasks to assist evaluation for more capable agents.

4 Challenges

Scaling reward modeling raises challenges around reward-model accuracy, feedback costs, distribution shift, loopholes, and robust optimization. The paper presents these as open research questions rather than solved obstacles.

Reward-model quality is critical because missing objective aspects can produce undesirable degenerate behavior.The agent’s behavior may depend on the learned reward model in a potentially fragile way.
The central scaling questions concern affordable feedback, distributional robustness, loophole prevention, proactive safety, and reliable policy optimization.
4.1 Amount of feedback: Sufficient reward-model accuracy depends on obtaining enough labeled data within a realistic budget and generalizing across the relevant state distribution.With infinite data from the right distribution and enough capacity, the correct reward function can be learned; practical budgets make generalization decisive.
4.1 Amount of feedback: Agents lacking human inductive biases may solve tasks in surprising or undesirable ways, suggesting that labeled data alone may be insufficient.
4.2 Feedback distribution: Off-policy reward prediction is needed to encourage unexplored positive trajectories and discourage unexplored negative ones, but principled solutions remain unclear.Possible mitigations include out-of-distribution detection with human deferral or widening the training distribution.
4.3 Reward hacking: Reward hacking occurs when agents exploit reward loopholes, creating tension between desired creative solutions and unintended behavior.Reward gaming targets misspecified rewards, while reward tampering interferes with the reward-computation process.
4.3 Reward hacking: Reward gaming has succeeded in several domains, but whether it can be overcome reliably in practice remains a major open question.Reported examples include gridworlds, Atari games, and continuous motor-control tasks.
4.4 Reward-result gap: Even a correct reward function may yield unaligned behavior if the agent fails to converge to an optimal policy.

5 Approaches

The proposed mitigation approaches tighten the feedback loop between users and agents and extend feedback beyond observed behavior. They aim to adapt reward models online while using hypothetical scenarios to anticipate unacceptable outcomes.

Frozen reward models can be exploited because discovered loopholes receive no additional user feedback or correction.
Online reward-model training lets user feedback adapt to the agent’s visited state distribution and correct discovered reward-model loopholes.Concurrent feedback provides a tighter loop between user judgments and agent behavior.
Off-policy feedback communicates that outcomes are undesirable before they occur and requires reward-model accuracy on unvisited states.Combined with model-based RL, it can support proactive evaluation of potential outcomes.
Generative models could create hypothetical counterfactual scenarios for proactive feedback, although distribution shift may make them difficult for humans to understand.

5.3 Leveraging existing data

Existing human-created video and prose contain information about human intentions despite lacking high-quality reward labels. The paper discusses unsupervised, hierarchical, and language-based ways to use this information.

Large collections of human-created video and prose contain useful information about human intentions even when they lack direct reward annotations.
Hierarchical reward models could let users provide low-level and high-level feedback at matching temporal levels.For writing, low-level feedback may target spelling, fluency, and tone, while high-level feedback targets plot and character development.
Natural-language-conditioned rewards can represent human-readable task labels and provide a separate channel that may be easier to protect and harder to spoof.
Natural language could make feedback more efficient, improve generalization, and support interpretability, especially for abstract high-level concepts.The reward model may require targeted training for representations to correspond neatly with natural-language expressions.
Annotating large existing media collections can be expensive; one estimate places the cost of annotating 500,000 hours of movies at about $50 million.

5.6 Model-based RL

Model-based reinforcement learning can incorporate the reward model into planning, allowing agents to evaluate unvisited actions and future outcomes. This supports proactive safety, faster adaptation, and defenses against reward tampering.

A model-based agent can use the reward model during planning to estimate rewards for actions it has not taken, provided off-policy accuracy holds.
Planning can reveal unacceptable outcomes before execution and help solicit user feedback about outcomes that have not yet occurred.
Environment models can back up reward-model changes to value estimates without additional environment interaction, enabling faster adaptation.
Evaluating future outcomes with the current reward model can provide principled defenses against reward tampering.Such agents have no incentive to alter their reward functions or manipulate the register holding the reward signal.

5.7 Side-constraints

Reward modeling can be supplemented with learned side-constraints that block unacceptable actions, though specifying those actions increases the human’s burden.

Blocking unacceptable actions can be more effective than assigning large negative rewards, which later rewards may compensate for.Reward hacking and errors in the agent’s world model can amplify this problem.
The same feedback techniques can train side-constraint models or enforce constraints during policy updates.
Side-constraints require humans to understand which actions may lead to unacceptable outcomes.In some domains, other agents may need to assist with this task through recursive reward modeling.

5.8 Adversarial training

Adversarial training can expose reward-model weaknesses and reward-hacking opportunities, but defining general perturbations and reviewing failures may make data requirements costly.

Adversarial training explicitly exposes models to perturbed inputs and is currently the most empirically effective strategy for mitigating neural-network adversarial inputs.
Agents could discover reward-model weaknesses, reward-hacking opportunities, and minimal perturbations causing unacceptable outcomes.This approach is analogous to red teams that seek attack strategies such as security vulnerabilities.
Users can review discovered failure cases and add them to the feedback dataset.
Adversarial training may increase data requirements beyond affordable limits even if it fixes the target problem.
Reward models should express uncertainty to support informative feedback collection and safer decisions under high uncertainty.
Bayesian methods distinguish irreducible from epistemic uncertainty, while ensembles provide a strong baseline for neural-network uncertainty estimation.
Calibration can align subjective uncertainty with empirical mistake frequency, but reliable handling of out-of-distribution states requires higher-quality estimates.

5.10 Inductive bias

Reward models must generalize appropriately beyond observed outcomes, making their inductive biases a central design challenge.

Because all possible outcomes cannot be represented in training, reward models need inductive biases that support appropriate generalization.Distributed representations and compositionality may help address the curse of dimensionality.
Standard piece-wise linear activations can extrapolate linearly far from training data, driving estimated rewards toward positive or negative infinity for extreme inputs.

6 Establishing trust

Establishing trust requires evaluating, interpreting, and verifying reward models and policies, while recognizing that reward modeling is not guaranteed to solve alignment.

Figure 5 compares learned rewards with true Atari rewards across nine games, revealing strong alignment in some games and poor alignment in others.
Safety certificates could provide third-party evidence of responsible development, regulatory compliance, or system trustworthiness.Possible forms include secret test-suite scores, interpretability evidence, and machine-checkable formal proofs.
Separating objective learning from behavior learning allows trust in the reward model and policy to be assessed separately.Reward alignment can be evaluated using user feedback, and the reward model may be easier to interpret and debug than the policy.
A plan generator can produce a human-readable course of action for optional user review before a plan executor implements it.
Clean, understood design choices can make behavior more predictable, while formally specified safety properties can become explicit design requirements.
Held-out tests can evaluate reward models and policies on simulated or adversarial environments designed to elicit misbehavior.
Formal verification could eventually verify properties of policies and reward functions, but current deep-learning methods lack strong theoretical guarantees.

7 Alternatives for agent alignment

The paper compares several alignment strategies with reward modeling, emphasizing trade-offs in scalability, feedback requirements, computational tractability, and trust. It identifies useful complementary roles for imitation, inverse reinforcement learning, myopic RL, expert-reasoning imitation, and debate.

7.1 Imitation learning: Imitation learning requires expert demonstrations, which may be more expensive than feedback, and human actions may omit crucial intuition or planning.The paper also questions whether imitation generalizes reliably beyond observed behavior.
7.1 Imitation learning: Imitation learning may serve as a stepping stone because trained agents can act as research assistants supporting other alignment efforts.The paper nevertheless considers it unlikely to remain competitive with other strategies in the longer term.
7.2 Inverse reinforcement learning: Inverse reinforcement learning treats behavior trajectories as feedback for recovering a reward function, but the problem is under-constrained because rewards are not uniquely identifiable from behavior alone.The paper notes that R = 0 is always a solution.
7.3 Cooperative inverse reinforcement learning: CIRL models reward modeling as a user-agent game, but its performance depends on the prior over reward functions and its optimal solution is not realistic to compute.CIRL therefore replaces direct reward specification with prior specification while presuming interactions that may be difficult to prescribe.
7.3 Cooperative inverse reinforcement learning: CIRL highlights that latent-reward optimization can help when feedback is unavailable for every state-action pair, while efficient communication of user intentions remains central.These insights motivate the paper’s broader reward-modeling direction.
7.4 Myopic reinforcement learning: Myopic RL lets users reward individual actions without reproducing the desired behavior, but it shifts credit assignment onto the user and limits performance.TAMER learns from near-term value feedback, while COACH learns from advantage-function feedback; such methods may suit tasks with easy human credit assignment or serve as building blocks.
7.5 Imitating expert reasoning: Imitating expert reasoning may improve interpretability and trust, but its economic competitiveness with recursive reward modeling depends on reasoning quality and whether Assumption 2 holds.The approach relies on the hope that improved models remain aligned when expert reasoning is aligned with the user.
7.6 Debate: Initial MNIST debate experiments boosted a sparse classifier’s accuracy, but more research is needed to assess whether debate scales as an alignment approach.The paper presents the experiments as promising rather than decisive.

8 Discussion

The discussion frames alignment as communicating a user’s task-specific intentions well enough for trust, using scalable reward modeling as the paper’s central research direction. It proposes empirical projects and argues that concrete technical challenges justify cautious optimism about applying alignment research to complex real-world domains.

Summary: The paper targets alignment of a single agent to a single user by communicating task-specific intentions rather than learning the entire preference payload.The stated goal is to enable trust in the trained agent.
Summary: Scaling reward modeling connects supervised learning for reward models with reinforcement learning for policies, while organizing AI-safety problems into one research narrative.The paper presents this direction as relevant to agents beyond human-level performance.
Concrete research projects: The proposed research program is empirically actionable through deep-RL experiments, harder tasks, and work on testing, interpretability, formal verification, and deep-RL theory.Existing games and simulated environments can provide synthetic user feedback and evaluate learned reward models when their programmed rewards are hidden.
Outlook: Real-world deployment lacks well-specified reward functions, so alignment research is presented as a way to address a potential bottleneck for complex domains.The paper characterizes the field as early-stage but views its scaling challenges as concrete technical problems for targeted research.