Source-linked AI summary
Deep reinforcement learning from human preferences
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei
TL;DR
Deep RL struggles with complex goals that are difficult to specify and costly to communicate through direct human feedback. This paper learns a reward function from human feedback and shows it can train complex behaviors across Atari and simulated robotics with feedback reduced by roughly three orders of magnitude.
Problem
Deep RL lacks practical ways to specify complex, poorly defined goals when reward functions are unavailable and direct human feedback is prohibitively expensive.
Method
The paper learns a reward function from human feedback and then optimizes that learned reward for deep RL tasks.
Results
Roughly 3 orders of magnitude lower interaction complexity enabled learning most tested Atari and robotics tasks, plus novel behaviors from about an hour of feedback.
Takeaways & Limitations
Human-preference feedback can be economically scaled to state-of-the-art RL systems for complex real-world task applications.
Takeaways & Limitations
Future work is needed to improve preference-learning efficiency and expand the range of tasks to which the approach applies.
Abstract
from arXiv · showhide
For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than one percent of our agent's interactions with the environment. This reduces the cost of human oversight far enough that it can be practically applied to state-of-the-art RL systems. To demonstrate the flexibility of our approach, we show that we can successfully train complex novel behaviors with about an hour of human time. These behaviors and environments are considerably more complex than any that have been previously learned from human feedback.
1 Introduction
The paper addresses deep RL tasks whose goals are complex or poorly specified by learning reward functions from non-expert human preferences over trajectory clips. It scales this approach to deep RL, showing complex behaviors in Atari and simulated robotics with limited human feedback.
- Motivation: Complex or poorly defined goals make it difficult to construct reward functions that capture human preferences.Hand-designed rewards can produce behavior that optimizes the reward without satisfying the intended preferences.
- Method: The method learns a reward function from human preferences while simultaneously training a policy to optimize the predicted reward.Humans compare short video clips of trajectory segments rather than assigning absolute numerical scores.
- Design goals: The approach is designed to support tasks that users can recognize but not demonstrate, non-expert teaching, large problems, and economical use of feedback.These objectives motivate learning from comparisons instead of requiring demonstrations or numerical rewards.
- Experiments: 15 minutes to 5 hours of feedback suffices to learn most original RL tasks in Atari and MuJoCo even when the reward function is unobservable.The experiments also examine novel behaviors such as performing a backflip or driving with the learned reward.
- Contribution: The key contribution is scaling human feedback to deep RL and learning much more complex behaviors than prior work.The environments include physics tasks with dozens of degrees of freedom and Atari tasks without hand-engineered features.
2 Preliminaries and Method
The method replaces environmental rewards with human preferences over trajectory segments, learns a reward estimate from those comparisons, and optimizes the resulting reward with reinforcement learning. It asynchronously cycles through policy interaction, preference querying, and supervised reward-model updates, using uncertainty to select informative comparisons.
- Preference-based objective: The approach replaces environmental rewards with preferences over trajectory segments, where σ1 ≻ σ2 denotes that a human prefers the first segment.A segment is a sequence of observations and actions, and the agent aims to learn behavior from these comparisons rather than an assumed reward signal.
- Training loop: The algorithm asynchronously updates a policy, queries humans on selected segment pairs, and trains a reward estimate to fit accumulated comparisons.The policy maximizes predicted rewards, while the learned reward mapping is optimized through supervised learning on human comparisons.
- Policy optimization: The learned reward estimate reduces the problem to traditional reinforcement learning, with policy-gradient methods preferred because the estimate may be non-stationary.The experiments use A2C for Atari games and TRPO for simulated robotics tasks.
- Human preference collection: Human overseers compare short trajectory-segment clips, selecting a preferred segment, declaring them equally good, or marking them incomparable.The clips are 1–2 seconds long; incomparable comparisons are excluded from the database.
- Active querying: The system selects preference queries by choosing trajectory pairs whose predicted preferences have the highest variance across an ensemble of reward predictors.It samples many segment pairs, evaluates them with each ensemble member, and uses prediction disagreement as a crude uncertainty estimate.
3 Experimental Results
Human-preference training nearly matched reward-based reinforcement learning on simulated robotics with 700 labels, while showing substantial but more variable performance on challenging Atari games. The method also learned complex novel behaviors in under an hour using hundreds to roughly 1,300 queries.
- Simulated robotics: 700 human labels nearly matched reinforcement learning on all eight simulated robotics tasks, with comparable mean performance but greater instability and variance.The experiments evaluated performance using the tasks’ true rewards.
- Simulated robotics: 1400 labels slightly outperformed training with the true reward, potentially because learned rewards provided beneficial shaping.The learned reward function assigned positive rewards to behaviors typically followed by high reward.
- Atari games: 5500 human queries produced substantial learning on most challenging Atari games, matching or exceeding reinforcement learning on some, while synthetic labels matched or approached it on BeamRider and Pong with 3300 labels.The method had more difficulty matching reinforcement learning in these environments overall.
- Atari games: Real human feedback was generally similar to or slightly worse than synthetic feedback, often matching synthetic performance with 40% fewer labels; on Ant, humans significantly outperformed synthetic feedback.The authors attribute the gap partly to labeling errors, contractor inconsistency, and uneven coverage of state space.
- Novel behaviors: 900 queries trained a Hopper to perform repeated backflips and land upright in less than an hour.The agent consistently performed the backflip, landed upright, and repeated the sequence.
- Novel behaviors: 800 queries trained a Half-Cheetah to move forward while standing on one leg in under an hour.This experiment used the same parameters as the preceding experiments.
- Novel behaviors: Roughly 1300 queries and 4 million environment frames trained an Enduro agent to stay alongside moving cars for a substantial fraction of an episode.The agent remained almost exactly even with other cars but became confused by background changes.
4 Discussion and Conclusions
The authors show that supervised reward modeling can make training deep RL from human preferences economically scalable, while identifying limits and future goals for broader, more efficient application.
- Limitations: The reported continuous-control tests did not cover Atari because the Atari reward model requires consecutive frames rather than a single frame.This limitation is specific to the tests described in the discussion.
- Contribution: The approach provides the first evidence that preference-based techniques can be economically scaled to state-of-the-art reinforcement learning systems.The authors frame this as a step toward practical deep RL applications on complex real-world tasks.
- Future work: Future work may improve the efficiency of learning from human preferences and expand the range of applicable tasks.The authors identify both efficiency and task coverage as directions for further research.
- Long-term goal: A long-term goal is to make learning from human preferences no more difficult than learning from a programmatic reward signal.This would help apply powerful RL systems to complex human values rather than low-complexity goals.
A Experimental Details
The experiments removed behavior-dependent termination signals so human preferences were the agent’s only guidance about desired behavior. Training used staged and annealed preference labeling, with Atari reward-predictor pretraining and uncertainty-based query selection.
- Episode handling: Variable-length episodes were removed because behavior-dependent termination conditions can encode task information unavailable in the reward function.This prevented termination signals from confounding learning from human preferences alone.
- Episode handling: Robotics termination conditions were replaced with penalties, while Atari life-loss and episode-end signals were withheld from the agent.Atari environments were reset internally but treated as single continuous episodes; synthetic oracle feedback used learned penalties except in Pong.
- Preference-label training: 200 epochs of reward-predictor pretraining were used in Atari before reinforcement-learning training began.Training initially compared trajectory segments from an untrained policy, then annealed labels to balance early predictor quality with adaptation to newly encountered states.
- Preference-label training: An ensemble of 3 predictors generated 10 times more candidate clip pairs than were shown to humans, selecting presented clips by maximum predictor variance.With real human feedback, label-rate annealing was attempted but remained approximate because contractors provided feedback unevenly.
A.1 Simulated Robotics Tasks
The simulated robotics experiments removed torque penalties because torques are not directly visible to human supervisors. Policies learned from preference comparisons used TRPO and a neural reward predictor, with exploration and labeling procedures adapted to changing rewards.
- Torque penalties were removed because torques are not directly visible to human supervisors and therefore do not represent human trajectory preferences well.
- TRPO optimized policies with γ = 0.995 and λ = 0.97, while a two-layer, 64-hidden-unit reward predictor used leaky ReLUs with α = 0.01.Compared trajectory segments lasted 1.5 seconds, spanning 15 to 60 timesteps depending on the task.
- 0.01 entropy bonus was used on all tasks except swimmer, which used 0.001, to encourage exploration under changing reward functions.Reward predictions were normalized to standard deviation 1.
- 25% of comparisons came from a randomly initialized policy, and labeling after T frames followed 2 ∗106/(T + 2 ∗106).
- The flexible reward-predictor architecture supported generalization beyond these tasks’ second degree polynomial reward functions.
A.2 Atari
The Atari experiments use standardized visual and optimization settings while inferring rewards from trajectory preferences rather than visible score information. Training uses a learned reward predictor with short segment comparisons and an asynchronously updated, recency-limited label buffer.
- Environment and preprocessing: Atari agents use standard environment wrappers, including frame preprocessing, life-loss episode termination, and rewards clipped to [−1, 1].Episodes begin with 0 to 30 no-ops; four frames are stacked, frameskip is 4, and adjacent frames are max-pooled.
- Environment and preprocessing: The score display is replaced with a constant black background across all seven games to prevent trivial reward inference from the score.BeamRider also blanks the enemy ship count, while Enduro blanks the speedometer.
- Policy and predictor training: Atari policies are optimized with synchronous A3C (A2C) using standard settings, including γ = 0.99, N = 16 workers, and 50 million training timesteps.The learning rate is 0.0007 and decays linearly to zero after 80 million timesteps, although runs train for only 50 million timesteps.
- Policy and predictor training: The reward predictor processes 84x84x4 stacked-frame inputs through four convolutional layers, a 64-unit fully connected layer, and a scalar output.Convolutional layers use batch normalization and dropout with α = 0.5 to reduce overfitting.
- Preference feedback: 25-timestep trajectory segments are compared for preference feedback, corresponding to 1.7 seconds at 15 fps with frame skipping.The system collects 500 comparisons from a randomly initialized policy at training start and decreases labeling roughly as 5 ∗106/(T + 5 ∗106).
- Preference feedback: The predictor trains asynchronously, processes roughly 1 label per 10 RL timesteps, and repeatedly samples only the latest 3,000 labels.The limited buffer gives greater weight to new labels when the data distribution shifts.
B Instructions Provided to Contractors
Contractors were asked to teach an AI to play Atari games by providing feedback on its performance.
- B.2 Atari: Contractors taught an AI to play Atari games by giving feedback on how well it was playing.