Reward learning from human preferences and demonstrations in Atari

1 Introduction

• Poor preference-guided exploration limits state coverage and query diversity, while preferences provide only a few hundred bits of human information per hour.Tasks with poorly defined goals motivate feedback-based alternatives; unlike demonstrations, judging outcomes requires less task familiarity but imitation cannot substantially exceed human performance.Source: §1 · pp. 1–2
• The method initializes the policy with DQfD demonstration pretraining, then combines demonstrations and trajectory preferences to train a reward model for improving that policy.The introduction frames this combination as addressing limitations of imitation learning and preference-only reward learning.Source: §1 · pp. 1–2
• The Arcade Learning Environment provides diverse, challenging Atari tasks with true rewards for objective evaluation and synthetic preferences for faster experimentation.The method is evaluated on Atari because the games benefit from nonlinear function approximation.Source: §1 · p. 2
• Demonstrations typically halve human time to a given performance and enable exploration-heavy games; direct human feedback lets the method surpass demonstrators in 2 of 9 games.The games named for exploration-heavy learning are Hero, Private Eye, and Montezuma’s Revenge.Source: §1 · p. 2, §1.1 · p. 2

1.1 Related work

• Prior deep-RL feedback work trains policies directly or handles noisy feedback and unsafe actions; Christiano et al. (2017) is the direct predecessor in tasks, rewards, architectures, and preference learning.The broader literature includes human ratings and rankings, with Warnell et al. (2017), Lin et al. (2017), and Saunders et al. (2018) representing distinct approaches.Source: §1.1 · p. 2
• Unlike prior demonstration-plus-deep-RL methods using hand-coded rewards, this work learns rewards from demonstrations and direct feedback rather than relying on fixed intrinsic objectives.Demonstration-based reward learning includes autolabeling demonstrations as preferable to the agent policy, closely related to generative adversarial imitation learning.Source: §1.1 · pp. 2–3

2.1 Setting

• The task supplies demonstrations and preference comparisons instead of environment rewards, with demonstrations provided initially and preferences collected as the agent trains.The agent aims to imitate demonstrations and maximize reward inferred from demonstrations and preferences.Source: §2.1 · p. 3

2.2 The training protocol

• The protocol behavior-clones the policy on demonstrations, gathers initial clips for preference labels, trains the reward model, then alternates policy learning and further preference-based model updates.The policy is pretrained a second time with reward-model rewards before the main loop because parts of the DQfD loss require reward labels on demonstrations.Source: §2.2 · pp. 3–4

2.3 Training the policy

• The policy uses DQfD, combining prioritized dueling double Q-learning with one- and three-step returns, a large-margin expert-action loss, and L2 regularization.It learns from expert and agent transitions in a replay buffer, with rewards computed from observations by the reward model.Source: §2.3 · p. 4

2.4 Training the reward model

• The reward model predicts reward from observations and learns from annotator comparisons of 25-step clips, including preferred or equally preferred judgments; incomparable pairs are discarded.The model is trained as a preference predictor using cross-entropy loss.Source: §2.4 · p. 4

2.5 Selecting and annotating the video clips

• Annotation clips are sampled uniformly from initial and training trajectories; some experiments add automatically labeled pairs favoring demonstrations, and most use a synthetic oracle reflecting Atari game rewards.Each initially requested clip can yield two extra labeled pairs without annotator input.Source: §2.5 · p. 5

3 Experimental results

• Preference feedback outperforms imitation in every game except Private Eye, and does so under every feedback condition in six of nine games.In Private Eye, preference feedback is seriously damaging except when demonstrations are used for labeling.Source: §3 · p. 6, Figure 1 · p. 6
• Demonstrations greatly help Hero, Montezuma’s Revenge, Private Eye, and Q*bert, but can hurt in Enduro, where the expert performs relatively poorly.In Q*bert, demonstrations permit better performance with 1100 labels than the full no-demos schedule; benefits appear to saturate quickly, with 50% of demonstrations performing similarly to the full-demo setup.Source: §3 · p. 6, Figure 1 · p. 6
• Human feedback is superior to synthetic feedback only in Beamrider, similar in most games, and clearly inferior in Breakout, Montezuma’s Revenge, and Pong.The reported causes include reward shaping that misaligns reward models and, in Montezuma’s Revenge, sensitivity to labeling errors.Source: Figure 1 · p. 6, §3 · p. 7
• Automatic preference labels improve performance in Private Eye, Hero, and Montezuma’s Revenge, with no significant differences on most other games.Breakout is an exception: its low-quality demonstrations hurt performance.Source: Figure 1 · p. 6, §3 · p. 7

3.1 Use of human time

• With only 3400 labels, the worst-performing game using demonstrations and automatic labels beats the median without demonstrations and 6800 labels.For each feedback schedule, most games benefit from demonstrations and their use in annotation; demonstrations with some preference feedback also ensure performance above random-agent scores in any game.Source: §3.1 · p. 7, Figure 2 · p. 7

3.2 Quality of reward model

• Synthetic preference models are reasonably well aligned with true reward in most games, while human feedback fails in Montezuma’s Revenge but succeeds there synthetically.Human feedback misses fine-grained distinctions in Q*bert over short timescales, though both models align over long timescales; in Montezuma’s Revenge, human penalties for death misalign with Atari score and label noise sensitivity also matters.Source: §3.2 · p. 8, Figure 3 · p. 8
• Frozen reward models are not significantly better than online training, and in Hero, Montezuma, and Private Eye agents exploit loopholes that raise predicted reward while diminishing true score.Online training can address such exploits interactively by labeling behaviors that appear in the policy but fail to improve game scores.Source: §3.2 · p. 8, Figure 4 · p. 9

Discussion

• With about 1000 preference comparisons, the combined method outperformed imitation learning in 7 out of 9 games and achieved superhuman performance on Pong and Enduro.Adding demonstrations to preferences typically produced substantial gains, especially in exploration-heavy games.Source: §4 · p. 9
• Although humans favored clips suggesting directional exploration, their feedback created reward pits that trapped agents in repetitive, fruitless behavior, while demonstrations supplied consistent exploration guidance (MacGlashan et al., 2017).Synthetic preference feedback proved more effective than feedback from humans.Source: §4 · p. 9
• No significant gains came from distributional or quantile RL, reward-policy weight sharing, action inputs, pretrained or semi-supervised reward models, loss phasing, or alternative demo annotation (Bellemare et al., 2017; Dabney et al., 2017).The alternative demo-annotation strategies are discussed in Appendix H.Source: §4 · p. 9
• Using DQN/DQfD rather than A3C showed reward learning was feasible across value-based and policy-gradient algorithms with comparable success (Christiano et al., 2017).Appendix F compares the scores of the two agents.Source: §4 · p. 9
• Reward hacking (Amodei et al., 2016) also affects hard-coded rewards (Lehman et al., 2018), but persisted only with frozen preferences; online human feedback was effective in preventing it.The persistent reward hacking observed occurred when preference feedback was frozen.Source: §4 · p. 9

A.1 Environment

• The Arcade Learning Environment uses the standard 18-action set and begins episodes with 0–90 no-ops.Source: §A.1 · p. 14
• Observations use max-pooling, action repeat, four-frame stacking, and grayscale resizing to 84×84.Source: §A.1 · p. 14
• Each four-frame stack counts as one observation and each repeated action as one actor step.Thus, one actor step corresponds to four Atari frames.Source: §A.1 · p. 14
• The score is replaced by a constant black background, and life-loss and episode-end signals are withheld, making play continuous.Source: §A.1 · p. 14
• Synthetic oracle feedback penalizes episode ends in every game except Pong, requiring the agent to learn that penalty.Source: §A.1 · p. 14

A.2 Expert demonstrations

• All experiments use the same per-game demonstrations, collected from an expert game tester and previously used by Hester et al. (2018).Source: §A.2 · p. 14
• The paper reports the demonstrations’ lengths and scores, but the supplied passage gives no values.Source: §A.2 · p. 14

Agent and reward model

• DQfD uses prioritized replay and a demonstration margin loss, recomputing reward-model values for each sampled batch; epsilon anneals from 0.1 to 0.01 over the first 105 actor steps.The setup also uses a 1e6 experience buffer, batch size 32, and learning updates every 4 steps.Source: §A.3 · p. 14
• The reward model maps stacked 84x84x4 frames through four convolutional layers and a 64-unit fully connected layer to a scalar; agent actions are excluded as they did not improve performance.The convolutional layers use 7x7, 5x5, 3x3, and 3x3 filters, each with 16 channels.Source: §A.3 · p. 14
• To limit overfitting, 1/e of the data is held out for validation and L2 regularization adapts to the gap between average validation and training loss.The regularization weight increases when validation loss is more than 50% higher than training loss and decreases when it is less than 10% higher.Source: §A.3 · p. 14
• The model adds a squared-reward loss, input Gaussian noise, batch normalization, and dropout, and assumes a 10% chance of random annotator responses when computing cross-entropy.It uses 0.1-amplitude input noise and replaces predicted preference probability P̂ with 0.9P̂ + 0.05 for cross-entropy.Source: §A.3 · p. 15
• Because comparison-trained reward scale is arbitrary, outputs are normalized to mean 0 and standard deviation 0.05 across the annotation buffer; no reward-model ensemble is used.The reward model is trained on batches of 16 segment pairs with Adam.Source: §A.3 · p. 15
• The annotation buffer grows through labels on 25-step clip pairs: the full schedule begins with 500 labels and reaches 6800 labels in a 50-million-step experiment.Schedules with 2, 4, and 6 times fewer labels total 3400, 1700, and 1133 labels, respectively.Source: §A.3 · p. 15

A.4 Training protocol

• Training ran for 500 iterations of 105 agent steps, keeping the reward model fixed within each iteration and retraining it afterward on 6,250 uniformly sampled batches.Each iteration collected trajectories and randomly sampled 25-step clips for annotation; synthetic labels were added after each iteration, while human labels arrived about every 6 hours (roughly 12 iterations).Source: §A.4 · p. 15
• Pretraining trained the agent from demonstrations, collected and labeled initial clip pairs, trained the reward model, then ran a final demonstration-only agent-training phase.The first agent phase used 20,000 batches, with noisy Q loss acting as a regularizer; reward-model training used 50,000 batches, and the final agent phase used 60,000. Some experiments added automatically labeled pairs favoring demonstration clips.Source: §A.4 · p. 15

B Performance as a function of human effort

• Figure 5 compares performance across games by joint labeling and demonstration effort, while Figure 6 summarizes best performance at each effort level with or without demonstrations.The tested setups include no demonstrations, demonstrations plus preferences, demonstrations plus preferences and initial automatic labels, and demonstrations with actual human preferences; preference schedules use the full 6,800-sample schedule or fractions of it.Source: B Performance as a function of human… · pp. 15–16, Figure 6 · p. 16, Figure 5 · p. 16
• With demonstrations, Hero, Montezuma’s Revenge, Private Eye, and Q*bert improve with added feedback; Enduro improves with added feedback only without demonstrations.The authors note that demonstrations work against feedback in Enduro because the agent easily beats the demonstrations.Source: Figure 5 · p. 16, Figure 6 · p. 16, B Performance as a function of human… · p. 17
• Beamrider peaks at low feedback and then regresses without demonstrations, but is feedback-driven with demonstrations; Pong is solved with little synthetic feedback in any setup.When preferences come from humans, Pong improves significantly with extra feedback.Source: B Performance as a function of human… · p. 17
• Breakout shows no clear feedback trend and is noisy, while Seaquest scores are limited because the agent does not learn to resurface.In Seaquest, removing episode boundaries lets extra shooting time offset the end-of-game penalty, and preferences poorly capture the reward-magnitude difference between collecting and surfacing divers versus shooting enemies.Source: B Performance as a function of human… · p. 17

Reward model training

• Enduro’s loss exceeded four times the entropy lower bound, yet its predicted reward correlated highly with game score and supported good play.The agent learned to play well even though the model failed to distinguish many clip pairs with small score differences.Source: C Reward model training · pp. 17–18, Figure 7 · p. 17
• Loss stayed close to the entropy bound in Beamrider, Breakout, Montezuma’s Revenge, and Private Eye, but was 50%-100% above it in Hero, Q*bert, and Seaquest.Figure 7 plots reward-model cross-entropy loss against average label entropy, the latter serving as a lower bound that rises with the share of indifferent labels.Source: C Reward model training · p. 17, Figure 7 · p. 17
• Enduro’s continuous scoring made similar clips differ by one point without being labeled indifferent, while the model learned salient events such as crashes instead of those small differences.Points accrue continuously in proportion to car speed, and crashes significantly reduce accumulated score.Source: C Reward model training · pp. 17–18

• Synthetic-preference reward models generally align well with true reward, especially over 1000-step intervals, while human-feedback models fail to align in Breakout, Montezuma’s Revenge, and Pong.Evaluation fixed the policy and reward model after successful full-schedule training; the synthetic result used the most successful seed, while only one human-feedback run was available.Source: Figure 8 · p. 18, D Reward model alignment · p. 18
• Sparse rewards yield fewer informative preference labels, and preferences provide no precise numeric feedback, making it hard to distinguish rewards of 10 versus 11 from 10 versus 100.Source: D Reward model alignment · p. 18
• Coarser alignment over short intervals makes reward models hackable in games where imprecisely timed rewards can be exploited.Source: D Reward model alignment · p. 18
• Human preferences can target goals unlike simulator rewards: in Montezuma’s Revenge, punishing deaths led to a passive policy, while prioritizing hits over points in Pong and Breakout could favor dropping the ball for an easy restart hit.Source: D Reward model alignment · p. 18

E Effects of label noise

• 5%–10% labeling mistakes have minor impact in most games but are significantly detrimental in Montezuma’s Revenge.Figure 9 uses synthetic feedback with the full label schedule and no automatic labels from demonstrations.Source: E Effects of label noise · p. 18, Figure 9 · p. 19
• Random labeling mistakes partially explain the relatively poor performance of human compared with synthetic annotators.Source: E Effects of label noise · p. 18
• Human-annotated experiments showed mistake rates between 5% and 10%.Source: E Effects of label noise · p. 18
• Figure 9 tests mislabeling rates using synthetic feedback, the full label schedule, and no automatic labels from demonstrations.This is the same setting as the human experiments.Source: Figure 9 · p. 19

F Comparison with policy gradient

• Christiano et al. (2017) evaluated preference feedback without demonstrations using the policy-gradient-based A3C agent.Source: F Comparison with policy gradient · p. 19
• The paper compares its no-demonstration scores from value-based DQN/DQfD with corresponding scores from Christiano et al. (2017).Source: F Comparison with policy gradient · p. 19

G Comparison with DQfD trained from true reward

• The comparison reports average scores for true-reward DQfD and learned-reward DQfD using full-schedule runs with autolabels.The learned-reward scores are the magenta bars in Figure 1.Source: G Comparison with DQfD trained from… · p. 19
• Although our method uses preference feedback on at most 340.000 agent steps (< 1% of experience), restricting true-reward DQfD to 1% of training steps would perform very poorly.The authors note that the comparison is not like-for-like because a synthetic oracle makes the true reward indirectly available to the agent.Source: G Comparison with DQfD trained from… · pp. 19–20

H Unsuccessful ideas

• Although quantile distributional RL was proposed to alleviate reward-scale sensitivity and improve performance (Dabney et al., 2017), neither distributional RL (Bellemare et al., 2017) nor quantile distributional RL stabilized training.Source: H Unsuccessful ideas · p. 20
• Weight sharing between policy and reward model, and copying policy weights to the reward model, destabilized training despite the prospect of shared representations helping performance (Jaderberg et al., 2017).Source: H Unsuccessful ideas · p. 20
• Pretrained CIFAR features slightly improved sample efficiency in a few games, while Tarvainen and Valpola (2017) brought little sample-complexity improvement and worsened stability; the fit of semi-supervision remains unclear.The authors could not determine whether the technique struggled with Atari visuals or reward learning’s problem structure.Source: H Unsuccessful ideas · p. 20
• Phasing out demonstration loss boosted Enduro performance but reduced performance in Montezuma’s Revenge and Private Eye, where demonstrations are critical.Source: H Unsuccessful ideas · p. 20
• Demo-demo preference pairs did not change performance, while small score-blanking misalignment between demonstrations and policy frames reduced demo-heavy game scores to zero, and added noise or regularization hurt performance.Source: H Unsuccessful ideas · p. 20