Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations

Introduction

• Designing rewards for desired behavior is difficult, and high-quality demonstrations can be hard to provide, motivating IRL to infer intrinsic rewards from demonstrations (Amodei et al., 2016).Source: §1 · p. 1
• Existing IRL methods typically make demonstrations appear near-optimal, while direct imitation also inherits suboptimal behavior; thus, both can fail to exceed a suboptimal demonstrator (Finn et al., 2016; Henderson et al., 2018; Torabi et al., 2018a).Source: §1 · p. 1
• T-REX learns a state-based reward from ranked demonstrations, assigning higher total return to higher-ranked trajectories so reinforcement learning can exceed even highly suboptimal demonstrations.Unlike standard IRL, which seeks a reward that justifies demonstrations, T-REX explains their ranking to extrapolate the user’s underlying intent.Source: §1 · pp. 1–2
• Pairwise ranking constraints help T-REX extrapolate returns to trajectories better than any observed demonstration and reduce overfitting to the limited state space visited by demonstrators.Ranked demonstrations provide diverse data from multiple state-space areas, helping the neural network learn what to do and what not to do.Source: §1 · p. 2
• T-REX often achieves more than twice the performance of the best-performing demonstration and outperforms state-of-the-art imitation learning algorithms on Atari and MuJoCo benchmarks.It also performs well with significant ranking noise and can learn by observing a novice improve noisily over time.Source: §1 · p. 2

2.4. Reward learning for video games

• The authors present their work as the first to significantly outperform demonstrators in high-dimensional tasks such as Atari without ground-truth rewards, demonstrator actions, or active preference queries.Given trajectory rankings, reward learning becomes binary classification and requires no environmental interactions.Source: §2 · p. 2, §2.4 · pp. 3–4

2.1. Learning from demonstrations

• Behavioral cloning and action-distribution imitation reproduce demonstrator actions, while IRL infers rewards for generalizing beyond demonstrated states; maximum-entropy IRL still ties performance to the demonstrator.Maximum-entropy methods seek rewards that make demonstrations near-optimal, with policy entropy helping disambiguate inference; the proposed approach instead learns performance-correlated features from rankings and solves one RL problem.Source: §2.1 · p. 2
• Preference-based policy learning and IRL use rankings, but prior methods use linear hand-crafted features and do not study extrapolation; scored-trajectory methods also had not been applied to high-dimensional deep IRL.The passage cites a survey of preference-based reinforcement learning by Wirth et al. (2017).Source: §2.1 · pp. 2–3

2.2. Learning from observation

• Learning from observation includes behavioral cloning from videos, multi-view video methods, and meta-learning approaches, while adversarial imitation removes action labels but has struggled to scale to Atari.The cited single-demonstration meta-learning approaches require training on a wide variety of similar tasks.Source: §2.2 · p. 3

2.3. Learning from suboptimal demonstrations

• Methods for highly suboptimal demonstrations require failure labels, known linear features, or a majority of expert examples to identify anomalous demonstrations.Standard reinforcement-learning initialization from suboptimal demonstrations uses a ground-truth reward, yet can still struggle to significantly outperform the demonstrator.Source: §2.3 · p. 3

2.4. Reward learning for video games

• Active trajectory-preference learning for Atari requires thousands of human labels, while Ibarz et al. (2018) also need demonstrator actions and can underperform the demonstrator despite active queries.Aytar et al. (2018) learn observation-based checkpoint rewards but rely on high-performance demonstrations and aim to imitate rather than extrapolate beyond them.Source: §2.4 · p. 3

3. Problem Definition

• The environment is modeled as an MDP with states, actions, transition probabilities, a reward function, and a discount factor.A policy maps states to probability distributions over actions.Source: §3 · p. 4
• Given ranked trajectories, the goal is to estimate the reward the demonstrator is trying to optimize and use it to learn a policy that outperforms the demonstrations.The reward function and demonstrator actions are not observed.Source: §3 · p. 4
• Only a qualitative ranking is required, based on an internal goal or intrinsic reward; rankings may come from pairwise preferences or ratings.The method must infer why some trajectories are better, even if relative scores are available.Source: §3 · p. 4

4. Method

• T-REX first infers a reward from demonstrations ranked worst to best, then uses reinforcement learning on that reward to seek a policy better than the demonstrator.The method has two stages: reward inference and policy optimization.Source: §4 · p. 4
• A neural reward is trained so predicted returns favor higher-ranked trajectories, using a softmax preference model and cross-entropy loss.The loss classifies which trajectory is preferable based on its predicted return.Source: §4 · p. 4
• T-REX samples demonstration pairs and length-L partial trajectories, using summed predicted rewards as logits and ranking-derived labels for training.These partial-pair examples provide many weak, noisy preferences, enabling expressive reward learning from a small number of ranked demonstrations.Source: §4 · p. 4

5.1.3. RESULTS

• T-REX significantly surpassed the suboptimal trajectories in every HalfCheetah and Hopper stage and beat BCO and GAIL across tasks and stages except Hopper Stage 2 and Ant Stage 2.Policies were trained with PPO to maximize the learned reward; the locomotion evaluations used forward distance traveled.Source: §5.1.3 · p. 5, Figure 2 · p. 5, §5.1.2 · p. 5
• In MuJoCo, strong correlation between T-REX’s learned reward and ground-truth returns enabled performance beyond demonstrations, but weaker correlation in Ant led to worse-than-demonstrator performance in Stage 2.The extrapolation analysis included trajectories with performance much better than the best demonstration provided to T-REX.Source: §5.1.3 · pp. 5–6, Figure 3 · p. 6

5.2.3. RESULTS

• T-REX outperformed BCO and GAIL on 7 of 8 Atari games and surpassed the best demonstration on 7 of 8, more than doubling its score on Beam Rider, Breakout, Enduro, and Q*bert.BCO performed below the average demonstration on every game, while GAIL exceeded it only on Space Invaders.Source: §5.2.3 · pp. 6–7, Table 1 · p. 7
• Atari reward extrapolation was accurate on Beam Rider, Breakout, Enduro, Seaquest, and Space Invaders, while learned rewards correlated less well on Pong, Q*bert, and Hero.T-REX overfit on Pong and scored no points on Hero; on Q*bert, PPO learned to score by inducing Coily to jump off an edge, a behavior absent from demonstrations.Source: §5.2.3 · p. 7, Figure 4 · p. 8

5.2.4. HUMAN DEMONSTRATIONS

• With novice human Atari demonstrations, T-REX significantly outperformed the best human demonstration on Q*bert, Space Invaders, and Video Pinball, but not on Montezuma’s Revenge or Ms Pacman.The demonstrations came from the Atari Grand Challenge Dataset and had ground-truth rankings.Source: §5.2.4 · p. 7

5.3. Robustness to Noisy Rankings

• On Stage 1 Hopper, T-REX was relatively robust to synthetic ranking noise of up to around 15% pairwise errors.Noise was produced by randomly swapping adjacent trajectories in ground-truth-ranked lists; results averaged nine runs per noise level.Source: §5.3 · p. 7, Figure 5 · p. 8
• Human ranking labels had 63%–88% pairwise accuracy against ground truth, yet T-REX still outperformed the demonstrator on 5 of 8 Atari games.Workers ranked demonstration videos in pairs, with six labels collected per pair and ambiguous or “Not Sure” majority labels removed.Source: §5.3 · pp. 7–8

5.3.1. LEARNING FROM TIME-BASED RANKINGS

• Using PPO training order as rankings, T-REX inferred meaningful rewards and achieved results comparable to ground-truth rankings on most stages, outperforming BCO and GAIL except on Ant Stage 2.This approach ranked trajectories by timestamp, requiring no demonstrator labels.Source: §5.3.1 · p. 8, Figure 5 · p. 8

• Combined with deep reinforcement learning, T-REX achieves better-than-demonstrator performance and outperforms state-of-the-art behavioral cloning and IRL methods.Source: §6 · p. 8
• T-REX learns to extrapolate intent from suboptimal ranked demonstrations and, to the authors’ knowledge, is the first IRL algorithm to significantly outperform demonstrators without external knowledge while scaling to high-dimensional Atari games.Source: §6 · p. 8
• T-REX is robust to modest ranking noise and can learn from automatically generated labels by watching a learner noisily improve over time.Source: §6 · p. 8

B.1. Policy performance

• The T-REX (time-ordered) row reports performance using demonstrations ranked by timestamps from a learning agent rather than explicit preference rankings.Table 1 presents the full MuJoCo experiment results, but the supplied passage does not provide specific values or comparisons.Source: §B.1 · p. 11

C. Behavioral Cloning from Observation

• BCO’s inverse transition models use 20,000 random-policy transitions and a DQN classifier trained with Adam, learning rate 0.0001, and L2 regularization 0.0001.For P(a|st, st+1), the two 4x84x84 frames are concatenated into an 8x84x84 input; the same architecture predicts actions from states.Source: C. Behavioral Cloning from Observation · p. 11
• Using only the best X% of demonstrations did not improve BCO over using all demonstrations (X = 100), likely because few demonstrations left too little training data.Both T-REX and BCO received the full demonstration set.Source: C. Behavioral Cloning from Observation · p. 11

D. Atari reward learning details

• Atari experiments used OpenAI Baselines PPO with default hyperparameters, an NVIDIA TITAN V GPU, and nine parallel workers.Source: D. Atari reward learning details · p. 11
• Reward learning masked score and lives in every game, plus game-specific HUD information in Beam Rider, Enduro, Seaquest, and Hero.The additional masks covered sector and enemy-ship counts, the dashboard’s lower half, divers and oxygen, and power and inventory, respectively.Source: D. Atari reward learning details · p. 11
• For Enduro, reward-network training used 2,000 randomly downsampled demonstrations and 10,000 Adam steps at learning rate 5e-5.Pairs of trajectories were randomly selected, cropped by 0–5 initial frames, and downsampled by retaining every third to sixth frame.Source: D. Atari reward learning details · p. 11
• Table 1 reports distance-based results for three robotic locomotion tasks, with means and standard deviations from 25 trials and comparisons of ground-truth-ranked versus time-ordered T-REX demonstrations.The caption distinguishes the best performance from suboptimal demonstrations from PPO’s best achievable performance under the ground-truth reward.Source: Table 1 · p. 12

B.2. Policy visualization

• Across Stages 1–3, T-REX turns suboptimal HalfCheetah demonstrations into faster crawling, partially stabilized walking, and finally near-optimal gaits.The Stage 1 policy resembles the crawling demonstration; Stage 2 resembles and partially stabilizes the best demonstration’s gait, while Stage 3 yields a near-optimal gait.Source: §B.2 · p. 11, Figure 1 · p. 12

E. Comparison to active reward learning

• T-REX exceeds the best demonstration on 7 of 8 games tested, versus DQfD+A on 3 of 9, using 12 ranked demonstrations without true rewards or thousands of active queries.DQfD+A extends DQfD, which surpasses the best demonstration on 14 of 41 Atari games.Source: E. Comparison to active reward learning · p. 13
• Despite demonstrations averaging 38 times better for DQfD+A, T-REX surpasses it on Q*Bert and Seaquest.T-REX uses 12 ranked demonstrations, compared with 4–7 demonstrations for DQfD+A.Source: E. Comparison to active reward learning · p. 13
• T-REX’s 12 ranked demonstrations require 66 pairwise comparisons, versus 3,400 preference labels for DQfD+A during training with ground-truth rewards.The 66 comparisons correspond to n · (n − 1)/2 comparisons among the demonstrations.Source: E. Comparison to active reward learning · p. 13

F.1. Human demonstrations

• The Atari Grand Challenge training set contained 12 unique human demonstrations selected by ground-truth return after duplicate scores were removed, and T-REX used the previously described hyperparameters.Demonstrations were collected for five Atari games.Source: §F.1 · p. 14
• T-REX outperformed the best human demonstration on Q*bert, Space Invaders, and Video Pinball, but not on Montezuma’s Revenge or Ms Pacman.Performance is reported in Table 3.Source: §F.1 · p. 14
• T-REX struggled on Montezuma’s Revenge and Ms Pacman, games requiring maze navigation and balancing objectives such as collecting objects and avoiding enemies.The paper notes similar difficulty on Hero and identifies extending T-REX to these settings as future work.Source: §F.1 · p. 14

F.2. Human rankings

• Workers compared side-by-side videos from 12 PPO demonstrations, judging which performed better or whether they were unsure.The study collected six labels for each of 132 possible sequences of two videos; majority votes formed human labels, while pairs without a majority or with a “Not Sure” majority were excluded from T-REX training.Source: §F.2 · p. 14
• T-REX optimized good policies for many games despite noisy human labels, but Enduro labels were too noisy for successful policy learning.PPO used three seeds, and the best final policy was averaged over 30 trials.Source: §F.2 · p. 14

G. Atari Reward Visualizations

• Masked 3×3 regions across four-frame stacks reveal pixel influence on predicted reward, producing attention heatmaps for minimum- and maximum-reward observations.The method compares predicted reward with and without each region masked, then sums absolute changes per pixel; the visualizations suggest the networks learn reward-relevant features.Source: G. Atari Reward Visualizations · p. 14
• Beam Rider, Breakout, and Enduro maps attend to gameplay cues including enemies and shots, remaining bricks and ball, and the controlled car and road.The maximum- and minimum-reward observations depict successful play or progress versus threats or falling behind.Source: Figure 2 · p. 16, Figure 3 · p. 17, Figure 4 · p. 18
• Hero and Pong maps attend to characters, maze or explosion features, and the ball and paddles, while Q*bert’s stair-focused maps are difficult to interpret.Hero’s maximum-reward observation is also difficult to interpret; its minimum shows a bomb killing the character rather than the wall.Source: Figure 5 · p. 19, Figure 6 · p. 20, Figure 7 · p. 21
• Seaquest maps focus on nearby enemies and the submarine, while Space Invaders maps attend to aliens and barriers in low-reward play and barriers in maximum-reward play.The maximum-reward Space Invaders observation shows all aliens destroyed despite no demonstration showing that outcome; the map does not attend to the controlled ship.Source: Figure 8 · p. 22, Figure 9 · p. 23