PaperlayerView exact revision on arXiv

Source-linked AI summary

Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations

Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan, Scott Niekum

arXiv:1904.06387v5cs.LGstat.ML

TL;DR

Existing IRL methods generally seek rewards that make demonstrations appear near-optimal, limiting their ability to exceed suboptimal demonstrators. T-REX learns rewards from approximately ranked observations and uses reinforcement learning to extrapolate beyond those demonstrations. It outperforms state-of-the-art imitation-learning and IRL methods across Atari and MuJoCo benchmarks, often achieving more than twice the best-demonstration performance.

  • Problem

    Existing IRL methods typically seek rewards that make demonstrations appear near-optimal, while high-quality demonstrations can be difficult to provide.

  • Method

    T-REX learns a state-based reward function from ranked demonstrations by making higher-ranked trajectories receive greater total return, then optimizes that reward with reinforcement learning.

  • Results

    T-REX outperforms state-of-the-art imitation-learning and IRL methods on Atari and MuJoCo tasks and often achieves more than twice the performance of the best demonstration.

  • Takeaways & Limitations

    Ranked, potentially poor demonstrations can support reward learning that extrapolates beyond observed performance, including when rankings are noisy or generated by a learner improving over time.

  • Takeaways & Limitations

    Performance can be worse than the demonstrator in Ant Stage 2 when the ranking correlation is not as strong.

Abstract

from arXiv · show

A critical flaw of existing inverse reinforcement learning (IRL) methods is their inability to significantly outperform the demonstrator. This is because IRL typically seeks a reward function that makes the demonstrator appear near-optimal, rather than inferring the underlying intentions of the demonstrator that may have been poorly executed in practice. In this paper, we introduce a novel reward-learning-from-observation algorithm, Trajectory-ranked Reward EXtrapolation (T-REX), that extrapolates beyond a set of (approximately) ranked demonstrations in order to infer high-quality reward functions from a set of potentially poor demonstrations. When combined with deep reinforcement learning, T-REX outperforms state-of-the-art imitation learning and IRL methods on multiple Atari and MuJoCo benchmark tasks and achieves performance that is often more than twice the performance of the best demonstration. We also demonstrate that T-REX is robust to ranking noise and can accurately extrapolate intention by simply watching a learner noisily improve at a task over time.

1. Introduction

T-REX addresses the failure of IRL to surpass suboptimal demonstrators by learning rewards from ranked demonstrations and extrapolating beyond observed performance. Across Atari and MuJoCo tasks, it outperforms strong imitation-learning baselines and often exceeds the best demonstration by more than twofold.

  • Motivation: High-quality demonstrations are difficult to provide, even for experts performing complex or high-frequency tasks.Examples include household chores, stock trading, and complex video games.
  • Motivation: Existing IRL methods typically learn rewards that make demonstrations appear near-optimal, producing suboptimal behavior when the demonstrator is suboptimal.This differs from inferring the demonstrator’s underlying intentions.
  • T-REX: T-REX learns a state-based reward function from ranked demonstrations by assigning greater total return to higher-ranked trajectories.The learned reward explains the ranking rather than merely justifying the demonstrations.
  • T-REX: Pairwise ranking constraints can regularize reward learning, helping T-REX identify ranking-correlated features and extrapolate beyond the visited state space.The paper notes that unequal rankings eliminate degenerate rewards such as an all-zero reward function.
  • Results: T-REX often achieves more than twice the performance of the best demonstration and outperforms state-of-the-art imitation-learning algorithms on Atari and MuJoCo tasks.It is also reported to tolerate significant ranking noise and learn from a novice who improves noisily over time.

2. Related Work

Prior work largely imitates actions, requires action labels, hand-crafted features, expert demonstrations, ground-truth rewards, or active supervision. T-REX instead learns from ranked observations and is presented as a scalable route to better-than-demonstrator policies in high-dimensional tasks.

  • Learning from demonstrations: The paper targets improvement over suboptimal demonstrators in high-dimensional reinforcement learning without hand-specified rewards or policy-learning supervision.It learns from demonstrations rather than requiring a reward function designed by a human.
  • T-REX: T-REX learns from ranked observations rather than actions and does not require ground-truth rewards or active preference queries.Given rankings, reward learning becomes a binary classification problem that does not require access to an MDP.
  • Behavioral cloning: Behavioral cloning imitates demonstrator actions but can suffer large generalization error without substantial human feedback and correction.Recent deep imitation methods model the demonstrator’s action distribution.
  • Inverse reinforcement learning: IRL infers rewards intended to generalize beyond demonstrated states, but prior approaches commonly rely on action labels or fail to scale to high-dimensional Atari tasks.The paper contrasts this with learning from observations without demonstrator actions.
  • Suboptimal demonstrations: Earlier approaches to surpass demonstrators required prior reward-feature signs, hand-crafted linear features, labeled failures, or a majority of expert demonstrations.These requirements constrain learning from highly suboptimal demonstrations.
  • T-REX: Unlike active preference-learning methods requiring thousands of labels during policy learning, T-REX uses an initial set of approximately ranked demonstrations.The paper presents this as enabling better-than-demonstrator policies without ongoing supervision.

3. Problem Definition

The problem is inverse reinforcement learning from observation without access to rewards or demonstrator actions. Given qualitatively ranked trajectories, the goal is to learn a parameterized reward and optimize a policy that can outperform the demonstrations.

  • Problem formulation: The environment is modeled as an MDP with states, actions, transition probabilities, a reward function, and a discount factor.A policy maps states to probability distributions over actions.
  • Problem formulation: The task assumes no access to the MDP reward function or the actions taken by the demonstrator.The learner instead observes ranked trajectories.
  • Problem formulation: The input is a qualitative ranking over demonstrations, which may come from pairwise preferences or ratings on a scale.The method must infer why some trajectories are better than others, even when relative scores provide the ranking.

4. Method

T-REX infers a state-based reward function from ranked demonstrations, then optimizes a policy against that learned reward. It expands supervision through partial trajectory pairs and trains preference predictions from predicted returns.

  • Pipeline: T-REX has two stages: reward inference from ranked demonstrations followed by policy optimization.The demonstrations are ordered from worst to best.
  • Reward inference: A neural network reward function assigns higher predicted total return to higher-ranked trajectories.The return is discounted and parameterized by the learned reward function.
  • Reward inference: The training objective uses a softmax probability and cross-entropy loss to classify which trajectory is preferable from predicted returns.This preference formulation follows Bradley-Terry and Luce-Shephard models.
  • Reward inference: T-REX augments the data with partial trajectory pairs, producing weakly supervised preference labels from many pairwise comparisons.This augmentation helps train expressive neural-network rewards from few ranked demonstrations.
  • Policy optimization: Reinforcement learning optimizes a policy using the learned reward function to seek better-than-demonstrator performance.

5. Experiments and Results

T-REX was evaluated on suboptimal MuJoCo and Atari demonstrations, learning rewards from rankings and optimizing them with PPO. It generally exceeded demonstrations and strong baselines, while performance depended on extrapolation quality and tolerated ranking noise.

  • MuJoCo experiments: T-REX used PPO-generated, increasingly better demonstrations for HalfCheetah, Hopper, and Ant, ranked by ground-truth returns.Each checkpoint supplied a 1,000-step trajectory; three suboptimality stages were evaluated.
  • MuJoCo experiments: The reward network learned from 5,000 pairs of partial trajectories labeled by rankings, then PPO optimized the inferred reward using an ensemble.The ensemble averaged five separately trained reward networks after normalization.
  • MuJoCo results: T-REX significantly outperformed suboptimal trajectories across all HalfCheetah and Hopper stages and exceeded BCO and GAIL on all tasks and stages except Hopper and Ant Stage 2.BCO and GAIL usually failed to surpass average demonstrations because they explicitly imitated demonstrator behavior.
  • Atari results: T-REX outperformed BCO and GAIL in 7 of 8 Atari games and exceeded the best demonstration in 7 of 8 games.It achieved more than double the best-demonstration score on Beam Rider, Breakout, Enduro, and Q*bert.
  • Reward extrapolation: Reward extrapolation was accurate for five Atari games, whereas weaker correlation caused poor performance on Pong and Hero but did not prevent Q*bert improvement.On Q*bert, PPO learned a repeatable scoring behavior absent from the demonstrations.
  • Robustness to noisy rankings: T-REX remained relatively robust to around 15% pairwise ranking errors and outperformed human demonstrators on 5 of 8 Atari games despite ranking accuracy between 63% and 88%.It also inferred meaningful rewards from noisy time-based rankings of a learning agent.

6. Conclusion

T-REX learns to extrapolate intent from suboptimal ranked demonstrations and achieves better-than-demonstrator performance while remaining robust to ranking noise and time-ordered labels.

  • T-REX extrapolates intent from suboptimal ranked demonstrations in high-dimensional tasks.
  • T-REX significantly outperforms demonstrators without external knowledge and scales to high-dimensional Atari games.
  • T-REX outperforms state-of-the-art behavioral cloning and IRL methods when combined with deep reinforcement learning.
  • T-REX is robust to modest ranking noise and can learn from labels generated by observing a learner improve noisily over time.

A. Code and Videos

The paper provides code and supplemental videos through its GitHub repository.

  • Code and supplemental videos are available at the ICML2019-TREX GitHub repository.

B. T-REX Results on the MuJoCo Domain

MuJoCo experiments compare T-REX with suboptimal demonstrations and PPO ground-truth performance, including time-ordered demonstrations from an improving learner. HalfCheetah visualizations show T-REX improving upon demonstrated gaits while preserving their style.

  • B. T-REX Results on the MuJoCo Domain: In HalfCheetah, T-REX preserves the crawling style of highly suboptimal demonstrations while optimizing a faster, better-than-demonstrator gait.
  • B. T-REX Results on the MuJoCo Domain: Table 1 measures robotic locomotion performance by total distance traveled, using the robot body's final x-position.
  • B. T-REX Results on the MuJoCo Domain: The table reports best suboptimal-demonstration performance above best achievable PPO performance under the ground-truth reward.
  • B. T-REX Results on the MuJoCo Domain: T-REX includes results from demonstrations ranked by ground-truth returns and from observing a learner, with timestamps determining ranking in the latter setting.

E. Comparison to active reward learning

T-REX exceeds demonstrator performance using few ranked demonstrations and no true rewards or thousands of active oracle queries, despite weaker demonstrations than DQfD+A.

  • T-REX achieved better-than-demonstrator performance on 7 of 8 games, compared with DQfD+A's 3 of 9 games.
  • T-REX used 12 ranked demonstrations and 66 comparisons without access to true rewards or thousands of active queries.
  • DQfD+A used demonstrations averaging 38 times better than T-REX's, yet T-REX surpassed DQfD+A on Q*Bert and Seaquest.

F.1. Human demonstrations

T-REX was evaluated using 12 ranked human demonstrations from five Atari games. It exceeded the best human demonstration on three games but failed to learn good control policies on two maze-navigation tasks.

  • F.1. Human demonstrations: 12 human demonstrations were selected after removing duplicate scores and sorting the remaining demonstrations by ground-truth return.The demonstrations came from the Atari Grand Challenge dataset and covered five Atari games.
  • F.1. Human demonstrations: T-REX outperformed the best human demonstration on Q*bert, Space Invaders, and Video Pinball.The result is reported for the real novice human demonstrations evaluated in Table 3.
  • F.1. Human demonstrations: T-REX did not learn a good control policy for Montezuma’s Revenge or Ms Pacman.These games require maze navigation and balancing objectives such as collecting objects and avoiding enemies.

F.2. Human rankings

Human rankings were collected by comparing pairs of demonstrations, aggregating six worker labels per pair by majority vote, and using the resulting preferences to train T-REX. T-REX learned good policies for many games despite noisy labels, although Enduro failed under excessive ranking noise.

  • F.2. Human rankings: Workers compared two demonstrations at a time and labeled which had better performance or whether they were unsure.The study used Amazon Mechanical Turk to obtain human rankings.
  • F.2. Human rankings: Six labels were collected for each of the 132 possible demonstration pairs, with majority vote used to determine the human label.The authors describe these labels as a worst-case lower bound on accuracy because workers did not generate the demonstrations.
  • F.2. Human rankings: T-REX optimized good policies for many games despite noisy human labels.Performance was measured using the best final PPO policy averaged over 30 trials, with three random seeds.
  • F.2. Human rankings: Enduro was a failure case where ranking labels were too noisy for successful policy learning.This identifies a practical boundary for learning from human rankings.

G. Atari Reward Visualizations

The Atari visualizations inspect which image regions influence T-REX’s predicted rewards by comparing maximum- and minimum-reward observations with attention maps. Across games, the learned reward often focuses on controllable objects, threats, progress indicators, or task-relevant structures.

  • G. Atari Reward Visualizations: Attention maps estimate regional reward influence by masking 3x3 regions across four stacked frames and measuring the resulting change in predicted reward.The mask uses each game’s default background color and moves with stride 1.
  • G. Atari Reward Visualizations: Beam Rider’s maximum-reward observation shows an enemy ship destroyed, while its minimum-reward observation shows an enemy shot destroying the player’s ship.Attention emphasizes enemy ships and the fired shot in the high-reward case.
  • G. Atari Reward Visualizations: Breakout’s reward weight concentrates on remaining bricks, with additional attention on the ball and paddle; the minimum-reward case has no destroyed bricks.The maximum-reward observation shows many bricks destroyed and the ball moving toward another brick.
  • G. Atari Reward Visualizations: Enduro’s attention covers the controlled car and road sides, while the minimum-reward observation shows the car falling behind other racers.Some attention also appears on the passed car and odometer.
  • G. Atari Reward Visualizations: Hero and Seaquest visualizations focus on controllable characters or submarines together with nearby walls, explosions, or enemies.Hero’s minimum-reward case involves a bomb killing the main character, while Seaquest’s involves an enemy about to hit the submarine.
  • G. Atari Reward Visualizations: Pong attention mainly targets the ball and paddles, while Q*bert attention focuses on the stairs despite the minimum-reward map being difficult to interpret.Q*bert’s maximum-reward observation comes from the second level, where stairs change from yellow to blue.
  • G. Atari Reward Visualizations: Space Invaders’ maximum-reward observation has all aliens destroyed with barriers intact, despite no demonstration achieving that state.The attention map focuses on the barriers rather than the controlled character’s location.
Loading 1904.06387v5…