PaperlayerView exact revision on arXiv

Source-linked AI summary

Third-Person Imitation Learning

Bradly C. Stadie, Pieter Abbeel, Ilya Sutskever

arXiv:1703.01703v2cs.LG

TL;DR

The paper tackles imitation learning when demonstrations are available only from a different viewpoint and reward functions are not manually specified. It combines domain confusion with adversarial imitation learning to learn domain-agnostic features, and reports successful results on pointmass, reacher, and inverted-pendulum tasks. The paper concludes that the method solves simple third-person imitation tasks, while evaluation is limited by reward-scale differences and the simple task settings considered.

  • Problem

    Imitation learning methods generally require first-person demonstrations, although third-person demonstrations are often easier to obtain and may not provide matching observations or actions.

  • Method

    The method combines domain confusion with Generative Adversarial Imitation Learning to learn domain-agnostic representations and a cost function from third-person demonstrations.

  • Results

    The algorithm successfully solves simple third-person imitation tasks in pointmass, reacher, and inverted-pendulum environments, with domain confusion essential for strong performance across all three experiments.

  • Takeaways & Limitations

    Third-person imitation learning can be performed without direct teacher–student state correspondence, and explicitly considering the third-person setting matters in these environments.

  • Takeaways & Limitations

    Evaluation is limited to simple tasks, and differing reward-function scales make variance difficult to compare across learning curves.

Abstract

from arXiv · show

Reinforcement learning (RL) makes it possible to train agents capable of achieving sophisticated goals in complex and uncertain environments. A key difficulty in reinforcement learning is specifying a reward function for the agent to optimize. Traditionally, imitation learning in RL has been used to overcome this problem. Unfortunately, hitherto imitation learning methods tend to require that demonstrations are supplied in the first-person: the agent is provided with a sequence of states and a specification of the actions that it should have taken. While powerful, this kind of imitation learning is limited by the relatively hard problem of collecting first-person demonstrations. Humans address this problem by learning from third-person demonstrations: they observe other humans perform tasks, infer the task, and accomplish the same task themselves. In this paper, we present a method for unsupervised third-person imitation learning. Here third-person refers to training an agent to correctly achieve a simple goal in a simple environment when it is provided a demonstration of a teacher achieving the same goal but from a different viewpoint; and unsupervised refers to the fact that the agent receives only these third-person demonstrations, and is not provided a correspondence between teacher states and student states. Our methods primary insight is that recent advances from domain confusion can be utilized to yield domain agnostic features which are crucial during the training process. To validate our approach, we report successful experiments on learning from third-person demonstrations in a pointmass domain, a reacher domain, and inverted pendulum.

1 INTRODUCTION

The paper addresses the difficulty of manually specifying rewards and the impracticality of collecting first-person demonstrations by developing unsupervised third-person imitation learning. Its approach uses domain-agnostic representations and succeeds across pointmass, reacher, and pendulum tasks.

  • Motivation: RL requires manually specified reward functions whose maximizers recover desired behavior.Inverse reinforcement learning addresses this weakness by inferring a reward function from expert trajectories.
  • Motivation: First-person imitation learning is difficult to collect, whereas third-person demonstrations are often easier and sometimes the only feasible source.Humans commonly learn by observing others perform tasks rather than receiving action-labeled trajectories from the agent’s viewpoint.
  • Contribution: The paper develops an unsupervised algorithm that infers correspondence between a teacher’s demonstration and the student’s environment without state correspondence.The goal is to let an agent observe another agent perform a task and accomplish the same task itself.
  • Approach: The method combines domain confusion and GAN-based imitation learning to recover domain-agnostic observation features and a cost function over those features.The formulation extends first-person RL-GAN to third-person imitation learning.
  • Results: The approach solves the paper’s simple third-person imitation tasks despite complicated pixel-level relationships between teacher demonstrations and student observations.The evaluated domains are pointmass, reacher, and pendulum.

2 RELATED WORK

Related imitation-learning methods include behavioral cloning, inverse reinforcement learning, GAN-based approaches, domain adaptation, and deep reinforcement learning. Their direct applicability is limited because third-person demonstrations differ in observations and actions from the imitator’s setting.

  • Imitation learning: Behavioral cloning learns an observation-to-action mapping from demonstrations using supervised learning.The approach may interleave learning with additional data collection.
  • Third-person setting: Third-person imitation makes existing behavioral cloning and inverse reinforcement learning difficult to apply because demonstrator and imitator observations and actions differ.Direct application would require a mapping between demonstrator-space and imitator-space observations and actions.
  • Third-person setting: Raw-sensory inverse reinforcement learning approaches are limited to first-person settings because matching raw sensory observations is impossible in the third-person setting.This limitation motivates representations that are not tied to a single viewpoint or domain.
  • Related techniques: The paper builds on GANs and applies gradient flipping from domain-confusion work within its optimization formulation.These connections link imitation learning with adversarial learning and domain adaptation.
  • Reinforcement learning: The proposed approach uses reinforcement learning from raw sensory data in the imitator domain and employs Trust Region Policy Optimization.Recent deep reinforcement learning advances make this setting practical.

3 BACKGROUND AND PRELIMINARIES

The background formalizes reinforcement learning and first-person imitation learning, then describes adversarial policy matching through a discriminator trained on expert and imitator state distributions.

  • Reinforcement learning: A finite-horizon discounted MDP is represented by state, action, transition, reward, initial-state, discount, and horizon components.The tuple is M = (S, A, P, r, ρ0, γ, T).
  • Reinforcement learning: RL seeks a parameterized policy that maximizes the expected discounted sum of rewards.The policy is written as πθ and its objective as η(πθ).
  • First-person imitation learning: First-person imitation learning omits the reward function and instead provides expert state traces generated by an unknown policy.Inverse reinforcement learning seeks a policy that performs as well as the expert under the unknown reward.
  • Adversarial imitation learning: The discriminator distinguishes states or state-action pairs from imitator and expert policies, while the imitator trains to make that distinction difficult.The desired equilibrium has the discriminator assign probability 0.5 in all cases.
  • Adversarial imitation learning: The paper abbreviates the discriminator’s minimization using cross-entropy loss with state inputs and expert-versus-non-expert class labels.CE denotes the standard cross entropy loss.

4 A FORMAL DEFINITION OF THE THIRD-PERSON IMITATION LEARNING PROBLEM

Third-person imitation learning is formulated across two MDPs: expert demonstrations come from one domain, while the learner must recover a policy that is optimal for its own domain’s reward.

  • Formal problem: The problem assumes expert and novice policies operate in two Markov Decision Processes, MπE and Mπθ.The domains need not provide a direct correspondence between their observed trajectories.
  • Formal problem: Given traces generated by an expert policy optimal under an unknown reward RπE, the learner recovers a policy πθ from those traces.The recovered policy should act optimally with respect to the novice-domain reward Rπθ.

5 A THIRD-PERSON IMITATION LEARNING ALGORITHM

The algorithm learns domain-agnostic features that support expert-versus-novice discrimination across different environments, then uses the discrimination signal as a reward for training the novice policy. It alternates optimization of the policy, discriminator, domain classifier, and feature extractor.

  • Architecture: The method extracts features from observations before classifying expert versus non-expert trajectories and their rollout domains.The feature extractor feeds both the trajectory discriminator and the domain classifier.
  • Domain confusion: The feature extractor is trained to retain information useful for classifying trajectories while removing information about whether rollouts come from expert or novice domains.This is formulated as minimizing classification loss while maximizing domain confusion.
  • Temporal inputs: The discriminator can use observations at times t and t+n to judge expert versus non-expert behavior from environmental change over time.This provides information unavailable from a static image alone.
  • Optimization: A gradient-reversal function acts as the identity during forward propagation and flips the sign during backpropagation through the domain loss.The resulting stochastic gradient ascent moves the extractor away from domain-discriminative features.
  • Training signal: Third-person demonstrations include expert and non-expert trajectories from the expert domain to provide sufficient discrimination signal.The complete formulation is summarized graphically in Figure 2.
  • Policy training: The novice policy uses −log DR as a reward and is trained with TRPO, while discriminator-related components are optimized alternately with stochastic gradient methods.The process continues by alternating generator and discriminator steps.

6 EXPERIMENTS

The experiments examine whether third-person imitation can transfer behavior from expert image-based rollouts in one domain to a policy operating in another domain. They also frame hyperparameter sensitivity as an experimental question.

  • Experimental questions: The experiments ask whether third-person imitation learning is possible in simple settings.The setting provides expert image-based rollouts in one domain and trains a policy in a different domain.
  • Experimental questions: A central evaluation tests whether a policy in a different domain can replicate the essence of behavior shown by expert rollouts.The question concerns transfer from demonstrations in one domain to execution in another.

2. Does the algorithm we propose benefit from both domain confusion and velocity?

This section asks whether the algorithm benefits from both domain confusion and velocity, and how sensitive it is to deployment hyperparameters.

  • Ablation question: The section evaluates whether the algorithm benefits from domain confusion.
  • Ablation question: The section evaluates whether the algorithm benefits from velocity.
  • Ablation question: The section considers sensitivity to the hyperparameters selected during deployment.

5. How does our method compare against some reasonable baselines?

The method learns reasonable policies from third-person demonstrations across three environments, with domain confusion essential to strong performance and multi-time-step input providing a marginal improvement. Compared with baselines, third-person imitation is competitive with first-person imitation, while directly applying a first-person policy to the third-person environment fails.

  • 6.2 EVALUATIONS: The algorithm recovers reasonable policies for reacher, inverted pendulum, and point environments after initially unstable training.Reward variability shrinks substantially after several iterations.
  • 6.2 EVALUATIONS: Domain confusion is essential for strong performance across all three experiments, while multi-time-step input only marginally improves results.Figure 5 compares removing domain confusion, removing multi-time-step input, or removing both.
  • 6.2 EVALUATIONS: The domain confusion coefficient λ trades off feature invariance against preserving information needed to recover an accurate cost.Too-low λ slows learning and leaves features non-invariant; too-high λ destroys information too quickly.
  • 6.2 EVALUATIONS: A four-frame look-ahead showed good performance across all tasks and was used in the other experiments, although no window was consistently optimal.Small windows provide little additional class signal, whereas large windows make causality difficult to interpolate.
  • 6.2 EVALUATIONS: Point performance follows a fairly linear decline with camera-angle difference, while reacher performance is more stochastic.The comparison uses final reward after 20 epochs of third-person imitation learning.
  • 6.2 EVALUATIONS: Third-person imitation is competitive with first-person imitation, whereas applying a first-person policy directly to the third-person agent fails miserably.Standard reinforcement learning with full state information and true reward calibrates the comparison.

7 DISCUSSION AND FUTURE WORK

The paper frames third-person imitation learning as an important problem and presents an algorithm capable of solving simple tasks. It identifies jointly training policy and cost features at the pixel level as a promising future direction.

  • The paper argues that third-person imitation learning will become increasingly important as first-person sample collection remains costly.
  • Jointly training policy and cost features at the pixel level could allow image features to be reused.

8 APPENDIX A: LEARNING CURVES FOR BASELINES

The appendix provides standalone learning-curve plots for the experimental baselines and reports the model components and training settings used in these experiments.

  • Standalone plots present learning curves for each experimental baseline, enabling closer examination of individual-curve variance.
  • Figures 10, 11, and 12 show performance comparisons for inverted pendulum, reacher, and point domains, respectively.
  • The joint feature extractor processes 50 x 50 RGB images through two convolutional and max-pooling layers using five 3-sized filters per layer.
  • The domain and class discriminators consume domain-agnostic convolutional outputs and produce two-class log probabilities through feed-forward layers and softmax.
  • ADAM trains the discriminators at learning rate 0.001, while the RL generator uses the off-the-shelf TRPO implementation in RLLab.
Loading 1703.01703v2…