PaperlayerView exact revision on arXiv

Source-linked AI summary

OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learning

Peter Henderson, Wei-Di Chang, Pierre-Luc Bacon, David Meger, Joelle Pineau, Doina Precup

arXiv:1709.06683v2cs.LG

TL;DR

Demonstrations may reflect diverse reward functions, complicating inverse reinforcement learning with a single reward-policy model. OptionGAN jointly learns reward and policy options from observed expert states, outperforming a single-policy approach across varied tasks, particularly transfer settings.

  • Problem

    Expert demonstrations can contain trajectories arising from diverse reward functions, rather than one underlying reward.

    Source: Introduction · p. 1

  • Method

    OptionGAN uses adversarial learning to jointly recover reward options and policy options from expert demonstrations.

    Source: Introduction · p. 1, Reward-Policy Options Framework · p. 3

  • Results

    The learned joint reward-policy options outperform the single-policy case across a variety of tasks, particularly in transfer settings.

    Source: Discussion · pp. 6–7

  • Takeaways & Limitations

    The learned options show temporal and interpretable cohesion without a specified call-and-return termination function.

    Source: Discussion · p. 7

Abstract

from arXiv · show

Reinforcement learning has shown promise in learning policies that can solve complex problems. However, manually specifying a good reward function can be difficult, especially for intricate tasks. Inverse reinforcement learning offers a useful paradigm to learn the underlying reward function directly from expert demonstrations. Yet in reality, the corpus of demonstrations may contain trajectories arising from a diverse set of underlying reward functions rather than a single one. Thus, in inverse reinforcement learning, it is useful to consider such a decomposition. The options framework in reinforcement learning is specifically designed to decompose policies in a similar light. We therefore extend the options framework and propose a method to simultaneously recover reward options in addition to policy options. We leverage adversarial methods to learn joint reward-policy options using only observed expert states. We show that this approach works well in both simple and complex continuous control tasks and shows significant performance increases in one-shot transfer learning.

Introduction

Expert demonstrations can span settings and reward functions, making a single reward-policy model a poor fit. The paper extends options to jointly decompose rewards and policies, with particular strength in one-shot transfer.

Introduction

  • OptionGAN learns specialized reward-policy options from varied expert demonstrations and outperforms a single reward approximator and policy network in most evaluated tasks, especially one-shot transfer.The method uses adversarial learning and can implicitly divide the demonstration state space.Source: Introduction · p. 1

Related Work

  • Prior work decomposes rewards from segmented demonstrations, while latent-task methods require a provided variable; OptionGAN instead encodes latent structure unsupervised for one-shot transfer.Reward decomposition examples include transition segmentation (Krishnan et al. 2016) and video segmentation (Sermanet, Xu, and Levine 2016); latent-task approaches include Hausman et al. 2017, Wang et al. 2017, and Li, Song, and Ermon 2017.Source: Related Work · pp. 1–2

Preliminaries and Notation

  • One-step options terminate at every state, and their policy-over-options can act as a mixture-of-experts gate over specialized intra-option policies.The paper relates one-step options to MoEs and describes a formulation that converges to them.Source: Preliminaries and Notation · p. 2
  • IRLGAN is GAIL without known expert actions: its discriminator uses observations alone to estimate rewards for policy optimization.TRPO is used for simple tasks and PPO for complex tasks.Source: Preliminaries and Notation · pp. 2–3

Reward-Policy Options Framework

The framework pairs each policy option with a reward option and learns both within an extension of the options framework for inverse reinforcement learning.

Reward-Policy Options Framework

  • Each option includes a reward function, and its intra-option policy is optimized with respect to that option’s local reward.The formulation extends the options tuple with rω.Source: Reward-Policy Options Framework · p. 3
  • The policy-over-options selects both an intra-option policy and its reward; the reward is statically bound to its worker, unlike the dynamically assigned intrinsic reward in Feudal RL.The paper denotes the policy-over-reward-options rΩ and sets it equal to πΩ.Source: Reward-Policy Options Framework · p. 3
  • The method reformulates the discriminator as a mixture-of-experts and reuses its gating function for policy options, with regularizers for information distribution, diversity, and sparsity.The authors state that the discriminator loss formulation leads the mixture to converge to one-step options.Source: Reward-Policy Options Framework · p. 3

Learning Joint Reward-Policy Options

The learning procedure represents policy and reward options as mixtures of experts that share a policy-over-options gate, allowing the discriminator update to train the gate and reward options together.

Learning Joint Reward-Policy Options

  • The policy and reward are each mixtures of specialized experts, weighted by the policy-over-options gating function.The policy mixture combines intra-option action distributions; the reward mixture combines reward-option outputs.Source: Learning Joint Reward-Policy Options · p. 3
  • The discriminator loss is a weighted mixture of reward-option sigmoid cross-entropy losses and updates the policy-over-options and reward-option parameters together.The passage states that the loss fits directly into the IRLGAN framework.Source: Learning Joint Reward-Policy Options · p. 3
  • The algorithm takes expert trajectories as input, updates policy-over-options parameters, and then updates policy options with constrained steps.This summarizes the listed input and update steps.Source: Learning Joint Reward-Policy Options · p. 4

6 end

OptionGAN learns reward options, intra-option policies, and a policy-over-options together, using expert specialization and regularization to shape option selection.

6 end

  • Policy-gradient updates weight intra-option actions by the policy-over-options, supporting end-to-end learning and smoother constraint-based updates.Once the gate specializes, gradients for unselected intra-option policies are weighted by zero.Source: §6 · p. 4, Regularization Penalties · p. 4

Mixture-of-Experts as Options

  • The specialization loss favors experts with below-average loss, moving the gate toward deterministic selection interpreted as one-step options in the optimal case.The paper reports empirical temporal coherence despite not explicitly using call-and-return termination.Source: Mixture-of-Experts as Options · p. 4

Sparsity and Variance Regularization

  • Three policy-over-options penalties encourage uniform option use across data, sparse activations per example, and varied activations.The target sparsity rate τ is .5 for all cases.Source: Sparsity and Variance Regularization · p. 4

Mutual Information Penalty

  • Pairwise mutual-information minimization between option distributions encourages reward-option specialization; entropy regularization performed worse and produced unstable learning.The mutual-information term is described as encouraging diversity while providing stable learning.Source: Mutual Information Penalty · p. 4
  • Figure 2 shows two interpretable behavior modes per option, with temporal cohesion and specialization across time in a sample rollout.The modes are described as specializing across a sample rollout trajectory.Source: Figure 2 · p. 5

Experiments

Across the evaluated settings, OptionGAN’s final option-based policies generally outperform the single-approximator case; the experiments include simple locomotion, varied-gravity transfer, and more complex tasks.

Experimental Setup

  • Evaluations average true-reward returns over 25 rollouts, with results averaged across 10 random-seed trials and shared hyperparameters across IRLGAN and OptionGAN.For four-option runs, the uniform-option regularizer is relaxed by setting λb = .01.Source: Experimental Setup · p. 5

Simple Tasks

  • On Hopper-v1, HalfCheetah-v1, and Walker2d-v1, OptionGAN performs as well as or better than the single-approximator setting and exhibits temporally coherent, interpretable options.The Hopper options take on distinct roles in landing and take-off/air travel despite no explicit termination function.Source: Simple Tasks · p. 5

Experiments

  • OptionGAN’s final policy outperforms the single-approximator case in nearly every evaluated setting.Table 1 reports true average return and standard error across 10 trials on 25 final evaluation rollouts.Source: Experiments · p. 5, Table 1 · p. 6

One-Shot Transfer Learning

  • In one-shot transfer, OptionGAN shows significant performance increases on varied-gravity tasks, though gains vary with option count and regularization.The novice learns on normal Earth gravity from 40 expert rollouts across four gravity settings, repeated for three locomotion tasks.Source: One-Shot Transfer Learning · pp. 5–6

Complex Tasks

  • Complex-task evaluations use a randomly placed wall for HopperSimpleWall-v0 and a changing target for RoboschoolHumanoidFlagrun-v1; PPO replaces TRPO for the latter because TRPO exploration was inadequate.Each task uses 40 expert rollouts.Source: Complex Tasks · p. 6

Ablation Investigations

At convergence, the policy-over-options gating is nearly deterministic; relaxing uniform activation can improve learning and reveal structure in gravity demonstrations.

Convergence of Mixtures to Options

  • Across 1M samples, 100% of gating activations fell within {0 + ϵ, 1 − ϵ} for ϵ = 0.1, and 98.72% fell within range ϵ = 1−3.The measurements cover 40 trials across four environments.Source: Convergence of Mixtures to Options · p. 6

Effect of Uniform Distribution Regularizer

  • Relaxing the uniform distribution constraint produced rapid performance increases, particularly in HalfCheetah-v1 with four options.The reported result is specific to the relaxed constraint and four-option setting.Source: Effect of Uniform Distribution… · p. 6
  • With a relaxed uniform penalty, options dropped out during training: two generally dropped slowly in Hopper and Walker, while one dropped within 20 iterations in HalfCheetah.The authors posit that HalfCheetah’s state space contains enough mutually exclusive information to divide across three options, while Hopper tasks settle less quickly.Source: Effect of Uniform Distribution… · p. 6

Latent Structure in Expert Demonstrations

  • Expert demonstrations showed clear separation among options, indicating that the policy-over-options learned the latent division of the original expert environments and specializes over experts, benefiting one-shot transfer learning.The authors suspect options farther from the target gravity are less specialized because closer options significantly cover their state spaces.Source: Latent Structure in Expert… · p. 6

Discussion

The paper extends options to jointly decompose rewards and policies, reporting advantages over a single-policy case, particularly in transfer. Learned options also show temporal and interpretable cohesion without a specified call-and-return termination function.

Discussion

  • The uniform-regularizer comparison reports final novice option shares of [.27, .21, .25, .25] at λb = 10.0 and [0., 0., .62, .38] at λb = .01.Figure 4 reports average πΩ across final sample novice rollouts.Source: Figure 4 · p. 7
  • Expert demonstration states show gravity-dependent option assignments: .75G states map significantly to Option 1, and 1.25G states to Option 0.The paper describes this as inherent structure in the demonstrations.Source: Figure 5 · p. 7
  • Joint reward-policy options outperform the single-policy case in a variety of tasks, particularly in transfer settings.The framework is presented as an extension of options within generative adversarial inverse reinforcement learning.Source: Discussion · pp. 6–7
  • Learned options demonstrate temporal and interpretable cohesion without specifying a call-and-return termination function.Figure 3’s columns cover simple locomotion curves, one-shot transfer experiments, and policy-over-options activations over time.Source: Discussion · p. 7, Figure 3 · p. 7
  • The MoE formulation is presented as potentially scalable to extremely large networks and applicable to reward shaping in forward RL and specialized MoEs in GANs.The authors characterize the framework as extendable and say the optionated networks capture problem structure effectively for one-shot transfer generalization.Source: Discussion · p. 7

Supplemental Material

The supplemental material notes expansions for the option-case discriminator loss and the regularization terms.

Supplemental Material

  • The expectation over the discriminator loss for the option case is expanded in the supplemental material.The passage identifies this as part of the Expanded Equations subsection.Source: Expanded Equations · p. 9
  • The supplemental material also expands the regularization terms.This is separately identified in the Expanded Equations subsection.Source: Expanded Equations · p. 9

Expert Collection

The section documents expert-trajectory sources and a shared evaluation setup, then lists task-specific demonstrations and optimizer and network choices for baseline and option models.

Expert Collection

  • OpenAI Gym expert state-sequence demonstrations come from TRPO-trained policies; Roboschool Flagrun demonstrations use its provided PPO pre-trained expert.The Gym experts were trained for 1,000 iterations.Source: Expert Collection · p. 9

Experimental Setup and Hyperparameters

  • Evaluations average true-reward returns over 25 rollouts and 10 random-seed trials, with shared hyperparameters across IRLGAN and OptionGAN.TRPO is used for policy optimization except in the Roboschool Flagrun experiment, which uses PPO.Source: Experimental Setup and Hyperparameters · p. 9

Simple Tasks and Transfer Tasks

  • Simple-task experts use 10 rollouts, while transfer experts use 40 rollouts drawn from four environment variations, with none from the novice environment.Figure 6 lists expert average returns across mixed environments: 3657.71 (Hopper-v1), 4181.97 (HalfCheetah-v1), and 4218.12 (Walker2d-v1).Source: Simple Tasks and Transfer Tasks · p. 9, Figure 6 · p. 10

IRLGAN

  • IRLGAN uses two 64-unit tanh hidden layers, a sigmoid discriminator, and three discriminator updates per policy update.The passage says entropy and l2 regularization worsened performance.Source: IRLGAN · p. 10

OPTIONGAN

  • OptionGAN uses shared two-layer policy and discriminator networks with option-specific outputs and a separate policy-over-options network.The two- and four-option cases use different λb regularization scaling factors; the other listed scaling factors are λe = 10.0 and λv = 1.0.Source: OPTIONGAN · pp. 10–11

RoboschoolHumanoidFlagrun-v1

  • Roboschool Flagrun uses PPO with a clipping objective, five Adam policy updates, and larger ReLU networks.The network sizes are (128, 128), and the clipping parameter is ϵ = .02.Source: RoboschoolHumanoidFlagrun-v1 · p. 11

Decomposition of Rewards over Expert Demonstrations

The policy-over-options shows structure across expert demonstrations: some experts use options more selectively, while others mix them. The passage relates this pattern to state-space coverage.

Decomposition of Rewards over Expert Demonstrations

Loading 1709.06683v2…