PaperlayerView exact revision on arXiv

Source-linked AI summary

A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models

Chelsea Finn, Paul Christiano, Pieter Abbeel, Sergey Levine

arXiv:1611.03852v3cs.LGcs.AI

TL;DR

The paper asks whether cost learning in generative modeling connects formally to cost learning in reinforcement learning, a question relevant to stability and scalability. It shows that sample-based MaxEnt IRL is equivalent to a GAN with an evaluable generator density, and uses this connection to motivate GAN-based EBM training. The paper further argues that GAN training can improve sample quality even when exact density evaluation is available, while identifying sampling and behavioral-cloning limitations.

  • Problem

    Cost-function learning is established in reinforcement learning but relatively new in generative modeling, motivating a formal connection between the two areas.

  • Method

    The paper derives an equivalence between sample-based MaxEnt IRL and GANs using a discriminator that incorporates efficiently evaluable generator densities, then extends the view to EBMs.

  • Results

    GAN training can significantly improve sample quality even when the generator density can be evaluated exactly.

  • Takeaways & Limitations

    The connection suggests GAN-based methods for training energy-based models and motivates combining adversarial modeling with density-evaluable generators.

  • Takeaways & Limitations

    Behavioral cloning can produce ineffective trajectories through compounding errors, while EBM sampling can struggle with multiple modes and diverse-sample generation.

Abstract

from arXiv · show

Generative adversarial networks (GANs) are a recently proposed class of generative models in which a generator is trained to optimize a cost function that is being simultaneously learned by a discriminator. While the idea of learning cost functions is relatively new to the field of generative modeling, learning costs has long been studied in control and reinforcement learning (RL) domains, typically for imitation learning from demonstrations. In these fields, learning cost function underlying observed behavior is known as inverse reinforcement learning (IRL) or inverse optimal control. While at first the connection between cost learning in RL and cost learning in generative modeling may appear to be a superficial one, we show in this paper that certain IRL methods are in fact mathematically equivalent to GANs. In particular, we demonstrate an equivalence between a sample-based algorithm for maximum entropy IRL and a GAN in which the generator's density can be evaluated and is provided as an additional input to the discriminator. Interestingly, maximum entropy IRL is a special case of an energy-based model. We discuss the interpretation of GANs as an algorithm for training energy-based models, and relate this interpretation to other recent work that seeks to connect GANs and EBMs. By formally highlighting the connection between GANs, IRL, and EBMs, we hope that researchers in all three communities can better identify and apply transferable ideas from one domain to another, particularly for developing more stable and scalable algorithms: a major challenge in all three domains.

1 Introduction

The paper establishes a mathematical equivalence between certain sample-based maximum entropy IRL methods and GANs when generator densities are evaluable, and extends this connection to energy-based models.

  • Motivation: GANs learn a generator’s cost function simultaneously with a discriminator, paralleling cost-function learning in inverse reinforcement learning.IRL, also called inverse optimal control, infers the cost underlying demonstrated behavior.
  • Core equivalence: Sample-based MaxEnt IRL becomes mathematically equivalent to GAN training when the generator density is efficiently evaluable and supplied to the discriminator.The modified discriminator incorporates generator likelihood values into its input or form.
  • Implication: GAN training can significantly improve sample quality even when the generator density can be evaluated exactly.The paper relates this effect to IRL’s ability to imitate behaviors that behavioral cloning cannot successfully learn.
  • Energy-based models: MaxEnt IRL is a special case of an energy-based model whose learned cost corresponds to the energy function and is trained by maximum likelihood.This relationship motivates using a particular GAN form to train EBMs.

2 Background

The background defines GANs, EBMs, and IRL, then describes MaxEnt IRL and guided cost learning as sample-based approaches for fitting costs and generating demonstrations. It also outlines likelihood, sampling, and behavioral-cloning limitations.

  • Generative adversarial networks: GANs train a generator and discriminator simultaneously, with the discriminator distinguishing generated samples from data and the generator seeking data-like outputs.The discriminator receives samples and outputs their probability of originating from the data distribution.
  • Energy-based models: EBMs assign energies to samples and model data with a Boltzmann distribution, but maximum-likelihood training requires an often-intractable partition function.Estimating the partition function commonly requires sampling from the model distribution during learning.
  • Energy-based models: MCMC sampling for EBMs can struggle with distinct modes and may take arbitrarily long to produce diverse samples.Approximate inference can miss modes, causing the energy function to assign them incorrectly low energy.
  • Inverse reinforcement learning: MaxEnt IRL models demonstrated trajectories with a Boltzmann distribution whose energy is the learned trajectory cost.Lower-cost trajectories have higher likelihood, while suboptimal trajectories become exponentially less probable as cost increases.
  • Guided cost learning: Guided cost learning estimates the MaxEnt IRL partition function by training a sampling distribution and applying importance sampling.It alternates between optimizing the cost and optimizing the sampling policy to reduce importance-sampling variance.
  • Guided cost learning: The optimal importance-sampling distribution matches the demonstration distribution, yielding both a learned cost function and a policy that generates demonstration-like samples.The method can scale to high-dimensional state and action spaces with nonlinear cost functions.
  • Guided cost learning: Importance-sampling estimates can have high variance when the sampling distribution fails to cover trajectories with high exp(−cθ(τ)) values.Guided cost learning addresses this coverage problem by mixing demonstration samples with generated samples.
  • Likelihood and behavioral cloning: Direct maximum likelihood can produce moment-matching distributions that allocate mass to negligible-probability regions when model capacity is limited.For sequential decisions, behavioral cloning additionally suffers compounding errors after policies leave the demonstrated state distribution.

3 GANs and IRL

The paper shows that a GAN with an efficiently evaluable generator density is mathematically equivalent to sample-based MaxEnt IRL. This modification also yields a GAN interpretation of guided cost learning and can improve training stability and sample quality even when likelihood is available.

  • GANs and IRL: GANs applied to IRL use the discriminator to represent learned cost and the generator to represent the policy.The correspondence is established for expert demonstrations and requires a particular discriminator form.
  • A special form of discriminator: Incorporating the evaluable generator density q(τ) into the discriminator makes its optimum independent of the generator.The discriminator is adjusted by subtracting logq(τ) from its sigmoid input, with logZ as the sigmoid bias.
  • A special form of discriminator: GAN training may improve samples from complex multimodal distributions even when the generator density can be evaluated and likelihood could be maximized directly.The paper motivates this as analogous to IRL imitating behaviors that behavioral cloning or direct maximum-likelihood regression may fail to learn successfully.
  • Equivalence between GANs and guided cost learning: For this Z, the discriminator gradient equals the MaxEnt IRL objective gradient, while the generator loss equals the cost minus the entropy of q(τ).Thus the discriminator and generator updates reproduce the corresponding guided-cost-learning updates.
  • Z estimates the partition function: The minimizing discriminator parameter Z is an importance-sampling estimator for the MaxEnt IRL partition function.This follows by setting the derivative of the discriminator loss with respect to Z to zero.
  • Discussion: The resulting GAN is precisely a sample-based algorithm for MaxEnt IRL and provides an adversarial strategy for training energy-based models.The connection is obtained by using a generator with efficiently evaluable densities and incorporating those densities into the discriminator.

4 GANs for training EBMs

The paper derives a GAN-based method for training energy-based models when generator densities are available. The method uses density information to obtain an unbiased partition-function estimate and remains consistent without exact generator sampling.

  • Partition-function estimation: Adversarial training can estimate the partition function for energy-based models by approximately sampling from the energy-induced distribution.Existing approaches use adversarial training to derive fast partition-function estimates.
  • Partition-function estimation: When generator density is available, the partition function admits an unbiased estimate using generated and real data.The mixture μ combines generated and real points, while q(x) and e_p(x) provide generator-density and data-density terms.
  • Adversarial optimization: The resulting loss updates the generator to minimize energy while maximizing entropy.
  • Adversarial optimization: Setting e_p(x)=p_θ(x) yields a GAN with discriminator output σ(E_θ(x)−logq(x)) and discriminator log-probability and generator log-odds losses.
  • Comparison with prior EBM-GANs: Using generator density produces an unbiased partition-function estimate without assuming the generator samples exactly from the energy-based model.This contrasts with Kim and Bengio’s biased estimator, which converges when the generator samples correctly.
  • Comparison with prior EBM-GANs: A related energy-based GAN uses an autoencoder discriminator and mean-squared reconstruction error as energy, rather than the discriminator form developed here.

5 Related Work

Related work connects adversarial training with imitation learning, sequential generation, actor-critic optimization, and energy-based modeling. The paper distinguishes its approach by using generator or policy density information where available.

  • Imitation learning: GAIL uses an adversarial structure to learn a policy matching expert demonstrations.
  • Imitation learning: The paper interprets GAIL as optimizing a cost learned through maximum entropy IRL, while its unconstrained discriminator leaves that cost implicit.Because GAIL does not use generator density, the discriminator is discarded and the policy is the final output.
  • Sequential generation: Prior sequential-generation methods merge maximum likelihood with known or surrogate rewards, whereas this work assumes the reward function is unknown.
  • Sequential generation: Yu et al. learn sequence-generation costs with GANs but do not incorporate known policy-density values into the discriminator.
  • Sequential generation: Their experiments report that maximum-likelihood pre-training is crucial for good performance, limiting practicality for recurrent generators without densities.
  • Optimization connections: Pfau and Vinyals relate GAN and actor-critic optimization, suggesting that stabilization techniques may transfer to imitation learning’s two-level optimization.

6 Discussion

The paper establishes an equivalence between generative adversarial modeling and maximum entropy inverse reinforcement learning, using generator likelihoods to obtain an unbiased energy estimate. It also proposes adversarial training as a way to train energy-based models and reduce reliance on MCMC.

  • Discussion: The paper shows an equivalence between generative adversarial modeling and maximum entropy inverse reinforcement learning.
  • Discussion: The derivation uses a discriminator that incorporates generator likelihoods, leading to an unbiased estimate of the underlying energy function.
  • Discussion: Future work includes combining density-providing deep generators with adversarial modeling for potentially more stable training and broader discrete-data applicability.
  • Discussion: The proposed EBM algorithm trains a neural network to sample from the distribution induced by the current energy.
  • Discussion: This approach could reduce the computational challenges of existing MCMC-based solutions.
Loading 1611.03852v3…