Source-linked AI summary
Connecting Generative Adversarial Networks and Actor-Critic Methods
David Pfau, Oriol Vinyals
TL;DR
GANs and actor-critic methods are difficult multilevel optimization problems, and the paper examines their close structural relationship. It formalizes GANs as modified actor-critic methods in a stateless environment where the actor cannot causally affect reward, then reviews stabilization strategies and related extensions. The connection is intended to encourage general techniques and exchange of ideas between the two communities.
Problem
GANs and actor-critic methods are difficult to optimize because multilevel systems with interacting private losses can exhibit instability, while stabilization strategies have largely developed separately.
Method
The paper compares GAN and actor-critic information flow, constructs an MDP interpretation of GANs, and reviews stabilization methods and related extensions.
Results
GANs can be viewed as modified actor-critic methods with blind actors in stateless MDPs, where the actor cannot causally affect reward and the true policy gradient is zero.
Takeaways & Limitations
The formal connection supports sharing stabilization ideas and developing general techniques for multilevel optimization with deep networks.
Takeaways & Limitations
The paper notes that compatible critics are not clearly extendable to GANs because the GAN MDP's true policy gradient is always zero.
Abstract
from arXiv · showhide
Both generative adversarial networks (GAN) in unsupervised learning and actor-critic methods in reinforcement learning (RL) have gained a reputation for being difficult to optimize. Practitioners in both fields have amassed a large number of strategies to mitigate these instabilities and improve training. Here we show that GANs can be viewed as actor-critic methods in an environment where the actor cannot affect the reward. We review the strategies for stabilizing training for each class of models, both those that generalize between the two and those that are particular to that model. We also review a number of extensions to GANs and RL algorithms with even more complicated information flow. We hope that by highlighting this formal connection we will encourage both GAN and RL communities to develop general, scalable, and stable algorithms for multilevel optimization with deep networks, and to draw inspiration across communities.
1 Introduction
Multilevel models combine several models with private losses, making ordinary optimization vulnerable to oscillation and collapse. GANs and actor-critic methods share this structure and motivate cross-community stabilization strategies.
- 1 Introduction: Hybrid multilevel losses pass information among models that optimize separate private objectives.This differs from standard single-objective optimization.
- 1 Introduction: Ordinary gradient descent can produce oscillations or collapse onto degenerate solutions in these systems.
- 1 Introduction: GANs and actor-critic methods both use a first model whose output is evaluated by a second model with special environmental information.The GAN discriminator accesses real samples, while the actor-critic critic accesses reward information.
- 1 Introduction: The two communities developed largely independent techniques for stabilizing training despite these close parallels.The paper reviews shared and model-specific approaches to these instabilities.
2 Algorithms
GANs and actor-critic methods are closely related bilevel optimization problems in which one model generates actions or samples and another evaluates them. The paper formalizes GANs as modified actor-critic methods and compares their training structures and stabilization strategies.
- Bilevel optimization: Bilevel optimization couples models that exchange information while minimizing separate losses, making ordinary gradient descent prone to oscillations or degenerate collapse.These problems differ from single-objective machine learning because no unified cost governs all models.
- Generative adversarial networks: GANs train a generator to produce samples and a discriminator to classify them as real or generated using a cross-entropy zero-sum game.The generator typically transforms fixed Gaussian noise, while the discriminator outputs a binary classification probability.
- Actor-critic methods: Actor-critic methods simultaneously learn a policy actor and an action-value critic, with policy updates based on value gradients and critic updates based on temporal-difference learning.The reviewed continuous-action methods include DPG, SVG(0), and NFQCA, differing mainly in online versus batch training.
- Shared structure: Both model classes use feedforward information flow from a generator or actor to an evaluator, while only the evaluator directly accesses real samples or reward information.The evaluator’s gradient information guides updates to the model producing samples or actions.
- Stabilization strategies: Table 1 categorizes GAN and actor-critic stabilization approaches by whether improved performance is demonstrated, not yet demonstrated, or inapplicable.The categories are encoded in green, yellow, and red, respectively.
- GANs as actor-critic: GANs correspond to actor-critic learning in a stateless MDP with blind actors, pixel-setting actions, cross-entropy critics, and actor updates suppressed when real images are shown.The actor cannot observe the state, and the environment chooses between an actor-generated image and a real image.
- GANs as actor-critic: Because the actor cannot causally affect reward in this MDP, the true policy gradient is zero and critic-driven actor updates become adversarial rather than complementary.The critic learns reward-predictive directions and then lowers value in directions taken by the actor.
3 Stabilizing Strategies
The paper reviews stabilization strategies shared across GANs and actor-critic methods, alongside techniques specific to each setting. These include methods for balancing optimization, preventing collapse or poor exploration, stabilizing targets, and improving gradient behavior.
- Shared strategies: Freezing one model when it becomes too strong can prevent degenerate GAN solutions and has an analogous use in actor-critic learning.In actor-critic methods, freezing the actor or critic is triggered when the TD error crosses a threshold.
- Shared strategies: Label smoothing replaces binary labels with ϵ and 1 − ϵ to keep generator gradients informative when discriminator predictions are highly confident.The paper suggests an analogous application when reinforcement-learning rewards are binary and critic gradients vanish.
- Shared strategies: Historical averaging penalizes parameter updates that move too far from the past average, helping prevent oscillations caused by models optimizing different objectives.Replay buffers in deterministic policy gradient methods are described as conceptually similar to fictitious play, but applicable only to the actor.
- Shared strategies: Minibatch discrimination reduces GAN mode collapse by classifying batches rather than individual images, paralleling entropy penalties that encourage exploration in reinforcement learning.Both approaches address insufficient diversity by encouraging broader generator samples or higher-entropy policies.
- Shared strategies: Batch normalization has supported scaling GANs to deep convolutional networks and has helped across many environments in deterministic policy gradient learning.Virtual batch normalization uses a constant reference batch to reduce prediction correlations, but its further benefit remains untested in that RL setting.
- Actor-critic-specific strategies: Replay buffers remove training-data correlations in discrete and continuous reinforcement learning, while target networks stabilize Q-learning by fixing one network during temporal-difference updates.Target networks are not considered applicable to GANs because the stateless GAN formulation removes the second action-value-function appearance in the Bellman recursion.
- Actor-critic-specific strategies: Entropy regularization addresses insufficient exploration in actor-critic methods and has an analogous role in preventing GAN generators from collapsing onto a few modes.The paper suggests that continuous-control exploration methods may transfer to increasing GAN sample diversity.
- Actor-critic-specific strategies: Compatible critics provide unbiased policy-gradient approximations under stated optimality and compatibility conditions, but the paper finds no natural GAN analogue.In the GAN MDP, the true policy gradient is always zero because every policy has value 0.5, so adversarial rather than compatible GANs are preferred.
4 Conclusions
The paper argues that deep learning combined with multilevel optimization has broad potential, and that GANs and actor-critic methods reveal useful connections between communities. It aims to encourage general techniques and exchange of ideas for difficult optimization and exploration problems.
- 4 Conclusions: GANs and actor-critic methods have made major impacts despite inherent difficulties in optimization and exploration.The paper presents their connection as a basis for developing and adopting general techniques across communities.
- 4 Conclusions: The paper hopes that connecting the two fields will encourage general techniques and freer exchange of ideas between machine-learning communities.This conclusion is framed within the broader promise of combining deep learning with multilevel optimization.
Supplemental Material
The supplemental material identifies other machine-learning problems that can be framed as multilevel optimization problems and notes that the paper reviews more complex extensions beyond basic GANs and actor-critic methods.
- Supplemental Material: Figure 2 presents other machine-learning problems that can be formulated as multilevel optimization problems.The supplied caption identifies the figure’s subject but does not specify its individual problems or visual encoding.
- Supplemental Material: The paper reviews extensions and other machine-learning problems with more complex information flow beyond the basic GAN and actor-critic methods.This discussion is presented as an overview of selected examples rather than an exhaustive treatment.
A GAN extensions
GAN extensions modify the divergence objective, discriminator structure, or latent-variable architecture while retaining or expanding the original information flow. These extensions include f-divergence bounds, energy-based GANs, inference networks, adversarial autoencoders, and InfoGANs.
- A GAN extensions: f-GANs interpret GAN training as minimizing a lower bound on an f-divergence between the generator and true distributions.The original GAN loss recovers a bound on Jensen-Shannon divergence for an appropriate convex f, while other choices yield KL or squared Hellinger bounds.
- A GAN extensions: Energy-based GANs replace the discriminator with an energy-based autoencoder trained using a contrastive loss.Both f-GANs and energy-based GANs preserve the original GAN’s information and gradient flow while changing its loss function.
- A GAN extensions: Several GAN extensions add an inference network to infer latent states for new data, including hybrid variational-autoencoder/GAN architectures and latent-state classification.The discriminator may classify latent states of true data induced by the inference network in addition to classifying generated images.
- A GAN extensions: Adversarial autoencoders learn the structure of a variational autoencoder’s approximate posterior adversarially.The paper locates this architecture in Figure 2c.
- A GAN extensions: InfoGANs augment the generative model’s latent state with units trained to maximize mutual information with the data.The stated purpose is to learn the model’s most important factors of variation.
B AC extensions
The reviewed actor-critic extensions broaden deterministic policy gradients to stochastic policies and span model-free to model-based learning.
- SVG generalizes deterministic policy gradients to stochastic policies across a spectrum from model-based to model-free algorithms.
- SVG(0) is a direct stochastic generalization of deterministic policy gradients.
- SVG(1) combines actor, critic, and model gradients with reward information to train the actor.
C Imitation Learning and Inverse Reinforcement Learning
Imitation learning and inverse reinforcement learning are related minimax problems with close connections to GANs and actor-critic methods. Extensions can form a three-level optimization problem, while a particular GAN extension matches guided cost learning in maximum entropy inverse reinforcement learning.
- Imitation learning, apprenticeship learning, and inverse reinforcement learning refer to closely related approaches for learning control policies from expert trajectories.
- Imitation learning can be formulated as a minimax problem that learns a cost function minimizing the optimal policy's advantage over the expert policy.
- Replacing direct policy optimization with an actor-critic method would produce a three-level optimization problem containing GANs and actor-critic as subproblems.
- When the generative-model density is known, the GAN objective is identical to the maximum-entropy inverse-reinforcement-learning objective.
- GAN training is identical to guided cost learning, which estimates the maximum-entropy inverse-reinforcement-learning partition function with a learned importance-sampling estimator.