PaperlayerView exact revision on arXiv

Source-linked AI summary

Energy-based Generative Adversarial Network

Junbo Zhao, Michael Mathieu, Yann LeCun

arXiv:1609.03126v4cs.LGstat.ML

TL;DR

GAN training traditionally uses a probabilistic binary discriminator, while this paper explores a more flexible energy-based formulation. EBGAN treats the discriminator as an energy function and demonstrates an auto-encoder implementation using reconstruction error. The paper reports better convergence patterns and scalability to high-resolution image generation, while its theoretical equilibrium recovers the data distribution.

  • Problem

    The paper addresses whether GANs can be formulated with flexible energy-based discriminators and loss functionals instead of a probabilistic binary classifier.

  • Method

    EBGAN treats the discriminator as an energy function and demonstrates an auto-encoder discriminator whose energy is reconstruction error.

  • Results

    EBGAN shows better convergence patterns and scalability to generate high-resolution images, with a hinge-loss Nash equilibrium satisfying pG*=pdata almost everywhere.

  • Takeaways & Limitations

    The energy-based perspective supports alternative discriminator architectures and loss functionals, including auto-encoders, within adversarial training.

  • Takeaways & Limitations

    The equilibrium characterization assumes there is no region where pdata(x)=0; in such regions, the optimal discriminator may take any value in [0,m].

Abstract

from arXiv · show

We introduce the "Energy-based Generative Adversarial Network" model (EBGAN) which views the discriminator as an energy function that attributes low energies to the regions near the data manifold and higher energies to other regions. Similar to the probabilistic GANs, a generator is seen as being trained to produce contrastive samples with minimal energies, while the discriminator is trained to assign high energies to these generated samples. Viewing the discriminator as an energy function allows to use a wide variety of architectures and loss functionals in addition to the usual binary classifier with logistic output. Among them, we show one instantiation of EBGAN framework as using an auto-encoder architecture, with the energy being the reconstruction error, in place of the discriminator. We show that this form of EBGAN exhibits more stable behavior than regular GANs during training. We also show that a single-scale architecture can be trained to generate high-resolution images.

1 INTRODUCTION

EBGAN reframes the GAN discriminator as an energy function that assigns low energy near the data manifold and higher energy elsewhere. This formulation supports alternative architectures and loss functionals, including an auto-encoder discriminator whose energy is reconstruction error.

  • Energy-based formulation: EBGAN views the discriminator as an energy function that assigns low values near high-density data regions and higher values outside them.The generator produces samples in regions assigned low energy by the discriminator.
  • Energy-based formulation: The energy-based formulation removes the need for an explicit probabilistic interpretation, allowing greater flexibility in discriminator architecture and training procedure.
  • Alternative discriminator designs: The original GAN’s probabilistic binary discriminator is one possible contrast function among many, rather than the only choice.
  • Alternative discriminator designs: EBGAN experimentally instantiates this framework with an auto-encoder discriminator whose energy is the reconstruction error.
  • Contributions: The paper contributes an energy-based adversarial formulation, a convergence result under a hinge loss, systematic experiments, and 256×256 ImageNet generation without a multi-scale approach.

2 THE EBGAN MODEL

EBGAN treats the discriminator as an energy function and trains the generator to produce low-energy samples while assigning higher energies to generated samples. The model uses a margin objective, establishes distribution-matching behavior at Nash equilibrium, and instantiates the discriminator as an auto-encoder whose energy is reconstruction error.

  • 2 THE EBGAN MODEL: EBGAN defines the discriminator as an energy function that assigns low values near high-density data regions and higher values elsewhere.The generator produces samples in regions assigned low energy, while the discriminator is trained to assign high energy to generated samples.
  • 2 THE EBGAN MODEL: The margin objective trains the discriminator to give real samples low energy and generated samples higher energy, while using a separate generator loss for stronger gradients.The generator loss retains non-zero gradients when the discriminator energy for generated samples exceeds the margin.
  • 2.2 OPTIMALITY OF THE SOLUTION: At Nash equilibrium, EBGAN's generator distribution equals pdata almost everywhere, and the discriminator output is constant within the margin.The theoretical analysis assumes infinite-capacity generator and discriminator models.
  • 2.3 USING AUTO-ENCODERS: The auto-encoder discriminator uses reconstruction error as its energy function instead of a binary logistic discriminator.The architecture is presented as an EBGAN instantiation, with the discriminator structured as an auto-encoder.
  • 2.3.1 CONNECTION TO THE REGULARIZED AUTO-ENCODERS: Auto-encoder energy functions can collapse to zero energy across the input space, motivating regularization that restricts low-energy reconstruction to a smaller region.The EBGAN generator supplies trainable contrastive samples that the discriminator should reconstruct with high energy.
  • 2.4 REPELLING REGULARIZER: The repelling regularizer discourages generated samples from clustering in one or a few modes of pdata.Its Pulling-away Term operates on encoder representations, attempts to orthogonalize pairwise sample representations, and is used in the generator loss but not the discriminator loss.

3 RELATED WORK

EBGAN situates GANs within an energy-based modeling framework and relates this view to probabilistic GANs and methods for stabilizing GAN training.

  • Energy-based GANs: EBGAN casts GANs into an energy-based model scope, connecting adversarial training with contrastive samples and energy functions.The related work positions noisy samples and contrastive divergence as relevant approaches.
  • GAN training stability: Prior GAN research addressed training stability through methods including noisy samples, architectural changes, and other stabilization techniques.The passage cites several approaches without specifying their individual effects.
  • Probabilistic GANs: Probabilistic GANs use a Gibbs distribution for energy-based density estimation but retain the computationally challenging partition function.This requires the energy function to be integratable.

4 EXPERIMENTS

The experiments evaluate EBGAN against GANs on MNIST, semi-supervised PI-MNIST, natural-image datasets, and ImageNet. They report more reliable MNIST training, improved Ladder Network performance, and high-resolution image generation.

  • Exhaustive grid search on MNIST: The grid search varied architectural choices and hyperparameters, including optimization settings, learning rates, and network depth.EBGAN models used Adam with learning rate 0.001, a fixed margin of 10, and tuned encoder depth with one decoder layer.
  • Exhaustive grid search on MNIST: The modified inception score I′ evaluated generated MNIST samples using an offline classifier trained on the entire MNIST training set.The formulation swaps the distribution order and omits the exponential operation, and was used only for this grid-search analysis.
  • Semi-supervised learning on MNIST: EBGAN-LN with a bottom-layer-cost Ladder Network profitably improved the Ladder Network’s performance on the PI-MNIST semi-supervised task.Table 2 reports error rates averaged over 15 random seeds across settings using 100, 200, and 1000 labels.
  • LSUN and CelebA: EBGAN was applied with deep convolutional architectures to generate 64 × 64 RGB images from LSUN bedrooms and aligned CelebA faces.The experiments compared EBGAN-PT with DCGAN under the same configuration and displayed generations side by side.
  • High-resolution ImageNet generation: EBGAN-PT generated ImageNet images at 128 × 128 using the full ImageNet-1k dataset and at 256 × 256 using a dog-breed subset.ImageNet-1k contains roughly 1.3 million images from 1000 categories.

5 OUTLOOK

The paper bridges GANs and auto-encoders through an energy-based perspective, reporting better convergence and scalability to high-resolution generation while identifying conditional models as future work.

  • EBGANs bridge GANs and auto-encoders by revisiting adversarial learning from an energy-based perspective.
  • The framework can incorporate a family of energy-based loss functionals described by LeCun et al. (2006).
  • The paper reports better convergence patterns and scalability for generating high-resolution images.
  • Conditional settings are identified as a promising direction for future research.

TWO INTERPRETATIONS OF GANS

GANs admit complementary interpretations centered on either the generator or the discriminator. In the discriminator-centered view, the generator produces contrastive samples that can support semi-supervised learning.

  • Generator-centered interpretation: In the generator-centered interpretation, the discriminator acts as a trainable objective that guides generated samples toward the data manifold.
  • Generator-centered interpretation: This interpretation trains the generator toward a set of possible desired outputs rather than one target output.
  • Discriminator-centered interpretation: In the discriminator-centered interpretation, the generator is trained to produce contrastive samples.
  • Discriminator-centered interpretation: Iteratively feeding contrastive samples can enhance the discriminator’s semi-supervised learning performance.

C APPENDIX: EXPERIMENT SETTINGS

The appendix specifies convolutional architectures, training conventions, grid-search evaluation, and selected configurations for 128×128 and 256×256 experiments.

  • Batch normalization, input scaling to [-1,1], Tanh generator outputs, ReLU nonlinearities, and Gaussian weight initialization define the training setup.
  • Models are evaluated with a modified inception score I′ based on the expected KL divergence between p(y) and p(y|x).The modification swaps the distribution order and omits the e(·) operation.
  • Figure 4 shows the best GAN or EBGAN generations selected by the highest I′ score from the grid search.
  • The EBGAN discriminator uses a deep convolutional auto-encoder with strided convolution in the encoder and fractional-strided convolution in the decoder.The margin is set to 80 for LSUN and 20 for CelebA.
  • The experiments use specified generator, encoder, and decoder architectures for 128×128 and 256×256 models.The 128×128 and 256×256 configurations differ in layer widths, noise-plane schedules, and margins.
  • Noise is injected into every generator layer by concatenating initialized 4D noise tensors with current feature maps.

D APPENDIX: SEMI-SUPERVISED LEARNING EXPERIMENT SETTING

The semi-supervised experiments use a tuned Ladder Network baseline and place the same architecture inside EBGAN, with a margin schedule for EBGAN-LN training.

  • The baseline is a bottom-layer-cost Ladder Network with a fully connected 784-1000-500-250-250-250 architecture.Batch normalization and ReLU follow each linear layer.
  • The baseline tunes reconstruction-cost weights and learning rates, uses Adam with β1=0.5, and trains for 120,000 steps.
  • The experiments use the same Ladder Network architecture inside EBGAN and train the resulting EBGAN-LN model like the auto-encoder EBGAN.
  • EBGAN-LN starts with margin 16 and decays it to 0 within the first 60,000 steps.

OTHER DETAILS

The experiments emphasize selecting an appropriate energy margin and document dataset, loss, and batch-normalization choices that affect reproducibility and optimization.

  • The EBGAN-LN experiment used unpadded 28×28 MNIST, whereas auto-encoder grid searches used zero-padded 32×32 MNIST without a notable observed difference.
  • The auto-encoder generally used the ℓ2 norm of reconstruction discrepancy, but EBGAN-LN used the vanilla ℓ2 loss from the Ladder Network implementation.
  • Batch normalization omitted learned γ and retained only bias β, so the baseline was not a strict reproduction of the cited published models.The authors state that the effect of this choice on learning remains unknown.
  • The energy margin m should balance the discriminator’s real- and fake-sample loss terms.The fake-sample term lies in [0, m], while the real-sample term depends on discriminator capacity and dataset complexity.
  • An excessively large m can broaden the range of convergent γ values and create opposing gradients that complicate optimization.

LSUN AUGMENTED VERSION TRAINING

For LSUN bedrooms, the authors trained an EBGAN auto-encoder on augmented 64×64 patches cropped from 96×96 original images and visualized its generations.

  • The augmented LSUN bedroom training used 64×64 patches cropped from 96×96 original images.
  • The resulting patch-based EBGAN auto-encoder generations are shown in Figure 11.

COMPARISON OF EBGANS AND EBGAN-PTS

The paper compares EBGAN with its pull-away-term variant across MNIST, LSUN bedroom, and CelebA generation settings using matched experimental configurations. The comparisons include whole-image and augmented-patch LSUN data, with visual results reported in Figures 9–14.

  • The pull-away term’s influence was evaluated on whole-image LSUN bedrooms, augmented-patch LSUN bedrooms, and CelebA.
  • Figure 9 shows EBGAN auto-encoder generations across m values from 1 to 32 under fixed architecture and optimization settings.
  • Figure 10 presents EBGAN-LN generations from unpadded 28×28 MNIST using identical experimental settings across different random seeds.
  • The comparison pairs used identical architectural and hyper-parameter settings, with the pull-away-term cost weight set to 0.1.
  • EBGAN and EBGAN-PT generations were compared for augmented-patch LSUN bedrooms in Figures 11 and 13.
  • EBGAN and EBGAN-PT generations were compared for whole-image LSUN bedrooms in Figure 12.
  • EBGAN and EBGAN-PT generations were compared on CelebA in Figure 14.
Loading 1609.03126v4…