Source-linked AI summary
Deep Directed Generative Models with Energy-Based Probability Estimation
Taesup Kim, Yoshua Bengio
TL;DR
Energy-based models require samples from their own estimated distributions during training, while MCMC may struggle to mix across sharp modes. This paper trains a deep directed generator to approximate that distribution alongside the energy function. The paper concludes that this two-network approach avoids the intractability using neural networks alone.
Problem
Energy-model training requires sampling from the model distribution to estimate its normalization-gradient term, but exact sampling is intractable and MCMC can mix poorly across sharp modes.
Source: §1 · p. 1, §2.2 · p. 3Method
The framework trains deep neural networks for an energy function and a directed generator whose samples approximate the energy-model distribution and supply negative-phase samples without MCMC.
Source: §1 · p. 2, §3 · p. 4, §3.2 · p. 6Results
The paper concludes that the intractability can be avoided using two separate models represented only by neural networks.
Source: §5 · p. 8Takeaways & Limitations
The framework provides a neural-network-only approach to energy-based probabilistic modeling, while explicit treatment of generator entropy remains future work.
Source: §5 · p. 8The generator regularizer approximates external-distribution entropy based on the assumption that internal entropy could affect it.
Source: §3.2 · p. 6
Abstract
from arXiv · showhide
Training energy-based probabilistic models is confronted with apparently intractable sums, whose Monte Carlo estimation requires sampling from the estimated probability distribution in the inner loop of training. This can be approximately achieved by Markov chain Monte Carlo methods, but may still face a formidable obstacle that is the difficulty of mixing between modes with sharp concentrations of probability. Whereas an MCMC process is usually derived from a given energy function based on mathematical considerations and requires an arbitrarily long time to obtain good and varied samples, we propose to train a deep directed generative model (not a Markov chain) so that its sampling distribution approximately matches the energy function that is being trained. Inspired by generative adversarial networks, the proposed framework involves training of two models that represent dual views of the estimated probability distribution: the energy function (mapping an input configuration to a scalar energy value) and the generator (mapping a noise vector to a generated configuration), both represented by deep neural networks.
1 Introduction
Energy-based models assign configurations scalar energies and define probabilities by normalizing their negated exponentials, but that normalization makes maximum-likelihood training difficult. The proposed framework pairs an energy function with a generator, using an entropy regularizer to address the generator’s missing-mode problem.
1 Introduction
- The normalization sum makes energy-model maximum-likelihood training intractable, and MCMC can mix poorly between sharp, separated modes.The negative-phase gradient requires samples from the model distribution; the proposed directed generator instead transforms an i.i.d. random sample without a sequential sampling process.Source: §1 · p. 1
- The framework represents the estimated distribution with complementary energy-function and generator models, training the generator in a GAN-like way using energy as a discriminator.The energy function encourages lower-energy generated samples, while a regularization term that indirectly maximizes generator entropy is reported as essential for more plausible samples.Source: §1 · p. 2
2 Energy-Based Probabilistic Models
Energy-based probabilistic models define distributions through energy functions, with training based on a likelihood objective whose negative phase requires model-distribution samples. The section explains products of experts, the sampling difficulty, a classifier interpretation of training, and the remaining costs of deep energy models.
2.1 Products of Experts
- A product of experts combines unnormalized energy terms, with examples including Student-t pattern detectors, logistic-regression feature detectors, and restricted Boltzmann machine experts.In the RBM description, the energy is low when its experts agree that an input is real.Source: §2.1 · p. 2
2.2 Learning and Intractability
- The negative-phase gradient requires samples from the energy model, but MCMC can take too long to mix across sharp modes, motivating a generator that approximates the model distribution.The positive phase lowers energy on training examples, while the negative phase raises it on model samples; the terms are equal when the model matches the data distribution.Source: §2.2 · p. 3
2.3 Training Models as a Classification Problem
- Energy-model training can be viewed as binary classification between data samples and model samples, and under the stated approximation this view yields the same update rule.The classifier treats the data distribution as one class and the energy-model distribution as the other; optimal training makes the categories difficult to discriminate.Source: §2.3 · pp. 3–4
2.4 Models with Multiple Layers
- Deep stochastic-layer models require difficult inference over hidden units, while deterministic deep energy networks avoid that inference but still need costly or high-variance MCMC sampling.Source: §2.4 · p. 4
3 The Proposed Model
The proposed framework represents an estimated distribution with an energy function and a generator, trained to align their complementary roles. Generated samples replace MCMC samples in the energy model’s negative phase, while the generator is trained to match the energy model.
3 The Proposed Model
- The framework pairs a deep energy model with a deep generative model, trained as complementary views of the estimated distribution and to approximately align.The energy model estimates the distribution with an energy function; the generator deterministically produces samples intended to match it.Source: §3 · p. 4, Figure 1 · p. 5
3.1 Deep Energy Model
- Generated samples approximate the negative phase of the energy-model update, which otherwise uses samples from the energy model distribution.The energy function is trained to distinguish training examples from generated samples; the generator is trained to align its sampling distribution with the energy model.Source: §3 · p. 4, §3.1 · p. 6
3.2 Deep Generative Model
- The generator samples prior noise and maps it deterministically to data, providing non-iterative samples for the energy model’s negative phase.Its distribution is trained toward the energy model’s distribution by minimizing KL divergence; an entropy regularizer is used to constrain variability and avert convergence to energy minima.Source: §3.2 · p. 6
3.3 Relation to Generative Adversarial Networks
- Unlike a GAN discriminator, the energy function is not described as becoming a trivial constant classifier as the generator approaches the data distribution.The paper also presents the trained energy function as a generic way to compare input configurations by relative probability.Source: §3.3 · p. 7
4 Experiments
Experiments on synthetic 2D data show that the energy model fits the training data and generated samples match its distribution. The paper also presents smooth MNIST latent-space interpolations and generated 64×64 CelebA and LSUN bedroom samples.
4 Experiments
- The synthetic-data evaluation reports that generated samples follow the energy-model distribution while the energy function fits the training data.The experiments used two-spiral and four-spin datasets, each with 10,000 points.Source: §4 · p. 7
- MNIST samples change smoothly as latent data z changes linearly, with interpolations passing through plausible intermediate digits.The model used fully connected layers without convolutions or image transformations.Source: §4 · p. 7, Figure 3 · p. 8
- The convolutional model generates 64×64 color-image samples for CelebA faces and LSUN bedrooms.The two datasets contain more than 200K face images and nearly 3M bedroom images, respectively.Source: §4 · p. 8, Figure 4 · p. 8
5 Conclusion
The paper concludes that two separate neural-network models can avoid the intractability of energy-based probabilistic modeling, and identifies generator entropy, semi-supervised learning, and visualization as future directions.
5 Conclusion
- The paper reports that intractability can be avoided using two separate deep models built only from neural networks.Source: §5 · p. 8
- Future work will explicitly address generator entropy and extend the deep energy model to semi-supervised learning.Source: §5 · p. 8
- The authors identify approximate visualization of the energy function on high-dimensional inputs as a useful future direction.Source: §5 · p. 8