Source-linked AI summary
Deep Generative Stochastic Networks Trainable by Backprop
Yoshua Bengio, Éric Thibodeau-Laufer, Guillaume Alain, Jason Yosinski
TL;DR
Deep unsupervised models remain difficult to train because multimodal distributions make marginalization, normalization, and MCMC mixing hard. The paper introduces GSNs, which learn local Markov-chain transitions with backpropagation instead of modeling P(X) directly; the resulting theory and experiments support consistent distribution capture and better mixing in latent-state models.
Problem
Deep unsupervised models remain difficult to train jointly, while multimodal distributions complicate inference, sampling, and learning.
Method
GSNs learn a Markov-chain transition operator and train its reconstruction distribution with gradient descent and backpropagation rather than directly modeling P(X).
Results
Under consistency conditions, the chain’s stationary distribution estimates the data-generating density, and latent-state GSNs mix better than shallower models without latent states.
Takeaways & Limitations
The framework supports generative architectures with intermediate noise, structured outputs, and missing values while avoiding intractable partition-function calculations.
Takeaways & Limitations
Local-noise designs require sufficiently large jumps to connect separated modes, trading off denoising difficulty against mixing between modes.
Abstract
from arXiv · showhide
We introduce a novel training principle for probabilistic models that is an alternative to maximum likelihood. The proposed Generative Stochastic Networks (GSN) framework is based on learning the transition operator of a Markov chain whose stationary distribution estimates the data distribution. The transition distribution of the Markov chain is conditional on the previous state, generally involving a small move, so this conditional distribution has fewer dominant modes, being unimodal in the limit of small moves. Thus, it is easier to learn because it is easier to approximate its partition function, more like learning to perform supervised function approximation, with gradients that can be obtained by backprop. We provide theorems that generalize recent work on the probabilistic interpretation of denoising autoencoders and obtain along the way an interesting justification for dependency networks and generalized pseudolikelihood, along with a definition of an appropriate joint distribution and sampling mechanism even when the conditionals are not consistent. GSNs can be used with missing inputs and can be used to sample subsets of variables given the rest. We validate these theoretical results with experiments on two image datasets using an architecture that mimics the Deep Boltzmann Machine Gibbs sampler but allows training to proceed with simple backprop, without the need for layerwise pretraining.
1 Introduction
The paper addresses the difficulty of jointly training deep unsupervised models by introducing Generative Stochastic Networks, which learn Markov-chain transitions rather than P(X) directly. The theory supports flexible architectures, missing-value and structured-output sampling, and experiments show improved mixing in latent-state models.
- Joint unsupervised training of deep models remains difficult despite progress in jointly training deep supervised models.
- GSNs estimate the data distribution indirectly by parametrizing a Markov-chain transition operator instead of P(X).This reframes density estimation as supervised-like function approximation, enabling maximum likelihood, gradient descent, and backpropagation without intractable partition functions.
- Under consistency conditions, learning a noisy reconstruction distribution makes the chain’s stationary distribution a consistent estimator of the data density.The corruption distribution may be purely local rather than requiring support over the entire visible-variable domain.
- GSNs extend to architectures with intermediate noise and support structured outputs, missing values, and related conditional sampling settings.
- Latent-state GSNs mix better than shallower models without latent states in the reported experiments.The architecture resembles deep Boltzmann machine Gibbs sampling but is trained with backpropagation without layerwise pretraining.
- The theory provides a novel justification for dependency networks and defines a proper joint distribution over visible variables learned by such models.
2 Summing over too many major modes
Highly multimodal distributions make inference, sampling, and learning difficult because approximating their sums and moving between modes can be intractable. GSNs instead learn generally local transition distributions, which often have fewer important modes and can be handled through function approximation.
- Graphical-model computations become difficult when many non-negligible modes contribute to the modeled distribution.The difficulty includes marginalization and estimating normalization-constant gradients.
- Structured-output sampling poses the same challenge when y and h are high-dimensional with a complex, highly multimodal joint distribution given x.
- MCMC estimation worsens for multimodal distributions because chains must traverse important modes and their mixing times, especially as modes become sharp and separated.
- GSNs rely on local Markov-chain transitions whose important modes are often confined to the neighborhood of the previous state.This makes transition partition-function gradients easier to approximate than those for the full data distribution.
- In the extreme unimodal denoising case, training reduces to predicting clean x from corrupted x̃ through function approximation.
- GSNs can also use multimodal output models, including mixture-of-Gaussians, RBMs, or NADE, when their parameter gradients are estimable.
- The approach avoids approximating partition-function gradients inside the training loop while retaining potential to capture rich distributions.
3 Generative Stochastic Networks
GSNs estimate the data distribution through a Markov-chain transition operator whose stationary distribution matches the target under suitable conditions. The theory supports local corruption, missing-input sampling, and dependency-network joints, while highlighting mixing, noise, and training limitations.
- Training framework: GSNs estimate P(X) indirectly by learning a Markov-chain transition operator rather than modeling the data density directly.The framework aims to make density estimation more like supervised function approximation.
- Local noise and ergodicity: Local corruption can preserve ergodicity while restricting each reconstruction distribution to nearby structure instead of every mode of P(X).The local-noise construction requires paths through the support with sufficiently small successive jumps.
- Stationary-distribution guarantees: Theorems show that, under the stated conditions, the stationary chain distribution has the same X-marginal as the data-generating distribution.The results establish equality of the joint state distributions across chain steps and consequently equality of their X-marginals.
- Limitations: The theory leaves the GSN training criterion for future work, and large noise can force unimodal reconstructions to average multiple modes.The paper notes that experiments used unimodal factorized reconstructions and identifies multimodal alternatives as future work.
- Missing inputs and structured outputs: GSNs can sample a conditional distribution by clamping observed variables while repeatedly resampling the unobserved variables.The paper states that this procedure samples from the appropriate conditional distribution.
- Dependency networks: If a dependency network has a stationary GSN chain, it defines a joint distribution even when its separately estimated conditionals are inconsistent.Consistent conditionals additionally yield a stationary distribution that consistently estimates the ground-truth joint distribution.
4 Experimental Example of GSN
The experiments evaluate GSNs on MNIST and TFD using deep stochastic architectures inspired by DBMs, trained with backpropagation. The models mix well, generate sharp samples, complete missing inputs stochastically, and achieve competitive TFD density estimates.
- Architecture and training: The architecture follows a DBM-like Gibbs-sampling computational graph while allowing efficient backpropagation-based training.Information propagates upward and downward, and walkback unfolds intermediate Markov-chain samples during training.
- Architecture and training: The walkback loss sums reconstruction negative log-likelihoods for target X across all 2^D reconstruction steps, where D is network depth.Intermediate samples are generated as if running the Markov chain from the input.
- Experimental setup: GSN experiments evaluated 2- and 3-hidden-layer networks on MNIST and TFD, alongside regular denoising auto-encoders.The models used tanh hidden units and Gaussian noise, following the referenced experimental setup.
- MNIST results: The MNIST GSN mixes well between classes, produces sharp digit images, and stochastically completes missing left halves while right halves remain clamped.The conditional chain explores plausible completion variations according to the trained conditional distribution.
- MNIST results: GSN and DBN-2 slightly outperformed samples drawn directly from the MNIST training set, possibly because mean-field outputs generated more prototypical samples.This explanation is presented as a possibility rather than a demonstrated mechanism.
- TFD results: The TFD GSN-3 achieved a test-set Parzen log-likelihood bound of 1890 ±29, compared with 1908 ±66 for DBN-2 and 604 ±15 for RBM.The DBN-2 comparison was statistically indistinguishable, while the setup was not retuned after MNIST experiments.
5 Conclusion
GSNs offer an alternative to maximum likelihood by training Markov-chain transition operators through backpropagation, using simpler reconstruction distributions to estimate the data-generating distribution. The approach supports conditional sampling and was evaluated on image datasets.
- GSNs train Markov-chain transition operators rather than directly modeling P(X), avoiding intractable marginalizations and partition functions.The stationary distribution of the chain is intended to estimate the data distribution.
- Reconstruction distributions are often simpler and approximately unimodal, making the training problem more like supervised function approximation than conventional unsupervised density estimation.The unimodality claim is provable in the limit of very small noise.
- Under mild conditions, denoising with noise injection and gradient descent can recover the data-generating distribution through a simple Markov chain.The noise must prevent perfect reconstruction, and the reconstruction distributions must be sufficiently powerful and consistent.
- The framework extends to local corruption and constrained or missing inputs, enabling conditional sampling over subsets of variables.These extensions allow structured-output modeling and stochastic completion of missing values.
A Supplemental Experimental Results
Supplemental experiments evaluate GSN sampling on MNIST and TFD against denoising auto-encoders and other generative models, while noting an unimplemented training-chain initialization procedure required by the theory.
- Experiments on MNIST and TFD evaluated GSN models with two and three hidden layers against regular one-hidden-layer denoising auto-encoders.The networks used tanh hidden units and Gaussian noise in the hidden layers, with standard deviation 2 in the described setup.
- The experiments did not implement the procedure proposed to initialize H0 from the previous H1 value for the same training example, leaving that theoretical requirement for future work.
- 214 ± 1.1 was the MNIST test-set Parzen log-likelihood bound for the two-layer GSN, versus -152 ± 2.2 for the single-layer DAE.
- On MNIST, the two-layer GSN exceeded the reported bounds for RBMs, two-hidden-layer DBNs, and a three-hidden-layer DBM under the stated Parzen-estimation setup.The cited comparisons were -244 ± 54 for RBMs, 138 ± 2 for a DBN-2, and 32 ± 2 for a DBM-3.
- On TFD, the GSN achieved 1890 ± 29, statistically indistinguishable from the DBN-2 at 1908 ± 66 and higher than the RBM at 604 ± 15.The TFD setup was not tuned after the MNIST experiments.