Source-linked AI summary
Generalized Denoising Auto-Encoders as Generative Models
Yoshua Bengio, Li Yao, Guillaume Alain, Pascal Vincent
TL;DR
The paper addresses the limited theory connecting denoising auto-encoder training with data-density estimation for discrete data, non-Gaussian corruption, other losses, and non-infinitesimal noise. It estimates reverse conditionals and alternates reconstruction with corruption in a Markov chain, establishing consistency and validating the approach empirically; walkback training improves convergence to the same target distribution.
Problem
Prior results connected DAEs to density scores only for continuous data, Gaussian corruption, squared-error loss, and infinitesimal noise.
Method
The paper trains a conditional model Pθ(X|˜X) and combines it with the known corruption process in an alternating Markov chain.
Results
The generated-sample distribution converges to the data-generating distribution when the conditional estimator is consistent and the Markov chain is ergodic.
Takeaways & Limitations
Denoising training can implicitly estimate the data-generating process and generate samples for broad classes of data, corruption processes, and reconstruction losses.
Takeaways & Limitations
The conditional estimator may be poor when X is far from ˜X because training mainly covers corrupted inputs near the original examples.
Abstract
from arXiv · showhide
Recent work has shown how denoising and contractive autoencoders implicitly capture the structure of the data-generating density, in the case where the corruption noise is Gaussian, the reconstruction error is the squared error, and the data is continuous-valued. This has led to various proposals for sampling from this implicitly learned density function, using Langevin and Metropolis-Hastings MCMC. However, it remained unclear how to connect the training procedure of regularized auto-encoders to the implicit estimation of the underlying data-generating distribution when the data are discrete, or using other forms of corruption process and reconstruction errors. Another issue is the mathematical justification which is only valid in the limit of small corruption noise. We propose here a different attack on the problem, which deals with all these issues: arbitrary (but noisy enough) corruption, arbitrary reconstruction loss (seen as a log-likelihood), handling both discrete and continuous-valued variables, and removing the bias due to non-infinitesimal corruption noise (or non-infinitesimal contractive penalty).
1 Introduction
The paper reframes denoising auto-encoders as implicit generative models, extending their interpretation beyond small Gaussian noise, continuous data, and squared-error reconstruction. It proposes sampling through alternating conditional reconstruction and corruption steps, with empirical validation and walkback training to improve sampling behavior.
- Motivation: Regularized auto-encoders capture input-distribution structure, but their pure unsupervised probabilistic interpretation remained unclear.Earlier work viewed reconstruction error as an energy function, while the connection to the data-generating distribution was less direct than for RBMs.
- Limitations of prior theory: Small-noise Gaussian DAEs with squared-error loss estimate the data density score, but only under restrictive conditions.The result applies to continuous-valued inputs and becomes valid only in the limit of small corruption, despite strong practical results with larger noise.
- Contribution: The proposed probabilistic interpretation applies to any data type, broad-support corruption process, and reconstruction loss expressible as a log-likelihood.This generalization covers both discrete and continuous variables without restricting the corruption to Gaussian noise or the loss to squared error.
- Contribution: Training estimates P(X|˜X), and alternating samples from this reconstruction distribution and C(˜X|X) yields a Markov chain for recovering P(X).The chain repeatedly reconstructs from corrupted inputs, applies the stochastic corruption process, and iterates.
- Empirical validation: The theory is validated on artificial non-parametric and real parametric data, while model-defined corruption improves sampling behavior.The resulting training procedure has surface similarity to contrastive divergence.
2 Generalizing Denoising Auto-Encoders
The generalized DAE learns a conditional distribution Pθ(X| ˜X) from corrupted examples and uses it in an alternating Markov chain to estimate P(X). Under consistency and ergodicity conditions, the chain’s asymptotic distribution converges to the data-generating distribution, while locality and corruption choices govern practical sampling behavior.
- 2.1 Definition and Training: The DAE trains a conditional model Pθ(X| ˜X) by predicting clean X from corrupted ˜X, using possibly regularized maximum likelihood.Training pairs sample X from P(X) and ˜X from C( ˜X|X).
- 2.2 Sampling: The sampling procedure alternates drawing X from Pθ(X| ˜X) and drawing ˜X from the known corruption process C( ˜X|X).This defines a homogeneous Markov chain whose transition operator is formed by integrating over corrupted states.
- 2.2 Sampling: The chain is not generally a proper Gibbs chain because the learned conditional and corruption process need not be consistent with a unique joint distribution.If the chain is ergodic, its stationary distribution can nevertheless define a joint distribution between X and ˜X.
- 2.3 Consistency: As n →∞, if Pθn(X| ˜X) is consistent and Tn is ergodic, the asymptotic sample distribution πn(X) converges to P(X).The result relies on convergence of the learned transition operator to the true operator, whose asymptotic marginal is P(X).
- 2.3 Consistency: Sufficient conditions include positive model and corruption probabilities over a bounded finite-volume region, ensuring irreducibility, aperiodicity, and recurrence.The bounded domain prevents the chain from wandering to infinity; avoiding zero corruption also prevents degenerate identity-like behavior.
- 2.4 Locality of the Corruption and Energy Function: Local corruption makes P(X| ˜X) simpler than complex multimodal P(X), but overly local corruption can create spurious modes that non-infinitesimal noise and walkback training help alleviate.The local conditional can approximate nearby regions well while becoming unreliable when X is far from ˜X.
3 Reducing the Spurious Modes with Walkback Training
Walkback training modifies corruption by following the model’s own random walks away from training examples, then trains the DAE to reconstruct the originals. Under stated assumptions, it estimates the same target distribution as ordinary denoising while potentially doing so more efficiently.
- Motivation: Local corruption can create spurious modes in poorly visited regions, while more noise or training reduces but does not eliminate the problem.The authors report a larger improvement from training the chain to walk back toward training examples.
- Walkback method: Walkback replaces the original corruption process with a model-dependent process that generates additional corrupted training examples along walks away from each training example.A geometric parameter controls walk length, with p = 0.5 as the default.
- Consistency: Under Corollary 1’s assumptions, minimizing the walkback training criterion produces an implicit distribution that consistently estimates the data-generating distribution.The proof treats the converged walkback corruption process as fixed and applies the consistency result to the resulting alternating chain.
- Implication: Walkback estimates the same distribution as the original denoising algorithm but may do so more efficiently by spending corruption steps where they help the model most.The paper links this efficiency claim to the way walkback explores the corruption space.
4 Experimental Validation
Experiments validate the generalized DAE theory in non-parametric artificial data and real binarized MNIST using neural-network DAEs. The figures compare estimated or generated distributions and show fewer spurious samples with walkback sampling.
- Non-parametric validation: The non-parametric experiment tests the asymptotic theory in a low-dimensional setting where enough data and capacity can be reached.Figure 3 compares the true, empirical, and denoising maximum-likelihood estimated distributions.
- MNIST validation: The MNIST experiment trains a 784-2000-784 DAE for 200 epochs on 50000 binarized examples with salt-and-pepper corruption and cross-entropy loss.Walkback training uses a chain of 5 steps to generate 5 corrupted examples for each training example.
- Sampling behavior: Figure 4 compares successive Markov-chain samples from plain and walkback sampling, with fewer spurious samples under walkback.The plain scheme appears on the left and the walkback scheme on the right.
5 Conclusion and Future Work
The paper concludes that generalized denoising training implicitly estimates the data-generating process and supports sampling through an alternating Markov chain. It also identifies multimodal conditional reconstruction distributions as potentially necessary for fully capturing the data distribution.
- Conclusion: A denoising model can implicitly estimate the underlying data-generating process, while alternating denoising and corruption samples provides a convergent sampling procedure.The conclusion states that this applies to any DAE when the corruption is non-degenerate and the chain converges.
- Conclusion: Walkback training is suggested as a variant that may converge faster to the same target distribution.The wording preserves the paper’s qualification that this faster convergence is suggested by the study.
- Future work: Fully capturing P(X) asymptotically may require P(X|˜X) to represent multimodal distributions over X given ˜X.This is presented as an insight from the theoretical results rather than an unconditional requirement for every setting.