PaperlayerView exact revision on arXiv

Source-linked AI summary

What Regularized Auto-Encoders Learn from the Data Generating Distribution

Guillaume Alain, Yoshua Bengio

arXiv:1211.4246v5cs.LGstat.ML

TL;DR

The paper asks what auto-encoders learn about the data-generating distribution. It analyzes regularized reconstruction criteria, showing that they estimate local density derivatives and can support approximate sampling from the estimated distribution.

  • Problem

    The paper addresses what auto-encoder reconstruction criteria learn about the input density and whether that information can be used for sampling.

  • Method

    It links denoising and contractive auto-encoders and analyzes regularized reconstruction error to characterize the learned reconstruction function.

  • Results

    Regularized auto-encoders estimate the score and second derivative of the log-density, while approximate Metropolis-Hastings can sample from the estimated model.

  • Takeaways & Limitations

    Regularized reconstruction error can provide an alternative to maximum likelihood, and score estimates can be used for approximate sampling without explicitly defining a density through a partition function.

  • Takeaways & Limitations

    Sampling can suffer from spurious probability maxima when the auto-encoder lacks sufficient capacity or training reaches a bad local minimum.

Abstract

from arXiv · show

What do auto-encoders learn about the underlying data generating distribution? Recent work suggests that some auto-encoder variants do a good job of capturing the local manifold structure of data. This paper clarifies some of these previous observations by showing that minimizing a particular form of regularized reconstruction error yields a reconstruction function that locally characterizes the shape of the data generating density. We show that the auto-encoder captures the score (derivative of the log-density with respect to the input). It contradicts previous interpretations of reconstruction error as an energy function. Unlike previous results, the theorems provided here are completely generic and do not depend on the parametrization of the auto-encoder: they show what the auto-encoder would tend to if given enough capacity and examples. These results are for a contractive training criterion we show to be similar to the denoising auto-encoder training criterion with small corruption noise, but with contraction applied on the whole reconstruction function rather than just encoder. Similarly to score matching, one can consider the proposed training criterion as a convenient alternative to maximum likelihood because it does not involve a partition function. Finally, we show how an approximate Metropolis-Hastings MCMC can be setup to recover samples from the estimated distribution, and this is confirmed in sampling experiments.

1. Introduction

The paper asks what reconstruction-based feature-learning criteria learn about the data-generating density and shows that regularized auto-encoders estimate local density structure, including the score. It further connects this estimate to approximate sampling from the learned distribution.

  • Motivation: Manifold learning focuses on identifying the lower-dimensional regions where the data-generating density concentrates.The setting assumes continuous inputs and a density concentrated near high-density manifolds.
  • Motivation: Reconstruction-based feature-learning algorithms raise the question of whether they learn the whole input density or only particular aspects of it.The paper frames this as a route toward understanding auto-encoders as implicit density models.
  • Contribution: The paper links denoising and contractive auto-encoders and studies what their reconstruction functions learn about the data-generating density.The main theoretical contribution concerns the density information recovered by minimizing the relevant regularized reconstruction criterion.
  • Contribution: The reconstruction function estimates the score, the direction of greatest density increase, and also the Hessian and local mean of the log-density.The local mean is the expected value in a small ball around the current location.
  • Sampling: An estimated score can support approximate Metropolis-Hastings sampling by approximating energy differences between proposed and current states.Experiments compare generated samples with training samples in multiple two-dimensional projections.

2. Contractive and Denoising Auto-Encoders

The section connects denoising and contractive auto-encoders, showing that small-noise denoising is equivalent to contraction of the whole reconstruction function. It motivates reconstruction-based estimation of density structure through the reconstruction function and its score-like behavior.

  • Geometric interpretation: Regularization preserves sensitivity to high-density-manifold variations while encouraging insensitivity to variations orthogonal to the manifold.The resulting reconstruction and representation capture manifold variation while mostly ignoring orthogonal variation.
  • Denoising and contractive criteria: Small Gaussian corruption with squared error makes a denoising auto-encoder similar to a contractive auto-encoder that contracts the whole reconstruction function.The corresponding contraction penalty coefficient is λ = σ2.
  • Denoising and contractive criteria: A contractive auto-encoder minimizes reconstruction error together with a penalty that encourages the reconstruction function to be less sensitive to its input.The section distinguishes contraction of the reconstruction r(·) from contraction of the encoder alone.
  • Optimal reconstruction: The optimal denoising reconstruction averages data-dependent contributions from a neighborhood of x, with larger noise incorporating a larger neighborhood.The weighted average is expressed through a density-dependent convolution or ratio of quantities.
  • Optimal reconstruction: As the corruption level σ approaches zero, the leading term of the optimal reconstruction is x, while the next term provides a score estimator.The score is the derivative of the log-density with respect to the input.
  • Assumptions and scope: The asymptotic results analyze non-parametric optimal solutions under smooth-density and small-noise assumptions, with quantities other than σ held fixed as σ →0.The reconstruction formula applies where p(x) ≠ 0.

3. Minimizing the Loss to Recover Local Features of p(·)

In the non-parametric, idealized setting, the optimal reconstruction function of a denoising or regularized contractive auto-encoder yields the score of the data density as noise vanishes. With sufficiently small known noise, the reconstruction can therefore provide a numerical approximation to ∂log p(x)/∂x.

  • The asymptotic optimal reconstruction function for DAE and RCAE models recovers the score ∂log p(x)/∂x as σ → 0.The result is stated for a non-parametric setting without parametric constraints on the reconstruction function.
  • A denoising auto-encoder trained only from samples can provide the score of the unknown data-generating density at input points.
  • The analysis asks what information about p can be inferred from a reconstruction function trained to minimize the empirical loss.
  • The ideal recovery argument assumes unlimited data, sufficient capacity, exact loss minimization, twice-differentiable reconstruction, known σ, and arbitrary numerical precision.
  • For fixed sufficiently small σ, the quantity derived from the reconstruction is close enough to ∂log p(x)/∂x for practical purposes.

3.3 Simple Numerical Example

Numerical experiments compare discrete RCAE and DAE score estimates in one dimension and around a two-dimensional spiral manifold. The estimates approach the true score as noise decreases, while the learned vector field points toward high-density manifold regions and reveals why reconstruction error is not itself an energy.

  • Simple Numerical Example: The one-dimensional experiment trains non-parametric DAE and RCAE models and tests whether their score estimates approach the true score as σ decreases.The density is chosen to make the behavior visually clear.
  • Simple Numerical Example: The discretized RCAE treats each reconstruction value at 1000 partition points as a free parameter and solves the resulting linear system exactly.
  • Vector Field Around a Manifold: In the two-dimensional spiral experiment, r(x)−x forms a vector field that points toward nearby high-density points on the data manifold.
  • Simple Numerical Example: RCAE and DAE score approximations are close to each other and converge toward the true score for σ ∈ {1.00, 0.31, 0.16, 0.06}.
  • Vector Field Around a Manifold: The vector field is near zero on the manifold, while low-density regions between manifold arms can act as sources because the vectors change sign.
  • Vector Field Around a Manifold: The analysis interprets reconstruction error as the norm of an approximate score rather than as an energy function.
  • Vector Field Around a Manifold: The score-like behavior is attributed to the opposition between reconstructing training examples and regularizing the reconstruction toward a constant function.

3.6 Limited Parameterization

The score-matching connection extends denoising auto-encoder results to a broad family of reconstruction functions, but specific parameterizations may not represent valid gradients of a potential energy. Gaussian corruption yields a regularized score-matching objective against a smoothed data distribution.

  • Limited Parameterization: A parameterized reconstruction function is not guaranteed to define a conservative vector field, so an associated energy function may not exist.
  • Limited Parameterization: Untied weights can produce better experimental results, while tied weights impose the gradient constraint V^T = W and can yield a non-normalizable energy.
  • Relation to Denoising Score Matching: The reconstruction-function result extends earlier denoising score-matching work to a large family of estimators, including non-parametric models.
  • Relation to Denoising Score Matching: For a known Gaussian noise scale, the optimal reconstruction function is obtained by minimizing the denoising criterion, which can be rewritten in score-matching form.
  • Relation to Denoising Score Matching: With Gaussian corruption, the denoising criterion is equivalent to score matching against a smoothed version of the data-generating density.Smoothing regularizes the target compared with matching the empirical distribution directly.

3.8 Estimating the Hessian

The Jacobian of the reconstruction function estimates the Hessian of the log-density after subtracting the identity matrix. This links local reconstruction behavior to manifold geometry through curvature in tangent and orthogonal directions.

  • The Hessian of the log-density can be estimated from the reconstruction Jacobian minus the identity matrix.
  • This relationship connects the derivative of the reconstruction function with the second derivative of the log-density and informs its geometric interpretation.
  • Near a density manifold, the score is nearly zero because points lie near a density ridge, while the Hessian distinguishes directions along and orthogonal to the manifold.
  • Because r(x) is close to x for small σ, the reconstruction Jacobian is close to the identity matrix.
  • In manifold directions, Jacobian eigenvalues near 1 correspond to faithful reconstruction and to log-density Hessian eigenvalues near 0.

4. Sampling with Metropolis-Hastings

The paper uses score estimates to approximate energy differences in Metropolis-Hastings sampling, enabling approximate draws from the estimated distribution. Sampling quality depends on training noise, model capacity, and MCMC mixing.

  • Energy differences: Energy differences between states can be approximated from the score using a first-order Taylor expansion or a discretized path integral.The path integral improves accuracy when the path remains in a region where the auto-encoder reliably estimates the energy gradient.
  • Metropolis-Hastings: A symmetric proposal combined with estimated energy differences supplies the acceptance ratio for approximate Metropolis-Hastings sampling.The ratio can use either the path-integral approximation or its discretized form, provided the trained model estimates ∂E/∂x accurately.
  • Sampling experiment: In a 10-dimensional density concentrated near a 1-d manifold, DAE samples convincingly model the original density when σtrain = σMH = 0.1.The experiment used isotropic Gaussian corruption with σtrain = 0.1 and set the Metropolis-Hastings noise parameter to σMH = 0.1.
  • Sampling limitations: Large training noise produces a blurry density, whereas very small noise leaves distant regions poorly modeled and can cause strange sampling behavior.Small proposal noise improves acceptance ratios but worsens mixing, while inadequate training can create spurious probability maxima.
  • Sampling limitations: Spurious maxima arise when the auto-encoder lacks capacity or training reaches a poor local minimum, causing MCMC to treat false attractors as high-probability regions.A failed DAE can produce an uncontrolled vector field away from the learned density manifold.

5. Conclusion

The conclusion identifies regularized auto-encoders as estimators of local density structure and shows that score estimates support approximate sampling. It also highlights limitations involving training, continuous inputs, and the small-noise regime.

  • 5. Conclusion: Regularized auto-encoders estimate local density properties, including the score, the second derivative of log-density, and the local mean.The result applies to denoising and a form of contractive auto-encoder training and challenges interpreting reconstruction error as an energy function.
  • Scope limitations: Finite data, small corruption noise, initialization, and non-convex optimization can produce undesirable reconstruction behavior far from training samples.These issues can affect learned vector fields and sampling outcomes outside the high-density manifold.
  • 5. Conclusion: Approximate Metropolis-Hastings sampling uses first-order energy differences derived from the score, and toy experiments recover samples resembling the training distribution.The approach avoids directly requiring an energy function or partition function for the training criterion.
  • Open questions: The theory relies on derivatives with respect to continuous inputs, leaving generalization to discrete data as an open question.Relative energy and ratio matching are identified as possible directions for discrete models.
  • Open questions: The analysis is restricted to denoising and contractive auto-encoders, while extending it to other reconstruction losses and corruption processes remains future work.Examples include Bernoulli cross-entropy, masking noise, and salt-and-pepper noise.
  • Scope limitations: The need for σ → 0 is described as troubling, motivating methods that can remove this limitation and handle both discrete and continuous variables.The paper also notes that score functions may be inconsistent when no analytic energy formulation is guaranteed.

6. Appendix

The appendix derives optimal reconstruction functions for denoising and contractive auto-encoders, showing their small-noise behavior is governed by derivatives of the data density. The expansions hold pointwise under explicit smoothness and support assumptions.

  • Optimal DAE solution: Theorem 1 derives the optimal DAE reconstruction function from expected quadratic loss with additive Gaussian corruption.The proof first minimizes the loss independently for each corrupted input, yielding a density-weighted reconstruction expression.
  • Small-noise expansion: For small corruption variance, the optimal reconstruction approaches the identity plus a term proportional to the density score, σ2∇log p(x) + o(σ2).The derivation uses Taylor expansion and Gaussian symmetry to obtain Eϵ[p(x + ϵ)ϵ] = σ2∇p(x) + o(σ2).
  • Scope of the expansion: The asymptotic formulas provide pointwise, not uniform, convergence as σ approaches zero.The required small-noise scale may depend on the evaluation point x.
  • Contractive criterion: Theorem 2 gives corresponding asymptotic expansions for the optimal reconstruction and its derivative under a regularized reconstruction loss.The result assumes a continuously differentiable density with support Rd and a twice-differentiable reconstruction function.

Proof

The proof converts the regularized objective into an optimality equation, expands its solution in σ2, and connects the resulting reconstruction to local density moments. The argument also situates these results relative to earlier interpretations of auto-encoder structure.

  • Proof: The proof uses the Euler–Lagrange equation to characterize the optimal reconstruction under the regularized loss.The reconstruction components can be optimized separately, producing a linear partial differential equation for each component.
  • Proof: At zero penalty, the optimal reconstruction is the identity r(x) = x, corresponding to perfect reconstruction.The proof then treats the solution as a power series in σ2 around this zero-penalty solution.
  • Proof: Taylor expansion of the optimality equation yields the leading small-σ2 correction involving the density gradient divided by the density.Higher-order terms are shown to contribute only at order σ4 or higher in the relevant derivative calculation.
  • Relation to prior work: The paper revisits earlier claims that reconstruction values and derivatives capture local density structure, replacing preliminary local-moment observations with more precise results.Earlier work had proposed using first and second derivatives to identify directions associated with preserving or increasing density.
  • Local moments: The analysis links the local mean of the density to its score and therefore identifies the optimal reconstruction with an estimate of the local mean.The reconstruction direction can loosely estimate the direction toward that local mean.

Proof

This proof develops the local-distribution analysis using normalized density inside a ball and Taylor expansions around its center. It tracks the integral terms needed to characterize local moments and their asymptotic errors.

  • Local mean expansion: The local-mean expansion is obtained by changing variables to center the integration at x0 and applying a Taylor expansion.Symmetry eliminates odd terms in integrals over the ball, while the remaining terms involve derivatives of p at x0.
  • Asymptotic error: The asymptotic remainder can dominate an exact δ4 term, so that higher-order coefficient cannot be usefully retained in the displayed expansion.The proof explicitly notes that the remainder is o(δ3).

Proof

The proof evaluates polynomial and quadratic integrals over centered balls using symmetry, scaling, and componentwise decomposition. These calculations supply the constants needed for the local-moment expressions.

  • Proof: Integrals of functions that are odd around the ball center vanish by symmetry.This cancellation removes terms containing an odd power of the centered coordinates.
  • Proof: Rescaling a ball of radius δ to the unit ball contributes the Jacobian factor δd to the integral.The change of variables yj = xj/δ yields the corresponding radius-dependent moment formulas.
  • Proof: The quadratic integral calculation decomposes componentwise and eliminates off-diagonal terms when i ≠ j.The remaining diagonal contributions produce the trace-based expression for a matrix quadratic form.
Loading 1211.4246v5…