PaperlayerView exact revision on arXiv

Source-linked AI summary

Denoising Criterion for Variational Auto-Encoding Framework

Daniel Jiwoong Im, Sungjin Ahn, Roland Memisevic, Yoshua Bengio

arXiv:1511.06406v2cs.LG

TL;DR

Standard variational autoencoder training becomes intractable when input corruption requires marginalizing the encoder over noise, while restrictive posterior families limit flexibility. The paper introduces a denoising variational lower bound with a noise-marginalized recognition network, and reports consistent improvements for VAE and IWAE when corruption is sensible.

  • Problem

    Input corruption makes the standard VAE lower bound intractable through encoder-noise marginalization, while standard approximate posteriors are limited in flexibility.

  • Method

    The paper introduces a denoising variational lower bound and a recognition network obtained by marginalizing input noise, with Monte Carlo training applicable to VAE and other inference-network models.

  • Results

    The proposed denoising method consistently improved performance for variational autoencoders and importance weighted autoencoders, surpassing vanilla VAE and IWAE with suitable corruption levels.

  • Takeaways & Limitations

    Denoising can provide broader and more robust approximate posterior distributions while retaining a tractable training criterion under sensible corruption.

  • Takeaways & Limitations

    Performance depends on choosing an appropriate corruption level, and a pixel-wise corruption distribution did not noticeably improve over a global corruption rate.

Abstract

from arXiv · show

Denoising autoencoders (DAE) are trained to reconstruct their clean inputs with noise injected at the input level, while variational autoencoders (VAE) are trained with noise injected in their stochastic hidden layer, with a regularizer that encourages this noise injection. In this paper, we show that injecting noise both in input and in the stochastic hidden layer can be advantageous and we propose a modified variational lower bound as an improved objective function in this setup. When input is corrupted, then the standard VAE lower bound involves marginalizing the encoder conditional distribution over the input noise, which makes the training criterion intractable. Instead, we propose a modified training criterion which corresponds to a tractable bound when input is corrupted. Experimentally, we find that the proposed denoising variational autoencoder (DVAE) yields better average log-likelihood than the VAE and the importance weighted autoencoder on the MNIST and Frey Face datasets.

1 INTRODUCTION

Variational inference has become efficient through inference networks, but standard VAE posterior families remain restrictive. This motivates combining denoising with variational auto-encoding and developing a tractable objective for corrupted inputs.

  • Variational inference advances: Inference networks use observation-conditioned approximate posteriors with shared parameters across latent variables, enabling efficient training on large-scale datasets.The approach benefits from re-parameterization and REINFORCE training techniques.
  • Limitations of standard VAEs: Standard VAEs restrict each latent dimension to an independent univariate Gaussian whose parameters are produced by a neural network.This factorized posterior family can be inadequate for more complex real-world problems.
  • Denoising motivation: Denoising criteria corrupt inputs but train models to recover the original data, improving robustness to noise in the high-dimensional observation space.Prior work links this criterion to generalization by making nearby manifold points robust to small observational perturbations.
  • Proposed framework: The paper introduces noise-marginalized recognition networks that can represent broader approximate posterior families, including mixtures of Gaussians.The proposed denoising variational lower bound is designed to make training tractable and efficient under input corruption.
  • Proposed framework: The proposed denoising criterion is presented as applicable not only to VAEs but also to IWAE and NVIL-style variational models.Its broader applicability follows from using inference networks for approximate distributions.

2 BACKGROUND

Variational inference replaces direct posterior approximation with optimization of a tractable variational family. VAEs use shared inference networks and hidden-layer noise, while denoising can distinguish training augmentation from reconstructing clean inputs.

  • Variational inference: Variational inference approximates the intractable posterior p(z|x) with a tractable distribution qφ(z) from a restricted parameterized family.Mean-field variational inference is an example that treats dependent variables as independent.
  • Denoising criterion: Denoising differs from ordinary input-noise augmentation because the model reconstructs the original data rather than the corrupted input.This distinction changes the training target, not merely the dataset contents.
  • Variational inference: The variational lower bound enables optimization by maximizing an objective instead of directly minimizing KL divergence to the intractable target distribution.The marginal log-likelihood is independent of the variational distribution, and the KL term is non-negative.
  • Variational autoencoders: A VAE defines a prior over latent variables, a neural-network observation model, and an observation-conditioned inference network qφ(z|x).The inference-network parameters are shared across observations.
  • Variational autoencoders: In the VAE lower bound, the reconstruction term corresponds to hidden-layer noise, while the KL term regularizes the approximate posterior toward the prior.The regularizer also maximizes the entropy of the injected noise.
  • Training and extensions: The inference-network framework can be trained with backpropagation through the re-parameterization trick and extended to models such as IWAE, NVIL, and DRAW.These models use inference networks for their approximate distributions.

3 DENOISING CRITERION IN VARIATIONAL FRAMEWORK

The denoising variational framework marginalizes inference over corrupted inputs, yielding a broader approximate-posterior family and a tractable denoising lower bound. Its effectiveness depends on sensible corruption, while additional stochastic layers can tighten the bound.

  • 3.1 THE DENOISING VARIATIONAL LOWER BOUND: Marginalizing a Gaussian inference network over corrupted inputs produces a mixture of Gaussian distributions, including an infinite mixture when nonlinear networks predict the parameters.With Bernoulli corruption on D binary dimensions, the resulting mixture has 2^D components sharing the inference-network parameters.
  • 3.1 THE DENOISING VARIATIONAL LOWER BOUND: The corruption procedure acts as a stochastic layer below the inference network, and its parameters can be learned by backpropagation with the re-parameterization trick.The model reconstructs the original clean input x rather than the corrupted input ˜x.
  • 3.1 THE DENOISING VARIATIONAL LOWER BOUND: Corrupting the input before inference makes the standard variational bound intractable because the encoder distribution must be marginalized over input noise.The denoising variational bound addresses this by moving the expectation over corruption outside the logarithm.
  • 3.1 THE DENOISING VARIATIONAL LOWER BOUND: The denoising bound is not automatically tighter than the VAE bound because its approximate posterior depends on the corruption distribution.An inefficient corruption distribution can destroy reconstruction-relevant information, so choosing a sensible corruption distribution is important.
  • 3.1 THE DENOISING VARIATIONAL LOWER BOUND: The denoising approximate posterior can cover a broader distribution class than the ordinary VAE posterior, potentially representing multiple posterior modes.The ordinary posterior family is included within the broader family induced by input corruption.
  • 3.2 TRAINING PROCEDURE: Maximizing the denoising objective minimizes expected posterior divergence across corrupted versions of each input, encouraging inference that maps noised inputs toward the true posterior.The procedure is approximated with Monte Carlo samples of corrupted inputs and latent variables, and can also be applied to IWAE.

4 EXPERIMENTS

Experiments evaluate denoising variational criteria across datasets, corruption settings, sample sizes, inference architectures, and augmentation strategies. Denoising generally improves performance when corruption is appropriately calibrated, while excessive noise and some design choices reduce its benefits.

  • Experimental setup: Experiments compare denoising VAE and IWAE variants across MNIST and Frey Face using different corruption distributions and noise levels.Salt-and-pepper noise is used for binary MNIST, while Gaussian noise is used for real-valued Frey Face.
  • Denoising performance: All denoising methods surpassed vanilla VAE and vanilla IWAE at suitable corruption levels, but large noise amounts performed worse than the vanilla models.The zero-corruption setting makes DVAE and DIWAE identical to VAE and IWAE, respectively.
  • Noise sensitivity: Both models were relatively insensitive to Gaussian versus salt-and-pepper noise but more sensitive to noise magnitude.The optimal corruption level was reported as lying in (0, 5], because excessive noise removes reconstruction information and increases train-test distribution mismatch.
  • Sample size: Increasing the sample size M accelerated convergence and improved converged log-likelihood, with VAE values of 94.97, 94.44, and 94.39 for M = 1, 5, and 10.The corresponding IWAE values were 93.17, 92.89, and 92.85; larger sample sizes require more computation, making M = 1 a practical choice.
  • Inference architecture: With GRU inference networks, VAE and IWAE showed overfitting behavior, whereas denoising mitigated it; DIWAE (GRU) performed best among the tested models.These results were reported on MNIST with different noise levels.
  • Data augmentation: On MNIST, data augmentation achieved 93.88 ± 0.08 for VAE and 92.51 ± 0.07 for IWAE, outperforming DVAE for VAE but underperforming DIWAE for IWAE.The reported DVAE and DIWAE values were 94.32 ± 0.37 and 93.83 ± 0.06, respectively.
  • Corruption design: Using pixel-wise corruption rates based on a mean image produced no noticeable improvement over a global corruption rate.The experiments therefore used simple global corruption rates, while parameterized corruption distributions were left for future work.

5 CONCLUSIONS

The paper introduces a denoising variational lower bound for variational inference with corrupted inputs. It reports improved performance for VAE and IWAE, while identifying sensible corruption distributions as an unresolved issue.

  • 5 CONCLUSIONS: The denoising variational lower bound can be tighter than the standard variational lower bound on noisy inputs when the corruption function is sensible.The criterion is presented for variational models whose approximate posterior is conditioned on the input.
  • 5 CONCLUSIONS: The method supports more flexible and robust approximate posterior distributions, including mixture-of-Gaussian forms, than standard training without corruption.The paper frames this as a claimed capability of the denoising criterion.
  • 5 CONCLUSIONS: Experiments consistently found performance improvements for both variational autoencoders and importance weighted autoencoders.The reported experiments used simple corruption distributions.
  • 5 CONCLUSIONS: How to obtain a sensible corruption distribution remains an important open question for broader application of the method.The paper suggests learning a parameterized corruption distribution or developing better heuristics as future directions.

APPENDIX

The appendix develops inequalities underlying the denoising variational lower bound and shows how repeated applications of a lemma establish the theorem. It also relates the objective to expected posterior-approximation error for noised inputs.

  • APPENDIX: Lemma 0 establishes an expectation inequality for nonnegative measurable functions using Jensen’s inequality.The appendix concludes that E_f(x)[log g(x)] ≤ E_f(x)[log f(x)].
  • APPENDIX: Lemma 1 provides an inequality for approximate posteriors in directed latent-variable models factorized as pθ(x, z) = pθ(x|z)p(z).The inequality is then used to derive a lower bound on marginal log-likelihood.
  • APPENDIX: Theorem 1 is proved by applying Lemma 1 consecutively across layers 1 through L.The appendix explicitly repeats the application for layers 1 and 2, then for layers 3 to L.
  • APPENDIX: The proof temporarily fixes θ to simplify the analysis.This is stated as an assumption in the proof setup.
  • APPENDIX: The derivation concludes that maximizing L_dvae is equivalent to minimizing expected KL divergence between the true and approximate posterior distributions for each noised input.The appendix states this equivalence after establishing the relevant equalities.
Loading 1511.06406v2…