PaperlayerView exact revision on arXiv

Source-linked AI summary

Denoising Adversarial Autoencoders

Antonia Creswell, Anil Anthony Bharath

arXiv:1703.01220v4cs.CVcs.LGstat.ML

TL;DR

For DAAEs, matching only the corrupted-input posterior to the prior leaves direct prior-based sample generation problematic. The paper proposes denoising adversarial autoencoder variants and an iterative sampling approach; the models do not consistently outperform one another on generative-task log-likelihood.

  • Problem

    When a DAAE matches qφ(z|x~) rather than the integrated posterior to the prior, prior draws are unlikely to yield samples consistent with p(x).

    Source: A. Drawing Samples From Autoencoders · p. 5

  • Method

    The paper proposes DAAE and iDAAE variants that use adversarial training to match different posteriors to a prior, and an iterative Markov chain procedure for DAAE sampling.

    Source: §VII · p. 12

  • Results

    DAAE and iDAAE do not consistently outperform one another in log-likelihood of synthesized samples on generative tasks.

    Source: G. Trade-offs in Performance · p. 12

  • Takeaways & Limitations

    For more consistent DAAE samples, the proposed procedure iteratively decodes, corrupts, and encodes rather than sampling from the prior and decoding directly.

    Source: D. Sampling DAAEs and iDAAEs · p. 8

Abstract

from arXiv · show

Unsupervised learning is of growing interest because it unlocks the potential held in vast amounts of unlabelled data to learn useful representations for inference. Autoencoders, a form of generative model, may be trained by learning to reconstruct unlabelled input data from a latent representation space. More robust representations may be produced by an autoencoder if it learns to recover clean input samples from corrupted ones. Representations may be further improved by introducing regularisation during training to shape the distribution of the encoded data in latent space. We suggest denoising adversarial autoencoders, which combine denoising and regularisation, shaping the distribution of latent space using adversarial training. We introduce a novel analysis that shows how denoising may be incorporated into the training and sampling of adversarial autoencoders. Experiments are performed to assess the contributions that denoising makes to the learning of representations for classification and sample synthesis. Our results suggest that autoencoders trained using a denoising criterion achieve higher classification performance, and can synthesise samples that are more consistent with the input data than those trained without a corruption process.

I. BACKGROUND

Autoencoders learn an encoder–decoder route from observations through latent variables to reconstructed or generated data. Denoising and prior regularisation offer different ways to shape representations, but each approach carries constraints on sampling or analytic tractability.

A. Autoencoders

  • Autoencoders estimate a data model by encoding observations into auxiliary variables and training a decoder to reconstruct or generate data from them.The encoder forms training pairs of latent variables and observations; the decoder learns from those pairs and can be sampled to obtain a data distribution.Source: A. Autoencoders · p. 2

B. Denoising Autoencoders (DAEs)

  • Denoising autoencoders reconstruct clean inputs from corrupted ones and can learn more robust, classification-useful representations, but their latent distribution is unknown for novel-sample generation.The ground-truth latent for a corrupted input is unknown, as is the distribution of encoded samples.Source: B. Denoising Autoencoders (DAEs) · p. 2

C. Variational Autoencoders

  • VAEs match qφ(z|x) to a chosen prior through a variational lower bound, enabling prior-based sampling, but analytic KL divergence is available only for limited distribution choices.For example, analytic computation is possible when both the posterior and prior are Gaussian.Source: C. Variational Autoencoders · pp. 2–3

D. Denoising Variational Autoencoders

Denoising makes the VAE variational objective difficult to compute for some posterior families, while adversarial training offers an alternative way to match distributions. The paper reviews this approach through adversarial training and adversarial autoencoders.

D. Denoising Variational Autoencoders

  • When the integrated posterior is a Gaussian mixture, the denoising variational lower bound can become analytically intractable; the DVAE instead optimizes an objective using qφ(z|x~).On a test dataset, Im et al. report lower negative variational lower bounds for the DVAE than for a regular VAE.Source: D. Denoising Variational Autoencoders · p. 3
  • The DVAE matches qφ(z|x~), rather than the integrated posterior, to the prior, so generation requires difficult sampling from that integrated posterior.The cited DVAE work does not address this sampling problem.Source: D. Denoising Variational Autoencoders · p. 3

A. Adversarial Training

  • Adversarial training matches generated samples to a target distribution by training a generator and discriminator in competition, without requiring the Jensen–Shannon divergence to be calculated directly.For an optimal discriminator, optimizing the generator is equivalent to minimizing that divergence.Source: A. Adversarial Training · p. 3

B. Adversarial Autoencoders

  • An adversarial autoencoder uses qφ(z|x) as both the encoder and adversarial generator, matching encoded samples to a chosen prior; adversarial matching permits more complex posteriors than typical VAE choices.The discriminator distinguishes prior samples from encoded samples, and the paper explores extending this model with denoising.Source: B. Adversarial Autoencoders · pp. 3–4

III. DENOISING ADVERSARIAL AUTOENCODER

The paper extends adversarial autoencoders with denoising and defines two ways to match latent distributions to a prior. These choices trade training efficiency against the sampling process.

III. DENOISING ADVERSARIAL AUTOENCODER

  • Denoising adversarial autoencoders combine reconstruction from corrupted inputs with adversarial matching of auxiliary-variable distributions to a prior.Both proposed variants are trained to approximately maximize the denoising variational lower bound.Source: §III · p. 4
  • The iDAAE matches the integrated posterior q̃φ(z|x) to the prior, whereas the DAAE matches the corrupted-input posterior qφ(z|x~).The iDAAE uses Monte Carlo integration during training.Source: §III · p. 4

A. Construction

IV. SYNTHESISING NOVEL SAMPLES

Sampling denoising autoencoders differs from sampling models that match the full posterior to a prior. For DAAEs, matching only the corrupted-input posterior makes direct prior-based synthesis problematic.

IV. SYNTHESISING NOVEL SAMPLES

  • The section reviews autoencoder sampling, identifies a DAAE sampling problem, and proposes an approach based on Markov chain sampling.The paper builds on earlier work on iterative sampling.Source: §IV · p. 4

A. Drawing Samples From Autoencoders

  • Figure 2 contrasts the encodings matched to the normal prior on CelebA: DAAE matches corrupted-data encodings, while iDAAE matches data encodings.The figure labels the corresponding distributions qφ(z|x~) and qφ(z|x).Source: Figure 2 · p. 5
  • Prior-based sampling is straightforward for the iDAAE, but nontrivial for the DAAE because only qφ(z|x~), not the integrated posterior q̃φ(z|x), is matched to the prior.As a result, prior draws are unlikely to yield samples consistent with p(x) for the DAAE.Source: A. Drawing Samples From Autoencoders · p. 5

B. Proposed Method For Sampling DAAEs

The proposed sampling method starts from a random latent vector and uses a Markov chain to obtain the latent distribution needed to synthesize novel samples without drawing from training data. Under stated conditions, the chain converges to its unique stationary distribution.

B. Proposed Method For Sampling DAAEs

  • A corrupted-data-conditioned posterior requires iterative synthesis, unlike sampling with the full posterior conditioned on original data.The paper introduces a novel sampling approach for adversarial autoencoders to address this challenge.Source: B. Proposed Method For Sampling DAAEs · p. 5
  • The chain starts from a random latent-space vector rather than a training sample, supporting synthesis of novel data samples.The target latent distribution is P(z) = ∫qφ(z|x)p(x)dx.Source: B. Proposed Method For Sampling DAAEs · p. 5
  • Under the stated assumptions, the transition operator defines a Markov chain with unique stationary distribution P(z) = ∫qφ(z|x)p(x)dx.The theorem assumes pθ(x|z) ≈ P(x|z) and an optimal adversarial pair; ergodicity also requires additive Gaussian corruption.Source: B. Proposed Method For Sampling DAAEs · p. 5
  • Sampling uses P(z), not the prior p(z), to condition pθ(x|z) when drawing samples from pθ(x).The method is for sampling trained models; its Markov chain does not update model parameters.Source: B. Proposed Method For Sampling DAAEs · pp. 5–6

V. IMPLEMENTATION

The implementation uses an encoder, decoder, and corruption process for image distributions, with parameters learned through autoencoder and adversarial training. The autoencoder reconstructs clean images from corrupted inputs, while adversarial training matches encoded latent samples to a prior.

V. IMPLEMENTATION

  • The implementation maps corrupted image samples to latent space with Eφ and maps latent samples to image space with Rθ, using corruption process C.The analysis in the preceding sections does not depend on a specific implementation choice.Source: §V · p. 6

A. The Autoencoder

  • The encoder and decoder are fully connected networks, trained to minimize pixel-wise binary cross-entropy between reconstructed and original uncorrupted images.The decoder uses ReLU intermediate layers and a sigmoid output; the encoder has ReLU intermediate layers and an unrestricted linear output layer.Source: A. The Autoencoder · p. 6
  • Reconstruction error alone does not match encoded distributions to the prior, so encoder parameters also require adversarial updates.Encoder and decoder parameters are learned under the autoencoder framework; encoder parameters are additionally updated under the adversarial framework.Source: A. The Autoencoder · p. 6, §V · p. 6

B. Adversarial Training

  • Adversarial training matches fake latent samples to prior samples by training a discriminator to classify them as fake or real.For DAAE, fake samples are encoded from corrupted inputs; for iDAAE, Monte Carlo sampling approximates them, and encoder parameters are updated to fool the discriminator.Source: B. Adversarial Training · p. 6

C. Sampling

The section compares direct-prior sampling from iDAAEs with iterative Markov-chain sampling from DAAEs. Across the reported experiments, iDAAEs outperformed AAEs at M=5, while even one DAAE sampling iteration improved image synthesis.

C. Sampling

  • iDAAEs and AAEs generate samples by decoding prior-drawn latents directly, while DAAEs require iterative sampling.Matching the full posterior during training is less computationally efficient, but makes sampling straightforward; matching the corrupted-sample posterior avoids additional sampling during training.Source: C. Sampling · p. 6

B. Datasets

  • The evaluation spans Omniglot, Sprites, and CelebA, with Omniglot split to test reconstruction or classification on character classes unseen during training.Omniglot has 1,623 character categories and 20 examples per character; the evaluation set contains the 659 categories excluded from training.Source: B. Datasets · p. 7, 1) Datasets: Omniglot · p. 7, 2) Datasets: Sprites · p. 7

C. Architecture and Training

D. Sampling DAAEs and iDAAEs

  • DAAE synthesis starts from a random latent sample and repeatedly decodes, corrupts, and encodes it; the resulting latent state can be decoded as a sample.The authors report that nine iterations yield varied Omniglot samples capturing multiple modes of the data-generating distribution.Source: D. Sampling DAAEs and iDAAEs · p. 8, 1) Sampling: Omniglot · p. 8
  • On Omniglot, iDAAE outperformed AAE on the evaluation set, while one DAAE sampling iteration improved log-likelihood over AAE; additional iterations worsened evaluation results.Evaluation uses character classes not seen in training, whereas testing uses the same classes; on testing, the fifth sample had better log-likelihood than the first, so likelihood trends differed by split.Source: Figure 3 · p. 8, 1) Sampling: Omniglot · p. 8
  • 2122 ± 5 was the iDAAE log-likelihood on Sprites, higher than the AAE's 2085 ± 5.After one sampling iteration, DAAE samples reached 2261 ± 5, higher than samples from either the iDAAE or AAE.Source: 2) Sampling: Sprites · pp. 8–9
  • On CelebA, DAAE samples had good structure but less variation, whereas iDAAE samples had more variation but less good structure.DAAE samples became sharper and less blurry from x(0) to x(20); the authors suggest mode-directed sampling may explain their similarity.Source: 3) Sampling: CelebA · p. 9
  • With M = 5, iDAAEs outperformed AAEs on all datasets, and one Markov-chain iteration improved image synthesis for DAAEs in every case.Initial DAAE samples were poor; relatively small M is convenient because iDAAE training time may increase linearly with M.Source: 3) Sampling: CelebA · p. 9

E. Reconstruction

Reconstruction error is measured against original samples, and the denoising models do not outperform the AAE uniformly across datasets. On CelebA, iDAAE reconstructions are highly faithful but slightly blurred.

Reconstruction evaluation

  • Reconstruction is evaluated by mean squared error between the reconstruction and the original test sample.The reconstruction task passes test samples through the trained encoder and decoder.Source: E. Reconstruction · p. 9

Reconstruction motivation

  • Reconstructions can indicate whether an encoding preserves sample features and whether the model overfits test samples.The authors expect noise during training to prevent overfitting and encourage more robust representations.Source: E. Reconstruction · pp. 9–10

2) Reconstruction: Sprites

3) Reconstruction: CelebA

  • On CelebA, iDAAE outperforms AAE and DAAE, while DAAE performs similarly to AAE.The iDAAE uses corruption σ = 0.1 and M = 10 integration steps; its reconstructions are highly faithful but slightly blurred.Source: 3) Reconstruction: CelebA · p. 10, Figure 7 · p. 10

F. Classification

The section evaluates learned encodings with SVM classification on Omniglot and CelebA, and examines sensitivity to DAAE hyperparameters. Denoising variants outperform AAE on some classification comparisons, but their relative performance depends on the task.

1) Classification: Omniglot

2) Classification: CelebA

  • CelebA classification uses linear SVMs on encodings to predict 12 facial attributes, with fixed noise σ = 0.25 and M = 10 for iDAAE.The comparison includes previously reported VAE and β-VAE results.Source: 2) Classification: CelebA · p. 11

Hyperparameter robustness

  • DAAE performs better with σ = 0.1 or 0.25 than with σ = 1.0, though the larger noise level does not break the model.The reported settings suggest reasonable performance without fine-tuning, although a full parameter sweep is computationally expensive.Source: 2) Classification: CelebA · p. 11, Figure 10 · p. 12

2) Classification: CelebA

G. Trade-offs in Performance

DAAE and iDAAE show benefits on some generation and reconstruction tasks, but neither consistently leads the other on generative log-likelihood. Their practical trade-offs include training and sampling cost, and the paper proposes denoising models and an iterative sampling approach.

G. Trade-offs in Performance

  • iDAAE may take longer to train than DAAE, although its integration steps can be parallelized when resources permit.Once trained, both models take the same time to compute classification encodings; as few as M = 5 training steps can improve classification scores.Source: G. Trade-offs in Performance · p. 12
  • Neither DAAE nor iDAAE consistently outperforms the other on synthesized-sample log-likelihood.iDAAE with M = 5 training integration steps and DAAE with one sampling step each lead to better-quality generated images.Source: G. Trade-offs in Performance · p. 12
  • Conflicting test and evaluation log-likelihoods do not clearly indicate how DAAE sampling iterations affect visual quality.The authors say visual inspection may be necessary.Source: G. Trade-offs in Performance · p. 12

VII. CONCLUSION

  • The paper proposes DAAE and iDAAE, matching the posterior conditioned on corrupted or original samples, respectively, to a prior with adversarial training.iDAAE uses Monte Carlo integration during training.Source: §VII · p. 12
  • For DAAE synthesis, the paper proposes iterative sampling with a Markov chain transition operator defined by the encoder, decoder, and training corruption process.The reported experiments compare the models with AAE on three datasets and three tasks: reconstruction, classification, and sample synthesis.Source: §VII · p. 12

APPENDIX A PROOFS

The appendix states conditions for stationarity and ergodicity of the sampling chain, gives examples of synthesized samples, and includes fragments of the iDAAE training algorithm.

APPENDIX A PROOFS

  • P(z) is stationary for the sampling chain when the initial latent variable is drawn from P(z).The proof follows the decoder, corruption, and encoder sampling steps to show the next latent variable is also from P(z).Source: Appendix A · p. 14
  • The chain is asserted to be ergodic with additive Gaussian corruption and an optimal adversarial pair, subject to the proof's reachability conditions.The proof invokes overlapping corruption supports and assumptions about the decoder and matched latent distribution.Source: Appendix A · p. 14

APPENDIX B EXAMPLES OF SYTHESISED SAMPLES

  • Figure 11 presents examples of randomly synthesized data samples.The passage supplies no further characterization of the examples.Source: Figure 11 · p. 14

APPENDIX C ALGORITHM FOR TRAINING THE IDAAE

  • The iDAAE training-algorithm excerpts include drawing a training batch, minimizing reconstruction cost, training the discriminator, and training the decoder to match the prior.These are listed as separate steps in the supplied algorithm excerpts.Source: Appendix C · p. 14

APPENDIX D ALGORITHM FOR MONTE CARLO INTEGRATION

MNIST provides a balanced, simple proof-of-concept dataset, while the experiments also test a 10-Gaussian-mixture prior. The models use fully connected networks and Gaussian corruption.

A. Datasets: MNIST

  • MNIST has 50,000 training, 10,000 validation, and 10,000 test images, balanced across ten digit classes; its simplicity motivates testing more complex datasets too.Images are 28-by-28 grayscale pixels.Source: A. Datasets: MNIST · p. 15

B. Architecture and Training: MNIST

  • The MNIST models use two-layer, 1,000-neuron encoder, decoder, and discriminator networks; most use a 10-unit encoding and 10D Gaussian prior.Training uses 100 epochs, learning rate 0.0002, batch size 64, and Gaussian noise with standard deviation 0.5 for DAAE and iDAAE models.Source: B. Architecture and Training: MNIST · p. 15, Table III · p. 15
  • An additional DAAE uses a prior comprising ten equally spaced 2D Gaussians around a circle, designed to test matching a mixture prior in latent space.The authors describe this prior as unrealistic for MNIST because it separates digit modes despite expected smooth transitions between similar digits.Source: B. Architecture and Training: MNIST · p. 15

C. Reconstruction: MNIST

On MNIST, denoising-model encodings outperform AAE encodings for classification, while reconstruction and sampling outcomes vary with model and prior. The later dataset passages specify likelihood-estimation procedures for Omniglot and Sprites.

C. Reconstruction: MNIST

  • iDAAE with M = 5 achieves the best MNIST reconstruction; M = 25 worsens error, and mixture-prior DAAE underperforms with mean-like reconstructions.The AAE reconstruction comparison is less challenging because its inputs are not corrupted before encoding.Source: C. Reconstruction: MNIST · p. 15

D. Sampling: MNIST

  • With a 10D Gaussian prior, five MC steps make samples clearer but omit digit 4; DAAE MC sampling may also lower test log-likelihood.The log-likelihood passage attributes the decrease possibly to samples tending toward mean samples and dropping modes.Source: Figure 12 · p. 16, D. Sampling: MNIST · p. 16, Table V · p. 17
  • For the 2D, 10-Gaussian-mixture prior, nine MC steps dramatically improve initially non-digit samples, but several data modes appear missing.The authors suggest the prior's structure may contribute to the missing modes.Source: Figure 13 · p. 16, D. Sampling: MNIST · p. 16
  • DAAE and iDAAE encodings outperform AAE encodings for MNIST classification, and all three beat PCA; score differences are small on this relatively easy dataset.The classifiers use RBF-kernel SVMs on encoded training samples and are evaluated on encoded test samples.Source: 1) Classification: MNIST · pp. 16–17, Table VI · p. 17

A. Omniglot

  • Omniglot sample log-likelihood is estimated with a Parzen window fit to 1,000 synthesized samples, with bandwidth determined on the testing dataset.Evaluation uses both evaluation and test datasets; a new synthesized sample set is used for the test-set calculation.Source: A. Omniglot · p. 17

B. Sprites

  • Sprites sample log-likelihood uses a Parzen window fit to 1,000 synthesized samples; the reported optimal bandwidth is 1.29.Source: B. Sprites · p. 17

APPENDIX G IDAAE ROBUSTNESS TO HYPER PARAMETERS

  • Appendix G presents an iDAAE figure and a separate figure titled “Robustness to hyper parameters.”Source: Figure 14 · p. 17, Figure 15
Loading 1703.01220v4…