Source-linked AI summary
Autoencoding beyond pixels using a learned similarity metric
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, Ole Winther
TL;DR
Generative models struggle with complex image distributions because element-wise similarity measures are poorly suited to visual data. The paper combines a VAE with a GAN, using discriminator features as a learned reconstruction metric, and reports improved visual fidelity alongside disentangled latent representations for face attributes.
Problem
Element-wise similarity measures are inadequate for complex image data because they do not capture properties of human visual perception or relevant invariances.
Method
The method combines VAEs and GANs, replacing the VAE's element-wise reconstruction error with an error based on learned hidden representations from the GAN discriminator.
Results
The method produces better image samples than models using element-wise errors and learns latent representations in which visual attributes can be modified through simple arithmetic.
Takeaways & Limitations
Learned similarity measures are presented as a promising step toward scaling generative models to more complex data distributions.
Takeaways & Limitations
Evaluation focuses on qualitative visual assessments of 64x64 images because traditional log-likelihood measures do not capture visual fidelity, and the learned similarity measure can produce noisy patterns in some cases.
Abstract
from arXiv · showhide
We present an autoencoder that leverages learned representations to better measure similarities in data space. By combining a variational autoencoder with a generative adversarial network we can use learned feature representations in the GAN discriminator as basis for the VAE reconstruction objective. Thereby, we replace element-wise errors with feature-wise errors to better capture the data distribution while offering invariance towards e.g. translation. We apply our method to images of faces and show that it outperforms VAEs with element-wise similarity measures in terms of visual fidelity. Moreover, we show that the method learns an embedding in which high-level abstract visual features (e.g. wearing glasses) can be modified using simple arithmetic.
1. Introduction
The paper argues that element-wise similarity metrics hinder generative modeling of complex data, especially images, and proposes learning a more suitable similarity measure. It combines VAEs and GANs so learned feature representations guide reconstruction, improving image samples and supporting disentangled latent attributes.
- The paper proposes an unsupervised scheme that learns and applies a similarity measure capable of capturing high-level structure relevant to the data distribution.This replaces hand-engineered or element-wise measures with a learned model.
- Element-wise metrics such as squared error poorly model human visual perception and can penalize small image translations that people barely notice.The paper motivates higher-level, sufficiently invariant image representations instead of pixel-wise similarity.
- VAE/GAN combines a VAE and GAN into an unsupervised generative model that simultaneously learns to encode, generate, and compare dataset samples.The approach uses the GAN discriminator's learned representations to define a more meaningful reconstruction signal for the VAE.
- Learned similarity measures produce better image samples than models trained with element-wise error measures.The paper evaluates this claim on image generation, emphasizing visual fidelity rather than only pixel-level reconstruction.
- Unsupervised training yields a latent image representation with disentangled factors of variation, allowing simple latent-space arithmetic to modify visual attributes.Face-image experiments show that arithmetic on the learned representation produces changes corresponding to labeled visual attributes.
2. Autoencoding with learned similarity
The VAE/GAN combines a VAE encoder with a GAN discriminator to replace pixel-wise reconstruction error with a learned feature-wise metric. This hybrid shares the decoder and discriminator-based signals while restricting loss updates to appropriate networks.
- Core approach: The VAE/GAN combines VAE encoding with GAN discrimination to obtain a more meaningful feature-wise reconstruction metric.The method transfers image representations learned by the discriminator into the VAE reconstruction objective while retaining the VAE encoder and GAN’s generative modeling strengths.
- Learned similarity: The method replaces the VAE’s element-wise reconstruction error with an error between hidden discriminator representations of real and reconstructed samples.For discriminator layer l, the observation model treats Dis_l(x) as Gaussian with mean Dis_l(˜x) and identity covariance.
- Training objective: Training uses a triple criterion combining prior regularization, discriminator-based reconstruction likelihood, and the GAN objective.The combined model is trained with L_prior, l_like, and L_GAN, while decoder parameters receive both reconstruction and adversarial signals.
- Model sharing: The decoder is shared with the GAN generator, mapping latent variables to data space for both reconstruction and adversarial generation.Because Dec and Gen perform the same latent-to-data mapping, the method uses Dec in the GAN objective instead of a separate generator.
- Practical considerations: The training procedure limits error signals to relevant networks because applying every component of the combined loss to every network can collapse the discriminator.The authors also report better results when the GAN error is not backpropagated to the encoder.
- Adversarial training: The GAN objective uses generated samples from both the prior p(z) and the encoder posterior q(z|x) to provide a more useful learning signal.For a given x, Dec(Enc(x)) is expected to resemble x more closely than Dec(z), making the negative sample more similar to the positive sample.
3. Related work
Related work addresses shortcomings of element-wise image distances through preprocessing, learned metrics, alternative reconstruction objectives, and adversarial generation. The VAE/GAN builds on deep feature representations and GANs’ avoidance of element-wise similarity measures.
- Metric engineering: Image preprocessing improves robustness to perturbations by engineering alternatives to simple element-wise distance measures.Examples include contrast normalization, gradient images, and pixel statistics gathered in histograms.
- Learned metrics: Supervised Siamese networks learn distances by minimizing similar-sample distances and maximizing dissimilar-sample distances with a max-margin cost.Because Siamese networks require supervised training, the authors say they cannot apply them directly to their unsupervised problem.
- Generative models: Prior generative-model work replaces or supplements element-wise errors with structural similarity, additional shape and edge outputs, GAN sharpening, or gradient-based measures.These approaches target improved image structure, sharpness, or reconstruction quality across grayscale images, videos, and other generative settings.
- GANs: GANs avoid element-wise similarity measures by generating samples indirectly and optimizing them to imitate the dataset according to the discriminator.This is presented as a likely explanation for their ability to produce high-quality images.
- Image embeddings: Image embeddings have been studied for expressing semantic relationships through simple arithmetic, including supervised generation of chairs from high-level information.This line of work motivates examining whether learned image representations capture structured visual factors.
- Deep features: Neural artistic style transfer motivates the VAE/GAN’s use of deep convolutional features, viewing discriminator reconstruction error as content and the GAN error signal as style.Unlike the style network’s feature correlations, the VAE/GAN style term comes from trying to fool the discriminator.
4. Experiments
Experiments on face images compare VAEs, a VAE with a learned discriminator distance, VAE/GAN, and GAN models, including generation, reconstruction, and attribute-conditioned synthesis. VAE/GAN produces sharper, more natural samples than VAEs, supports semantic latent-attribute manipulation, and improves attribute recognition over an ordinary VAE, while learned-distance and semi-supervised settings expose limitations.
- Experimental setup: The comparison included a plain VAE, a VAE with a pretrained GAN discriminator as learned distance, VAE/GAN, and GAN.The learned distance was measured at a discriminator layer after three successive 2× downsamplings.
- Experimental setup: 64×64 CelebA face images with 40 binary attributes were used for unsupervised training, using aligned and cropped images without attributes.The dataset contains 202,599 images.
- Face generation and reconstruction: VAE/GAN and pure GAN generated sharper images with more natural textures and face parts than the plain VAE, while VAEDisl produced noisy artifacts despite sharper off-center images.The plain VAE became blurry off-center, whereas VAEDisl’s artifacts were attributed to harsh downsampling.
- Face generation and reconstruction: VAE/GAN reconstructed test-set images similarly to random samples, whereas GAN reconstruction was unavailable because GAN lacks an encoder.VAEDisl generated noisy blue patterns in some reconstructions, possibly because its GAN-based similarity measure collapsed to zero.
- Attribute manipulation: Adding visual attribute vectors to latent representations produced reconstructions reflecting semantic concepts such as eyeglasses and bangs, though correlated attributes caused imperfect changes.Attribute vectors were computed as the difference between mean latent vectors for images with and without each binary attribute.
- Attribute similarity: Conditioned VAE/GAN samples yielded significantly better visual attributes and smaller recognition error than an ordinary VAE, while GAN performed poorly, likely because of training instability.Results were not directly comparable with Yan et al. because preprocessing and regression models differed.
- Scope and limitations: Semi-supervised VAE/GAN experiments on CIFAR-10 and STL-10 did not reach state-of-the-art results, with high intra-class variation proposed as a possible cause.The setup used unsupervised pretraining followed by fine-tuning with a small number of labeled examples.
5. Discussion
The method is a VAE–GAN hybrid that uses a learned similarity measure and achieves high visual fidelity, while also exposing limitations in the choice of discriminator features.
- The learned distance measure enables an image encoder-decoder network to generate images with unprecedented visual fidelity in the experiments.
- The learned representation can disentangle variation factors and reveal high-level visual attributes in latent space.The authors suggest combining many unlabeled images with a small labeled set to discover latent features.
- The model combines VAE and GAN training, making it a hybrid rather than a straightforward extension of either framework.The decoder is trained both as a VAE decoder and as a GAN generator; alternatively, the method can be viewed as a GAN with an additional network constraining p(z).
- The discriminator may provide useful features despite being trained to distinguish generated from real samples rather than to measure similarity directly.The authors report that convolutional features were good enough in this setting to improve on element-wise image distances.
- The method's image quality depends on discriminator features whose suitability as a similarity measure is not obvious.The discussion proposes testing better features, including representations from face-trained Siamese or pretrained feedforward networks.
- The results show visual fidelity competitive with GANs, which the paper characterizes as state of the art in that regard.