Source-linked AI summary
Generating Images with Perceptual Similarity Metrics based on Deep Networks
Alexey Dosovitskiy, Thomas Brox
TL;DR
Image-generation models trained with image-space losses often produce over-smoothed results, motivating a more perceptually faithful objective. The paper introduces DeePSiM, which measures feature-space distances and combines them with an adversarial natural-image prior. Across autoencoding, variational autoencoding, and feature inversion, the method produces sharper, more natural-looking outputs and preserves perceptually important detail.
Problem
Image-space losses often yield over-smoothed generated images, especially when feature representations leave fine-detail locations uncertain.
Method
DeePSiM uses weighted feature, adversarial, and pixel-space losses to measure perceptual similarity while encouraging realistic generated images.
Results
Across autoencoding, variational autoencoding, and AlexNet inversion, DeePSiM preserves fine structures, produces realistic image statistics, and substantially improves high-level reconstructions.
Takeaways & Limitations
Feature-space losses can reconstruct perceptually important details from low-dimensional representations and outperform typical image-space losses across three image-generation tasks.
Takeaways & Limitations
The paper does not solve sampling multiple distinct pre-images for one feature vector because training data provide only one image per conditioning vector.
Abstract
from arXiv · showhide
Image-generating machine learning models are typically trained with loss functions based on distance in the image space. This often leads to over-smoothed results. We propose a class of loss functions, which we call deep perceptual similarity metrics (DeePSiM), that mitigate this problem. Instead of computing distances in the image space, we compute distances between image features extracted by deep neural networks. This metric better reflects perceptually similarity of images and thus leads to better results. We show three applications: autoencoder training, a modification of a variational autoencoder, and inversion of deep convolutional networks. In all cases, the generated images look sharp and resemble natural images.
1. Introduction
The paper argues that image-space losses often blur generated images because they penalize uncertain detail locations, whereas deep feature distances better capture perceptual similarity. DeePSiM combines feature similarity with a natural-image prior and improves image generation across three applications.
- Motivation: Image-space squared Euclidean loss often produces blurry reconstructions when fine-detail locations are uncertain.Feature representations may not preserve the precise locations of all details.
- Motivation: Deep convolutional features can capture perceptual similarity because they tolerate small deformations while preserving sensitivity to edges and textures.The distribution of fine details matters more perceptually than their exact locations.
- Method: DeePSiM combines feature-space similarity with adversarial training to favor realistic outputs among possible reconstructions.Feature distance alone is insufficient because contractive representations map many natural and non-natural images to the same vector.
- Applications: The method is applied to autoencoding, variational autoencoding, and inversion of AlexNet representations.These applications test image compression, generative modeling, and reconstruction from high-level activations.
- Results: DeePSiM preserves fine structures in compressed images, produces realistic image statistics, and substantially improves reconstructions from high-level AlexNet activations.The reconstructions indicate that predicted class probabilities retain texture, color, and position information.
2. Related work
The related-work review situates DeePSiM among neural image-generation models, perceptual image-similarity metrics, and adversarial methods. It emphasizes concurrent work with a similar feature-space and adversarial idea but a narrower scope.
- Neural image generation: Neural image-generation research includes restricted Boltzmann machines, deep variants, autoencoders, and stochastic neural networks.These models have been used for unsupervised learning and generative modeling.
- Perceptual similarity: Prior perceptual-similarity measures include the visible differences predictor, spatio-temporal quality models, perceptual distortion metrics, and SSIM.SSIM compares local statistics of image patches.
- Adversarial generation: GANs provide adversarial image-generation training but can be difficult to optimize, with oscillation, divergence, or partial distribution coverage reported in practice.Several modifications address training stability, including multi-scale generation and convolutional-deconvolutional architectures with batch normalization.
- Adversarial generation: Conditional GANs generate images conditioned on variables such as object classes or previous video frames.The paper distinguishes its approach from conditional GANs despite a similar-looking conditioning structure.
- Closest related work: Larsen et al. concurrently combined feature-space similarity and adversarial training, but applied it only to a face-image variational autoencoder using discriminator features.The paper presents DeePSiM as broader across image types and applications.
3. Model
The model trains a generator using perceptual feature distances, adversarial realism constraints, and optionally image-space loss. Its architecture combines generator, discriminator, and comparator networks, with comparator choice and adversarial training providing the main design flexibility.
- Loss design: DeePSiM combines feature, adversarial, and pixel-space losses as a weighted objective for supervised image generation.The model is defined for input-target pairs and image targets, but the framework allows arbitrary vector inputs and outputs.
- Architecture: The generator produces outputs, the discriminator distinguishes generated from natural images, and the comparator computes image features.These three convolutional networks implement the core model architecture.
- Loss design: Feature loss compares comparator representations rather than pixels, while feature-only training can produce artifacts because many non-natural images share a feature vector.A natural-image prior is therefore needed to constrain generated images to the natural-image manifold.
- Loss design: Adversarial training learns a natural-image prior by training the discriminator against generated and real images while training the generator to fool it.The approach avoids manually designing the prior, although adversarial training is unstable and hyperparameter-sensitive.
- Architectures: The networks use up-convolutional generators and leaky ReLU nonlinearities, while AlexNet inversion additionally feeds features to the discriminator.Training uses Adam, batch size 64, and 500,000–1,000,000 mini-batch iterations.
- Architectures: Comparators include pretrained AlexNet, VideoNet, randomly initialized AlexNet, and Exemplar-CNN, with exact comparison layers specified per experiment.Comparator networks may be fixed or trained and can be part of the generator or discriminator.
4. Experiments
The experiments evaluate DeePSiM across autoencoding, variational generation, and AlexNet inversion. Compared with image-space losses, DeePSiM preserves perceptually important details and produces more realistic reconstructions, while comparator choice and adversarial training affect outcomes.
- The experiments cover autoencoder training, variational autoencoding, and inversion of AlexNet representations.
- Autoencoder: DeePSiM autoencoders preserve more texture details and produce naturally looking, non-blurry reconstructions despite higher Euclidean loss.AlexNet comparison can corrupt fine details, whereas Exemplar-CNN may fail to preserve exact color because of comparator invariance.
- Autoencoder: DeePSiM-learned autoencoder features achieve significantly better classification accuracy, indicating more semantically meaningful information.
- Variational autoencoder: VAE models trained with DeePSiM produce images with realistic statistics, whereas the usual squared-Euclidean loss produces very blurry samples.VideoNet features yield qualitatively similar samples to AlexNet features, suggesting supervised comparator training may not be necessary.
- Inverting AlexNet: DeePSiM reconstructs high-level AlexNet features well, with CONV5 reconstructions near-perfect and fully connected reconstructions preserving colors, positions, and major image features.The method performs fairly well in both image-space and feature-space Euclidean error.
- Inverting AlexNet: Iterative re-encoding changes reconstructions little, indicating that important perceptual features are preserved, while removing image-space or adversarial loss introduces distinct problems.Image-space loss causes over-smoothing; omitting adversarial loss produces noisy or unstable results and border artifacts, while sampling multiple pre-images was not addressed.
5. Conclusion
The paper proposes image-generation losses based on feature-space distances and finds them superior to typical image-space losses across three tasks. The authors also identify feature selection and realism control as open directions.
- Feature-space losses outperform typical image-space losses across auto-encoding, VAE generation, and feature inversion.The paper reports perceptually important detail reconstruction even from very low-dimensional representations.
- The paper evaluates several feature spaces and identifies finding task-appropriate optimal features as an area requiring more research.
- Figure 11 illustrates image interpolation by interpolating between features in AlexNet’s FC6 and FC8 layers.
Appendix
Appendix results examine feature-space interpolation, preservation of position and color, VAE samples, and iterative reconstruction across AlexNet layers. They show qualitative behavior of the proposed approach and contrast it with image-space-loss reconstructions.
- Figure 12 examines preservation of object position and color in deep AlexNet layers.
- Figure 13 presents generated images from interpolations between features of natural images across CONV5, FC6, FC7, and FC8.The figure compares interpolation behavior at different AlexNet layers.
- VAE samples using VideoNet loss are qualitatively similar to other samples, indicating AlexNet initialization is unnecessary.
- Iterative encoding and reconstruction show that image-space-loss training fails to preserve features, causing reconstructions to quickly diverge from the original image.