Source-linked AI summary
Learning to Generate Images with Perceptual Similarity Metrics
Jake Snell, Karl Ridgeway, Renjie Liao, Brett D. Roads, Michael C. Mozer, Richard S. Zemel
TL;DR
Pixel-wise image-synthesis losses can produce blurry images and fail to match human quality judgments. The paper trains deterministic and stochastic autoencoders with differentiable MS-SSIM and compares them with pixel-wise objectives. Across reconstruction, representation, classification, and super-resolution evaluations, perceptually optimized models receive stronger reported support, including reliable human preference for their reconstructions.
Problem
Pixel-wise losses used in image synthesis can yield blurry results and fail to capture human judgments of image quality.
Method
The paper trains deterministic and stochastic autoencoders with differentiable SSIM-family losses, especially MS-SSIM, and compares them with MSE and MAE objectives.
Results
Perceptually optimized networks outperform pixel-wise alternatives across reconstruction, representation, classification, and super-resolution evaluations, with SSIM preferred over MSE in most tested images.
Takeaways & Limitations
Perceptually grounded training objectives can be applied to image-synthesis and transformation architectures while producing representations judged more favorably for image quality.
Abstract
from arXiv · showhide
Deep networks are increasingly being applied to problems involving image synthesis, e.g., generating images from textual descriptions and reconstructing an input image from a compact representation. Supervised training of image-synthesis networks typically uses a pixel-wise loss (PL) to indicate the mismatch between a generated image and its corresponding target image. We propose instead to use a loss function that is better calibrated to human perceptual judgments of image quality: the multiscale structural-similarity score (MS-SSIM). Because MS-SSIM is differentiable, it is easily incorporated into gradient-descent learning. We compare the consequences of using MS-SSIM versus PL loss on training deterministic and stochastic autoencoders. For three different architectures, we collected human judgments of the quality of image reconstructions. Observers reliably prefer images synthesized by MS-SSIM-optimized models over those synthesized by PL-optimized models, for two distinct PL measures ($\ell_1$ and $\ell_2$ distances). We also explore the effect of training objective on image encoding and analyze conditions under which perceptually-optimized representations yield better performance on image classification. Finally, we demonstrate the superiority of perceptually-optimized networks for super-resolution imaging. Just as computer vision has advanced through the use of convolutional architectures that mimic the structure of the mammalian visual system, we argue that significant additional advances can be made in modeling images through the use of training objectives that are well aligned to characteristics of human perception.
1. Introduction
Image synthesis networks commonly use pixel-wise losses, but these losses can produce blurry reconstructions and may not reflect human judgments of quality. The paper instead studies perceptually grounded losses and reports benefits for reconstruction, representation learning, classification, and super-resolution.
- Image generation spans synthetic images, handwritten digits, and naturally cluttered scenes, making it a broad testbed for generative models.
- Autoencoders compress images into low-dimensional feature vectors and decode those vectors into reconstructed images.
- MSE and MAE are standard reconstruction losses, but both can yield blurry, low-pass-filtered synthesized images.
- The paper replaces conventional losses with perceptually grounded losses and evaluates them on image reconstruction and generation.
- Perceptually optimized representations are also evaluated for image classification and single-image super-resolution.
2. Background and Related Work
Image-synthesis systems rely on reconstruction or generation objectives that can be computationally difficult or poorly aligned with human perception. This paper develops an autoencoder-based approach using differentiable SSIM-family losses, including multiscale comparisons.
- Neural networks for image synthesis: Deterministic autoencoders reconstruct inputs directly, whereas probabilistic autoencoders model image distributions and support stochastic generation.
- Neural networks for image synthesis: Boltzmann-machine and deep-belief-network approaches require costly MCMC steps to approximate derivatives of an intractable partition function.
- Neural networks for image synthesis: GANs train coupled generator and discriminator networks, but their objectives are inter-dependent and lack a single common objective.
- Neural networks for image synthesis: Autoencoders are emphasized because their encoders produce compact representations that can transfer to other tasks.
- Perception-Based Error Metrics: Pixel-wise MSE, MAE, and likelihood measures can disagree with human quality judgments for contrast changes and impulse noise.
- Structural Similarity: SSIM compares luminance, contrast, and structure, while MS-SSIM extends the comparison across multiple image scales.The multiscale variant downsamples images iteratively and applies contrast and structure comparisons at every scale.
- Structural Similarity: SSIM-family losses are selected partly because their gradients are analytically simple and inexpensive to compute.
- Structural Similarity: The training objective minimizes a loss based on structural-similarity scores between original and reconstructed images.
3. Deterministic Autoencoders
The paper compares perceptual similarity losses with pixelwise losses for deterministic autoencoders, using human judgments across fully connected and convolutional architectures. Human observers generally preferred SSIM/MS-SSIM reconstructions, especially for images with fine or background detail.
- Experimental setup: The experiments compare SSIM or MS-SSIM against MSE and MAE across fully connected and convolutional autoencoders.The fully connected study uses SSIM versus MSE, while the convolutional study uses MS-SSIM versus both pixelwise losses.
- Human preferences: In the first study, a plurality of images had over 90% participant agreement preferring SSIM over MSE, with almost no majority preference for MSE.Participants judged 100 CIFAR-10 image triplets.
- Qualitative comparisons: SSIM reconstructions were unanimously preferred for eight images because they preserved object details lost in MSE reconstructions.The examples include images ordered by participant preference, with MSE, original, and SSIM reconstructions shown from left to right.
- Qualitative comparisons: For the least-preferred SSIM examples, a majority still preferred SSIM in seven of eight images, while MSE won in only one image by 60%.The SSIM reconstructions retained at least as much visible detail, and inconsistent ratings may indicate roughly equal quality.
- Human preferences: In the second study, SSIM was chosen for 86.25% of 2000 judged images, corresponding to nearly a 7:1 preference ratio over MSE.Individual participants preferred SSIM on between 63% and 99% of trials.
- Convolutional autoencoders: In convolutional reconstructions, MS-SSIM’s advantage appeared strongest for fine object and background detail, but was smaller for simpler, homogeneous, less-textured images.Even when MSE or MAE ranked best, MS-SSIM reconstructions reportedly had no obvious defects relative to the alternatives.
4. Probabilistic Autoencoders
The paper adapts variational autoencoders to arbitrary differentiable image losses through the Expected-Loss VAE, then evaluates perceptual versus pixelwise objectives for reconstruction and generation.
- EL-VAE formulation: The Expected-Loss VAE replaces the probabilistic decoder with a deterministic prediction and combines expected image loss with KL regularization.The objective supports arbitrary differentiable losses and weights reconstruction against regularization.
- EL-VAE training methodology: The constant C controls the trade-off between reconstruction emphasis and the KL regularizer in EL-VAE training.Increasing C emphasizes reconstructions but increases KL divergence and worsens generated samples.
- EL-VAE training methodology: Loss normalization and statistical model selection were used to address scaling differences and choose C for each image-specific objective.MSE, MAE, and MS-SSIM were normalized by expected training-set loss before selecting C.
- EL-VAE results: MS-SSIM was ranked highest in 992 of 1000 human judgments of EL-VAE reconstructions, outperforming MSE and MAE on this evaluation.The study used randomly selected STL-10 test images and latent posterior modes.
- EL-VAE results: Random samples from MS-SSIM-optimized EL-VAEs contained greater detail and structure than samples from MSE- or MAE-optimized models.The qualitative comparison sampled latent codes from the prior and decoded them into images.
5. Classification with Learned Representations
The paper tests whether perceptually aligned training objectives produce latent representations tied more closely to image factors of variation, using Yale B faces and supervised predictors.
- Motivation: The classification experiments evaluate whether perceptually optimized objectives yield representations more closely tied to image variation than pixelwise objectives.This section extends the paper’s reconstruction and generation comparisons to learned image encodings.
- Experimental setup: Experiments used the Extended Yale B Faces dataset, labeled by identity and illumination azimuth and elevation.The dataset contains 2,414 grayscale images of 38 individuals, resized to 48 × 48 pixels.
- Experimental setup: Deterministic convolutional autoencoders were trained with MSE, MAE, and MS-SSIM, then their bottleneck representations were evaluated for downstream prediction.The study examined whether the learned representations captured factors of variation in face images.
- Experimental setup: A 32-unit ReLU bottleneck was used, with linear-kernel SVMs predicting identity and RBF-kernel SVRs predicting illumination azimuth and elevation.Models were trained after the autoencoders converged, with hyperparameters selected using validation data.
6. Image Super-Resolution
The paper applies MS-SSIM to super-resolution through SRCNN, comparing it with MSE and MAE using standard image-quality metrics and visual inspection.
- Method: SRCNN was used as the baseline architecture for comparing MSE, MAE, and MS-SSIM losses in 4× super-resolution.The model consists of convolutional and fully connected ReLU layers.
- Method: Training used 5 million randomly cropped 33 × 33 patches from a subset of ImageNet.Evaluation used Set5, Set14, and BSD200, with measures computed on the Y channel of YCbCr color space.
- Evaluation: The alternatives were evaluated with PSNR and SSIM on three standard super-resolution test datasets.The comparison followed the SRCNN evaluation protocol.
- Results: Visual comparisons at 4× magnification show MS-SSIM improving resolution, removing ringing artifacts, and enhancing contrast.The cited examples include ringing in a bottom-row image and fabric contrast in a third-row image.
7. Discussion and Future Work
Perceptually grounded losses improve reconstruction quality, representation utility, and super-resolution across the reported experiments. The discussion also motivates extending perceptual objectives beyond differentiable SSIM-based losses.
- Findings: SSIM-optimized images receive higher human quality judgments than pixel-loss-optimized images across multiple neural network architectures.The comparison covers MSE and MAE pixel-wise objectives.
- Findings: Perceptually optimized representations are better suited for predicting content-related image attributes.
- Findings: Perceptual losses can be substituted into current state-of-the-art architectures in place of pixel losses such as MSE.The paper highlights this flexibility in its super-resolution results.
- Interpretation: MS-SSIM may encourage encoding relevant low- and mid-level image structure, potentially benefiting fine-grained classification.The paper presents fine-grained classification usefulness as a conjecture.
- Future Work: Future work could combine multiple perceptually grounded losses, including black-box quality measures learned through a forward model.Human preference judgments could further refine these losses.
A.1. Fully-Connected Autoencoders
The deterministic autoencoder experiments use strongly compressed fully connected and convolutional architectures for image reconstruction. Their differing input sizes, bottlenecks, and datasets support comparisons across network architectures.
- Fully-Connected Architecture: Fully connected autoencoders process 32 × 32 images through a 256-feature binary bottleneck.The encoder expands 1024 inputs to 8192 features before reducing dimensionality by factors of two.
- Fully-Connected Architecture: Binary bottleneck activations are constrained to −1 or +1 to enforce strong signal compression.
- Training Setup: Training uses CIFAR-10 for fully connected models and STL-10 for convolutional models, with validation sets determining when training stops.
- Convolutional Architecture: Convolutional autoencoders process 96 × 96 images through three stride-2 convolutional layers with 128, 256, and 512 filters.Spatial resolution decreases from 96 × 96 to 12 × 12.
- Convolutional Architecture: The convolutional experiments compare bottlenecks of 128 and 512 units but report only the 128-unit network because qualitative performance was similar.
A.3. Training Results
Each deterministic autoencoder performs best according to the metric used for its optimization. The fully connected results quantify this alignment, while the convolutional results are presented in Table 3.
- Deterministic Results: 69.71% of held-out images have better reconstruction MSE from the MSE-optimized fully connected network.
- Deterministic Results: 97.33% of held-out images have better reconstruction SSIM from the SSIM-optimized fully connected network.
- Metric Alignment: Each network performs best on its own training metric.Convolutional held-out reconstruction losses are reported in Table 3.
- EL-VAE Setup: EL-VAE encoders use three convolutional layers, a 128-unit bottleneck, and a Gaussian MLP approximate posterior with prior N(0, I).
- EL-VAE Setup: EL-VAEs are trained on the unlabeled STL-10 data until training-set performance stops improving, without regularization.
C. Classification Details
The classification experiments train deterministic convolutional autoencoders on Extended Yale B Faces and evaluate representations with support-vector methods. Different prediction tasks use different kernels and hyperparameter searches.
- Autoencoder Training: Classification experiments use deterministic convolutional autoencoders with a 32-unit bottleneck on Extended Yale B Faces.
- Evaluation: Azimuth and elevation prediction use RBF-kernel SVRs with C selected from {1, 10, 100, 1000}.
- Evaluation: Identity classification uses a linear-kernel SVM with C selected from 0.01 through 10000.
- Evaluation: The classifier hyperparameters are selected by three-fold cross-validation.
D. Image Super-Resolution Details
The super-resolution experiments train only on the luminance channel after converting images to YCbCr, using bicubic downsampling and specified SGD settings. MS-SSIM training uses five scales with a downsampling ratio of two at each scale.
- Super-resolution training and testing use only the Y channel after RGB images are converted to YCbCr.
- Bicubic downsampling generates the training and testing data, while bicubic upsampling restores chroma channels for RGB visualization.
- The models use SGD with momentum, a fixed learning rate of 1.0e−3, and momentum of 0.9.
- No weight decay is used because the model is simple and the authors observed no overfitting.
- MS-SSIM training uses 5 scales, downsampling the image by a ratio of 2 at each scale.