Source-linked AI summary
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, Oliver Wang
TL;DR
Human-aligned image similarity remains difficult because visual similarity is subjective and existing metrics are limited. This paper introduces a large-scale human-judgment dataset and evaluates deep features across architectures and training signals, finding that they outperform classic metrics and share perceptual structure across representations.
Problem
Image similarity remains an open problem because visual patterns are high-dimensional and similarity judgments are subjective, while existing metrics may not align with human perception.
Method
The paper collects a varied dataset of human perceptual judgments and evaluates deep features across architectures, tasks, and supervision levels against classic metrics.
Results
Deep features outperform classic metrics across architectures and supervision levels, while randomly initialized networks achieve much lower performance.
Takeaways & Limitations
Perceptual similarity appears to be an emergent property shared by visual representations learned through supervised, self-supervised, and unsupervised objectives.
Abstract
from arXiv · showhide
While it is nearly effortless for humans to quickly assess the perceptual similarity between two images, the underlying processes are thought to be quite complex. Despite this, the most widely used perceptual metrics today, such as PSNR and SSIM, are simple, shallow functions, and fail to account for many nuances of human perception. Recently, the deep learning community has found that features of the VGG network trained on ImageNet classification has been remarkably useful as a training loss for image synthesis. But how perceptual are these so-called "perceptual losses"? What elements are critical for their success? To answer these questions, we introduce a new dataset of human perceptual similarity judgments. We systematically evaluate deep features across different architectures and tasks and compare them with classic metrics. We find that deep features outperform all previous metrics by large margins on our dataset. More surprisingly, this result is not restricted to ImageNet-trained VGG features, but holds across different deep architectures and levels of supervision (supervised, self-supervised, or even unsupervised). Our results suggest that perceptual similarity is an emergent property shared across deep visual representations.
1. Motivation
Perceptual similarity remains difficult because human judgments depend on high-order structure, context, and pairwise comparisons that pixel-based metrics fail to capture. This study introduces a large-scale human-judgment dataset and shows that deep visual representations, across architectures and training signals, align strongly with perceptual judgments.
- Motivation: Visual similarity remains a wide-open problem because visual patterns are high-dimensional, correlated, and subjectively judged.
- Motivation: Pixel-based measures such as ℓ2 distance and PSNR assume pixel-wise independence, so they can assign small changes to perceptually severe distortions such as blurring.
- Motivation: Human similarity judgments depend on high-order structure, context, and pairwise comparisons, making direct function fitting potentially intractable.
- Key findings: Deep network activations correspond to human perceptual judgments across architectures and outperform commonly used metrics on a newly collected large-scale database.
- Key findings: Self-supervised and unsupervised networks can perform comparably well, while random initialization performs much worse, indicating that training signal is crucial.
- Contributions: The study introduces a highly varied perceptual similarity dataset containing 484k human judgments, parameterized distortions, real algorithm outputs, and just noticeable difference judgments.
2. Berkeley-Adobe Perceptual Patch Similarity (BAPPS) Dataset
BAPPS is a large-scale, diverse dataset of human perceptual judgments built from 2AFC comparisons and validated with JND tests. It spans traditional, CNN-based, and real-algorithm distortions, using 64 × 64 patches to cover many distortion types and emphasize lower-level similarity.
- Judgment collection: BAPPS collects perceptual judgments primarily through 2AFC tests asking which of two distorted patches is closer to a reference, and validates them with JND judgments.In the 2AFC task, participants compare patches x0 and x1 against original patch x; people spent approximately 3 seconds per judgment.
- Distortion coverage: The dataset includes 20 traditional distortions and 308 sequentially composed distortions spanning noise, blur, shifts, corruptions, and compression artifacts.Traditional perturbation severity is parameterized, and pairs of distortions are composed sequentially to expand the space of possible distortions.
- Distortion coverage: CNN-based distortions simulate deep-learning artifacts by varying tasks, architectures, and losses, including autoencoding, denoising, colorization, and superresolution.These distortions are designed to mimic plausible outputs from deep-learning algorithms.
- Real algorithm outputs: The dataset also contains perceptual judgments for outputs from real algorithms in superresolution, frame interpolation, video deblurring, and colorization.Superresolution examples use ×2, ×3, and ×4 upsampling tracks, while other applications include outputs from multiple flow-, CNN-, phase-, deblurring-, and colorization-based methods.
3. Deep Feature Spaces
This section defines feature-space distances by normalizing and channel-scaling activations, then aggregating distances across spatial locations and layers. It evaluates multiple architectures and supervision settings, and introduces LPIPS variants that calibrate, fine-tune, or train networks using perceptual judgments.
- Network architectures: The study evaluates SqueezeNet, AlexNet, and VGG architectures, including five VGG convolutional layers and AlexNet conv1-conv5 layers.SqueezeNet was designed to be extremely lightweight at 2.8 MB, while AlexNet may more closely match the architecture of the human visual cortex.
- Network architectures: It additionally evaluates self-supervised representations based on puzzle-solving, cross-channel prediction, learning from video, and generative modeling.The evaluated networks are publicly available and include variants of AlexNet.
- Network activations to distance: Feature distances use unit-normalized activations from multiple layers, channel-wise scaling, ℓ2 distance, spatial averaging, and layer aggregation.Setting w_l = 1∀l is equivalent to computing cosine distance.
- Training on our data: LPIPS includes lin, tune, and scratch variants that respectively learn linear weights, fine-tune pretrained networks, or train networks from random Gaussian weights on perceptual judgments.For VGG, the lin configuration learns 1472 parameters.
4. Experiments
Experiments show that deep representations correlate well with perceptual judgments across supervision types and transfer to real-world image-processing tasks. Performance improves through perceptual calibration, while end-to-end tuning can reduce transfer performance.
- Evaluation protocol: Validation measures agreement with five human pairwise judgments per triplet, awarding partial credit according to the fraction of human preferences matched.For example, predicting a choice favored by four of five humans receives 80% credit.
- Representation learning: Self-supervised tasks perform on par with classification networks, indicating that many tasks can induce representations that transfer well to perceptual judgments.Evaluated tasks include generative modeling, puzzle solving, cross-channel prediction, and foreground-object segmentation from video.
- Representation learning: Network structure and data-oriented filters improve correlation with perceptual judgments, whereas randomly initialized Gaussian weights provide little improvement.The comparison contrasts deep networks with weights drawn from a Gaussian against learned representations.
- Metric behavior: Deep and traditional metrics disagree qualitatively: deep embeddings are more sensitive to blur, while BiGANs often treat correlated noise as a smaller distortion than SSIM.The comparison uses BiGANs and SSIM on traditional distortions.
- Transfer to real-world tasks: Linear calibration improves performance across SqueezeNet, AlexNet, and VGG on real-algorithm tasks, with 11 of 12 scores improving.The starting performances are 64.0%, 65.0%, and 62.6%, respectively; tuning throughout instead lowers transfer performance.
5. Conclusions
The results indicate that networks trained on challenging visual prediction and modeling tasks learn world representations that correlate well with perceptual judgments. This conclusion aligns with findings that self-supervised and unsupervised representations support semantic tasks and that computer-vision representations model macaque neural activity.
- Networks trained on challenging visual prediction and modeling tasks learn representations that correlate well with perceptual judgments.
- Self-supervised and unsupervised objectives similarly produce representations effective for semantic tasks.
- Computer-vision representations trained on visual tasks also become effective models of neural activity in macaque visual systems.
Appendix
The appendix provides full quantitative details, training details, and results on the TID2013 dataset in separate sections.
- Appendix A: Appendix A presents the full quantitative details.
- Appendix B: Appendix B discusses the training details.
- Appendix C: Appendix C reports results on the TID2013 dataset.
A. Quantitative Results
Quantitative evaluation spans low-level metrics and networks with random, unsupervised, self-supervised, supervised, and perceptually learned representations across multiple validation sets. Learned linear calibration reaches state-of-the-art on real algorithm tests, while tuned models on defined distortions approach human performance.
- A. Quantitative Results: Figures 7–9 plot performance separately for traditional and CNN-based distortions and real algorithm applications.The real algorithm applications include superresolution, frame interpolation, video deblurring, and colorization.
- A. Quantitative Results: Learned linear weights on the Alex model achieve state-of-the-art results on the real algorithms test set.The calibrated models learn a linear layer on each channel, unlike out-of-the-box versions that weight channels equally.
- A. Quantitative Results: Table 5 reports 2AFC scores across methods and test sets, with higher scores indicating better performance.The evaluation includes low-level metrics and networks with random, unsupervised, self-supervised, supervised, and perceptually learned representations.
- A. Quantitative Results: 80.6% and 81.4% for Alex – tune and VGG – tune, respectively, are nearly equal to human performance of 82.6% on the defined distortions.The passage attributes this result to a training set of 150k patch pairs and 300k judgments being nearly large enough to explore the traditional and CNN-based distortions.
B. Model Training Details
The model maps two distances to a perceptual similarity score using a small learned network, trained with constrained linear weights and a staged learning-rate schedule. Preliminary experiments found this learned approach outperformed a fixed-margin ranking loss.
- Network and loss: The scoring network G takes two distances, (d0, d1), and predicts a score ĥ ∈ (0, 1) using two 32-channel FC-ReLU layers, a 1-channel FC layer, and a sigmoid.The final loss function is specified in Equation 2.
- Network and loss: A learned network worked better than a ranking loss that enforced a constant margin between patch-pair distances.The comparison was made in preliminary experiments.
- Optimization: Training used 5 epochs at learning rate 10^-4, 5 epochs with linear decay, batch size 50, and two judgments per training patch pair.When the two judgments were split, the classification target h was set to 0.5.
- Optimization: The linear-layer weights were constrained to be non-negative by projecting negative weights to 0 at every iteration.This prevents larger distances in a feature from making two patches closer in the learned distance metric.
C. TID2013 Dataset
On the TID2013 dataset, deep classification networks outperform traditional perceptual metrics on the paper’s perturbations. AlexNet remains near the highest traditional metric even when scores are averaged across multiple image scales and layers without calibration.
- C. TID2013 Dataset: The evaluation tests TID2013 images at five resolutions, using smaller dimensions of 128, 192, 256, 384, and 512.Scores are computed across these resolution settings.
- C. TID2013 Dataset: 73.3% is achieved by one tested deep classification network on the traditional perturbations, versus 61.4% for FSIMc and 59.9% for ℓ2.FSIMc is described as close to ℓ2, while the tested deep classification networks score higher.
- C. TID2013 Dataset: AlexNet produces scores near FSIMc, the highest metric, even when averaging across all tested scales and layers without further calibration.The comparison uses the AlexNet architecture and the FSIMc metric.
D. Changelog
The paper’s revisions progressed from an initial preprint to a CVPR camera-ready version that reorganized results and training details into the Appendix, clarified key distinctions, and added linear weights for two architectures.
- The v1 release was the initial preprint version.
- The v2 CVPR camera-ready moved TID results, SSIM-versus-BiGAN results, and some training details into the Appendix to meet the 8-page limit.
- The revision clarified that SSIM was not designed for geometric distortions and that the dataset is for perceptual similarity rather than IQA.
- The revision added linear weights for the Squeeze-lin and VGG-lin architectures in Figure 10, along with miscellaneous text edits.