PaperlayerView exact revision on arXiv

Source-linked AI summary

The Perception-Distortion Tradeoff

Yochai Blau, Tomer Michaeli

arXiv:1711.06077v4cs.CV

TL;DR

Image restoration evaluation faces a conflict between distortion and perceptual quality, and the paper asks whether this conflict is fundamental rather than a flaw in particular metrics. It proves a perception-distortion bound for arbitrary distortion measures, characterizes perceptual quality distributionally, and shows that GANs can approach the bound. The resulting perception-distortion-plane methodology enables more informative comparisons of restoration algorithms, including super-resolution methods.

  • Problem

    Existing evaluations use distortion measures or perceptual scores, but perceptually better restoration can have worse distortion scores, raising whether this tradeoff is fundamental.

  • Method

    The paper defines a perception-distortion function, analyzes the deviation between reconstructed and natural-image distributions, and studies GANs as a way to approach the resulting bound.

  • Results

    The paper proves that distortion and perceptual quality are at odds for all distortion measures and identifies an unattainable region with a monotone boundary.

  • Takeaways & Limitations

    Evaluating restoration methods with paired distortion and no-reference perceptual metrics on the perception-distortion plane provides a more informative comparison than using distortion alone.

  • Takeaways & Limitations

    The bound used in one analysis is generally not tight, and in a scalar Gaussian example Dmax can be smaller than 2Dmin depending on noise level.

Abstract

from arXiv · show

Image restoration algorithms are typically evaluated by some distortion measure (e.g. PSNR, SSIM, IFC, VIF) or by human opinion scores that quantify perceived perceptual quality. In this paper, we prove mathematically that distortion and perceptual quality are at odds with each other. Specifically, we study the optimal probability for correctly discriminating the outputs of an image restoration algorithm from real images. We show that as the mean distortion decreases, this probability must increase (indicating worse perceptual quality). As opposed to the common belief, this result holds true for any distortion measure, and is not only a problem of the PSNR or SSIM criteria. We also show that generative-adversarial-nets (GANs) provide a principled way to approach the perception-distortion bound. This constitutes theoretical support to their observed success in low-level vision tasks. Based on our analysis, we propose a new methodology for evaluating image restoration methods, and use it to perform an extensive comparison between recent super-resolution algorithms.

1 INTRODUCTION

The paper proves a fundamental perception-distortion tradeoff: near an unattainable boundary, improving distortion necessarily sacrifices perceptual quality, and vice versa. It formalizes this tradeoff for all distortion measures, supports GANs as a principled approach to the bound, and proposes plane-based evaluation.

  • Motivation: Recent restoration methods can improve reconstruction accuracy without improving visual quality, with perceptually superior algorithms often scoring worse on PSNR or SSIM.This pattern is commonly attributed to shortcomings in existing distortion measures.
  • Core result: The paper proves that an unattainable region exists in the perception-distortion plane, regardless of the algorithmic scheme.Its monotone boundary means nearby algorithms can improve distortion or perceptual quality, but not both simultaneously.
  • Core result: The perception-distortion tradeoff holds for all distortion measures, not only MSE or SSIM.The analysis therefore provides a complementary explanation rather than attributing the phenomenon solely to particular criteria.
  • Formal framework: The perception-distortion function separates attainable from unattainable operating points by relating best perceptual quality to a maximal allowable average distortion.The paper follows the logic of rate-distortion theory and considers any estimator.
  • Implications: GANs provide a principled way to approach the perception-distortion bound, supporting their observed advantages in image restoration.The paper uses this framework to compare recent super-resolution methods on the perception-distortion plane.

2 DISTORTION AND PERCEPTUAL QUALITY

The paper distinguishes reference-based distortion from reference-independent perceptual quality and characterizes perceptual quality through natural-image statistics and real-versus-fake discrimination. It emphasizes ensemble-level distributional quality because single-image likelihood can reward outputs that ignore the input.

  • Distortion and perceptual quality: Full-reference distortion measures quantify a reconstructed image by its discrepancy from a ground-truth reference, whereas perceptual quality concerns resemblance to a natural image.The paper notes that similarity to a high-quality reference does not always imply high perceptual quality.
  • Perceptual quality: Perceptual quality is independent of reference-image similarity and is associated with deviations from natural image statistics.No-reference measures estimate this quality without depending on a reference image.
  • Human opinion based quality assessment: In real-versus-fake testing, optimal discrimination success is 2dTV(pX, p ˆ X) + 1/2, where dTV measures distributional deviation.The success probability falls to 1/2 when generated and natural-image distributions match.
  • Distributional quality: GAN-based restoration uses an adversarial loss to minimize a distance between generated-image and natural-image distributions.This distribution-matching approach has been applied across super-resolution, inpainting, compression, deblurring, and image-to-image translation.
  • Single image quality vs. image ensemble quality: Average log-likelihood can rate a reconstruction algorithm highly even when it always outputs the same natural-looking image and ignores its input.The paper therefore focuses on ensemble quality rather than quality of individual images alone.
  • Single image quality vs. image ensemble quality: The decomposition E[QLL( ˆX)] = −dKL(p ˆ X, pX) −H(p ˆ X) shows that the entropy term discourages output diversity.Dropping that term yields distributional-divergence-based quality measures.

3 PROBLEM FORMULATION

The section defines distortion as reconstruction error and perceptual quality as similarity between output and natural-image distributions, then analyzes their tradeoff in non-invertible restoration settings. Examples show that MMSE and MAP optimization can produce distribution shifts, while the theorem extends this limitation to arbitrary distortion measures.

  • 3 PROBLEM FORMULATION: Image restoration estimates x from degraded y through pY|X, while non-invertibility means x cannot be recovered with zero error.The estimator produces outputs according to p_Xhat|Y.
  • 3 PROBLEM FORMULATION: Distortion measures dissimilarity between x and x_hat, whereas perceptual quality measures how closely reconstructed outputs follow natural-image statistics.Perceptual quality can be represented by a divergence between p_Xhat and p_X.
  • 3 PROBLEM FORMULATION: Perfect perceptual quality requires p_Xhat = p_X, but outputs sampled independently from natural images can still have large distortion.Thus perceptual fidelity and reference-image accuracy are distinct objectives.
  • 3.3 Arbitrary distortion measures: A distortion measure is distribution preserving when its mean-distortion minimizer satisfies p_Xhat = p_X, but this property may hold only in a single setting.The paper motivates stability across noise levels and restoration tasks as the practically relevant requirement.
  • 3.1 The square-error distortion: MMSE estimates average possible explanations and can leave the natural-image manifold, producing blurry reconstructions unlike the binary MNIST image distribution.In the experiment, each 280 × 280 image contains 100 randomly selected 28 × 28 digit or blank images.
  • 3.2 The 0 −1 distortion: MAP minimizes binary 0 −1 distortion but can still alter the output distribution: when log(p1/p0) > 1/2, it never predicts 0.At higher noise levels, the image experiment reports MAP estimates increasingly dominated by blank images.
  • 3.3 Arbitrary distortion measures: In a linear Gaussian example, the minimal KL deviation decreases as allowable MSE increases, with a convex tradeoff that becomes more severe at higher noise.The plotted quantity is the minimal KL distance subject to a maximal MSE constraint.
  • 3.3 Arbitrary distortion measures: For non-invertible degradations, no distortion metric is stably distribution preserving.Theorem 1 states that no measure remains distribution preserving throughout a total-variation neighborhood of the data distribution.

4 THE PERCEPTION-DISTORTION TRADEOFF

The perception-distortion function characterizes the best perceptual quality attainable at a given distortion level, revealing a fundamental tradeoff for arbitrary distortion measures. For square-error distortion, perfect perceptual quality can be reached with at most twice the minimum distortion, equivalent to a 3 dB PSNR sacrifice.

  • P(D) gives the minimal distributional deviation from natural images attainable by an estimator under distortion constraint D.It separates attainable from unattainable regions in the perception-distortion plane.
  • Theorem 2 establishes a perception-distortion tradeoff for any distortion measure when the divergence is convex in its second argument.The resulting function is monotonically non-increasing and convex under the stated assumption.
  • The tradeoff is most severe at low distortion and high perceptual quality, so small accuracy gains in the low-distortion regime can sharply reduce the ability to fool discriminators.The same steepness implies that improving already-low perceptual quality can require a large distortion increase.
  • For square-error distortion, perfect perceptual quality is achievable with MSE 2Dmin, corresponding to no more than a 3 dB PSNR sacrifice.The construction draws the restoration output from the posterior distribution pX|Y.
  • Dmax can be substantially below the general 2Dmin bound in the scalar Gaussian example, so the theorem is not generally tight.

5 TRAVERSING THE TRADEOFF WITH A GAN

GAN training provides a tunable way to traverse the perception-distortion tradeoff by balancing distortion and adversarial losses. In denoising experiments, the resulting estimators approximately follow the theoretical tradeoff and span a range from blurry and accurate to sharp and inaccurate.

  • GAN-based traversal: The generator loss combines MSE distortion with an adversarial perceptual-quality loss, with λ controlling their tradeoff.The adversarial loss corresponds to a divergence between the generated and natural-image distributions.
  • GAN-based traversal: Varying λ in the generator loss produces estimators along the perception-distortion function.λ acts as a Lagrange multiplier, so changing it varies the allowable distortion D.
  • Denoising experiment: In WGAN denoising, the estimator curve is monotonically decreasing and transitions from blurry and accurate to sharp and inaccurate.The curve is adjacent to, though not coincident with, the analytic perception-distortion bound.
  • Denoising experiment: The WGAN curve connecting estimators is a good approximation to the theoretical perception-distortion tradeoff for λ ∈[0, 0.3].Distortion is measured by MSE and perceptual quality by Wasserstein distance between pX and p ˆ X.
  • Denoising experiment: The denoising WGAN matches the MAP estimator’s distortion with better perceptual quality and nearly matches the random-draw estimator’s perceptual quality at lower distortion.These comparisons place the WGAN estimator favorably against both deterministic and distribution-matching baselines.

6 PRACTICAL METHOD FOR EVALUATING ALGO-

The paper evaluates restoration algorithms on a perception-distortion plane because low distortion and high perceptual quality can be incompatible. It uses admissibility and paired NR/FR metrics to compare algorithms and identify those nearest the attainable boundary.

  • Evaluation methodology: The perception-distortion plane exposes an unattainable lower-left region where improving perceptual quality requires higher distortion nearby.This motivates evaluating algorithms jointly rather than with distortion or perceptual quality alone.
  • Admissibility: An algorithm is admissible when no algorithm in the group dominates it by achieving both better perceptual quality and less distortion.Algorithms that are not dominated lie closest to the perception-distortion bound.
  • Metric limitations: When NR metrics are inaccurate, perceptual quality should instead be measured with human opinion scores or a discriminator trained against natural images.The paper gives blind deblurring with large blurs as an example setting where NR metrics may fail.
  • Empirical tradeoff: Across the tested plots, NR and FR metrics are anti-correlated near the unattainable region, including IFC, VIF and VGG2,2.The tradeoff therefore extends beyond MSE and SSIM to measures considered more aligned with visual quality.
  • Empirical tradeoff: Among algorithms closest to the bound, increasing IFC distortion coincides with improving perceptual quality.This visual comparison illustrates the local anti-correlation near the boundary.
  • Metric behavior: Near the perception-distortion bound, FR metrics disagree with human judgments favoring SRGAN, whereas NR metrics successfully identify its perceptual quality.FR metrics can correlate with perceptual quality farther from the unattainable region.
  • Evaluation methodology: The proposed practical comparison pairs a no-reference perceptual metric with a full-reference distortion metric on the perception-distortion plane.For 16 4× super-resolution algorithms, the study uses NIQE for perception and metrics including RMSE, SSIM, MS-SSIM, IFC, VIF and VGG2,2 for distortion.

7 CONCLUSION

The paper concludes that distortion and perceptual quality fundamentally conflict, including for distortion measures often considered perceptual. It therefore recommends evaluating restoration methods with paired NR and FR metrics.

  • Conclusion: Lower distortion forces an algorithm’s output distribution to deviate further from natural-scene statistics.The paper presents this as the central counter-intuitive perception-distortion phenomenon.
  • Conclusion: The tradeoff appears empirically across popular distortion measures, including measures considered well-correlated with human perception.The conclusion states that no distortion measure alone is suitable for assessing image restoration methods.
  • Conclusion: A pair of NR and FR metrics places each algorithm on the perception-distortion plane for a more informative comparison.The proposed methodology accounts jointly for perceptual quality and distortion.
  • Theoretical setup: The theoretical analysis studies maximal discrimination success between real images and algorithm outputs under a 0.5 prior for the two hypotheses.This is formulated as a Bayesian hypothesis-testing problem using the MAP decision rule.
  • Theoretical setup: The discrete denoising examples show that MSE and 0−1 loss are not distribution preserving for estimating X from Y = X + N.The examples analyze MMSE and MAP estimators under Gaussian noise.

APPENDIX C PROOF OF THEOREM 1

The proof establishes that stable distribution preservation would force a unique optimal estimator matching the posterior, while non-invertible degradation yields multiple optimal estimators. This contradiction rules out stably distribution-preserving distortion metrics in that setting.

  • Definitions: A non-invertible degradation is defined by positive posterior probability across a non-singleton Sx for a set Sy with positive probability.This condition ensures multiple possible reconstructions remain compatible with observations in Sy.
  • Proof strategy: Under stable distribution preservation, the unique optimal estimator satisfies p ˆ X∗|Y = pX|Y.The proof shows the optimal conditional estimator depends only on the posterior pX|Y.
  • Non-uniqueness: When p ˆ X∗|Y = pX|Y is optimal under non-invertible degradation, any estimator supported on the posterior-minimizing set can also be optimal.Different supports within Sx yield distinct optimal conditional distributions.
  • Non-uniqueness: Two estimators with disjoint supports inside Sx are both optimal yet have positive total-variation distance, proving optimal-estimator non-uniqueness.The construction applies over every y in Sy.
  • Theorem conclusion: The unique-estimator result and the non-uniqueness result contradict each other under a non-invertible degradation.Therefore, no stably distribution-preserving distortion metric exists in that setting.

APPENDIX D DERIVATION OF EXAMPLE 1

The appendix derives the constrained optimization for Example 1 and proves that the perception-distortion function is non-increasing and convex. It also analyzes the scalar Gaussian estimator case through the MSE constraint and KL objective.

  • Scalar Gaussian example: The estimator distribution is Gaussian because ˆX=aY=a(X+N), enabling the KL distance between zero-mean normal distributions to define the objective.The appendix then derives the MSE between X and ˆX.
  • Scalar Gaussian example: For negative a, replacing it with a positive value preserves dKL while not increasing MSE, so analysis can restrict attention to a≥0.This follows from symmetry of dKL and the inequality MSE(|a|)≤MSE(a).
  • Scalar Gaussian example: The MSE constraint is empty below Dmin; for D ≥ Dmin, feasible values satisfy a−≤a≤a+.For D > Dmin, a+ increases monotonically with D, broadening the feasible set.
  • Scalar Gaussian example: For Dmin < D ≤ D0, the optimal parameter is the largest feasible value, a=a+(D), with D0 defined by a+(D0)=1/.The optimization is illustrated in Fig. 12.
  • Perception-distortion function: P(D) is non-increasing because its feasible constraint set does not shrink as D increases.The proof follows the rate-distortion theorem strategy.
  • Perception-distortion function: P(D) is convex because conditional mixture estimators preserve the required distributional structure and convexity of d(·,·) bounds the mixed objective.The proof uses pˆXλ|Y=λpˆX1|Y+(1−λ)pˆX2|Y and establishes the result for all λ∈[0,1].

APPENDIX F PROOF OF THEOREM 3

The appendix proves Theorem 3 by analyzing a distribution-preserving estimator constructed from conditional independence and identical conditional distributions. It shows that this estimator has MSE precisely twice that of the MMSE estimator.

  • Distribution-preserving estimator: The estimator ˆX of (14) attains perfect perceptual quality.The proof uses the fact that X and ˆX are conditionally independent and identically distributed given Y.
  • Distribution-preserving estimator: The distribution-preserving estimator has MSE precisely twice the MSE of the MMSE estimator.The result follows from the orthogonality principle together with the preceding conditional-distribution identities.

APPENDIX G WGAN ARCHITECTURE AND TRAINING DETAILS (SEC. 5)

The appendix specifies the WGAN denoising architecture and training setup, including a generator loss that combines MSE and adversarial terms. Training uses fixed epoch, batch-size, optimizer, and learning-rate schedules for each λ.

  • Training details: Each λ setting is trained for 35 epochs with batch size 64 using ADAM with β1=0.5 and β2=0.9.The generator and discriminator initial learning rates are 10−3 and 10−4, respectively.
  • Training details: Both learning rates are halved every 10 epochs during training.The discriminator uses 5×5 unpadded convolutional filters.

APPENDIX H SUPER-RESOLUTION EVALUATION DETAILS (SEC. 6) AND ADDITIONAL COMPARISONS

The appendix documents the super-resolution evaluation protocol and extends perception-distortion comparisons across multiple no-reference metrics, full-reference metrics, image formats, and algorithm sets. The reported pattern is consistent across these evaluation variations, with metric-specific artifacts affecting some methods.

  • Evaluation protocol: Low-resolution inputs are produced by factor-4 bicubic downsampling, and super-resolution results are evaluated using established NR and FR metrics plus a pretrained VGG-19 network.The appendix lists BRISQUE, BLIINDS-II, NIQE, SSIM, MS-SSIM, IFC, VIF, and the Ma et al. NR method.
  • Cross-metric pattern: The same qualitative pattern appears across the additional NR metrics and RGB evaluations as in the NIQE comparisons.This supports the appendix’s stated generalization across accurately predictive NR methods.
  • Metric sensitivity: Perceptual rankings for Johnson et al. and SRResNet-VGG2,2 vary across NR metrics, likely because of differing sensitivity to crosshatch artifacts.Johnson et al. is omitted from NIQE plots because its NIQE score is 13.55, far outside the plotted range.
  • Y-channel comparisons: Figures 14–16 plot 15 or 16 algorithms using the Ma et al., BRISQUE, or BLIINDS-II NR metrics against RMSE, SSIM, MS-SSIM, IFC, VIF, and VGG2,2 distortion measures.These evaluations use the y-channel alone.
  • RGB comparisons: Figures 17–18 repeat perception-distortion plots on three-channel RGB images using multiple NR metrics and RMSE, SSIM, and MS-SSIM.The RGB comparisons use 16 algorithms.
Loading 1711.06077v4…