PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff

Yochai Blau, Tomer Michaeli

arXiv:1901.07821v4cs.LGcs.CVcs.ITstat.ML

TL;DR

Lossy compression traditionally optimizes distortion at a fixed rate, although distortion and perceptual quality need not agree. The paper formalizes rate, distortion, and perception jointly using a distribution-based perceptual definition, then proves and illustrates their tradeoff. Its main result is that enforcing better perceptual quality generally elevates the rate-distortion curve, requiring a sacrifice in rate or distortion.

  • Problem

    Minimizing distortion alone does not necessarily produce good perceptual quality, motivating a theory that also accounts for perception.

  • Method

    The paper defines R(D, P) using Blau and Michaeli’s perceptual-quality measure, analyzes its general properties, derives a Bernoulli-source closed form, and evaluates the tradeoff on toy MNIST compression.

  • Results

    Enforcing higher perceptual quality elevates the rate-distortion function, so achieving good perception requires sacrificing either rate or distortion.

  • Takeaways & Limitations

    Evaluating compression methods only by rate-distortion curves can be misleading; evaluation should also include a no-reference perceptual-quality measure.

  • Takeaways & Limitations

    Closed-form solutions for the rate-distortion-perception function are harder to obtain than classical rate-distortion solutions, with the binary-source case described as a limited-applicability exception.

Abstract

from arXiv · show

Lossy compression algorithms are typically designed and analyzed through the lens of Shannon's rate-distortion theory, where the goal is to achieve the lowest possible distortion (e.g., low MSE or high SSIM) at any given bit rate. However, in recent years, it has become increasingly accepted that "low distortion" is not a synonym for "high perceptual quality", and in fact optimization of one often comes at the expense of the other. In light of this understanding, it is natural to seek for a generalization of rate-distortion theory which takes perceptual quality into account. In this paper, we adopt the mathematical definition of perceptual quality recently proposed by Blau & Michaeli (2018), and use it to study the three-way tradeoff between rate, distortion, and perception. We show that restricting the perceptual quality to be high, generally leads to an elevation of the rate-distortion curve, thus necessitating a sacrifice in either rate or distortion. We prove several fundamental properties of this triple-tradeoff, calculate it in closed form for a Bernoulli source, and illustrate it visually on a toy MNIST example.

1. Introduction

The paper argues that rate-distortion analysis must incorporate perceptual quality because minimizing distortion alone can produce poor perceptual quality. It formulates and analyzes a rate-distortion-perception tradeoff, showing that improved perception generally raises the rate-distortion curve.

  • Motivation: Minimizing distortion alone does not necessarily produce good perceptual quality, motivating a generalized rate-distortion theory.Prior work found that perceptually improved reconstructions can incur increased distortion, while distortion minimization can make output statistics deviate from ground-truth statistics.
  • Contribution: The paper adopts Blau and Michaeli’s mathematical perceptual-quality definition and proves a triple tradeoff among rate, distortion, and perception.Its central observation is that enforcing higher perceptual quality elevates the rate-distortion function.
  • Tradeoff: Higher perceptual quality requires sacrificing either rate or distortion, as illustrated by the elevated constrained rate-distortion curves.Figure 1 contrasts Shannon’s unconstrained curve with curves under progressively tighter perceptual constraints.
  • Analysis: The rate-distortion-perception function R(D, P) gives the minimal achievable rate for specified distortion D and perception index P.The analysis derives a Bernoulli-source closed form and proves monotonicity and convexity under minor assumptions for full-reference distortion measures.
  • Experiment: The paper illustrates the three-way tradeoff with a toy MNIST image-compression experiment, finding that the phenomena are dominant at low bit rates.The experiment varies rate, distortion, and perception using an encoder-decoder network with a generative model.

2. Background

The background distinguishes rate-distortion objectives from perceptual quality, which concerns whether decoded outputs resemble natural source signals. It explains how these criteria can conflict and how perceptual quality is operationalized through distributional divergence.

  • Rate-Distortion Theory: Rate-distortion theory studies the tradeoff between bits per sample and expected reconstruction distortion under an encoder-decoder mapping.The mapping may be stochastic and is represented by a conditional distribution p_ˆX|X.
  • Rate-Distortion Theory: Classical rate-distortion functions are known in closed form for only a few source distributions and simple distortion measures, although they are generally non-increasing and convex.Examples of simple measures include squared error and Hamming distance.
  • Perceptual Quality: Perceptual quality measures whether an output is perceived as a valid natural sample, independently of its similarity to the input.It is associated with the deviation of the output distribution p_ˆX from the natural-signal distribution pX.
  • Perceptual Quality: The paper defines a lower-is-better perceptual quality index as the divergence d(pX, p_ˆX) between source and output distributions.The divergence may be Kullback-Leibler or Wasserstein, and the choice that best matches human perception remains under study.
  • Perceptual Quality: Perceptual quality and distortion can diverge: sampling directly from pX gives perfect perceptual quality but poor distortion, while distortion minimization need not yield good perceptual quality.The conflict is described as fundamental for non-invertible degradations.

3. The Rate-Distortion-Perception Tradeoff

The rate-distortion-perception function extends rate-distortion theory by requiring perceptual quality alongside rate and distortion. For Bernoulli sources and under general assumptions, stricter perception constraints can elevate the rate-distortion curve, while perfect perceptual quality under MSE has a bounded rate-distortion cost.

  • Definition and setup: The rate-distortion-perception function extends rate-distortion theory by incorporating a perception index alongside rate and distortion.It lower-bounds the best achievable rate for specified distortion and perception constraints, although general achievability is not proved.
  • Bernoulli source: For a Bernoulli source with Hamming distortion and total-variation perception, sufficiently tight perception constraints cause R(D, P) to depart from the unconstrained rate-distortion function.When P ≤ p, the perception constraint becomes active whenever the distortion constraint is sufficiently loose.
  • Bernoulli source: Stricter perceptual-quality constraints elevate the rate-distortion curve, requiring a sacrifice in rate or distortion as allowed distortion increases.At D = 0, all curves merge because lossless reconstruction preserves the source distribution; the curves separate as D grows.
  • Theoretical properties: At lower bit rates, the tradeoff between distortion and perceptual quality becomes stronger, whereas higher rates can improve perception at constant distortion.This behavior appears in equi-rate level sets and cross sections of the three-dimensional function R(D, P).
  • Theoretical properties: Under A1, R(D, P) is convex, and it is monotonically non-increasing in D and P; under A2, perfect-perception and unconstrained curves differ.A2 is sufficient to guarantee some D0 with R(D0, 0) > R(D0, ∞), though failure of A2 does not imply equality.
  • Theoretical properties: For squared-error distortion, perfect perceptual quality requires no higher rate than unconstrained compression at half the target distortion, giving a no-more-than-2-fold MSE increase.The bound is not generally tight, so some settings may achieve perfect perception with a smaller distortion increase.

4. Experimental Illustration

The MNIST experiments visualize how rate, distortion, and perceptual quality trade off, using neural encoder-decoder pairs trained with squared-error and deep-feature distortions. Perceptual constraints improve visual quality, but elevate rate-distortion curves and do not ensure digit accuracy.

  • 4. Experimental Illustration: The experiments train neural encoder-decoder pairs while varying representation dimension, quantization levels, and a perception-distortion weighting parameter.Rate is controlled by the encoder output dimension and quantization levels, while the weighting parameter balances distortion against perceptual quality.
  • 4.1. Squared-Error Distortion: 98 MNIST encoder-decoder pairs show that constraining perceptual quality elevates the rate-distortion curve, requiring higher rate and/or distortion.Perceptual quality is indicated by discriminator loss approximating Wasserstein distance, and rate is measured in bits per digit.
  • 4.1. Squared-Error Distortion: At fixed rate, distortion and perceptual quality trade off more strongly at low bit-rates, while fixed distortion allows improved perception through increased rate.These relationships are visualized through equi-rate lines and cross-sections of R(D, P).
  • 4.1. Squared-Error Distortion: At extremely low bit-rates, perceptual constraints produce visually pleasing reconstructions, but at 2 bits most reconstructions fail to preserve digit identity.Improved perceptual quality therefore does not imply improved accuracy; Shannon-curve reconstructions are more accurate on average.
  • 4.2. Advanced Distortion Measures: Replacing MSE with deep-feature distortion does not eliminate the perception tradeoff, although the tradeoff appears somewhat weaker.Minimizing distortion alone is somewhat more visually pleasing with the advanced distortion, while perception-constrained reconstructions retain greater variability and less blur.

5. Conclusion

The paper concludes that perceptual quality conflicts with rate and distortion in lossy compression. It characterizes this triple tradeoff and argues that rate-distortion curves alone may mislead method comparisons.

  • 5. Conclusion: Keeping decoded-signal statistics similar to source statistics requires higher distortion or rate.The paper frames this as a fundamental conflict between perceptual quality and the other two compression properties.
  • 5. Conclusion: The characterized triple tradeoff was empirically illustrated in image compression, suggesting rate-distortion curves alone may mislead comparisons.

Supplementary Material

The supplementary material contains proofs, a rate relationship for memoryless stationary sources, the Bernoulli rate-distortion-perception function, and experimental training details.

  • Supplementary Material: The supplement provides proofs for Theorems 1 and 2 and proves the relation between encoder-decoder rate and R(D, P) for memoryless stationary sources.
  • Supplementary Material: It derives the rate-distortion-perception function R(D, P) for a Bernoulli random variable.
  • Supplementary Material: It specifies training and architecture details for the experiments and discusses the perceptual loss used in Section 4.2.

A. Proof of Theorem 1

The theorem analysis establishes that R(D, P) is non-increasing and convex under stated assumptions, and that perception constraints can make it differ from the traditional rate-distortion function.

  • A. Proof of Theorem 1: R(D, P) is non-increasing in both distortion D and perception index P because enlarging the constraint set cannot increase the minimum mutual information.
  • A. Proof of Theorem 1: Assuming convexity of the divergence in its second argument, convex combinations of feasible conditional distributions establish convexity of R(D, P).The proof combines mutual-information convexity with convexity of the perceptual divergence constraint.
  • A. Proof of Theorem 1: The convexity proof concludes directly by combining the intermediate inequalities establishing the required bound.
  • A. Proof of Theorem 1: Under assumption A2, perfect perceptual quality cannot generally make R(D, P) coincide with the unconstrained rate-distortion function.The contradiction uses an independent reconstruction at zero mutual information whose support cannot match the full source support.
  • A. Proof of Theorem 1: The contradiction is completed because perfect perceptual quality would require the source support to lie within the minimizing set, contrary to assumption A2.

B. Proof of Theorem 2

The proof constructs a modified decoder that achieves perfect perceptual quality while controlling its distortion and rate. It uses posterior sampling and establishes the needed information and distortion relations.

  • Rate-distortion pair: Replacing ˆX with E[X|ˆX] lowers MSE without increasing the rate, so the considered pair remains on Shannon’s rate-distortion curve.
  • Posterior-based construction: Posterior sampling produces a new decoded output whose distribution matches the source distribution, yielding perfect perceptual quality.The modified output is drawn from pX|ˆX, and its marginal distribution is identical to pX.
  • Information bound: The modified output preserves the relevant mutual-information bound through the data processing inequality for X → ˆX → ˜X.
  • Conclusion: The proof combines the construction, constraint membership, convexity, and monotonicity of R(D, P) to obtain the theorem’s rate comparison.

C. Perception aware lossy compression of a memoryless stationary source

This section extends lossy compression analysis to memoryless stationary sources with both distortion and perception constraints. It shows that constraining perception raises the required rate at a fixed distortion level, while noting an achievability gap.

  • General rate bound: For a memoryless stationary source, any encoder-decoder pair with average distortion D and perception index P has rate at least R(D, P).The lower bound is evaluated at the pair’s distortion and perception levels.
  • General rate bound: R(D, P) is defined as the minimal achievable rate for specified distortion and perception constraints, with proofs relying on convexity and monotonicity.
  • Perception constraint: R(·, 0) > R(·, ∞) under the stated assumptions, so perfect perceptual quality requires a higher rate than unconstrained perception at constant distortion.
  • Scope: Achievability of R(D, P) is left for future work, although the lower-bound proof is sufficient to establish that a rate-distortion-perception tradeoff necessarily exists.

D. Derivation of the rate-distortion-perception function R(D, P) of a Bernoulli source

The Bernoulli-source derivation gives a piecewise characterization of R(D, P) under Hamming distortion and total-variation perception. The perception constraint is active only in part of the parameter range and disappears when P is sufficiently large.

  • Parameterization: The derivation parameterizes pˆX|X by a and b and evaluates mutual information, Hamming distortion, and total-variation divergence under those parameters.
  • Finite perception constraint: When D ≤ D1, the Bernoulli solution is independent of P; when D > D1, the total-variation constraint is active.
  • Zero-rate regime: The zero-rate solution occurs when a = b, making X and ˆX independent; a random source-distributed output satisfies the perception constraint but requires distortion to prevent triviality.
  • Overall solution: For P < p, the overall Bernoulli-source solution is assembled from regimes separated by thresholds D1 and D2.
  • Overall solution: For P ≥ p, the solution is independent of P and equals Shannon’s classic Bernoulli-source rate-distortion curve.

E. Architecture and training parameters for the experiments in Sec. 4

The experiments use encoder, decoder, and discriminator networks for MNIST compression, with training objectives combining distortion and perceptual losses. Quantization uses fixed, evenly spaced codebook centers and nearest-neighbor assignment.

  • Architecture: The MNIST experiments use encoder, decoder, and discriminator architectures specified in Table 1, with training objectives defined separately for distortion and perceptual-loss settings.
  • Training configuration: The experiments train 98 encoder-decoder pairs across encoder dimensions, quantization levels, and tradeoff coefficients listed in Table 2.
  • Training configuration: For each dimension and quantization level, a λ = 0 model uses distortion only and is trained for 25 epochs.
  • Optimization: Optimization uses ADAM with β1 = 0.5 and β2 = 0.9, batch size 64, scheduled learning rates, and specified convolutional filter sizes and padding.
  • Quantization: The quantization layer uses fixed, evenly spaced bin centers in [−1, 1], with nearest-neighbor assignment in the forward pass and soft assignment for backward-pass gradients.

F. The perceptual loss in the experiment in Sec. 4.2

The experiment combines squared-error distortion with a perceptual loss computed in a pre-trained classifier’s deep-feature space. For MNIST, the classifier is trained specifically on digit images, and the second convolutional layer is selected because it gives the best perceptual quality.

  • Loss design: Perceptual loss is combined with squared-error distortion because minimizing perceptual loss alone does not produce pleasing results.The paper notes that this combination is also common in related work using additional distortion terms.
  • Loss design: The perceptual loss is the squared error between deep features produced by a pre-trained convolutional network.This operationalizes perceptual similarity in the network’s feature space rather than directly in image space.
  • MNIST-specific network: The standard VGG network is unsuitable for MNIST, so the authors pre-train a digit-classification network achieving over 99% accuracy.The network architecture is reported in Table 3.
  • Selected configuration: The second convolutional layer is used in the experiment because it yields the best perceptual quality in the layer comparison.Training used stochastic gradient descent for 30 epochs with batch size 30.
  • Layer comparison: The experiment evaluates perceptual loss on the first convolutional, second convolutional, and first fully connected layers.Figure 9 shows perceptual loss alone, while Figure 10 compares it with MSE and uses weights optimized for visual quality.
Loading 1901.07821v4…