On Perceptual Lossy Compression: The Cost of Perceptual Reconstruction and An Optimal Training Framework

1. Introduction

• High perceptual quality can raise the rate-distortion curve, yet quantitative analysis of the resulting distortion increase was lacking.The paper frames its questions around quantifying this effect and building a framework for perfect-perception reconstruction.Source: §1 · p. 2
• At a fixed bit rate, perfect perception doubles the lowest achievable MSE; an MSE-optimized encoder is also optimal under the perfect-perception constraint.The analysis further makes adversarial loss unnecessary for optimizing the encoder.Source: §1 · p. 2
• The proposed GAN conditions its discriminator on an MSE-optimized encoder and achieves the lowest MSE under perfect perception, outperforming distortion-plus-adversarial-loss training.The framework does not require distortion loss to train the perceptual decoder.Source: §1 · p. 2
• Experiments on MNIST verify the theoretical finding and show that the perception constraint's effect on the rate-distortion trade-off accords with the predicted doubling of MSE.The experiments also demonstrate the proposed framework's superiority over the traditional distortion-plus-adversarial-loss framework.Source: §1 · p. 2
• Unlike a prior result giving at most a twofold MSE increase, this analysis derives an exact doubling by directly analyzing the perception-constrained compression formulation.The analysis also concludes that distortion loss, on pixels or deep features, is unnecessary for decoder training.Source: §1 · p. 2

2.1. Perceptual Quality

• Perceptual quality measures how natural restored samples look, independently of their similarity to a reference, using divergence between source and reconstruction distributions.The divergence is zero when the distributions match.Source: §2.1 · p. 2

2.2. Rate-Distortion-Perception Tradeoff

• Shannon’s rate-distortion theory characterizes the tradeoff between bits per symbol and expected distortion, with the rate-distortion function depending on a distortion measure such as MSE.The rate-distortion function is convex and non-increasing in distortion.Source: §2.2 · p. 3
• Adding a perception constraint extends the rate-distortion formulation to a three-way rate-distortion-perception tradeoff; perfect perception requires identical source and reconstruction distributions.Prior work bounds the MSE rate-distortion function under perfect perception by a scaled version of the unconstrained curve, but the actual elevation remains unclear.Source: §2.2 · p. 3

2.3. Adversarial Loss for Perceptual Reconstruction

• Conventional GAN compression training balances distortion, adversarial, and entropy terms, but choosing the adversarial-loss weight makes jointly optimizing perception and distortion difficult.A small weight can leave perceptual quality unsatisfactory, while a large weight can prevent good distortion optimization.Source: §2.3 · p. 3
• The proposed GAN framework targets the lowest distortion under perfect perception while avoiding a balance between distortion and adversarial losses.The framework is also based on GAN.Source: §2.3 · p. 3

3.1. Analysis on the Rate-Distortion Tradeoff Under Perfect Perception Constraint

• At a fixed bit rate, perfect perception doubles the lowest achievable MSE distortion, equivalently relating the constrained rate-distortion function to the unconstrained function at half the distortion.This result is stated for squared-error distortion.Source: §3.1 · p. 5
• An encoder optimal without a perception constraint at distortion D/2 is also optimal under perfect perception at distortion D.This result allows the encoder to be optimized independently of the perception constraint.Source: §3.1 · p. 5

3.2. A Framework for Perfect Perceptual Reconstruction in Lossy Compression

• The proposed procedure first trains (E, G1) using MSE loss, then fixes E and adversarially trains G2 to match conditional output distributions given the compressed representation.The conditional discriminator receives both the data and bit stream Z.Source: §3.2 · pp. 5–6
• Unlike DAL, the framework uses its discriminator to assess whether decoded data is consistent with the bit stream, avoiding a balance between distortion and adversarial losses.The authors state that this enables lowest-MSE reconstruction under the perfect-perception constraint without a distortion constraint on G2.Source: §3.2 · p. 6
• The decoder G2 receives random noise while being optimized to satisfy the conditional-distribution constraint associated with perfect perceptual quality.The passage says the constraint holds when G2 and the discriminator are optimal.Source: §3.2 · p. 6

4. Experimental Illustration

• The experiments use MNIST to demonstrate the two-fold relation and compare the proposed framework with DAL.The authors note that the rate-distortion-perception function lacks an analytical expression for a general data distribution.Source: §4 · p. 6

4.1. Rate-Distortion Tradeoff with or without Perception Constraint

• G2’s MSE converges more slowly than G1’s but approaches twice G1’s MSE at each of six tested bit rates.The authors attribute G2’s slower convergence to alternating joint training with J; G2 uses adversarial loss without MSE loss.Source: Figure 3 · p. 7, §4.1 · p. 7
• Across 32 bit rates, the fitted R(D/2, ∞) curve closely approaches R(D, 0), empirically supporting the two-fold relation.The experiment compares 32 MSE-only encoder-decoder pairs with 32 pairs trained using the proposed framework.Source: §4.1 · p. 7
• G2 produces clearer, sharper images than G1, remaining recognizable at bit rate 2, but may reconstruct a different number.G2 adds noise to generate details, which can change the original number’s typeface, inclination, or thickness.Source: Figure 4 · p. 7, §4.1 · pp. 7–8
• The proposed method has better perceptual quality than DAL, even when DAL’s MSE is larger than G2’s.The comparison uses DAL settings with λ ∈ {0.1, 1, 5} at bit rate R = 4.Source: Figure 5 · p. 8, §4.1 · p. 8

4.2. Related Works

• Unlike end-to-end DAL, the proposed GAN first trains an MSE-optimized encoder, then trains the decoder adversarially while conditioned on that encoder.The authors relate their conditioned discriminator to Mentzer et al. (2020), while distinguishing their adversarial-only decoder training and MSE-only encoder training.Source: §4.2 · p. 8
• Applying the framework to color image compression is left for future work because of hardware demands and the challenge of tuning a large GAN.Source: §4.2 · p. 8

• At fixed bit rate, perfect perception exactly doubles the lowest achievable MSE, and experiments support this result and the proposed framework’s superiority over DAL.Source: §5 · p. 8

• Symmetry of Δ makes W symmetric, so transposing an optimal solution produces another feasible optimum.Source: A. Proof of Lemma 1 · p. 12
• Averaging the optimum and its transpose gives a feasible solution to (10).Source: A. Proof of Lemma 1 · p. 12
• Strict convexity of x log x supports equality of the averaged solution’s objective with the optimum, leading to the lemma’s conclusion.Source: A. Proof of Lemma 1 · p. 13

B. Proof of Theorem 1

• The proof represents the encoder and decoder through joint distribution matrices L and Q, respectively, for the source-output and reconstruction-output pairs.These matrices are defined using the joint distributions of Y with Z and Ŷ with Z.Source: B. Proof of Theorem 1 · p. 13
• The feasible region of (36) is a subset of that of (35), and (L∗, Q∗) is optimal to both; therefore, any optimum of (36) is also an optimum of (35).Source: B. Proof of Theorem 1 · p. 14
• The optimal pair (L*, Q*) is an optimal solution to (39), and the proof uses this result to establish optimality of the self-paired solutions.The argument compares their objective values and concludes that F(L*, L*) = F(Q*, Q*) = F(L*, Q*).Source: B. Proof of Theorem 1 · pp. 14–15

C. Derivation of equation (20)

• Equation (20) follows using E[Y |Z] = E[Ŷ |Z].This is the stated justification for step (d).Source: C. Derivation of equation (20) · p. 16
• The supplied derivation identifies equal conditional expectations and equal distributions as reasons supporting equation (20).The intermediate step (e) is also attributed to a stated condition, but that condition is not specified in these passages.Source: C. Derivation of equation (20) · p. 16
• The derivation also uses the fact that Y and Ŷ have the same distribution.The passage identifies this as the justification for step (f).Source: C. Derivation of equation (20) · p. 16

D. Degenerate problem

• G2 can converge to an MSE far from twice G1’s MSE, with clearer but incorrect output numbers despite sufficient bit-stream information.The passage contrasts G1’s correct outputs with G2’s incorrect outputs.Source: D. Degenerate problem · p. 16
• The authors attribute this degeneration to alternating adversarial training converging to a poor point.Source: D. Degenerate problem · p. 16
• Pretraining discriminator J on matched and mismatched sample-encoder pairs is reported to reduce degeneration in intensive experiments.The pairs are (x_i, E(x_i)) and (x_j, E(x_i)) for i ≠ j.Source: D. Degenerate problem · p. 16