PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimally Controllable Perceptual Lossy Compression

Zeyu Yan, Fei Wen, Peilin Liu

arXiv:2206.10082v1cs.CVeess.IV

TL;DR

Distortion and perceptual quality conflict, and achieving different points on their tradeoff has typically called for different decoders. This paper uses two constructed decoders and optimal training frameworks to attain arbitrary tradeoffs by interpolation. Experiments validate the theoretical result and demonstrate the frameworks' practical effectiveness.

  • Problem

    Distortion and perceptual quality are at odds, while flexible optimal control of their tradeoff remains unclear.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    The paper linearly interpolates between an MMSE decoder and a perfect-perception decoder, and proposes two optimal frameworks for training the latter.

    Source: §1 · p. 2

  • Results

    The theory establishes arbitrary optimal D-P tradeoffs from the two decoders, and experiments on MNIST, depth images, and RGB images validate the result and framework effectiveness.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Two paired decoders provide a theoretical foundation for achieving optimal D-P tradeoffs without training a different decoder for each quality level.

    Source: §1 · p. 2, §6 · p. 9

    Although the augmented formulation is theoretically optimal for any λ in [0, 1), practical performance still requires tuning λ because Wasserstein-1 approximation and finite model capacity intervene.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Recent studies in lossy compression show that distortion and perceptual quality are at odds with each other, which put forward the tradeoff between distortion and perception (D-P). Intuitively, to attain different perceptual quality, different decoders have to be trained. In this paper, we present a nontrivial finding that only two decoders are sufficient for optimally achieving arbitrary (an infinite number of different) D-P tradeoff. We prove that arbitrary points of the D-P tradeoff bound can be achieved by a simple linear interpolation between the outputs of a minimum MSE decoder and a specifically constructed perfect perceptual decoder. Meanwhile, the perceptual quality (in terms of the squared Wasserstein-2 distance metric) can be quantitatively controlled by the interpolation factor. Furthermore, to construct a perfect perceptual decoder, we propose two theoretically optimal training frameworks. The new frameworks are different from the distortion-plus-adversarial loss based heuristic framework widely used in existing methods, which are not only theoretically optimal but also can yield state-of-the-art performance in practical perceptual decoding. Finally, we validate our theoretical finding and demonstrate the superiority of our frameworks via experiments. Code is available at: https://github.com/ZeyuYan/Controllable-Perceptual-Compression

1. Introduction

Distortion and perceptual quality are at odds, leaving the D-P tradeoff as a central challenge in lossy compression. The paper shows that two constructed decoders and two theoretically optimal training frameworks suffice to achieve arbitrary tradeoffs, with experiments validating the theory and framework performance.

1. Introduction

  • With squared-error distortion and squared Wasserstein-2 perception, linear interpolation between an MMSE decoder and a perfect-perception decoder achieves arbitrary points on the optimal D-P bound.The interpolation factor quantitatively controls perceptual quality.Source: §1 · pp. 1–2
  • Two optimal training frameworks construct the perfect-perception decoder: direct decoding from compressed representations or post-processing an MMSE encoder-decoder pair.The direct framework is more compact; the post-processing framework is more flexible for existing distortion-optimized systems, and both apply beyond RGB images.Source: §1 · p. 2
  • Experiments on MNIST, depth images, and RGB images validate the theoretical finding and show that the frameworks can achieve state-of-the-art perceptual quality in practice.The paper reports satisfactory practical performance on RGB images for its augmented framework.Source: §1 · p. 2
  • The augmented framework adds MMSE-decoder conditioning to Yan et al. (2021)'s adversarial training, retaining theoretical optimality while addressing its unsatisfactory RGB-image performance.The paper attributes the earlier difficulty to adversarial loss alone being insufficient for stable RGB decoding.Source: §1 · p. 2
  • Unlike Freirich et al. (2021)'s restoration result, this work jointly optimizes a compression encoder and decoder and derives the tradeoff through a different analysis.It also proposes an augmented optimal formulation and two optimal training frameworks for controllable compression.Source: §1 · p. 2

2. Related Works

Prior work studies perceptual losses and the distortion-perception tradeoff, but optimal flexible control remains unresolved. Existing approaches either require separate models or use heuristic interpolation.

2.1. Perceptual Lossy Compression

  • Perceptual compression commonly combines distortion and adversarial or deep-feature losses, but these methods are not optimal for perceptual decoding and typically require separate models for different tradeoffs.Yan et al. (2021) gives an optimal adversarial-training framework for perfect perception, but reports unsatisfactory results on more complex data such as RGB images.Source: §2.1 · p. 2

2.2. Distortion-Perception Tradeoff

  • Optimal flexible D-P control remains unclear: varying decoders can approximate the tradeoff but requires different decoders, while decoder interpolation has been described as heuristic and non-optimal.Prior work also establishes distortion-perception tension and a shared encoder for compression with or without a perception constraint, but the latter result covers only perfect perception.Source: §2.2 · pp. 2–3

3. Theory for Optimally Controllable D-P Tradeoff

Because distortion and perceptual quality trade off, conventional loss balancing requires different models for different settings. The paper proves that paired decoders can instead reach any point on the D-P bound by interpolation, with an MMSE encoder optimal across perception constraints.

3.1. Perceptual Lossy Compression

  • Perceptual quality measures how natural decoded samples look, regardless of similarity to the reference; it can be quantified by deviation between source and output distributions.The passage names KL divergence and Wasserstein distance as examples of deviation measures.Source: §3.1 · p. 3
  • Distortion-plus-adversarial loss combines paired-sample distortion with an adversarial loss that promotes decoded outputs resembling the natural-sample distribution.Its balance parameter λ controls the tradeoff, but the unconstrained formulation cannot attain perfect perception without potentially excessive distortion.Source: §3.1 · p. 3

3.2. Theoretical Results for Optimally Controllable D-P Tradeoff

  • Two decoders—an MMSE decoder and a perfect-perception decoder paired with its encoder—can reach any point on the D-P bound through linear interpolation.The theorem also states that the MMSE encoder is optimal for every perception constraint P ≥ 0.Source: §3.2 · p. 4
  • Varying interpolation factor α controls the distortion-perception tradeoff; at α = 0, decoding distortion is exactly twice its value at α = 1.The result uses MSE distortion and squared Wasserstein-2 distance for perceptual quality.Source: §3.2 · p. 4
  • The D-P tradeoff function D(P) is nonincreasing and convex.The paper relates these properties to the first and second derivatives of P with respect to D.Source: §3.2 · pp. 4–5

4. Optimal Training Frameworks for Perfect Perceptual Decoding

The section presents two optimal training frameworks for perfect perceptual decoding, enabling interpolation between an MMSE decoder and a perfect perceptual decoder to reach arbitrary D-P tradeoff points. An augmented training formulation preserves perfect-perception optimality while adding supervision from the MMSE reconstruction.

4. Optimal Training Frameworks for Perfect Perceptual Decoding

  • The initial framework trains an MMSE encoder-decoder pair, fixes the encoder, then trains a perceptual decoder using conditional Wasserstein-1 adversarial training; adversarial-only training is unsatisfactory on RGB images.The Wasserstein-1 objective uses a conditional discriminator to distinguish decoded and source samples paired with the compressed representation.Source: §4 · p. 5

4.1. An Augmented Optimal Formulation for Perfect Perceptual Decoding Learning

  • Adding an ℓ2 supervision term measuring distance from the MMSE decoding Xd leaves the augmented formulation optimal for perfect perceptual decoding when λ ∈ [0, 1).The paper attributes improved training stability to the added supervision from Xd.Source: §4.1 · p. 5
  • In practice, λ still needs tuning because Wasserstein-1 is approximated and decoding models have finite capacity.The theorem states that any λ ∈ [0, 1) satisfies the optimal condition, but practical performance varies with λ.Source: §4.1 · p. 5

4.2. Proposed Optimal Training Frameworks

  • Both frameworks can reach arbitrary D-P bound points by interpolating between the MMSE decoder and perfect perceptual decoder; Framework B also applies to existing distortion-optimized systems.Framework A has a more compact structure, while Framework B can replace its MMSE pair with a distortion-only compression system such as BPG.Source: §4.2 · p. 6

5. Experiments

Experiments on MNIST, depth images, and RGB images examine the proposed interpolation-based tradeoff and training frameworks. The results report clearer MNIST outputs at similar distortion, improved depth-image perceptual quality, and RGB tradeoffs across two frameworks.

5. Experiments

  • The experiments evaluate Framework A on MNIST and Framework B on SUNCG depth images, and compare proposed-framework perception scores on RGB images at two bit rates.The RGB experiment uses COCO2014, and the empirical perception-distortion curves are plotted on KODAK.Source: §5 · p. 6, §5.2 · p. 7, §5.3 · p. 8

5.1. Results on MNIST

  • On MNIST, Framework A’s output is clearer than traditional DAL at similar MSE, while increasing α makes outputs blurrier but lowers distortion.The empirical MSE approximately exhibits the predicted jump at β = 0.5.Source: Figure 3 · p. 7, §5.1 · p. 7

5.2. Results on Depth Images

  • On SUNCG depth images, Gp yields much better perceptual quality than MMSE decoding and HiFiC, while interpolated point clouds become blurrier as α increases.The comparison visualizes reconstructed point clouds; HiFiC is not distinctly better than MMSE around noisy edges.Source: Figure 4 · p. 7, §5.2 · p. 8

5.3. Results on RGB Images

  • On RGB images, Frameworks A and B outperform networks trained by directly altering λ, while their perception-distortion curves are similar.At α = 0.5, HiFiC has a better perception score than the proposed frameworks at similar distortion; the reported PI score deteriorates as α increases while distortion decreases.Source: Figure 5 · p. 8, §5.3 · p. 8

6. Conclusion

The conclusion states that two specifically constructed decoders suffice for arbitrary distortion-perception tradeoffs through interpolation. Experiments on MNIST, depth images, and RGB images support the theoretical result and the frameworks’ effectiveness.

6. Conclusion

  • Two specifically constructed decoders suffice for arbitrary points of the distortion-perception tradeoff.The conclusion identifies these as the MMSE decoder and a perfect perceptual decoder, whose outputs are interpolated.Source: §6 · p. 8
  • The proposed training frameworks are presented as theoretically optimal ways to construct a perfect perceptual decoder.The paper positions these frameworks as going beyond existing heuristic frameworks.Source: §6 · pp. 8–9
  • Experiments on MNIST, depth images, and RGB images support the theoretical result and demonstrate the frameworks’ effectiveness.Source: §6 · p. 8

A. Proof of Theorem 1.

The proof establishes that an MMSE encoder paired with an interpolated decoder is optimal across the distortion-perception bound. It first treats the decoder for a fixed encoder, then shows the encoder and pair attain the required optimum.

A. Proof of Theorem 1.

  • For any α in (0, 1), the MMSE encoder Ed is optimal for the unconstrained formulation when paired with an optimal decoder.The proof identifies Ed and the corresponding joint distribution choice as attaining equality in the bound.Source: A. Proof of Theorem 1 · p. 13
  • With Ed fixed, the optimal decoder is the linear interpolation αGd(Zd) + (1 − α)Gp(Zd).The proof uses the perfect-perception decoder’s matching joint distribution with the latent representation to obtain this decoder.Source: A. Proof of Theorem 1 · p. 13
  • Comparing the interpolated pair with an arbitrary optimal pair establishes that it attains the D-P bound.The proof concludes that the equalities hold for any competing pair, so the constructed pair is optimal for the constrained formulation.Source: A. Proof of Theorem 1 · p. 13

B. Proof of Theorem 2.

The proof relates the optimality condition in terms of the decoded MMSE output to the condition in terms of the latent representation. It also establishes the different optimal output behavior for λ below one and above one.

B. Proof of Theorem 2.

  • For 0 ≤ λ < 1, an optimal solution to (12) matches the joint distribution of the source and MMSE decoding output.The proof derives this condition first in terms of the augmented variables and then expresses it for X and Xd.Source: B. Proof of Theorem 2 · p. 14
  • For an optimal MMSE decoder, the mapping from Zd to Xd is bijective, making the matching condition in terms of Xd equivalent to the condition in terms of Zd.Source: B. Proof of Theorem 2 · p. 14
  • When λ > 1, the only optimal solution to (12) is the MMSE decoding output Xd.Source: B. Proof of Theorem 2 · p. 14

C. Applying Framework B to Post-process the BPG Codec

Framework B can post-process BPG even though BPG is not optimal in MSE, and the reported SUNCG example shows its output closer to ground truth than BPG’s.

C. Applying Framework B to Post-process the BPG Codec

  • Although Framework B is theoretically optimal when its encoder-decoder pair is MMSE, it also applies to existing distortion-optimized compression systems.The paper notes the strict MMSE-pair condition in practice while describing the broader applicability of the framework.Source: C. Applying Framework B to… · p. 14
  • Framework B replaces its encoder-decoder pair with BPG at QP=40 and trains a post-processing network to improve BPG’s perceptual quality.The parameters match those used in Section 5.2.Source: C. Applying Framework B to… · p. 16
  • The post-perceptual BPG output is much closer to ground truth than the BPG output on a typical SUNCG sample.The comparison is shown using point clouds for visual clarity.Source: Figure 6 · p. 15, C. Applying Framework B to… · p. 16

D. Result on RGB Images

On RGB images, both proposed frameworks produce more detail and sharper edges than the MMSE decoder. Lower λ trades lower PSNR for improved perception indices, while very low values add visible artifacts.

D. Result on RGB Images

  • Both Framework A and Framework B produce more detail and sharper edges than the MMSE decoder in the Kodak examples.The compared RGB samples are shown in Figures 7–9.Source: D. Result on RGB Images · p. 16
  • As λ decreases, PSNR decreases while each of the three perception indices improves on Kodak.The quantitative comparison is reported in Table 2 for Framework A.Source: D. Result on RGB Images · p. 16, Table 2 · p. 18
  • When λ ≤0.1, visible detail improves significantly, but artifacts also increase.For λ ≥0.2, the outputs look similar to MMSE decoding.Source: D. Result on RGB Images · p. 16
  • More intensive tuning of adversarial training, hyperparameters, and network architecture may further improve perceptual quality.The authors describe adversarial-training tuning as involved.Source: D. Result on RGB Images · p. 16
Loading 2206.10082v1…