PaperlayerView exact revision on arXiv

Source-linked AI summary

Lossy Image Compression with Conditional Diffusion Models

Ruihan Yang, Stephan Mandt

arXiv:2209.06950v8eess.IVcs.CVcs.LGstat.ML

TL;DR

The paper addresses perceptual limitations in learned image compression by replacing conventional Gaussian or Laplacian decoders with a conditional diffusion model. It combines hierarchical variational compression with diffusion-based reconstruction and reports stronger perceptual performance than GAN baselines while remaining comparable to competitive baselines on classical distortion.

  • Problem

    Learned compression models optimized for MSE or MAE can produce blurry reconstructions, motivating more expressive decoders for perceptual image quality.

  • Method

    The approach combines transform coding and hierarchical variational autoencoders with a conditional diffusion decoder that reconstructs images from an entropy-coded content latent while synthesizing texture variables.

  • Results

    The approach outperforms GAN baselines in three of four rate-perception metrics, including FID, while achieving comparable classical-distortion performance to highly competitive baselines.

  • Takeaways & Limitations

    Diffusion-based decoding provides a compression framework whose performance can be tuned toward perceptual metrics while retaining competitive distortion results.

  • Takeaways & Limitations

    At low bitrates, perceptual compression may hallucinate lower-level details and misrepresent identity, raising fairness and trustworthiness concerns.

Abstract

from arXiv · show

This paper outlines an end-to-end optimized lossy image compression framework using diffusion generative models. The approach relies on the transform coding paradigm, where an image is mapped into a latent space for entropy coding and, from there, mapped back to the data space for reconstruction. In contrast to VAE-based neural compression, where the (mean) decoder is a deterministic neural network, our decoder is a conditional diffusion model. Our approach thus introduces an additional ``content'' latent variable on which the reverse diffusion process is conditioned and uses this variable to store information about the image. The remaining ``texture'' variables characterizing the diffusion process are synthesized at decoding time. We show that the model's performance can be tuned toward perceptual metrics of interest. Our extensive experiments involving multiple datasets and image quality assessment metrics show that our approach yields stronger reported FID scores than the GAN-based model, while also yielding competitive performance with VAE-based models in several distortion metrics. Furthermore, training the diffusion with $\mathcal{X}$-parameterization enables high-quality reconstructions in only a handful of decoding steps, greatly affecting the model's practicality. Our code is available at: \url{https://github.com/buggyyang/CDC_compression}

1 Introduction

The paper proposes a transform-coding image compressor that replaces deterministic Gaussian or Laplacian decoding with conditional diffusion, separating encoded content from synthesized texture. Experiments report strong perceptual performance and competitive distortion results, while X-prediction enables fast reconstruction.

  • Proposed approach: A discrete content latent stores image information, while diffusion texture variables are synthesized during decoding.The content latent conditions the denoising process rather than fully specifying a deterministic reconstruction.
  • Motivation: The model can be modified to enhance perceptual metrics, addressing the limitations of VAE decoders that directly optimize low MSE or MAE and may produce blurry outputs.The motivation is to use a more expressive conditional generative model for perceptual image compression.
  • Empirical findings: Across four datasets and up to sixteen image-quality metrics, the approach reports best performance in FID while remaining comparable with leading compression models.The evaluation spans distortion, perceptual-reference, and no-reference perceptual metrics.
  • Proposed approach: The framework combines hierarchical transform coding with a conditional diffusion decoder to reconstruct images from an encoded contextual latent.The encoder maps images to a latent used as diffusion context during reconstruction.
  • Training and decoding: The diffusion-derived loss captures a richer decoding distribution than traditional VAE distortion terms and supports high-quality reconstructions in a handful of denoising steps.The authors derive the objective from a variational upper bound on the diffusion model’s implicit rate-distortion function.

2 Related Work

Related work progresses from classical and VAE-based transform codecs toward perceptual compression using adversarial or diffusion generative models. The paper positions its conditional diffusion codec as variational and tailored to entropy-coded image compression.

  • Transform-coding compression: End-to-end learned codecs challenge JPEG, BPG, and WEBP using nonlinear transform coding implemented through hierarchical VAEs.These codecs jointly transform data into latent representations and entropy-code them into short bit strings.
  • Transform-coding compression: VAE-based codecs commonly use Gaussian or Laplacian decoders and optimize MSE or MAE distortion.Their learned priors support entropy coding of latent representations.
  • Perceptual compression: Perceptual compression methods target realism at low bitrates, but distortion and realism constitute a fundamental tradeoff within a given architecture.Adversarial losses have been used to improve perceptual quality, although GAN training can be unstable and design-sensitive.
  • Diffusion-based compression: Diffusion models offer high perceptual quality with stable training, and prior work has begun applying them to compression.The paper’s approach differs by combining diffusion with variational, entropy-coding-oriented compression rather than using a non-variational model.

3 Method

The method combines transform-coding compression with a conditional diffusion decoder that reconstructs images from an entropy-coded content latent while synthesizing texture variables during decoding. Its objective balances bitrate and a diffusion-induced reconstruction distortion, while X-prediction enables efficient iterative decoding and stochasticity supports perceptual-quality control.

  • Latent-variable model: The diffusion variational autoencoder stores image content in a semantic latent z while generating residual texture variables x1:N at decoding time.Because x1:N are synthesized rather than compressed, stochastic decoding can produce variable reconstructions.
  • Compression and decoding: The encoder maps images to a quantized latent z, whose discrete prior is entropy-coded, and the decoder reconstructs images through denoising conditioned on z.The prior is hierarchical, and the conditional decoder replaces the usual simplistic Gaussian or Laplacian decoder.
  • Training objective: The rate-distortion objective combines the cost of coding z under its prior with Lupper, a diffusion-induced distortion measuring reconstruction from z.Lupper is obtained as a variational upper bound to the diffusion model’s negative data log likelihood and is distinct from conventional VAE distortion.
  • Fast decoding using X-prediction: X-prediction directly reconstructs x0 from noisy inputs and z, inheriting autoencoder-like single-step reconstruction while successive steps refine perceptual quality.When noise is large, the model relies more on z; when noise is small, the noisy image carries more reconstruction information.
  • Fast decoding using X-prediction: X-prediction achieves comparable compression performance to ε-prediction with only a handful of decoding steps instead of hundreds, without post-processing.The method uses a smaller test-time number of denoising steps than the training schedule can contain.
  • Perceptual optimization: Additional perceptual losses can steer the quality of compressed images toward selected perceptual metrics.The optional losses are introduced alongside the conditional diffusion compression loss.

4 Experiments

Experiments evaluate Conditional Diffusion Compression across datasets, baselines, metrics, and decoding choices. Results show a tunable tradeoff: deterministic decoding favors distortion, while stochastic and perceptual-loss-informed decoding favors perceptual quality.

  • Experimental Setup: The evaluation spans multiple test datasets and sixteen image-quality metrics, including distortion and perceptual measures.The main paper presents eight widely used metrics, with eight additional metrics in the appendix.
  • Baseline Comparisons: CDC is compared with HiFiC for perceptual quality and DGML and NSC for rate-distortion performance.HiFiC uses adversarial, LPIPS, and MSE losses, while DGML and NSC are VAE-based models.
  • Distortion Metrics: CDC with ρ = 0 performs favorably on distortion metrics, with on-par FSIM, SSIM, and MS-SSIM scores against the best baselines, excluding PSNR.The ρ = 0 model is not specifically optimized for these three metrics.
  • Distortion vs. Perception: CDC with ρ = 0.9 performs best on perceptual metrics, whereas CDC with ρ = 0 achieves better model-based distortion performance.The stochastic, LPIPS-informed version uses γ = 0.8, while the base version uses deterministic decoding.
  • Ablation Studies: Increasing decoding steps improves perceptual results under stochastic decoding but does not substantially improve distortion under deterministic decoding.The X-prediction model uses 17 decoding steps in the DIV2K comparison, while ε-prediction requires hundreds for comparable performance.
  • Ablation Studies: Increasing γ adds detail and rugged texture, with γ = 0.8 giving the lowest LPIPS and best qualitative outcomes; values above 0.8 increase distortion and reduce perceptual quality.All images in the qualitative comparison share the same bpp.

5 Conclusion & Discussion

The paper presents a diffusion-based transform-coding compression approach with promising rate–perception performance and competitive classical distortion. It identifies future compression improvements and risks of perceptual misrepresentation at low bitrates.

  • The proposed method combines transform coding with a denoising decoder for iterative reconstruction of compressed images.
  • The approach outperforms GAN baselines in three of four rate–perception metrics, including FID, while remaining comparable to highly competitive baselines on classical distortion.
  • Further compression improvements may come from integrating autoregressive entropy models or iterative encoding, which the paper leaves for future research.
  • Prioritizing perceptual quality at low bitrates can risk hallucinated lower-level details and misrepresenting identities, raising fairness and trustworthiness concerns.

B Architectures

The architecture uses a U-Net denoising module alongside a hyperprior-based compression design. Its memory efficiency exceeds HiFiC, but iterative diffusion decoding is slower than baseline methods.

  • Denoising module: The denoising module uses a U-Net with ResNet, attention, and convolutional up/downsampling blocks across six downsampling and six upsampling units.
  • Architecture overview: Figure 6 visualizes the proposed model architecture, with additional specifications documented alongside the architecture description.
  • Hyperprior: The hyperprior follows Minnen et al. (2018), and the hyper latent y uses 256 channels.
  • Efficiency: The model has superior memory efficiency to HiFiC, but diffusion decoding takes approximately 1 second per image, slower than baselines.

D Additional explanation on experiment metrics

The paper uses several image-quality metrics to assess distortion and perceptual similarity, including FID, LPIPS, and PieAPP.

  • FID measures the Fréchet-distance divergence between compressed-image and ground-truth latent-feature distributions.
  • LPIPS measures the l2 distance between latent embeddings extracted from VGG-Net or AlexNet.
  • PieAPP provides a different perceptual-score measurement based on a model.

E Supplemental Ablation Study

The trade-off parameter ρ controls whether training prioritizes perceptual quality or traditional distortion. Larger ρ improves perceptual quality while generally worsening distortion, with little benefit beyond ρ > 0.9.

  • ρ values of 0, 0.32, 0.64, and 0.9 are evaluated through rate-distortion curves on COCO.
  • Higher ρ values improve perceptual quality but worsen distortions in most scenarios.
  • Values exceeding ρ > 0.9 are not viable because perceptual quality no longer shows noticeable improvement.

F Additional visualization of the compressed images and decoding variability visualization

The visualizations compare CDC and HiFiC reconstructions with ground truth at specified bit rates, and illustrate decoding variability from different random seeds. Stochastic decoding preserves the same latent variable while allowing low-level textural distinctions.

  • Ground-truth images are included as reference visualizations for the compressed-image comparisons.
  • CDC Xθ(ρ = 0.9) is visualized at bpp=0.205 and bpp=0.398 alongside HiFiC reconstructions at bpp=0.207 and bpp=0.456.
  • Different random seeds for xN ∼N(0, γ2I) can produce low-level textural distinctions when stochastically decoding the same latent variable z with γ = 0.8.

G Visualizations of the Decoding Process

The decoding-process visualization tracks the texture variable across five time steps and contrasts deterministic reconstruction with stochastic denoising. Deterministic decoding transforms a grayscale image directly into the reconstruction, whereas stochastic decoding dynamically denoises noise.

  • The texture variable xn is visualized at n = {0%N, 30%N, 60%N, 90%N, 100%N} over N decoding steps.
  • Deterministic decoding transforms a grayscale image into the reconstruction, while stochastic decoding dynamically denoises noise during the process.

H Additional Rate-Distortion(Perception) Results

Additional rate-distortion-perception results are shown for COCO, Tecnick, Kodak, and DIV2K. The figures specify 500 decoding steps for the ϵθ model.

  • COCO: Figure 16 reports rate-distortion-perception results for the COCO dataset using 500 decoding steps for the ϵθ model.
  • Tecnick: Figure 17 reports rate-distortion-perception results for the Tecnick dataset using 500 decoding steps for the ϵθ model.
  • Kodak: Figure 18 reports rate-distortion-perception results for the Kodak dataset using 500 decoding steps for the ϵθ model.
  • DIV2K: Figure 19 reports rate-distortion-perception results for the DIV2K dataset using 500 decoding steps for the ϵθ model and includes the complete 16 metrics.
Loading 2209.06950v8…