PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Generative Models for Distribution-Preserving Lossy Compression

Michael Tschannen, Eirikur Agustsson, Mario Lucic

arXiv:1805.11057v2cs.LGstat.ML

TL;DR

The paper asks how lossy compression can preserve the training-data distribution rather than produce degraded deterministic reconstructions. It formulates DPLC and develops Wasserstein-based generative-model methods, including Wasserstein++, to learn stochastic decoders across rates. The resulting systems provide artifact-free reconstructions from zero-rate generation to almost perfect high-rate reconstruction, while improving over previous methods in stochastic decoding.

  • Problem

    DPLC addresses the problem of optimizing rate-distortion performance while requiring reconstructed samples to follow the training-data distribution, especially in the extreme-compression regime.

  • Method

    The paper first learns a generative model using WGAN, WAE, or their Wasserstein++ combination, then learns a rate-constrained stochastic decoder and encoder using Wasserstein distance.

  • Results

    The proposed systems produce essentially artifact-free reconstructions across rates, spanning generative modeling at zero bitrate and almost perfect reconstruction at high bitrate.

  • Takeaways & Limitations

    DPLC provides a framework that smoothly connects extreme compression with generative modeling while preserving the reconstruction distribution across rates.

  • Takeaways & Limitations

    The approach cannot simply replace the high-capacity generative mapping with a rate-constrained encoder, because low rates severely restrict the required model capacity.

Abstract

from arXiv · show

We propose and study the problem of distribution-preserving lossy compression. Motivated by recent advances in extreme image compression which allow to maintain artifact-free reconstructions even at very low bitrates, we propose to optimize the rate-distortion tradeoff under the constraint that the reconstructed samples follow the distribution of the training data. The resulting compression system recovers both ends of the spectrum: On one hand, at zero bitrate it learns a generative model of the data, and at high enough bitrates it achieves perfect reconstruction. Furthermore, for intermediate bitrates it smoothly interpolates between learning a generative model of the training data and perfectly reconstructing the training samples. We study several methods to approximately solve the proposed optimization problem, including a novel combination of Wasserstein GAN and Wasserstein Autoencoder, and present an extensive theoretical and empirical characterization of the proposed compression systems.

1 Introduction

Deep neural compression methods can improve visual quality at low bitrates by synthesizing realistic content, but this paper formalizes a stronger goal: reconstructions should preserve the data distribution at every rate. Its DPLC framework spans stochastic generation at zero bitrate, progressive reconstruction at intermediate rates, and perfect reconstruction at sufficiently high rates.

  • DNN-based compression methods outperform traditional codecs across image, speech, and video compression under several distortion measures.They can also adapt to specific domains and support fast processing through modern data-processing pipelines.
  • At low image bitrates, learning-based methods synthesize realistic local content and avoid blur or blocking, although they may incur larger PSNR distortions.GANs further improve visual quality by synthesizing content beyond local texture, such as building facades.
  • DPLC constrains reconstructions to follow the original data distribution while retaining more image content as bitrate increases.The decoder produces i.i.d. data-distribution samples at zero bitrate and can achieve perfect reconstruction at high enough rates.
  • The proposed approaches use Wasserstein distance and deep generative models to learn stochastic decoders with theoretically characterizable reconstruction distributions and bitrate-dependent distortion bounds.The framework first learns a generative model, then learns a stochastic decoder and corresponding encoder.
  • Wasserstein++ combines WGAN and WAE to target high sample quality, good mode coverage, and a structured latent space suitable for an encoder.The paper evaluates the approach on CelebA and LSUN bedrooms and presents it as an effective DPLC system.

2 Problem formulation

The paper replaces deterministic lossy reconstructions with a rate-constrained stochastic decoder whose outputs match the training-data distribution. This preserves artifact-free generation across rates while connecting zero-rate generative modeling to high-rate perfect reconstruction.

  • Standard lossy compression uses an encoder and decoder to minimize expected distortion subject to an R-bit code.The encoder maps inputs to one of 2^R codes, and the decoder maps codes back to the input space.
  • Deterministic compression has at most 2^R distinct reconstructions, causing increasing blur or blocking as rates decrease and a constant reconstruction at R = 0.Allowing stochastic encoders and decoders alone does not resolve this problem.
  • DPLC optimizes the standard rate-distortion objective while requiring reconstructed samples to follow the training-data distribution.The decoder is allowed to be stochastic.
  • The distribution constraint is intended to enforce artifact-free reconstructions at all rates.As R approaches zero, the solution becomes a generative model; at sufficiently large rates, it guarantees perfect reconstruction.

3 Deep generative models for distribution-preserving lossy compression

The paper formulates distribution-preserving lossy compression as rate-distortion optimization constrained by similarity between reconstructed and data distributions. Its generative-model approach separates distribution learning from rate-constrained reconstruction, enabling a rate-dependent tradeoff with theoretical guarantees.

  • The regularized formulation addresses the difficulty of enforcing the distribution constraint, which at zero bitrate requires learning an exact generative model of the unknown data distribution.
  • Challenges of the extreme compression regime: At finite rates, distortion-only optimization can produce constant, mean-biased reconstructions, motivating stochastic generative decoders for extreme compression.
  • Proposed method: The decoder is decomposed as D = G ◦ B, with G trained as a generative model and stochastic B jointly trained with the encoder to minimize distortion.
  • Learning the generative model: Wasserstein distance is used because its transportation cost can match the distortion measure, allowing the reconstruction distortion to be related analytically to the generator's distribution mismatch.
  • Learning the generative model: WGAN provides a dual optimization route, while WAE factorizes couplings through an encoder; the proposed procedure first learns G⋆ and then learns B ◦ E with G⋆ fixed.
  • Reconstructing the Wasserstein distance: Theorem 1 shows that the procedure's distortion approaches Wd(PX, PG⋆(Z)) with an additive error decaying exponentially in R, while distribution mismatch is determined by G⋆ and independent of R.
  • Reconstructing the Wasserstein distance: Using a generator trained with an f-divergence can yield larger asymptotic distortion than one minimizing Wasserstein distance defined by the reconstruction distortion.

4 Unsupervised training via Wasserstein++

The unsupervised training procedure parameterizes the generator, stochastic function, and encoder with neural networks and enforces latent-prior matching. Wasserstein++ combines WGAN and WAE objectives to balance sample sharpness, mode coverage, and encoder-compatible latent structure.

  • The generator, stochastic function, and encoder are parameterized as deep neural networks and trained from data using stochastic gradient descent.
  • The constraint B(E(X)) ∼ PZ can be enforced with MMD or GAN-based regularization, which is easier than matching the complex data distribution directly.
  • Wasserstein++: WAE supplies an encoder-compatible latent structure, whereas WGAN does not necessarily produce such structure, particularly in high-dimensional latent spaces.
  • Wasserstein++: WAE tends to produce less sharp samples than WGAN but is less prone to mode dropping because its reconstruction term penalizes missing modes.
  • Wasserstein++: Wasserstein++ combines the primal and dual Wasserstein forms through a convex combination and alternates critic updates with generator and encoder updates.
  • Wasserstein++: The critic is trained on interpolated latent samples, while the generator is trained on encoder-produced samples for both WGAN and WAE loss terms.
  • Wasserstein++: Training the critic on interpolated latent samples is described as introducing robustness to latent-space distribution mismatch.

5 Empirical evaluation3

The evaluation measures reconstruction distortion, distributional fidelity, and conditional variability on CelebA and LSUN bedrooms across bitrates. DPLC models generally trade increasing stochastic image generation at lower rates against reconstruction quality, with Wasserstein++ often providing the strongest distributional fidelity.

  • Evaluation setup: The evaluation uses MSE for reconstruction quality, FID for distribution matching, and conditional pixel variance for reconstruction variability.Testing metrics are computed on held-out data; PV estimates conditional variability across repeated reconstructions.
  • Generator quality: Wasserstein++ produces lower sample and reconstruction FID than WAE but slightly higher sample FID than WGAN-GP on both datasets.The comparison reflects a trade-off between distributional fidelity and reconstruction distortion in generator training.
  • CelebA: On CelebA, BPG obtains the lowest MSE overall, while WAE-based DPLC obtains the lowest MSE among the DPLC models.The comparison does not include advanced context-model entropy coding for BPG.
  • CelebA: On CelebA, DPLC with Wasserstein++ attains the lowest reconstruction FID among DPLC variants, while its FID decreases as bitrate increases.CAE and BPG exhibit sharp low-rate FID degradation associated with blur and compression artifacts, respectively.
  • Stochastic decoding: For all DPLC models, conditional pixel variance increases as bitrate decreases, progressing from little variability toward generated image content and random faces.GC has essentially zero PV except at zero bitrate, where it is comparable to DPLC.
  • LSUN bedrooms: On LSUN bedrooms, Wasserstein++ provides the lowest FID by a large margin at all positive rates, whereas WAE has high reconstruction FID because it models the dataset poorly.WGAN-GP follows the MSE and PV trends but its reconstruction FID increases notably as bitrate decreases, plausibly because of mode coverage issues.
  • Baseline comparison: GC uses injected noise for stochastic decoding at low rates but suffers mode collapse and substantially higher low-rate reconstruction FID than Wasserstein++.At high bitrates, GC largely ignores the injected noise.

6 Related work

Related work spans neural compression, GAN-based image compression, hierarchical generative representations, and distribution-preserving quantization. The paper distinguishes DPLC by combining rate-distortion optimization with an explicit distribution constraint and stochastic decoding.

  • Neural compression: DNN compression methods have been applied to images, speech, and video, with compressive autoencoders and recurrent networks among the dominant architectures.These methods can adapt to domains and support low-bitrate compression through learned representations.
  • GAN-based compression: Prior GAN-based compression work uses adversarial losses for artifact suppression or image-content generation, but does not demonstrate a properly working stochastic decoder in the cited approach.Other work uses GAN generators with deterministic distortion minimization rather than solving the DPLC problem.
  • Generative representations: Hierarchical VAE-style representations partially preserve and partially generate image content but lack an explicit rate-distortion formulation and quantized code.This distinguishes that framework from the rate-constrained DPLC setup.
  • Generative-model combinations: Wasserstein++ is conceptually related to VAE-GAN but combines WAE and WGAN objectives that minimize the same Wasserstein cost function.The cited comparison presents this shared cost as making Wasserstein++ more principled conceptually.
  • Distribution-preserving quantization: Earlier distribution-preserving quantization work studies moment preservation and dither-based methods that preserve a variable’s distribution.These approaches concern scalar quantization outside machine learning.

7 Conclusion

The paper formulates distribution-preserving lossy compression and develops generative-model-based solutions, including Wasserstein++. Its systems span generative modeling at zero bitrate to near-perfect reconstruction at high bitrate while producing stochastic low-bitrate decoders.

  • Problem: DPLC optimizes the rate-distortion tradeoff while requiring reconstructed samples to follow the training-data distribution.This is the paper’s central problem formulation.
  • Approach: Wasserstein++ combines WAE and WGAN, and the paper analytically characterizes the resulting compression systems.The conclusion identifies Wasserstein++ as a principal proposed approach.
  • Results: The systems cover the spectrum from learning a data generative model at zero bitrate to almost perfect reconstruction at high bitrate.The paper reports essentially artifact-free reconstructions across rates.
  • Takeaway: The framework improves over previous methods by producing stochastic decoders at low bitrates, thereby addressing the DPLC objective.The conclusion presents this as the first effective solution claimed by the paper.
  • Future scope: Future work includes scaling the approach to full-resolution images and applying it to data types beyond images.These directions mark the stated scope boundary of the current study.

A Proof of Theorem 1

The proof constructs a rate-constrained stochastic encoder-decoder whose generated outputs preserve the prior distribution, then bounds distortion using nearest-neighbor quantization. It also notes that optimal vector quantization is generally computationally difficult.

  • The proof constructs a stochastic function ˆF by quantizing F⋆(X) with a nearest-neighbor quantizer and sampling within the corresponding Voronoi region.The encoder uses the quantizer output, while B samples from the prior conditioned on that region.
  • The construction preserves the prior distribution because B(E(X)) is distributed as PZ, making G⋆(B(E(X))) distributed as G⋆(Z).
  • The distortion bound follows by upper-bounding d(X,G⋆(ˆF(X))) and the quantization error, then taking expectations and applying the stated inequalities.The lower bound follows because the optimization includes rate-constrained mappings B ◦ E.
  • Optimal vector quantization is generally NP hard, although stronger assumptions on PZ can yield exponential convergence without optimal quantization.

B Hyperparameters and architectures

The experiments specify separate training settings for WAE, WGAN-GP, Wasserstein++, the DPLC components, and the CAE baseline, using convolutional architectures with normalization and nonlinearities.

  • Hyperparameters: The DPLC function B ◦ E and CAE baseline use separate training settings, with CAE removing the MMD regularization coefficient.
  • Architectures: The architecture notation specifies convolutional kernel size, stride, filter count, normalization, and activation choices.Suffixes b and l denote batch normalization and layer normalization with leaky ReLU, respectively.
  • Architectures: The listed networks include convolutional encoders and decoders, a generator G with transposed convolutions, and bottleneck mappings with fully connected layers.
  • Hyperparameters: WAE, WGAN-GP, and Wasserstein++ use listed Adam rates, regularization, batch sizes, iteration counts, and learning-rate schedules, with LSUN training doubling iterations.For LSUN bedrooms, λMMD is 300 for WAE and Wasserstein++, and schedules are scaled accordingly.

C The Wasserstein++ algorithm

Wasserstein++ jointly initializes a WAE encoder, generator, and WGAN discriminator, then mixes encoded and prior latent samples before generating interpolated data samples.

  • The procedure uses MMD regularization, a WGAN coefficient, gradient-penalty coefficient, critic-iteration count, minibatch size, a characteristic kernel, and Adam parameters.
  • Wasserstein++ initializes the WAE encoder Fφ, generator Gθ, and WGAN discriminator parameters before training.
  • Each minibatch samples training data, encoded latent variables from Fφ(xi), and independent latent variables from the prior PZ.
  • The algorithm interpolates encoded and prior latent samples using ηi and νi before generating samples with Gθ.The resulting latent variables and generated outputs combine reconstruction-oriented and prior-driven components.

D Visual examples

Visual examples compare random generations and DPLC reconstructions across CelebA and LSUN bedrooms at multiple rates. Wasserstein++ produces sharp reconstructions, while other methods show blur, reduced variability, or mode collapse in specific settings.

  • Random samples: WGAN-GP and Wasserstein++ generate sharper random CelebA and LSUN bedroom samples than WAE.
  • CelebA: For CelebA, WAE, WGAN-GP, and Wasserstein++ DPLC reconstructions become more variable as bitrate decreases.
  • CelebA: The CelebA baseline using G ◦ B ◦ E shows no reconstruction variability except at 0 bpp, while CAE reconstructions become increasingly blurry as rate decreases.
  • LSUN bedrooms: For LSUN bedrooms, WAE reconstructions are blurry at all rates, whereas WGAN-GP is blurry except at 0 bpp and Wasserstein++ remains quite sharp.
  • LSUN bedrooms: The LSUN baseline produces a stochastic decoder at very low rates but suffers from mode collapse, while CAE becomes increasingly blurry as rate decreases.
Loading 1805.11057v2…