PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Compression

Shibani Santurkar, David Budden, Nir Shavit

arXiv:1703.01467v2cs.CV

TL;DR

Traditional codecs lack adaptability and can fail to degrade gracefully, motivating compression methods that use generative models. The paper develops generative image and video codecs, reporting improved compression and noise tolerance while noting limits in demonstrated image complexity and perceptual objectives.

  • Problem

    Traditional hand-crafted codecs are data-agnostic and lack adaptability, while pixel-level objectives can produce perceptually implausible blurry reconstructions.

  • Method

    The paper pre-trains a generative decoder, trains an encoder against it, and extends compression to video through latent-space interpolation and coding.

  • Results

    Generative compression shows orders-of-magnitude improvement in image and video compression factor and noise tolerance compared with traditional schemes.

  • Takeaways & Limitations

    Generative compression can provide visually pleasing reconstructions at deeper compression levels and graceful degradation because transmitted data describes images using shared semantic understanding.

  • Takeaways & Limitations

    The demonstrated generative compression results use relatively simple 64x64 examples, leaving generalization to larger images and more complex semantics as an open question.

Abstract

from arXiv · show

Traditional image and video compression algorithms rely on hand-crafted encoder/decoder pairs (codecs) that lack adaptability and are agnostic to the data being compressed. Here we describe the concept of generative compression, the compression of data using generative models, and suggest that it is a direction worth pursuing to produce more accurate and visually pleasing reconstructions at much deeper compression levels for both image and video data. We also demonstrate that generative compression is orders-of-magnitude more resilient to bit error rates (e.g. from noisy wireless channels) than traditional variable-length coding schemes.

1 Introduction

Traditional codecs use hand-crafted transforms that lack adaptability, while generative compression aims to preserve meaningful quality as bandwidth drops or errors occur, potentially at much deeper compression levels.

  • 1 Introduction: Traditional codecs are agnostic to compressed data and do not degrade gracefully when bandwidth drops or transmission errors occur.Graceful degradation means user experience worsens but remains meaningful.
  • 1 Introduction: Orders-of-magnitude improvement may be possible for lossy compression through generative compression compared with existing algorithms.The paper focuses on lossy compression because it relaxes exact reconstruction requirements.
  • 1 Introduction: Rate-distortion compression maps inputs into latent codes, quantizes those codes, and entropy-codes them before synthesis reconstruction.The objective balances latent-code rate against output distortion.
  • 1 Introduction: Hand-crafted linear codecs may be suboptimal for natural images, may not generalize to emerging media, and can take years to develop and standardize.Examples include JPEG’s discrete cosine transforms and JPEG2000’s multi-scale wavelets.

2 Generative Models for Image Compression

Neural generative models offer alternatives to hand-crafted compression transforms, but GANs lack an encoder and likelihood-based autoencoders can produce blurry reconstructions.

  • 2 Generative Models for Image Compression: Learned neural codecs replace hand-crafted analysis and synthesis transforms with encoder and decoder functions trained from data.Examples include compressive autoencoders, nonlinear gain-control models, and LSTM-based or fully convolutional approaches.
  • 2 Generative Models for Image Compression: Generative decoders can transform samples from a prior latent distribution into realistic images, with GAN training using an auxiliary discriminator.The discriminator distinguishes samples from the true data distribution and generated distribution.
  • 2 Generative Models for Image Compression: GANs can reconstruct high-quality images from latent codes but lack the inference function needed to encode images into those codes.They map from latent space to image space, but not vice versa.
  • 2 Generative Models for Image Compression: VAEs provide an encoder by approximating the posterior latent distribution, but Gaussian-likelihood training is equivalent to pixel-level L2 loss and leads to blurry reconstructions.Pixel-level L2 loss correlates poorly with human perception.

3 Neural Codecs for Generative Compression

The proposed neural codecs pre-train generative decoders and then learn encoders against them, extending the approach to video through latent-space interpolation and delta coding.

  • 3 Neural Codecs for Generative Compression: NCode first adversarially pre-trains a decoder as a generative model, then trains an encoder against that fixed decoder using a distortion loss.The architecture uses DCGAN-style convolutional networks and also investigates lossy latent quantization.
  • 3 Neural Codecs for Generative Compression: NCode combines pixel distortion with perceptual features from a pretrained convolutional network to improve reconstructed-image plausibility.The fourth convolutional layer of ImageNet-pretrained AlexNet supplies the perceptual feature representation.
  • 3 Neural Codecs for Generative Compression: The model differs from related hybrid approaches through its non-adaptive adversarially pretrained decoder, hybrid perceptual/pixel loss, and vanilla autoencoder design.Unlike an ALI-based pipeline, NCode explicitly includes pixel-level distortion to preserve similarity to the input.
  • 3.1 Generative Video Compression: The paper presents neural video compression at sub-MPEG rates and models missing frames by interpolating along paths in a learned latent manifold.The approach assumes sufficiently small temporal gaps can be approximated by linear interpolation in latent space.
  • 3.1 Generative Video Compression: Video latents can be further compressed by transmitting frame-to-frame differences and applying Huffman entropy coding.The paper reports greater entropy-coding gains for latent differences than for independently sampled latent vectors.

4 Experiments

Experiments evaluate NCode for images and MCode for video against conventional codecs, emphasizing perceptual quality, deep compression, and noisy-channel robustness. NCode improves visual or semantic reconstruction at comparable or substantially higher compression, while MCode exploits temporal latent-space structure for further bitrate reduction.

  • Experimental setup: The benchmarks use CelebA, UT Zappos50K, MIT Places, and KTH actions, comparing neural codecs with JPEG/JPEG2000, Toderici et al., and MPEG4 (H.264).Image performance uses PSNR, SSIM, visual inspection, and CIFAR-10 classification; video performance uses mean frame-wise PSNR and SSIM.
  • NCode image compression: NCode(100,5) yields higher-quality reconstructions than JPEG/2000 at approximately 4-fold higher compression, while NCode(25,4) reaches an order-of-magnitude greater compression with recognizable images.NCode(25,2) typically remains plausible with respect to underlying class semantics even under over-compression.
  • NCode image compression: At approximately 190-fold compression, NCode(25,4) produces more recognizable CIFAR-10 images than Toderici et al. at approximately 150-fold and JPEG/2000 at approximately 15-fold compression.Recognizability is measured using a ConvNet trained independently on uncompressed CIFAR-10 images.
  • Robustness to noisy channels: NCode remains comparatively robust to bit errors: at ε = 10^-2, PSNR degrades by only approximately 1dB, whereas JPEG loses more than 7dB at ε ∼10^-4.The robustness analysis attributes the contrast to explicit latent-space coordinates versus variable-length entropy-coded representations.
  • MCode video compression: At comparable compression levels, frame-by-frame MCode provides higher-quality video than MPEG4, with better preservation of background texture and limb sharpness despite similar PSNR.The comparison uses KTH hand-waving sequences and mean frame-wise PSNR/SSIM averaged over test videos.
  • MCode video compression: Interpolating between latent vectors enables order-of-magnitude bitrate reduction over MPEG4 while producing more visually plausible sequences.Transmitting Huffman-encoded latent differences provides a further 20%-50% lossless compression on average; Figure 5 evaluates N = 2, 4, and 8.
  • Robustness to noisy channels: MCode offers a robustness-versus-compression choice by operating with or without inter-frame Huffman coding, which adds 20%-50% performance improvement on average.The robustness analysis extends to video MCode when inter-frame entropy coding is absent.

5 Large Image Compression

Generative compression is extended from simple 64x64 examples toward larger, more complex images, but its current large-image limitations largely reflect instability in generative modeling. Improved GANs already produce recognizable 256x256 ImageNet reconstructions.

  • The earlier demonstrations used relatively simple 64x64 images, leaving generalization to larger images with more complex class semantics as an open question.
  • Current limitations are attributed to generative modeling generally rather than to generative compression itself.
  • Adversarial-training instabilities cause GANs to fail for large images.
  • On 256x256 ImageNet animal-subset test samples, Wasserstein GAN produces recognizable reconstructions unlike the vanilla DCGAN used throughout the paper.

6 Discussion

The paper frames generative compression as an alternative to hand-crafted codecs, using generative models to constrain reconstructions to semantically appropriate natural images. It reports potential orders-of-magnitude gains in compression factor and noise tolerance, graceful degradation at deeper compression, and a possible cached-manifold path toward practical deployment.

  • Generative compression replaces hand-crafted codecs by training the synthesis transform as a generative model and using it as a non-adaptive decoder in an autoencoder.
  • The decoder confines reconstructions to a compact set of natural images enriched for the appropriate class semantics.
  • Orders-of-magnitude improvement is reported for image and video compression in both compression factor and noise tolerance versus traditional schemes.
  • Generatively compressed images degrade gracefully at deeper compression levels because transmitted data is a description relative to the receiver’s shared understanding of natural-image semantics.
  • Practical deployment could use synchronized cached manifolds representing common objects and scenes, avoiding explicit transmission of network weights for previously encountered concepts.
Loading 1703.01467v2…