PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Adversarial Networks for Extreme Learned Image Compression

Eirikur Agustsson, Michael Tschannen, Fabian Mentzer, Radu Timofte, Luc Van Gool

arXiv:1804.02958v3cs.CVcs.LG

TL;DR

Deep compression systems are typically optimized for distortion metrics, motivating objectives that better capture visual quality at very low bitrates. This paper proposes GAN-based generative compression, achieving strong user-study preferences and additional storage savings when semantic label maps enable selective synthesis.

  • Problem

    Deep compression systems are typically optimized for PSNR or MS-SSIM, motivating training objectives beyond these traditional distortion metrics for very low bitrates.

  • Method

    The paper develops a principled GAN framework for full-resolution compression, combining encoder-decoder generative compression with adversarial training and extending it conditionally using semantic label maps during discriminator training.

  • Results

    Generative compression achieves state-of-the-art visual quality in user studies, with models preferred to BPG despite BPG using up to 95%, 124%, and 181% more bits across evaluated datasets and domains.

  • Takeaways & Limitations

    At low bitrates, generative compression can provide dramatic bitrate savings, while domain restriction and selective synthesis with semantic labels provide additional storage savings.

  • Takeaways & Limitations

    For selective generative compression, the semantic label map requires 0.036 bpp on average for downscaled 1024×512px Cityscapes images, creating relatively large overhead compared with preserved image content.

Abstract

from arXiv · show

We present a learned image compression system based on GANs, operating at extremely low bitrates. Our proposed framework combines an encoder, decoder/generator and a multi-scale discriminator, which we train jointly for a generative learned compression objective. The model synthesizes details it cannot afford to store, obtaining visually pleasing results at bitrates where previous methods fail and show strong artifacts. Furthermore, if a semantic label map of the original image is available, our method can fully synthesize unimportant regions in the decoded image such as streets and trees from the label map, proportionally reducing the storage cost. A user study confirms that for low bitrates, our approach is preferred to state-of-the-art methods, even when they use more than double the bits.

1. Introduction

The paper proposes GAN-based generative compression for full-resolution images below 0.1 bpp, synthesizing details that cannot be stored while preserving overall content. User studies report strong visual-quality advantages over established codecs at extreme bitrates, and semantic conditioning enables selective synthesis with additional bitrate savings.

  • Motivation: Deep compression systems are commonly optimized for PSNR or MS-SSIM, motivating training objectives beyond traditional distortion metrics at very low bitrates.Adversarial losses are presented as a candidate because they capture global semantics and local texture.
  • Approach: The proposed framework applies GANs to full-resolution image compression at target bitrates below 0.1 bpp.The framework is described as a principled GAN framework for extreme image compression.
  • Generative compression: Generative compression preserves overall image content while synthesizing fine structures such as tree leaves and building windows.GC does not require semantic label maps during training or deployment.
  • Selective generative compression: Selective generative compression fully synthesizes selected image regions from semantic label maps while preserving user-defined regions in detail.The system can combine preserved and synthesized content across multiple object boundaries while preserving image semantics.
  • Selective generative compression: Over 50% bitrate reductions are achieved in selective generative compression without notably degrading image quality.The operation combines preserved image content with synthesized content in semantically constrained regions.

2. Related work

Prior learned compression systems use neural architectures and entropy models while typically optimizing classical distortion measures. Related adversarial approaches differ in objective design, evaluation, and training procedures from the paper’s principled GAN formulation.

  • Deep compression: Deep compression commonly uses autoencoders or recurrent neural networks to transform images into entropy-coded bitstreams.Context models are often used to capture the distribution of the bitstream and reduce coding rates.
  • Generative models: GANs generate realistic, sharp images, while conditional GANs support image-to-image translation across domains such as maps and satellite imagery.The related work notes that GANs can scale to high resolutions for some datasets.
  • Adversarial compression: One prior compression system adds adversarial training to an MS-SSIM-optimized model but provides no ablation study isolating the adversarial scheme’s benefit.The paper contrasts this with its own study of GAN formulation benefits against MSE and MS-SSIM objectives.
  • Adversarial compression: The paper’s GAN loss is distinguished from a prior pairwise real/fake classification loss by its interpretation in terms of divergences between probability distributions.The prior approach also uses heuristic loss reweighting and manually thresholded alternation between generator and discriminator training.
  • Adjacent applications: Other related work applies GANs to thumbnail compression, removes artifacts from engineered codecs, or uses adversarial training for single-image super-resolution.Spatial bitrate allocation based on saliency is also related to the paper’s selective compression mode.

3. Background

The background formulates learned compression as encoding images into quantized latent representations and balancing perceptual distortion against bitrate. GANs extend this framework by modeling image distributions, including conditional distributions when side information is available.

  • Generative Adversarial Networks: GANs train a generator and discriminator through a min-max objective to approximate the data distribution.The generator maps samples from a fixed prior toward the data distribution while the discriminator distinguishes generated from real data.
  • Conditional Generative Adversarial Networks: Conditional GANs give both generator and discriminator access to side information to model the conditional distribution of images.The side information is denoted s, and the modeled distribution is p_x|s.
  • Deep Image Compression: Deep compression encodes an image with an encoder and quantizer, then reconstructs it from the quantized latent representation using a decoder.The quantized representation is entropy-coded into a bitstream, while the decoder produces the reconstruction.
  • Deep Image Compression: The rate-distortion objective trades perceptual similarity against the entropy of the encoded representation.The entropy H(ŵ) measures the average number of bits, and β controls the bitrate contribution when entropy is estimated differentiably.
  • Deep Image Compression: The bitrate can instead be controlled through the entropy bound by setting β = 0 and adjusting the quantizer levels or latent dimensionality.This avoids modeling entropy explicitly as a loss term, though it may produce suboptimal bitrates.

4. GANs for extreme image compression

The proposed framework combines learned compression with adversarial generation under a bitrate constraint. It supports unconditional generative compression that preserves content while synthesizing unavailable detail, and selective modes that use semantic guidance to choose generated regions.

  • Generative Compression: The framework encodes x into a quantized representation ŵ, optionally adds noise, and uses a decoder/generator to reconstruct an image consistent with both the image distribution and encoded content.The architecture combines an encoder, quantizer, generator, and discriminator in a learned compression system.
  • Generative Compression: The objective balances distortion, GAN loss, and entropy-related terms through λ, while the discriminator continues optimizing the GAN divergence.The final objective has different dynamics from a standard GAN because the latent vector contains information about a real image.
  • Generative Compression: Constraining H(ŵ) prevents exact recovery, forcing the encoder and generator to balance image preservation against realistic synthesis.The paper describes this bitrate limitation as crucial to generative compression.
  • Generative Compression: GC synthesizes unavailable texture, such as tree detail, instead of representing it as a blurry region, while the distortion term also helps stabilize GAN training.The paper states that the distortion term tends to prevent mode collapse.
  • Generative Compression: GC (D+) gives a semantic label map only to the discriminator during training, so trained encoders and generators do not require semantics at encoding or decoding.The extension is conditional during training but keeps inference independent of the semantic map.
  • Selective Generative Compression: Selective generative compression uses a binary heatmap to preserve selected regions while fully synthesizing zero-mask regions constrained by semantic labels.Random-instance training preserves 25% of instances, whereas random-box training preserves a randomly positioned box.

5. Experiments

Experiments evaluate generative compression across natural-image and street-scene datasets against engineered and learned baselines. The evaluation emphasizes bitrate, perceptual quality, and semantic plausibility because conventional distortion metrics do not capture realistic synthesis well.

  • Architecture, Losses, and Hyperparameters: β = 0 and five quantization centers control the bitrate through the latent entropy bound, while arithmetic coding reduces experimental rates below that bound.For GC with C = 2 bottleneck channels, the bound gives 0.0181 bpp, and arithmetic coding yields 8.8% smaller bitrates than the upper bound.
  • Datasets: GC models are trained on 188k Open Images images and evaluated on Kodak and RAISE1K, while semantic-label experiments use Cityscapes.The study also investigates constrained domains and semantic information available during training.
  • Evaluation: Figure 3 compares bitrate in bpp and PSNR in dB for GC, BPG, and an MSE baseline on Cityscapes.GC reconstructions are described as sharper and more textured despite BPG and the MSE baseline having higher PSNR.
  • Baselines: The method is compared with BPG, AEDC, and an MSE-only baseline sharing GC’s architecture and training parameters.The Cityscapes AEDC model uses bottleneck depth C = 4 and achieves 0.07 bpp with slightly better MS-SSIM than BPG at the same bitrate on validation data.
  • Evaluation: Human evaluation is used because PSNR and MS-SSIM do not measure the realism of synthesized content or the perception-distortion trade-off.The user study compares GC models with BPG and AEDC using pairwise side-by-side judgments.
  • Semantic Quality: SC evaluation measures semantic preservation and plausible blending in synthesized regions rather than relying solely on classical distortion metrics.The objective matches the training goal for selective generative compression.

6. Results

Across natural-image and street-scene benchmarks, generative compression is preferred at extreme bitrates and can reduce storage further through selective synthesis. The results also show stronger semantic preservation and visual detail than classical or autoencoder baselines, while performance is most constrained for complex generated objects.

  • Generative compression: BPG uses 95% and 124% more bits than GC models preferred by users on Kodak and RAISE1K, respectively.These GC models use C = 4 and were trained on Open Images, despite distribution shift between training and testing data.
  • Generative compression: On Cityscapes, BPG uses 181% more bits than the C = 2 GC model preferred by users.For C = 8, BPG still requires 21–49% more bits than the GC models across all three data sets.
  • Generative compression: GC produces finer detail and more convincing tree, water, and sky textures than BPG, while BPG, AEDC, and MSE produce smoothing, blocking, or blur.Scenes involving humans remain the principal challenge for GC.
  • Selective generative compression: SC preserves semantics somewhat better than pix2pixHD, while BPG, AEDC, and MSE have considerably lower mean IoU.The comparison is reported on the Cityscapes validation set as a function of bitrate, including SC instance and box modes.
  • Selective generative compression: Selective synthesis reduces bitrate by 50% or more without notably degrading visual quality for repetitive structures such as trees, streets, and sky.The networks merge preserved and generated content across object boundaries; complex synthesized objects such as buildings and people look worse.
  • Selective generative compression: Semantic label maps cost 0.036 bpp on average for downscaled 1024×512px Cityscapes images, creating relatively large overhead versus preserved image content.The paper states that this cost vanishes as image size increases because the mask can be stored as a dimension-independent vector graphic.

7. Conclusion

The paper concludes that GAN-based generative compression achieves substantial bitrate savings at low rates, while selective synthesis enables additional savings when semantic label maps are available.

  • GAN-based generative compression delivers dramatic bitrate savings compared with methods optimized for classical objectives at low bitrates.The comparison is evaluated by visual quality in a user study.
  • Constraining the application domain to street scenes provides additional storage savings.
  • Selective generative compression combines fully synthesized image content with preserved regions when semantic label maps are available.
  • Future work includes controlling GC's spatial bit allocation and combining SC with saliency information to decide which regions to preserve.

A. Comparison with State-of-the-art

The paper compares generative compression with learned and engineered state-of-the-art methods using qualitative analysis, emphasizing visual quality at substantially lower bitrates.

  • The comparison table distinguishes methods by compression approach and evaluation metrics, including PSNR and MS-SSIM on Kodak.
  • Rippel et al. use 29–179% more bits, while the proposed models produce images of comparable or better quality.
  • BPG remains visually competitive with the current state-of-the-art, consistent with reported 8.41% bitrate savings in PSNR.
  • The proposed method produces comparable visual quality on some examples while using much fewer bits than the available images from the current state-of-the-art work.

B. Training Details

The models use ADAM optimization with fixed training schedules, while normalization differs between most training and the latter half of Open Images training.

  • ADAM is used with a learning rate of 0.0002 and mini-batch size 1.
  • The networks are trained for 150000 iterations on Cityscapes and 280000 iterations on Open Images.
  • Instance normalization is used except during the second half of Open Images training, when the generator/decoder uses fixed batch statistics.The authors found this reduced artifacts and color shift.

C. Data set and Preprocessing Details

The experiments use Open Images for GC training, Kodak and RAISE1K for evaluation, and Cityscapes for SC, with preprocessing and coding tailored to each setting.

  • GC training uses 200k randomly sampled Open Images images, reduced to an effective set of 188k after preprocessing.Images are rescaled and certain insufficiently downscaled or highly saturated images are discarded.
  • GC models are evaluated on 24 Kodak images and 20 randomly selected RAISE1K images, with images rescaled so their longer side is 768px.
  • SC evaluation relies on Cityscapes, using the same preprocessing as GC and requiring semantic label maps during training and deployment.
  • SC semantic label maps are encoded by quantizing vector-graphic coordinates, coding them relative to preceding boundary coordinates, and applying arithmetic coding.
  • Header sizes are excluded for all baseline methods to ensure fair comparison.

E. Architecture Details

The proposed architectures encode images and optional semantic maps into quantized latent codes, then decode them with generators trained against multi-scale adversarial discrimination. The SC variant uses semantic features for spatially informed generation and higher-resolution latent codes.

  • GC architecture: GC encodes an image and optional label map into a W/16 × H/16 × C feature map, with C ∈ {2, 4, 8}, then quantizes it over L centers.The encoder first produces 960 channels before projecting to C channels; the quantized code is stored.
  • GC architecture: The GC encoder uses six convolutional layers, including four 2-strided convolutions, followed by projection to C channels and quantization.The listed GC encoder is c7s1-60, d120, d240, d480, d960, c3s1-C, q.
  • SC architecture: SC extracts semantic-map features at the latent spatial resolution and concatenates them with the quantized image code before generation.Its encoder downsamples by 8×, producing dim(ˆw) = W/8 × H/8 × C, while the generator first reduces this representation to W/16 × H/16 × 960.
  • Adversarial training: Both GC and SC use a multi-scale discriminator that measures divergence between real-image and generated-image distributions locally and globally.The discriminator is based on the multi-scale architecture of [44].

F. Visuals

The visual evaluations compare the proposed models with BPG and prior learned compression methods across Kodak, RAISE1k, and Cityscapes. They also inspect latent samples, semantic synthesis, and reconstructions using estimated label maps.

  • Dataset comparisons: The user-study visual sections show the first five images from Kodak, RAISE1k, and Cityscapes alongside BPG outputs at similar bitrates.Figures 10–12 provide these dataset-specific examples, while the surrounding text identifies them as inputs to the user study.
  • Comparisons with [34]: Compared with [34], the model trades texture fidelity for fewer artifacts or smoother structures, with image-dependent differences in eyes, faces, and backgrounds.The comparisons report less texture but fewer artifacts on a knob, smoother jaw and hat lines but worse eye reconstruction, and smoother backgrounds with less bird texture.
  • Comparisons with [31] and BPG: Against [31] and BPG, the model sometimes produces sharper grass, sky, faces, and fewer ringing artifacts, but can reconstruct fences, lighthouses, or entire images worse.The visual comparisons explicitly describe both sharper or less blurry regions and cases with significantly worse reconstruction using only a fraction of the bits.
  • Latent representations: Uniformly sampled GC latent codes produce domain-specific or generic visual patches, whereas uniformly sampled MSE codes produce low-frequency noise.Cityscapes samples resemble street signs, buildings, and trees; Open Images samples are more colorful and generic.
  • Latent representations: Sampling the GC decoder with an improved WGAN generator yields sharper images than analogous sampling with an MSE model.The experiment learns the distribution of ˆw = E(x) and compares generated samples from the two models.
  • Semantic conditioning: SC visualizations demonstrate class-specific synthesis, seamless merging of preserved and generated content across object boundaries, and reconstructions driven by estimated semantic label maps.Examples synthesize vegetation, sky, sidewalk, ego vehicle, and wall, including cases where a preserved box crosses object boundaries.
Loading 1804.02958v3…