Source-linked AI summary
Lossy Image Compression with Compressive Autoencoders
Lucas Theis, Wenzhe Shi, Andrew Cunningham, Ferenc Huszár
TL;DR
The paper addresses the difficulty of optimizing autoencoders for flexible lossy image compression because compression includes non-differentiable operations. It introduces differentiable approximations for training while retaining efficient architectures, achieving performance similar to or better than JPEG 2000 and enabling optimization for specific content and metrics.
Problem
Lossy image compression is difficult to optimize with autoencoders because quantization and coding costs are non-differentiable, despite the need for more flexible codecs.
Method
The method uses continuous differentiable approximations for discrete quantization and entropy-rate estimation while retaining rounding-based quantization in the compression pipeline.
Results
The approach achieves performance similar to or better than JPEG 2000 for perceptual quality; CAE significantly outperformed JPEG 2000 at 0.375 bpp and 0.5 bpp in MOS.
Takeaways & Limitations
The end-to-end trained framework can be optimized for specific content and arbitrary metrics, using efficient architectures suitable for large-image decoding.
Takeaways & Limitations
Results are highly image dependent, with individual-image results provided separately.
Abstract
from arXiv · showhide
We propose a new approach to the problem of optimizing autoencoders for lossy image compression. New media formats, changing hardware technology, as well as diverse requirements and content types create a need for compression algorithms which are more flexible than existing codecs. Autoencoders have the potential to address this need, but are difficult to optimize directly due to the inherent non-differentiabilty of the compression loss. We here show that minimal changes to the loss are sufficient to train deep autoencoders competitive with JPEG 2000 and outperforming recently proposed approaches based on RNNs. Our network is furthermore computationally efficient thanks to a sub-pixel architecture, which makes it suitable for high-resolution images. This is in contrast to previous work on autoencoders for compression using coarser approximations, shallower architectures, computationally expensive methods, or focusing on small images.
1 INTRODUCTION
The paper targets flexible lossy compression with autoencoders, addressing both changing media requirements and the non-differentiability that makes direct optimization difficult. It reports performance similar to or better than JPEG 2000 while using efficient neural-network architectures.
- Neural networks had not surpassed existing codecs for lossy image compression, although autoencoders had shown promising results mainly on small images.
- Autoencoders could support compression tailored to computational, bandwidth, content, and media-format requirements that vary across applications.Examples include asymmetric encoder-decoder complexity for server-to-device communication and adaptation to lightfield, 360-video, or VR content.
- Quantization makes lossy compression non-differentiable, complicating gradient-based training of neural networks.The paper therefore aims to optimize the autoencoder’s rate-distortion tradeoff directly while approximating rounding-based quantization and coding costs.
- The proposed approach achieves performance similar to or better than JPEG 2000 for perceptual quality.
- The framework uses efficient neural-network architectures intended to support near-real-time decoding of large images on low-powered consumer devices.
2 COMPRESSIVE AUTOENCODERS
The compressive autoencoder combines learned encoding, decoding, and entropy modeling with differentiable approximations that enable rate–distortion optimization despite quantization and discrete coding. The approach retains ordinary quantization in the forward pass, estimates rates continuously, and supports flexible rate control and deep convolutional architectures.
- A compressive autoencoder consists of an encoder f, decoder g, and probabilistic model Q for entropy coding.Q assigns bit costs to representations according to their frequencies.
- Quantization remains in the forward pass, while its derivative is replaced with that of a smooth approximation during backpropagation.This preserves the information bottleneck while making gradient-based training possible.
- Quantization and differentiable alternatives: The method preserves exact decoder gradients under a single forward and backward pass, unlike alternatives that require separate rounded and approximate decoder passes.The decoder gradient is exact when the distortion function d is differentiable.
- Entropy rate estimation: A continuous differentiable density q upper-bounds the non-differentiable bit cost, allowing encoder training through sampled uniform noise.The resulting estimate is differentiable in the latent representation when q is differentiable.
- Rate control: Scale parameters λ let a pretrained autoencoder be finetuned for different rate–distortion tradeoffs without storing many separate models.Scales can be shared across spatial dimensions and interpolated for finer bit-rate control.
- Related work: Compared with related approaches, the framework uses standard deep convolutional networks, entropy coding, and direct rate–distortion optimization rather than recurrent or variational formulations.The cited alternatives include RNN-based progressive coding and variational autoencoders with restrictions on equivalent distortion formulations.
3 EXPERIMENTS
The experiments evaluate the compressive autoencoder’s architecture, training procedures, rate control, and compression quality against established codecs and an RNN-based method. The method achieves strong perceptual results, while image-dependent artifacts and encoding-time costs remain practical considerations.
- Architecture: The decoder uses sub-pixel convolution layers to upsample efficiently after residual processing.Sub-pixel convolution reorganizes many-channel tensors into outputs with fewer channels and larger spatial extent.
- Training: Incremental training activates coefficient-mask entries progressively, then reduces the learning rate from 10^-4 to 10^-5 after all coefficients are enabled.Networks were trained for up to 10^6 updates, although good performance was usually reached earlier.
- Rate control: Fine-tuning scale parameters and interpolating nearby scales provides control over additional rate-distortion trade-offs without retraining every model from scratch.All other network parameters remain fixed during scale fine-tuning.
- Quantitative evaluation: In terms of SSIM, the method outperforms all other tested methods, while PSNR is slightly worse than JPEG 2000 at low and medium bit rates and slightly better at high bit rates.MS-SSIM scores are similar across methods except at very low bit rates, and results are highly image dependent.
- Qualitative evaluation: At low bit rates, CAE can produce sharper lines and fewer artifacts than competing methods but can also create hair artifacts and skin discolouration; at higher rates, these problems disappear.CAE artifacts are described as smoother and sometimes Gabor-filter-like than JPEG 2000 artifacts.
- Perceptual evaluation: CAE and JPEG 2000 achieve higher mean opinion scores than JPEG and the RNN-based method at all tested bit rates, with CAE significantly exceeding JPEG 2000 at 0.375 bpp and 0.5 bpp.The reported significance levels are p < 0.05 and p < 0.001, respectively.
4 DISCUSSION
The paper presents end-to-end trained compressive autoencoders that reach JPEG 2000-level performance on high-resolution images while supporting flexible optimization. It identifies future work in optimizing for perceptually meaningful metrics.
- The architecture achieves better performance than JPEG 2000 on SSIM and MOS scores.The result uses an efficient convolutional architecture, simple rounding-based quantization, and simple entropy coding.
- End-to-end training enables optimization for arbitrary metrics and is demonstrated at this performance level on high-resolution images.The authors describe this as the first such demonstration to their knowledge.
- Perceptually relevant metrics suitable for optimization remain underdeveloped across varied distortions and image pairs.Existing metrics correlate well with human perception for certain distortion types, but broader optimization is more challenging.
- Future work will explore optimizing compressive autoencoders for different metrics, including neural-network-based metrics and GAN-based alternatives.
A.1 GRADIENT OF CLIPPING
The clipping gradient is redefined so optimization remains informative when reconstructed pixel values exceed the valid range. This aligns the backward-pass signal with test-time squared error.
- The clipping operation is assigned a constant gradient during backpropagation.
- For squared loss, the redefined gradient preserves an error signal when the reconstructed value exceeds 255.Without redefinition, clipping would produce a zero error signal; without clipping, the signal would depend on irrelevant values above 255.
- The backward-pass signal is proportional to the test-time error incurred after clipping.
A.2 DIFFERENT MODES OF JPEG
The evaluation compares optimized and unoptimized JPEG across two chroma-subsampling modes on the Kodak dataset. Chroma subsampling and image-specific Huffman optimization improve average performance.
- The comparison covers JPEG 4:2:0 and 4:4:4 modes, each with optimized and predefined Huffman tables.Optimized JPEG stores an image-specific Huffman table, while unoptimized JPEG uses a predefined table.
- On average, chroma-subsampled and optimized JPEG perform better on the Kodak dataset.Header bits are excluded, but optimized JPEG includes the bits required to store its Huffman table.
A.3 COMPRESSION VS DIMENSIONALITY REDUCTION
The analysis separates information loss from dimensionality reduction and quantization, while also comparing JPEG modes and rate-specific CAEs. It finds quantization is the dominant visible source of degradation.
- Compression vs Dimensionality Reduction: Dimensionality reduction has minimal visual effect compared with quantization in compressive-autoencoder reconstructions.The encoder output can be 3× smaller than the input while dimensionality-reduction effects remain almost imperceptible.
- Different Modes of JPEG: Figure 7 compares JPEG 4:2:0, JPEG 4:4:4, and optimized versions of both modes on the Kodak dataset.Optimized Huffman tables perform better than default Huffman tables.
- Compression vs Dimensionality Reduction: Quantization introduces visible artefacts in the reconstructed images.
- Compression vs Dimensionality Reduction: The rate-distortion objective is rescaled to place its rate-control parameter in a more intuitive range.The objective combines a coefficient-based rate term with a distortion term involving the reconstructed image.
- Compression vs Dimensionality Reduction: The high-, medium-, and low-bit-rate CAEs use different α values and output-channel counts.High and medium bit rates use α = 0.01 and α = 0.05 with 96 channels; low bit rate uses α = 0.2 with 64 channels.
A.5 COMPARISON WITH VAE
The paper compares its approach with training using additive uniform noise instead of rounding, under a shared autoencoder architecture at high bit-rates.
- Replacing rounding with additive uniform noise during training provides an alternative approach attributed to Ballé et al. (2016).
- With mean-squared error distortion, this rate-distortion optimization is equivalent to training a variational autoencoder with a Gaussian likelihood and uniform encoder.
- Both approaches use the same autoencoder architecture and training procedure for high bit-rates.
A.6 COMPLETE IMAGES
The section presents complete images corresponding to the crops in Figure 5, comparing original images with reconstructions from three compression methods.
- Each example shows the original image at top left and reconstructions from CAE, JPEG 2000, and Toderici et al. (2016b).
- The CAE reconstruction is placed at top right, JPEG 2000 at bottom left, and Toderici et al. (2016b) at bottom right.
- The complete images are intended to be viewed on a monitor screen.