Source-linked AI summary
Full Resolution Image Compression with Recurrent Neural Networks
George Toderici, Damien Vincent, Nick Johnston, Sung Jin Hwang, David Minnen, Joel Shor, Michele Covell
TL;DR
Neural image compression lacked evidence of a single network competitive across compression rates and arbitrary image sizes. This paper develops recurrent full-resolution compression architectures with entropy coding and reports better-than-JPEG average performance on MS-SSIM and PSNR-HVS rate-distortion AUCs, with and without entropy coding.
Problem
Evidence was limited for a single neural network competitive across compression rates and arbitrary image sizes, with prior work limited to 32×32 images.
Method
The paper combines recurrent encoder-decoder architectures, binarization, residual reconstruction, and entropy coding to compress full-resolution images at variable rates without retraining.
Results
On average, the models achieve better than JPEG performance on both MS-SSIM AUC and PSNR-HVS AUC, with and without entropy coding.
Takeaways & Limitations
The architectures provide a set of recurrent image-compression models that perform well across the evaluated perceptual metrics and coding settings.
Takeaways & Limitations
Existing perceptual metrics do not reliably correlate with human vision across distortion types, making it difficult to identify a best model.
Abstract
from arXiv · showhide
This paper presents a set of full-resolution lossy image compression methods based on neural networks. Each of the architectures we describe can provide variable compression rates during deployment without requiring retraining of the network: each network need only be trained once. All of our architectures consist of a recurrent neural network (RNN)-based encoder and decoder, a binarizer, and a neural network for entropy coding. We compare RNN types (LSTM, associative LSTM) and introduce a new hybrid of GRU and ResNet. We also study "one-shot" versus additive reconstruction architectures and introduce a new scaled-additive framework. We compare to previous work, showing improvements of 4.3%-8.8% AUC (area under the rate-distortion curve), depending on the perceptual metric used. As far as we know, this is the first neural network architecture that is able to outperform JPEG at image compression across most bitrates on the rate-distortion curve on the Kodak dataset images, with and without the aid of entropy coding.
1. Introduction
The section motivates neural image compression that remains competitive across compression rates and arbitrary image sizes. It proposes addressing both stronger patch-based residual encoding and entropy coding of long-term dependencies, while evaluating quality with human visual system–inspired measures.
- Prior work achieved better-than-state-of-the-art compression rates for a given quality regardless of input image, but was limited to 32×32 images.
- The paper targets a neural network competitive across compression rates on images of arbitrary size.
- It addresses both stronger patch-based residual encoding and entropy coding that captures long-term dependencies between image patches.
- Quality assessment should use a human visual system–inspired measure rather than relying only on PSNR or Lp differences.
- Earlier neural compression methods included variational recurrent autoencoders for images up to 64×64 pixels and non-variational recurrent networks for variable-rate encoding.
- This work extends common fixed-rate bottleneck approaches by supporting variable-rate compression beyond thumbnail-sized images.
2. Methods
The methods use a shared recurrent encoder–binarizer–decoder architecture that progressively encodes residuals into binary representations. The paper evaluates recurrent-unit variants and reconstruction frameworks, including a content- and iteration-dependent gain extension for additive reconstruction.
- Architecture: Each compression network comprises an encoding network, stateless binarizer, and decoding network, with recurrent components in the encoder and decoder.Inputs are encoded into binary codes for storage or transmission, and the decoder reconstructs the image.
- Recurrent components: The recurrent units use separate convolutions for the incoming input and recurrent hidden state, with larger 3×3 hidden kernels improving compression curves over 1×1 kernels.Hidden kernels are 3×3 in D-RNN#3 and D-RNN#4 and 1×1 elsewhere.
- Architecture: 1/8 bpp is produced per iteration from each 32×32×3 input, corresponding to 192:1 compression after one iteration before entropy coding.Each iteration reduces the image to a 2×2×32 binarized representation.
- Recurrent components: The experiments compare LSTM, Associative LSTM, and GRU-based units, finding Associative LSTMs effective only in decoders and using non-associative LSTMs in their encoders.The GRU formulation adds residual connections inspired by ResNet and Highway Networks.
- Reconstruction frameworks: The reconstruction frameworks include one-shot reconstruction, additive reconstruction, and a gain-extended additive architecture that scales each target residual using content- and iteration-dependent patch gains.One-shot predicts the full image at each iteration, whereas additive reconstruction sums residual reconstructions across iterations.
3. Entropy Coding
The entropy-coding section addresses submaximal binary-code entropy by adding lossless, fully convolutional coding that processes codes progressively and estimates conditional probabilities for arithmetic coding. It introduces BinaryRNN, based on PixelRNN, to model spatial and temporal dependencies in binary codes, while noting that multi-iteration coding must capture redundancy between iterations.
- Motivation: Entropy coding can improve compression because generated codes are not necessarily maximal-entropy or able to exploit large-scale visual redundancy.The image encoder is treated as a fixed binary-code generator in this section.
- Coding framework: The lossless entropy coders are fully convolutional and process binary codes progressively, using raster-scan order within each encoding iteration.Codes have the form c(y, x, d) with size H × W × D, where D is m × the number of iterations.
- Coding framework: Conditional probabilities P(c(y, x, d) | T(y, x, d)) are estimated from prior stream-order bits, with arithmetic coding producing compression lengths governed by cross entropy.The context T(y, x, d) depends only on previous bits in stream order.
- BinaryRNN: BinaryRNN adapts PixelRNN to single-layer binary-code compression, combining neighboring codes with recurrent state summaries that capture short-term and long-term dependencies.Its state line has size 1 × W × k and carries information from previously decoded binary codes.
- Multiple iterations: For multiple iterations, duplicating the single-iteration coder would miss redundancy between iterations, motivating information transfer from previous layers into the iteration-specific line LSTM.The baseline assigns each iteration its own line LSTM.
4. Results
Results on Kodak show that model rankings depend on the perceptual metric, bitrate, training data, and reconstruction architecture. High-Entropy training generally improves performance, while progressive entropy coding benefits all models, with larger gains expected for large images.
- Training data: High-Entropy training generally improved models over the 32×32 dataset, indicating the value of training on hard-to-compress examples.The High Entropy dataset was formed by selecting tiles with the worst PNG compression ratios.
- Architecture results: On 32×32 data, GRU with One Shot reconstruction ranked highest on both MS-SSIM and PSNR-HVS.LSTM with Residual Scaling ranked second in MS-SSIM, while Residual GRU ranked second in PSNR-HVS.
- Architecture results: On High-Entropy data, LSTM One Shot achieved the highest MS-SSIM but the worst PSNR-HVS, whereas Residual GRU One Shot achieved the highest PSNR-HVS.GRU One Shot ranked second in both metrics, illustrating that the metrics do not always agree.
- Entropy coding: All models benefited from progressive entropy coding, with best-model gains of 5% at 2 bpp and 32% at 0.25 bpp on Kodak.The Kodak gains were limited by the dataset’s relatively low-resolution images, while larger images are expected to benefit more.
- Entropy coding: Baseline LSTM entropy coding reduced bitrate by 25% at 2 bpp and 57% at 0.25 bpp.The progressive entropy coder was trained for a specific image encoder.
5. Discussion
The paper presents RNN-based compression models that outperform JPEG on average across both MS-SSIM and PSNR-HVS AUC, with and without entropy coding. It also identifies video-derived codecs and jointly trained, larger-patch entropy coding as important next challenges.
- Discussion: The proposed architecture combines RNN compression, content-based residual scaling, and a new GRU variation, whose models achieved the highest PSNR-HVS on the high-entropy training dataset.Existing perceptual metrics do not fully capture the models’ distortions, making it difficult to identify a single best model.
- Discussion: On average, the models outperform JPEG on both MS-SSIM AUC and PSNR-HVS AUC, with and without entropy coding.Entropy coding improves performance because early recurrent encoder iterations produce spatially correlated codes.
- Discussion: Future work targets video-derived codecs such as WebP and joint training of the BinaryRNN entropy coder with the patch-based encoder on larger patches.Larger patches could balance patch-encoder efficiency against the entropy coder’s predictive power.