PaperlayerView exact revision on arXiv

Source-linked AI summary

Full Resolution Image Compression with Recurrent Neural Networks

George Toderici, Damien Vincent, Nick Johnston, Sung Jin Hwang, David Minnen, Joel Shor, Michele Covell

arXiv:1608.05148v1cs.CV

TL;DR

Neural compression had limited evidence of a single model competitive across compression rates and arbitrary image sizes. This paper develops RNN-based compression methods that, on average, outperform JPEG on both AUC MS-SSIM and AUC PSNR-HVS, with and without entropy coding.

  • Problem

    Prior work showed recurrent neural compression across rates but was limited to 32×32 images, leaving a gap in models competitive across rates and arbitrary image sizes.

    Source: §1 · p. 1

  • Method

    The paper combines a stronger patch-based residual encoder with an entropy coder designed to capture long-term dependencies between image patches.

    Source: §1 · p. 1

  • Results

    On average, the models outperform JPEG on both AUC MS-SSIM and AUC PSNR-HVS, with and without entropy coding.

    Source: §5 · p. 7

  • Takeaways & Limitations

    The authors report that their models perform well under the selected perceptual metrics but that the evidence does not identify a single best model.

    Source: §5 · p. 7

    Available perceptual metrics do not correlate well with human vision overall and may correlate only for particular types of distortion.

    Source: §5 · p. 9

Abstract

from arXiv · show

This paper presents a set of full-resolution lossy image compression methods based on neural networks. Each of the architectures we describe can provide variable compression rates during deployment without requiring retraining of the network: each network need only be trained once. All of our architectures consist of a recurrent neural network (RNN)-based encoder and decoder, a binarizer, and a neural network for entropy coding. We compare RNN types (LSTM, associative LSTM) and introduce a new hybrid of GRU and ResNet. We also study "one-shot" versus additive reconstruction architectures and introduce a new scaled-additive framework. We compare to previous work, showing improvements of 4.3%-8.8% AUC (area under the rate-distortion curve), depending on the perceptual metric used. As far as we know, this is the first neural network architecture that is able to outperform JPEG at image compression across most bitrates on the rate-distortion curve on the Kodak dataset images, with and without the aid of entropy coding.

1 Introduction

The paper targets a neural compressor that works across rates and arbitrary image sizes by improving both patch-based residual encoding and long-range entropy coding. It evaluates perceptual quality with PSNR-HVS and MS-SSIM and extends prior variable-rate methods beyond thumbnail-sized images.

1 Introduction

  • The paper targets compression across rates and arbitrary image sizes by strengthening patch-based residual encoding and entropy coding of long-term dependencies between patches.Toderici et al. [2016] demonstrated rate-competitive recurrent compression but only for 32×32 images and without modeling long-range patch dependencies.Source: §1 · p. 1
  • Because PSNR and Lp differences do not fully reflect human sensitivity to distortions, the authors evaluate with PSNR-HVS and MS-SSIM [Wang et al., 2003].They note that most perceptual metrics are non-differentiable, limiting direct optimization for them.Source: §1 · pp. 1–2

1.1 Previous Work

  • The work extends neural compression beyond thumbnail-sized images while supporting variable-rate encoding, unlike most networks that use a fixed rate determined by bottleneck size.Prior work applied variational recurrent autoencoders to images up to 64×64 pixels [Gregor et al., 2016] and used recurrent networks for variable-rate encoding [Toderici et al., 2016].Source: §1.1 · p. 2

2 Methods

The compression system uses recurrent encoders and decoders with a binarizer to encode residuals over successive iterations. The methods explore recurrent-unit variants and reconstruction schemes, including a content-dependent scaling extension to additive reconstruction.

2 Methods

  • Each iteration encodes the current residual into binary codes and decodes them into a reconstruction, with recurrent states carried forward while network weights are shared.The binarizer is stateless; for a 32×32×3 input, each iteration produces a 2×2×32 representation, or 1/8 bpp (192:1 compression after one iteration, before entropy coding).Source: §2 · pp. 2–3
  • Using 3×3 hidden kernels in decoder units D-RNN#3 and D-RNN#4 consistently improved compression curves over the 1×1 hidden kernels used in Toderici et al. [2016].Source: §2 · p. 3

2.1.1 LSTM

  • The LSTM recurrent units use convolutional transformations on both the input and hidden state to update their cell and hidden states.The input and hidden-state convolutions use the spatial extents and depths specified for the units in Figure 1.Source: §2.1.1 · p. 3

2.1.2 Associative LSTM

  • Associative LSTM extends LSTM with a holographic representation, but experiments used it only in the decoder and used standard LSTM in the encoder.The paper reports that Associative LSTMs were effective only when used in the decoder.Source: §2.1.2 · pp. 3–4

2.1.3 Gated Recurrent Units

  • The GRU variant adds residual connections across iterations and between the input and output of the recurrent block, using coefficients αx = αh = 0.1.This formulation is inspired by ResNet and is intended to speed convergence.Source: §2.1.3 · p. 4

2.2 Reconstruction Framework

  • One-shot reconstruction predicts the full image at every iteration, whereas additive reconstruction predicts each residual and sums the iteration outputs.In one-shot reconstruction, only the previous residual is passed forward; experiments found that also passing the original image did not improve reconstructions.Source: §2.2 · p. 4
  • Content- and iteration-dependent gains extend additive reconstruction to accommodate residuals that shrink at different rates across image patches.A convolutional network estimates the gain from the previous residual; the gain scales the current residual before encoding and reverses that scaling during decoding.Source: §2.2 · pp. 4–5

3 Entropy Coding

Because learned binary codes may retain entropy and spatial redundancy, the paper adds lossless entropy coding to improve compression. It models code probabilities conditioned on preceding bits and uses arithmetic coding to encode them.

3 Entropy Coding

  • The framework predicts each binary code from a context of preceding bits in stream order, then uses arithmetic coding; expected ideal length is the cross entropy between true and estimated probabilities.Codes are processed progressively and, within each iteration, in raster-scan order.Source: §3 · p. 5

3.1 Single Iteration Entropy Coder

  • For one code layer, a BinaryRNN based on PixelRNN [Oord et al., 2016] estimates probabilities line by line with a 1×3 LSTM convolution and a state line summarizing previous lines.The state has size 1 × W × k, with k = 64 in practice, and captures short- and long-term dependencies.Source: §3.1 · p. 6

3.2 Progressive Entropy Coding

  • For multiple iterations, the progressive coder augments each iteration’s line-LSTM input with information estimated from the previous iteration, allowing it to capture cross-iteration redundancy.Unlike code inputs in the current iteration, previous-iteration codes are fully available, so computing this additional input requires no masked convolution.Source: §3.2 · p. 6

4 Results

On Kodak images, the experiments compare recurrent compression models using MS-SSIM and PSNR-HVS, with AUC summarizing rate-distortion performance. Training on hard-to-compress patches and adding entropy coding generally improved results.

Evaluation Metrics

  • The evaluation used MS-SSIM and PSNR-HVS on the 24-image Kodak dataset, ranking models by area under the rate-distortion curve.Higher metric values indicate closer matches to reference images; the paper notes that the metrics can disagree and models may differ across bit rates.Source: §4 · p. 7

Training Setup

  • Models generally performed better when trained on the High Entropy dataset than on the 32×32 dataset, supporting the use of hard-to-compress training examples.The High Entropy set samples 100 tiles with the worst PNG compression ratio from each of 6 million web images sized 1280×720.Source: §4 · pp. 6–7

Architectures

  • With 32×32 training data, GRU One Shot led both metrics; with High Entropy training, LSTM One Shot led MS-SSIM but had the worst PSNR-HVS, while Residual GRU One Shot led PSNR-HVS.On High Entropy data, GRU One Shot ranked second in both metrics; with 32×32 data, LSTM with Residual Scaling ranked second in MS-SSIM and Residual GRU ranked second in PSNR-HVS.Source: §4 · p. 7

Entropy Coding

  • All models benefited from entropy coding; best-model gains ranged from 5% at 2 bpp to 32% at 0.25 bpp, while Baseline LSTM savings ranged from 25% at 2 bpp to 57% at 0.25 bpp.The authors note that Kodak's relatively low-resolution images limit the gains and that entropy coding's benefit is more apparent on large images.Source: §4 · p. 7

5 Discussion

The paper presents RNN-based image-compression models that on average outperform JPEG on AUC MS-SSIM and AUC PSNR-HVS, with and without entropy coding. The discussion highlights limits in perceptual evaluation and entropy-coder training, and identifies stronger video-codec-derived methods as a future challenge.

5 Discussion

  • The proposed RNN compression architecture combines content-based residual scaling with a new GRU variation, whose high-entropy-trained model achieved the highest PSNR-HVS among the models.Source: §5 · p. 7
  • The models achieve better-than-JPEG average performance on AUC MS-SSIM and AUC PSNR-HVS, both with and without entropy coding.The authors caution that existing perceptual metrics do not capture the models’ distortions well enough to declare one model best.Source: §5 · p. 7, Figure 5 · p. 8, Figure 6 · p. 9
  • The authors identify outperforming video-codec-derived methods such as WebP on large images as the next challenge, since these methods reuse already-decoded patches.Source: §5 · p. 9
  • Jointly training BinaryRNN and the patch-based encoder on larger patches could trade off encoder efficiency against entropy-coder predictive power.Currently, encoder efficiency is fixed during its training, and BinaryRNN’s efficiency is limited by the encoder’s entropy.Source: §5 · p. 9
  • Available perceptual metrics correlate poorly with human vision overall, or only for particular distortion types, limiting direct optimization for perceptual quality.A metric that correlated with human raters across distortion types could be incorporated into the loss function.Source: §5 · p. 9

A Supplementary Materials

The supplement presents Kodak image comparisons between JPEG and several neural compression approaches at nominal bitrates of 0.5, 1.0, and 2.0 bpp. It cautions that the bitrate accounting favors JPEG and overstates the networks’ final rates.

A Supplementary Materials

  • The supplement compares JPEG 420 with four neural approaches, showing full original images and detailed crops for each at nominal bitrates of 0.5, 1.0, and 2.0 bpp.The neural approaches are LSTM Residual Scaling trained on the 32x32 dataset, GRU One-shot trained on the 32x32 dataset, GRU One-shot trained on the high-entropy dataset, and Residual GRU One-shot trained on the 32x32 dataset.Source: A Supplementary Materials · p. 11
  • The visual comparison favors JPEG because its header size is excluded while nominal network rates omit entropy coding, making final network rates lower and JPEG rates slightly higher.Final network rates are about 15% lower than nominal at 0.5 bpp and about 5% lower at 2 bpp.Source: A Supplementary Materials · p. 11

Related papers

Loading 1608.05148v1…