PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-end Optimized Image Compression

Johannes Ballé, Valero Laparra, Eero P. Simoncelli

arXiv:1611.01704v3cs.CVcs.IT

TL;DR

The paper addresses lossy image compression by jointly optimizing nonlinear transforms and quantization for rate–distortion performance. Its end-to-end model uses cascaded convolutional nonlinearities and a continuous quantization relaxation, achieving better rate–distortion performance than JPEG and JPEG 2000 for most images and bit rates, with improved perceptual quality across all tested images and bit rates.

  • Problem

    Lossy image compression must balance entropy rate against quantization distortion, while high-dimensional optimal quantization is generally intractable.

  • Method

    The method jointly optimizes cascaded nonlinear analysis and synthesis transforms with uniform quantization using a continuous relaxation and an R+λD objective.

  • Results

    The method improves rate–distortion performance over JPEG and JPEG 2000 for most images and bit rates, while MS-SSIM improves across all test images and bit rates.

  • Takeaways & Limitations

    Nonlinear transform coding produces more natural compressed images than JPEG and JPEG 2000, whose linear-transform artifacts remain severe.

  • Takeaways & Limitations

    The architecture choices are somewhat ad hoc, and alternative architectures could potentially yield significant performance improvements.

Abstract

from arXiv · show

We describe an image compression method, consisting of a nonlinear analysis transformation, a uniform quantizer, and a nonlinear synthesis transformation. The transforms are constructed in three successive stages of convolutional linear filters and nonlinear activation functions. Unlike most convolutional neural networks, the joint nonlinearity is chosen to implement a form of local gain control, inspired by those used to model biological neurons. Using a variant of stochastic gradient descent, we jointly optimize the entire model for rate-distortion performance over a database of training images, introducing a continuous proxy for the discontinuous loss function arising from the quantizer. Under certain conditions, the relaxed loss function may be interpreted as the log likelihood of a generative model, as implemented by a variational autoencoder. Unlike these models, however, the compression model must operate at any given point along the rate-distortion curve, as specified by a trade-off parameter. Across an independent set of test images, we find that the optimized method generally exhibits better rate-distortion performance than the standard JPEG and JPEG 2000 compression methods. More importantly, we observe a dramatic improvement in visual quality for all images at all bit rates, which is supported by objective quality estimates using MS-SSIM.

1 INTRODUCTION

The paper frames lossy image compression as a rate–distortion problem and proposes jointly optimizing nonlinear analysis and synthesis transforms around a quantized latent representation.

  • 1 INTRODUCTION: Lossy compression trades off the entropy of a discretized representation against quantization error, with applications requiring different rate–distortion operating points.The objective depends on the desired balance between rate and distortion.
  • 1 INTRODUCTION: High-dimensional optimal quantization is intractable, motivating transform coding with independent quantization of transformed elements and subsequent entropy coding.Conventional methods typically use linear transforms, such as JPEG's discrete cosine transform and JPEG 2000's multiscale wavelets.
  • 1 INTRODUCTION: The proposed framework uses cascaded convolutional nonlinear transforms, uniform scalar quantization, and an approximate nonlinear inverse to reconstruct images.Its GDN nonlinearity is inspired by biological visual neurons and has been effective at Gaussianizing image densities.
  • 1 INTRODUCTION: Analysis and synthesis parameters are jointly optimized with stochastic gradient descent for any desired rate–distortion point using a continuous relaxation of quantization.The relaxation replaces quantization with additive uniform noise in the probability model.

2 CHOICE OF FORWARD, INVERSE, AND PERCEPTUAL TRANSFORMS

The transforms use cascaded convolutional stages with biologically inspired local gain control, while the synthesis path reverses the analysis operations and the perceptual metric is set to MSE.

  • 2 CHOICE OF FORWARD, INVERSE, AND PERCEPTUAL TRANSFORMS: Local nonlinear gain control is used to reduce higher-order dependencies in linear filter responses, extending transform coding beyond decorrelation alone.The operation is inspired by models of biological visual neurons.
  • 2 CHOICE OF FORWARD, INVERSE, AND PERCEPTUAL TRANSFORMS: The analysis transform has three stages of convolution, subsampling, and GDN, producing the latent vector y from the input image x.Each stage begins with affine convolution and downsampling, then concludes with GDN.
  • 2 CHOICE OF FORWARD, INVERSE, AND PERCEPTUAL TRANSFORMS: The synthesis transform reverses the stage order, replaces downsampling with upsampling, and uses IGDN as an approximate inverse of GDN.Its output is the reconstructed image obtained from the quantized latent representation.
  • 2 CHOICE OF FORWARD, INVERSE, AND PERCEPTUAL TRANSFORMS: The nonlinear transforms warp code space so fixed uniform scalar quantization acts as a parametric form of vector quantization in image space.The rate–distortion objective is R+λD, where λ selects the operating trade-off.
  • 2 CHOICE OF FORWARD, INVERSE, AND PERCEPTUAL TRANSFORMS: The perceptual transform is set to the identity and distortion is measured with MSE, enabling interpretable comparison with existing MSE-optimized methods and color-image optimization.The paper notes that a reliable perceptual metric for color images was not available.

3 OPTIMIZATION OF NONLINEAR TRANSFORM CODING MODEL

The model optimizes a rate–distortion objective using a fixed uniform quantizer in code space and nonlinear transforms, replacing quantization with continuous noise during training. Under specific assumptions, the relaxed objective corresponds to variational-autoencoder optimization, but actual compression is evaluated with discrete bit rates and broader perceptual metrics may break the equivalence.

  • Rate–distortion objective: The objective minimizes R+λD over transform and entropy-model parameters, with λ selecting the desired rate–distortion trade-off.The rate is approximated through entropy because a properly designed entropy code achieves rates only slightly above entropy.
  • Rate–distortion objective: A fixed uniform scalar quantizer operates in code space while nonlinear transforms warp that space to approximate vector quantization.This avoids directly solving optimal high-dimensional quantization in image space.
  • Continuous relaxation: Because quantization derivatives are zero almost everywhere, additive independent uniform noise provides a continuous relaxation for stochastic-gradient optimization.The noise matches the quantization-bin width and approximates both the quantized distribution and quantization-error moments.
  • Continuous relaxation: The relaxed loss uses a jointly trained, nonparametric piecewise-linear density model whose differentiability makes it suitable for stochastic optimization.The marginal code distributions are modeled independently and smoothed by the uniform-noise convolution.
  • Relationship to variational generative image models: With MSE distortion and the specified generative model and approximate posterior, the relaxed rate–distortion objective is equivalent to a variational-autoencoder objective up to additive constants.The KL terms correspond respectively to a constant, distortion, and rate under these definitions.
  • Relationship to variational generative image models: The variational-autoencoder correspondence is conditional: it is not guaranteed for non-Euclidean metrics or perceptual transforms, and continuous optimization is evaluated separately from actual discrete bit rates.The paper assesses the quality of the rate and distortion approximations empirically.

4 EXPERIMENTAL RESULTS

Experiments show that the relaxed training objective tracks quantized rate–distortion values well, while the optimized nonlinear method produces more natural compressed images and stronger perceptual quality than JPEG and JPEG 2000.

  • Optimization: The model was trained separately for each λ using joint stochastic optimization of the transforms and marginal probability models.Optimization used 6507 ImageNet images and produced separate transforms and probability models for each rate–distortion trade-off.
  • Relaxation validation: The continuously relaxed loss closely approximates quantized rate–distortion values, with small distortion variance and rate bias disappearing for finer quantization.A constant bias would only change the effective λ and would not affect compression performance.
  • Rate-matched comparison: Figure 6 matches bit rates across JPEG, JPEG 2000, and the proposed method while varying each method’s quality or λ settings.The figure compares three bit rates using cropped image regions.
  • Visual quality: The proposed method preserves smooth contours and sharp edges at low rates, whereas JPEG and JPEG 2000 show blocking, aliasing, and ringing artifacts.The comparison used heavily compressed images at equal or greater bit rates.
  • Rate–distortion performance: Perceptual advantages hold for all tested images and bit rates, while PSNR rate–distortion performance is better than both baselines for most images, especially at low rates.Figure 7 evaluates perceptual quality with MS-SSIM and fidelity with PSNR.

5 DISCUSSION

The discussion frames the method as an end-to-end nonlinear transform-coding system whose visual and rate–distortion gains extend beyond conventional linear representations. It also identifies provisional evidence about the efficiency of GDN/IGDN and broader links to unsupervised learning.

  • Discussion: The paper presents nonlinear transform coding with end-to-end rate–distortion optimization, improving performance over JPEG and JPEG 2000 for most images and bit rates.The authors also report substantially improved MS-SSIM across all tested images and bit rates.
  • Relation to representations: The analysis transform can be interpreted as a linear transform followed by a complex vector quantizer, linking the method to optimized image representations.The discussion compares this interpretation with sparse coding, steerable pyramids, curvelets, and dual-tree complex wavelets.
  • Related work: Unlike a related hierarchical image representation, this work optimizes directly for rate–distortion performance rather than using generative modeling as its objective.The distinction is presented as a difference between the approaches’ modeling goals.
  • Nonlinearity choice: Preliminary experiments suggest GDN/IGDN can reach comparable performance with fewer stages than rectified-linear units, but this efficiency conclusion remains provisional because experiments are limited.The authors are testing simpler nonlinearities to determine how much performance and visual quality depend on biologically inspired joint nonlinearities.
  • Broader implications: The rate–distortion objective is presented as an instance of unsupervised learning or density estimation, suggesting possible transfer of its framework to supervised learning questions.The discussion specifically mentions whether additive-noise relaxation could improve robustness to small perturbations in classification networks.

6 APPENDIX

The appendix details the network architecture, optimization procedures, entropy coding implementation, and evaluation observations, including limitations of adaptive coding and rate–distortion summaries.

  • 6.1 NETWORK ARCHITECTURE AND OPTIMIZATION: The transforms use three convolutional stages with downsampling and GDN in analysis, while synthesis uses analogous stages with IGDN.The grayscale architecture is specified with layer connectivity, filter sizes, channel counts, and parameter totals in Figure 8.
  • 6.1 NETWORK ARCHITECTURE AND OPTIMIZATION: GDN simplifies the nonlinear parameterization by fixing exponents and constraining its channel-coupling weight matrix to be symmetric.This choice reflects the reduced importance of tail behavior for the rate–distortion objective.
  • 6.1 NETWORK ARCHITECTURE AND OPTIMIZATION: The full model is optimized separately for each λ on 6507 ImageNet images using Adam, with parameterization and stabilization procedures for filters and GDN/IGDN parameters.Additional procedures include DCT filter coefficients, positivity constraints, symmetry enforcement, filter renormalization, spline-based marginal densities, and adaptive spline ranges.
  • 6.2 ENTROPY CODE: The entropy code uses CABAC to encode image metadata, λ, and quantized latent values, converting each quantized value into binary decisions with exponential Golomb fallback outside the modeled range.Quantized elements are traversed by channel and raster-scan position; CABAC contexts encode the resulting binary decisions.
  • 6.2 ENTROPY CODE: Adaptive entropy coding provides relatively small gains over a well-designed non-adaptive code because raster-scan ordering only crudely exploits spatial adaptation.The appendix also notes that rate–distortion behavior varies substantially across images and bit rates, making aggregate summaries difficult to interpret.
Loading 1611.01704v3…