PaperlayerView exact revision on arXiv

Source-linked AI summary

Practical Full Resolution Learned Lossless Image Compression

Fabian Mentzer, Eirikur Agustsson, Michael Tschannen, Radu Timofte, Luc Van Gool

arXiv:1811.12817v3eess.IVcs.CVcs.LG

TL;DR

Practical learned lossless image compression remains underdeveloped because prior learned compression focused on lossy settings and autoregressive decoding is slow. L3C addresses this with a fully parallel hierarchical probabilistic model that jointly learns auxiliary representations and entropy predictors. It outperforms several engineered codecs and is orders of magnitude faster than PixelCNN-based approaches, while learned representations outperform predefined alternatives.

  • Problem

    Prior learned image-compression research focused on lossy compression, and learned lossless models lacked practical bitrate and runtime comparisons with classical codecs.

  • Method

    L3C jointly trains fully parallel hierarchical feature extractors and predictors to model the image and auxiliary representations for adaptive arithmetic coding.

  • Results

    L3C outperforms PNG, JPEG2000, and WebP, while running 5.06 · 10^2× faster than MS-PixelCNN and 5.31 · 10^4× faster than PixelCNN++.

  • Takeaways & Limitations

    Learning auxiliary representations is crucial: L3C significantly outperforms predefined RGB-based representations and makes learned lossless compression practical for full-resolution images.

  • Takeaways & Limitations

    Comparisons with PixelCNN and MS-PixelCNN are not completely fair because their code is unavailable and hardware and software frameworks differ.

Abstract

from arXiv · show

We propose the first practical learned lossless image compression system, L3C, and show that it outperforms the popular engineered codecs, PNG, WebP and JPEG 2000. At the core of our method is a fully parallelizable hierarchical probabilistic model for adaptive entropy coding which is optimized end-to-end for the compression task. In contrast to recent autoregressive discrete probabilistic models such as PixelCNN, our method i) models the image distribution jointly with learned auxiliary representations instead of exclusively modeling the image distribution in RGB space, and ii) only requires three forward-passes to predict all pixel probabilities instead of one for each pixel. As a result, L3C obtains over two orders of magnitude speedups when sampling compared to the fastest PixelCNN variant (Multiscale-PixelCNN). Furthermore, we find that learning the auxiliary representation is crucial and outperforms predefined auxiliary representations such as an RGB pyramid significantly.

1. Introduction

The paper addresses the gap between theoretically possible learned lossless compression and practical systems by introducing a fully parallel learned model. L3C jointly learns hierarchical auxiliary representations and predictors, achieving competitive compression rates and much faster decoding than autoregressive alternatives.

  • Motivation: Learned lossless compression had received little practical evaluation against classical codecs, despite likelihood-based models being theoretically usable with arithmetic coding.Prior work focused mainly on lossy compression, while bitrate and runtime comparisons with PNG, WebP, JPEG2000, and FLIF were largely absent.
  • Motivation: Autoregressive models are computationally expensive because decoding requires a forward pass for every predicted sub-pixel in raster-scan order.This sequential operation limits practical use for full-resolution images.
  • Method: Learning the auxiliary feature representation is crucial because predefined choices such as a multiscale RGB pyramid produce suboptimal performance.The learned representation helps predictors model both the image and the auxiliary features.
  • Method: L3C uses jointly trained, fully parallel feature extractors and predictors to model the image together with hierarchical auxiliary representations.The system encodes both image and auxiliary features with adaptive arithmetic coding and predicts their distributions across scales.
  • Results: L3C outperforms WebP, JPEG2000, and PNG in compression rates, with PNG beaten by a large margin, while being only marginally outperformed by FLIF.The model is optimized for discrete log-likelihood and remains conceptually simpler than FLIF.
  • Results: 5.06 · 10^2× faster than MS-PixelCNN and 5.31 · 10^4× faster than PixelCNN++ are the reported speedups for L3C.In practice, decoding uses S = 3 parallel forward passes over pixels together with adaptive arithmetic coding.

2. Related Work

Related work spans autoregressive discrete models, engineered lossless codecs, learned context models, and continuous likelihood models. These approaches differ in factorization, transformations, auxiliary representations, and the practical constraints imposed by arithmetic coding.

  • Likelihood-Based Generative Models: PixelRNN and PixelCNN model pixels sequentially, requiring network evaluations equal to the number of predicted sub-pixels, 3 · W · H.PixelCNN uses masked convolutions to condition each pixel distribution on previous pixels.
  • Engineered Codecs: Engineered codecs use reversible transformations and entropy coding, with PNG, WebP, and JPEG2000 employing increasingly involved spatial or fragment-based representations.The supplied passage introduces these codec design families but does not provide a complete comparison.
  • Context Models in Lossy Compression: Lossy-compression context models commonly encode learned image representations with shallow autoregressive models over latents.L3C is related to hierarchical encoder-decoder designs but trains its representation for lossless rather than lossy compression.
  • Continuous Likelihood Models for Compression: Continuous likelihood models face practical compression challenges because discrete likelihoods require non-trivial integration and arithmetic-coding memory can grow exponentially without factorization.The domain-size issue follows from factorizing a distribution over the image variables.

3. Method

L3C uses a hierarchical, fully parallel probabilistic model that jointly represents the image and learned auxiliary features for adaptive entropy coding. Its learned quantized hierarchy and mixture predictors are trained end-to-end to model these representations efficiently.

  • Lossless compression: Arithmetic coding converts predicted symbol probabilities into a compressed bitstream, approaching the entropy lower bound for long streams.For natural images, adaptive coding uses conditional probabilities because pixels are strongly correlated rather than i.i.d.
  • Hierarchical architecture: L3C jointly models the image and a hierarchy of auxiliary representations instead of factorizing pixels autoregressively one sub-pixel at a time.The auxiliary representations simplify modeling by providing multiscale features, while the model remains non-autoregressive over spatial locations.
  • Hierarchical architecture: The hierarchy uses learned feature extractors and predictors, with each predictor conditioned on higher-scale features and each extractor producing quantized auxiliary features.Predictor features summarize coarser auxiliary representations, and the final predictor receives only the highest-scale representation.
  • Quantization: Each auxiliary feature is quantized by nearest-neighbor assignment to one of L discrete levels, while differentiable soft quantization supplies gradients during training.The reported implementation uses L = 25 evenly spaced levels in [−1, 1].
  • Mixture model: Conditional distributions are modeled with discretized logistic mixtures, which reduce memory and sparse-gradient issues while assigning similar probabilities to neighboring values.RGB channels use a weak autoregression within each pixel, whereas entries at different spatial locations are conditionally independent given predictor features.
  • Loss: The training loss is the negative log-likelihood of the joint image-and-representation model and decomposes into cross-entropies for the different representations.Gradients propagate through the learned representation targets; stopping this propagation substantially worsens performance.

4. Experiments

The experiments evaluate L3C against learned and engineered codecs across multiple image datasets, and test architectural and representation choices. Results show that learned representations matter, while runtime comparisons use specified batch sizes, hardware, and preprocessing conditions.

  • Models: L3C is compared with RGB Shared and RGB baselines using bicubic subsampling, while L3C learns its representations.The RGB Shared baseline trains one predictor, whereas the RGB baseline trains three scale-specific predictors.
  • Datasets: 362 551 Open Images training images support evaluation on Open Images, DIV2K, and RAISE-1k datasets.Images are downscaled and filtered; crops processed sequentially when GPU memory is insufficient, which the authors note biases against L3C.
  • Architecture ablations: Stopping gradients through learned-representation targets significantly worsens performance and prevents substantial reductions in their cross-entropy.This ablation supports optimizing the learned representations jointly rather than blocking these gradients.
  • Representation ablations: Learning the feature representations is crucial, because a predefined multiscale RGB pyramid produces suboptimal performance.The experiments also find increasing the number of levels more beneficial than increasing the number of channels.

5. Results

L3C achieves strong lossless compression while substantially reducing the runtime associated with PixelCNN-based approaches. Its learned representations support lower bitcost than predefined RGB representations, with speed traded against some learned-baseline and PixelCNN comparisons.

  • 5.1. Compression: L3C outperforms WebP and JPEG2000 by up to 3.3%, while FLIF marginally outperforms L3C.
  • 5.1. Compression: Learning feature extractors improves L3C over RGB baselines on all datasets, with the RGB baselines’ outputs at least 7.8% larger everywhere.
  • 5.2. Comparison with PixelCNN: L3C is at least 5.06 · 10^2× faster than MS-PixelCNN on 320 × 320 crops, with larger speedups over original PixelCNN.
  • 5.2. Comparison with PixelCNN: On ImageNet32, L3C outputs are 20.6% larger than MS-PixelCNN and 24.4% larger than original PixelCNN, trading bitcost for orders-of-magnitude speed.
  • 5.3. Encoding / Decoding Time: L3C has encoding times comparable to classical approaches but slower decoding, while combined encoding and decoding is faster than FLIF or has better bitrate than PNG, WebP, and JPEG2000.

6. Conclusion

The paper concludes that L3C provides a practical fully parallel hierarchical model for lossless image compression. It outperforms several classical codecs and benefits substantially from learned rather than predefined representations, while PixelCNN-based full-resolution compression is much slower.

  • L3C is a fully parallel hierarchical probabilistic model with auxiliary feature representations.
  • L3C outperforms PNG, JPEG2000, and WebP on all evaluated datasets.
  • Learning the representations is crucial because L3C significantly outperforms RGB Shared and RGB baselines using predefined heuristic representations.
  • PixelCNN-based methods require two to five orders of magnitude longer than L3C for lossless full-resolution image compression.

A.1. Changes in Version 3

Version 3 corrects preprocessing artifacts that affected both learned and non-learned compression results. The revised datasets use PNG storage, uncropped validation images divided into memory-fitting crops, and randomized training downscaling.

  • The earlier preprocessing produced a distinctive image distribution that neural networks learned and that also made non-learned codecs compress more easily.
  • The revised preprocessing saves images as PNGs, avoids validation rescaling, and randomizes the training downscaling factor to broaden artifact variation.

A.2. Encoding and Decoding Details

The encoding and decoding procedure obtains probability distributions for the hierarchical representations and visualizes the decoding steps across scales. The implementation first constructs the CDF needed by arithmetic decoding before processing each scale.

  • For each scale, the implementation first obtains a CDF matrix on the CPU and then performs an arithmetic-decoder pass.
  • The detailed decoding steps use z(0) = x and are visualized in Figure A4.

Encoding

Encoding first obtains the hierarchical representations and predicted distributions, then uses adaptive arithmetic coding to encode the auxiliary features and RGB image values. The highest-scale auxiliary representation uses a uniform prior, while subsequent CDFs are derived from predicted logistic distributions.

  • Encoding: A single forward pass obtains all scale representations and distribution predictions before arithmetic encoding.The network produces z(s) and f(s) for every scale, enabling parallel prediction of probabilities for x and auxiliary features.
  • Encoding: The highest-scale auxiliary representation z(S) is encoded with a uniform prior requiring log2(L) bits per symbol.This avoids calculating a CDF for s = 3 in the reported timing setup.
  • Encoding: The RGB-scale CDF is obtained from predicted means and scales using the logistic-distribution CDF.The RGB means are updated using the input x before constructing the intervals needed for arithmetic coding.
  • Encoding: For each scale and channel, predicted CDFs are used with adaptive arithmetic coding to encode z(s) and concatenate the resulting sub-bitstreams.Each sub-bitstream includes its dimensions encoded as a UINT16 triplet to support unique decoding.

Decoding

Decoding proceeds hierarchically from the lowest-resolution auxiliary representation upward, using each decoded representation to obtain the next CDF. After the auxiliary features are recovered, the RGB channels are decoded sequentially and concatenated into the image.

  • Decoding: The decoder first recovers z(S) from the bitstream using the uniform prior used during encoding.This initializes the hierarchy without requiring a predicted CDF.
  • Decoding: Feeding z(S) to D(S) produces f(S) and the CDF needed to decode z(S−1) with adaptive arithmetic decoding.The same procedure is repeated because each lower-scale representation depends on the preceding decoded scale.
  • Decoding: The decoder repeats the hierarchical process to recover z(S), …, z(1) and the corresponding feature parameters.Each accompanying CDF is obtained after decoding the representation at the preceding scale.
  • Decoding: Given f(1), the decoder obtains the CDF for the red channel, decodes it, and then updates the next-channel mean using the recovered value.The green and blue channels are decoded analogously using the previously decoded channels.
  • Decoding: Concatenating the decoded channels x1, x2, and x3 yields the final decoded image x.The RGB channels are reconstructed after the auxiliary hierarchy has been decoded.

A.2.1 Hardware Used

The implementation measurements use a Titan X (Pascal) GPU and Intel Xeon E5-2680 v3 CPU, with timing procedures reported for CDF computation and arithmetic decoding. The appendix also describes parallelization, memory handling, codec comparisons, and learned representations.

  • A.2.1 Hardware Used: Timing measurements were collected on a Titan X (Pascal) GPU and Intel Xeon E5-2680 v3 CPU.These hardware details define the platform for the reported runtime results.
  • A.2.1 Hardware Used: The encoder can parallelize across scales after one forward pass, while the decoder remains sequential across scales because each representation predicts the next.Channels of auxiliary representations could additionally be parallelized because they are modeled independently.
  • A.2.1 Hardware Used: CDF tensors are stored in managed memory through a custom CUDA kernel to avoid expensive GPU-to-CPU copying for arithmetic coding.For a 512 × 512 image, the RGB CDF has dimensions H × W × 257 and requires 257MB per channel.
  • A.2.1 Hardware Used: The timing tables use a simple arithmetic coding module, whereas state-of-the-art adaptive entropy coders typically require milliseconds per MB.Thus, the reported coding times depend partly on the implementation used for measurement.
  • A.2.1 Hardware Used: On ImageNet64, L3C outputs are 23.8% larger than MS-PixelCNN and 19.4% larger than the original PixelCNN, but smaller than all classical approaches.The comparison reports bits per sub-pixel and notes that the higher bit cost is traded against orders-of-magnitude speed gains.
  • A.2.1 Hardware Used: Batch-size experiments show that per-image sampling time decreases as batch size increases, with overhead dominating at batch size 30.The comparison at batch size 30 is described as biased against L3C because its small network supports larger batches.
  • A.2.1 Hardware Used: The representations preserve global image structure across scales, while smaller s values model more detail.The authors identify these representations as potential features for image understanding on partially decoded images.
  • A.2.1 Hardware Used: The RGB Shared baseline uses one predictor D(1), whereas the RGB baseline trains multiple predictors.The figures show the architectural distinction between shared and separate predictor designs.
Loading 1811.12817v3…