Source-linked AI summary
Learning End-to-End Lossy Image Compression: A Benchmark
Yueyu Hu, Wenhan Yang, Zhan Ma, Jiaying Liu
TL;DR
End-to-end learned image compression lacked a systematic survey and benchmark despite substantial progress. This paper surveys the field, benchmarks existing and proposed methods, and introduces a coarse-to-fine hyperprior framework. The proposed method achieves more consistent gains across content and resolutions, while benchmark results also reveal cross-metric gaps and scope limitations.
Problem
End-to-end learned image compression lacks a systematic survey and benchmark for thorough method comparison, while models must adapt across resolutions, bit-rates, and distortions.
Method
The paper surveys learned compression by network architecture, entropy model, and rate control, then proposes a coarse-to-fine hyperprior framework for multiresolution context modeling.
Results
The proposed method achieves more consistent gains across different content and resolutions, with coding performance improvements and low time complexity on parallel computing hardware.
Takeaways & Limitations
The survey and benchmark clarify learned image-compression design methodologies, challenges, and future research directions while supporting further exploration of higher-efficiency compression.
Takeaways & Limitations
Cross-metric evaluation reveals that strong optimization for one metric may not deliver expected visual quality, motivating better assessment techniques.
Abstract
from arXiv · showhide
Image compression is one of the most fundamental techniques and commonly used applications in the image and video processing field. Earlier methods built a well-designed pipeline, and efforts were made to improve all modules of the pipeline by handcrafted tuning. Later, tremendous contributions were made, especially when data-driven methods revitalized the domain with their excellent modeling capacities and flexibility in incorporating newly designed modules and constraints. Despite great progress, a systematic benchmark and comprehensive analysis of end-to-end learned image compression methods are lacking. In this paper, we first conduct a comprehensive literature survey of learned image compression methods. The literature is organized based on several aspects to jointly optimize the rate-distortion performance with a neural network, i.e., network architecture, entropy model and rate control. We describe milestones in cutting-edge learned image-compression methods, review a broad range of existing works, and provide insights into their historical development routes. With this survey, the main challenges of image compression methods are revealed, along with opportunities to address the related issues with recent advanced learning methods. This analysis provides an opportunity to take a further step towards higher-efficiency image compression. By introducing a coarse-to-fine hyperprior model for entropy estimation and signal reconstruction, we achieve improved rate-distortion performance, especially on high-resolution images. Extensive benchmark experiments demonstrate the superiority of our model in rate-distortion performance and time complexity on multi-core CPUs and GPUs. Our project website is available at https://huzi96.github.io/compression-bench.html.
1 INTRODUCTION
Image compression balances visual quality against bit-rate, while end-to-end learning jointly redesigns compression components. This paper surveys and benchmarks learned methods, then proposes a coarse-to-fine hyperprior framework with improved coding performance and low parallel-computing complexity.
- Lossy image compression preserves critical visual information while reducing bit-rate, requiring trade-offs between reconstruction quality and coding rate.
- Traditional codecs use transform, quantization, and entropy coding to decorrelate signals and discard less important information.Examples include DCT in JPEG and DWT in JPEG 2000.
- End-to-end learned compression jointly improves latent decorrelation and entropy estimation to control bit-rate and distortion more effectively.
- The field lacked a systematic survey and benchmark that thoroughly summarized and compared learned image compression methods.
- The paper surveys existing methods, benchmarks them against learned and traditional codecs, and analyzes rate-distortion performance across bit-rates, resolutions, and implementation complexity.
- The proposed coarse-to-fine hyperprior framework improves coding performance while keeping time complexity low on parallel computing hardware.
2 PROBLEM FORMULATION
End-to-end learned compression jointly optimizes transform, quantization, entropy estimation, and reconstruction under a rate-distortion objective. This flexibility improves modeling but creates optimization, complexity, and rate-control challenges.
- Problem formulation: The encoder transforms an image into a discrete latent code, and the decoder reconstructs the image from that code.The encoder parameters are learned, while the decoder maps the quantized representation back to pixels.
- Problem formulation: Rate-distortion optimization minimizes R + λD, balancing the bit-rate required to encode the latent code against reconstruction distortion.Different λ values represent different rate-distortion trade-offs for application requirements.
- Problem formulation: Because the latent distribution is unknown, an entropy model estimates it so the rate can be formulated through cross entropy for entropy coding.The real latent distribution is denoted p_y, while q_y is its entropy-model estimate.
- Problem formulation: Joint optimization makes all compression components trainable and allows direct targeting of differentiable distortion metrics, unlike handcrafted component tuning.The approach can optimize metrics such as SSIM, MS-SSIM, and perceptual difference, but good end-to-end optimization remains difficult.
- Problem formulation: Full-resolution processing supplies more context and avoids blocking effects, but increases complexity because deeper networks are needed to enlarge the receptive field.Rate control also remains costly: many methods require separate models for different rate-distortion trade-offs, while variable-bit-rate single models can be time-consuming.
3 OVERVIEW OF PROGRESS IN RECENT YEARS
Recent learned image-compression research has advanced through trainable quantization, neural transforms, entropy models, and application-specific objectives. The paper organizes these developments and motivates systematic comparison across methods.
- Historical progress: Learned image compression has progressed from recurrent neural networks to methods whose performance exceeds JPEG and approaches or surpasses BPG.The paper summarizes major milestones in Table 1 and traces the field’s development.
- Trainable compression: Differentiable approximations to quantization, including additive uniform noise, straight-through rounding, and soft-to-hard vector quantization, enable end-to-end training.These methods address the nondifferentiability of true quantization.
- Neural transforms: Neural transforms use GDN, residual blocks, attention, non-local, invertible, and recurrent architectures to reduce spatial redundancy and improve representation modeling.The transform is a critical component for producing compact latent representations.
- Entropy modeling: Entropy models evolved from elementwise-independent assumptions toward explicit estimation of dependencies in latent representations.The paper summarizes these entropy-modeling solutions in Table 2.
- Application-specific objectives: Emerging methods target machine vision and human perception at low bit-rates, including generative synthesis of complex textured regions.Conditional GANs can use low-bit-rate representations as guidance for synthesizing such areas.
- Open challenges: The survey identifies unresolved problems and motivates benchmark analysis of backbones, entropy models, and other components.The paper uses this analysis to identify areas where learned frameworks can be improved.
4 BACKBONES FOR IMAGE COMPRESSION
Learned compression backbones are organized into one-time feed-forward and multistage recurrent frameworks. They trade training simplicity, rate flexibility, parameter storage, runtime, and rate-distortion performance differently.
- Backbone categories: VAE-based compression backbones encode images into compact latent vectors under dimensionality reduction and entropy constraints.The paper divides backbone architectures into one-time feed-forward and multistage recurrent frameworks.
- One-time feed-forward frameworks: Increasing GDN architecture complexity can improve compression, but reported gains are less significant than those from contributions such as hyperpriors.The paper contrasts this behavior with deeper networks in image recognition, where added depth commonly brings further gains.
- One-time feed-forward frameworks: Feed-forward architectures include GDN-based, residual block-based, and multiscale networks, with hyperprior models among the GDN-based variations.These categories are illustrated in Fig. 1.
- Multistage recurrent frameworks: Recurrent frameworks progressively encode residuals, using reconstructed-stage errors as inputs for subsequent stages and combining stage outputs for final reconstruction.Stateful variants propagate feature states between stages, while incremental reconstruction adds decoded stage outputs together.
- Framework trade-offs: Recurrent models naturally support variable-rate compression and share weights across stages, whereas feed-forward models require multiple instances for variable bit-rates.Shared weights can reduce parameter storage, but residual and image signals make recurrent training more challenging.
- Framework trade-offs: Feed-forward networks are easier to train because they are shallower and avoid back-propagation through time, while recurrent models require more execution time.Existing works report higher rate-distortion performance for one-time feed-forward architectures, but variable-bit-rate support remains a major adoption barrier.
5 ENTROPY MODELS
Entropy models estimate latent distributions to reduce coding rate, and their design interacts with the jointly optimized transform. The paper emphasizes adaptability to resolution, bit-rate, distortion, and computational hardware.
- Entropy-model role: Entropy coding estimates latent-representation probability distributions and constrains entropy to reduce the bit-rate.The required coding rate is bounded by the information entropy of the represented symbols.
- Entropy-model survey: The paper organizes entropy-modeling solutions in Table 2 and presents typical variations for comparison.The table describes different entropy models used in learned image compression.
- Entropy-model design: Ideal entropy coding requires precise instance-specific joint-distribution estimation, but early methods simplified the problem by assuming independent latent elements.Residual spatial redundancy remains even after transform optimization, motivating more advanced entropy models.
- Joint optimization: In learned compression, improving the entropy model also changes the latent patterns produced by the analysis transform because all components are jointly optimized.Entropy-model design therefore needs to account for the structure of the rest of the compression pipeline.
- Open requirements: Learned compression models should adapt across resolutions, bit-rates, and distortions while exploiting parallel hardware for higher efficiency.The paper identifies high-resolution compression and large-scale GPU parallelism as growing practical requirements.
6 PROPOSED COARSE-TO-FINE MODEL
The proposed framework uses multilayer hyperprior conditioning to model latent distributions coarse to fine, while preserving and aggregating information for reconstruction. This design addresses limitations of sequential context models and information loss in conventional hypertransforms.
- Coarse-to-Fine Hyperprior Modeling: The framework combines an analysis transform, quantization, synthesis transform, and entropy modeling to optimize learned image compression end to end.The analysis transform produces latent representation X, which is quantized before entropy coding and reconstructed by the synthesis transform.
- Coarse-to-Fine Hyperprior Modeling: State-of-the-art entropy models combine context models and hyperpriors, but sequential conditioning is difficult to accelerate and less scalable across image sizes.Hyperpriors must also remain compact and informative enough to estimate conditional probabilities accurately, particularly for large resolutions.
- Coarse-to-Fine Hyperprior Modeling: The proposed multilayer conditioning framework introduces hyperpriors Y and Z, forming a coarse-to-fine model whose innermost latent distribution is approximately factorized.The additional layer reduces the dimension of Z and squeezes out redundancy through hypertransforms.
- Coarse-to-Fine Hyperprior Modeling: Each upper-layer latent distribution is estimated from all elements in the previous layer, allowing information from a larger area to model long-term image correlations.The paper states that this structure improves compression performance, especially for high-resolution images.
- Network Architecture: The innermost hyperprior uses a zero-mean Gaussian model with channelwise independent and spatially shared scale values σ.For the outer latent distributions, means and scales are predicted from the preceding hyperrepresentation; the rate-distortion objective is L_RD = R + λD.
- Signal-Preserving Hyper Transform: The signal-preserving hypertransform expands the first-layer representation, avoids nonlinear activation there, and uses smaller filters and 1 × 1 convolutions in later layers.This design targets information loss caused by channel reduction, downsampling, large kernels, and ReLUs in the original transforms.
7 EVALUATION
The evaluation uses established natural-image datasets for testing learned compression methods. Kodak provides varied 512 × 768 images, while Tecnick and CLIC include different image characteristics and higher-resolution content.
- Testing Datasets: Kodak contains 24 high-quality natural images at 512 × 768 resolution and is widely used to evaluate image compression.Its varied content and textures are sensitive to compression artifacts.
- Testing Datasets: Tecnick is another commonly used testing dataset, with the SAM-PLING test set used for evaluation in some studies.
- Testing Datasets: CLIC provides yearly updated test images captured by mobile phones and professional cameras, with average resolutions of 1913 × 1361 and 1803 × 1175, respectively.
7.2 Rate-Distortion Performance
The benchmark evaluates learned compression across datasets, resolutions, and codecs, finding that entropy-model design drives rate-distortion gains and that the proposed method offers more consistent performance.
- Few studies had thoroughly evaluated rate-distortion performance across varied images while comparing learned methods with established anchors.
- The evaluation averages bpp, PSNR, and MS-SSIM across Kodak, Tecnick, and CLIC 2019, comparing learned methods with JPEG, BPG, and VVC.
- Entropy-model design is the main driver of rate-distortion improvement, with hyperpriors producing a major gain over earlier contextual models.
- Network depth is comparatively less important than designing networks to model image priors effectively, although sufficient width and parameters remain important.
- Normalization and hyperprior models bring significant improvements, while the proposed method achieves more consistent gains across content and resolutions.
7.3 Studies on the Proposed Method
The proposed method is studied through coarse-to-fine hyperprior and information-aggregation ablations, with particular benefits reported for high-resolution images and fewer reconstruction artifacts at low rates.
- 7.3.1 Coarse-to-Fine Modeling: The coarse-to-fine hyperprior is evaluated across 4K, 1080p, and 540p subsets using BD-Rate relative to a single-layer hyperprior anchor.
- 7.3.1 Coarse-to-Fine Modeling: CVPR17-RNN results are omitted on CLIC 2019 because the available code does not support that dataset’s resolutions.
- 7.3.2 Information Aggregation Reconstruction: Aggregation ablations vary feature form, feature granularity, and fusion stage, with rate-distortion curves reported for the alternatives.
- 7.3.1 Coarse-to-Fine Modeling: The coarse-to-fine model achieves more significant BD-Rate reductions on high-resolution images than the original single-layer model.
- 7.3.2 Information Aggregation Reconstruction: The IAR subnetwork aggregates image representations at different granularities to use transmitted information for reconstruction.
7.4 Cross-Metric Evaluation
Cross-metric evaluation shows that optimizing PSNR or MS-SSIM can favor that metric while producing different perceptual-distortion behavior, exposing a gap between assessment objectives.
- Perceptual cross-evaluation uses averaged L2 distances between VGG-16 feature maps of original and reconstructed images.
- Models trained with MS-SSIM have less perceptual distortion at a given PSNR, whereas PSNR-trained models have less at a given MS-SSIM.
- Models optimized for one metric show higher perceptual distortion at the same level of that metric than models not optimized for it.
- Among MS-SSIM-trained models, stronger MS-SSIM-bpp performance tends to coincide with larger perceptual distortion at a fixed MS-SSIM.
- The experiments identify a gap between metrics, particularly for models with better performance on one metric.
7.5 Discussion
The discussion compares learned compression with VVC and examines visual quality and computational behavior. VVC retains stronger rate-distortion performance and lower decoder complexity, while learned methods offer flexibility and parallelization potential.
- 7.5.1 Time Complexity: The time benchmark measures encoding and decoding for the proposed method, a context-model method, and VTM-8 on a CPU and GPU.
- 7.5.2 Visual Quality: The proposed method reconstructs images with fewer artifacts than representative learned and hybrid codecs at lower bit-rates.
- 7.5.1 Time Complexity: VTM has no thread-level parallelism, so multi-core devices do not accelerate it.
- 7.5.2 Comparison with VVC: VVC achieves better rate-distortion performance than the benchmarked end-to-end learned methods and has significantly lower decoder complexity.
- 7.5.2 Comparison with VVC: Learned methods are more flexible because neural networks can learn encoder and decoder operations end-to-end and can potentially use parallel computing.
8 CONCLUSION
The paper benchmarks learned image-compression methods, analyzes their challenges, and proposes a coarse-to-fine hyperprior framework evaluated across image contents and resolutions.
- The study systematically benchmarks existing learned image-compression methods and discusses their insights and challenges.
- The proposed coarse-to-fine hyperprior framework addresses multiresolution context-modeling issues in image compression.
- Thorough evaluations compare existing and proposed methods across images with varied content and resolutions.
- Cross-metric evaluation identifies joint optimization for machine intelligence systems and human perception as a future research direction.