Source-linked AI summary
Improved Lossy Image Compression with Priming and Spatially Adaptive Bit Rates for Recurrent Networks
Nick Johnston, Damien Vincent, David Minnen, Michele Covell, Saurabh Singh, Troy Chinen, Sung Jin Hwang, Joel Shor, George Toderici
TL;DR
The paper addresses lossy image compression with recurrent convolutional networks by improving perceptual training, hidden-state context, and spatial bit allocation. Combining these techniques improves MS-SSIM rate-distortion performance beyond recent neural methods and many standard codecs, including WebP, while SABR and diffusion introduce distinct computational or signaling considerations.
Problem
Recurrent models use a fixed local bit rate across each image, while perceptual metrics pose differentiability challenges and L1 or L2 losses are only loosely related to perception.
Method
The method combines SSIM-weighted L1 training, hidden-state priming and diffusion, and spatially adaptive bit rates in a recurrent image-compression architecture.
Results
Combining the three techniques improves the MS-SSIM rate-distortion curve, surpassing recent neural methods and many standard codecs, including BPG 420, WebP, JPEG2000, and JPEG.
Takeaways & Limitations
Priming improves reconstruction quality with lower inference and training computational requirements than diffusion, while SABR’s adaptivity more than offsets its height-map overhead.
Takeaways & Limitations
The reported setting includes a smaller batch size and lower learning rate due to memory constraints, and retraining for SABR could yield better performance.
Abstract
from arXiv · showhide
We propose a method for lossy image compression based on recurrent, convolutional neural networks that outperforms BPG (4:2:0 ), WebP, JPEG2000, and JPEG as measured by MS-SSIM. We introduce three improvements over previous research that lead to this state-of-the-art result. First, we show that training with a pixel-wise loss weighted by SSIM increases reconstruction quality according to several metrics. Second, we modify the recurrent architecture to improve spatial diffusion, which allows the network to more effectively capture and propagate image information through the network's hidden state. Finally, in addition to lossless entropy coding, we use a spatially adaptive bit allocation algorithm to more efficiently use the limited number of bits to encode visually complex image regions. We evaluate our method on the Kodak and Tecnick image sets and compare against standard codecs as well recently published methods based on deep neural networks.
1. Introduction
The paper extends recurrent neural-network image compression with priming, spatially adaptive bit rates, and SSIM-guided loss, reporting improved rate-distortion performance over standard codecs and learned methods under MS-SSIM.
- Method: Recurrent autoencoding progressively encodes residual information, using early iteration codes for low bit rates and later codes for higher bit rates.Each iteration combines information from the current residual with context stored in recurrent hidden states.
- Method: Priming runs iterations before code emission or reconstruction, expanding decoder context and enabling more complex functions in early iterations.The architecture also examines hidden-state priming and diffusion as mechanisms for improving spatial information propagation.
- Method: SABR dynamically adjusts the bit rate across each image according to local image complexity.This targets the inefficient fixed local bit rate imposed by conventional recurrent representations.
- Method: SSIM-guided pixel-wise loss improves perceptual training relative to unweighted alternatives.The paper describes the loss as part of a broader training approach intended to improve generalization and reconstruction quality.
- Results: The combined techniques produce an MS-SSIM rate-distortion curve that exceeds standard codecs and other neural-network-based methods.Figure 1 reports Bjøntegaard Delta rate savings relative to JPEG for the Kodak dataset.
2. Related Work
Related work establishes neural-network image compression through autoencoders and recurrent progressive encodings, while noting that few such systems improve upon JPEG or match JPEG2000 on typical web images.
- Standard codecs: JPEG remains widely used, while JPEG2000, WebP, and BPG provide more sophisticated alternatives; BPG is described as the most coding-efficient public codec.These standards frame the comparison target for learned compression methods.
- Open challenge: Few neural-network-based systems improve upon JPEG or match JPEG2000 on typical digital images found on the web.This limitation motivates comparisons between newer learned methods and established codecs.
- Neural compression: Deep neural networks have been applied to lossy image compression by extending autoencoders and quantizing bottleneck layers or latent variables.Multiple bit rates have been addressed through multiple models, learned quantization scaling, or recurrent transmission of encoded subsets.
- Recurrent approaches: Recurrent autoencoders generate progressive codes whose prefixes support different bit rates as the number of iterations increases.This approach transmits only a subset of the progressive code.
3. Methods
The method combines recurrent convolutional compression with DSSIM-weighted training, hidden-state priming and diffusion, and spatially adaptive bit allocation. These techniques improve spatial context, allocate bits to complex regions, and support better reconstructions without necessarily increasing transmitted bits.
- Network Architecture: The encoder and decoder use feed-forward convolutional units at the ends and convolutional GRUs in the intermediate layers.The encoder receives the residual between the original image and the previous reconstruction; the first residual is the original image.
- Hidden-state Priming: Hidden-state priming adds k recurrent steps before emitted codes or reconstructions, initializing hidden states without adding the discarded encoder bits to the bit stream.Diffusion inserts additional priming steps between emitting iterations, increasing spatial support and improving results at the cost of runtime and training time.
- Spatially Adaptive Bit Rates: Spatially adaptive bit rates assign bits according to local image complexity rather than using a fixed nominal rate across the image.The evaluation algorithm allocates tile bits to meet a target quality, using 16 × 16 tiles and assignments in 32-bit steps.
- SSIM Weighted Loss: The weighted L1 loss uses local DSSIM to emphasize image blocks whose perceptual distortion exceeds the average.This weighting aims to make perceptual quality more uniform across the image while retaining a differentiable pixel-wise loss.
4. Training
The training experiments compare perceptual-loss weighting, hidden-state priming, and diffusion, while also examining their computational costs. Diffusion improves reconstruction quality as its steps increase, but requires substantially more computation and training time.
- Training setup: The experiments train baseline, DSSIM, priming, and diffusion models to isolate improvements from the proposed techniques.Baseline and DSSIM models use 3.8M steps; diffusion models use 2.2M steps with k values from 0 to 3.
- Diffusion trade-offs: For k = 1 to 3, diffusion repeats encoder and decoder diffusion steps before every iteration rather than only before the first iteration.The diffusion models use DSSIM training and repeat the steps k times before producing the next bits or reconstruction.
- Training setup: All methods in Table 2 use DSSIM training loss and vary the number of diffusion steps between iterations.The tested diffusion configurations range from 0 to 3 steps.
5. Results
The proposed recurrent compression system is evaluated on Kodak and Tecnick using RGB-domain quality metrics and comparisons with learned and standard codecs. Its best model improves over prior neural methods and generally outperforms standard codecs, while SABR was applied without retraining.
- Evaluation setup: The evaluation reports AUC for PSNR, SSIM (dB), and MS-SSIM (dB) in the RGB domain.SSIM and MS-SSIM use the transform −10 log10(1 − Q), where Q is SSIM or MS-SSIM.
- Evaluation setup: Results are reported on Kodak’s 24 images and the larger Tecnick SAMPLING dataset of 100 1200 × 1200 images.Tecnick is presented as more representative of contemporary, higher-resolution content.
- Best-model comparison: On Kodak, the full method outperforms existing codecs at all but the lowest bit rate, where BPG 420 matches its performance.Figure 6 reports MS-SSIM (dB) and compares learned methods with standard codecs.
- Ablation results: DSSIM training improves AUC for every reported metric on both Kodak and Tecnick compared with the L1-trained baseline.The comparison covers PSNR, SSIM, and MS-SSIM.
- Ablation results: Priming improves over the non-priming DSSIM model, and the Prime model continues improving with additional training.Priming increases runtime, but its extra steps occur only before the first iteration.
- Best-model comparison: On Tecnick, the full method outperforms existing codecs except at the lowest bit rate, where BPG 420 has a small advantage.Figure 7 reports MS-SSIM (dB) on the larger Tecnick dataset.
- Ablation results: More diffusion improves reconstruction quality through the three tested steps, while computation increases nearly linearly with k+1 and training time rises substantially.The authors focus on the Prime model for later comparisons because of these trade-offs.
6. Conclusion
The paper reports that perceptually weighted loss, hidden-state priming, and spatially adaptive bit rates improve recurrent image compression. Together, these techniques improve MS-SSIM rate-distortion performance beyond recent neural methods and several standard codecs.
- Perceptually weighted L1 loss improved MS-SSIM, SSIM, and PSNR.The loss weights pixel-wise training using perceptual similarity.
- Hidden-state priming further improves reconstruction quality with lower inference and training computational requirements than diffusion.The improvement is probably related to content-dependent hidden-state initialization.
- Spatially adaptive bit rates reduce bits in easier-to-compress image regions, offsetting the overhead of transmitting the SABR height map.The method allocates bits according to local image complexity.
- Combining the three techniques surpasses recent neural-network compression methods and many standard codecs on the MS-SSIM rate-distortion curve.The compared standard codecs include BPG 420, WebP, JPEG2000, and JPEG.