Source-linked AI summary
Learning Convolutional Networks for Content-weighted Image Compression
Mu Li, Wangmeng Zuo, Shuhang Gu, Debin Zhao, David Zhang
TL;DR
CNN image compression is challenged by nondifferentiable quantization and the need to estimate discrete entropy for rate control. The paper addresses these issues with a content-weighted importance map, binary quantization, and a proxy function enabling end-to-end optimization. At low bit rates, the system outperforms JPEG and JPEG 2000 in SSIM and produces sharper, more textured images with fewer artifacts.
Problem
CNN-based lossy compression must handle nondifferentiable quantization and continuous estimation of entropy rate defined on discrete codes.
Method
A content-weighted importance map controls continuous rate allocation, while binary quantization and a proxy function support end-to-end optimization of the compression framework.
Results
The system outperforms JPEG and JPEG 2000 by SSIM at low bit rates and produces sharp edges, rich textures, and fewer artifacts.
Takeaways & Limitations
Content-aware spatially variant bit allocation is associated with better retention of perceptual structures and reduced visual artifacts in low-rate compression.
Abstract
from arXiv · showhide
Lossy image compression is generally formulated as a joint rate-distortion optimization to learn encoder, quantizer, and decoder. However, the quantizer is non-differentiable, and discrete entropy estimation usually is required for rate control. These make it very challenging to develop a convolutional network (CNN)-based image compression system. In this paper, motivated by that the local information content is spatially variant in an image, we suggest that the bit rate of the different parts of the image should be adapted to local content. And the content aware bit rate is allocated under the guidance of a content-weighted importance map. Thus, the sum of the importance map can serve as a continuous alternative of discrete entropy estimation to control compression rate. And binarizer is adopted to quantize the output of encoder due to the binarization scheme is also directly defined by the importance map. Furthermore, a proxy function is introduced for binary operation in backward propagation to make it differentiable. Therefore, the encoder, decoder, binarizer and importance map can be jointly optimized in an end-to-end manner by using a subset of the ImageNet database. In low bit rate image compression, experiments show that our system significantly outperforms JPEG and JPEG 2000 by structural similarity (SSIM) index, and can produce the much better visual result with sharp edges, rich textures, and fewer artifacts.
1. Introduction
The paper proposes content-weighted image compression to address nondifferentiable quantization and discrete entropy estimation. Its importance map enables spatially adaptive bit allocation, continuous rate control, and end-to-end CNN optimization.
- JPEG and JPEG 2000 use handcrafted transformations and separate codec optimization, performing poorly at low rates with blurring, ringing, and blocking artifacts.
- CNN compression remains difficult because quantization is nondifferentiable and entropy rate on discrete codes requires a continuous approximation.
- The content-weighted importance map adapts local bit allocation to spatially varying image content, assigning more bits to detailed regions.
- The importance-map sum provides continuous compression-rate control without an additional entropy-rate estimate.
- A binarizer quantizes encoder outputs, while a proxy function supplies differentiable backward propagation for joint end-to-end optimization.
- In low-bit-rate tests, the system outperforms JPEG and JPEG 2000 quantitatively and visually, retaining sharp edges and textures with fewer artifacts.It was trained on a subset of ImageNet and tested on Kodak.
2. Related Work
Prior image-compression standards rely on handcrafted transforms and separate codec optimization, while learned methods use continuous relaxations of quantization and entropy losses. The proposed system instead uses content-aware local code lengths and binary quantization.
- JPEG applies DCT to 8×8 patches, quantizes frequency components, and uses a Huffman-encoding variant for compression.
- JPEG 2000 uses a multi-scale transform, whereas these standards separately optimize codec stages.
- Deep learning methods include recurrent and full-resolution CNN-based systems for lossy image compression, alongside state-of-the-art lossless models.
- Ballé et al. replace rounding with additive uniform noise, while Theis et al. smooth the rounding derivative and upper-bound discrete entropy loss.
- Unlike related convolutional autoencoders with spatially invariant code lengths, this system uses content-aware local code lengths and binary encoder-output quantization.
3. Content-weighted Image Compression
The framework jointly learns convolutional encoding, importance-guided bit allocation, binarization, and decoding. It replaces discrete rate estimation with a continuous importance-map objective and uses proxy gradients to enable end-to-end training.
- Framework: The codec combines a convolutional encoder, binarizer, importance-map network, and convolutional decoder.The encoder produces E(x), the binarizer forms binary codes, the importance network produces P(x), and the decoder reconstructs the image.
- Binarizer: The binarizer assigns 1 to encoder outputs above 0.5 and 0 otherwise.Because sigmoid constrains encoder outputs to [0, 1], the thresholded binary representation is well-defined.
- Binarizer: A piecewise-linear proxy replaces the binarizer only during back-propagation, while the true binary function remains used in forward propagation.This addresses the binarizer’s zero-almost-everywhere gradient and permits encoder updates.
- Importance map: The content-weighted importance map assigns more bits to regions with sharp edges or rich textures and fewer bits to smooth regions.The map is a one-channel feature map learned from intermediate encoder features and has the same spatial size as the encoder output.
- Importance map: Quantizing the importance map creates a mask that trims binary codes, so each location receives a content-dependent number of bits.The final code is c = M(p) ◦ B(e), and the quantized importance map is also included in the code.
- Rate-distortion objective: The sum of the continuous importance map serves as rate loss, with a threshold r penalizing code length only above the target.This avoids optimizing the quantized code-length expression directly while providing compression-rate control.
4. Convolutional entropy encoder
The convolutional entropy encoder further compresses binary codes and importance maps using context-adaptive arithmetic coding with CNN-based probability prediction.
- Encoding binary code: Arithmetic coding is applied within a CABAC framework to compress binary codes and importance maps after the main compression model.The coding schedule scans each binary map left to right and row by row, skipping bits masked by zero importance.
- Context modeling: The context for each bit combines neighboring bits and neighboring maps in a 5 × 5 × 4 cuboid.Unavailable bits are explicitly encoded using values that distinguish the bit being predicted, masked bits, boundary positions, and uncoded bits.
- Probability prediction: A CNN receives each bit’s context cuboid and predicts the probability that the bit equals 1.This replaces a frequency table, whose size would be too large for the task.
- Probability prediction: The convolutional entropy encoder is trained on contexts extracted from binary feature maps generated by the trained encoder.The same encoder is extended to compress binary maps representing the quantized importance map.
5. Experiments
Experiments on Kodak images evaluate rate-distortion performance, visual quality, importance-map behavior, and convolutional entropy coding against established baselines and ablations.
- Experimental setup: The model is trained on about 10,000 high-quality ImageNet images cropped into 128 × 128 patches and tested on Kodak.Compression rate is evaluated in bits per pixel, with distortion measured using MSE, SSIM, and PSNR.
- Quantitative evaluation: On Kodak, the proposed system outperforms JPEG, JPEG 2000, and Ballé [1] in SSIM, while PSNR is similar to JPEG 2000 and Ballé [1].Its MSE is slightly better than JPEG 2000 and Ballé [1], whereas JPEG has the worst MSE performance.
- Visual quality evaluation: At low compression rates, the proposed system produces fewer noticeable artifacts and more pleasing images than JPEG, JPEG 2000, and Ballé [1].The compared artifacts include blurring, ringing, blocking, and color distortion.
- Experimental analyses on important map: Removing the importance map makes the baseline poorer than JPEG 2000 and Ballé [1] in MSE, PSNR, and SSIM.The comparison is used to validate the importance map’s necessity within the proposed model.
- Experimental analyses on important map: The importance map allocates bits first to salient edges at low rates, then to weaker edges, mid-scale textures, and small-scale textures as the rate increases.The reported allocation pattern is consistent with human visual perception.
- Entropy encoder evaluation: Both binary codes and importance maps can be further compressed by the convolutional entropy encoder, whose full model performs best among four variants.The convolutional encoder also outperforms standard CABAC with a small context and either coding method outperforms JPEG 2000 in SSIM.
6. Conclusion
The paper concludes that content-weighted compression combines non-entropy rate control, spatially variant bit allocation, binary quantization, and end-to-end learning to improve visual quality.
- Conclusion: The importance map provides non-entropy rate control and allocates more bits to salient regions.A straight-through estimator enables end-to-end learning, while convolutional entropy coding further compresses the codes and importance map.
A. Network Architecture
The network architecture section specifies the convolutional encoder and decoder architectures and their activation choices.
- Network architecture: Table 1 and Table 2 provide the network architectures of the convolutional encoder and decoder, respectively.The supplied table captions identify the architectural references, while the implementation description specifies the activation pattern.
- Network architecture: Except for the last layer, each convolutional layer uses ReLU; the encoder’s final layer uses Sigmoid to constrain outputs to (0, 1).The decoder has no nonlinear layer after its final convolutional layer.
B. Binarizing scheme for importance map
The importance map is quantized and converted into a binary representation for compression with binary arithmetic coding. The resulting binary map is determined directly from the quantized importance map and its number of feature maps depends on the number of importance levels.
- The quantized importance map Q(p) is binarized as b = B(Q(p)) for compression with binary arithmetic coding.The binary importance map has shape nb × w × h.
- The number of binary importance-map feature maps nb satisfies 2nb−1 < L ≤2nb, where L is the number of importance levels.
- Equation 13 provides a direct way to calculate the binary importance map from the quantized importance map.
C. Experiments supplementary
Using the importance map produces more detailed textures and better visual quality than the corresponding model without it. The supplementary material also points to additional high-resolution results and experiments.
- The model with an importance map produces more detailed textures and better visual quality than the model without one.The comparison is shown in Figure 9.
- The importance map gives the model more ability to model textures and edges in low bit rate image compression.
- Additional high-resolution results and expanded appendix experiments are available through the referenced project website.