PaperlayerView exact revision on arXiv

Source-linked AI summary

A Unified End-to-End Framework for Efficient Deep Image Compression

Jiaheng Liu, Guo Lu, Zhihao Hu, Dong Xu

arXiv:2002.03370v3eess.IVcs.CV

TL;DR

Learning-based image compression can improve compression performance, but state-of-the-art methods can require substantial computation. EDIC combines channel attention, a Gaussian mixture entropy model, and decoder-side enhancement; it reports comparable image-compression performance to Minnen’s method with decoding over 150 times faster, and improvement for video compression.

  • Problem

    State-of-the-art learning-based image compression methods improve compression performance but incur significant computational cost, raising the question of improving efficiency without substantially increasing computation.

    Source: §I · p. 1

  • Method

    EDIC uses an autoencoder-style framework combining channel attention over latent relationships, a Gaussian mixture entropy model, and decoder-side enhancement.

    Source: §I · p. 2, B. Channel Attention Scheme · p. 4, D. Decoder-side Enhancement · p. 5

  • Results

    EDIC reports comparable compression performance to Minnen’s method and decoding over 150 times faster for 768 × 512 images, while also reporting improved video-compression performance.

    Source: §I · p. 2, §V · p. 11

  • Takeaways & Limitations

    The framework is reported to be readily usable for video compression and to outperform existing state-of-the-art methods for image and video compression.

    Source: §V · p. 11

Abstract

from arXiv · show

Image compression is a widely used technique to reduce the spatial redundancy in images. Recently, learning based image compression has achieved significant progress by using the powerful representation ability from neural networks. However, the current state-of-the-art learning based image compression methods suffer from the huge computational cost, which limits their capacity for practical applications. In this paper, we propose a unified framework called Efficient Deep Image Compression (EDIC) based on three new technologies, including a channel attention module, a Gaussian mixture model and a decoder-side enhancement module. Specifically, we design an auto-encoder style network for learning based image compression. To improve the coding efficiency, we exploit the channel relationship between latent representations by using the channel attention module. Besides, the Gaussian mixture model is introduced for the entropy model and improves the accuracy for bitrate estimation. Furthermore, we introduce the decoder-side enhancement module to further improve image compression performance. Our EDIC method can also be readily incorporated with the Deep Video Compression (DVC) framework to further improve the video compression performance. Simultaneously, our EDIC method boosts the coding performance significantly while bringing slightly increased computational cost. More importantly, experimental results demonstrate that the proposed approach outperforms the current state-of-the-art image compression methods and is up to more than 150 times faster in terms of decoding speed when compared with Minnen's method. The proposed framework also successfully improves the performance of the recent deep video compression system DVC. Our code will be released at https://github.com/liujiaheng/compression.

I. INTRODUCTION

Traditional codecs use manually designed transforms and prediction tools, while learning-based compression offers trainable representations but can incur substantial decoding cost. EDIC targets that trade-off with channel attention, a Gaussian mixture entropy model, and decoder-side enhancement.

A. Traditional Image and Video Compression

  • Traditional methods cannot be optimized using large-scale training, which may limit their performance.Image compression targets spatial redundancy, while video compression targets temporal redundancy.Source: §I · p. 1, A. Traditional Image and Video… · p. 2

I. INTRODUCTION

  • For 768 × 512 images, average GPU decoding time is 0.013 seconds for Ballé’s method and 2.426 seconds for Minnen’s method.The passage attributes the difference to Minnen’s sequential per-pixel decoding versus parallel convolutional reconstruction in earlier approaches.Source: §I · p. 1
  • EDIC combines channel attention, a Gaussian mixture model, and decoder-side enhancement, reporting comparable compression performance to Minnen’s method with decoding over 150 times faster at 768 × 512.The framework is also reported to improve video compression performance.Source: §I · p. 2
  • EDIC differs from concurrent works [21], [22] by building on rather than the slower, using channel-wise rather than spatial attention, and also applying its technologies to video compression.The authors describe the video results as promising and note that [21], [22] do not discuss that application.Source: §I · p. 2

A. Traditional Image and Video Compression

  • Traditional image and video codecs rely on manually designed techniques, including transforms and block-based motion estimation and compensation.JPEG uses DCT; JPEG2000 uses a discrete wavelet transform, while BPG uses prediction-transform coding with 35 encoding modes.Source: A. Traditional Image and Video… · p. 2

B. Learning based Image and Video Compression

Learning-based compression uses neural networks to model image and video data, with approaches spanning recurrent and convolutional autoencoders, entropy priors, and spatial-redundancy methods. The section also highlights the rate-distortion and computational trade-offs that motivate efficient designs.

B. Learning based Image and Video Compression

  • Learning-based image compression progressed from recurrent neural networks to CNN autoencoders optimized for rate-distortion, with later methods using hyper-priors and autoregressive priors to improve entropy modeling and compression.The cited work reports performance comparable to or better than BPG in MS-SSIM, while the cited autoregressive approach reports better PSNR than BPG.Source: B. Learning based Image and Video… · pp. 2–3
  • The figure’s convolutional layers are specified by filter count, kernel size, and stride; N and M set channel counts, and Φ denotes estimated Gaussian-mixture parameters.The caption also defines downsampling and upsampling arrows, quantization, and arithmetic encoding and decoding.Source: Figure 1 · p. 3
  • Autoregressive priors improve compression but can raise computational cost nearly 100 times, motivating more efficient learning-based frameworks.The passage notes that state-of-the-art methods, [18] outperform traditional methods such as BPG.Source: B. Learning based Image and Video… · p. 3
  • Other image-compression approaches exploit multi-scale relationships, adversarial training, latent importance maps, or inpainting-based prediction and residual coding.The cited works include multi-scale decomposition, GAN-based compression [16], and importance-map guidance.Source: B. Learning based Image and Video… · p. 3
  • Learning-based video compression includes hybrid coding with neural networks, frame interpolation, 3D autoencoders, and latent-space residual coding.The cited systems also estimate motion and blending coefficients or use interpolation loops.Source: B. Learning based Image and Video… · p. 3

III. PROPOSED METHODS

EDIC uses an autoencoder-style compression pipeline with a hyperprior path to estimate entropy-model parameters. It optimizes distortion and bitrate together through a rate-distortion objective.

A. Overall Architecture for Image Compression

  • The encoder maps image x to latent representation y, which is quantized and arithmetic-coded before the decoder reconstructs image x̂.A hyper-encoder derives hyperprior information and latent z from y; quantized z is coded, then the hyper-decoder estimates entropy-model parameters Φ.Source: A. Overall Architecture for Image… · pp. 3–4
  • The training objective is L = λD + R = λd(x, x̂) + H(ŷ) + H(ẑ), balancing distortion and bitrate through λ.The distortion metric may be mean square error or MS-SSIM.Source: A. Overall Architecture for Image… · p. 4
  • The bitrate terms are approximated by the entropies of the quantized image latents ŷ conditioned on hyperlatents ẑ and of ẑ itself.The distributions are denoted p̂_y|z(ŷ|ẑ) and p̂_z(ẑ), respectively.Source: A. Overall Architecture for Image… · p. 4

B. Channel Attention Scheme

The channel attention scheme uses global average pooling and nonlinear transforms to capture channel relationships in latent representations. It reweights features in the encoder and hyper-encoder before quantization and entropy coding.

B. Channel Attention Scheme

  • Channel attention exploits relationships among channels in latent representations y and z, using global average pooling to obtain channel-wise statistics from an input feature map.The input feature map X has height I, width J, and channel dimension C.Source: B. Channel Attention Scheme · p. 4
  • The scheme applies nonlinear transforms to the pooled statistics to capture channel-wise relationships and produce attention values.The output channel-wise attention values are used to re-scale the input feature map, with a residual operation also included in the implementation.Source: B. Channel Attention Scheme · p. 4
  • The module uses global average pooling (GAP) and fully connected (FC) layers.The figure caption identifies these abbreviations in the module structure.Source: Figure 2 · p. 4
  • Reweighted feature maps from the channel attention module are applied to subsequent quantization and entropy coding modules.The module is integrated into the encoder and hyper-encoder networks.Source: B. Channel Attention Scheme · p. 4

C. Gaussian Mixture Model for Entropy Estimation

The section introduces a Gaussian mixture model (GMM) for estimating the distribution of latent representations, addressing limits of a single Gaussian model. Its bit-allocation analysis reports greater savings, especially in edge regions.

C. Gaussian Mixture Model for Entropy Estimation

  • The GMM saves more bits than a single Gaussian model, especially in edge regions.In Fig. 4, brighter regions indicate where the GMM saves more bits.Source: Figure 4 · p. 5, C. Gaussian Mixture Model for Entropy… · p. 5
  • The hyper-encoder and hyper-decoder estimate the mean and variance parameters of the Gaussian model for latent representations.The hyperprior scheme models the latent representations using a Gaussian distribution and represents z with a factorized entropy model.Source: C. Gaussian Mixture Model for Entropy… · pp. 4–5
  • A single Gaussian model has limited representation ability for complex contents, motivating the use of a GMM for latent representations y.The section frames accurate bitrate estimation as critical and describes the single-Gaussian latent model used in prior learning-based systems.Source: C. Gaussian Mixture Model for Entropy… · pp. 4–5
  • The GMM estimates mixture parameters with convolutional layers; the implementation uses two Gaussian models and predicts their means, variances, and weights.For two models, the module outputs 5 × N channels; for F ≥ 3 models, it can output 3 × F × N channels.Source: C. Gaussian Mixture Model for Entropy… · p. 5

D. Decoder-side Enhancement

The decoder-side enhancement module refines reconstructed images with residual blocks that predict high-frequency components. EDIC is also applied to video compression, where the framework handles motion and residual information.

D. Decoder-side Enhancement

  • The decoder-side module applies residual blocks after reconstruction to restore the image and predict high-frequency components.It uses a 3-to-32-channel convolution, three enhancement blocks with three residual blocks each, then a final convolution and residual operation.Source: D. Decoder-side Enhancement · p. 5, Figure 5 · p. 5

E. Extension for Video Compression

1) Motion Estimation and Compression:

The video framework estimates and compresses motion, uses it to predict each frame, then compresses the residual and reconstructs the frame. EDIC modules are applied to both motion and residual coding, with the framework optimized for rate-distortion trade-offs.

1) Motion Estimation and Compression:

2) Motion Compensation, Residual Compression and Frame Reconstruction:

  • The residual is the difference between the current and predicted frames; decoded residuals are added to the prediction to reconstruct the frame.The residual encoder quantizes its representation before the decoder reconstructs the residual.Source: 2) Motion Compensation, Residual… · p. 6

3) Optimization of the framework:

  • The framework minimizes a rate-distortion objective combining frame distortion with the estimated bitrates of motion and residual latent representations.The objective is defined for each time step.Source: 3) Optimization of the framework · p. 6
  • EDIC replaces DVC’s Ballé-based compression of motion and residual information with its channel-attention, Gaussian-mixture bitrate-estimation, and decoder-side enhancement modules.The Gaussian mixture model estimates latent bitrates, while decoder-side enhancement improves reconstructed motion and residual quality.Source: 3) Optimization of the framework · p. 6, Figure 7 · p. 6

IV. EXPERIMENTS

On Kodak image compression, EDIC is comparable to Minnen’s and Lee’s methods at low bitrates and performs better at high bitrates under both PSNR and MS-SSIM. The experiments evaluate the method on Kodak and describe its training setup.

IV. EXPERIMENTS

  • Image rate-distortion performance is averaged over Kodak PhotoCD images; training uses 20,745 high-quality images and randomly cropped 256 × 256 patches.The experiments also evaluate video compression on HEVC test sequences, using Vimeo-90k for training.Source: §IV · p. 6

A. Performance and implementation details for Image Compression

B. Performance and implementation details for Video Compression

On HEVC video sequences, EDIC outperforms DVC and H.264 in PSNR and is comparable to H.265; under MS-SSIM, it outperforms all three for almost all test classes. The video implementation follows DVC settings.

B. Performance and implementation details for Video Compression

  • The video implementation follows DVC settings, using seven-frame Vimeo clips for training and HEVC test sequences with varied resolutions and contents for evaluation.For MSE training, the model is pretrained at λ = 4096 and fine-tuned at λ values from 256 to 2048.Source: B. Performance and implementation… · p. 8
  • On HEVC, EDIC outperforms DVC and H.264 in PSNR and is comparable to H.265; under MS-SSIM it outperforms all three for almost all test classes.The video rate-distortion curves are shown for PSNR and MS-SSIM.Source: B. Performance and implementation… · p. 9, Figure 12 · p. 9

C. Ablation study

The ablation compares EDIC modules against a baseline using a single-Gaussian entropy model. Each module improves performance over the baseline, with attention adding about 0.2 dB and decoder-side enhancement improving every tested model.

C. Ablation study

  • The baseline uses a single Gaussian, with modules added individually and in combinations under the same training strategy.The baseline removes GMM, attention, and decoder-side enhancement; its hyper-decoder predicts mean and variance parameters.Source: 1) Effiectiveness of Each module · pp. 9–10
  • Fig. 14 juxtaposes reconstructed samples from ground truth, JPEG, BPG, Ballé’s method, Minnen’s method, and EDIC for kodim23.png.Source: Figure 14 · p. 10
  • Adding each module improves performance over the baseline; attention adds about 0.2 dB, and decoder-side enhancement improves any tested model.The tested variants add attention, GMM, or decoder-side enhancement individually, omit enhancement from EDIC, or combine all three modules.Source: 1) Effiectiveness of Each module · p. 10

2) Results when using different numbers of Gaussian Models:

Three Gaussian models perform similarly to two, while visualizations show bit allocation varies by image region. The conclusion presents EDIC as an end-to-end image-compression framework also applicable to video compression.

2) Results when using different numbers of Gaussian Models:

  • Three Gaussian models achieve similar performance to two, indicating that increasing the model count does not significantly improve EDIC.The implementation uses two Gaussian models; three models are formed by changing the GMM module’s output channels and adding softmax to normalize weights.Source: 2) Results when using different… · p. 10

D. Visualization

  • Brighter bit-allocation regions indicate more bits; EDIC allocates fewer bits to smooth areas and more to edges, while reconstructed images score higher in PSNR and viewing quality at close compression ratios.The visualizations use latent representations and reconstructed samples from Kodak images.Source: D. Visualization · p. 10, Figure 15 · p. 11

V. CONCLUSION

  • EDIC combines channel-wise attention, GMM bitrate estimation, and decoder-side enhancement in an end-to-end framework that can be used for video compression.The conclusion reports superiority over existing state-of-the-art methods for image and video compression.Source: §V · pp. 10–11
Loading 2002.03370v3…