PaperlayerView exact revision on arXiv

Source-linked AI summary

Image and Video Compression with Neural Networks: A Review

Siwei Ma, Xinfeng Zhang, Chuanmin Jia, Zhenghui Zhao, Shiqi Wang, Shanshe Wang

arXiv:1904.03567v2cs.CV

TL;DR

The paper addresses the need for improved image and video compression as visual data growth outpaces compression-ratio gains and traditional hybrid coding faces further-improvement challenges. It reviews neural-network methods, including deep-learning integration with HEVC and end-to-end coding frameworks. The review reports substantial gains for neural networks combined with hybrid video coding, while end-to-end image compression remains behind HEVC.

  • Problem

    Visual data growth exceeds compression-ratio improvement, while traditional hybrid coding faces increasing challenges in achieving further performance gains.

  • Method

    The paper systematically reviews neural-network image and video compression, covering HEVC-integrated techniques, end-to-end frameworks, optimization methods, and semantic-visual representation.

  • Results

    Neural networks combined with traditional hybrid video coding achieve significant improvement over HEVC, while end-to-end neural image compression outperforms JPEG2000 but struggles against HEVC.

  • Takeaways & Limitations

    Neural networks offer promising prediction, compact representation, parallel-friendly computation, and flexible end-to-end optimization for future image and video compression.

Abstract

from arXiv · show

In recent years, the image and video coding technologies have advanced by leaps and bounds. However, due to the popularization of image and video acquisition devices, the growth rate of image and video data is far beyond the improvement of the compression ratio. In particular, it has been widely recognized that there are increasing challenges of pursuing further coding performance improvement within the traditional hybrid coding framework. Deep convolution neural network (CNN) which makes the neural network resurge in recent years and has achieved great success in both artificial intelligent and signal processing fields, also provides a novel and promising solution for image and video compression. In this paper, we provide a systematic, comprehensive and up-to-date review of neural network based image and video compression techniques. The evolution and development of neural network based compression methodologies are introduced for images and video respectively. More specifically, the cutting-edge video coding techniques by leveraging deep learning and HEVC framework are presented and discussed, which promote the state-of-the-art video coding performance substantially. Moreover, the end-to-end image and video coding frameworks based on neural networks are also reviewed, revealing interesting explorations on next generation image and video coding frameworks/standards. The most significant research works on the image and video coding related topics using neural networks are highlighted, and future trends are also envisioned. In particular, the joint compression on semantic and visual information is tentatively explored to formulate high efficiency signal representation structure for both human vision and machine vision, which are the two dominant signal receptor in the age of artificial intelligence.

I. INTRODUCTION

Image and video compression reduce statistical, spatial, visual, and temporal redundancies to provide high-quality services under transmission and storage constraints. This review examines neural-network approaches as traditional coding frameworks face challenges in further improving performance.

  • Image and video compression support high-quality services despite limited transmission and storage capabilities.
  • Image compression evolved from entropy and transform coding toward prediction and quantization techniques integrated in standards such as JPEG and JPEG 2000.JPEG combines block DCT, DC prediction, quantization, and entropy coding; JPEG 2000 uses wavelet transforms and EBCOT.
  • Video coding primarily exploits temporal redundancy through inter-frame prediction and motion compensation within hybrid prediction/transform frameworks.Temporal redundancy arises from high correlation between successive frames captured over short intervals.
  • HEVC extends hybrid coding with 33 angular intra prediction modes, DC mode, and planar mode based on neighboring reconstructed pixels.
  • CNNs provide adaptable, efficiently trained predictors whose prediction quality has surpassed rule-based predictors, motivating their use in image and video compression.The review organizes neural-network compression developments across image methods, video methods, optimization techniques, and newer coding paradigms.

II. INTRODUCTION OF NEURAL NETWORK AND IMAGE/VIDEO COMPRESSION

This section introduces neural networks alongside block-based image coding and hybrid video coding. It highlights neural-network compression as a parallel-friendly, end-to-end alternative trajectory to sequential, independently optimized coding tools.

  • A. Neural Network: Neural networks use layered processing units connected by weights and nonlinear activation functions, with multilayer training commonly performed by backpropagation and stochastic gradient descent.
  • B. Image and Video Compression: JPEG partitions images into 8×8 blocks, applies block-DCT, and compresses quantized coefficients through entropy coding.
  • B. Image and Video Compression: Hybrid video standards such as MPEG-2, H.264/AVC, and HEVC combine transform and prediction, with HEVC adding spatial intra modes and efficient inter prediction.
  • B. Image and Video Compression: Block-based standards operate sequentially and tool-by-tool, limiting parallelism and performance gains compared with end-to-end optimization.
  • The review traces neural-network compression through a technical roadmap and presents representative image and video techniques.

III. PROGRESS OF NEURAL NETWORK BASED IMAGE COMPRESSION

Neural-network image compression progresses from MLP-based transforms and predictors toward integrated end-to-end frameworks. These methods use compact representations, spatial context, and learned optimization, while early designs face rate adaptability constraints.

  • A. Multi-layer Perceptron based Image Coding: MLPs can approximate continuous functions and were applied to image compression by designing unitary transforms for spatial data.The neuron output uses an activation function, bias term, and adjustable weights.
  • A. Multi-layer Perceptron based Image Coding: Early end-to-end neural image coding integrated spatial transformation, transform-coefficient quantization, and binary coding into one optimization problem.
  • A. Multi-layer Perceptron based Image Coding: A decomposition/decision neural network selected binary code combinations, but fixed code counts made adaptation to variable compression ratios difficult.
  • A. Multi-layer Perceptron based Image Coding: Dimension-reduction networks used autoencoder bottlenecks to reduce data dimensionality, while quantization and entropy coding remained separate modules.
  • A. Multi-layer Perceptron based Image Coding: MLP-based predictive coding exploited left and upper spatial context to generate nonlinear image predictors, with higher-order models targeting edges and contours.
  • A. Multi-layer Perceptron based Image Coding: Nested Training Algorithms and alternative connection structures were investigated to reduce training time and improve MLP-based compression efficiency.

B. Random Neural Network based Image Coding

Random neural networks represent image signals as spikes rather than spatial-domain values optimized by gradient backpropagation. Early image-coding systems used encoder/decoder networks whose intermediate layer produced compressed bits.

  • B. Random Neural Network based Image Coding: Random neural networks transmit unit-amplitude spikes modeled as a Poisson process, with positive and negative signals representing excitation and inhibition.
  • B. Random Neural Network based Image Coding: An early random-neural-network image codec used a feedforward encoder/decoder with one intermediate layer that produced compressed bits and reconstructed images.

C. Convolutional Neural Network based Coding

CNN-based coding supports end-to-end image compression but must address quantization’s zero-gradient problem during training. Learned transforms and entropy models can surpass earlier standards, although practical deployment remains constrained by computational considerations.

  • CNNs characterize correlations between neighboring pixels through cascaded convolution operations, supporting hierarchical statistical representations for compression.
  • Quantization produces zero gradients almost everywhere, preventing straightforward backpropagation-based training of end-to-end CNN compression models.
  • Ballé et al. introduced analysis and synthesis transforms and simulated quantization with additive uniform noise to enable end-to-end rate-distortion optimization.The synthesis transform inverses the analysis transform, while convolution, subsampling, and normalization stages are jointly optimized.
  • Learned image compression improved on JPEG2000 using PSNR and MS-SSIM, while later hyperprior and context models achieved performance similar to or better than HEVC intra coding.
  • The review identifies hardware support and energy-efficiency analysis as open practical concerns because autoregressive components are not easily parallelizable.

D. Recurrent Neural Network based Coding

RNN-based image coding uses recurrent memory and progressive multi-iteration architectures to support variable-bitrate compression. These methods address rate control differently from CNN approaches by restricting coding bits through scaled-additive coding.

  • RNNs store recent behaviors in memory units whose recurrent connections transmit transformed information from earlier executions to adapt processing to the current input context.
  • RNN-based image compression still faces difficulties propagating gradients for rate estimation.
  • Toderici et al. used scaled-additive coding to restrict the number of coding bits instead of approximating rate estimation as in CNN compression.
  • The proposed architecture performs multi-iteration progressive compression and supports variational bitrate control through encoding, binarization, and decoding modules.The encoding and decoding networks contain recurrent components.

E. Generative Adversarial Network based Coding

GAN-based coding uses adversarial training to improve decoded-image perceptual quality, while neural networks are also integrated into HEVC intra-prediction modes. Reported gains include substantially smaller files and bitrate savings, but complexity and operating conditions constrain some methods.

  • Generative Adversarial Network based Coding: GAN compression jointly trains generator and discriminator networks, using adversarial loss to improve the subjective quality of reconstructed images.
  • Generative Adversarial Network based Coding: GAN-based compression produced files 2.5 times smaller than JPEG and JPEG2000, 2 times smaller than WebP, and 1.7 times smaller than BPG across quality levels measured by MS-SSIM.
  • Generative Adversarial Network based Coding: Conceptual and extreme-compression GAN frameworks target semantic content generation, including bitrates below 0.1 bpp and different degrees of content generation.
  • HEVC-integrated Coding: Neural video-coding research explores deep-learning improvements to HEVC modules including intra prediction, inter-prediction, quantization, entropy coding, and loop filtering.
  • HEVC-integrated Coding: IPFCN adds a fully connected intra-prediction mode competing with HEVC’s 35 modes and achieved up to 3.0% average BD-rate saving, including up to 4.4% for 4K class-A videos.Its decoding complexity increased by more than 200 times relative to HM-16.9.
  • HEVC-integrated Coding: CNN-based CTU down/upsampling achieved around 5.5% average bitrate saving at low bitrate, but only 0.7% luma saving under common HEVC QPs and degraded when test QPs deviated from training QPs.

B. Neural Network based Inter Prediction

Neural networks improve HEVC inter prediction by generating fractional-pixel references or virtual reference frames. These approaches reduce prediction error and achieve bitrate savings, but fractional-pixel CNNs require many specialized models.

  • Fractional-pixel prediction: FRCNN uses a CNN regressor to generate fractional pixels from an integer-motion-compensated reference block toward the current coding block.Separate CNNs are trained for different fractional-pixel positions, including three half-pixel positions.
  • Fractional-pixel prediction: 3.9%, 2.7% and 1.3% bitrate savings are achieved over HM-16.7 under LDP, LDB and RA configurations, respectively.The gains are reported for the FRCNN approach.
  • Fractional-pixel prediction: FRCNN performance requires up to 120 CNN models trained for slice types, QPs, videos and coding configurations.This model specialization is identified as a cost of the approach.
  • Virtual reference frames: DVRF generates a virtual reference frame with CNN-based frame-rate up conversion, allowing co-located blocks to serve as inter-prediction blocks without motion vectors.The network uses the two nearest bidirectional reference frames as inputs.
  • Virtual reference frames: 4.6% bitrate savings over HM-16.9 and 0.7% over JEM-7.1 are reported on average for DVRF.The method is illustrated in the cited DVRF figures and summarized in Table II.

C. Neural Network based Quantization and Entropy Coding for Video Coding

Neural networks are applied to video quantization and entropy coding to adapt compression to perceptual visibility and improve probability prediction for syntax elements. Reported gains include bitrate savings at matched perceptual quality and for intra-mode coding.

  • Neural-network quantization: An adaptive two-step quantization strategy uses neural networks to predict visibility thresholds and derive CTU quantization steps from patch features.The second-stage regression uses model parameters α, β and γ related to patch features.
  • Neural-network quantization: 11% bitrate saving is obtained for the luma channel against HEVC at the same SSIM-based perceptual quality.The result is reported for the adaptive quantization strategy.
  • Neural-network entropy coding: CNN prediction of 35 intra-prediction-mode probabilities improves CABAC coding by about 9.0% for 8×8 CUs.The input most probable modes are represented as a 35-dimensional one-hot vector, and the output contains probabilities for the 35 modes.
  • Neural-network entropy coding: The same probability-prediction principle is extended to motion vectors, coefficients and transform indices, including variable-length coding of transform indices.The review notes that CNN-based entropy-coding work remains limited, particularly for quantized transform coefficients.

D. Neural Network based Loop Filtering

CNN loop filters remove compression artifacts within or after the HEVC coding loop, with spatial-temporal and content-aware designs improving filtering. Their coding gains are accompanied by model-storage and decoding-complexity burdens.

  • In-loop filtering: RHCNN applies a 13-layer residual highway CNN to HEVC loop filtering, using convolutional layers, ReLUs and identity skip connections.Different QP and frame-type combinations can require separate models.
  • In-loop filtering: STResNet uses reconstructed-frame pairs and the corresponding uncompressed frame to learn spatial-temporal residual filtering.Its model parameters Θ are trained through a loss comparing predicted and uncompressed frames.
  • In-loop filtering: Content-aware multi-model CNN filters train multiple filters and a discriminative selector to choose an optimal filter during testing.The selector helps remove coding overheads associated with filter selection.
  • Efficiency limitations: CNN loop filters require multiple QP-specific models, increasing memory burdens for video codecs.Combining QPs into bands or feeding QP as an input can reduce storage, with usually marginal performance loss relative to QP-dependent models.
  • Efficiency limitations: Decoding time for one deep-learning loop-filter method reaches 15010% without ALF and 12800% with ALF, compared with 123% for ALF itself.The review identifies pruning and quantization as possible optimization directions.
  • Post-filtering: Post-filters use supervised CNNs, universal models, or neighboring high-quality frames to reduce artifacts in compressed images and video.These approaches are reviewed separately from in-loop filtering.

E. New Video Coding Frameworks Based on Neural Network

Neural networks are also used to redesign video coding frameworks and accelerate HEVC decisions. Reviewed systems exploit spatial-temporal prediction or learned classification, with early fully learned coding showing performance comparable to H.264/AVC.

  • Fully learned frameworks: Fully learned video coding with VoxelCNN performs predictive coding through spatial-temporal coherence, followed by iterative analysis/synthesis and binarization.The reviewed scheme lacks entropy coding but still demonstrates the potential of learned video coding.
  • Fully learned frameworks: The VoxelCNN-based scheme shows performance comparable with H.264/AVC despite lacking entropy coding.This result is presented as evidence of potential for future video coding.
  • Predictive representations: LSTM encoder-decoder models learn video representations that can predict future frames through autoencoder and future-predictor networks.The cited approach predicts a long future sequence rather than only one future frame.
  • HEVC optimization: Neural-network fast mode decisions classify blocks and predict coding parameters to skip unnecessary HEVC rate-distortion calculations.A coarse gradient analysis separates homogeneous and edge blocks before CNN processing of edge blocks.

VI. CONCLUSIONS AND OUTLOOK

Neural networks show promise for image and video compression, including substantial gains when combined with traditional hybrid video coding, but practical deployment and broader signal-fidelity goals remain open challenges.

  • End-to-end neural image compression remains in its infancy: it outperforms JPEG2000 but still struggles against HEVC.
  • Neural networks combined with traditional hybrid video coding achieve significant performance improvement over HEVC.
  • Neural networks offer content adaptivity, larger receptive fields, and flexible end-to-end optimization compared with handcrafted coding models.
  • Future compression systems should address both semantic fidelity for machine vision and visual fidelity for human viewing.
  • RD-guided training and adaptive switching across multiple networks are proposed for handling diverse image and video structures.
  • Computation and memory burdens constrain practical deployment, and jointly optimizing compression performance with efficiency remains insufficiently studied.
  • Hierarchical visual signal representations jointly compress feature descriptors and visual content, using decoded features to assist global-motion handling and preserve analysis performance.
  • Neural networks have high prediction and representation efficiency, parallel-friendly hardware suitability, and potential for further compression advances.
Loading 1904.03567v2…