PaperlayerView exact revision on arXiv

Source-linked AI summary

DVC: An End-to-end Deep Video Compression Framework

Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, Zhiyong Gao

arXiv:1812.00101v3eess.IVcs.CV

TL;DR

Video compression systems traditionally use predictive coding with separately designed components, leaving open how to jointly optimize motion and residual compression end to end. DVC combines neural motion estimation and auto-encoder compression modules under a single rate-distortion loss, outperforming H.264 on PSNR and matching H.265 on MS-SSIM.

  • Problem

    Conventional video compression components are not jointly optimized, and learning-based video compression must address motion representation and whole-system rate-distortion optimization.

  • Method

    DVC jointly learns motion estimation, motion compression, and residual compression with neural networks under a single rate-distortion loss.

  • Results

    DVC outperforms H.264 on PSNR and is on par with H.265 on MS-SSIM.

  • Takeaways & Limitations

    The framework provides a promising end-to-end basis into which improved optical flow, image compression, bidirectional prediction, and rate-control techniques can be plugged.

Abstract

from arXiv · show

Conventional video compression approaches use the predictive coding architecture and encode the corresponding motion information and residual information. In this paper, taking advantage of both classical architecture in the conventional video compression method and the powerful non-linear representation ability of neural networks, we propose the first end-to-end video compression deep model that jointly optimizes all the components for video compression. Specifically, learning based optical flow estimation is utilized to obtain the motion information and reconstruct the current frames. Then we employ two auto-encoder style neural networks to compress the corresponding motion and residual information. All the modules are jointly learned through a single loss function, in which they collaborate with each other by considering the trade-off between reducing the number of compression bits and improving quality of the decoded video. Experimental results show that the proposed approach can outperform the widely used video coding standard H.264 in terms of PSNR and be even on par with the latest standard H.265 in terms of MS-SSIM. Code is released at https://github.com/GuoLusjtu/DVC.

1. Introduction

DVC addresses the lack of end-to-end optimization in conventional video compression by jointly learning its key components with neural networks. It reports better PSNR than H.264 and comparable MS-SSIM to H.265.

  • 1. Introduction: Conventional video compression uses hand-crafted modules that are not jointly optimized end to end.The paper identifies joint optimization of the whole compression system as a desirable improvement.
  • 1. Introduction: End-to-end video compression is challenging because motion must be generated and compressed for video-specific rate-distortion objectives.The paper highlights both the difficulty of learning suitable motion information and the need to optimize motion and residual information together.
  • 1. Introduction: DVC jointly learns motion estimation, motion compression, and residual compression in a single end-to-end model.Its components include neural implementations of motion estimation, compensation, residual and motion compression, quantization, and bit-rate estimation.
  • 1. Introduction: The model jointly optimizes key compression components through a single loss function based on the rate-distortion trade-off.The loss balances reconstructed-frame distortion against the number of encoding bits.
  • 1. Introduction: DVC provides a bridge between conventional video compression and deep model design, allowing improved optical-flow and image-compression models to be plugged into the framework.The authors position the model as a starting point for researchers in video compression, computer vision, and deep learning.
  • 1. Introduction: DVC outperforms H.264 on PSNR and is on par with H.265 on MS-SSIM.These are the paper's headline comparisons for compression quality.

2. Related Work

Related work spans handcrafted image and video codecs, module-specific deep-learning improvements, and an RNN-based video codec. DVC differs by jointly learning motion estimation and compression with whole-system rate-distortion optimization.

  • 2. Related Work: Traditional image codecs rely on handcrafted transforms and separately optimized modules, which may limit compression performance.JPEG is cited as an example using DCT, coefficient quantization, and entropy coding.
  • 2. Related Work: Deep image-compression methods use recurrent or convolutional auto-encoder networks to learn nonlinear representations.These approaches attracted attention as alternatives to handcrafted image-compression techniques.
  • 2. Related Work: Traditional video codecs such as H.264 and H.265 follow predictive coding but cannot be jointly optimized end to end.Their compression components are manually designed despite their efficiency.
  • 2. Related Work: Many deep video-compression studies improve individual modules rather than construct an end-to-end compression scheme.The cited modules include intra prediction, residual coding, mode decision, entropy coding, and post-processing.
  • 2. Related Work: The closest RNN-based approach uses traditional block motion estimation and does not jointly optimize motion processing with other components under rate-distortion training.DVC instead uses DNN-based motion estimation and compression within whole-system rate-distortion optimization.
  • 2. Related Work: Motion estimation is a key video-compression component, with traditional codecs using block-based methods that support hardware implementation.Learning-based optical flow motivates pixel-level motion estimation in the proposed framework.

3. Proposed Method

The proposed framework replaces hand-crafted predictive-coding modules with neural networks while preserving motion, residual, reconstruction, and entropy-coding stages. These components are jointly trained under a rate-distortion objective that accounts for both decoded-frame distortion and representation bit costs.

  • Overview of the Proposed Method: The framework maintains one-to-one correspondence with traditional video compression while replacing hand-crafted transformations with neural components.The proposed stages cover motion estimation and compression, motion compensation, residual transformation and quantization, entropy coding, and frame reconstruction.
  • Motion Estimation and Compression: A CNN estimates optical flow as motion information, and an auto-encoder-style MV network compresses and reconstructs that flow before motion compensation.The optical flow is transformed into a quantized motion representation and decoded into reconstructed motion information.
  • Transform, Quantization and Inverse Transform: A nonlinear residual encoder-decoder replaces the traditional linear transform, maps residuals to representations, quantizes them, and reconstructs the residuals.The quantized residual representation is passed to the residual decoder to obtain the reconstructed residual.
  • Entropy Coding and Bit-Rate Estimation: During training, bit-rate estimation networks model the probability distributions of quantized motion and residual representations to estimate their entropy costs.At testing, the quantized motion and residual representations are coded into bits and transmitted to the decoder.
  • Motion Compensation Network: The motion compensation network warps the previous reconstructed frame and refines it with the reference frame and motion vector to produce the predicted frame.This pixel-wise approach is designed to provide more accurate temporal information and avoid blockness artifacts associated with block-based compensation.
  • Training Strategy: The single rate-distortion loss combines reconstruction distortion with the bit costs of motion and residual representations.The distortion is measured between the original and reconstructed frames using MSE, while λ controls the rate-distortion trade-off.

4. Experiments

Experiments evaluate DVC against established codecs and analyze how motion estimation, compensation, buffering, and motion representation affect rate-distortion performance. DVC generally outperforms H.264 and approaches or exceeds H.265 on MS-SSIM.

  • Experimental Setup: DVC is evaluated on UVG and HEVC Class B, C, D, and E sequences using PSNR, MS-SSIM, and bits per pixel.Training uses Vimeo-90k, while the evaluation datasets provide diversified content and resolutions.
  • Experimental Results: On UVG, DVC achieves about 0.6dB higher quality than Wu et al. at the same Bpp while using one previous reference frame.Wu et al. uses bidirectional prediction and two neighboring frames, leaving scope for multiple-reference-frame extensions.
  • Experimental Results: DVC outperforms H.264 on most datasets for both PSNR and MS-SSIM and achieves similar or better performance than H.265 on MS-SSIM.The loss uses MSE, yet the reconstructed videos retain reasonable MS-SSIM quality.
  • Ablation Study and Model Analysis: Jointly training motion estimation reduces optical-flow cost from 0.044bpp to 0.029bpp while increasing warped-frame PSNR from 27.33dB to 28.17dB.The comparison contrasts fixed pretrained motion estimation with rate-distortion-optimized joint training.
  • Ablation Study and Model Analysis: Removing motion compensation lowers PSNR by about 1.0dB, while replacing buffered reconstructed frames with original frames removes about 0.2dB at the same Bpp.These ablations assess refinement of warped frames and the online reconstructed-frame buffering strategy.
  • Ablation Study and Model Analysis: Without motion information, frame-by-frame coding causes more than a 2dB PSNR drop, and learned zero-magnitude flow regions reduce motion bit costs.In the visual flow analysis, encoding cost decreases from 0.045bpp to 0.038bpp for the joint-trained flow map.

5. Conclusion

The paper presents a fully end-to-end deep learning framework for video compression that combines predictive coding with neural nonlinear representations. Experiments show gains over H.264 and the recent learning-based system, while the framework can accommodate further coding improvements.

  • 5. Conclusion: The proposed framework combines the classic predictive coding scheme with the nonlinear representation ability of deep neural networks.The conclusion positions DVC as a framework for applying deep learning to video compression.
  • 5. Conclusion: Experimental results show that DVC outperforms H.264 and a recent learning-based video compression system.The conclusion reports the comparison without specifying additional numerical values.
  • 5. Conclusion: The framework can incorporate new techniques for optical flow, image compression, bidirectional prediction, and rate control.These components are described as readily pluggable into the proposed framework.

A.1. BDBR and BD-PSNR(BD-MSSSIM) Results

BDBR and BD-PSNR(BD-MSSSIM) compare bitrate savings or quality gains against H.264 at matched quality or bitrate. DVC saves 19.22% bitrate by PSNR and over 29% by MS-SSIM, while H.265 saves 25.06% and 21.73%, respectively.

  • BDBR measures average bitrate savings at the same PSNR or MS-SSIM, while BD-PSNR and BD-MSSSIM measure gains at the same bitrate.These metrics quantify compression efficiency relative to a baseline.
  • 19.22% bitrate savings: DVC versus H.264 when distortion is measured by PSNR.
  • 25.06% bitrate savings: H.265 versus H.264 when distortion is measured by PSNR.
  • More than 29% bitrate savings: DVC versus H.264 when measured by MS-SSIM, compared with H.265's 21.73%.
  • DVC outperforms H.264 in PSNR and MS-SSIM and is comparable to or better than H.265 in these metrics, respectively.

A.2. Compression Performance on the HEVC Class C and Class D

On HEVC Class C and Class D datasets, DVC continues to outperform H.264 in PSNR and MS-SSIM while reducing bitrate relative to H.264.

  • DVC outperforms H.264 in PSNR and MS-SSIM on the HEVC Class C and Class D datasets.
  • 3.88% bitrate savings: DVC versus H.264 on the HEVC Class C dataset.
  • 9.68% bitrate savings: DVC versus H.264 on the HEVC Class D dataset.

A.3. Result Using Kinetics Dataset as the Train Dataset

When trained on the Kinetics dataset, DVC outperforms H.264 and the baseline method in the reported comparison.

  • DVC trained on Kinetics outperforms H.264 in the reported video compression comparison.
  • DVC trained on Kinetics also outperforms the baseline method.

Appendix B. Experimental Settings

The experiments compare DVC with H.264 and H.265 using specified FFmpeg settings and evaluate RGB-channel PSNR, MS-SSIM, and bits per pixel. Bpp is computed from compressed-frame bit cost and resolution, while sequence PSNR averages frame-level PSNRs.

  • H.264 and H.265 compressed videos are generated with FFmpeg using the very fast preset and low-latency tuning.
  • H.264 encoding uses libx264 with CRF Q, GOP-bf 2, and zerolatency-related settings in the specified command.
  • H.265 encoding uses libx265 with CRF Q and keyint set to GOP in the specified command.
  • FR, N, Q, and GOP represent frame rate, encoded-frame count, quality, and GOP size; N is 100 for HEVC datasets, Q is 15, 19, 23, 27, and GOP is 10 for HEVC or 12 for UVG.
  • PSNR and MS-SSIM are evaluated on the RGB channel, with sequence PSNR obtained by averaging PSNR across all video frames.
  • Bpp is calculated as R/W/H for a compressed frame with resolution W × H and bit cost R.

Appendix C. Network Architecture of Our

The motion compensation network uses a warped frame representation and residual blocks with a pre-activation structure.

  • The network represents motion-compensated output using the warped frame w(ˆx_t−1, ˆv_t).
  • The motion compensation network is presented in Figure 3.
  • Residual blocks use a pre-activation structure.
Loading 1812.00101v3…