PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning for Video Compression with Hierarchical Quality and Recurrent Enhancement

Ren Yang, Fabian Mentzer, Luc Van Gool, Radu Timofte

arXiv:2003.01966v7eess.IVcs.CV

TL;DR

Video compression needs better rate-distortion performance than handcrafted codecs provide, while prior learned methods do not exploit hierarchical quality for references and enhancement. HLVC combines three quality layers, specialized compression networks, and quality-guided recurrent enhancement. Experiments report state-of-the-art learned video compression and better PSNR and MS-SSIM than x265's “Low-Delay P (LDP) very fast” mode.

  • Problem

    Prior learned video compression methods do not learn hierarchical quality for high-quality references and multi-frame post-processing.

  • Method

    HLVC compresses video with three decreasing-quality layers using image compression, BDDC, and SMDC, then applies quality-weighted WRQE enhancement.

  • Results

    HLVC achieves state-of-the-art learned video compression performance and outperforms x265's “Low-Delay P (LDP) very fast” mode in PSNR and MS-SSIM.

  • Takeaways & Limitations

    Hierarchical quality supports compression and enhancement of lower-quality frames using high-quality information without additional enhancement bit-rate.

  • Takeaways & Limitations

    The frame structure is manually set; future work should learn prediction and hierarchical structures automatically.

Abstract

from arXiv · show

In this paper, we propose a Hierarchical Learned Video Compression (HLVC) method with three hierarchical quality layers and a recurrent enhancement network. The frames in the first layer are compressed by an image compression method with the highest quality. Using these frames as references, we propose the Bi-Directional Deep Compression (BDDC) network to compress the second layer with relatively high quality. Then, the third layer frames are compressed with the lowest quality, by the proposed Single Motion Deep Compression (SMDC) network, which adopts a single motion map to estimate the motions of multiple frames, thus saving bits for motion information. In our deep decoder, we develop the Weighted Recurrent Quality Enhancement (WRQE) network, which takes both compressed frames and the bit stream as inputs. In the recurrent cell of WRQE, the memory and update signal are weighted by quality features to reasonably leverage multi-frame information for enhancement. In our HLVC approach, the hierarchical quality benefits the coding efficiency, since the high quality information facilitates the compression and enhancement of low quality frames at encoder and decoder sides, respectively. Finally, the experiments validate that our HLVC approach advances the state-of-the-art of deep video compression methods, and outperforms the "Low-Delay P (LDP) very fast" mode of x265 in terms of both PSNR and MS-SSIM. The project page is at https://github.com/RenYang-home/HLVC.

1. Introduction

The paper targets more efficient high-quality video compression by replacing handcrafted, non-end-to-end codecs with HLVC's hierarchical quality layers and recurrent enhancement. High-quality references support compression and enhancement of lower-quality frames, improving rate-distortion performance without added enhancement bits.

  • HLVC compresses frames in three hierarchical layers with highest, medium, and lowest quality, respectively.
  • High-quality frames improve neighboring-frame compression at the encoder and enhancement at the decoder.The decoder enhancement uses high-quality information from neighboring frames without bit-rate overhead.
  • BDDC uses compressed layer-1 frames as bi-directional references, while SMDC uses one motion map to estimate motions among several frames.These designs target improved reference use and reduced motion-information bit-rate.
  • WRQE weights recurrent cells by quality features and receives compressed frames and bit-stream information for multi-frame enhancement.
  • HLVC achieves state-of-the-art learned video compression performance and outperforms x265's “Low-Delay P (LDP) very fast” mode.

2. Related works

Prior learned video compression methods replace codec components with neural networks, but generally do not learn hierarchical quality for reference-assisted compression and multi-frame enhancement. HLVC addresses this gap by encoding quality information into the bit stream and using it during enhancement.

  • Learned video compression methods replace traditional codec components with DNNs, including motion compensation, residual compression, interpolation, and autoregressive modeling.
  • Existing methods generally do not learn hierarchical quality, limiting high-quality references for compressing other frames and for multi-frame post-processing.
  • Earlier video-enhancement methods include single-frame and multi-frame approaches, but they were designed as post-processing modules for traditional codecs.
  • HLVC encodes each frame's compression quality into the bit stream so enhancement can use accurate quality information alongside compressed frames.

3. The proposed approach

HLVC organizes video frames into three decreasing-quality layers and combines hierarchical references with specialized compression and recurrent enhancement networks. High-quality information supports compression and enhancement of lower-quality frames, while SMDC reduces motion-map bits by sharing motion information across frames.

  • Framework: HLVC compresses frames in three layers with decreasing quality, using image compression for layer 1, BDDC for layer 2, and SMDC for layer 3.Layer 1 frames consume the highest bit-rates and provide references for neighboring frames.
  • Bi-Directional Deep Compression (BDDC): BDDC compresses layer 2 using previous and upcoming compressed layer 1 frames as bi-directional references.Its motion-estimation subnet uses a pyramid network to handle large motions between reference and target frames.
  • Single Motion Deep Compression (SMDC): SMDC compresses multiple layer 3 frames with a single motion map, reducing motion-information bit-rate by exploiting correlated neighboring-frame motions.The network uses compressed frames from layers 1 and 2 as references and applies inverse motion for prediction.
  • Weighted Recurrent Quality Enhancement (WRQE): WRQE enhances decoded frames with recurrent cells weighted by quality features, using compressed frames and quality information from the bit stream.Quality information guides multi-frame enhancement, particularly for low-quality layer 3 frames, without additional stored information.
  • Bi-Directional Deep Compression (BDDC): BDDC performs motion compression and compensation before residual compression, encoding quantized motion and residual representations into bits.Reference frames are backward-warped using compressed motions, then merged for motion compensation.

4. Experiments

Experiments evaluate HLVC on non-overlapping video datasets using PSNR, MS-SSIM, rate-distortion curves, BDBR, and component ablations. HLVC outperforms prior learned methods broadly and improves on H.265 in the reported settings, while ablations quantify the roles of hierarchical quality, single-motion coding, and WRQE.

  • Experimental settings: HLVC is evaluated on JCT-VC and UVG, with PSNR and MS-SSIM as quality metrics and x265 “LDP very fast” as a comparison standard.Training uses Vimeo-90k for BDDC and SMDC and 142 Xiph/VQEG videos for WRQE; test sets do not overlap with training data.
  • Rate-distortion performance: HLVC’s MS-SSIM rate-distortion curves outperform learned approaches and H.265 across low-to-high bit-rates on JCT-VC and UVG.For PSNR, HLVC outperforms DVC and Wu et al.; it exceeds H.265 on JCT-VC and at high bit-rates on UVG.
  • Bit-rate reduction: −35.94% average BDBR versus H.265 is achieved by HLVC’s MS-SSIM model, while its PSNR model achieves −6.10% average BDBR on MS-SSIM.The MS-SSIM model remains at −27.77% BDBR without WRQE, and the PSNR model outperforms DVC on MS-SSIM.
  • Bit-rate reduction: −4.46% average PSNR BDBR versus H.265 is achieved by HLVC’s PSNR model, which also beats H.265 on 14 of 20 test videos.The model has 7.83% bit-rate overhead on JCT-VC Class C but remains best among learned methods in PSNR.
  • Hierarchical quality: Hierarchical quality improves low-quality layer-3 frames while reducing their bit-rate, because higher-quality layers provide references for compression.From the baseline to baseline+HQ, layers 1 and 2 gain bit-rate and PSNR, whereas layer 3 gains PSNR at lower bit-rate.
  • Single motion strategy: SMDC reduces motion-map bits by 23.4%, from 0.0175 bpp to 0.0134 bpp, while PSNR rises from 29.26 dB to 29.47 dB.The total bit-rate decreases from 0.0973 bpp to 0.0969 bpp as more bits become available for residual coding.
  • Recurrent enhancement: WRQE improves low-quality frames by around 1 dB in the BasketballPass example and benefits more from hierarchical quality than from a non-hierarchical model.The enhancement uses quality-dependent recurrent weighting and high-quality neighboring frames to recover information in distorted frames.

5. Conclusion and future work

The paper concludes with HLVC, which combines hierarchical quality layers with recurrent enhancement for learned video compression. Its frame structure is manually set, leaving automatic prediction and hierarchy design as future work.

  • Conclusion: HLVC compresses three frame layers at decreasing quality using image compression, BDDC, and SMDC, then enhances multiple frames with WRQE.WRQE uses compressed frames, quality information, and bit-rate information for recurrent enhancement.
  • Conclusion: The approach’s experiments validate the effectiveness of hierarchical quality and recurrent enhancement for learned video compression.
  • Future work: The frame structure is manually set, motivating future DNNs that automatically design prediction and hierarchical structures.

6. Details of our framework

The framework uses learned motion estimation, compensation, and reconstruction components, with implementation details specified for BDDC, SMDC, and WRQE subnets. Motion estimation is pretrained using warped-frame MSE before joint optimization.

  • ME subnet: The ME subnet uses a 5-level pyramid network and is pretrained by minimizing MSE between warped and target frames.It is then jointly optimized with the whole BDDC network.
  • ME subnet: MSE-optimized motion achieves higher warped-frame PSNR than ground-truth-optical-flow training, improving motion compensation.Figure 10 shows example warped frames and their PSNR values.
  • MC and RC subnets: The MC and RC subnets use CNN-based auto-encoders with the same structure, including GDN and IGDN transformations.Their encoder and decoder parameters are listed in Tables 2 and 3.
  • MP subnet: The MP subnet follows the motion compensation network of [22], using 3 × 3 convolutions, 64 filters in intermediate layers, 3 output filters, and ReLU activations.
  • SMDC network: SMDC reuses the previously described ME, MC, MP, and RC subnet architectures.
  • WRQE network: The WRQE WG subnet uses a bidirectional LSTM with 256 hidden units, 5×5 convolutions, 24 intermediate filters, 3 output filters, and ReLU.

7. Additional experiments

Additional experiments examine implementation settings, motion estimation, SMDC behavior, visual quality, GOP flexibility, and comparisons with alternative x265 configurations and DVC.

  • Experimental settings: The experiments specify x264 and x265 command-line configurations, including quality settings for JCT-VC and UVG datasets.Quality values are 15, 19, 23, 27 for JCT-VC and 11, 15, 19, 23 for UVG.
  • SMDC motion estimation: SMDC estimates multiple-frame motions from a single compressed motion map through inverse operations derived from the motion from frame 2 to frame 0.The resulting motion-compensated frames are illustrated in Figure 12.
  • SMDC motion estimation: SMDC generates x̃1 with higher PSNR than x̃2, likely because x̃1 uses bi-directional motion and a shorter distance to the reference frame.Table 4 reports average PSNR values across JCT-VC videos.
  • SMDC benefits: SMDC reduces motion-information bit-rate by compressing a single motion map and can improve residual compression through higher-quality x̃1.
  • Visual results: The lowest-quality HLVC frames show fewer compression artifacts than H.265 while using a lower average whole-video bit-rate.Figure 13 compares PSNR and MS-SSIM models with H.265.
  • Different GOP sizes: The method adapts to GOP size 12 by inserting additional SMDC modules, achieving BDBR = 1.53% on UVG with PSNR.
  • x265 comparisons: Against x265 hierarchical B, HLVC achieves BDBR = −9.85% and −10.55% on MS-SSIM for the medium and very fast modes, respectively.HLVC does not outperform x265 hierarchical B on PSNR, where BDBR = 20.87%.
  • Ablation comparisons: Against DVC, the baseline+HQ and baseline+HQ+SM ablations achieve BDBR = −5.50% and −7.37%, respectively, on JCT-VC.These ablations compare hierarchical and IPPP frame structures using DVC as the anchor.
Loading 2003.01966v7…