PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Deep Video Compression by Resolution-adaptive Flow Coding

Zhihao Hu, Zhenghao Chen, Dong Xu, Guo Lu, Wanli Ouyang, Shuhang Gu

arXiv:2009.05982v1cs.CV

TL;DR

Compressing pixel-level optical flow is a challenge for learning-based video compression because single-resolution representations do not handle varied motion patterns effectively. RaFC adapts resolution at frame and block levels, and the authors report that it outperforms recent state-of-the-art deep-learning video compression methods on four benchmarks.

  • Problem

    Learning-based video compression faces the challenge of compressing pixel-level optical flow when single-resolution representations must handle varied motion patterns.

    Source: §1 · p. 2

  • Method

    RaFC uses multi-resolution flow maps and motion features, selecting their resolutions at frame and block levels using a rate-distortion criterion.

    Source: §1 · p. 2

  • Results

    RaFC outperforms recent state-of-the-art deep-learning video compression methods across four benchmark datasets.

    Source: §5 · p. 14

  • Takeaways & Limitations

    For smooth local regions, coarse motion-feature blocks save encoding bits without substantially sacrificing distortion.

    Source: §1 · p. 2

Abstract

from arXiv · show

In the learning based video compression approaches, it is an essential issue to compress pixel-level optical flow maps by developing new motion vector (MV) encoders. In this work, we propose a new framework called Resolution-adaptive Flow Coding (RaFC) to effectively compress the flow maps globally and locally, in which we use multi-resolution representations instead of single-resolution representations for both the input flow maps and the output motion features of the MV encoder. To handle complex or simple motion patterns globally, our frame-level scheme RaFC-frame automatically decides the optimal flow map resolution for each video frame. To cope different types of motion patterns locally, our block-level scheme called RaFC-block can also select the optimal resolution for each local block of motion features. In addition, the rate-distortion criterion is applied to both RaFC-frame and RaFC-block and select the optimal motion coding mode for effective flow coding. Comprehensive experiments on four benchmark datasets HEVC, VTL, UVG and MCL-JCV clearly demonstrate the effectiveness of our overall RaFC framework after combing RaFC-frame and RaFC-block for video compression.

1 Introduction

RaFC addresses the difficulty of compressing pixel-level optical flow across varied motion patterns by selecting resolutions at frame and block levels. The combined framework achieves state-of-the-art video compression performance on four benchmark datasets.

  • Deep video compression replaces conventional codec modules, including motion estimation and compensation, with neural networks.Source: §1 · pp. 1–2
  • RaFC targets the difficulty of compressing pixel-level optical flow when a single-resolution representation cannot handle varied motion patterns effectively.The framework adopts multi-resolution representations for flow maps and motion features.Source: §1 · p. 2
  • RaFC-frame selects flow-map resolution per frame, favoring high resolution for complex global motion and low resolution for simple motion.Source: §1 · p. 2
  • The overall RaFC framework combines frame-level flow-map selection with block-level motion-feature selection.The two schemes use multi-resolution representations and select resolutions based on the rate-distortion criterion.Source: §1 · p. 2
  • RaFC achieves state-of-the-art video compression performance on HEVC Class E, VTL, UVG, and MCL-JCV.Source: §1 · p. 2

2 Related Work

Prior image-compression work includes recurrent, auto-encoder, and spatially adaptive approaches, while video compression progresses from hand-crafted codecs to learned systems. RaFC differs by adapting flow-map and motion-feature resolutions for end-to-end optical-flow compression.

2.1 Image Compression

  • Image compression approaches include progressive RNNs, auto-encoders, and spatial methods that predict distributions or use importance maps.Source: §2.1 · p. 3

2.2 Video Compression

  • Traditional H.264 and H.265 use hand-crafted motion estimation and compensation, while early neural video methods were not end-to-end trainable.Source: §2.2 · p. 3
  • Recent video methods use optical flow, but full-resolution flow networks designed for accurate motion maps are not optimal for compression; 3D auto-encoding also remains limited in capturing fine-scale motion.Source: §2.2 · pp. 3–4
  • RaFC contrasts with these approaches by compressing optical-flow maps in an end-to-end trainable framework.Source: §2.2 · p. 4

3 Methodology

RaFC integrates motion and residual coding in a hybrid video-compression pipeline, using rate-distortion optimization to select motion representations at frame and block levels. Multi-resolution flow maps and motion features let the system adapt these choices globally and locally.

3.1 System Overview

  • The hybrid pipeline encodes optical flow as quantized motion features, uses reconstructed flow for motion compensation, encodes the prediction residual, and adds it back to reconstruct the frame.The reconstructed frame is stored as the reference for the next frame.Source: §3.1 · p. 4

3.2 Problem Formulation

  • The system optimizes rate plus weighted distortion, with rate comprising motion and residual bits, and uses rate-distortion optimization to choose the candidate mode with the minimum value.Distortion measures the difference between input and reconstructed frames using MSE or MS-SSIM.Source: §3.2 · p. 6

3.3 Resolution-adaptive Flow Coding (RaFC)

  • RaFC-frame generates two-resolution optical flow maps and selects the map with the lower rate-distortion value for each frame.The paper reports promising results with two-scale flow maps, while noting that more resolutions can be used.Source: §3.3 · pp. 6–7
  • RaFC-block builds multi-scale motion features and selects a resolution for each block using its rate-distortion value.The indicator map records the selected representation for each block; the paper illustrates its generation with a 64 × 64 input and a 2 × 2 map.Source: §3.3 · p. 8
  • The indicator map guides masking of quantized multi-scale features, and coarser features are upsampled before the motion representations are reorganized.The reorganized features are used to obtain the reconstructed flow map.Source: §3.3 · pp. 8–9
  • The combined framework applies block-level feature selection to each candidate frame-level flow map, then chooses the resulting motion mode with the lowest rate-distortion value.The two schemes select resolutions at different stages: flow-map input to the motion-vector encoder and motion-feature output from it.Source: §3.3 · p. 9

4 Experiment

RaFC is evaluated on four benchmark datasets, with experiments comparing its compression performance and analyzing the contributions of its frame- and block-level schemes. Results also examine motion-bit savings and how selected resolutions vary with bitrate.

4.1 Experimental Setup

  • The evaluation uses HEVC Class E, UVG, MCL-JCV, and VTL, with PSNR, MS-SSIM, and bits per pixel as measures.Vimeo-90k clips of seven 448 × 256 frames are used for training; the model is trained in two stages and fine-tuned with MS-SSIM for that criterion.Source: §4.1 · p. 10
  • Training uses PyTorch with CUDA, batch size 4, and Adam; the reported training time is about six days.The learning rate is 1e-4 for the first 1,800,000 steps and 1e-5 for the remaining steps.Source: §4.1 · p. 10

4.2 Experimental Results

  • “Ours*” achieves the best results on all four datasets and gains about 0.5 dB over DVC at 0.1 bpp on UVG.The hyperprior version “Ours” also outperforms DVC on all datasets; “Ours*” further improves results.Source: Figure 4 · p. 11, §4.2 · p. 12

4.3 Ablation Study and Model Analysis

  • RaFC-frame improves on DVC by 0.5 dB at 0.055 bpp on UVG, and combining RaFC-frame with RaFC-block improves further.The ablation uses the hyperprior entropy model for all methods.Source: §4.3 · p. 12, Figure 5 · p. 13
  • RaFC saves up to 70% of motion-information bits at the same PSNR as DVC on HEVC Class E.The reported comparison covers average PSNR over predicted frames; motion bits also form a significantly smaller share of total bits than with DVC.Source: §4.3 · p. 13
  • RaFC-block often selects small blocks near moving-object boundaries and large blocks in smooth areas.The visualization shows a reconstructed optical-flow map and its corresponding block selections for a HEVC Class E frame.Source: Figure 6 · p. 14, §4.3 · p. 14
  • At lower bitrates, low-resolution flow maps and larger blocks are more common; higher bitrates favor high-resolution maps and smaller blocks.The analysis relates larger blocks at low bitrates to saving bits for motion coding.Source: Table 1 · p. 13, Table 2 · p. 13, §4.3 · p. 14

5 Conclusion

RaFC combines frame-level and block-level schemes to compress motion information by selecting resolutions for global and local motion patterns. Experiments on four benchmark datasets show that the framework outperforms recent state-of-the-art deep-learning video-compression methods.

5 Conclusion

  • RaFC combines frame-level flow-map resolution selection with block-level motion-feature resolution selection to address global and local motion patterns.The conclusion reports that experiments on four benchmark datasets show better performance than recent state-of-the-art deep-learning video-compression methods.Source: §5 · p. 14

A Results on the HEVC Class B, Class C and Class D datasets

Supplementary results cover HEVC Class B, C, and D in addition to the four datasets reported in the main paper. The method is competitive with H.265 at low bitrates in PSNR and outperforms all other baselines in MS-SSIM.

A Results on the HEVC Class B, Class C and Class D datasets

  • On HEVC Class B, C, and D, the method is competitive with H.265 at low bitrates in PSNR and outperforms all other baselines in MS-SSIM.The main paper reports HEVC Class E, VTL, UVG, and MCL-JCV results; these additional HEVC classes are reported in the supplementary material.Source: A Results on the HEVC Class B, Class C… · p. 17

B The Command Line for H.264 and H.265

The appendix gives FFmpeg command lines for generating H.264 and H.265 compressed videos from uncompressed YUV input. It specifies the shared input settings and the codec-specific options and parameter values.

B The Command Line for H.264 and H.265

  • Both commands set the pixel format to yuv420p and specify input dimensions, frame rate, frame count, and output file.The H.264 command uses libx264 with zerolatency tuning; the H.265 command uses libx265 with zerolatency tuning.Source: B The Command Line for H.264 and H.265 · p. 17
  • The quality parameter Q is set to 19, 23, 27, and 31; GoP size is 10 for the HEVC dataset and 12 for other datasets.FR and N denote frame rate and the number of frames to be encoded.Source: B The Command Line for H.264 and H.265 · p. 17

C Results of Our Method and H.265 Using Variable GoP Sizes

Under variable GoP sizes, RaFC remains competitive with H.265 in PSNR and outperforms it in MS-SSIM on HEVC Class B, C, and D sequences.

C Results of Our Method and H.265 Using Variable GoP Sizes

Loading 2009.05982v1…