PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Predictive Video Compression with Bi-directional Prediction

Woonsung Park, Munchurl Kim

arXiv:1904.02909v1cs.CV

TL;DR

Deep-learning video compression has received less attention than deep image compression. BP-DVC Net combines bidirectional prediction, optical-flow estimation, and temporal-context entropy coding; the authors report better compression performance than existing video codecs.

  • Problem

    Deep-learning video compression has received less attention than deep image compression.

    Source: §1 · p. 1

  • Method

    BP-DVC Net uses refined optical flows to warp past and future frames for bidirectional prediction, while temporal context among neighboring-frame feature maps supports entropy coding.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    The proposed video compression scheme showed better compression performance than existing video compression codecs.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Temporal context among neighboring-frame feature maps improves coding efficiency.

    Source: §1 · p. 2

    The cited prior temporal-context method is limited in coding efficiency because its auto-encoder must both compress the current frame and perform another task.

    Source: §3.6 · p. 6

Abstract

from arXiv · show

Recently, deep image compression has shown a big progress in terms of coding efficiency and image quality improvement. However, relatively less attention has been put on video compression using deep learning networks. In the paper, we first propose a deep learning based bi-predictive coding network, called BP-DVC Net, for video compression. Learned from the lesson of the conventional video coding, a B-frame coding structure is incorporated in our BP-DVC Net. While the bi-predictive coding in the conventional video codecs requires to transmit to decoder sides the motion vectors for block motion and the residues from prediction, our BP-DVC Net incorporates optical flow estimation networks in both encoder and decoder sides so as not to transmit the motion information to the decoder sides for coding efficiency improvement. Also, a bi-prediction network in the BP-DVC Net is proposed and used to precisely predict the current frame and to yield the resulting residues as small as possible. Furthermore, our BP-DVC Net allows for the compressive feature maps to be entropy-coded using the temporal context among the feature maps of adjacent frames. The BP-DVC Net has an end-to-end video compression architecture with newly designed flow and prediction losses. Experimental results show that the compression performance of our proposed method is comparable to those of H.264, HEVC in terms of PSNR and MS-SSIM.

1. Introduction

The paper addresses deep-learning video compression, a less-studied area than deep-learning image compression, with an end-to-end codec built around bidirectional prediction. BP-DVC Net combines optical-flow estimation, prediction and entropy coding that uses temporal context.

1. Introduction

  • Deep-learning image compression has been intensively studied, while deep-learning video compression has drawn less attention despite video codecs’ use of temporal prediction.Conventional codecs use I-, P-, and B-frames; the paper identifies B-frame coding as the most efficient of these three types.Source: §1 · p. 1
  • BP-DVC Net uses optical flow from future and past frames for bidirectional prediction, then efficiently encodes the resulting residuals.The paper incorporates a B-frame structure, which it describes as more coding-efficient than I- or P-frame coding.Source: §1 · pp. 1–2
  • Optical-flow networks on both encoder and decoder sides estimate flows without transmitting them, while refinement targets accurate flows over distant frames within a GOP.The proposed refinement subnet uses a U-Net structure and a novel optical-flow loss.Source: §1 · p. 2
  • The entropy-coding network exploits temporal context among feature maps of neighboring frames to improve coding efficiency.The paper frames this as a contribution of its end-to-end video-compression scheme.Source: §1 · p. 2

2. Related Work

The related work reviews learned image and video compression, optical-flow estimation, and frame interpolation. It situates BP-DVC Net among end-to-end video codecs and optical-flow-based prediction approaches.

2.1. Deep learning-based image compression

  • CNN autoencoders model bottleneck features for entropy coding; input-adaptive and context-adaptive models [19] outperform BPG, while outperform JPEG2000.The paper adopts an LSTM-based autoencoder as its baseline structure with progressive coding.Source: §2.1 · p. 2

2.2. Deep learning based video compression

  • Learned video-compression methods either replace conventional codec components with DNNs or use end-to-end autoencoders; CNN autoencoders have also been proposed for P-frame prediction [30] [31].The latter predict the current frame using only a previous frame.Source: §2.2 · p. 2

2.3. Deep learning based optical flow estimation

  • PWC-Net uses a learnable feature pyramid and warping of CNN feature maps; the paper uses it as a pretrained optical-flow estimator for bidirectional prediction.Earlier approaches discussed include FlowNetS and FlowNetC, SpyNet, and PWC-Net.Source: §2.3 · p. 3

2.4. Deep learning based frame interpolation

  • Frame interpolation methods include convolution-filtering, phase-based, and optical-flow-based approaches,, and, respectively.The paper proposes optical-flow-based bidirectional prediction with a refinement subnet to improve flow accuracy.Source: §2.4 · p. 3

3. Proposed BP-DVC Net

This section presents BP-DVC Net’s end-to-end video-compression architecture, combining bidirectional prediction, optical-flow estimation, and a ConvLSTM auto-encoder with temporal-context entropy coding. It describes the separation of prediction from residue compression, the dedicated prediction loss and flow-refinement network, and flow estimation at both encoder and decoder so motion information need not be transmitted.

3. Proposed BP-DVC Net

  • BP-DVC Net uses U-Nets for optical-flow estimation and bidirectional prediction, plus a ConvLSTM auto-encoder that progressively reduces the prediction residue.The residue is fed to the auto-encoder at iteration 0, which outputs successive reconstructions.Source: §3.1 · pp. 3–4

3.3.1 Proposed Optical flow estimation network

  • In a 12-frame GOP, frames 1 and 13 are encoded first, then frame 7 is bi-predicted; the scheme derives decoder-side flows without transmitting them.Unlike the cited earlier approach, which transmitted both flow fields, the model uses forward and backward flows estimated between encoded reference frames; experiments use M1,2, M3,3 and M6,6.Source: §3.1 · p. 4, Figure 2 · p. 4
  • The flow network approximates flows under a smooth-motion assumption, then refines them with a small U-Net when long frame distances or fast motion make that assumption unreliable.The decoder can predict the approximated flows from frames available to both encoder and decoder.Source: §3.3.1 · p. 5

3.3.2 Proposed Bi-directional prediction network

  • A second U-Net predicts the current frame by combining two neighboring frames warped with refined flows and their warped contexts, and trains end-to-end with the flow and auto-encoder networks.The prediction network uses a prediction loss.Source: §3.3 · p. 4, §3.3.2 · p. 5

3.5. Proposed entropy coding model

  • For M1,2, the entropy model uses 11 masked 3D-convolution layers with 128 channels and an adjacent frame’s binary code as temporal context for predicting the current code.A skip connection adds the neighboring-frame code to the entropy network output.Source: §3.5 · p. 6, Figure 4 · p. 6

3.3.2 Proposed Bi-directional prediction network

  • A separate bidirectional prediction network produces residuals for the ConvLSTM-based auto-encoder, allowing prediction and residue compression to focus on separate losses.The prediction network uses the proposed prediction loss, while the auto-encoder uses the reconstruction loss.Source: §3.6 · p. 6
  • An optical-flow refinement subnet improves bidirectional prediction; the authors contrast it with Wu et al. using Farnebäck flow and coarse block-motion estimation.The passage says this refinement leads to improved overall compression performance.Source: §3.6 · p. 6
  • Unlike Wu et al., BP-DVC Net estimates bidirectional optical flow at both encoder and decoder, avoiding optical-flow transmission and saving bitrate.Wu et al. transmit motion information, including optical flow; BP-DVC Net derives the motion information at the decoder from its optical-flow estimation network.Source: §3.6 · p. 6

4. Experiments

BP-DVC Net is evaluated against conventional and learned codecs on several video datasets, with ablations examining optical-flow estimation, bidirectional prediction, and entropy coding. The reported comparisons cover PSNR, distortion, and bitrate.

Experimental conditions

  • Training uses Kinetics videos preprocessed to 352×288 and evaluates on VTL, UVG, and HEVC test sequences.The training set contains 30K videos and 450K frames; evaluation uses raw video datasets.Source: §4.1 · pp. 6–7

Experimental results

  • BP-DVC Net outperforms H.264 and Wu’s method on most datasets in PSNR and has similar or better PSNR performance than H.265.The rate-distortion comparisons cover VTL, UVG, and HEVC datasets.Source: Figure 5 · p. 7, §4.2 · p. 7

Experimental conditions

  • Distortion is measured with PSNR and MS-SSIM, while bitrate is measured in bits per pixel; lower Bpp and higher distortion-metric values indicate better reconstruction.Source: §4.1 · p. 7

Ablation study

  • Optical-flow estimation improves coding efficiency over optical transmission and works without transmitting motion information to the decoder.Source: §4.3 · p. 8
  • The bidirectional prediction network has a larger influence on compression performance than the auto-encoder alone, allowing the auto-encoder to focus on residual compression.Source: §4.3 · p. 8
  • The proposed entropy model improves coding efficiency by about 5% using temporal context from neighboring frames, compared with about 2% for the model in.Source: §4.3 · p. 8

5. Conclusion

BP-DVC Net combines bidirectional prediction, optical-flow estimation, and temporal context in entropy coding. The paper reports better compression performance than existing codecs, while noting that its fixed I-frame/B-frame combination may limit performance.

5. Conclusion

  • The proposed scheme is a deep end-to-end video compression model based on bidirectional prediction, with optical-flow estimation and prediction losses improving the base auto-encoder.Temporal context from neighboring frames’ features is also applied in the entropy model to improve compression performance slightly.Source: §5 · p. 8
  • The paper reports better compression performance than existing video compression codecs.Source: §5 · p. 8
  • The fixed combination of I- and B-frames may limit performance; finding an optimal mix of I-, P-, and B-frames is future work.Source: §5 · p. 8

1. The proposed optical flow refinement sub-net

The optical-flow refinement sub-net refines approximate flows that may become less accurate under large motion. It uses a U-Net structure and predicts flow residuals.

1. The proposed optical flow refinement sub-net

  • Large motion may reduce the accuracy of approximated optical flows, motivating refinement before their use.Source: §1 · p. 11
  • The U-Net-based refinement sub-net predicts only optical-flow residuals, using skip connections between interim convolution layers and between the network input and output.Source: §1 · p. 11, Figure A1 · p. 11

2. The proposed bi-directional prediction network

The bidirectional prediction network estimates the current frame from two decoded neighboring frames. It warps both frames and their context features with optical flows before combining them as network input.

2. The proposed bi-directional prediction network

  • The proposed bidirectional prediction network uses a U-Net structure.Source: §2 · p. 11
  • The network predicts the current frame from two decoded neighboring frames, using their respective optical flows to warp both frames and their ResNet18 context features.Source: §2 · p. 12
  • The warped neighboring frames and context features are concatenated and fed to the prediction network, which is trained to produce an output close to the current frame.Source: §2 · p. 12

3. The performance comparisons of R-D curves in terms of MS-SSIM

BP-DVC Net outperforms H.264 [1], H.265, and Wu’s method on MS-SSIM rate-distortion curves. It also outperforms HEVC on MS-SSIM-versus-rate curves for datasets where its PSNR-versus-rate results were inferior.

3. The performance comparisons of R-D curves in terms of MS-SSIM

  • BP-DVC Net exceeds HEVC on MS-SSIM-versus-rate curves for datasets where it showed inferior PSNR-versus-rate results.Source: §3 · p. 12
  • BP-DVC Net yields decoded-image subjective quality comparable to or better than HEVC.Source: §3 · p. 12
  • BP-DVC Net outperforms H.264 [1], H.265, and Wu’s method in MS-SSIM rate-distortion comparisons.Figure A3 covers the VTL, UVG, and HEVC datasets.Source: Figure A3 · p. 12, §3 · p. 12

4. The comparison of subjective quality with both H264 [1] and H265 [2]

The Foreman and YachtRide comparisons show fewer coding artifacts in BP-DVC Net reconstructions than in AVC/H.264 and HEVC. The comparisons also highlight sharper edges around the auricle and helmet brim in Foreman.

4. The comparison of subjective quality with both H264 [1] and H265 [2]

Related papers

Loading 1904.02909v1…