PaperlayerView exact revision on arXiv

Source-linked AI summary

M-LVC: Multiple Frames Prediction for Learned Video Compression

Jianping Lin, Dong Liu, Houqiang Li, Feng Wu

arXiv:2004.10290v1eess.IVcs.CVcs.LG

TL;DR

Low-latency learned video compression methods were limited in their use of previous frames as references. M-LVC explicitly uses multiple reference frames and motion-vector fields, adds refinement networks, and jointly trains the scheme; it outperforms existing learned low-latency methods and H.265 on the reported metrics, while its tested speed remains below real-time decoding.

  • Problem

    Prior low-latency learned video-compression methods were limited in using previous frames as references, with some using only one previous frame.

  • Method

    M-LVC uses multiple reference frames and associated MV fields for prediction and compensation, adds MV and residual refinement networks, and jointly optimizes modules with rate-distortion training.

  • Results

    M-LVC outperforms existing learned video-compression methods for low-latency mode and performs better than H.265 in PSNR and MS-SSIM.

  • Takeaways & Limitations

    Explicit multiple-frame prediction and refinement provide the paper's supported route to improved low-latency learned video compression performance.

  • Takeaways & Limitations

    For 352×256 sequences, the Proposed model runs at 2.7fps encoding and 5.9fps decoding, and the authors identify real-time decoding as future work.

Abstract

from arXiv · show

We propose an end-to-end learned video compression scheme for low-latency scenarios. Previous methods are limited in using the previous one frame as reference. Our method introduces the usage of the previous multiple frames as references. In our scheme, the motion vector (MV) field is calculated between the current frame and the previous one. With multiple reference frames and associated multiple MV fields, our designed network can generate more accurate prediction of the current frame, yielding less residual. Multiple reference frames also help generate MV prediction, which reduces the coding cost of MV field. We use two deep auto-encoders to compress the residual and the MV, respectively. To compensate for the compression error of the auto-encoders, we further design a MV refinement network and a residual refinement network, taking use of the multiple reference frames as well. All the modules in our scheme are jointly optimized through a single rate-distortion loss function. We use a step-by-step training strategy to optimize the entire scheme. Experimental results show that the proposed method outperforms the existing learned video compression methods for low-latency mode. Our method also performs better than H.265 in both PSNR and MS-SSIM. Our code and models are publicly available.

1. Introduction

M-LVC targets low-latency learned video compression by explicitly using multiple previous frames and associated motion vectors as references. It adds refinement modules and joint rate-distortion optimization, achieving stronger reported compression performance than prior learned methods and H.265.

  • Motivation: M-LVC addresses low-latency learned video compression, where prior approaches restricted references to temporally previous frames and often used only one frame.The paper motivates expanding reference information to better exploit temporal correlation.
  • Key idea: Multiple reference frames generate ensemble predictions for the current frame and help extrapolate the next motion-vector field, reducing residual and MV coding cost.The method explicitly combines multiple motion-compensation hypotheses and predicts subsequent motion vectors from multiple MV fields.
  • Contributions: Four modules are introduced: multiple-frame MV prediction, multiple-frame motion compensation, MV refinement, and residual refinement.The ablation study is reported to demonstrate gains from these modules.
  • Training: All modules are jointly optimized with a single rate-distortion loss and a step-by-step training strategy.The training strategy progressively adds modules before jointly fine-tuning them.
  • Results: M-LVC outperforms existing learned video compression methods in low-latency mode and performs better than H.265 in both PSNR and MS-SSIM.Experiments cover different datasets, resolutions, and content types.

2. Related Work

Learned video compression replaces parts of conventional coding pipelines with trained neural components. Low-latency methods differ in how they use temporal references, with M-LVC explicitly using multiple decoded frames and motion-vector fields rather than relying on one frame or an implicit latent state.

  • Learned compression: Learned image compression uses data-driven nonlinear transforms and learned probability estimation instead of handcrafted techniques used in conventional codecs.CNN- and LSTM-based auto-encoders are among the learned image-compression approaches discussed.
  • Prior video compression: Related learned video-compression work is comparatively limited and includes interpolation-based methods that combine traditional motion estimation with image compression.The cited interpolation approaches generate remaining frames hierarchically from compressed key frames.
  • Low-latency methods: Low-latency methods replace conventional motion-estimation, motion-compression, motion-compensation, and residual-compression components with CNN-based components.DVC is described as directly compressing motion information while using one previous frame for compensation.
  • Reference-frame design: Unlike DVC's single reference frame and Rippel et al.'s latent-state approach, M-LVC explicitly uses multiple decoded frames and multiple decoded MV fields.The paper describes the explicit design as more scalable and interpretable than implicit latent-state use.

3. Proposed Method

The proposed low-latency learned video compression scheme uses multiple reconstructed frames and motion-vector fields throughout prediction, refinement, and compensation. Its modules are trained progressively but jointly optimized with a single rate-distortion objective.

  • Compression Workflow: The scheme defines pixel-wise motion and residual representations, compressing motion-vector differences and residuals with separate auto-encoder networks.The decoded motion-vector difference is added to the predicted motion field, while the decoded residual is refined before frame reconstruction.
  • Multiple-Frame Prediction: Multiple reconstructed motion-vector fields predict the current field, while multiple reference frames generate a more accurate motion-compensated prediction.The MV prediction network uses previous reconstructed MV fields in a multi-scale coarse-to-fine process; the compensation network uses four previous reference frames.
  • Refinement: MV refinement reduces quantization errors in reconstructed motion, especially when zero-valued motion differences would otherwise cause inaccurate compensation.The previous reconstructed frame guides refinement, and experiments report improved compression efficiency when it is provided to the refinement network.
  • Refinement: Residual refinement reduces compression error in the decoded residual, particularly at low bit rates, using previous reference frames.The refined residual is added to the predicted frame, which is then cached for coding subsequent frames.
  • Training Strategy: Training from scratch produced unbalanced rates, so the authors add and train modules step by step before jointly fine-tuning the complete system with one rate-distortion loss.The staged procedure progressively introduces motion compensation, compression, multi-frame MV prediction, and both refinement networks.

4. Experiments

Experiments evaluate the proposed low-latency learned video compression scheme across datasets, reference-frame settings, ablations, visualizations, and runtime. The results show gains from multiple-frame prediction and refinement modules, while step-by-step training is important and computational efficiency remains limited.

  • Experimental Setup: The evaluation uses Vimeo-90k for training and HEVC Classes B, C, D, E plus seven 1080p UVG sequences for testing.Training clips are cropped to 192×192 and 16 frames; evaluation uses PSNR, MS-SSIM, and bits per pixel.
  • Experimental Results: The proposed method outperforms DVC and Wu ECCV2018 by a large margin and surpasses H.265 in both PSNR and MS-SSIM.On HEVC Class B at 0.226 bpp, it achieves about 1.2 dB coding gain over DVC.
  • Ablation Study: 0.5∼0.7 dB gain at the same bpp is achieved by adding MAMVP-Net to the baseline.The network also reduces MV coding from 0.042 bpp for the original MV to 0.027 bpp for the MVD.
  • Ablation Study: 0.15 dB gain at high bit rates and about 0.4 dB at low bit rates are achieved by MVRefine-Net.Including the reconstructed previous frame in this network provides about 0.1 dB additional gain consistently.
  • Ablation Study: 0.1∼0.25 dB gain comes from multiple-reference motion compensation, while Residual Refine-Net adds about 0.3 dB at low rates and about 0.2 dB at high rates.These comparisons replace single-reference compensation with MMC-Net and then add residual refinement.
  • Ablation Study: Training the proposed model from scratch produces very poor results, with 0.0002 bpp for MVD and 0.2431 bpp for residual on Kimono at λ = 16.The reported imbalance motivates the step-by-step training strategy.
  • Encoding and Decoding Time: The proposed model runs at 2.7 fps for encoding and 5.9 fps for decoding on a 352×256 sequence using one Titan Xp GPU.The authors identify real-time decoding as future work because MAMVP-Net adds substantial encoding and decoding time.

5. Conclusion

The paper concludes that multiple reference frames improve motion compensation and MV prediction in an end-to-end learned low-latency video compression scheme. MV and residual refinement address compression error, while joint rate-distortion optimization and step-by-step training support the full system.

  • Conclusion: The scheme uses multiple reference frames for motion compensation and motion vector prediction to remove temporal redundancy in low-latency compression.The conclusion identifies these as central components of the proposed end-to-end system.
  • Conclusion: MV and residual refinement modules compensate for auto-encoder compression error and enhance reconstruction quality.Both refinement stages are integrated into the proposed scheme.
  • Conclusion: All modules are jointly optimized with a single rate-distortion loss and a step-by-step training strategy.The conclusion presents this as the optimization approach for the complete system.
  • Conclusion: Experiments show that the method outperforms existing learned video compression methods for low-latency mode.The conclusion reports the overall experimental outcome without specifying a dataset or metric.

A. Proposed Method

The method refines motion vectors and residuals using warped features from multiple previous frames and current predictions. Its architectures use convolutional networks, including dilated convolutions and U-Net-style residual blocks.

  • MV refinement network: The MV refinement network extracts features from decoded motion vectors and the previous decoded frame, then warps multiple-vector features toward the current frame.The warped features are combined with features from the predicted MV and previous decoded frame.
  • MV refinement network: A dilated convolution-based network processes the warped and current features to obtain the final reconstructed motion vector.The dilated convolutions provide a larger receptive field.
  • Residual refinement network: The residual refinement network warps features from four previous decoded frames toward the current frame before further convolutional processing.It also uses features of the predicted current frame and reconstructed residual.
  • Residual refinement network: A U-Net-based CNN with multiple residual blocks produces the refined residual from the warped features and current-frame features.The figure caption specifies 3×3 kernels and 48 output channels within each residual block.

B.1. Ablation Study of Our MAMVP-Net

The ablation study compares four MAMVP-Net variants differing in scale and alignment on the HEVC Class D dataset, using reconstruction quality and bit-rate as evaluation measures.

  • Compared variants: Four MAMVP-Net variants are compared: single-scale without alignment, single-scale with alignment, multi-scale without alignment, and multi-scale with alignment.The proposed multi-scale with alignment model is compared with three simplified versions.
  • Evaluation: The variants are tested on the HEVC Class D dataset using reconstruction quality and bit-rate.The reported reconstruction-quality metric is PSNR, with bit-rates measured in bpp.
  • Result: The proposed multi-scale model with alignment achieves the highest reconstruction quality with the lowest reported bit-rate.

B.2. Visual Results of Our MV Refine-Net

Visualizations on the Kimono sequence show how MV refinement changes decoded motion differences after compression. The refinement restores some non-zero differences removed under the bit-rate constraint.

  • Visual comparison: After compression, the decoded MVD contains more zeros than the original MVD because of the bit-rate constraint.
  • Effect of refinement: MV Refine-Net restores some non-zero MVDs and improves motion-vector accuracy.The visualization compares the original MVD, decoded MVD, and refined MVD.

B.3. Visual Results of Our MMC-Net

Visual results on BasketballPass compare predicted frames produced with Add MVRefine-Net and Add MMC-Net at λ = 64. The former is smoother visually, while the latter has the higher reported PSNR.

  • Visual comparison: The comparison uses the original frame and predictions from Add MVRefine-Net and Add MMC-Net at λ = 64.
  • Quantitative and visual result: 31.97dB PSNR is reported for the Add MMC-Net prediction, versus 31.42dB for Add MVRefine-Net.The Add MVRefine-Net prediction is described as much smoother, including in the wall area.

B.4. Visual Results of Our Residual Refine-Net

The Residual Refine-Net restores details lost when residual quantization smooths the decoded residual, improving the reconstructed frame.

  • The decoded residual is much smoother than the original because of the rate constraint.
  • Residual Refine-Net restores some image details and improves reconstruction quality.

B.5. Compression Performance on the HEVC Class C and E Datasets

On HEVC Class C and E datasets, the proposed method substantially outperforms DVC and matches or exceeds H.265 in PSNR and MS-SSIM.

  • The proposed method outperforms DVC by a large margin on HEVC Class C and E datasets.
  • The proposed method achieves on-par or better compression performance than H.265 in PSNR and MS-SSIM.

B.6. Comparison with Other Learned Video Compression Methods

The paper compares its low-latency method with learned and traditional video codecs across datasets, while noting scenario, objective, and codec-setting differences.

  • Djelouah ICCV2019 achieves a 0.25 ∼0.7dB PSNR gain over the proposed method on MCL JCV.Djelouah targets random-access scenarios and uses an autoregressive entropy model, unlike the proposed low-latency method.
  • Rippel ICCV2019 outperforms the proposed method by about 0.005 MS-SSIM on Xiph 1080p.Rippel is optimized for MS-SSIM, whereas the proposed method is optimized for MSE.
  • The proposed method achieves competitive PSNR with x264 and is on par with x265 in MS-SSIM on UVG and HEVC Class B and D datasets.
  • Figure 15 compares Djelouah ICCV2019, Rippel ICCV2019, and the proposed method on two datasets.
  • Figure 16 reports PSNR in the top row and MS-SSIM in the bottom row for H.264, H.265, DVC, Wu’s method, and the proposed method.
Loading 2004.10290v1…