PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Video Compression using Spatio-Temporal Priors

Haojie Liu, Tong Chen, Ming Lu, Qiu Shen, Zhan Ma

arXiv:1902.07383v2eess.IV

TL;DR

The paper addresses the challenges of accurately representing motion and reconstructing frames for temporal inter coding. It proposes a neural compression framework combining spatial and temporal priors, and reports approximately 38% average BD-Rate improvement over HEVC MP.

  • Problem

    Temporal inter coding requires accurate motion representation and high-fidelity reconstructions for compensation.

    Source: §2.2 · p. 3

  • Method

    The neural video-compression framework jointly uses spatial, temporal, and spatio-temporal priors to encode intra texture, motion, and residuals.

    Source: §1 · p. 1

  • Results

    ≈38.12% average BD-Rate gain over HEVC MP was reported.

    Source: §4 · p. 4

  • Takeaways & Limitations

    The reported BD-Rate gains over H.264/AVC and HEVC constitute state-of-the-art coding performance.

    Source: §5 · p. 4

    Bidirectional temporal prediction is deferred to future work.

    Source: §1 · p. 1

Abstract

from arXiv · show

The pursuit of higher compression efficiency continuously drives the advances of video coding technologies. Fundamentally, we wish to find better "predictions" or "priors" that are reconstructed previously to remove the signal dependency efficiently and to accurately model the signal distribution for entropy coding. In this work, we propose a neural video compression framework, leveraging the spatial and temporal priors, independently and jointly to exploit the correlations in intra texture, optical flow based temporal motion and residuals. Spatial priors are generated using downscaled low-resolution features, while temporal priors (from previous reference frames and residuals) are captured using a convolutional neural network based long-short term memory (ConvLSTM) structure in a temporal recurrent fashion. All of these parts are connected and trained jointly towards the optimal rate-distortion performance. Compared with the High-Efficiency Video Coding (HEVC) Main Profile (MP), our method has demonstrated averaged 38% Bjontegaard-Delta Rate (BD-Rate) improvement using standard common test sequences, where the distortion is multi-scale structural similarity (MS-SSIM).

1. INTRODUCTION

The paper introduces an end-to-end neural video-compression framework that combines spatial and temporal priors across intra texture, motion, and residual coding. It reports approximately 38% BD-Rate improvement over HEVC MP.

  • The framework jointly uses spatial, temporal, and joint spatio-temporal priors for intra texture, frame reconstruction, and temporal predictive residual coding.Source: §1 · p. 1
  • Spatial priors come from downscaled same-image representations, while ConvLSTM temporal priors capture long-short dependencies across previously processed frames; motion uses optical flow.Source: §1 · p. 1
  • Evaluation used low-delay IPPP coding, with the first frame intra-coded and subsequent frames inter-coded using unidirectional forward prediction.Source: §1 · p. 1
  • Bidirectional prediction is deferred to future work.Source: §1 · p. 1
  • ≈38% BD-Rate improvement over HEVC MP was reported, with distortion measured using MS-SSIM in decibel scale.Source: §1 · p. 2

2. NEURAL VIDEO COMPRESSION: FROM MODEL-DRIVEN TO DATA-DRIVEN SOLUTION

The section presents NVC as data-driven coding built around learned spatial and temporal priors for intra texture, motion, and residuals.

2.1. Neural Intra Coding

  • Intra coding combines autoregressive context and hyperpriors to predict Gaussian mean and variance, while ICN fuses decoded hyperpriors with latent features for reconstruction.The encoder and decoder use ResNet with GDN activations; quantization is approximated with uniform noise in training and rounding at inference.Source: §2.1 · pp. 2–3

2.2. Neural Inter Coding

  • Inter coding encodes optical flow for reference-frame warping, then processes warped frames degraded by noisy flow or occlusion with ten residual blocks and embedded resampling.The processing network enlarges the receptive field to improve warped-frame quality.Source: §2.2 · p. 3
  • A ConvLSTM recurrent network captures and augments high-frequency priors from reference-frame features to derive a frame for temporal prediction.The input feature vector is extracted from the decoded reference frame.Source: §2.2 · p. 3

2.3. Neural Residual Coding

  • Residual coding augments the intra architecture with ConvLSTM temporal priors to improve residual probability modeling.Its input concatenates the reconstructed frame with ICN hyperprior features.Source: §2.3 · p. 3
  • Residual probabilities assume a Gaussian distribution and are calculated from the cumulative distribution function.The modeled values are pixels of the residual frame.Source: §2.3 · pp. 3–4

3. END-TO-END TRAINING STRATEGY

Training proceeds from separately pretrained intra and flow networks to joint rate-distortion optimization, using a five-frame training sequence from Vimeo.

3. END-TO-END TRAINING STRATEGY

  • The intra and flow networks are pretrained first, then jointly trained with the pretrained networks for overall rate-distortion optimization.The loss combines MS-SSIM distortion, warp loss measured with L1 and total variation, and intra- and inter-frame bit rates.Source: §3 · p. 4
  • The training loss weights adapt to the target overall bit rate and the percentage of inter-coding bits allocated to flow.The warp loss uses L1 norm and total variation loss.Source: §3 · p. 4
  • Training uses Vimeo crops of 192×192 pixels and five frames, sharing weights across the remaining time steps.The learning rate starts at 10^-4 and is halved every 30 epochs.Source: §3 · p. 4

4. EXPERIMENTAL STUDIES

The evaluation reports an average BD-Rate gain over HEVC MP and lower bit rates than competing codecs at similar MS-SSIM quality. Comparisons use common test sequences and an IPPP coding setup.

4. EXPERIMENTAL STUDIES

  • The evaluation measures BD-Rate with MS-SSIM in dB and compares NVC with H.264/AVC HP and HEVC MP.x264 and x265 are used for the respective conventional codecs.Source: §4 · p. 4
  • ≈38.12% average BD-Rate gain is reported over HEVC MP across standard test sequences.The comparison uses IPPP mode with the same GOP of 8.Source: §4 · p. 4
  • At MS-SSIM close to 0.988, NVC snapshots show significant bit-rate reductions against H.264/AVC HP and HEVC MP.Fig. 3 presents snapshots at similar quality; r denotes bit rate in bits per pixel and d denotes MS-SSIM.Source: §4 · p. 4, Figure 3 · p. 4

5. CONCLUSION

The conclusion presents a jointly learned spatio-temporal-prior framework with reported BD-Rate gains over H.264/AVC and HEVC. It identifies bidirectional prediction and adaptive bit allocation as future directions.

5. CONCLUSION

  • The proposed joint spatio-temporal-prior framework reports 38% average BD-Rate gains over H.264/AVC and HEVC.The conclusion describes this as state-of-the-art coding performance.Source: §5 · p. 4
  • Spatial priors come from downscaled image representations, while temporal priors are captured by a recurrent network.Source: §5 · p. 4
  • Future work includes bidirectional temporal prediction and adaptive bit-rate allocation among intra texture, flow motion, and residuals.Source: §5 · p. 4
Loading 1902.07383v2…