PaperlayerView exact revision on arXiv

Source-linked AI summary

Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression

Jiahao Li, Bin Li, Yan Lu

arXiv:2207.05894v1eess.IVcs.CVcs.MM

TL;DR

Neural video codecs often reuse image-codec entropy models and therefore do not fully exploit spatial-temporal dependencies. This paper introduces latent and dual spatial priors plus content-adaptive quantization, achieving 18.2% bitrate saving over H.266 (VTM) on UVG at PSNR.

  • Problem

    Most neural video codecs directly reuse image-codec entropy models and do not fully exploit spatial-temporal characteristics when modelling quantized latents.

  • Method

    The codec combines a latent prior for temporal dependencies, a parallel-friendly dual spatial prior, and spatial-channel-wise adaptive quantization.

  • Results

    18.2% bitrate saving over H.266 (VTM) is achieved on UVG at PSNR using the highest compression-ratio configuration.

  • Takeaways & Limitations

    The entropy model supports smooth rate adjustment in a single model while improving rate-distortion performance through content-adaptive bit allocation.

  • Takeaways & Limitations

    The entropy model assumes that q(ỹ_t) follows a Laplace distribution when estimating distribution parameters.

Abstract

from arXiv · show

For neural video codec, it is critical, yet challenging, to design an efficient entropy model which can accurately predict the probability distribution of the quantized latent representation. However, most existing video codecs directly use the ready-made entropy model from image codec to encode the residual or motion, and do not fully leverage the spatial-temporal characteristics in video. To this end, this paper proposes a powerful entropy model which efficiently captures both spatial and temporal dependencies. In particular, we introduce the latent prior which exploits the correlation among the latent representation to squeeze the temporal redundancy. Meanwhile, the dual spatial prior is proposed to reduce the spatial redundancy in a parallel-friendly manner. In addition, our entropy model is also versatile. Besides estimating the probability distribution, our entropy model also generates the quantization step at spatial-channel-wise. This content-adaptive quantization mechanism not only helps our codec achieve the smooth rate adjustment in single model but also improves the final rate-distortion performance by dynamic bit allocation. Experimental results show that, powered by the proposed entropy model, our neural codec can achieve 18.2% bitrate saving on UVG dataset when compared with H.266 (VTM) using the highest compression ratio configuration. It makes a new milestone in the development of neural video codec. The codes are at https://github.com/microsoft/DCVC.

1 INTRODUCTION

The paper targets entropy modelling in neural video codecs by jointly exploiting spatial and temporal dependencies, while adding content-adaptive quantization for single-model rate adjustment and improved rate-distortion performance.

  • Motivation: Most neural video codecs reuse image-codec entropy models instead of fully exploiting video’s spatial-temporal characteristics.Existing approaches mainly optimize latent representations through data flows or network structures.
  • Hybrid entropy model: The proposed latent prior uses previous-frame quantized latents to predict current-frame distributions and squeeze temporal redundancy.Cascaded training forms a propagation chain connecting current latents with longer-range reference frames.
  • Hybrid entropy model: The dual spatial prior reduces spatial redundancy through a parallel-friendly two-step coding scheme that exploits odd-even and cross-channel contexts.Unlike serialized autoregressive priors, it can select context from both odd and even positions.
  • Adaptive quantization: The entropy model generates spatial-channel-wise quantization steps that support smooth rate adjustment in one model and content-adaptive bit allocation.Global, channel-wise, and spatial-channel-wise quantization factors are combined to adapt rates to video contents.
  • Results: 18.2% bitrate saving over H.266 (VTM) is achieved on UVG at PSNR using the highest compression-ratio configuration.The corresponding MS-SSIM saving is 35.1%.

2 RELATED WORK

Related work spans residual, conditional, and 3D autoencoder video-coding frameworks, while entropy-model research increasingly targets temporal and spatial dependencies and single-model rate adjustment.

  • Video-coding frameworks: Residual coding follows motion-compensated prediction with subsequent residual coding, while conditional coding uses temporal frames or features as conditions.Conditional coding has lower or equal entropy bound than residual coding.
  • Entropy models: Most existing codecs directly reuse image-codec entropy models, whereas newer work investigates conditional entropy coding, temporal context priors, and recurrent entropy models.These approaches primarily emphasize temporal correlation, with limited spatial modelling.
  • Entropy models: Autoregressive spatial priors can exploit spatial correlation but impose serialized coding and slow inference.The paper seeks an entropy model that jointly captures spatial-temporal correlation with low complexity.
  • Paper positioning: The proposed latent prior and time-efficient dual spatial prior address temporal-spatial modelling, while multi-granularity quantization enables smooth rate adjustment in one model.Prior methods typically require separate models for each rate point.

3 PROPOSED METHOD

The proposed codec combines temporal and spatial priors to estimate latent distributions efficiently, while multi-granularity quantization enables adaptive rate control. Its dual spatial prior uses two-step coding to improve context modeling without additional coding delay.

  • Framework Overview: The codec uses conditional coding with temporal context generation, contextual encoding/decoding, and frame reconstruction.The current frame is transformed into a quantized latent representation, entropy-coded, decoded, and reconstructed through a frame generator.
  • Hybrid Spatial-Temporal Entropy Model: The entropy model estimates the latent distribution from hyper priors, temporal context, previous decoded latents, and other known information.These inputs provide complementary information, while cascaded training connects latent representations across multiple frames.
  • Hybrid Spatial-Temporal Entropy Model: The latent prior exploits temporal correlation by using the previous frame’s quantized latent representation to predict the current distribution.Cascaded training further builds long-range connections among latent representations across frames.
  • Hybrid Spatial-Temporal Entropy Model: The dual spatial prior splits latents into channel chunks and codes complementary checkerboard positions in two steps.First-step coded positions are fused into contexts for the second step, expanding spatial context while maintaining parallel-friendly processing.
  • Hybrid Spatial-Temporal Entropy Model: The dual spatial prior mines spatial and cross-channel correlations, doubling the spatial-context scope without additional coding delay.The two chunks are added before arithmetic encoding, and the expanded context improves distribution prediction.
  • Rate Adjustment in Single Model: The codec addresses single-model rate adjustment with adaptive multi-granularity quantization, including spatial-channel-wise steps generated by the entropy model.Content-adaptive steps support smooth rate adjustment and allocate smaller steps to important information.

4 EXPERIMENTAL RESULTS

Experiments evaluate the codec across standard datasets, baselines, ablations, rate adjustment, and complexity. The proposed entropy model improves compression through temporal and spatial priors while supporting single-model rate control.

  • 4.2 Comparisons with Previous SOTA Methods: 4.7% average bitrate saving over VTM is achieved across all datasets for PSNR, while MS-SSIM reaches 46.4% average bitrate saving.VTM is used as the anchor in the BD-Rate comparisons.
  • 4.3 Ablation Study: Removing the latent prior increases bitrate by 10.1%, while using only hyper prior or latent prior increases bitrate by 15.6% or 19.3%, respectively.The hyper prior remains the most important input, but the latent prior provides significant additional savings.
  • 4.3 Ablation Study: 14.1% bitrate saving comes from the dual spatial prior, including 6.2% improvement over the checkerboard prior.The auto-regressive prior saves an additional 12.0% bitrate but is not adopted because of very slow encoding and decoding.
  • 4.3 Ablation Study: 11.8% bitrate loss occurs without multi-granularity quantization, and removing spatial-channel-wise quantization increases bits by 8.9%.The spatial-channel-wise step supports content-adaptive bit allocation across video contents.
  • 4.3 Ablation Study: W-Net saves more than 10% bitrate over residual-block networks and 7.9% over a single U-Net, while adding some complexity remains a consideration.The model uses W-Net as a compromise between compression performance and complexity.
  • 4.4 Rate Adjustment: The codec supports smooth rate adjustment by manually varying the global quantization step during testing, using a single model.Figure 6 contrasts this behavior with methods requiring separate models for each rate point.
  • 4.5 Model Complexity: The codec reduces training and storage burden through single-model rate adjustment, while encoding and decoding are slightly slower than Sheng 2021.It is faster than DCVC, whose auto-regressive prior is parallel-unfriendly.

5 CONCLUSION

The paper presents an entropy model that captures spatial and temporal dependencies while enabling single-model rate adjustment. Its codec achieves higher compression than VTM and reports 18.2% bitrate saving on UVG.

  • 5 CONCLUSION: The latent prior exploits correlations across multiple frames, while the dual spatial prior expands parallel-friendly spatial context and captures cross-channel redundancy.The entropy model also generates spatial-channel-wise quantization steps for content-adaptive quantization.
  • 5 CONCLUSION: 18.2% average bitrate saving is achieved on UVG against VTM using its highest compression ratio configuration.The conclusion identifies this result as an important milestone for neural video coding.

Appendices

The appendices provide supplementary material for the proposed hybrid spatial-temporal entropy modelling approach.

  • Appendices: The appendices provide supplementary material for hybrid spatial-temporal entropy modelling in neural video compression.

A NETWORK ARCHITECTURE

The codec combines multi-scale contextual processing with an entropy model that uses hyper, temporal-context, and previous-frame latent priors. Its dual spatial prior supports two-step probability estimation and spatial-channel-wise quantization, while W-Net-based generation improves receptive field and reconstruction.

  • Entropy model: The entropy model consumes hyper, temporal-context, and previous-frame latent priors to model the current quantized latent representation.The hyper prior and temporal context prior each have 192 channels, while the latent prior has 96 channels.
  • Entropy model: The entropy model estimates distribution parameters in two checkerboard coding steps and generates quantization steps at spatial-channel-wise granularity.The second step fuses latent representations coded in the first step before estimating the remaining distribution parameters.
  • Motion-vector encoder and decoder: Motion-vector coding uses an encoder-decoder pair whose multi-granularity quantization and entropy model follow the latent representation pipeline, with motion-specific inputs.The motion-vector encoder outputs a 64-channel latent at 16x downsampled resolution.
  • Frame generator: The frame generator uses a W-Net-based structure instead of plain residual blocks to enlarge the receptive field and improve generation ability.It takes a 32-channel high-resolution feature and a 64-channel original-resolution context, producing the reconstructed frame and a feature for the next frame.
  • Neural image codec: For intra coding, the neural image codec applies similar multi-granularity quantization and entropy modelling, but its entropy-model input contains only the hyper prior.A U-Net is also used to improve image-codec generation ability.

B SETTINGS OF TRADITIONAL CODECS

The experiments compare the proposed codec with x265, HM-16.20, and VTM-13.2 using specified encoder configurations and command-line settings. Traditional-codec comparisons use low-delay configurations with high compression settings and controlled reference-frame and bit-depth choices.

  • Reference codecs: The evaluation compares x265, HM-16.20, and VTM-13.2, representing H.265 and H.266 reference encoders.HM and VTM use low-delay configurations with the highest compression ratio.
  • Reference codecs: The HM and VTM comparisons use four reference frames and 10-bit internal bit depth.The settings are taken from the corresponding low-delay encoder configurations.
  • Encoder commands: The x265 command uses yuv420p input, the veryslow preset, and zerolatency tuning.The command is part of the traditional-codec evaluation setup.
  • Encoder commands: The VTM command specifies low-delay configuration, 8-bit input and output depths, 4:4:4 chroma format, and an intra period of 32.The command also sets frame rate, frame count, dimensions, and level 6.2.

C RATE-DISTORTION CURVES

The paper reports rate-distortion curves on each dataset for both PSNR and MS-SSIM, showing strong compression performance across a wide operating range.

  • Rate-distortion results: The codec achieves state-of-the-art compression ratio across a wide rate range on the evaluated datasets.The rate-distortion curves report both PSNR and MS-SSIM results.

D COMPARISON UNDER DIFFERENT INTRA PERIOD SETTINGS

The paper evaluates both the practical intra period 32 setting and the less commonly used intra period 12 setting. Under intra period 12, the codec is reported to outperform prior neural and traditional codecs.

  • Intra period 32: Intra period 32 is used because prior work reported that intra period 12 harms compression ratio; for HM, period 32 saved 23.8% bitrate on average.The authors describe intra period 32 as closer to practical usage.
  • Intra period 12: Under intra period 12, the codec significantly outperforms previous state-of-the-art neural and traditional codecs in PSNR BD-rate comparisons.VTM with the standard low-delay configuration does not support intra period 12, so VTM-13.2* is used as the anchor.

E VISUAL COMPARISONS

Visual comparisons indicate that the codec achieves higher reconstruction quality without increasing bitrate, including more accurate stripe-color restoration than previous neural codecs.

  • The codec achieves higher reconstruction quality without increasing bitrate compared with previous neural and traditional video codecs.In the RaceHorses example, previous neural codecs show obvious color distortion, whereas this codec restores more accurate stripe color.

F OTHER DETAILS

The codec redesigns its entropy model and quantization mechanism while reusing several components from prior DCVC-based codecs. Its quantizer and motion-compensation pipeline use multi-granularity quantization steps, with additional evaluations reported through BD-Rate tables and rate-distortion figures.

  • The entropy model, quantization mechanism, and frame generator are redesigned from Sheng 2021, while several motion and temporal-context components are reused.Reused components include motion estimation, motion vector coding, temporal context mining, and contextual encoding and decoding.
  • Quantizer: The quantizer divides the latent representation by the quantization step, subtracts the mean, rounds to the nearest integer, and arithmetic-encodes the result.Decoding reverses these operations by arithmetic-decoding the rounded representation, adding the mean, and multiplying by the quantization step.
  • Quantizer: Quantization steps are selected at multiple granularities: a user-set global step is scaled by channel-wise steps to reflect differing channel importance.The described mechanism differs from fixed-QS codecs and supports target-rate control through a global quantization setting.
  • Motion estimation and motion compensation: Motion compensation estimates a dense full-resolution motion vector, transforms it into a latent representation, quantizes and encodes it, then uses the reconstructed vector to warp features and extract temporal context.Motion-vector quantization also uses multi-granularity quantization steps, with quantization and entropy modeling similar to the latent representation pipeline.
  • The evaluation reports image-codec and intra-period-12 BD-Rate comparisons, alongside rate-distortion curves across UVG, MCL-JCV, HEVC classes, and RGB.Additional visual examples cover RaceHorses from HEVC D and videoSRC03 from MCL-JCV.
Loading 2207.05894v1…