PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Practical Real-Time Neural Video Compression

Zhaoyang Jia, Bin Li, Jiahao Li, Wenxuan Xie, Linfeng Qi, Houqiang Li, Yan Lu

arXiv:2502.20762v2eess.IVcs.CV

TL;DR

Existing NVCs need faster practical coding because operational costs, including memory I/O and function-call overhead, can bottleneck speed beyond computational cost. DCVC-RT reduces these costs through implicit temporal modeling and single low-resolution latents, while adding integerization and rate control. It reaches 125.2/112.8 fps encoding/decoding for 1080p video and saves 21% bitrate versus H.266/VTM, with an int16 speed limitation on current hardware.

  • Problem

    Practical NVCs still struggle to balance coding speed with rate-distortion performance because operational costs can dominate computational cost.

  • Method

    DCVC-RT reduces operational complexity using implicit temporal modeling, single low-resolution latents, model integerization, and module-bank-based rate control.

  • Results

    125.2/112.8 fps encoding/decoding is achieved for 1080p video, with 21% bitrate savings compared with H.266/VTM.

  • Takeaways & Limitations

    The codec combines real-time 1080p coding with high compression ratio and practical cross-device consistency and rate-control functionality.

  • Takeaways & Limitations

    Int16 mode is slower than fp16 because current hardware has limited optimization for int16 inference.

Abstract

from arXiv · show

We introduce a practical real-time neural video codec (NVC) designed to deliver high compression ratio, low latency and broad versatility. In practice, the coding speed of NVCs depends on 1) computational costs, and 2) non-computational operational costs, such as memory I/O and the number of function calls. While most efficient NVCs prioritize reducing computational cost, we identify operational cost as the primary bottleneck to achieving higher coding speed. Leveraging this insight, we introduce a set of efficiency-driven design improvements focused on minimizing operational costs. Specifically, we employ implicit temporal modeling to eliminate complex explicit motion modules, and use single low-resolution latent representations rather than progressive downsampling. These innovations significantly accelerate NVC without sacrificing compression quality. Additionally, we implement model integerization for consistent cross-device coding and a module-bank-based rate control scheme to improve practical adaptability. Experiments show our proposed DCVC-RT achieves an impressive average encoding/decoding speed at 125.2/112.8 fps (frames per second) for 1080p video, while saving an average of 21% in bitrate compared to H.266/VTM. The code is available at https://github.com/microsoft/DCVC.

1 University of Science and Technology of China 2 Microsoft Research Asia

The passage lists four Microsoft contact addresses associated with the section.

  • The listed contact handles are libin and li.jiahao.
  • The listed contact handles also include wenxie and yanlu.
  • All four addresses use the microsoft.com domain.

1. Introduction

The introduction argues that practical NVC progress now depends on real-time coding, not compression ratio alone. DCVC-RT addresses operational bottlenecks with efficiency-focused designs and supports practical deployment features.

  • Motivation: Practical NVCs must improve real-time coding because compression ratio is no longer the primary challenge.The remaining challenge is balancing coding speed with rate-distortion performance and deployability.
  • Motivation: Operational costs such as memory I/O and frequent function calls, rather than computational cost alone, limit practical coding speed.These costs arise from tensor sizes, hardware communication, and autoregressive entropy-model calls.
  • Approach: DCVC-RT removes explicit motion modules and uses implicit temporal modeling to reduce operation frequency while reallocating capacity to frame coding.
  • Approach: A single low-resolution latent representation reduces latent-wise memory I/O overhead compared with progressive downsampling.The proposed latent resolution is 1/8 of the original image size.
  • Practical functionality: Integerization supports cross-device consistency, while a module-bank-based mechanism provides rate-control functionality.

2. Related Works

Related work has substantially improved NVC compression efficiency and practical functionality, but real-time video coding remains difficult. Existing approaches trade off speed, quality, encoding cost, latency, or hardware requirements.

  • Compression efficiency: Recent NVCs outperform leading traditional codecs such as ECM, shifting attention from rate-distortion performance toward functionality and adaptability.
  • Real-time coding: Real-time neural video coding remains underexplored, with existing methods falling short on 1080p performance, encoding latency, or common hardware requirements.Some methods require optimization-based encoding or multi-GPU pipelines.
  • Practical functionality: Cross-device consistency is addressed through deterministic integer calculations or auxiliary calibration bitstreams because floating-point calculations can be nondeterministic.

3. Rethink the Complexity Problem in NVCs

The paper redefines NVC acceleration around operational complexity, showing that memory-related and modular overheads can dominate computational cost. This motivates reducing operational complexity while preserving computational capacity for important modules.

  • Complexity analysis: Reducing channel count decreases computational complexity quadratically but improves inference time only approximately linearly.This indicates that computational cost alone does not determine coding speed.
  • Complexity analysis: Operational complexity includes latent representation size, which affects memory I/O, and module count, which affects operation and function-call overhead.
  • Key findings: Operational complexity, rather than computational complexity, is the main speed bottleneck, so reducing computational cost alone yields marginal inference-time gains.
  • Key findings: Latent representation size dominates at high computational complexity, whereas module count becomes more important at low computational complexity.
  • Design implication: The design preserves computational capacity in critical modules while eliminating less essential ones to improve the rate-distortion-complexity trade-off.

4. Towards Practical Real-Time NVC

DCVC-RT targets practical real-time neural video coding by reducing operational complexity alongside computational cost. Its design combines single-scale latents, implicit temporal modeling, adaptable rate control, and integerized inference for efficient and consistent deployment.

  • Latents at Single Low Resolution: 3.6× faster encoding at 1/8 scale demonstrates the speed advantage of single low-resolution latents over progressive downsampling.The comparison keeps computational complexity equal in the ablation study.
  • Latents at Single Low Resolution: Single 1/8-scale latent learning reduces operational overhead while preserving comparable compression quality under equal model capacity.At 1/8 scale, the receptive field surpasses progressive downsampling and achieves a comparable 0.3% BD-Rate; 1/16 scale instead degrades compression ratio.
  • Implicit Temporal Modelling: Implicit temporal modeling replaces explicit motion estimation and compensation with a simple feature extractor and joint latent processing.The temporal context is concatenated with the current latent, reducing module count and operation frequency.
  • Implicit Temporal Modelling: 3.4× faster encoding makes implicit temporal modeling practical for real-time applications while slightly improving BD-Rate on small motions and reducing it on large motions.It also surpasses explicit motion modeling in scene-change scenarios, where motion cannot be effectively modeled.
  • Module-Bank-Based Rate Control: A module-bank-based rate-control scheme models varied hyperprior distributions across quantization parameters and achieves about 3% bit savings.Separate vector banks extend rate adjustment across encoder, decoder, feature extractor, and reconstruction modules, while hierarchical quality control adjusts frame-level qp offsets.
  • Model Integerization: 16-bit model integerization uses deterministic integer calculations to ensure consistent coding outputs across devices.The implementation maps floating-point features to int16 with K1 = 512, uses int32 convolution accumulators, and applies a lookup table for Sigmoid.

5. Experiments

DCVC-RT is evaluated across video datasets, resolutions, codecs, and hardware, showing strong compression efficiency and real-time coding speed. Its integerized mode preserves compression performance but is slower than fp16 because of limited hardware optimization.

  • DCVC-RT saves 14.0% bitrate against VTM in RGB, compared with 15.8% savings for DCVC-FM.The RGB comparison closely matches DCVC-FM’s compression performance.
  • On UVG, DCVC-RT performs better than VTM across the quality range and is strongest below 0.02 bpp, but loses performance above 40 dB.The high-quality drop is attributed to the lightweight model design; a larger model achieves the highest compression ratio across bitrate ranges while remaining real-time.
  • DCVC-RT consistently improves coding speed by at least 20× across tested resolutions and GPUs, reaching 4K 30 fps on an A100 and 1080p 30 fps on an RTX 2080 Ti.The complexity analysis reports comparable compression ratios against DCVC-DC and DCVC-FM.
  • DCVC-RT (int16) preserves compression gains, outperforming VTM by 18.3%, while reaching 1080p 30 fps on an RTX 4090 and 720p 24 fps on an RTX 2080 Ti.The integerized mode supports 16-bit calculations with minimal compression impact.
  • Int16 coding is significantly slower than fp16 because modern GPUs generally lack dedicated int16 optimization, especially on the A100.On the A100, optimized Tensor Cores make fp16 processing more than four times faster than int16.

6. Conclusion and Limitation

The paper presents DCVC-RT as a practical real-time neural video codec that targets high compression ratio, low latency, and versatility. Its integerized mode remains slower than fp16 because current hardware provides limited optimization for int16 inference.

  • DCVC-RT targets high compression ratio, low latency, and broad versatility as a practical real-time neural video codec.
  • The codec identifies operational cost as the primary coding-speed bottleneck and reduces it through implicit temporal modeling and a single low-resolution latent representation.The design also includes model integerization and module-bank-based rate control for practical adaptability.
  • DCVC-RT achieves 110 fps 1080p coding with 21% bitrate savings compared with H.266/VTM.The paper characterizes this as a landmark result for neural video codec evolution.
  • Int16 coding remains slower than fp16 because current hardware offers limited optimization for int16 inference.The authors expect future hardware optimization and engineering to reduce this gap.

Appendices

The appendices document evaluation settings and explain key DCVC-RT design choices, including conditional coding and a weighted SiLU activation. They also describe the two-step distribution estimation scheme adopted to favor coding speed.

  • Test Settings: The supplementary material evaluates traditional codecs in YUV420 and RGB colorspaces using their best settings for fair, comprehensive comparison.The YUV420 comparison includes HM, VTM, and ECM with officially provided configuration files; RGB testing converts YUV420 inputs using BT.709.
  • Module Structures: DCVC-RT follows a conditional coding manner and uses a weighted SiLU function, WSiLU(x) = x · Sigmoid(α · x).The weighting parameter α is set to 4 by default.
  • Entropy Modeling: The two-step distribution estimation scheme trades a slight performance drop for significantly faster coding.The scheme is selected over more complex entropy models because of its speed advantage.

B.3. Parallel Coding

DCVC-RT reduces latency by running independent neural-network operations and entropy-coding tasks concurrently. This parallel coding approach yields measurable encoding and decoding speedups while supporting the codec’s integerized convolution pipeline.

  • Encoding: During encoding, yq and zq entropy coding is parallelized with decoder-independent processing because these symbols do not depend on decoder inference.Reconstruction generation is typically skipped during encoding when the reference feature is buffered beforehand.
  • Decoding: During decoding, zq entropy decoding runs concurrently with feature extraction, followed by parallel processing of yq entropy decoding and the remaining feature extractor.The two-step scheme first decodes y1q, whose decoding time covers feature-extractor inference time.
  • B.3. Parallel Coding: 12% encoding and 9% decoding speedups result from parallelizing independent entropy-coding and network-inference operations.Entropy coding can run on the CPU while concurrent network inference runs on the GPU without severe hardware resource contention.
  • Model Integerization: The integerization workflow maps floating-point inputs, weights, and biases to int16 representations before convolution and output clipping.The implementation uses K2 = 8192 and K1 = 512 for the integer convolution formulation.

C.2. Results on Rate-Control

DCVC-RT provides rate control within a single model by adjusting the quantization parameter and using a bank of learned rate modules. Interpolation expands the supported rate range beyond the original learned points.

  • Rate Control: Adjusting the quantization parameter enables DCVC-RT to modulate bitrates for applications such as streaming and real-time communication.Figure 9 illustrates rate adjustment on the Cactus sequence.
  • Rate-Adjustment Module Bank: The module bank learns 64 rate points containing latent-modulation vectors and factorized modules for coding z.These components provide rate adjustment within a single model.
  • Rate-Adjustment Module Bank: Interpolation between adjacent module-bank vectors expands the supported rate number to arbitrarily larger values.For a rate between the i-th and i + 1-th modules, linear interpolation produces an intermediate vector.

C.5. Model Integerization Results

Integerized DCVC-RT improves cross-device consistency while retaining most floating-point compression performance. The lightweight model performs strongly at low quality but loses performance at high rates, whereas a larger variant improves that range.

  • Cross-Device Consistency: Nondeterministic floating-point calculations can accumulate cross-device errors into visible artifacts after around 30 frames, while model integerization ensures consistency.The cross-platform example encodes on an NVIDIA A100 GPU and decodes on an RTX 2080Ti.
  • Integerization Results: Approximately 3% BD-Rate decrease separates int16 from fp16 DCVC-RT, indicating only a minor compression-performance change.Tables 6 and 7 report int16 rate-distortion performance.
  • RGB Results: 14.0% average bits saving versus VTM is reported for RGB with intra-period −1, compared with 15.8% for DCVC-FM.The comparison uses BD-rate under the RGB format and all-frame intra-period −1 settings.
  • Visual Comparison: DCVC-RT reconstructs more detailed and clearer structures than VTM and DCVC-FM in visual examples, even at lower bitrates.The visual comparison is presented in Figure 11.
  • Rate-Distortion Curves: At high rates, the lightweight DCVC-RT model’s performance declines because its capacity is limited compared with larger models.Across tested datasets, the model is comparable or superior to DCVC-FM at low quality and surpasses ECM, but declines at high rates.
  • C.9. Performance for a Large Model: Increasing capacity produces DCVC-RT-Large, which improves compression and addresses the small model’s high-bitrate performance drop.The large model uses more channels and DCB blocks and maintains lower complexity than DCVC-FM.
Loading 2502.20762v2…