PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Video Compression with Feature Modulation

Jiahao Li, Bin Li, Yan Lu

arXiv:2402.17414v2cs.CVeess.IV

TL;DR

NVC needs wider quality control and better performance across long prediction chains to become practical. DCVC-FM uses feature modulation, including latent-feature quantization scaling and periodically refreshed temporal features, while supporting both RGB and YUV. Under a single intra-frame setting, it achieves 29.7% bitrate saving over DCVC-DC with 16% MAC reduction.

  • Problem

    NVC remains limited by narrow single-model quality ranges and quality degradation under long prediction chains or large intra-period settings.

  • Method

    DCVC-FM uses learnable quantization-scaled latent features and periodically refreshed temporal features within a conditional coding framework.

  • Results

    29.7% bitrate saving over DCVC-DC is achieved with 16% MAC reduction under a single intra-frame setting.

  • Takeaways & Limitations

    DCVC-FM supports wide quality ranges, long prediction chains, both RGB and YUV colorspaces, and low-precision inference within a practical NVC design.

  • Takeaways & Limitations

    Low-precision inference remains far from real-time, and floating-point entropy coding has cross-platform issues.

Abstract

from arXiv · show

The emerging conditional coding-based neural video codec (NVC) shows superiority over commonly-used residual coding-based codec and the latest NVC already claims to outperform the best traditional codec. However, there still exist critical problems blocking the practicality of NVC. In this paper, we propose a powerful conditional coding-based NVC that solves two critical problems via feature modulation. The first is how to support a wide quality range in a single model. Previous NVC with this capability only supports about 3.8 dB PSNR range on average. To tackle this limitation, we modulate the latent feature of the current frame via the learnable quantization scaler. During the training, we specially design the uniform quantization parameter sampling mechanism to improve the harmonization of encoding and quantization. This results in a better learning of the quantization scaler and helps our NVC support about 11.4 dB PSNR range. The second is how to make NVC still work under a long prediction chain. We expose that the previous SOTA NVC has an obvious quality degradation problem when using a large intra-period setting. To this end, we propose modulating the temporal feature with a periodically refreshing mechanism to boost the quality. %Besides solving the above two problems, we also design a single model that can support both RGB and YUV colorspaces. Notably, under single intra-frame setting, our codec can achieve 29.7\% bitrate saving over previous SOTA NVC with 16\% MACs reduction. Our codec serves as a notable landmark in the journey of NVC evolution. The codes are at https://github.com/microsoft/DCVC.

1. Introduction

Neural video compression offers a potential alternative to increasingly challenging residual coding, but practical deployment requires broader quality control and robust long-chain prediction. DCVC-FM addresses these issues through feature modulation and reports strong compression and efficiency results.

  • Traditional residual coding faces diminishing compression improvements alongside sharply increasing complexity, motivating neural video compression.
  • Conditional coding provides a lower entropy bound and more flexible learned context than residual coding, supporting its use in neural video codecs.
  • 3.8 dB is the average quality range supported by prior single-model NVCs, which is insufficient for practical products.
  • DCVC-FM modulates current-frame latent features with a learnable quantization scaler trained using uniform quantization-parameter sampling.The sampling mechanism improves harmonization between encoding and quantization processes.
  • Long prediction chains cause temporal error accumulation, while frequent intra-frame insertion improves quality at the cost of compression efficiency.For H.265/HM, intra-period 32 provides 23.8% average bitrate saving over intra-period 12.
  • 29.7% bitrate saving over DCVC-DC is achieved with 16% MAC reduction under a single intra-frame setting.DCVC-FM also supports both RGB and YUV colorspaces within one model and demonstrates low-precision inference.

2. Related Work

Neural image and video codecs increasingly use hierarchical, transformer-enhanced, and feature-domain designs, with conditional coding offering greater temporal-context flexibility than residual coding. Despite recent progress, NVC practicality remains limited by narrow quality ranges and small intra-period settings.

  • Neural Image Coding: NIC models commonly use hyperpriors and hierarchical frameworks, with transformers, diffusion models, and lightweight designs explored to improve performance or efficiency.NIC standardization is already under consideration.
  • Residual Coding-Based NVC: Early and many subsequent NVCs follow residual coding, separately compressing motion vectors and residuals while strengthening components such as optical-flow estimation and reference-frame usage.This paradigm includes methods for complex motion areas and multiple temporal references.
  • Conditional Coding-Based NVC: Conditional coding has greater potential because learned feature-domain temporal context is not restricted to the predicted pixel-domain frame or sub-optimal subtraction.Its temporal correlation with the current frame can be learned automatically and its context can be flexibly designed.
  • Conditional Coding-Based NVC: Recent conditional NVCs improve context modeling through temporal feature propagation and combined spatial-temporal entropy models, and DCVC-DC outperforms the under-developing ECM.These advances increase context diversity for compression.
  • Open Problems: NVC practicality remains constrained by limited single-model quality ranges and small intra-period settings, including values such as 10, 12, and 32.DCVC-HEM and DCVC-DC support variable bitrates in one mode but still provide insufficient quality range for varied requirements.

3. Proposed Method

DCVC-FM extends conditional coding with feature modulation to support a wider quality range and more robust long-chain prediction. It also adds rate control, content-adaptive modulation, and periodically refreshed temporal features.

  • 3.1. Overview: DCVC-FM is built on DCVC-DC and uses motion, temporal-context, and frame-coding functions for conditional video compression.Motion is estimated and coded before temporal context conditions frame reconstruction; the resulting feature propagates to the next frame.
  • 3.2. Wide Quality Range in a Single Model: 11.4 dB PSNR range is supported, compared with 3.8 dB for DCVC-DC, through a learnable quantization scaler and uniform quantization-parameter sampling.The sampling mechanism exposes the codec to different rate–distortion trade-offs and harmonizes encoding with quantization.
  • 3.4. Implementation: The codec includes a single model for RGB and YUV colorspaces and supports low-precision inference through implementation improvements.These additions target broader NVC practicality beyond the two feature-modulation mechanisms.
  • 3.2. Wide Quality Range in a Single Model: A global quantization scaler is selected from integer qt, while spatial-channel-wise scalers provide content-adaptive modulation at each feature position.The implementation uses 64 adjustable qt values; spatial-channel-wise scaling is learned through the entropy model.
  • 3.2. Wide Quality Range in a Single Model: DCVC-FM demonstrates rate control by adjusting qt for each frame, with actual bitrate close to fluctuating target bitrates in high- and low-target scenarios.The examples illustrate feasibility rather than a new rate-control design.
  • 3.3. Long Prediction Chain: Longer-video training and periodic temporal-feature refresh address error accumulation under intra-period –1 prediction chains.The refresh period is 32 in the implementation, and refreshed context is extracted from the previous reconstructed frame instead of propagated features.

4. Experimental Results

DCVC-FM is evaluated across RGB and YUV420 settings, quality ranges, long prediction chains, and complexity configurations. It consistently improves compression over prior NVC and traditional codec baselines, while feature modulation mitigates quality degradation and expands controllable quality.

  • RGB colorspace: DCVC-FM achieves 20.3% average bitrate saving over VTM in RGB under 96 frames and intra-period 32.
  • YUV420 colorspace: 24.1% average bitrate saving over VTM is achieved in YUV420 under 96 frames and intra-period 32, compared with 17.0% for DCVC-DC.
  • Long prediction chains: 25.3% bitrate saving over VTM is maintained under intra-period –1 with 96 frames, while DCVC-DC declines from 17.0% to 12.2%.
  • Long prediction chains: With all frames and intra-period –1, DCVC-FM outperforms VTM by 25.5%, while DCVC-DC incurs a 12.8% bitrate increase.Using DCVC-DC as anchor, DCVC-FM achieves 29.7% average bitrate saving across datasets.
  • Quality range: DCVC-FM expands the MCL-JCV quality range to 11.4 dB, compared with 3.7 dB for DCVC-DC.The reported ranges are [31.40, 42.80] for DCVC-FM and [38.66, 42.32] for DCVC-DC.
  • Ablation and complexity: Ablations attribute 16% MAC reduction to structure optimization and 29.7% bitrate saving to periodically refreshing temporal features after longer-video training.The wider quality range and joint RGB/YUV support introduce 3.4% and 4.8% bitrate increases, respectively, relative to preceding ablations.
  • Ablation and complexity: 16-bit inference significantly reduces runtime and halves memory usage, although 32-bit inference is slightly slower despite lower MACs.The optimized implementation is important: without it, bitrate increases by 87.3%.

5. Conclusion and Limitation

The paper concludes that feature modulation addresses wide quality-range support and long prediction-chain degradation while adding joint RGB/YUV operation and low-precision inference. It also identifies deployment limitations in speed and cross-platform entropy coding.

  • Conclusion: Feature modulation enables finely controlled quality ranges through quantization-scaler learning and improves long-chain prediction through periodically refreshed temporal features.
  • Implementation: 16-bit inference reduces memory usage, while the implementation reports complexity comparisons for low-precision operation.
  • Conclusion: DCVC-FM supports both RGB and YUV colorspaces and enables low-precision floating-point inference.
  • Limitation: DCVC-FM remains far from real-time speed, and floating-point inference has a cross-platform entropy-coding issue.

Appendices

The supplementary material provides additional documentation for the proposed DCVC-FM neural video codec.

  • The document identifies itself as supplementary material for the proposed DCVC-FM neural video codec.

A. Test Settings

The appendices define comparative test settings across YUV420 and RGB, including codec configurations, refresh-period analysis, and rate-control implementation details.

  • Test Settings: Comparisons cover NVCs and traditional codecs in both YUV420 and RGB colorspaces.
  • YUV420 colorspace: YUV420 comparisons test HM, VTM, and ECM because traditional codecs and practical applications mainly use this colorspace.
  • RGB colorspace: RGB test videos are converted from YUV420 using BT.709, following prior evaluation protocols.
  • Test Settings: Traditional codecs use their best coding tools and reference structures for both YUV420 and RGB comparisons.
  • Temporal feature modulation: A refresh period of 32 is treated as a trade-off because smaller periods increase bitrate cost and larger periods allow error-contaminated propagation.Performance begins to decay above period 32, at 64 and 96; the best period may vary across videos.
  • Rate control: The rate-control demonstration adjusts quantization parameter qt only on even frames to accommodate frame-level bitrate fluctuation.The paper characterizes this as a simple feasibility demonstration rather than an advanced rate-control algorithm.

D. Reimplemented grid sample

The implementation reduces numerical error in the grid sample function for low-precision tensors, improving fidelity relative to the default function.

  • 16.149% error occurs when 16-bit tensor values are directly fed to the default grid sample function.
  • 0.026% error is obtained with the reimplemented and improved grid sample function.
  • The evaluation document reports RD curves for all datasets using all frames and intra period = –1.

F. Smooth Quality Adjustment in Single Model

DCVC-FM supports 64 quality levels in one model and produces smooth quality adjustment without outliers, enabling precise rate control.

  • 64 quality levels are supported by DCVC-FM in a single model.The experiments test 16 RD points for codec comparisons, while the model itself supports all 64 levels.
  • Smooth quality adjustment is achieved across the tested levels, with no outliers in the RD curves.The paper identifies this smoothness as a prerequisite for precise rate control.
  • The rate-control algorithm updates buffer sizes and the quality value q using current and average frame sizes.The algorithm clips q to the range 0–63 before returning the updated control state.
  • Quality adjustments use conditional q increments or decrements based on buffer deviation and current frame size.The listed thresholds include 10 × current frame size and 0.5 × current frame size conditions.

G. Visual Comparison

The visual comparisons present four examples of DCVC-FM reconstructions against ECM and DCVC-DC, using rate-distortion comparisons in YUV420.

  • Four visual examples compare DCVC-FM with traditional codec ECM and previous SOTA NVC DCVC-DC.
  • The rate-distortion curves compare HEVC B, HEVC C, and HEVC D datasets in YUV420 colorspace with intra-period = –1.Each row represents a dataset, and columns show overall, relatively low, and relatively high quality ranges.
  • DCVC-FM reconstructs textures with greater clarity without additional bitrate costs relative to ECM and DCVC-DC.

H. Traditional Codec Using B Frame Config

The main comparison uses low-delay-B, while hierarchical-B results are also reported; the paper notes a substantial compression-ratio gap and leaves surpassing the best traditional codec in HieB for future work.

  • The main paper uses the low-delay-B setting because DCVC-FM targets the low-delay scenario.Hierarchical-B is a random-access configuration that breaks the low-delay requirement.
  • Surpassing the best traditional codec in the HieB setting is identified as future work.
  • The referenced RD curves for HEVC E, UVG, and MCL-JCV use YUV420 colorspace and intra-period = –1.Each row represents a dataset, with columns for overall, relatively low, and relatively high quality ranges.
Loading 2402.17414v2…