PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Video Compression with Diverse Contexts

Jiahao Li, Bin Li, Yan Lu

arXiv:2302.14402v3eess.IVcs.CVcs.MM

TL;DR

Neural video codecs use limited contexts, while traditional codecs obtain gains from more modes at high computational cost. DCVC-DC expands temporal and spatial context diversity through hierarchical quality guidance, group-based offsets, and quadtree partitioning. It achieves 23.5% bitrate saving over the previous SOTA NVC and surpasses ECM in RGB and YUV420 colorspaces.

  • Problem

    Existing NVCs have limited context extraction and utilization, motivating methods that improve context learning and use while maintaining low computational cost.

  • Method

    DCVC-DC increases temporal context diversity through hierarchical quality guidance and group-based offset fusion, and increases spatial diversity with quadtree partitioning.

  • Results

    23.5% bitrate saving is achieved over the previous SOTA NVC, and DCVC-DC surpasses ECM in both RGB and YUV420 colorspaces.

  • Takeaways & Limitations

    Diverse contexts enable DCVC-DC to push NVC compression performance beyond the previous SOTA NVC and ECM comparisons reported in the paper.

  • Takeaways & Limitations

    Training uses fixed distortion weights for hierarchical quality patterns, although reinforcement learning may better handle the associated time-series weight decisions.

Abstract

from arXiv · show

For any video codecs, the coding efficiency highly relies on whether the current signal to be encoded can find the relevant contexts from the previous reconstructed signals. Traditional codec has verified more contexts bring substantial coding gain, but in a time-consuming manner. However, for the emerging neural video codec (NVC), its contexts are still limited, leading to low compression ratio. To boost NVC, this paper proposes increasing the context diversity in both temporal and spatial dimensions. First, we guide the model to learn hierarchical quality patterns across frames, which enriches long-term and yet high-quality temporal contexts. Furthermore, to tap the potential of optical flow-based coding framework, we introduce a group-based offset diversity where the cross-group interaction is proposed for better context mining. In addition, this paper also adopts a quadtree-based partition to increase spatial context diversity when encoding the latent representation in parallel. Experiments show that our codec obtains 23.5% bitrate saving over previous SOTA NVC. Better yet, our codec has surpassed the under-developing next generation traditional codec/ECM in both RGB and YUV420 colorspaces, in terms of PSNR. The codes are at https://github.com/microsoft/DCVC.

1. Introduction

Neural video codecs have limited context extraction and utilization, motivating DCVC-DC’s efficient expansion of temporal and spatial context diversity. The resulting codec reports higher compression efficiency than prior NVC and ECM.

  • Motivation: Traditional codecs gain coding efficiency from expanded coding modes, but rate-distortion optimization makes this approach computationally expensive.ECM can require up to half an hour to encode a 1080p frame.
  • Motivation: Most NVCs use limited context extraction, often relying on a single optical flow, which can lead to parameter uncertainty and local optima.Adding traditional codec-like modes can improve context diversity but introduces substantial computational cost through RDO.
  • Temporal contexts: DCVC-DC increases context diversity by learning hierarchical quality patterns and high-quality long-term temporal contexts across frames.The approach is based on the DCVC and DCVC-HEM frameworks and aims to alleviate quality degradation during feature propagation.
  • Temporal contexts: Group-based offset diversity and cross-group fusion strengthen optical-flow-based temporal context mining for complex or large motions.Multiple offsets provide complementary motion alignments and reduce warping errors.
  • Spatial contexts: A quadtree-based partition increases spatial context diversity when encoding latent representations and improves distribution estimation.Compared with checkerboard and dual-spatial models, it supports more diverse correlation modelling while remaining parallel-efficient.
  • Results: 23.5% bitrate saving is achieved over the previous SOTA NVC, while DCVC-DC surpasses ECM in both RGB and YUV420 colorspaces.The reported comparison uses the best compression-ratio configuration for traditional codecs.

2. Related Work

Related neural video codecs have progressed from residual coding toward more flexible conditional coding, but their context modes remain less diverse than traditional codecs. DCVC-DC addresses this gap with time-efficient temporal and spatial context designs.

  • Neural compression: Neural image codecs show coding gains from combining hyperprior, autoregressive, global, and long-range contexts.These approaches motivate increasing context diversity in neural compression.
  • Neural video compression: Residual NVCs predict frames in the pixel domain and remove redundancy through subtraction, whereas conditional NVCs learn feature-domain contexts.Conditional coding offers more flexible design of context definition, learning, and usage.
  • Context diversity: Existing NVCs generally rely on a single optical flow, unlike traditional codecs’ translational, affine, partition, and bi-prediction modes.Single-flow designs are described as vulnerable to epistemic uncertainty and local optima.
  • Context diversity: DCVC-DC provides diverse contexts without additional inference cost through high-quality temporal contexts, offset diversity, and quadtree partitioning.The designs are described as time-efficient alternatives to adding traditional codec modes searched by RDO.

3. Proposed Method

DCVC-DC increases context diversity for neural video compression across temporal and spatial dimensions while preserving parallel-efficient coding. It learns hierarchical quality patterns and diverse motion-aligned contexts, then uses quadtree-partitioned entropy coding.

  • 3.2. Hierarchical Quality Structure: Hierarchical quality training periodically generates high-quality frames and feature representations to provide long-term temporal contexts and reduce error propagation.Frame weights in the rate-distortion loss follow the hierarchical structure, guiding this pattern during training rather than imposing it only at inference.
  • 3.3. Group-Based Offset Diversity: DCVC-DC predicts residual offsets and modulation masks from decoded motion information, then fuses multiple offset-warped feature groups for diverse temporal contexts.Each feature group has separate offsets, while the decoded motion vector provides the base alignment.
  • 3.3. Group-Based Offset Diversity: Cross-group reordering increases interactions among offset-warped groups before fusion without increasing complexity.The reordered groups are fused in contiguous sets, enabling more combinations of temporal contexts.
  • 3.4. Quadtree-Based Partition: Quadtree-based entropy coding partitions the latent representation into four channel groups and codes spatial positions over four parallel steps.Previously coded positions and temporal, hyperprior, and prior-frame contexts are used to predict subsequent probability mass functions.
  • 3.4. Quadtree-Based Partition: The quadtree schedule uses 0, 4, 4, and 8 spatial neighbours across its four steps, averaging twice as many neighbours as checkerboard and dual spatial models.Different groups code different spatial positions at each step, expanding the available neighbourhoods while retaining parallel coding.

4. Experimental Results

Experiments evaluate DCVC-DC against neural and traditional codecs across RGB and YUV420 settings, then analyze its context components and complexity trade-offs. The results show improved compression efficiency, including gains over ECM, while structural optimizations reduce MACs but do not directly reduce runtime.

  • Experimental Settings: Experiments use Vimeo-90k for training and evaluate 96-frame low-delay sequences with an intra period of 32.BD-Rate measures compression ratio, with negative values indicating bitrate savings.
  • RGB Results: 23.5% bitrate saving over DCVC-HEM is achieved on RGB videos measured by PSNR, while DCVC-DC saves 6.4% over ECM on average.The RGB comparisons use BT.709 conversion in the reported tables.
  • RGB Results: 47.6% bitrate saving over VTM is achieved with MS-SSIM on RGB videos, compared with 10.4% for ECM over VTM.DCVC-DC attains the state-of-the-art compression ratio across a wide bitrate range.
  • YUV420 Results: 17.0% average bitrate saving over VTM is achieved in YUV420 measured by weighted PSNR, and DCVC-DC also outperforms ECM on average.The weighted PSNR uses component weights of (6,1,1)/8, and the same network structure supports RGB and YUV420 with different finetuning.
  • Ablation Study: Quadtree partition reduces average BD-Rate from 21.3% to 14.7%, while cross-group interaction reduces it by 6.3% over the preceding configuration.Hierarchical quality structure produces an additional 8.4% gap, indicating benefits from high-quality temporal contexts.
  • Complexity: 19.4% fewer MACs are reported than DCVC-HEM, although actual encoding and decoding time is higher because depthwise convolution has lower computational density under equal MACs.The complexity comparison is tested with 1080p input on an NVIDIA 2080TI.

5. Conclusion and Limitation

DCVC-DC boosts neural video compression by exploiting diverse temporal and spatial contexts, surpassing ECM in both RGB and YUV420 colorspaces. Its hierarchical quality-pattern training still relies on fixed distortion weights, identified as a limitation.

  • Conclusion: DCVC-DC uses long-term, high-quality temporal contexts, cross-group motion alignments, and quadtree partitions to increase context diversity.These designs target error propagation, long-range correlation, complex motion, and spatial context diversity.
  • Conclusion: DCVC-DC surpasses ECM in both RGB and YUV420 colorspaces.The paper describes this as an important milestone for neural video compression.
  • Limitation: The hierarchical quality-pattern training uses fixed distortion weights similar to those in traditional codecs.The authors suggest reinforcement learning as future work for making better temporally dependent weight decisions.

Appendices

The appendices provide supplementary material for the proposed DCVC-DC neural video compression model.

  • Appendices: The appendices provide supplementary material for the proposed neural video compression model, DCVC-DC.They accompany the paper's diverse-contexts approach.

A. Network Structure

The network structure combines diverse-context mechanisms with parallel-efficient computation and variable-bitrate support. Its implementation includes group-based offset prediction, quadtree entropy coding, and convolutional optimizations.

  • A. Network Structure: DCVC-DC is based on DCVC-HEM and emphasizes diverse contexts to improve compression efficiency.Hierarchical quality-pattern learning is mainly performed during training by adjusting the distortion weight in the loss.
  • A. Network Structure: Group-based offset prediction uses decoded motion vectors, reconstructed frames, and propagated features to generate offsets and masks.The implementation divides 48 propagated-feature channels into 16 groups with 2 offsets per group.
  • A. Network Structure: Quadtree partition-based entropy coding predicts latent-distribution parameters across four coding steps using temporal, hyperprior, and previously coded-position contexts.The quantized previous-frame latent representation is also used in distribution-parameter prediction.
  • A. Network Structure: Depthwise separable convolutions and 1x1 regular convolutions reduce computation in the entropy model's DepthConvBlock.The depthwise convolution uses a 3x3 kernel, while the other regular convolution layers use 1x1 kernels.
  • A. Network Structure: Unequal channel settings reduce computation while allocating more channels to the quantized latent representation.Propagated and motion-aligned features use 48 channels, while the quantized latent representation uses 128 channels.
  • A. Network Structure: The codec supports variable bitrates in a single model by using a user-controlled quantization parameter and learnable quantization-step tables.The quantization parameter controls bitrate, while quantization operations are adjusted at higher resolutions for finer rate control.

B. Test Settings

The test settings compare neural and traditional codecs in YUV420 and RGB colorspaces using specified conversion pipelines and codec configurations. RGB testing uses BT.709 conversion and YUV444 as the traditional-codec internal colorspace, while ECM remains under development.

  • B. Test Settings: The evaluation compares neural and traditional codecs in both YUV420 and RGB colorspaces using the pipeline shown in Figure 12.The traditional-codec benchmarks include HM, VTM, and ECM with their respective low-delay configuration files.
  • B. Test Settings: YUV420 testing performs no colorspace conversion.This is the direct YUV420 pipeline described for the test settings.
  • B. Test Settings: RGB testing converts raw YUV420 videos to RGB using BT.709, matching the conversion used to generate the HEVC datasets.The raw RGB videos for those datasets are unavailable, so the videos are converted back using BT.709.
  • B. Test Settings: Traditional codecs convert RGB videos to YUV444 internally before converting reconstructed videos back to RGB for distortion calculation.Using YUV444 as the internal colorspace achieves the best performance in the compared traditional-codec pipelines.
  • B. Test Settings: 21.2% bitrate increase occurs when using setting 5 instead of the selected setting 1 for traditional-codec RGB testing.Setting 1 uses YUV444 as the internal colorspace and is reported as the best-performing configuration.
  • B. Test Settings: ECM-5.0 is still under development and has bugs supporting YUV444 for RGB-video testing that were fixed and verified by the authors.After the fix, ECM-5.0 performs better than VTM-17.0, with similar bitrate savings over VTM-17.0 in YUV420.

C. Results in RGB colorspace with BT.601

Under the stronger VTM-17.0 anchor, DCVC-DC achieves significant bitrate savings for BT.601-converted RGB videos, while the comparison uses PSNR and MS-SSIM quality measures.

  • DCVC-DC does not require retraining when testing RGB videos converted using BT.601.
  • The comparison uses a stronger baseline than the VTM-13.2 version used in prior work, with VTM-13.2 averaging 5.0% higher bitrate than VTM-17.0.
  • 18.0% average bitrate saving is achieved by DCVC-DC over VTM-17.0 for BT.601-converted RGB videos measured by PSNR.
  • VTM-17.0 is used as the anchor, with BD-rate comparisons reported for PSNR and MS-SSIM.

D. Rate-Distortion Curves

The reported RD curves cover RGB and YUV420 videos across broad bitrate ranges. DCVC-DC achieves SOTA compression ratio across these datasets and has smaller PSNR variance than VTM-17.0.

  • DCVC-DC has smaller PSNR variance than VTM-17.0, while hierarchical quality patterns improve compression ratio with negligible visual degradation.
  • DCVC-DC achieves SOTA compression ratio across a wide bitrate range on all reported datasets.
  • The RD curves include RGB videos in BT.709 and YUV420 videos without conversion.

E. Visual Comparison

Visual comparisons show DCVC-DC reconstructing clearer textures without increasing bitrate relative to VTM-17.0 and ECM-5.0. The accompanying figures specify the evaluated colorspaces and quality measures.

  • DCVC-DC reconstructs clearer textures without increasing bitrate compared with VTM-17.0 and ECM-5.0.
  • The RGB comparisons with BT.709 report PSNR in the left column and MS-SSIM in the right column.
  • The YUV420 comparison covers UVG, MCL-JCV, and HEVC B, C, D, and E.
  • Figure 17 provides four visual comparison examples.
Loading 2302.14402v3…