PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Contextual Video Compression

Jiahao Li, Bin Li, Yan Lu

arXiv:2109.15047v2eess.IVcs.CVcs.MM

TL;DR

Predictive residue coding is a sub-optimal way to exploit temporal redundancy, motivating the paper’s questions about defining, using, and learning conditions. DCVC answers them with learnable feature-domain context used across encoding, decoding, and entropy modeling. It reports 26.0% bitrate saving over x265 veryslow for 1080p standard test videos, while leaving further condition design open.

  • Problem

    Predictive residue coding uses simple subtraction and is theoretically no more efficient than conditional coding, motivating better ways to exploit frame correlations.

  • Method

    DCVC defines conditions as learnable feature-domain contextual features and uses them in the encoder, decoder, and entropy model, with feature-domain MEMC guiding context learning.

  • Results

    26.0% bitrate saving over x265 using the veryslow preset is reported for 1080p standard test videos.

  • Takeaways & Limitations

    The extensible framework supports conditional coding with richer context for reconstructing high-frequency contents and allows conditions to be flexibly designed.

  • Takeaways & Limitations

    The framework uses a single reference frame, does not supervise context channels, and leaves multiple-reference-frame design and temporal reconstruction stability for future work.

Abstract

from arXiv · show

Most of the existing neural video compression methods adopt the predictive coding framework, which first generates the predicted frame and then encodes its residue with the current frame. However, as for compression ratio, predictive coding is only a sub-optimal solution as it uses simple subtraction operation to remove the redundancy across frames. In this paper, we propose a deep contextual video compression framework to enable a paradigm shift from predictive coding to conditional coding. In particular, we try to answer the following questions: how to define, use, and learn condition under a deep video compression framework. To tap the potential of conditional coding, we propose using feature domain context as condition. This enables us to leverage the high dimension context to carry rich information to both the encoder and the decoder, which helps reconstruct the high-frequency contents for higher video quality. Our framework is also extensible, in which the condition can be flexibly designed. Experiments show that our method can significantly outperform the previous state-of-the-art (SOTA) deep video compression methods. When compared with x265 using veryslow preset, we can achieve 26.0% bitrate saving for 1080P standard test videos.

1 Introduction

The paper argues that residue coding is a sub-optimal form of conditional coding and proposes DCVC, which uses learnable feature-domain context across coding components. The framework is extensible and improves compression efficiency over prior methods.

  • Motivation: Residue coding removes temporal redundancy through subtraction, imposing the strong assumption that each current pixel correlates only with its predicted counterpart.The paper contrasts this with conditional coding, whose entropy is theoretically no greater than residue coding.
  • Motivation: The paper frames conditional coding around three questions: what the condition is, how to use it, and how to learn it.
  • Proposed framework: DCVC defines the condition as learnable contextual features with arbitrary dimensions and uses them in the contextual encoder, decoder, and entropy model.Feature-domain motion estimation and motion compensation guide context extraction, while the entropy model becomes temporally adaptive.
  • Results: 26.0% bitrate saving over x265 using the veryslow preset is reported for 1080p standard test videos.The introduction also reports 16.4% bitrate saving over previous state-of-the-art deep-learning methods.
  • Proposed framework: The framework extends earlier conditional-coding work by jointly addressing encoding, decoding, and entropy modeling without requiring handcrafted content filtering.
  • Contributions: The contributions include spatial-temporal or temporal-prior entropy modeling, richer feature-domain context for high-frequency reconstruction, and an extensible conditional-coding design.

2 Related works

Prior deep video compression work includes non-delay-constrained and delay-constrained predictive approaches, alongside extensions from image compression. DCVC instead applies conditional coding across encoding, decoding, and entropy modeling in a delay-constrained setting.

  • Deep image compression: Deep image compression progressed from compressive autoencoders toward stronger entropy models, including factorized, hyper-prior, autoregressive-context, and Gaussian-mixture approaches.
  • Deep video compression: Non-delay-constrained video methods may use future references or groups of pictures, but they increase delay and GPU memory cost.
  • Deep video compression: Delay-constrained methods reference only previous frames and replace traditional hybrid-codec modules with neural networks.
  • DCVC positioning: DCVC belongs to the delay-constrained category for broader applicability, including real-time communication.
  • DCVC positioning: Unlike earlier conditional-coding methods limited to entropy modeling or encoding, DCVC applies conditional coding to encoding, decoding, and entropy modeling.

3 Proposed method

DCVC replaces residue coding with conditional coding using learned feature-domain context. The framework conditions encoding, decoding, and entropy modeling on context, while feature-domain motion compensation and adaptive context use target both compression efficiency and reconstruction quality.

  • 3.1 The framework of DCVC: DCVC replaces fixed residue subtraction with conditional coding, using a learned context rather than directly encoding the frame difference.The contextual encoder and decoder learn correlations between the current frame and context to remove redundancy.
  • 3.1 The framework of DCVC: Higher-dimensional feature-domain context provides richer, more correlated information, with different channels able to capture different content types.The paper illustrates channels that focus differently, including on high-frequency contents.
  • 3.1 The framework of DCVC: Both frame encoding and decoding are conditioned on context, producing quantized latent codes that the contextual decoder converts into the reconstructed frame.This design lets the network automatically learn how to use contextual information instead of relying on fixed subtraction.
  • 3.4 Results: DCVC outperforms DVCPro across the reported bitrate ranges and reduces reconstruction error for new content caused by motion.The visual comparison reports BPP values of 0.0306 for DCVC and 0.0359 for DVCPro.
  • 3.2 Entropy model: The entropy model combines spatial and temporal priors with hyperprior side information, while a temporal-only variant enables parallel operations for faster encoding and decoding.Spatial priors target higher compression ratio, whereas temporal priors support acceleration.
  • 3.3 Context learning: Feature-domain motion compensation generates context by extracting reference-frame features, warping them with decoded motion, and refining the warped representation.The refinement network addresses spatial discontinuities introduced by warping.

4 Experimental results

DCVC is evaluated against deep-learning and traditional codecs across resolutions, quality metrics, entropy-model choices, and intra-frame coding settings. It achieves broad bitrate advantages, especially on high-resolution videos, with only a small measured inference-time increase over DVCPro.

  • Evaluation setup: DCVC is tested on Vimeo-90k-trained models using HEVC Classes B–E plus MCL-JCV and UVG videos.HEVC test classes span 1080P, 480P, 240P, and 720P; MCL-JCV and UVG provide additional 1080P sequences.
  • Evaluation setup: PSNR and MS-SSIM rate-distortion curves evaluate bitrate cost against reconstruction quality.Figure 5 uses BPP and PSNR, while Figure 6 uses MS-SSIM and fine-tuned DL-based codecs.
  • Compression results: 23.9%, 25.3%, and 26.0% bitrate savings are achieved against x265 veryslow on MCL-JCV, UVG, and HEVC Class B, respectively.Savings are also 5.8%, 17.5%, and 11.9% on HEVC Classes C, D, and E; DCVC outperforms DVCPro and x265 across the tested video characteristics.
  • Compression results: DCVC achieves larger improvements on high-resolution videos because higher-dimensional feature context carries richer information for reconstructing high-frequency contents.The reported explanation links high-resolution video textures with the usefulness of feature-domain context.
  • Complexity: 2268G MACs for DCVC versus 2014G for DVCPro represent about a 13% increase, while 1080P inference time rises by only about 1%.Measured P40-GPU inference times are 857 ms for DCVC and 849 ms for DVCPro, attributed mainly to GPU parallelism.
  • Ablation studies: Ablations show that concatenating RGB prediction or context features improves compression, while temporal and spatial priors each help and together improve performance further.The final hyper-prior-plus-temporal-prior solution is preferred when trading complexity against compression ratio; the temporal prior also supports parallel operations.
  • Baseline comparisons: With the same SOTA DL-based intra-frame coding, DCVC significantly outperforms DVCPro in comparisons including public and retested baselines.DVC and DVCPro are retested with the same intra-frame coding for fair comparison; Figure 7 also distinguishes public results and cheng2020-anchor intra coding.

5 Discussion

The discussion frames DCVC as a conditional-coding solution with a lower entropy bound than residue coding, while emphasizing that condition design remains open. The paper also identifies channel redundancy, single-reference context, and temporal stability as unresolved issues.

  • Discussion: DCVC replaces the assumption that inter-frame prediction is always most efficient with conditional coding that can learn temporal and spatial correlations.The paper states that this adaptability is especially relevant to encoding new contents.
  • Open questions: Condition definition, usage, and learning remain open questions despite DCVC providing one effective solution.The framework is described as extensible, with potential for more efficient condition design.
  • Limitations and future work: The method does not supervise context channels, so redundancy across channels may limit use of high-dimensional context.Future work targets eliminating channel redundancy to maximize context utilization.
  • Limitations and future work: Context generation uses a single reference frame, leaving multi-reference conditional coding as a promising direction.The paper notes that traditional codecs have shown performance gains from using more reference frames.
  • Limitations and future work: Temporal stability of reconstruction quality is not considered and may require post-processing or additional training supervision.The paper gives temporal-stability loss as an example of possible supervision.

Appendices

The appendices provide detailed network structures, training strategies, and additional experimental results supporting the proposed DCVC framework.

  • Appendices: The appendices contain detailed network structures, training strategies, and additional experimental results.These materials are included to demonstrate the effectiveness of DCVC.

A Network Architecture

The architecture extracts and refines contextual features, then uses them alongside current-frame information for contextual encoding and decoding. Its entropy models encode latent codes and motion vectors, while the overall loss accounts for four bitstream components.

  • Contextual encoder and decoder: Contextual encoding concatenates the current frame with context and produces 16x down-sampled latent codes with dimension 96.The contextual decoder upsamples these codes and concatenates the result with context to generate the reconstruction.
  • Feature extraction and context refinement: Feature extraction and context refinement each contain a convolution layer and a residual block.The networks remain shallow to limit complexity.
  • Training loss: The total rate-distortion loss accounts for current-frame latents, motion-vector latents, and their two corresponding hyper priors.These four components are represented by ŷt, ĝt, ẑt, and ŝt.
  • Motion vector generation: Motion-vector generation combines optical-flow motion estimation with motion-vector encoding, decoding, and refinement networks.The encoder and decoder structures follow DVCPro, with refinement included in the decoder.
  • Entropy model: The temporal-prior entropy network uses convolution layers with stride 2 and GDN, while other entropy components follow an image-compression model.Motion-vector latent codes also have a corresponding entropy model.

B Progressive training

Training proceeds progressively from motion estimation to reconstruction, contextual coding, and finally end-to-end optimization. This schedule is designed to stabilize training before optimizing the complete rate-distortion objective.

  • Loss design: The bitstream contains four latent-code components, so the total rate-distortion loss sums their bitrate costs with distortion.The components are current-frame and motion-vector latents plus their corresponding hyper priors.
  • Motivation: Directly applying Lall initially can be unstable because the model may favor generating context without MEMC.The authors therefore use progressive training after initializing motion estimation with a pretrained optical-flow network.
  • Progressive schedule: Step 1 warms up motion estimation, motion-vector encoding, and decoding using the motion-estimation loss.The warped frame is produced by applying the decoded motion vector to the previous reconstructed frame.
  • Progressive schedule: Step 2 freezes motion generation and trains the remaining modules solely for reconstruction quality.This encourages context that better reconstructs high-frequency contents.
  • Progressive schedule: Step 3 adds bitrate cost while keeping the motion-generation part frozen and trains contextual coding.It trains the framework except for the frozen motion-generation module.
  • Progressive schedule: Step 4 reopens motion generation and performs end-to-end training with the complete loss Lall.The progressive strategy is reported to stabilize model training.

C Details of experimental settings

Experiments train on Vimeo-90k and test on three commonly used video-compression datasets. Comparisons use specified intra-frame models, FFMPEG settings, and GOP configurations.

  • Datasets: Training uses the Vimeo-90k septuplet dataset, while testing uses MCL-JCV, UVG, and HEVC standard test videos.The datasets are commonly used for video-compression research.
  • Intra frame coding: Intra coding uses cheng2020-anchor for MSE and hyperprior for MS-SSIM from CompressAI.These are selected as the best corresponding models provided by CompressAI.
  • Model settings: DCVC trains four MSE models with λ values 256, 512, 1024, and 2048, and four MS-SSIM models with λ values 8, 16, 32, and 64.CompressAI quality indices 3 through 6 provide the corresponding intra-frame models.
  • FFMPEG settings: The x264 and x265 tests use the veryslow preset and constant quantization parameters rather than veryfast and constant rate factor settings.These choices target higher compression ratio and avoid rate-control effects.
  • FFMPEG settings: The command-line settings encode width, height, frame rate, frame count, quantization parameter, and GOP size.These variables are represented by W, H, FR, N, QP, and GOP.
  • GOP settings: Default GOP sizes are 10 for HEVC videos and 12 for non-HEVC videos, with 100 and 120 tested frames respectively.Each setting covers 10 GOPs.

D Test on larger GOP size

The larger-GOP experiment tests whether DCVC retains or improves its bitrate advantage when temporal dependencies span more frames. The reported examples associate feature-domain context with better reconstruction and reduced error propagation.

  • Larger GOP settings: 3x default GOP settings use sizes 30 for HEVC videos and 36 for non-HEVC videos.The tested frame counts are 30 for HEVC and 36 for non-HEVC videos.
  • Bitrate saving: 24.0% bitrate saving replaces 16.2% for the 1080P HEVC Class B dataset.The paper attributes the improvement to feature-domain context helping reconstruct high-frequency contents and alleviate error propagation.
  • Error propagation: In BasketballDrive, DVCPro's PSNR decreases from 34.1 dB to 26.6 dB in the first GOP, whereas DCVC decreases only to 31.8 dB.The example compares PSNR and bit cost with GOP size 30 across 60 tested frames.

E Ablation study

The ablation studies examine context dimensionality, MEMC, and the learning of context, showing that 64-dimensional context and MEMC substantially support performance while better context learning remains open.

  • Channel dimension of context: 64-Dim context further boosts performance over 3-Dim and 16-Dim models for most datasets, whereas 256-Dim provides little improvement and loses performance on HEVC Class E.The authors attribute the 256-Dim issue to unstable training without extra supervision and adopt the 64-Dim model.
  • Motion estimation and motion compensation (MEMC): Removing MEMC causes a large performance drop for both DVCPro and DCVC, confirming that MEMC helps both residue coding and conditional coding.Without MEMC, the models directly use the previous decoded frame as prediction or condition.
  • Motion estimation and motion compensation (MEMC): 22.1% bitrate saving over DVCPro (w/o MEMC) is achieved by DCVC (w/o MEMC), exceeding the 12.7% improvement of DCVC in pixel-domain context over DVCPro.These comparisons show a larger conditional-coding advantage when MEMC is disabled.
  • Future directions: The paper identifies better context learning as future work, including transformer-based exploration of global correlations and larger receptive fields.The current approach uses MEMC to learn the context.

F Visual comparison

Visual comparisons show that DCVC reconstructs higher-quality frames than DVCPro without increasing bitrate, particularly for color, texture, and stripe details.

  • Visual comparison: DCVC achieves much higher reconstruction quality than DVCPro without increasing bitrate cost.Examples include reduced color distortion and unexpected textures, plus clearer stripe texture in basketball clothing.
Loading 2109.15047v2…