PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Context Mining for Learned Video Compression

Xihua Sheng, Jiahao Li, Bin Li, Li Li, Dong Liu, Yan Lu

arXiv:2111.13850v2cs.CVcs.LGeess.IV

TL;DR

Learned video compression needs richer temporal context modeling, but auto-regressive entropy models hinder parallel decoding. The paper mines multi-scale contexts from propagated features, re-fills them throughout the compression scheme, and reports bit-rate savings against official H.265 and H.266 reference software.

  • Problem

    Existing learned video compression can lose texture and motion information when deriving a single-scale context from previously decoded frames.

  • Method

    The scheme stores propagated features, learns multi-scale temporal contexts from them, and re-fills those contexts into several compression modules without an auto-regressive entropy model.

  • Results

    14.4% bit rate saving over H.265/HEVC HM in PSNR and 21.1% over H.266/VVC VTM in MS-SSIM are reported for intra period 32.

  • Takeaways & Limitations

    The scheme achieves higher compression ratios than existing learned video codecs while pursuing a more parallelization-friendly decoder.

Abstract

from arXiv · show

We address end-to-end learned video compression with a special focus on better learning and utilizing temporal contexts. For temporal context mining, we propose to store not only the previously reconstructed frames, but also the propagated features into the generalized decoded picture buffer. From the stored propagated features, we propose to learn multi-scale temporal contexts, and re-fill the learned temporal contexts into the modules of our compression scheme, including the contextual encoder-decoder, the frame generator, and the temporal context encoder. Our scheme discards the parallelization-unfriendly auto-regressive entropy model to pursue a more practical decoding time. We compare our scheme with x264 and x265 (representing industrial software for H.264 and H.265, respectively) as well as the official reference software for H.264, H.265, and H.266 (JM, HM, and VTM, respectively). When intra period is 32 and oriented to PSNR, our scheme outperforms H.265--HM by 14.4% bit rate saving; when oriented to MS-SSIM, our scheme outperforms H.266--VTM by 21.1% bit rate saving.

I. INTRODUCTION

Video compression is important because video data dominates internet traffic and efficient compression reduces transmission and storage costs. The paper targets limitations in learned video compression by mining richer temporal contexts while avoiding an auto-regressive entropy model.

  • Video data contributes to most internet traffic, creating demand for efficient compression to reduce transmission and storage costs.
  • Existing learned video compression methods include residual coding, volume coding, entropy coding, and conditional coding schemes.
  • DCVC uses a single-scale temporal context from previously decoded frames, which lose texture and motion information because they contain only three channels.
  • The proposed temporal context mining module learns richer, multi-scale contexts from propagated features rather than previously reconstructed frames.
  • The learned contexts are re-filled into the contextual encoder-decoder, frame generator, and temporal context encoder to compress and reconstruct current frames.
  • 14.4% bit rate saving over H.265/HEVC HM in PSNR and 21.1% over H.266/VVC VTM in MS-SSIM are reported without auto-regressive entropy modeling.

II. RELATED WORK

Traditional video codecs use established hybrid coding frameworks, while learned video compression replaces or augments these components with neural networks. The paper positions its temporal-context approach against both learned methods and stronger official reference implementations of traditional codecs.

  • A. Traditional Video Compression: Traditional video standards include H.264/AVC, H.265/HEVC, and H.266/VVC, which use prediction, transform, quantization, entropy coding, and loop filtering.
  • B. Learned Video Compression: Learned video compression replaces components of traditional motion-compensated prediction and residual coding with jointly optimized convolutional neural networks.
  • B. Learned Video Compression: The proposed scheme estimates and compresses motion vectors between the current frame and previously decoded frame, while mining multi-scale contexts from propagated features.
  • B. Learned Video Compression: Decoded frames and propagated features are retained to support compression of subsequent frames, with learned contexts re-filled into multiple compression modules.
  • A. Traditional Video Compression: The evaluation uses official reference software JM, HM, and VTM to represent higher compression potential rather than relying only on faster x264 and x265 implementations.

III. METHODOLOGY

The scheme mines multi-scale temporal contexts from propagated features and re-fills them into compression components to better learn and use temporal information.

  • III. METHODOLOGY: The proposed scheme aims to improve learned video compression by mining and re-filling temporal contexts.The approach is based on temporal context mining and temporal context re-filling.
  • III. METHODOLOGY: Motion vectors are estimated from the current and previously decoded frames using a neural network-based motion estimation module.The motion estimation module is based on pre-trained Spynet.
  • III. METHODOLOGY: The estimated motion vectors are compressed and reconstructed lossily using an auto-encoder with a hyper prior structure.The reconstructed motion vector is denoted as ˆvt.
  • III. METHODOLOGY: The TCM module generates multi-scale temporal contexts from the propagated feature Ft−1 instead of the previously decoded frame ˆxt−1.This replaces single-scale context generation from reconstructed frames with feature-based temporal context mining.
  • III. METHODOLOGY: The learned temporal contexts are re-filled into the contextual encoder-decoder, frame generator, and temporal context encoder.The contextual encoder-decoder and frame generator compress and reconstruct the current frame, while the feature is propagated to the next frame.
  • III. METHODOLOGY: Largest-scale contexts emphasize texture and color channels, whereas smallest-scale contexts mainly focus on regions with large motion.The visualization illustrates distinct information captured at different context scales.

5) Temporal Context Encoder:

The temporal context mining pipeline extracts, warps, merges, and refines multi-scale features to represent non-uniform motion and texture, while avoiding an auto-regressive entropy model for parallel decoding.

  • 5) Temporal Context Encoder:: The temporal context encoder uses multi-scale temporal contexts to generate a lower-dimensional temporal prior for latent representations.The temporal prior is used with the hyper prior in Laplacian parameter estimation.
  • 5) Temporal Context Encoder:: The scheme omits the auto-regressive entropy model to keep decoding processes parallelization-friendly.The factorized entropy model is used for the hyper prior, with Laplace distributions modeling latent representations.
  • 5) Temporal Context Encoder:: The TCM module learns contexts from propagated features because previously decoded frames contain only three channels and lose texture and motion information.The propagated feature is stored before the last convolutional layer used to obtain the reconstructed frame.
  • 5) Temporal Context Encoder:: A hierarchical structure generates multi-scale contexts because single-scale context may not describe spatial-temporal non-uniform motion and texture well.Largest-scale channels focus on texture or color, while smallest-scale channels mainly focus on regions with large motion.
  • 5) Temporal Context Encoder:: The TCM module extracts three levels of multi-scale features from the propagated feature using convolutional layers and residual blocks.The extraction module has L levels, with three levels used in the paper.
  • 5) Temporal Context Encoder:: Decoded motion vectors are downsampled and used to warp multi-scale propagated features at corresponding scales.Each downsampled motion vector is divided by 2 before warping.
  • 5) Temporal Context Encoder:: Upsampled features are concatenated with same-scale warped features and refined through a convolutional layer and residual block.The refinement module learns a residue that is added to the propagated feature representation.
  • 5) Temporal Context Encoder:: The multi-scale contexts are re-filled into the contextual encoder-decoder, frame generator, and temporal context encoder.This architecture-wide re-filling is illustrated in Fig. 5.

C. Temporal Context Re-filling

Temporal context re-filling distributes the learned multi-scale contexts across the compression pipeline, where they support temporal prediction and temporal entropy modeling.

  • C. Temporal Context Re-filling: The learned multi-scale temporal contexts are re-filled into the contextual encoder-decoder, frame generator, and temporal context encoder.The paper states that this re-filling substantially improves the scheme’s compression ratio.

1) Contextual Encoder-Decoder and Frame Generator:

The contextual encoder-decoder and frame generator use multi-scale contexts throughout current-frame compression and reconstruction, while the temporal context encoder forms a temporal prior for entropy modeling.

  • 1) Contextual Encoder-Decoder and Frame Generator:: The largest-scale context is concatenated with the current frame before contextual encoding.Additional context scales are concatenated during mapping from the frame to its latent representation.
  • 1) Contextual Encoder-Decoder and Frame Generator:: The contextual decoder uses multi-scale contexts to map the quantized latent representation to a feature, which the frame generator converts into the reconstructed frame.The decoder uses ¯C1t, while the frame generator concatenates ¯C0t with the decoded feature.
  • 1) Contextual Encoder-Decoder and Frame Generator:: The feature before the frame generator’s last convolutional layer is propagated to help compress the next frame.A bottleneck residual block reduces the complexity of the middle layer after concatenation increases channel count.
  • 1) Contextual Encoder-Decoder and Frame Generator:: The temporal context encoder concatenates multi-scale contexts to produce a lower-dimensional temporal prior instead of using a single-scale context.The temporal prior is fused with the hyper prior to estimate latent-representation means and variances.

D. Loss Function

The scheme jointly optimizes a rate-distortion loss combining distortion with motion and contextual latent bit rates, using MSE or 1−MS-SSIM as distortion.

  • The loss function combines distortion and rate terms for motion-vector and contextual latent representations.The motion and contextual rates include their associated hyper priors.
  • Distortion is measured between the input frame and reconstructed frame using mean-square error or 1−MS-SSIM.
  • The model is trained step-by-step for stability, with sequential training frames used during the last five epochs to alleviate error propagation.
  • Training uses Vimeo-90k crops, while evaluation covers UVG, MCL-JCV, and HEVC datasets with varied video content.MCL-JCV-26 excludes four animation sequences considered different from the natural videos in the training dataset.

2) Implementation Details:

The implementation and comparison use multiple rate-controlled models, standard distortion metrics, and low-delay configurations for learned and traditional codecs. At intra period 32, the proposed scheme achieves substantial BD-rate savings against HM and VTM* and improves subjective detail retention.

  • Implementation Details: Four models are trained with λ values 256, 512, 1024, and 2048, while MS-SSIM models use λ values 8, 16, 32, and 64.Training uses AdamW, batch size 4, two NVIDIA V100 GPUs, and takes 2.5 days.
  • Implementation Details: bpp measures bit cost per pixel, while PSNR and MS-SSIM measure distortion between decoded and original frames.
  • Comparison Setting: The comparison uses low-delay x264, x265, JM-19.0, HM-16.20, and VTM-13.2, with codec-specific CPU or GPU settings and matched I-frame compression for learned codecs.
  • Results: 14.4% bit rate saving over HM is achieved in PSNR, while 21.1% bit rate saving over VTM is achieved in MS-SSIM at intra period 32.On HEVC Class RGB, the scheme also outperforms VTM* by 14.2% in PSNR, while retaining a 25.4% PSNR gap to the best VTM configuration.

3) Model Complexity and Encoding/Decoding Time:

The proposed scheme has moderate model size and computational cost, while removing the auto-regressive entropy model substantially reduces decoding time. Its practical deployment remains constrained by complexity and hardware differences in codec comparisons.

  • Encoding/Decoding Time: Encoding and decoding measurements include model inference, entropy processing, CPU-GPU transfer, and full frame-to-bitstream or bitstream-to-frame processing.
  • Practical Boundary: The scheme's encoding and decoding comparisons with HM and VTM are affected by its use of an expensive GPU versus their general-purpose CPU.
  • Practical Boundary: The paper concludes that complexity, especially decoding complexity, must be reduced before practical use.

C. Ablation Study

Ablations examine feature propagation, temporal-context mining and re-filling, component placement, hierarchy depth, context count, feature dimension, and added network layers. Results support richer propagated-feature contexts and show diminishing returns beyond three levels or contexts.

  • Component Effectiveness: The ablation baseline combines feature propagation, temporal context mining, and temporal context re-filling.The study evaluates the contribution of learning and utilizing temporal contexts.
  • Temporal Context Source: Propagated features may contain more temporal information than reconstructed frames.
  • Component Placement: Temporal contexts are removed separately from the encoder, decoder, frame generator, and entropy model to assess their component-specific influence.
  • TCM Levels: Bit rate saving improves as TCM levels increase but begins to saturate at three levels.Single-scale variants use only the level-0 temporal context.
  • Re-filled Contexts: Bit rate saving improves with more re-filled contexts, but performance improvement is not obvious beyond three contexts.The scheme therefore employs 3 temporal contexts.
  • Network Capacity Control: Simply adding residual blocks to make MACs comparable cannot greatly improve the compression ratio.The comparison modifies feature extraction, context refinement, and frame generator modules in 1L1C models.

4) Influence of the Dimension of Propagated Features:

The experiments examine feature-dimension choices and the scheme’s behavior across an extended bit-rate range. Lower propagated-feature dimensions cause only slight losses, while performance remains strong over wider bit rates, despite a stated complexity boundary for practice.

  • Influence of the Dimension of Propagated Features: 64 features match RLVC’s DPB size, while 48, 15, and 9 match the DPB sizes of FVC, MLVC, and four-reference-frame codecs.These dimensions were selected for hardware- and codec-size comparisons.
  • Influence of the Dimension of Propagated Features: Lower propagated-feature dimensions cause only a slight loss, allowing users to adjust dimension according to hardware capability.
  • Wider Bit Rate Range: Additional models extend the trained bit-rate range for PSNR and MS-SSIM evaluation, while competing released models remain limited to four bit-rate points.The proposed scheme uses λ values of 128 and 4096 for PSNR, and 4 and 128 for MS-SSIM.
  • Wider Bit Rate Range: The proposed scheme continues to work well over a wider range of bit rates on HEVC Class RGB.Figure 12 reports rate-distortion performance at lower and higher bit rates.
  • Discussion: Despite strong MS-SSIM results against VTM, the scheme remains far behind VTM in PSNR and requires substantially lower decoding complexity for practical use.
Loading 2111.13850v2…