PaperlayerView exact revision on arXiv

Source-linked AI summary

FVC: A New Framework towards Deep Video Compression in Feature Space

Zhihao Hu, Guo Lu, Dong Xu

arXiv:2105.09600v2eess.IVcs.CV

TL;DR

Pixel-space operations in learning-based video compression may produce inaccurate motion estimation or less effective compensation. FVC performs the major coding operations in feature space, using compressed deformable-convolution offsets and multi-frame feature fusion, and achieves state-of-the-art performance on four benchmark datasets.

  • Problem

    Pixel-space operations in learning-based video compression may suffer from inaccurate motion estimation or less effective motion compensation.

  • Method

    FVC performs motion estimation, motion compression, motion compensation, and residual compression in feature space, with deformable compensation and multi-frame feature fusion.

  • Results

    FVC achieves state-of-the-art performance on the HEVC, UVG, VTL, and MCL-JCV benchmark datasets.

  • Takeaways & Limitations

    The framework demonstrates the effectiveness of performing video compression operations in feature space across four benchmark datasets.

  • Takeaways & Limitations

    The multi-frame feature fusion module assumes that the dimensions of the fused feature representations match.

Abstract

from arXiv · show

Learning based video compression attracts increasing attention in the past few years. The previous hybrid coding approaches rely on pixel space operations to reduce spatial and temporal redundancy, which may suffer from inaccurate motion estimation or less effective motion compensation. In this work, we propose a feature-space video coding network (FVC) by performing all major operations (i.e., motion estimation, motion compression, motion compensation and residual compression) in the feature space. Specifically, in the proposed deformable compensation module, we first apply motion estimation in the feature space to produce motion information (i.e., the offset maps), which will be compressed by using the auto-encoder style network. Then we perform motion compensation by using deformable convolution and generate the predicted feature. After that, we compress the residual feature between the feature from the current frame and the predicted feature from our deformable compensation module. For better frame reconstruction, the reference features from multiple previous reconstructed frames are also fused by using the non-local attention mechanism in the multi-frame feature fusion module. Comprehensive experimental results demonstrate that the proposed framework achieves the state-of-the-art performance on four benchmark datasets including HEVC, UVG, VTL and MCL-JCV.

1. Introduction

FVC addresses limitations of pixel-space video compression by performing motion estimation, compensation, and residual compression in feature space. It combines deformable compensation with multi-frame feature fusion and reports state-of-the-art performance on four benchmark datasets.

  • Traditional video codecs use hand-designed modules that cannot be optimized end-to-end from large-scale video data.
  • Pixel-level video compression can suffer from unreliable motion estimation and residual information, motivating feature-space operations.
  • FVC performs motion estimation, motion compensation, and residual compression in the feature space.
  • Deformable compensation compresses offset maps and uses deformable convolution to predict current-frame features more accurately.
  • Multi-frame feature fusion combines features from multiple previous frames through non-local attention for better current-frame reconstruction.
  • FVC achieves state-of-the-art performance on HEVC, UVG, VTL, and MCL-JCV.

2. Related Work

Image compression research has progressed from handcrafted transforms to learning-based methods that encode images into latent feature representations. Auto-encoder and recurrent approaches have achieved state-of-the-art image compression performance.

  • Traditional image compression methods reduce spatial redundancy using handcrafted techniques such as DCT.
  • Learning-based image compression methods use recurrent or auto-encoder architectures to represent and reconstruct images through feature space.
  • Learning-based image compression methods have achieved state-of-the-art performance for image compression.

2.2. Video Compression

Video compression standards use hybrid coding to reduce spatial and temporal redundancy, while learning-based codecs extend this structure with neural networks. Pixel-level flow and residual coding remain difficult because unreliable estimates can degrade compression performance.

  • Hybrid video compression uses motion compensation and residual coding to reduce spatial and temporal redundancy.
  • Learning-based video compression has become a research direction that follows the traditional hybrid coding framework.
  • Most existing approaches estimate and compress pixel-level optical flow and residual information.
  • Unreliable pixel-level flow maps or residual information can degrade the compression performance of learning-based video codecs.

2.3. Deformable Convolution

Deformable convolution uses learned offset maps to model complex motion in feature space. FVC incorporates this operation into video compression, compresses its offsets, and combines reconstructed features from multiple frames.

  • Deformable convolution with learned offset maps enhances neural-network modeling and has been applied to action recognition and video super-resolution.
  • FVC uses feature-space offset maps for motion estimation and deformable convolution for motion compensation.
  • The offset maps are compressed before their reconstructed versions guide motion compensation.
  • The framework fuses reconstructed features from multiple previous frames to produce the final reconstructed feature and frame.

3.1. Overview

FVC processes video compression primarily in feature space, extracting representations, estimating and compressing motion, compensating with deformable operations, compressing residuals, and fusing temporal context for reconstruction.

  • Feature Extraction: The framework encodes the current and previous reconstructed frames into feature representations before compression and reconstruction.Feature extraction uses a strided convolution followed by residual blocks.
  • Deformable Compensation: Deformable compensation estimates offset maps from consecutive-frame features, compresses them, and uses reconstructed offsets for feature prediction.The procedure comprises motion estimation, motion compression, and motion compensation.
  • Residual Compression: Residual compression encodes the residual between the current and predicted features, then adds the reconstructed residual to obtain an initial reconstructed feature.The residual feature is compressed with an auto-encoder style network.
  • Multi-frame Feature Fusion: Multi-frame feature fusion combines features from multiple previous reconstructed frames with the initial reconstructed feature to improve the final reconstruction.The supplied passage identifies fusion of three previous reconstructed frames and the initial reconstructed representation.
  • Frame Reconstruction and Entropy Coding: The feature decoder transforms the final reconstructed feature into the reconstructed frame, while quantized motion and residual features are entropy-coded into bitstreams.A bit-estimation network is used during training.

3.2. Deformable Compensation

The deformable compensation module performs motion estimation and compression in feature space, then uses reconstructed offsets with deformable convolution to predict the current feature.

  • Motivation: Feature-space deformable compensation addresses inaccurate frame prediction from pixel-level optical-flow and motion-compensation pipelines.The paper links inaccurate prediction to extra redundancy in subsequent residual compression.
  • Motion Estimation and Compression: A lightweight two-layer network estimates an offset map from current and reference features, and an auto-encoder compresses and reconstructs it.The offset map is transformed to a quantized latent representation before decoding.
  • Motion Compensation: Deformable convolution applies the reconstructed offset map to the reference feature to generate the predicted current feature.The offsets control sampling locations, whose feature values are fused into output values.
  • Deformable Convolution: The deformable convolution design uses channel groups, two offset directions, and 3 × 3 kernels for feature sampling.The figure caption specifies G = 8 channel groups and horizontal and vertical offset directions.

3.3. Multi-frame Feature Fusion

Multi-frame feature fusion improves reconstructed features by deformably compensating multiple previous frames and refining them with non-local and self-attention mechanisms.

  • Multi-frame Prediction: The module first generates predicted feature representations from multiple previous frames using a shared feature-space deformable compensation module.The supplied passage refers to multiple previous-frame representations.
  • Attention Refinement: Non-local attention refines predicted representations according to their similarity with the initial reconstructed feature.The same mechanism also refines the initial reconstructed feature through self-attention.
  • Architecture: The module reuses the deformable compensation architecture while providing a separate detailed non-local attention block.The supplied implementation description uses a previous-frame prediction as an example.
  • Attention Refinement: For each spatial location, the module compares feature vectors, produces an attention map with softmax, and reweights spatial positions in reference patches.This process yields refined feature vectors across the feature map.
  • Feature Fusion: The refined representations are concatenated and passed through a convolution layer, with the initial reconstructed feature added back to produce the final feature.The operation is repeated for each spatial location.

3.4. Residual Compression and Other Details

FVC compresses residual features with an auto-encoder-style network and uses learned entropy and quantization procedures for end-to-end video compression.

  • Residual Compression: Residual feature compression uses the same network architecture as offset-map compression to simplify the system.The residual feature map is compressed after deformable compensation.
  • Bitrate Estimation: A hyperprior entropy model estimates bitrate during training, while the time-consuming autoregressive model is omitted to reduce computational complexity.The entropy model is used within the whole learning-based video compression framework.
  • Quantization: Training approximates quantization by adding uniform noise, whereas inference uses direct rounding.This differentiable approximation supports end-to-end optimization during training.

3.5. Loss Function

FVC optimizes a rate-distortion objective combining offset-map and residual-feature coding rates with reconstruction distortion. The distortion term can use MSE or MS-SSIM, controlled by λ.

  • Rate-Distortion Objective: The objective is RD = R + λD = Ro + Rr + λd(Xt, ˆXt).Ro and Rr are the bits for offset maps and residual feature maps, respectively.
  • Rate Terms: The total rate is decomposed into offset-map rate Ro and residual-feature rate Rr.Both rates contribute directly to the optimized objective.
  • Distortion and Trade-off: The distortion d(Xt, ˆXt) measures differences between the input and reconstructed frames using MSE or MS-SSIM.λ controls the rate-distortion trade-off.

4. Experiments

Experiments evaluate FVC across four benchmark datasets against traditional and learning-based codecs, with ablations isolating feature-space operations and multi-frame fusion. FVC reports broad rate-distortion gains, while the ablations support contributions from deformable compensation, feature-level residual coding, NLA, and MFF.

  • Experimental Setup: FVC is evaluated on HEVC, UVG, MCL-JCV, and VTL using BDBR, PSNR, and MS-SSIM comparisons.The experiments include traditional H.265 and learning-based baselines, including DVC and enhanced DVC*.
  • Experimental Results: More than 18% overall bit-rate savings are reported across all benchmark datasets relative to H.265.On HEVC Class D, FVC saves 18.39% bit-rate, compared with 8.29% for LU ECCV20 and 1.77% for HU ECCV20.
  • Experimental Results: FVC outperforms DVC and DVC* by a large margin on all datasets, achieves better PSNR than H.265 at all bit-rates, and leads baselines in MS-SSIM.DVC* uses the same compression network for pixel-level optical flow and residual maps, providing a matched comparison for FVC's feature-space operations.
  • Ablation Study: Removing NLA lowers performance nearly 0.2dB, while removing MFF lowers performance 0.5dB at 0.3bpp on HEVC Class D.These results support multi-frame feature fusion with non-local attention.
  • Ablation Study: Feature-space residual compression improves FVC-basic by 0.3dB at 0.38bpp, while feature-space motion compensation improves it by 1.2dB at 0.4bpp.The comparisons use variants with pixel-space residual or motion operations.
  • Deformable Compensation: FVC improves predicted-frame quality by 1.75dB at 0.017bpp over DVC* on HEVC Class C.The comparison uses the same compression method and auto-encoder-style network for motion information compression.
  • Complexity: FVC requires 548ms coding time and about 26M parameters, with more than 24M parameters in offset-map and residual-feature compression networks.The timing is measured on 1920×1080 videos using a single 2080TI GPU.

5. Conclusion

The paper concludes that FVC performs deep video compression in feature space through deformable compensation, feature-level residual compression, and multi-frame feature fusion. It reports promising results on HEVC, UVG, VTL, and MCL-JCV.

  • Conclusion: FVC combines deformable compensation, feature-level residual compression, and multi-frame feature fusion in a feature-space video compression framework.The framework compresses offset maps and uses reconstructed offsets for deformable-convolution motion compensation.
  • Conclusion: FVC achieves promising results on the HEVC, UVG, VTL, and MCL-JCV datasets by performing its operations in feature space.The multi-frame fusion module uses deformable compensation and non-local attention to refine reconstructed features.
Loading 2105.09600v2…