PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model

Ren Yang, Fabian Mentzer, Luc Van Gool, Radu Timofte

arXiv:2006.13560v4eess.IVcs.CV

TL;DR

Existing video-compression methods use few reference frames and independently model latent PMFs, limiting their use of temporal correlation. RLVC addresses this with recurrent auto-encoders and a recurrent probability model, and the reported experiments show stronger compression performance than prior learned methods and specified x265 settings.

  • Problem

    Existing video-compression approaches use limited reference frames and independently estimate latent PMFs, restricting exploitation of temporal correlation.

  • Method

    RLVC combines recurrent auto-encoders in the encoder and decoder with a recurrent probability model conditioned on previous latent representations.

  • Results

    The experiments report that RLVC outperforms previous learned approaches and x265's default LDP setting on PSNR and MS-SSIM, and x265's slowest setting on MS-SSIM.

  • Takeaways & Limitations

    Recurrent auto-encoding expands the reference-frame range, while recurrent conditional probability modeling improves coding efficiency within the proposed framework.

Abstract

from arXiv · show

The past few years have witnessed increasing interests in applying deep learning to video compression. However, the existing approaches compress a video frame with only a few number of reference frames, which limits their ability to fully exploit the temporal correlation among video frames. To overcome this shortcoming, this paper proposes a Recurrent Learned Video Compression (RLVC) approach with the Recurrent Auto-Encoder (RAE) and Recurrent Probability Model (RPM). Specifically, the RAE employs recurrent cells in both the encoder and decoder. As such, the temporal information in a large range of frames can be used for generating latent representations and reconstructing compressed outputs. Furthermore, the proposed RPM network recurrently estimates the Probability Mass Function (PMF) of the latent representation, conditioned on the distribution of previous latent representations. Due to the correlation among consecutive frames, the conditional cross entropy can be lower than the independent cross entropy, thus reducing the bit-rate. The experiments show that our approach achieves the state-of-the-art learned video compression performance in terms of both PSNR and MS-SSIM. Moreover, our approach outperforms the default Low-Delay P (LDP) setting of x265 on PSNR, and also has better performance on MS-SSIM than the SSIM-tuned x265 and the slowest setting of x265. The codes are available at https://github.com/RenYang-home/RLVC.git.

I. INTRODUCTION

Existing video-compression methods use non-recurrent structures and limited references, restricting temporal-correlation modeling. RLVC addresses this with recurrent auto-encoding and probability modeling across frames.

  • I. INTRODUCTION: Existing handcrafted and learned codecs use non-recurrent structures, limiting reference frames and temporal-correlation exploitation.Previous learned methods also estimate latent-representation PMFs independently for each frame.
  • I. INTRODUCTION: RLVC uses recurrent networks to represent inputs, reconstruct outputs, and model PMFs for entropy coding.Its recurrent auto-encoder places recurrent cells in both the encoder and decoder.
  • I. INTRODUCTION: All previous frames can serve as references for compressing the current frame, expanding temporal information beyond limited-reference approaches.The encoder and decoder recurrently process frame sequences and their hidden states.
  • I. INTRODUCTION: The recurrent probability model estimates q(y_t | y<t) instead of independent q(y_t), which is expected to lower cross entropy and bit-rate.The conditional PMF is modeled from previous latent representations because consecutive frames are temporally correlated.
  • I. INTRODUCTION: The paper proposes recurrent auto-encoding and recurrent probability modeling to exploit temporal correlation across a large range of frames.The contributions identify expanded references and temporally conditional PMF estimation as the central innovations.

II. RELATED WORKS

Prior learned video codecs are largely non-recurrent or use limited temporal context. RLVC processes one frame with previous hidden states, enabling larger temporal context with finite memory and recurrent entropy modeling.

  • II. RELATED WORKS: Earlier learned video-compression methods adopted end-to-end DNN frameworks, but existing codecs generally lacked recurrent compression models.Prior approaches used at most limited reference frames or non-recurrent structures.
  • II. RELATED WORKS: RLVC takes one frame and previous hidden states as inputs, avoiding the GPU-memory-limited temporal groups required by 3D auto-encoders.This design allows the network to move recurrently through a larger temporal range with finite memory.
  • II. RELATED WORKS: RLVC uses motion compensation and recurrent auto-encoders to compress motion and residual information.An optical-flow network estimates motion between the current and previously compressed frames, while separate RAEs process motion and residuals.
  • II. RELATED WORKS: The recurrent probability model estimates temporally conditional PMFs for latent representations during arithmetic coding.Temporal conditioning is expected to reduce cross entropy relative to independent PMF estimation and thereby reduce bit-rate.

B. Recurrent Auto-Encoder (RAE)

The RAE inserts recurrent cells into both encoder and decoder pathways so previous-frame information informs current latent representations and reconstructions. This expands the usable reference range beyond non-recurrent auto-encoders.

  • B. Recurrent Auto-Encoder (RAE): The two RAEs share the same architecture and are used for motion and residual compression.The section denotes their inputs with a common x_t for simplicity.
  • B. Recurrent Auto-Encoder (RAE): Non-recurrent auto-encoders process only the current input and latent representation, failing to exploit temporal correlation across consecutive frames.Their encoder and decoder do not receive recurrent information from previous frames.
  • B. Recurrent Auto-Encoder (RAE): The proposed RAE inserts recurrent cells into both encoder and decoder networks.A ConvLSTM in the encoder carries information from previous frames through hidden states.
  • B. Recurrent Auto-Encoder (RAE): The RAE generates latent representations and reconstructions recurrently, allowing previous frames to serve as references for the current frame.This expands the usable reference range beyond the limited references of non-recurrent approaches.

C. Recurrent Probability Model (RPM)

The RPM recurrently models each latent representation’s conditional PMF from previous latent representations, targeting lower cross entropy and bit-rate than independent modeling. It uses location-specific discretized logistic PMFs whose parameters are generated recurrently and then applied in adaptive arithmetic coding.

  • Conditional entropy modeling: RPM models the current latent representation conditionally on all previous latent representations rather than using an independent PMF.This targets the lower conditional cross entropy expected from temporal correlation.
  • Conditional entropy modeling: The conditional PMF factorizes across the N 3D locations of the latent representation, with a separate distribution estimated for each element.Each element y_it is associated with a location-specific conditional PMF.
  • Discretized logistic PMF: Each quantized latent element is modeled with a discretized logistic distribution integrated over its quantization interval.The interval spans y_it − 0.5 to y_it + 0.5.
  • RPM architecture and coding: The RPM recurrently generates location-specific distribution parameters from previous latent representations and uses the resulting PMFs for adaptive arithmetic coding.The recurrent network produces µ_t and s_t before coding the latent representation into a bit stream.

D. Training

The paper trains separate RLVC models for MS-SSIM and PSNR using a uni-directional LDP structure and recurrent components. Training proceeds progressively before end-to-end optimization on sequential frames, while experiments compare rate-distortion performance with learned methods and x265 settings.

  • Training setup: RLVC uses separate models optimized for MS-SSIM and PSNR, with distortion defined as 1 − MS-SSIM or MSE respectively.The approach uses a uni-directional Low-Delay P structure.
  • Training setup: The first frame is compressed as an I-frame, while later P-frames use the proposed RPM for motion and residual latent representations.The first P-frame lacks previous latent representations and is handled separately; subsequent P-frames use RPM-based coding.
  • Entropy coding: The RPM network estimates conditional PMFs for later frames, whose expected actual bit-rate equals the conditional cross entropy under the training-distribution assumption.Two RPM networks are applied to motion and residual latent representations.
  • Training procedure: The recurrent model is trained progressively on the first P-frame and then end-to-end on sequential training frames.The procedure first trains motion estimation, then adds recurrent auto-encoding and motion compensation before final joint training.
  • Optimization: The training procedure uses quantization relaxation, Adam optimization, model-specific λ values, and learning-rate reduction after convergence.The learning rate starts at 10^-4 and is reduced to 10^-6 during final training.

A. Settings

The experiments evaluate RLVC on JCT-VC and UVG datasets across multiple resolutions and compare it with learned video compression methods and several x265 configurations. The x265 comparisons vary presets, tuning modes, and LDP structure while controlling quality through CRF values.

  • Baselines: The learned-compression comparison includes HLVC, Liu et al., Habibian et al., DVC, Cheng et al., and Wu et al.These methods represent recent learned video compression approaches used for comparison.
  • Baselines: The x265 comparison includes LDP very fast, LDP default, default, SSIM default, slowest, and SSIM slowest settings.These settings differ in preset, LDP structure, and SSIM tuning.
  • Encoding settings: CRF controls compression quality, with lower CRF corresponding to higher quality; the experiments use dataset-specific CRF ranges.JCT-VC uses CRF values 15, 19, 23, and 27, while UVG uses 7, 11, 15, 19, and 23.

B. Performance

RLVC achieves leading learned video-compression performance across MS-SSIM and PSNR comparisons, while outperforming several x265 settings under different quality metrics. Its higher encoding complexity is a documented trade-off.

  • Comparison with learned approaches: RLVC outperforms the compared learned approaches on MS-SSIM and PSNR rate-distortion performance across UVG and JCT-VC.The MS-SSIM model is competitive with Agustsson et al. and outperforms the other learned approaches; the PSNR model outperforms the compared PSNR-optimized models.
  • Comparison with learned approaches: 18 of 20 test videos achieve the best learned-approach results, while the PSNR model performs best among learned approaches on every test video.The reported results cover UVG and JCT-VC Class B.
  • Comparison with learned approaches: RLVC exceeds HLVC despite using only uni-directional IPPP without post-processing, whereas HLVC uses bi-directional prediction, hierarchical structure, and post-processing.The comparison isolates the reported advantage of the recurrent framework under a less elaborate coding structure.
  • Comparison with x265: 43.78% average bit-rate reduction is achieved against x265 (LDP very fast) using MS-SSIM BDBR.RLVC also outperforms x265 (LDP default), x265 (default), x265 (SSIM default), and x265 (slowest) on the reported average comparison.
  • Computational complexity: Higher encoding complexity is the cost of the recurrent autoencoders and probability model, although decoding is faster than HLVC and some other approaches.RLVC exceeds 30 fps on 240p videos and excludes entropy-coding time from the reported measurements.

C. Ablation studies

Ablations show that both recurrent components improve compression: RAE improves rate-distortion performance, while RPM further reduces bit-rate through temporal conditional modeling. Sequential temporal state also benefits later P-frames.

  • Ablation setup: The baseline removes recurrent cells from the autoencoders and replaces RPM with a factorized spatial entropy model.This baseline is used to assess each recurrent component separately.
  • The proposed RAE: Adding recurrent cells to the autoencoders improves the baseline rate-distortion curve.The dual recurrent cells encode residual information between current and previous inputs, while the decoder uses encoded residuals and previous outputs.
  • The proposed RPM: Adding RPM to BL+RAE significantly reduces bit-rate at the same compression quality.The result supports lower temporally conditional cross entropy than independent cross entropy.
  • The proposed RPM: RPM also outperforms the hyperprior spatial entropy model at the same compression quality.The comparison includes the hyperprior’s side-information overhead.
  • The proposed RPM: The temporal prior uses all previous latent representations without the overhead bit-rate required to compress hyperprior side information.This distinguishes RPM from the compared spatial prior and hyperprior formulation.
  • Sequential prior vs. one-frame prior: Resetting recurrent states before the sixth P-frame decreases its rate-distortion performance compared with the sequential prior.The sequential prior transfers hidden states through P-frames, allowing earlier temporal information to contribute.
  • Change of PSNR and bpp along consecutive P-frames: Both PSNR and bit-rate decrease along consecutive P-frames as their distance from the I-frame increases.The paper attributes the bit-rate trend to richer temporal priors available to RAE and RPM, while quality can decline through predictive-frame dependencies.

D. Combining RPM with spatial probability models

The RPM can be combined with spatial probability models, but the hyperprior combination yields only a slight bit-rate improvement. Among evaluated GOP structures, bi-IPPP with GOP = 13 performs best, while GOP = 20 remains competitive without obvious degradation.

  • Spatial probability models: The RPM is flexible enough to combine with spatial probability models such as the hyperprior model.The paper illustrates a combined RPM and hyperprior model.
  • Spatial probability models: Bit-rate reduction is < 1% when RPM is combined with the hyperprior spatial probability model.The authors attribute the slight gain to strong temporal correlation among video frames.
  • GOP structures: GOP = 13 with bi-IPPP using N = M = 6 achieves the best performance among the evaluated GOP structures.This structure is used in the paper’s subsequent experiments.
  • GOP structures: GOP = 20 with bi-IPPP remains competitive with GOP = 13 and causes only slight performance degradation.The conclusion describes RLVC as compatible with various GOP sizes and especially adjustable to GOP = 20.
  • GOP structures: Bi-directional prediction achieves better performance than uni-directional prediction for RLVC and other traditional and learned approaches.The paper notes that bi-directional prediction is also used by previous learned video compression methods.

B. Error propagation

RLVC can compress beyond its training sequence length, but performance degradation from error propagation still appears at later P-frames. The paper also positions recurrent compression as a possible direction for improving deep-network components in traditional codecs.

  • Error propagation: PSNR drops around 0.5 dB from frame 7 to frame 13 after the sequence exceeds the models’ training length.The analysis uses 12 consecutive P-frames with GOP = 13 in the uni-IPPP setting, while training uses 6 P-frames plus an I-frame.
  • Error propagation: Error propagation exists in RLVC, although frame 13 can outperform frame 11 at low bit-rates and remain comparable at high bit-rates.A similar pattern occurs between frames 3 and 7, where the later frame performs better at low bit-rates.
  • Conclusion: The proposed recurrent autoencoders and RPM expand the range of reference frames beyond previous learned approaches and handcrafted standards.The conclusion links this expansion to exploiting temporal correlation in video frames.
  • Future directions: Recurrent networks could further improve traditional codecs that currently use non-recurrent deep networks, including HEVC-based methods.The paper presents this as a possible direction rather than an established result.
  • Limitations: The framework still relies on warping and motion compensation to reduce temporal redundancy.The paper identifies eliminating this dependency as future work.

A. Performance on conversational video

On conversational and high-resolution video datasets, RLVC generally outperforms learned baselines and several x265 settings, with performance depending on metric, bit-rate, and prediction configuration. Its strongest comparisons are in MS-SSIM, while PSNR results remain mixed against slower or bidirectional x265 settings.

  • JCT-VC Class E: RLVC outperforms DVC and Liu et al. on JCT-VC Class E in both MS-SSIM and PSNR.Class E is a conversational video dataset used to test generalization.
  • JCT-VC Class E: On JCT-VC Class E, the RLVC MS-SSIM model outperforms x265 LDP default and LDP very fast at all bit-rates.It also outperforms all other listed x265 settings at medium and high bit-rates in MS-SSIM, and is comparable at low bit-rates.
  • JCT-VC Class E: On JCT-VC Class E, the RLVC PSNR model beats x265 LDP veryfast and x265 LDP default when bpp > 0.05.It does not outperform x265 default or slowest in PSNR on this dataset.
  • MCL-JCV: On MCL-JCV, RLVC’s MS-SSIM model outperforms x265 LDP default and LDP very fast, plus several other x265 settings at high bit-rates.The dataset contains 30 videos at 1920 × 1080 resolution.
  • MCL-JCV: On MCL-JCV, RLVC’s PSNR model outperforms the learned approach of Djelouah et al. despite using only IPPP prediction.Djelouah et al. uses bi-directional prediction, whereas the proposed approach uses IPPP mode.
Loading 2006.13560v4…