PaperlayerView exact revision on arXiv

Source-linked AI summary

Feedback Recurrent Autoencoder for Video Compression

Adam Golinski, Reza Pourreza, Yang Yang, Guillaume Sautiere, Taco S Cohen

arXiv:2004.04342v1cs.LGcs.CVstat.ML

TL;DR

The paper addresses video compression that is not end-to-end optimized for rate-distortion on representative data. It proposes a low-latency learned codec using common components, and reports that it outperforms compared codecs on high-resolution UVG, particularly in a streaming-relevant rate range.

  • Problem

    Codecs in use are only minimally learning-based and are not end-to-end optimized for rate-distortion on large, representative video datasets, despite video comprising over 60% of internet traffic.

    Source: §1 · p. 1

  • Method

    The paper proposes a low-latency autoencoder video codec using standard building blocks, recurrent feedback, and explicit optical-flow estimation.

    Source: §1 · p. 2

  • Results

    At 0.09–0.13 bpp, the method outperforms compared learned and traditional codecs on high-resolution UVG MS-SSIM/rate curves.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Online frame-by-frame compression suits video conferencing and cloud gaming, while content-specific fine-tuning holds promise for reducing bandwidth further.

    Source: §1 · p. 2

    Low-bitrate models can show slight color shifts in flat regions, sometimes noticeable at I-frame-to-P-frame transitions and contributing to flicker.

    Source: Color shift · p. 14

Abstract

from arXiv · show

Recent advances in deep generative modeling have enabled efficient modeling of high dimensional data distributions and opened up a new horizon for solving data compression problems. Specifically, autoencoder based learned image or video compression solutions are emerging as strong competitors to traditional approaches. In this work, We propose a new network architecture, based on common and well studied components, for learned video compression operating in low latency mode. Our method yields state of the art MS-SSIM/rate performance on the high-resolution UVG dataset, among both learned video compression approaches and classical video compression methods (H.265 and H.264) in the rate range of interest for streaming applications. Additionally, we provide an analysis of existing approaches through the lens of their underlying probabilistic graphical models. Finally, we point out issues with temporal consistency and color shift observed in empirical evaluation, and suggest directions forward to alleviate those.

1 Introduction

The paper presents a low-latency learned video codec built from standard components, combining recurrent feedback with explicit optical-flow learning. It reports strong UVG rate-distortion results against learned and traditional codecs and analyzes existing approaches through their probabilistic models.

1 Introduction

  • Video makes up over 60% of internet traffic, while codecs in use are only minimally learning-based and are not end-to-end optimized for rate-distortion on representative video data.Lossy compression is presented as important for reducing bandwidth and storage and increasing internet scalability.Source: §1 · p. 1
  • Online compression sends frames as soon as they are recorded, supporting applications such as video conferencing and cloud gaming.The authors also say content-specific codec fine-tuning holds promise for further bandwidth reduction in both applications.Source: §1 · p. 2
  • The architecture combines residual-block encoder and decoder networks with decoder-to-encoder recurrent feedback and encoder-side motion estimation trained using an explicit optical-flow loss.The paper reports that the optical-flow loss improves decoder-side flow and consequently rate-distortion performance.Source: §1 · p. 2
  • On high-resolution UVG MS-SSIM/rate curves, the method outperforms all compared methods, especially in the streaming-relevant 0.09–0.13 bpp range.The paper also reports state-of-the-art rate-distortion performance against learned approaches and competitive or better performance than AVC and HEVC under equivalent settings.Source: §1 · p. 2
  • The paper analyzes differences and connections among existing learned video-compression methods by detailing their sequential latent-variable models.Source: §1 · p. 2

2 Methodology

The method formulates causal video compression as a rate–distortion problem and builds a low-latency I/P-frame codec from autoencoders, optical-flow compensation, and recurrent feedback. It extends the baseline with decoder-to-encoder state feedback and explicit optical-flow supervision.

2.1 Problem setup

  • The codec maps video frames to discrete latents and reconstructions, balancing rate and distortion with MS-SSIM in RGB space as the distortion metric.The rate is modeled as −log PZ(z); the rate–distortion objective is equivalent to a β-VAE evidence lower bound under the stated encoder conditions.Source: §2.1 · pp. 2–3

2.2 Overview of the proposed method

  • Online causal compression encodes each frame without future context, using I-frames and P-frames but no B-frames.The sequence is divided into groups of pictures beginning with an I-frame, with separate encoders, decoders, and priors for I- and P-frames.Source: §2.2 · p. 3

2.3 Network architecture

  • I-frame compression uses the encoder and decoder design of, while the method focuses on exploiting prior information for P-frame compression.The rate model also ignores constant practical entropy-coder overhead, which is negligible with many bits per stream.Source: §2.2 · p. 3, §2.3 · p. 4

Baseline architecture

  • The baseline encodes each P-frame from the current frame and previous reconstruction, then reconstructs it by warping that frame with decoded optical flow and adding a residual.The P-frame latents may retain temporal correlation with earlier flow and residual values, a redundancy that multiple reference frames alone do not exploit.Source: Baseline architecture · pp. 4–5

Feedback recurrent module

  • A feedback recurrent module summarizes decoder history and shares its recurrent state with the encoder to exploit higher-order temporal redundancy.The architecture uses generic convolutional recurrent modules such as Conv-GRU to model history relevant to the next frame.Source: Feedback recurrent module · p. 5

Latent quantization and prior model

  • The method uses nearest-neighbor codebook quantization and a time-independent prior factorized across P-frame latents, modeled with a gated PixelCNN.The I-frame prior is a separate network with the same architecture; the paper states that temporal factorization does not limit the model’s ability to capture an empirical data distribution.Source: Latent quantization and prior model · p. 6

Explicit optical flow estimation module

  • Because rate–distortion training alone led the decoder to rely almost solely on residuals, the method adds encoder-side MENet flow estimation and dedicated flow losses.The extra losses apply only during the first few thousand iterations; early on, they use the previous ground-truth frame instead of the inaccurate reconstructed frame, while the encoder input remains a warped previous reconstruction.Source: Explicit optical flow estimation module · pp. 6–7

3 Graphical Model Analysis

The analysis compares learned video-compression approaches by their sequential latent-variable and inference structures. It contrasts Markov and recurrent designs, showing how decoder recurrence can preserve a flexible video distribution even with a time-factorized prior.

Modeling framework

  • The analysis focuses on how sequential generative and inference models structure latent variables, and whether they can represent flexible data distributions and encode their dependencies.The paper frames rate–distortion training as connected to amortized variational inference.Source: §3 · pp. 8–9

Markov model

  • The Markov scheme used by Lu et al. and low-latency HEVC/AVC makes each decoded frame depend on the previous decoded frame and current latent, constraining the video distribution to adjacent-frame dependence.It also assumes successive flow and residual differences are independent across time.Source: §3 · p. 9

Recurrent prior

  • Liu et al. add a ConvLSTM prior across time, yielding a marginalized video distribution without conditional-independence assumptions between frames.This corresponds to the graphical model with dashed connections in Fig. 3(a).Source: §3 · p. 9

Decoder recurrence

  • A recurrent decoder state, used by Rippel et al. and this paper, summarizes earlier transmitted latents and permits a flexible video distribution alongside a time-factorized prior.Factorization permits parallel entropy decoding across time steps, and the paper states that recurrent connections from past frames to the encoder latents are unnecessary.Source: §3 · p. 9

Noncausal models

  • The graphical models of Han et al. and Habibian et al. break causality because encoding a group of pictures must wait until all its frames are available.The former uses a global latent for time-invariant features; the latter connects frames and latents across different time steps.Source: §3 · p. 9

4 Experiments

On UVG, the method outperforms the compared learned approaches and, above 0.09 bpp, traditional codecs; ablations highlight explicit flow estimation, while low-rate outputs show temporal and color issues.

4.1 Training setup

  • Training used cropped, downscaled Kinetics400 clips and jointly trained I- and P-frame networks from scratch across five β values, with GoP size 8.The Kinetics400 subset used high-quality videos; random 160×160 crops were used for training.Source: Datasets · p. 10, Training · p. 10

4.2 Comparison with other methods

4.3 Ablation study

  • Removing explicit optical-flow estimation costs at least 0.002 MS-SSIM below 0.16 bpp; removing feedback or recurrence causes only minor losses below 0.1 bpp.The authors suspect limited recurrent-module capacity and receptive field contribute to the small gain from the feedback recurrent module.Source: §4.3 · pp. 13–14

4.4 Empirical observations

  • Low-rate outputs show GoP-boundary flicker: P-frame quality and rate decline through the GoP, while MS-SSIM’s color-shift invariance contributes slight shifts.The authors note that the metric-related aspect is not reflected in Fig. 7.Source: Temporal consistency · p. 14, Color shift · p. 14

5 Conclusion

The proposed autoencoder video codec combines feedback recurrence with explicit optical-flow estimation, achieving strong rate-distortion performance in low-latency comparisons while leaving temporal consistency, color fidelity, and computational complexity for future work.

5 Conclusion

  • The codec combines a feedback recurrent module with explicit optical-flow estimation for learned lossy video compression.It achieves state-of-the-art performance among learned video compression methods and comparable or better rate-distortion than classic codecs in low-latency mode.Source: §5 · p. 14
  • Future work targets improved temporal consistency, color fidelity, and reduced computational complexity.The stated aim is to pave the way to a practical real-time learned video codec.Source: §5 · p. 14

A Model and training details

The appendix details the encoder, decoder, and optical-flow network architectures, a BatchNorm training adjustment, and the provenance of images used in figures.

A Model and training details

  • Fig. 9 details the encoder and decoder, while Fig. 10 details the optical-flow estimation network.The prior-model architecture follows Section B.2 of.Source: A Model and training details · p. 18
  • The P-frame network uses transposed convolution, with C1 = 32, C2 = 10, and C3 = 32; the I-frame network removes two decoder layers and uses C2 = 3.The I-frame encoder takes x0 as its only input; much of the architecture is inherited from Fig. 2 in [29] and Fig. 9 in [15].Source: Figure 9 · p. 18
  • BatchNorm is switched to evaluation mode at iteration 40k because feedback across unrolled time steps prevents consistent normalization during training.After the switch, BatchNorm parameters are no longer updated.Source: A Model and training details · pp. 18–19

B Images used in figures

  • Figure videos are produced by Netflix and carry a CC BY-NC-ND 4.0 license.The passage identifies the source video as Netflix Tango.Source: B Images used in figures · p. 19

C ffmpeg and HM experiments

The appendix describes how the H.264, H.265, and HM baselines were configured and how bitrate and MS-SSIM were measured.

C ffmpeg and HM experiments

  • H.264 and H.265 baselines use ffmpeg and HM tools, with native YUV-1080p-8bit UVG video as input.The appendix gives separate ffmpeg commands for low-latency and default settings.Source: C ffmpeg and HM experiments · pp. 19–20
  • The ffmpeg runs use 1080 × 1920 video at 120 fps, with RATE values of 10, 20, 37, 62, 87, and 112 Mbps mapping to 0.04, 0.08, 0.15, 0.25, 0.35, and 0.45 bits/pixel.The low-latency GOP size is 12; ffmpeg uses its default medium preset.Source: C ffmpeg and HM experiments · p. 20
  • HM runs use LowDelayP.cfg and QP values of 20, 22, 25, 30, 35, and 40 to vary encoded quality and bitrate.The sequences contain 600 or 300 frames.Source: C ffmpeg and HM experiments · p. 20
  • MS-SSIM is measured in RGB at video level, and bitrate is normalized by frame dimensions and frame rate.Bitrate comes from compressed-file size for ffmpeg and HM log files for HM.Source: C ffmpeg and HM experiments · p. 20

C.1 YUV to RGB conversion inconsistencies

The authors compare three ways of converting UVG YUV video to RGB and report inconsistent model performance across the resulting versions.

C.1 YUV to RGB conversion inconsistencies

  • The comparison covers direct ffmpeg conversion of UVG-1080p-8bit, OpenCV YUV2BGR I420 conversion, and ffmpeg conversion from UVG-4K-10bit to RGB 1080p.The reported results use the OpenCV conversion.Source: §C.1 · p. 20, Figure 11 · p. 21
  • Model performance differs across the converted UVG versions; the authors suggest chroma upsampling interpolation as a potential cause.In YUV420, U and V channels are subsampled and must be upsampled before conversion to RGB.Source: §C.1 · p. 20

D Qualitative examples

The qualitative examples compare reconstructed frames and distortion maps for the proposed method and H.265. They show different bit-allocation patterns and highlight both temporal degradation and color shifts.

D Qualitative examples

  • The figures overlay MS-SSIM and MSE maps on grayscale originals; the proposed method’s BPP maps additionally visualize latent entropy across frames.Columns show every other frame in a GoP 8 cycle from I-frame to I-frame.Source: D Qualitative examples · p. 21
  • The BPP maps show bits distributed more uniformly across I-frames, but concentrated around moving-object edges in P-frames.The maps are derived from latent entropy, averaged across channels and bilinearly upsampled to image resolution.Source: D Qualitative examples · p. 21
  • The method reconstructs edges sharply but leaves the pink suit dimmer, a color shift visible in MSE but not captured by MS-SSIM; H.265 blurs edges while preserving colors.The methods optimize or are evaluated with different distortion metrics: MS-SSIM for the proposed method and PSNR for traditional codecs.Source: D Qualitative examples · p. 21
  • MS-SSIM error maps worsen from I-frames to P-frames, while consecutive P-frames use significantly less rate than the first I-frame.The authors describe these qualitative patterns as confirming Fig. 7’s quantitative analysis.Source: D Qualitative examples · pp. 21–22

E Theoretical justification of the feedback recurrent module

The theoretical analysis argues that a causal encoder needs prior latent codes, but not earlier input frames. The appendix then explains how the paper’s and related methods’ probabilistic graphical models encode their dependencies.

E Theoretical justification of the feedback recurrent module

  • With causal encoding and decoding, the encoder can be parameterized as z_t = f_enc(x_t, z_<t): it needs the current frame and prior latents, not earlier inputs.The analysis uses average frame-wise mutual information under a rate constraint and treats it as a guide to data dependencies, not perceptual quality.Source: E Theoretical justification of the… · pp. 22, 25

F Graphical modeling considerations

  • The graphical-model appendix explains the reasoning behind the models presented in Fig. 3.Its discussion derives dependency structures for the proposed method and related approaches.Source: F Graphical modeling considerations · p. 25

Wu et al. [43]

  • For Wu et al. [43], each latent depends on the current frame and earlier latents through the deterministic reconstruction process; the generative prior has no edges between latents.The generative model links consecutive reconstructions, while the inference model also links earlier latents to the current latent.Source: Wu et al. [43] · pp. 25–26

Liu et al. [27]

  • Liu et al. [27] add past-to-current latent edges in the generative model through a time-autoregressive prior, while leaving the inference model unchanged.Source: Liu et al. [27] · p. 26

Rippel et al. [33] and ours

Han et al. [16]

  • The graphical models are derived from Han et al. equations: Eqs. (4) and (6) for the generative model, and Eq. (5) for inference.Source: Han et al. [16] · p. 26

Habiban et al. [15]

  • Habiban et al. [15] use an autoregressive temporal prior and 3D convolutions whose receptive field lets each latent depend on every input frame in the block.The resulting generative and inference models have fully connected links between the block’s latents and inputs.Source: Habiban et al. [15] · p. 26

F.1 Full flexibility of the marginal PX(x1:T )

  • For Liu et al. and the proposed/Rippel model, eliminating latents yields a marginal over frames with no conditional-independence assumptions between them.In the proposed/Rippel graph, eliminating z_1 connects all observed frames in the induced factor.Source: §F.1 · p. 27

G.1 MS-SSIM color shift artifact

The authors test whether equal MS-SSIM can conceal color shifts by optimizing image quality under a fixed MS-SSIM constraint. The experiment shows that substantial color changes can coexist with the same MS-SSIM score.

G.1 MS-SSIM color shift artifact

  • The experiment perturbs an image with additive Gaussian noise, then optimizes for lower PSNR while constraining MS-SSIM to the perturbed image’s score.The chosen equal-MS-SSIM level is 0.966, corresponding to the average quality of the second-lowest-rate model.Source: §G.1 · p. 27
  • Images can have very low PSNR, mainly from drastic uniform color shifts, while remaining at equal distance under MS-SSIM.The illustrated original, noisy perturbation, and two optimized images all have MS-SSIM 0.966.Source: §G.1 · p. 28, Figure 15 · p. 28
  • The resulting color-shift example helps explain color artifacts observed in the model outputs.Source: §G.1 · p. 28

G.2 MS-SSIM corner artifact

The standard MS-SSIM implementation can underweight corner pixels during training, allowing the network to allocate fewer bits there and produce blank corners. Replicate padding removes the artifact.

G.2 MS-SSIM corner artifact

  • Unpadded Gaussian convolutions underweight corner pixels in the training distortion, so the rate-distortion model can assign them fewer bits and leave decoded corners blank.The authors report that this issue does not affect MS-SSIM when used for evaluation.Source: §G.2 · p. 29
  • Replicate-padding the image before the Gaussian convolution eliminates the corner artifact.Source: §G.2 · p. 29, Figure 16 · p. 29
  • The paper retained the standard MS-SSIM implementation for both training and evaluation to remain consistent with other work.Source: §G.2 · p. 29
Loading 2004.04342v1…