Source-linked AI summary
Video Compression With Rate-Distortion Autoencoders
Amirhossein Habibian, Ties van Rozendaal, Jakub M. Tomczak, Taco S. Cohen
TL;DR
The paper addresses lossy video compression with a simple generative model that combines a discrete-latent 3D autoencoder and an autoregressive entropy-coding prior. It reports stronger performance than compared learned methods, while extending compression to semantic, adaptive, and multimodal settings. The authors conclude that learned codecs can provide benefits beyond compression performance, although practical rate-distortion coding relies on deterministic encoders because bits-back coding is difficult and limited to lossless compression.
Problem
Video compression is an important application for generative modeling, but deep learning-based video compression had received comparatively little attention.
Method
The paper jointly trains an autoencoder with a discrete latent space and an autoregressive prior using a rate-distortion objective, with a deterministic encoder for lossy compression.
Results
The method outperforms recent learned video compression methods and performs on par with H.265/HEVC, while multimodal experiments report a factor-of-4 advantage over HEVC/H.265 and AVC/H.264.
Takeaways & Limitations
Semantic weighting prioritizes important regions, domain fine-tuning supports effective compression of specific video distributions, and multimodal training exploits redundancy across modalities.
Takeaways & Limitations
Bits-back coding, which would make encoder randomness free in the coding cost, is difficult to implement and can only be used for lossless compression.
Abstract
from arXiv · showhide
In this paper we present a a deep generative model for lossy video compression. We employ a model that consists of a 3D autoencoder with a discrete latent space and an autoregressive prior used for entropy coding. Both autoencoder and prior are trained jointly to minimize a rate-distortion loss, which is closely related to the ELBO used in variational autoencoders. Despite its simplicity, we find that our method outperforms the state-of-the-art learned video compression networks based on motion compensation or interpolation. We systematically evaluate various design choices, such as the use of frame-based or spatio-temporal autoencoders, and the type of autoregressive prior. In addition, we present three extensions of the basic method that demonstrate the benefits over classical approaches to compression. First, we introduce semantic compression, where the model is trained to allocate more bits to objects of interest. Second, we study adaptive compression, where the model is adapted to a domain with limited variability, e.g., videos taken from an autonomous car, to achieve superior compression on that domain. Finally, we introduce multimodal compression, where we demonstrate the effectiveness of our model in joint compression of multiple modalities captured by non-standard imaging sensors, such as quad cameras. We believe that this opens up novel video compression applications, which have not been feasible with classical codecs.
1. Introduction
The paper presents a theoretically grounded generative approach to lossy video compression using autoencoders and autoregressive models. It evaluates architectural choices and extends the method to semantic, adaptive, and multimodal compression.
- Video compression matters economically because the paper cites an estimate that 80% of internet traffic will soon be video.
- The method uses autoencoders and autoregressive models, with a discrete latent space and deterministic encoder interpretable within the VAE framework.The paper relates rate-distortion objectives to the VAE loss terms and argues that stochastic encoders increase bitrate without improving distortion.
- The best model uses a 3D-convolutional ResNet autoencoder and a temporally conditioned gated PixelCNN prior, achieving better rate-distortion than compared learned methods.The evaluation varies 2D versus 3D autoencoders and autoregressive prior types, and compares against learned and traditional codecs under restricted settings.
- Semantic compression allocates more bits to objects of interest, such as people, by weighting rate and distortion losses for detected or segmented regions.The regions of interest can be extracted using off-the-shelf object detection or segmentation networks such as Mask R-CNN.
- Adaptive compression fine-tunes a model to the video distribution of a target domain, with autonomous-car footage yielding large rate-distortion improvements.The paper contrasts this with classical codecs, for which domain-specific fine-tuning is often not feasible.
- Multimodal compression jointly compresses multiple video modalities and outperforms HEVC/H.265 and AVC/H.264 by a factor of 4.The method exploits redundancy in videos from depth, stereo, or multi-view cameras.
2. Related Work
The related work reviews learned image and video compression, including autoregressive, interpolation-based, stochastic variational, and flow/residual approaches. The paper extends rate-distortion autoencoders to video with 3D convolutions and conditional autoregressive priors.
- Learned image compressors use low-dimensional representations, quantization approximations, and entropy minimization to compress discrete image codes.
- The paper’s closest prior work is a rate-distortion autoencoder for image compression, which it extends to video.Extensions include conditional autoregressive priors, 3D convolutions for multiple frames, and removal of spatial importance maps without compression-performance loss.
- Autoregressive video models can be powerful but scale poorly to larger videos and may be limited to lossless compression, motivating entropy coding of smaller latent codes.
- Stochastic variational video compression uses separate latent variables for frames and inter-frame dependencies, whereas this paper uses one latent space and a simpler model.
- Flow and residual modeling can improve distortion, but the cited method remains outperformed by HEVC/H.265 at low bit rates on benchmark datasets.The paper describes these approaches as promising directions that may further improve its results.
3. Rate-Distortion Autoencoders & VAEs
The paper frames lossy compression as learning discrete latents that balance coding cost against reconstruction distortion. Its rate-distortion objective connects to the VAE variational bound, while motivating deterministic encoders because practical lossy compression cannot use bits-back coding.
- Latent-variable formulation: The model learns discrete latent variables that preserve information needed to approximately reconstruct the input.The prior and decoder are treated as parameterized distributions in the latent-variable model.
- Compression interpretation: The VAE compression interpretation separates reconstruction distortion from the coding cost of transmitting latent variables.The decoder likelihood describes reconstruction, while the prior determines the latent coding cost.
- Compression interpretation: KL[q|p] discounts cross-entropy by encoder entropy H[q], because bits-back coding can recover that entropy in lossless compression.The paper notes that bits-back coding is difficult to implement and unavailable for lossy compression.
- Rate-distortion objective: The rate-distortion loss uses cross-entropy for latent coding and reconstruction distortion, with β controlling their tradeoff.For discrete latents, adding encoder entropy yields a valid bound; the resulting loss is used for compression.
- Rate-distortion objective: Deterministic encoders are optimal for fixed prior and likelihood because encoder noise increases bitrate without improving distortion.With deterministic discrete encoders, H[q] = 0, making the rate-distortion loss equivalent to the variational bound.
4. Methodology
The methodology compresses eight-frame video chunks into discrete spatial latents, entropy-codes them with autoregressive priors, and decodes them using a convolutional autoencoder. It compares frame-independent, temporally conditioned, and recurrent priors while optimizing MS-SSIM distortion and latent coding rate.
- 4.1. Preprocessing: The model processes eight-frame video chunks with 160-pixel crops and preserves the input’s temporal dimension during encoding.Inputs have shape T × C × H × W with T = 8, while spatial encoding uses stride s = 8.
- 4.2. Autoencoder: The autoencoder produces a discrete latent code with 32 channels and spatial dimensions 20 × 20 for each frame.The encoder uses no temporal stride and reduces each 160-pixel spatial dimension by a factor of eight.
- 4.2. Autoencoder: The evaluated autoencoders use fully convolutional residual architectures, with versions based on separate 2D frame processing or 3D convolutions.The architecture includes residual connections, batch normalization, and ReLU nonlinearities.
- 4.3. Autoregressive Prior: The latent code is entropy-coded with adaptive arithmetic coding driven by a gated PixelCNN prior over latent variables.The prior can model frames independently or condition predictions on temporal context.
- 4.3. Autoregressive Prior: Temporal conditioning lets the prior predict each latent from earlier positions in the same frame and information from previous frames.Frame conditioning uses the complete previous latent frame, while recurrent conditioning summarizes earlier frames with a GRU hidden state.
- 4.4. Losses: Training measures distortion with MS-SSIM and rate with the deterministic encoder’s latent log-likelihood.At decoding, reconstructed latents pass through the autoencoder decoder to produce the video reconstruction.
5. Experiments
The experiments evaluate datasets, training choices, autoencoder and prior architectures, codec comparisons, and three learned-compression extensions. Results favor spatio-temporal autoencoding, temporal conditioning, semantic weighting, domain adaptation, and multimodal redundancy exploitation.
- Datasets: The evaluation uses Kinetics, UVG, SDV, Human Activity, Dynamics, and Berkeley MHAD for training, benchmarking, ablation, semantic, adaptive, and multimodal experiments.Kinetics supplies training videos; UVG benchmarks codecs; SDV supports ablations; Human Activity supports semantic compression; Dynamics supports autonomous-car adaptation; MHAD supplies quad-camera streams.
- Ablation studies: The ablations compare 2D versus 3D autoencoders and frame-based, previous-frame-conditioned, and Conv-GRU-conditioned autoregressive priors.Models are trained on Kinetics and evaluated on SDV, with MS-SSIM as distortion loss and cross-entropy as rate loss.
- Ablation studies: A previous-frame-conditioned prior substantially improves frame-based encoding, while Conv-GRU conditioning adds only a marginal improvement over the previous frame.The advantage is particularly pronounced with a frame autoencoder.
- Ablation studies: The 3D autoencoder substantially outperforms the 2D autoencoder, with a larger difference than that between 2D and 3D priors.The authors suggest that 3D autoencoders produce largely temporally decorrelated latents, reducing the need for temporal prior conditioning.
- Comparison to state of the art: The proposal consistently outperforms other learned codecs and approaches classical codecs, particularly in the 0.10−0.25 bpp range.The comparison includes AVC/H.264 and HEVC/H.265 under default settings.
6. Conclusion
The conclusion presents deterministic-encoder rate-distortion autoencoders with autoregressive priors as an effective video-compression approach. It highlights strong codec performance and extensions that exploit semantics, domain specialization, and multimodal redundancy.
- Conclusion: The method combines a variational-autoencoder interpretation with a deterministic encoder, an autoencoder, and an autoregressive prior.The theoretical analysis favors deterministic encoders for lossy compression because bits-back coding cannot be used.
- Conclusion: The best model outperforms recent learned video codecs without flow estimation or interpolation and performs on par with H.265/HEVC.Three-dimensional spatio-temporal autoencoders are reported to reduce the need for temporal conditioning in the prior.
- Conclusion: Semantic, adaptive, and multimodal extensions demonstrate learned compression beyond overall compression performance.They prioritize important regions, finetune pretrained models to domain-specific videos, and exploit redundancy in quad-camera footage.
7. Supplementary Material
The supplementary material describes the autoencoder and autoregressive code-model architectures, quantization, conditioning, and joint rate-loss training. It also specifies how the prior predicts discrete latent codes.
- Architectures: The architecture uses three-dimensional convolutional and transposed-convolutional layers in the autoencoder and four-layer PixelCNN blocks in the prior.The supplementary figures specify channel counts, kernels, strides, same-padding, and conditioning-featuremap dimensions.
- Autoencoder: The encoder outputs continuous latent variables that are quantized with a learned codebook into one-hot probabilities.The decoder embeds these probabilities with the same codebook to obtain an approximation that is decoded into the predicted video.
- Quantization: As τ→∞, quantization becomes deterministic by concentrating probability on the closest codebook center.In practice, τ = 10^7 produces one-hot vectors for 32-bit floats, while backward gradients use a softmax with τ = 1.
- Autoregressive code-model: The autoregressive code-model predicts each latent-code probability conditioned on preceding latent elements.It uses a PixelCNN architecture, with conditioning injected into autoregressive blocks for frame- and GRU-conditioned models.
- Training: The code-model is trained with cross-entropy rate loss, and gradients also train the encoder to produce latents that are easier to predict.The authors do not detach the rate-loss gradient from the encoder parameters.
C. Evaluation procedure
The evaluation uses FFMPEG 2.8.15-0 for H.264/AVC and H.265/HEVC baselines with default settings unless otherwise reported.
- Evaluation procedure: H.264/AVC and H.265/HEVC baselines use FFMPEG 2.8.15-0 with default settings.The evaluation varies codec settings only when explicitly reported elsewhere.
C.2. Data preprocessing
The evaluation uses a shared FFMPEG-based frame extraction pipeline and reports bitrate consistently across neural and classical codecs. Rate loss provides the neural-network bitrate proxy, while MS-SSIM is computed with a benchmarked implementation.
- Data preprocessing: Frames are extracted from raw source videos as PNGs with FFMPEG, using the same data-loading pipeline for neural networks and baselines.YUV-to-RGB conversion may theoretically introduce distortion, but it is imperceptible in practice.
- Rate measurement: Classical-codec bitrate is measured as total filesize divided by the total number of pixels.
- Evaluation setup: Figure 11 reports rate-distortion results on UVG for classical and learned compression methods, with H.264 and H.265 evaluated at GoP=12.
- Rate measurement: Neural-network rate is represented by the rate loss converted into bits per pixel.The rate loss gives expected bitrate under adaptive arithmetic coding, and expected bpp correlates highly with actual bpp.
- Distortion measurement: MS-SSIM is calculated with an in-house implementation benchmarked against TensorFlow 2 using the original power factors.
D.1. Comparison to other methods
The comparison evaluates learned and traditional video compression methods under specified codec settings. Under restrictive GoP=12 settings, the proposed model exceeds H.265/HEVC rate-distortion performance above 1.2 bpp, while semantic results include background comparisons for traditional codecs.
- Codec settings: The evaluation uses FFMPEG default GoP=25 and preset=medium instead of commonly restrictive GoP and fast-preset settings.Restrictive settings can worsen traditional-codec compression performance and are often chosen to match neural-network evaluation conditions.
- Comparison to other methods: Under restrictive GoP=12 settings, the proposed model has better rate-distortion performance than H.265/HEVC above 1.2 bpp.
- Comparison to other methods: Figure 11 compares the end-to-end method with other learned compression methods and classical codecs on UVG.
- Semantic compression: Figure 12 includes background performance for traditional codecs alongside semantic-compression results.Figure 7a omits H.264/AVC and H.265/HEVC background performance to avoid clutter.
D.3. Adaptive Compression
Adaptive compression fine-tunes models for predictable domains such as autonomous-driving footage. In qualitative results, adaptation significantly improves road markings, while the generic model avoids the clear block artifacts seen with H.265/HEVC.
- Adaptive compression: The adapted model significantly improves road markings in autonomous-driving footage.
- Qualitative comparison: H.265/HEVC shows clear block artifacts around road markings, whereas the generic model avoids those artifacts but produces somewhat blurry edges.
- Adaptive compression: Expanding perspective motion in road-driving footage is a predictable pattern that neural networks can exploit but classical algorithms would find difficult to engineer manually.
- Qualitative comparison: Figure 13 presents qualitative results for adaptive compression, including H.265/HEVC at 0.025 BPP and the generic model at 0.030 BPP.