Source-linked AI summary
Learned Video Compression
Oren Rippel, Sanjay Nair, Carissa Lew, Steve Branson, Alexander G. Anderson, Lubomir Bourdev
TL;DR
Existing video codecs are hard-coded and ML-based video rate control remains challenging, motivating a more adaptable learned codec. The paper presents an end-to-end low-latency model with learned compensation, propagated state, joint signal compression, and spatial rate control. It outperforms commercial codecs across nearly the entire bitrate range, while real-time deployment remains limited by computational efficiency.
Problem
Existing codecs are hard-coded and ML-based compression lacks a single high-performing model supporting multiple bitrates comparable to separate bitrate-specific models.
Method
The paper learns video coding end-to-end for low-latency mode using arbitrary compensation, a propagated learned state, joint compression of motion and residual, and spatial rate control.
Results
The approach outperforms all existing video codecs across nearly the entire bitrate range; competing codecs produce codes up to 60% larger on standard-definition videos.
Takeaways & Limitations
The work demonstrates an ML-based video codec that surpasses commercial codecs in low-latency mode and supports learned spatial bitrate allocation.
Takeaways & Limitations
The presented approach supports only low-latency mode, and its current computational speed is insufficient for real-time deployment.
Abstract
from arXiv · showhide
We present a new algorithm for video coding, learned end-to-end for the low-latency mode. In this setting, our approach outperforms all existing video codecs across nearly the entire bitrate range. To our knowledge, this is the first ML-based method to do so. We evaluate our approach on standard video compression test sets of varying resolutions, and benchmark against all mainstream commercial codecs, in the low-latency mode. On standard-definition videos, relative to our algorithm, HEVC/H.265, AVC/H.264 and VP9 typically produce codes up to 60% larger. On high-definition 1080p videos, H.265 and VP9 typically produce codes up to 20% larger, and H.264 up to 35% larger. Furthermore, our approach does not suffer from blocking artifacts and pixelation, and thus produces videos that are more visually pleasing. We propose two main contributions. The first is a novel architecture for video compression, which (1) generalizes motion estimation to perform any learned compensation beyond simple translations, (2) rather than strictly relying on previously transmitted reference frames, maintains a state of arbitrary information learned by the model, and (3) enables jointly compressing all transmitted signals (such as optical flow and residual). Secondly, we present a framework for ML-based spatial rate control: namely, a mechanism for assigning variable bitrates across space for each frame. This is a critical component for video coding, which to our knowledge had not been developed within a machine learning setting.
1. Introduction
The paper introduces an end-to-end learned video codec for low-latency coding that addresses the limited adaptability of hard-coded codecs. It combines learned compensation, state propagation, joint signal compression, and spatial rate control, and outperforms existing codecs across nearly the entire bitrate range.
- Motivation: Video codecs serve rapidly growing traffic but remain largely hard-coded and poorly adaptable to diverse uses such as social sharing, object detection, and VR streaming.Video consumed more than 70% of internet traffic in 2016 and was expected to grow threefold by 2021.
- Approach: The proposed codec is learned end-to-end for low-latency mode, where each frame relies only on information from the past.The paper identifies this setting as important for live transmission.
- Evaluation: Across nearly the entire bitrate range, the approach outperforms existing codecs; relative code sizes are up to 60% larger for competing codecs on SD videos.On 1080p videos, H.265 and VP9 are up to 20% larger and H.264 up to 35% larger relative to the proposed method.
- Contributions: The system learns arbitrary spatio-temporal compensation beyond translation, enabling more accurate predictions for patterns such as out-of-plane head rotation.Traditional codecs cannot predict a profile face from a frontal view using simple translations.
- Contributions: Rather than relying only on reference frames and optical flow, the model propagates an autonomously learned state that can retain information over longer periods.The state is designed to maximize information retention across frames.
- Contributions: Motion and residual signals are jointly compressed through one bottleneck, allowing bitrate allocation between them to depend on frame complexity.Traditional methods compress motion and residual separately, limiting their tradeoff.
- Contributions: The codec introduces flexible motion fields, multi-flow scene representations, and ML-driven spatial rate control for variable bitrates across each frame.Spatial rate control assigns different bitrates to different spatial locations while supporting multiple rates in one model.
2. Model Architecture
The architecture progressively replaces the traditional flow-residual pipeline with jointly compressed signals, a learned propagated state, and arbitrary compensation. These changes address limited expressiveness and memory while improving bitrate efficiency.
- General structure: The video coder maps each frame and previous state to a transmitted bitstream and reconstructed frame.The state represents prior memory propagated across time.
- Step #1: ML formulation of the flow-residual paradigm: The initial ML formulation estimates, compresses, and reconstructs optical flow, then uses it for motion compensation and separately compresses the residual.The residual is defined as the difference between the target frame and its motion-compensated approximation.
- Step #2: Joint compression: The architecture then jointly compresses flow and residual through a single encoder-decoder bottleneck.This removes redundancies and lets the model distribute bandwidth according to input complexity.
- Step #3: Propagation of a learned state: A learned state replaces pixel-space reference frames and flows, allowing arbitrary information to be propagated across frames.State updates pass through a low-bandwidth bottleneck, and reconstructed frames are generated from the updated state.
- Step #3: Propagation of a learned state: Introducing the learned state results in 20-40% bitrate savings.The result is reported in Figure 9.
- Step #4: Arbitrary compensation: Arbitrary compensation generalizes flow-based prediction beyond simple translations by generating multiple flows and reference frames.The model can learn compensation that changes values rather than only moving pixels.
3. Coding procedure
The coding procedure converts a learned tensor into a variable-length bitstream by bitplane decomposition and adaptive entropy coding. A codelength regularizer shapes sparsity, while feedback adjusts its strength toward a target bitcount.
- Coding procedure: The coding procedure maps a fixed-size tensor c to a variable-length bitstream e by exploiting redundancy injected during training.The tensor values lie in [-1, 1] and the bitstream length is variable.
- Bitplane decomposition: Bitplane decomposition converts c into B binary planes, truncating precision; the implementation uses B = 6.Each scalar value is represented by a six-bit binary expansion.
- Adaptive entropy coding: Adaptive entropy coding maps the binary tensor to a bitstream using probabilities conditioned on neighboring pre-transmitted bits.The context leverages structure within and across bitplanes.
- Adaptive codelength regularization: The codelength regularizer reduces the entropy of the binary representation by making higher-index bitplanes increasingly sparse.This shaping is designed to improve the entropy coder's efficiency.
- Adaptive codelength regularization: The scalar αi is modulated by feedback so the observed average codelength approaches a target bitcount.Training monitors average codelength and adjusts αi according to the discrepancy from the target.
4. Spatial Rate Control
The paper develops spatial rate control for neural video compression, assigning different bitrates across each frame while selecting among multiple learned codelayers. A rate controller chooses local rates using estimated rate-distortion slopes.
- Motivation: Spatial rate control assigns arbitrary bitrates independently across spatial locations in each frame.The rates can respond to reconstruction complexity, network conditions, and quality guarantees.
- Motivation: A single fixed-bitrate model can accumulate error because coding complexity varies across space and time.Areas that are difficult to reconstruct at a given quality become more difficult in subsequent frames.
- Spatial multiplexing framework: The framework uses R distinct codelayers, each associated with a different rate and trained with its own entropy coder and regularization.The rate map selects which codelayer is active at each spatial location.
- Spatial multiplexing framework: At each location, a rate map selects one active codelayer through binary masks, and the bitstream encodes active values plus the rate mask.The decoder needs the rate mask to reconstruct the selected spatial configuration.
- Spatial multiplexing framework: The spatial multiplexer splits the encoder into R lightweight branches and sums corresponding decoder outputs in a common space.Each encoder and decoder branch contains a single convolution.
- Rate controller algorithm: The controller selects the largest local rate whose estimated rate-distortion slope meets threshold λ.It estimates slopes between neighboring rates for each spatial location and quality metric.
5. Results
The evaluation benchmarks the learned low-latency codec against mainstream codecs on standard-definition and high-definition datasets using rate-distortion, relative-size, ablation, and runtime analyses. The method is evaluated with MS-SSIM-based quality measurement and includes a spatial rate-control ablation.
- Metrics and color space: BPP measures total encoded file size, including header information, averaged across all video pixels.Final reconstructions are evaluated with MS-SSIM, while intermediate motion-compensated reconstructions use the Charbonnier loss.
- Evaluation setup: The benchmarks include HEVC/H.265, AVC/H.264, VP9, and the HEVC HM reference implementation in low-latency configurations.H.264 and H.265 are tested with default and slower presets, while B-frames are removed.
- Evaluation setup: The study evaluates 34 CDVL SD videos and 22 Xiph 1080p HD videos using standard video-compression test sets.The datasets contain 15,650 SD frames and 11,680 HD frames; HD videos are center-cropped to height 1024.
- Compression results: Figure 8 reports average MS-SSIM as a function of BPP and average compressed sizes relative to WaveOne at representative MS-SSIM levels.Relative-size comparisons use shared valid MS-SSIM ranges of 0.990-0.998 for SD and 0.980-0.994 for HD.
- Ablation studies: The ablations vary learned state, flow-based motion compensation, and spatial rate control, evaluating configurations with and without the rate-control framework.The comparison includes the full model, a previous-frame-and-flow model without learned state, and a naïve model without motion compensation.
- Runtime: The decoder runs at around 10 frames/second and the encoder at around 2 frames/second on VGA videos using an NVIDIA Tesla V100.The paper states that this speed is insufficient for real-time deployment.
6. Conclusion
The paper concludes that its learned codec outperforms commercial codecs across nearly the entire bitrate range in low-latency mode. It identifies real-time efficiency and B-frame support as future directions.
- 6. Conclusion: The work introduces an ML-based video codec that outperforms all commercial codecs across nearly the entire bitrate range in low-latency mode.The supported setting uses only past information for each frame.
- 6. Conclusion: Future work targets improved computational efficiency for real-time coding and extension to temporal interpolation modes using B-frames.The current approach supports only low-latency mode.
A.1. CDVL SD
The CDVL SD appendix describes how the standard-definition evaluation videos were retrieved and filtered from the Consumer Digital Video Library.
- A.1. CDVL SD: The SD videos were retrieved by searching the Consumer Digital Video Library for VGA resolution at original and excellent quality levels.Near-duplicate videos were reduced by retaining only the first retrieved instance.
- A.1. CDVL SD: The appendix lists the retrieved CDVL video names used for the standard-definition evaluation.The list includes videos from the Bennet-Watt and NTIA collections.
A.2. Xiph HD
The Xiph HD evaluation uses 1080p videos from the Xiph test collection, comprising the listed sequences.
- A.2. Xiph HD: The Xiph test videos are available from the Xiph media video collection.The passage provides the collection URL.
- A.2. Xiph HD: All videos used from this collection have 1080p resolution.
- A.2. Xiph HD: The evaluated sequences include aspen, blue_sky, controlled_burn, crowd_run, dinner, ducks_take_off, in_to_tree, life, old_town_cross, park_joy, pedestrian_area, red_kayak, and riverbed.
- A.2. Xiph HD: Additional evaluated sequences include rush_field_cuts, rush_hour, snow_mnt, speed_bag, station2, sunflower, touchdown_pass, tractor, and west_wind_easy.