Source-linked AI summary
Content Adaptive and Error Propagation Aware Deep Video Compression
Guo Lu, Chunlei Cai, Xiaoyun Zhang, Li Chen, Wanli Ouyang, Dong Xu, Zhiyong Gao
TL;DR
Learning-based video codecs suffer from accumulated inter-frame reconstruction errors and fixed encoders that do not adapt to video content. The paper uses multi-frame rate-distortion training and online encoder updating, achieving better benchmark compression performance without increasing model size or decoding complexity.
Problem
Learning-based video compression suffers from error propagation and encoders that are not adaptive to different video contents.
Method
The method jointly optimizes rate-distortion cost over multiple consecutive frames and updates encoder parameters online while keeping the decoder unchanged.
Results
The proposed method outperforms state-of-the-art learning-based video codecs without increasing model size or decoder complexity.
Takeaways & Limitations
The approach alleviates error accumulation and enables content-adaptive coding during inference.
Takeaways & Limitations
Online rate-distortion optimization increases encoder computational complexity, with more iterations required for complex-motion sequences.
Abstract
from arXiv · showhide
Recently, learning based video compression methods attract increasing attention. However, the previous works suffer from error propagation due to the accumulation of reconstructed error in inter predictive coding. Meanwhile, the previous learning based video codecs are also not adaptive to different video contents. To address these two problems, we propose a content adaptive and error propagation aware video compression system. Specifically, our method employs a joint training strategy by considering the compression performance of multiple consecutive frames instead of a single frame. Based on the learned long-term temporal information, our approach effectively alleviates error propagation in reconstructed frames. More importantly, instead of using the hand-crafted coding modes in the traditional compression systems, we design an online encoder updating scheme in our system. The proposed approach updates the parameters for encoder according to the rate-distortion criterion but keeps the decoder unchanged in the inference stage. Therefore, the encoder is adaptive to different video contents and achieves better compression performance by reducing the domain gap between the training and testing datasets. Our method is simple yet effective and outperforms the state-of-the-art learning based video codecs on benchmark datasets without increasing the model size or decreasing the decoding speed.
1 Introduction
Learning-based video compression faces error propagation from inter-frame prediction and limited adaptation to different video contents. The proposed method addresses both through multi-frame rate-distortion training and online encoder updating while keeping the decoder unchanged.
- Traditional video compression relies on hand-designed techniques and engineered modules rather than end-to-end learning.
- Single-frame training ignores how reconstruction errors affect subsequent frames, allowing error to accumulate through inter-frame predictive coding.
- Learning-based encoders remain fixed after training and therefore cannot adapt their coding behavior to different video contents.
- The proposed joint training minimizes rate-distortion cost over multiple consecutive frames to exploit long-term temporal information.
- Online encoder updating adjusts encoder parameters per frame using the rate-distortion objective, adapts to content, and leaves the decoder unchanged.
- The method outperforms state-of-the-art learning-based codecs without increasing model size or decoder complexity.
2 Related Work
Video compression has evolved from hand-crafted hybrid coding toward end-to-end neural frameworks. Existing learning-based codecs still train on single-frame distortion and lack content-adaptive encoders, leaving error propagation insufficiently addressed.
- Traditional video compression uses motion-compensated prediction and transform coding within a block-based hybrid framework.
- End-to-end video compression replaces or jointly learns major coding components, including motion, residual, and entropy representations.
- Existing learning-based codecs optimize single-frame distortion without considering error propagation across video frames.
- Their encoders are also not adaptive to different video content despite achieving comparable or better performance than H.264.
3 Motivations Related to Learning Based Video Compression System
Inter-prediction propagates reconstruction errors across frames, reducing later-frame quality. Traditional encoders address content variation through rate-distortion-based mode selection, whereas learned encoders may remain suboptimal across domains.
- Inter-prediction causes distorted reconstructed frames to serve as references, propagating error to subsequent frames.
- The DVC 5th-frame PSNR is 33.52dB, falling to 33.37dB for the 6th frame and 31.50dB for the 50th frame.
- The DVC+EPA curve demonstrates that error propagation can be alleviated to improve compression performance.
- Traditional encoders select rate-distortion-optimal modes, using large blocks for homogeneous regions and small blocks for complex regions.
- Domain gaps in resolution or motion magnitude can make training-dataset encoders suboptimal for testing datasets.
4 Proposed Method
The proposed method extends deep video compression with error-propagation-aware training and content-adaptive encoder updating. It jointly optimizes multiple temporal steps and updates only the encoder during inference while leaving the decoder unchanged.
- DVC framework: The DVC baseline uses auto-encoder networks to compress motion and residual information under a rate-distortion objective.Its loss combines distortion with the bits required for compressed motion and residual latents.
- DVC framework: The baseline ignores reconstruction-error influence on subsequent frames and uses a fixed encoder that cannot adapt to the current frame.These dependencies lead to error propagation and can deteriorate inference-time compression performance.
- Error propagation aware training: Error-propagation-aware training jointly considers compression performance across the current and subsequent frames that depend on the current reconstruction.The procedure obtains a sequence of rate-distortion costs and optimizes multiple temporal steps; T is set to 5 in the experiments.
- Error propagation aware training: DVC+EPA improves the 50th-frame PSNR by over 0.61dB over baseline DVC, reaching 32.11dB versus 31.50dB.The reported gain becomes larger as the time step increases.
- Content adaptive encoding: Online encoder updating optimizes encoder parameters for each input frame using the rate-distortion objective while keeping the decoder unchanged.The scheme uses the original frame available at the encoder side and updates the encoder to reduce the training-testing domain gap.
- Content adaptive encoding: Online updating changes residual features and improves optical-flow detail, raising warped-frame PSNR from 33.40dB to 34.13dB.The updated bits map allocates fewer background bits, while coding bits decrease from 0.056bpp to 0.051bpp.
5 Experiments
Experiments evaluate the proposed codec across multiple datasets, distortion metrics, baselines, and ablations. Results show improved compression, especially when error propagation is stronger, while online encoder updating improves rate-distortion performance at added encoder-side computation.
- Experimental setup: Experiments use Vimeo-90k for training and HEVC, VTL, UVG, and MCL-JVC sequences for evaluation under PSNR, MS-SSIM, and bpp metrics.HEVC includes Classes B, C, and D; VTL uses 20 sequences at 352 × 288, UVG uses 1920×1080 high-frame-rate videos, and MCL-JVC contains 24 videos at 1920 × 1080.
- Comparison with state-of-the-art methods: The proposed method outperforms H.265 in MS-SSIM and achieves better compression than a recent learning-based method on the VTL dataset despite using one reference frame.The UVG evaluation also reports a large-margin improvement over another state-of-the-art learning-based codec using MS-SSIM.
- Comparison with state-of-the-art methods: The proposed method outperforms DVC by a large margin with variable GoP sizes, while remaining generally comparable to H.265 at the default setting.The authors attribute stronger gains at larger GoP sizes to more severe error propagation and note that traditional codecs use additional techniques such as multiple reference frames and adaptive quantization.
- Ablation study: 0.2dB improvement is obtained on HEVC Class C with GoP=20 when using error propagation aware training instead of baseline DVC.The training strategy considers temporal neighboring frames to alleviate error accumulation.
- Ablation study: 5.49% bitrate saving at GoP size 10 increases to 10.59% at GoP size 50, showing larger gains for longer prediction groups.Increasing the training interval from T = 3 to T = 5 also raises bitrate saving from 2.12% to 5.59%.
- Ablation and computational complexity: Online encoder updating improves compression by more than 0.5dB, while full encoder updating outperforms latent-feature or partial last-layer updating.The adaptive encoder can require more iterations for complex motion, increasing encoder computation while producing approximately 1dB improvement on HEVC Class C.
6 Conclusion
The paper proposes a content-adaptive, error-propagation-aware deep video compression method. It reduces error accumulation during training and improves compression performance without increasing model size or reducing decoding speed.
- The method alleviates error accumulation during the training stage.
- Online encoder updating enables content-adaptive coding during inference.
- The approach improves compression performance without increasing model size or decreasing decoding speed.
- Experiments show performance surpasses state-of-the-art learning-based video compression methods.