Source-linked AI summary
Video Compression through Image Interpolation
Chao-Yuan Wu, Nayan Singhal, Philipp Krähenbühl
TL;DR
Video compression is important but conventional codecs rely heavily on hand-designed engineering. This paper introduces an end-to-end deep codec based on repeated image interpolation, achieving performance above several prevailing codecs and on par with H.264.
Problem
Video compression is increasingly important, yet competitive video codecs remain largely hand designed and hand optimized.
Method
The codec encodes anchor frames and reconstructs intervening frames through hierarchical deep image interpolation, using compressible information to disambiguate reconstructions.
Results
The codec outperforms deep baselines, MPEG-4 Part 2, and H.261 in compression rate and visual quality, while matching state-of-the-art H.264.
Takeaways & Limitations
Repeated deep image interpolation achieves state-of-the-art video compression without sophisticated heuristics or excessive engineering.
Takeaways & Limitations
Runtime and real-time compression engineering were not considered and remain future research directions.
Abstract
from arXiv · showhide
An ever increasing amount of our digital communication, media consumption, and content creation revolves around videos. We share, watch, and archive many aspects of our lives through them, all of which are powered by strong video compression. Traditional video compression is laboriously hand designed and hand optimized. This paper presents an alternative in an end-to-end deep learning codec. Our codec builds on one simple idea: Video compression is repeated image interpolation. It thus benefits from recent advances in deep image interpolation and generation. Our deep video codec outperforms today's prevailing codecs, such as H.261, MPEG-4 Part 2, and performs on par with H.264.
1 Introduction
The paper introduces an end-to-end deep video codec that treats video compression as repeated image interpolation. It reports stronger results than several conventional codecs and performance comparable to H.264.
- Core idea: Video compression is framed as repeated image interpolation using recent advances in deep image generation and interpolation.Anchor frames are encoded with deep image compression, while remaining frames are reconstructed by interpolating neighboring anchors.
- Results: 0.080 BPP: the model achieves MS-SSIM 0.984, compared with 0.980 for H.264 and 0.946 for MPEG-4 Part 2.On Blender’s Tears of Steel, the model has better visual quality than MPEG-4 Part 2 and comparable quality to H.264.
- Core idea: The codec adds a small compressible code to disambiguate otherwise non-unique interpolations and reconstruct frames faithfully.The main technical challenge is designing a compressible interpolation network.
- Architecture: The codec uses increasingly powerful interpolation architectures, incorporating offline block-motion or optical-flow estimates to handle moving patterns.A hierarchical interpolation scheme further reduces bitrate by interpolating between progressively closer reference frames.
- Evaluation: Across VTL, UVG, and a 2.8M-frame Kinetics subset, the codec outperforms deep baselines, MPEG-4 Part 2, and H.261, while matching H.264.Compression rate and visual quality are evaluated with MS-SSIM and PSNR.
2 Related Work
Prior codecs exploit temporal redundancy through hand-designed block motion estimation and frame referencing, while learning-based video compression remains comparatively underexplored. The paper positions deep interpolation as a general alternative.
- Image compression: Deep image compression achieves strong spatial compression but does not exploit the temporal redundancy that makes video compression effective.The paper extends deep image-compression ideas toward variable-rate video compression.
- Video compression: Traditional video codecs divide frames into macroblocks and use I-, P-, and B-frames with motion estimates and residual images.H.264 and HEVC organize these references hierarchically.
- Video compression: Traditional video compression requires intensive engineering and makes joint optimization difficult.The paper addresses this with a hierarchical neural codec trained end-to-end without hand-engineered heuristics or filters.
- Learning-based video compression: Learning-based video compression is largely unexplored because modeling temporal redundancy is difficult, and prior postprocessing work may not generalize beyond domain-specific videos.The paper presents what it describes as the first general deep network for video compression.
- Image interpolation and extrapolation: Image interpolation methods estimate motion through spatially varying kernels or flow fields, then combine forward and backward predictions.Such methods work well for small timesteps but struggle at larger timesteps, where interpolation is non-unique and side information is needed.
3 Preliminary
The preliminary formulation represents video frames with binary codes and balances bitrate against reconstruction fidelity. It builds from recurrent deep image compression and extends it with temporal motion and interpolation information.
- Problem formulation: The encoder maps video frames to binary codes and the decoder reconstructs them, with objectives of minimizing total bitrate and reconstruction error.Reconstruction error is measured with an L1 loss.
- Image compression: Deep image compression progressively encodes residuals between the original frame and the previously reconstructed image.The encoder and decoder update latent Conv-LSTM states across iterations.
- Image compression: The recurrent image-compression model supports variable bitrate encoding through the choice of iteration count K.All iterations share the same architecture and parameters.
- Video compression: Conventional video codecs compress P-frames using a motion estimate and residual image, then reconstruct each color frame from those components.The residual captures appearance changes not explained by motion, and both signals are entropy coded.
- Proposed extension: The paper augments image interpolation with motion information and a compressible bottleneck layer.This provides a learned mechanism for representing information needed beyond interpolation from reference frames.
4 Video Compression through Interpolation
The codec compresses key frames and reconstructs remaining frames through conditional interpolation, progressively adding motion and residual information to improve fidelity. A hierarchical design reduces bitrate by interpolating between increasingly close reference frames, while bitrate selection balances distortion and rate.
- Codec structure: Every n-th frame is encoded as an I-frame, while the remaining n−1 frames are interpolated as R-frames; in practice, n = 12.The codec uses Toderici et al.'s image compression algorithm for I-frames and then applies interpolation to the remaining frames.
- Basic interpolation: Blind interpolation reconstructs frames from key-frame context alone, favoring compression rate over image quality because R-frames add no unseen information.The model extracts multiscale U-net feature maps from two key frames and jointly trains the context and interpolation networks.
- Motion compensation: Motion-compensated interpolation warps context features using optical flow or block-motion estimates, allowing the decoder to focus on image creation rather than motion estimation.Block motion is easier to compress, whereas optical flow preserves finer details.
- Residual interpolation: Residual motion-compensated interpolation combines motion compensation with compressed residual information to capture both motion and appearance differences.The residual component supplies information not explained by the reference frames.
- Hierarchical interpolation: Hierarchical interpolation uses previously decompressed frames as references, reducing bitrate through closer temporal offsets while limiting error propagation to practical hierarchy depths.The system extends a binary hierarchy from n = 8 to n = 12 frames, but performance drops when extending to n = 27 because interpolations become more distant.
- Bitrate optimization: Beam-search bitrate selection prunes combinations outside the MS-SSIM-versus-bitrate envelope because bitrate choices at one hierarchy level affect lower-level errors.The globally optimal combination is infeasible to enumerate, so the method retains approximately O(m) combinations in practice.
5 Experiments
The experiments evaluate the codec across datasets, interpolation designs, motion representations, entropy coding, and established video codecs. Results show that hierarchical, motion-aware interpolation with entropy coding achieves strong compression-quality trade-offs and competitive performance against prior codecs.
- Datasets and Protocol: 2.8M frames across Kinetics, VTL, and UVG support evaluation over varied resolutions, dates, quality levels, and video contents.Kinetics supplies training, validation, and testing data; VTL and UVG provide raw-video evaluation at 352 × 288 and 1920 × 1080 resolutions.
- Ablation study: Motion-compensated interpolation works significantly better than vanilla interpolation, because motion information disambiguates the interpolation.The reported BPP includes the size of motion vectors.
- Ablation study: Block motion estimation outperforms optical flow because its motion information is smaller, easier to compress, and losslessly compressible.A hypothetical losslessly compressed optical-flow model performs better than motion vectors, but compressible optical-flow estimation is left for future work.
- Ablation study: 52% BPP savings at low bitrate and at least 10% at high bitrate show that entropy coding improves every interpolation model.Short-time-frame interpolation reaches image-model visual quality at one or two orders of magnitude fewer bits per pixel.
- Comparison to prior work: The codec outperforms MPEG-4 Part 2 and H.261, performs on par with H.264, and approaches HEVC across the evaluated datasets.On high-resolution UVG, it outperforms H.264 by a good margin and matches HEVC in PSNR.
- Comparison to prior work: At 0.12 ± 0.01 BPP, the codec produces faithful images without blocky artifacts and matches state-of-the-art H.264 qualitatively.The qualitative comparison includes MPEG-4 Part 2, H.264, and the proposed method.
6 Conclusion
The paper presents a deep video codec based on repeated image interpolation, achieving strong compression performance without sophisticated heuristics or excessive engineering. Runtime and real-time compression remain unaddressed engineering considerations.
- 6 Conclusion: Repeated deep image interpolation forms the basis of the codec, with compressible information added to disambiguate interpolations.This supports faithful reconstruction rather than pure hallucination.
- 6 Conclusion: The authors report better performance than MPEG-4 Part 2 and H.261 while matching state-of-the-art H.264.
- 6 Conclusion: Runtime and real-time compression are not considered and are identified as important directions for future research.
- 6 Conclusion: The codec achieves state-of-the-art performance without sophisticated heuristics or excessive engineering.
A Model Details
The model details describe feature fusion choices, architecture-specific resolutions, probability estimation for adaptive arithmetic coding, and validation-based model selection.
- A Model Details: Fusing U-net features at resolution W^2 × H^2 yields good performance across models.Fusing more resolutions improves performance slightly but requires more memory and computation.
- A Model Details: The encoder uses W^4 × H^4 features for M1,2 and M3,3.
- A Model Details: The M3,3 decoder uses W^4 × H^4 and W^8 × H^8 features.
- A Model Details: Models are selected according to validation-set performance.
- A Model Details: Adaptive arithmetic coding estimates bit probabilities with a 3D-Pixel-CNN containing 11 masked-convolution layers.
B Bitrate Optimization
The bitrate optimization procedure explores combinations of iteration counts hierarchically, retaining promising configurations for deeper levels and final experiments.
- B Bitrate Optimization: The algorithm selects promising iteration-count combinations from the performance envelope at the first hierarchy level.Selected (K0, K1)-combinations proceed to the next level.
- B Bitrate Optimization: The final combinations returned by the optimization algorithm are used in the paper’s experiments unless otherwise noted.
- B Bitrate Optimization: K0, K1, K2, and K3 denote iteration counts for the I-frame, M6,6, M3,3, and M1,2 models, respectively.