VCT: A Video Compression Transformer

Fabian Mentzer; George Toderici; David Minnen; Sung-Jin Hwang; Sergi Caelles; Mario Lucic; Eirikur Agustsson

Source-linked AI summary

VCT: A Video Compression Transformer

Fabian Mentzer, George Toderici, David Minnen, Sung-Jin Hwang, Sergi Caelles, Mario Lucic, Eirikur Agustsson

arXiv:2206.07307v2Jun 15, 2022cs.CVcs.LGeess.IV

TL;DR

Neural video compression methods often rely on complex architectural biases such as motion prediction and warping. VCT independently maps frames to representations and uses a transformer to model temporal dependencies, outperforming prior methods without those biases.

Problem
Existing neural video compression methods increasingly use complex architectural biases and priors, making them harder to implement and dependent on matching data characteristics.
Method
VCT independently maps input frames to representations and uses a transformer-based temporal entropy model to predict future representations from past frames.
Results
VCT outperforms previous methods on standard video compression datasets without motion prediction or warping components.
Takeaways & Limitations
The results show that transformer-based video compression can learn useful temporal relationships and handle varied video patterns without hand-designed motion or warping components.
Takeaways & Limitations
The model uses independence assumptions in its representation factorization, leaving alternatives such as channel autoregression and vector quantization for future work.

Abstract

from arXiv · show

We show how transformers can be used to vastly simplify neural video compression. Previous methods have been relying on an increasing number of architectural biases and priors, including motion prediction and warping operations, resulting in complex models. Instead, we independently map input frames to representations and use a transformer to model their dependencies, letting it predict the distribution of future representations given the past. The resulting video compression transformer outperforms previous methods on standard video compression data sets. Experiments on synthetic data show that our model learns to handle complex motion patterns such as panning, blurring and fading purely from data. Our approach is easy to implement, and we release code to facilitate future research.

1 Introduction

VCT replaces motion prediction, warping, and residual compensation with a transformer that models dependencies among independently encoded frame representations. This simpler design avoids temporal error propagation and outperforms prior methods while learning motion patterns directly from data.

1 Introduction: VCT replaces flow prediction, warping, and residual compensation with a transformer-based temporal entropy model.The transformer predicts distributions for quantized representations, which are then losslessly compressed with entropy coding.
1 Introduction: The transformer outperforms previous methods on standard video compression datasets despite using no motion or warping components.The authors report that it wins in both PSNR and MS-SSIM on MCL-JCV and UVG.
1 Introduction: The model independently maps each frame to a quantized representation, then uses prior representations to predict the current representation's distribution.A decoder reconstructs each frame from its representation, while better distribution prediction reduces the required bitrate.
1 Introduction: VCT avoids complex state transitions and temporal error propagation because each reconstruction does not depend on previous reconstructions.Warping-based methods can propagate visual errors forward and require additional residual bits to correct them.
1 Introduction: Attention over all representation symbols is impractical at 1080p, so VCT introduces independence assumptions that shrink the attention matrix and enable parallel execution.A typical representation contains approximately 1.6 million symbols, making naive attention a roughly 1.6M×1.6M matrix.
1 Introduction: The approach is designed for straightforward implementation, and the authors release code and models to support future research.The paper explicitly presents the model as easy to implement.

2 Related Work

Related neural video compression methods largely build on CNN-based motion estimation, warping, residual correction, and recurrent or autoregressive temporal models. VCT instead applies transformer-based temporal modeling to independently encoded representations.

2 Related Work: Transformers had already replaced or supplemented components of neural image compression, including hyperpriors and convolutional blocks.These developments motivated applying transformer architectures beyond image compression.
2 Related Work: Earlier neural video codecs commonly predict optical flow, transmit compressed flow and residuals, and correct warped reconstructions.This line of work extended traditional motion-compensation pipelines with flow and residual components.
2 Related Work: Neural video compression research has also explored scale-space flow, recurrent decoders, ConvLSTMs, and additional temporal context.Scale Space Flow became a building block for several approaches, while recurrent models were used for decoding or entropy modeling.
2 Related Work: Other methods avoid direct pixel-space flow by using 3D autoregressive models, feature-space flow, iterative quantization, or feature-space residual compensation.These approaches still rely on CNNs, ConvLSTMs, or motion estimation for temporal modeling in different parts of the codec.

3 Method

VCT independently encodes frames into quantized representations, then uses a transformer-based temporal entropy model to predict and losslessly code their distributions. The design replaces motion-specific components with learned temporal dependencies while retaining causal block-wise autoregression.

Frame encoding and temporal coding: VCT independently encodes each frame into a quantized representation, then entropy-codes it using transformer-predicted distributions conditioned on prior representations.This separates lossy frame transform coding from lossless entropy coding.
Frame encoding and temporal coding: The entropy model exploits temporal redundancy by predicting conditional symbol distributions, reducing bits when its predictions match the observed representations.Entropy coding uses more frequent values with fewer bits, while imperfect predictions remain lossless but less efficient.
Transformer-based temporal entropy model: The transformer processes current spatial blocks with overlapping context from the two preceding representations, using independence assumptions to make attention computationally practical.Blocks in the current representation are predicted from larger corresponding blocks in the two previous representations.
Transformer-based temporal entropy model: A temporal mixer combines information from previous blocks, while masked within-block autoregression predicts token PMFs conditioned on that context and already transmitted tokens.A learned start token ensures the sender and receiver produce identical causal distributions for entropy coding.
Evaluation: On MCL-JCV and UVG, VCT outperforms all compared methods in both PSNR and MS-SSIM despite using no motion or warping components.The rate-distortion comparison reports bits per pixel and megabits per second for these data sets.
Architectures: The model uses standard transformer components, including separate temporal-processing transformers and a masked conditional transformer for current-block prediction.The architecture uses Tsep, Tjoint, and Tcur with learned positional embeddings and masked attention for causality.

4 Experiments

The experiments train VCT on large-scale Internet video clips and evaluate it against neural and standard codecs on established benchmarks and synthetic motion datasets.

Datasets: VCT is trained on one million nine-frame Internet video clips and evaluated on MCL-JCV and UVG.MCL-JCV contains thirty 1080p videos, while UVG is the second common benchmark dataset.
Synthetic data: Synthetic datasets generate 512×512 videos from still images to study shifting, blurring or sharpening, and other parameterized temporal patterns.Each dataset varies a parameter x and contains 100 videos for each value.
Baselines: The comparison includes HEVC, H.264, DVC, SSF, ELF-VC, FVC, DCVC, RLVC, and CNN-based temporal entropy modeling.For fairness, B-Frames are disabled in the standard codecs, while other codec constraints are not imposed.
Metrics: Evaluation reports PSNR and MS-SSIM in RGB, with models trained using MSE or a 200 · (1 − MS-SSIM) objective.The MS-SSIM objective is used for Stage III models.

5 Results

VCT outperforms the compared methods on benchmark rate-distortion performance and learns useful temporal behavior across synthetic motion patterns and decoding stages.

Comparison to State of the Art: VCT outperforms all compared neural methods in both PSNR and MS-SSIM on MCL-JCV and UVG despite using no motion or warping components.The comparison is presented as rate-distortion graphs in Fig. 4.
Synthetic data: ≈45% lower R-D loss than SSF occurs for shifting videos, while VCT matches HEVC when shifts are multiples of 16.SSF saturates at about x = 10, whereas VCT’s representation shifts exactly one symbol for stride-aligned shifts.
Synthetic data: VCT beats HEVC and SSF on blurring or sharpening and maintains near-constant R-D loss as fading increases.SSF has ≈20% lower R-D loss for blurring than sharpening, while neither baseline explicitly supports fading.
Visualizing certainty during decoding: As more tokens are decoded, the transformer becomes more certain and the reconstructed image sharpens, while showing some implicit second-order motion modeling.The visualization conditions on k ∈ {0, 2, 13} decoded tokens.
Ablations: Conditioning on one previous frame reduces bitrate by −58%, two previous frames add −6%, and latent residual prediction increases PSNR by 0.7dB at equal bitrate.More temporal context produced no further gains in this ablation.

6 Conclusion and Future Work

The paper concludes that VCT simplifies neural video compression while matching or surpassing prior methods without explicit motion priors, and identifies longer memory and distribution factorization as future directions.

Conclusion: VCT outperforms previous methods without explicit motion prediction or warping, using only a two-frame window into the past.The authors present this as an architectural simplification of neural video compression.
Future Work: Future work could add longer-term memory or arbitrary reference frames for video types that may benefit from broader temporal context.The conclusion frames these as possibilities rather than established improvements.
Future Work: Alternative distribution factorizations, including vector quantization and channel autoregression, remain unexplored design choices.The authors also suggest changing the independence assumptions used to split representations into blocks.
Societal Impact: The authors suggest that reduced bandwidth and storage requirements could benefit video conferencing, streaming, and knowledge preservation.These effects are presented as hoped-for societal impact.

NeurIPS Checklist

The checklist reports coverage of contributions, limitations, ethics, assets, and experimental documentation, while noting that training data and error bars are not fully available.

Checklist: The authors report that they described their contributions, scope, limitations, and potential negative societal impacts.The checklist points readers to the relevant paper sections.
Experimental Reporting: The experiments report training details and compute resources but do not report error bars across repeated runs.The authors say multiple runs usually ended at similar final losses.
Ethics and Data: The paper reports no new data, personally identifiable information, offensive content, crowdsourcing, or human subjects research.The checklist marks the corresponding ethics and data questions as not applicable.
Reproducibility: The paper states that code will be released, but training data cannot be released and full reproducibility assets are therefore unavailable.The checklist marks code release as planned but the requested data and instructions package as unavailable.

A.1 Main Text Image auto-encoder (E, D) details

The main image auto-encoder uses convolutional and residual blocks, with residual placement varied to study decoder performance.

A.1 Main Text Image auto-encoder (E, D) details: The encoder E is four stride-2 convolutional blocks, while decoder D combines transposed convolutions with residual blocks.Both convolution types use 5×5 kernels, leaky ReLU activations, and dED = 192 filters.
A.1 Main Text Image auto-encoder (E, D) details: The default decoder uses fewer residual blocks at higher resolutions, following the pattern 4220 across transpose-convolution stages.Residual blocks are represented by R, and transposed convolutions by T.
A.1 Main Text Image auto-encoder (E, D) details: Decoder variants 0000 and 2222 test whether removing residual blocks or moving them later affects performance on MCL-JCV.The 2222 variant places residual blocks in higher-resolution features, making them more expensive.

A.2 Public Code Release: Simplified Training from Scratch

The public code release provides a simplified training setup, while supplementary experiments examine data scaling and CNN-versus-transformer temporal modeling.

A.2 Public Code Release: Simplified Training from Scratch: Training the main encoder and decoder from scratch is unstable, so the released setup uses a lightweight ELIC architecture for Stage III.The setup trains directly from scratch with learning rate 1E−4 for 750k steps and does not require hyperprior training in the reported excerpt.
A.2 Public Code Release: Simplified Training from Scratch: 50k training clips are sufficient for VCT to outperform the CNN-based SSF method on MCL-JCV.VCT benefits from additional training data, whereas SSF does not benefit from a significant increase in dataset size.
A.4 Transformers vs. CNNs: Using one frame of context instead of VCT’s two increases bitrate by 18%.This ablation isolates context length from the CNN-versus-transformer comparison.
A.4 Transformers vs. CNNs: Replacing VCT’s transformer with a CNN increases bitrate by around 50%, indicating that the transformer accounts for most of the remaining gap.The comparison uses a CNN temporal entropy model and includes a reproduced CNN baseline trained on the authors’ data.
A.2 Public Code Release: Simplified Training from Scratch: The supplementary figures report comparisons on MCL-JCV and UVG using PSNR and MS-SSIM, alongside a comparison to Liu et al.The supplied captions identify the datasets and metrics but do not state numerical outcomes.