Source-linked AI summary
VRT: A Video Restoration Transformer
Jingyun Liang, Jiezhang Cao, Yuchen Fan, Kai Zhang, Rakesh Ranjan, Yawei Li, Radu Timofte, Luc Van Gool
TL;DR
Video restoration requires using neighboring but misaligned frames, while existing sliding-window and recurrent approaches restrict parallel processing or long-range modelling. VRT uses multi-scale temporal mutual self-attention and parallel warping to reconstruct frames in parallel. It outperforms state-of-the-art methods by up to 2.16dB across five restoration tasks and fourteen benchmark datasets.
Problem
Video restoration must exploit multiple neighboring frames despite their misalignment, while recurrent methods limit parallelization and long-range dependency modelling.
Method
VRT is a multi-scale framework that uses temporal mutual self-attention and parallel warping to align, extract, and fuse frame features for parallel reconstruction.
Results
VRT achieves state-of-the-art performance across five video-restoration tasks and fourteen benchmark datasets, with gains up to 2.16dB.
Takeaways & Limitations
VRT provides parallel computation and long-range dependency modelling for video restoration while jointly aligning and fusing multi-frame information.
Takeaways & Limitations
VRT has larger model size and longer runtime per frame than recurrent models, although it may support distributed deployment.
Abstract
from arXiv · showhide
Video restoration (e.g., video super-resolution) aims to restore high-quality frames from low-quality frames. Different from single image restoration, video restoration generally requires to utilize temporal information from multiple adjacent but usually misaligned video frames. Existing deep methods generally tackle with this by exploiting a sliding window strategy or a recurrent architecture, which either is restricted by frame-by-frame restoration or lacks long-range modelling ability. In this paper, we propose a Video Restoration Transformer (VRT) with parallel frame prediction and long-range temporal dependency modelling abilities. More specifically, VRT is composed of multiple scales, each of which consists of two kinds of modules: temporal mutual self attention (TMSA) and parallel warping. TMSA divides the video into small clips, on which mutual attention is applied for joint motion estimation, feature alignment and feature fusion, while self attention is used for feature extraction. To enable cross-clip interactions, the video sequence is shifted for every other layer. Besides, parallel warping is used to further fuse information from neighboring frames by parallel feature warping. Experimental results on five tasks, including video super-resolution, video deblurring, video denoising, video frame interpolation and space-time video super-resolution, demonstrate that VRT outperforms the state-of-the-art methods by large margins ($\textbf{up to 2.16dB}$) on fourteen benchmark datasets.
1 Computer Vision Lab, ETH Zurich, Switzerland 2 Meta Inc. 3 KU Leuven, Belgium
The paper lists affiliations in Switzerland, the United States, and Belgium, along with author contact information and a project repository.
- The listed affiliations are ETH Zurich's Computer Vision Lab in Switzerland, Meta Inc., and KU Leuven in Belgium.
- Author contact information is provided through vision.ee.ethz.ch and fb.com email domains.
- The paper provides a GitHub repository at github.com/JingyunLiang/VRT.
1. Introduction
Video restoration must exploit multiple neighboring frames despite their misalignment. VRT addresses limitations of sliding-window and recurrent approaches through parallel computation and long-range dependency modelling.
- Video restoration reconstructs high-quality frames from multiple low-quality frames using misaligned neighboring frames.
- Sliding-window methods repeatedly process input frames and incur inefficient feature utilization and higher computation costs.
- Recurrent methods limit parallelization, weaken long-range dependency modelling, and suffer performance drops on few-frame videos.
- VRT enables parallel computation and long-range dependency modelling for video restoration.
- VRT uses shifted non-overlapping clips, temporal mutual self-attention, and parallel warping within a multi-scale framework.
- VRT adaptively aligns and fuses supporting-frame features into reference frames through mutual attention.
- VRT achieves state-of-the-art performance across five video-restoration tasks, with gains up to 2.16dB on benchmark datasets.
2. Related Work
Related work spans CNN-based sliding-window and recurrent video-restoration frameworks, while Transformer-based video models mainly target recognition or feature fusion. VRT provides a restoration-oriented multi-scale framework for parallel reconstruction and temporal information fusion.
- Learning-based, especially CNN-based, methods are the primary workhorse for video restoration.
- Sliding-window methods usually process short frame sequences and predict the center frame, whereas recurrent methods propagate prior predictions.
- Many Transformer video models extend attention temporally but are designed mainly for visual recognition rather than restoration.
- VRT reconstructs T high-quality frames in parallel from T low-quality frames within a multi-scale framework.
- A CNN-Transformer video super-resolution model does not fully exploit local patch information and suffers border artifacts during testing.
3. Video Restoration Transformer
VRT restores video frames in parallel by combining multi-scale feature processing, temporal mutual self attention, and parallel warping. Its attention modules align and fuse neighboring frames while shifted clips and larger temporal windows support cross-clip and long-range interactions.
- Overall Framework: VRT extracts and reconstructs T high-quality frames from T low-quality frames in parallel for multiple video restoration tasks.The framework separates feature extraction from reconstruction and supports video super-resolution, deblurring, denoising, and related tasks.
- Overall Framework: Multi-scale processing aligns features at different image resolutions to handle object and camera motions across scales.Features are downsampled and then progressively upsampled, with skip connections between features at the same scales.
- Mutual Attention: Mutual attention lets reference-frame elements query supporting-frame elements and aggregate their weighted features for alignment and fusion.When one attention weight dominates, the operation equals warping a supporting-frame element to the corresponding reference-frame position; otherwise, it acts as soft warping.
- Temporal Mutual Self Attention: TMSA combines mutual attention with self attention because mutual attention alone does not preserve information from the reference frame.Mutual attention performs bidirectional alignment between two frames, while self attention extracts and preserves current-frame features.
- Temporal Mutual Self Attention: TMSA partitions sequences into non-overlapping two-frame clips, applies mutual and self attention, and shifts clips every other layer for cross-clip interactions.The shifted design reduces exhaustive frame-pair complexity from O(T^2) to O(T).
- Parallel Warping: Parallel warping fuses neighboring-frame features at each network stage to address large motions that spatial window partitioning may handle poorly.Neighboring features are warped toward the current frame and concatenated with the original feature.
4. Experiments
Across five video-restoration tasks, VRT achieves strong quantitative and visual results while modeling long sequences in parallel, with measured trade-offs in runtime and model size.
- Video Super-Resolution: VRT achieves the best video super-resolution performance for bicubic and blur-downsampling degradations across the reported datasets.It outperforms VSRT by up to 0.57dB on REDS short sequences and EDVR by 0.50∼1.57dB on different datasets.
- Video Super-Resolution: VRT restores all frames simultaneously rather than predicting them frame by frame, supporting long-sequence temporal modeling.Training on longer sequences further increases PSNR by 0.52dB, while performance remains strong on both short and long sequences.
- Visual Results: VRT produces sharp edges and fine details in video super-resolution and removes motion blur while restoring faithful details in deblurring.Competitors are described as suffering from distorted textures, lost details, incomplete blur removal, or unsharp edges.
- Video Deblurring: 1.47dB and 0.0299 gains in PSNR and SSIM over ARVo on DVD demonstrate VRT’s leading video-deblurring performance.VRT also reports significant PSNR gains of 2.15dB on GoPro and 1.99dB on REDS.
- Video Denoising: 0.82∼2.16dB PSNR improvements over existing methods show strong video-denoising performance across two benchmark datasets.The model is trained as one non-blind model for noise levels σ ∈[0, 50].
- Video Frame Interpolation and Space-Time Super-Resolution: VRT achieves best or competitive video-frame-interpolation results and provides a strong space-time super-resolution baseline despite not being task-specific.For space-time video super-resolution, cascading VFI and VSR improves PSNR by 1.03dB on Vid4.
- Ablation Study: Ablations show that reducing scales degrades performance, while increasing the attention window to 8 yields a 0.18dB improvement.Mutual attention benefits feature fusion but cannot alone preserve reference-frame information, motivating complementary design choices.
5. Conclusion
VRT is a multi-scale transformer for video restoration that jointly extracts, aligns, and fuses information across frames. Experiments across five restoration tasks show gains of up to 2.16dB on fourteen benchmark datasets.
- VRT jointly extracts, aligns, and fuses information from different frames at multiple resolutions.
- Its multi-scale framework combines temporal mutual self attention with parallel warping for feature extraction, alignment, and fusion.Mutual attention supports implicit flow estimation and feature warping, while self attention extracts features.
- Up to 2.16dB gains were reported across video super-resolution, deblurring, denoising, frame interpolation, and space-time video super-resolution.The experiments covered fourteen benchmark datasets.