Source-linked AI summary
EDVR: Video Restoration with Enhanced Deformable Convolutional Networks
Xintao Wang, Kelvin C. K. Chan, Ke Yu, Chao Dong, Chen Change Loy
TL;DR
Video restoration must align multiple frames despite large motion and fuse information affected by diverse motion and blur. EDVR addresses these challenges with coarse-to-fine deformable PCD alignment and temporal-spatial TSA fusion, winning all four NTIRE19 tracks and outperforming published state-of-the-art methods on video super-resolution and deblurring.
Problem
Video restoration requires accurate multi-frame alignment and effective fusion under occlusion, large motion, misalignment, and blur.
Method
EDVR uses feature-level PCD alignment in a coarse-to-fine framework and TSA fusion that weights information across frames, locations, and channels.
Results
EDVR wins all four NTIRE19 video restoration and enhancement tracks and outperforms state-of-the-art methods on video super-resolution and deblurring benchmarks.
Takeaways & Limitations
EDVR provides a unified framework with alignment and fusion designs that generalize across video super-resolution and deblurring tasks.
Takeaways & Limitations
Performance decreases 0.5-1.5 dB when training and testing data distributions mismatch, indicating substantial dataset bias.
Abstract
from arXiv · showhide
Video restoration tasks, including super-resolution, deblurring, etc, are drawing increasing attention in the computer vision community. A challenging benchmark named REDS is released in the NTIRE19 Challenge. This new benchmark challenges existing methods from two aspects: (1) how to align multiple frames given large motions, and (2) how to effectively fuse different frames with diverse motion and blur. In this work, we propose a novel Video Restoration framework with Enhanced Deformable networks, termed EDVR, to address these challenges. First, to handle large motions, we devise a Pyramid, Cascading and Deformable (PCD) alignment module, in which frame alignment is done at the feature level using deformable convolutions in a coarse-to-fine manner. Second, we propose a Temporal and Spatial Attention (TSA) fusion module, in which attention is applied both temporally and spatially, so as to emphasize important features for subsequent restoration. Thanks to these modules, our EDVR wins the champions and outperforms the second place by a large margin in all four tracks in the NTIRE19 video restoration and enhancement challenges. EDVR also demonstrates superior performance to state-of-the-art published methods on video super-resolution and deblurring. The code is available at https://github.com/xinntao/EDVR.
1. Introduction
EDVR addresses the alignment and fusion challenges posed by realistic video restoration benchmarks with a unified framework built around PCD alignment and TSA fusion. It wins all four NTIRE19 video restoration and enhancement tracks and outperforms state-of-the-art methods on video restoration benchmarks.
- Motivation: REDS introduces larger and more complex motions than existing datasets, creating a challenging benchmark for video restoration.The NTIRE19 benchmark supports fair comparisons and progress in video restoration.
- Motivation: Accurate multi-frame correspondence and effective feature fusion are difficult under occlusion, large motion, and severe blurring.Existing approaches often underuse temporal redundancy, making alignment and fusion central design challenges.
- EDVR Framework: EDVR combines Pyramid, Cascading and Deformable alignment with Temporal and Spatial Attention fusion in a framework applicable to super-resolution and deblurring.PCD performs feature-level alignment, while TSA emphasizes informative features across frames and spatial locations.
- EDVR Framework: Temporal attention weights neighboring features using their element-wise correlation with the reference frame, followed by spatial attention over locations and channels.The design accounts for unequal informativeness caused by imperfect alignment and blur.
- Results: EDVR wins all four NTIRE19 video restoration and enhancement tracks and shows superior performance to state-of-the-art methods on video super-resolution and deblurring.The four tracks cover clean and blurred video super-resolution and clean and compression-artifact video deblurring.
2. Related Work
Prior video restoration methods use optical flow, implicit motion compensation, or direct multi-frame fusion, but occlusion, large motion, and blur complicate alignment. EDVR builds on deformable convolution-based feature alignment while extending it with PCD and attention-based fusion.
- Video Restoration: Video restoration methods commonly use temporal alignment, but accurate optical flow is difficult under occlusion and large motions.TOFlow reports that standard optical flow is not always the optimal motion representation for video restoration.
- Video Restoration: DUF and TDAN avoid explicit optical-flow estimation through implicit motion compensation and outperform flow-based methods.TDAN aligns input frames at the feature level using deformable convolutions.
- Video Deblurring: Video deblurring methods may directly fuse multiple frames without explicit alignment because blur increases motion-estimation difficulty.EDVR instead seeks information from multiple frames using alignment, with a deblurring module added before alignment when blur is present.
- Deformable Convolution: EDVR adopts deformable convolution as the basic operation for its PCD alignment module, inspired by TDAN's feature-level alignment.Deformable convolutions learn offsets that sample beyond regular local neighborhoods.
3. Methodology
EDVR is a unified video-restoration framework that processes neighboring frames through feature-level deformable alignment and attention-based fusion before reconstruction. Its design includes task-specific handling for high-resolution or blurry inputs and a cascaded second stage for severe motion blur.
- 3.1. Overview: EDVR takes consecutive low-quality frames, aligns neighboring features to a reference frame with PCD, fuses them with TSA, and reconstructs the restored output.For video super-resolution, the output is formed by adding a predicted residual to a directly upsampled image.
- 3.1. Overview: High-resolution inputs are downsampled before most computation, while blurry inputs receive PreDeblur processing before alignment to improve alignment accuracy.The final upsampling layer restores the original input resolution.
- 3.2. Alignment with Pyramid, Cascading and Deformable Convolution: Learnable offsets are predicted from concatenated neighboring and reference features, and bilinear interpolation handles fractional sampling locations.This deformable alignment operates on feature maps rather than directly estimating optical flow.
- 3.2. Alignment with Pyramid, Cascading and Deformable Convolution: PCD performs feature-level deformable alignment in a pyramid and cascaded coarse-to-fine manner, propagating coarse estimates upward before refinement.The module uses a three-level pyramid and is jointly learned with the full framework without additional supervision or optical-flow pretraining.
- 3.3. Fusion with Temporal and Spatial Attention: TSA weights aligned features using spatial-specific temporal attention based on reference-neighbor similarity, then applies spatial attention to the fused representation.Attention-modulated features are aggregated by convolution, while spatial masks use a pyramid design to enlarge the attention receptive field.
- 3.4. Two-Stage Restoration: A second, shallower EDVR stage refines first-stage outputs because severe blur can impair motion compensation and detail aggregation.The cascade is intended to remove severe motion blur and reduce inconsistency among output frames.
4. Experiments
Experiments evaluate EDVR on standard video super-resolution and deblurring benchmarks, ablate its PCD and TSA modules, and assess performance on the NTIRE19 REDS challenges.
- Comparisons with State-of-the-art Methods: EDVR is evaluated on Vid4, Vimeo-90K-T, and REDS4 for 4× video super-resolution, plus REDS4 for video deblurring.The comparisons include published state-of-the-art methods and use RGB or luminance evaluation as specified by each table.
- Comparisons with State-of-the-art Methods: EDVR achieves comparable performance to DUF on Vid4 and significantly outperforms state-of-the-art methods on Vimeo-90K-T and REDS4.On Vid4, it outperforms methods other than DUF by a large margin; qualitative results show more accurate texture recovery.
- Comparisons with State-of-the-art Methods: EDVR outperforms state-of-the-art video deblurring methods by a large margin on REDS4 and recovers clear details from extremely blurry images.The comparison includes DeepDeblur, DeblurGAN, SRN-Deblur, and DBN.
- Ablation Studies: The PCD alignment module improves the baseline by 0.2 dB with four deformable convolutions and by nearly 0.4 dB over that design at roughly the same computational cost.The ablation also reports smaller and cleaner flow after PCD alignment, indicating improved handling of large and complex motions.
- Ablation Studies: The TSA attention module adds a 0.14 dB gain over the PCD model with similar computation, while lower-motion frames and regions tend to receive higher attention.The attention pattern indicates that these frames and regions are more informative for reconstruction.
- Ablation Studies: Training and testing on different dataset distributions causes performance decreases of 0.5–1.5 dB, demonstrating substantial dataset bias.The paper reports this bias when the training and testing distributions diverge.
- Evaluation on REDS Dataset: EDVR wins all four NTIRE19 video restoration and enhancement tracks, and two-stage restoration improves performance by around 0.5 dB.Self-ensemble is helpful in the first stage but provides only marginal improvement in the second; the second stage helps recover details in extremely blurry cases.
5. Conclusion
EDVR is a unified framework designed for video restoration and enhancement, centered on PCD alignment and TSA fusion. It won all four NTIRE19 challenge tracks and outperformed existing methods on several video super-resolution and deblurring benchmarks.
- EDVR combines PCD alignment and TSA fusion in a unified framework for diverse video restoration tasks.
- EDVR won all four NTIRE19 video restoration and enhancement challenge tracks.
- EDVR demonstrated superior performance to existing methods on several video super-resolution and deblurring benchmarks.