Source-linked AI summary
BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment
Kelvin C. K. Chan, Shangchen Zhou, Xiangyu Xu, Chen Change Loy
TL;DR
Video super-resolution must exploit complementary information across misaligned frames while handling long-term propagation and alignment challenges. BasicVSR++ redesigns BasicVSR with second-order grid propagation and flow-guided deformable alignment, achieving a 0.82 dB PSNR gain over BasicVSR with similar parameters and strong results on other restoration tasks.
Problem
Video super-resolution requires aggregating complementary information across misaligned frames, while recurrent models still face difficult long-term transmission and feature-alignment problems.
Method
BasicVSR++ redesigns BasicVSR with second-order grid propagation and flow-guided deformable alignment.
Results
0.82 dB in PSNR is gained over BasicVSR on REDS4 with similar numbers of parameters, while BasicVSR++ also generalizes to compressed video enhancement and succeeds in NTIRE 2021 challenges.
Takeaways & Limitations
Enhanced propagation and alignment improve information aggregation across video frames while maintaining efficiency and supporting multiple video restoration tasks.
Abstract
from arXiv · showhide
A recurrent structure is a popular framework choice for the task of video super-resolution. The state-of-the-art method BasicVSR adopts bidirectional propagation with feature alignment to effectively exploit information from the entire input video. In this study, we redesign BasicVSR by proposing second-order grid propagation and flow-guided deformable alignment. We show that by empowering the recurrent framework with the enhanced propagation and alignment, one can exploit spatiotemporal information across misaligned video frames more effectively. The new components lead to an improved performance under a similar computational constraint. In particular, our model BasicVSR++ surpasses BasicVSR by 0.82 dB in PSNR with similar number of parameters. In addition to video super-resolution, BasicVSR++ generalizes well to other video restoration tasks such as compressed video enhancement. In NTIRE 2021, BasicVSR++ obtains three champions and one runner-up in the Video Super-Resolution and Compressed Video Enhancement Challenges. Codes and models will be released to MMEditing.
1. Introduction
Video super-resolution must aggregate complementary information across misaligned frames, but recurrent methods still face difficult long-term transmission and feature-alignment problems. BasicVSR++ redesigns BasicVSR with second-order grid propagation and flow-guided deformable alignment to improve information aggregation.
- VSR requires complementary information from misaligned video frames, making cross-frame restoration challenging.
- Recurrent frameworks propagate latent features to exploit long-term dependencies and can be more compact than sliding-window methods.
- BasicVSR uses bidirectional propagation and optical-flow feature warping to exploit information from the entire input video.
- BasicVSR’s rudimentary propagation and alignment limit information aggregation, especially for fine details in occluded and complex regions.
- BasicVSR++ introduces second-order grid propagation and flow-guided deformable alignment to propagate and aggregate information more effectively.Grid propagation adds aggressive bidirectional, second-order connections; flow-guided deformable alignment addresses optical-flow fragility and deformable-alignment training difficulty.
- 0.82 dB in PSNR is gained over BasicVSR on REDS4 with similar numbers of parameters.
2. Related Work
Related work establishes recurrent propagation and feature alignment as important components for video processing, while BasicVSR++ uses a temporal grid rather than a multi-scale grid design. The paper focuses experiments on BasicVSR as the state-of-the-art VSR comparator.
- Recurrent Networks: Recurrent networks are used in video super-resolution, deblurring, and frame interpolation.
- Recurrent Networks: BasicVSR demonstrates that bidirectional propagation better exploits temporal features than unidirectional propagation.
- Recurrent Networks: BasicVSR also shows that feature alignment helps align highly relevant but misaligned features.
- Grid Connections: BasicVSR++ uses a grid structure for bidirectional propagation across time rather than the multi-scale grids used in other vision tasks.
3. Methodology
BasicVSR++ redesigns recurrent video super-resolution with second-order grid propagation and flow-guided deformable alignment to improve feature aggregation across misaligned frames.
- BasicVSR++ extracts frame features, propagates them through second-order grid propagation, aligns them with flow-guided deformable alignment, then reconstructs images using convolution and pixel-shuffling.
- Second-Order Grid Propagation: Grid propagation alternates backward and forward propagation, repeatedly revisiting information from different frames for feature refinement.
- Second-Order Grid Propagation: Second-order connections relax BasicVSR’s first-order Markov assumption, aggregating information from different spatiotemporal locations.
- Flow-Guided Deformable Alignment: Flow-guided deformable alignment pre-aligns features with optical flow, then estimates deformable-convolution offsets and masks from the prealigned features.
- Flow-Guided Deformable Alignment: Instead of directly predicting offsets, the alignment module learns residual offsets relative to optical flow, while deformable convolution operates on unwarped features.
- Flow-Guided Deformable Alignment: For second-order propagation, two warped features and their flows are concatenated to estimate offsets jointly, avoiding doubled computation and preserving complementary information.
4. Experiments
BasicVSR++ is evaluated across multiple video restoration datasets and degradations, achieving state-of-the-art quantitative performance and recovering fine details in challenging examples.
- Quantitative Comparisons: BasicVSR++ achieves state-of-the-art performance on all datasets under both degradations.This result is reported in Table 1.
- Quantitative Comparisons: 1.3 dB: BasicVSR++ outperforms EDVR by up to 1.3 dB in PSNR while using 65% fewer parameters.EDVR is described as a large-capacity sliding-window method.
- Quantitative Comparisons: 0.82 dB: the lighter BasicVSR++ (S) improves over BasicVSR by 0.82 dB in PSNR with comparable parameters and runtime.The same comparison reports a 0.57 dB improvement over IconVSR.
- Qualitative Comparisons: BasicVSR++ restores fine details in qualitative comparisons, including wheel spokes, stairs, and building structure that competing methods do not recover.These examples are shown on REDS4, Vimeo-90K-T, and Vid4.
5. Ablation Studies
Ablation studies attribute improvements to the proposed propagation and alignment designs, which enhance detail recovery, alignment sharpness, and temporal smoothness.
- Component Ablations: 0.14–0.46 dB: each proposed component brings considerable PSNR improvement in the ablation study.The study gradually inserts components into a baseline.
- Propagation Ablation: Second-order propagation and two propagation iterations improve performance, while further increases yield only 0.05 dB in PSNR.The final design therefore keeps both the order and iteration count at two.
- Propagation Ablation: Second-order grid propagation restores fine details and complex textures that variants without the proposed components render as blurry.The benefit is most noticeable where the current frame provides limited reconstruction information.
- Alignment Ablation: Flow-guided deformable alignment produces offsets similar to optical flow while retrieving information from multiple neighboring locations.This provides more flexibility than aggregating information from only one motion-indicated spatial location.
- Alignment Ablation: Optical-flow alignment produces blurry edges, whereas the proposed alignment yields sharper features that preserve more details.The difference is attributed to gathering information from neighboring locations rather than relying only on spatial-warping interpolation.
- Alignment Ablation: 2.17 dB: using optical flow only as offset-fidelity supervision reduces PSNR by 2.17 dB relative to the full model, while omitting flow causes training collapse.Directly incorporating optical flow into the network provides more explicit guidance.
- Compressed Video Enhancement: Compressed-video outputs from BasicVSR++ contain fewer artifacts and clearer details.The qualitative comparison is presented in Figure 10.
- Temporal Consistency: BasicVSR++ shows smoother temporal transitions than EDVR and BasicVSR, whose profiles exhibit noise or discontinuity.The comparison links EDVR’s noise to flickering artifacts and BasicVSR’s discontinuity to less smooth temporal behavior.
6. NTIRE 2021 Challenge Results
BasicVSR++ achieved leading results in NTIRE 2021 video restoration challenges, including recognition in both video super-resolution and compressed video enhancement.
- BasicVSR++ won the NTIRE 2021 video super-resolution track with a compact and efficient structure.
- BasicVSR++ obtained two champions and one runner-up in the compressed video enhancement challenge.
- BasicVSR++ reduced compressed-video artifacts and produced higher-quality outputs across three evaluated patches.
7. Conclusion
The paper redesigns BasicVSR with enhanced propagation and alignment components for video super-resolution, while maintaining efficiency and generalizing to other restoration tasks.
- BasicVSR++ redesigns BasicVSR with two components targeting propagation and alignment performance in video super-resolution.
- BasicVSR++ outperforms existing state-of-the-art methods by a large margin while maintaining efficiency.
- The designs generalize to other video restoration tasks, including compressed video enhancement.
A. Network Architecture
The network uses specified residual-feature extraction and propagation branches, plus a second-order deformable alignment module driven by features and optical flows from multiple timesteps.
- The initial feature extraction uses 5 residual blocks, while each propagation branch uses 7 residual blocks.
- The network feature channel is set to 64, and pretrained SPyNet serves as the flow network.
- Second-order alignment concatenates prealigned features and optical flows from different timesteps before estimating offsets and masks.
- The deformable alignment uses a kernel size of 3 and 16 deformable groups.
B. Experimental Settings
Experiments train on REDS and Vimeo-90K and evaluate across multiple video datasets under 4× bicubic or blur-downsampling degradations, using documented architectural and optimization settings.
- Datasets: Training uses REDS and Vimeo-90K, with REDS45 as test data and REDSval46 as validation data for REDS.
- Datasets: Evaluation also uses Vid4, UDM10, and Vimeo-90K-T as test sets alongside Vimeo-90K.
- Network implementation: The Co and Cm modules share six initial layers and can be implemented with convolution stacks followed by channel splitting.
- Degradations: All models use 4× downsampling with either Bicubic or Blur Downsampling degradations.
- Degradations: Blur Downsampling applies Gaussian blurring with σ=1.6 before subsampling every four pixels.
C. Qualitative Comparisons
BasicVSR++ qualitatively restores fine details across REDS4, UDM10, Vimeo-90K-T, and Vid4, producing plausible results in the reported examples.
- BasicVSR++ restores fine details and produces plausible qualitative results across the reported video restoration examples.The comparisons cover REDS4, UDM10, Vimeo-90K, and Vid4.
- 33.61 dB is reported for the ground-truth reference, while EDVR reaches 31.28 dB and Bicubic reaches 25.93 dB.
- 32.62 dB is reported for BasicVSR++ (ours), compared with 32.25 dB for IconVSR in the shown example.