Source-linked AI summary
MuCAN: Multi-Correspondence Aggregation Network for Video Super-Resolution
Wenbo Li, Xin Tao, Taian Guo, Lu Qi, Jiangbo Lu, Jiaya Jia
TL;DR
Existing VSR methods commonly rely on error-prone one-to-one correspondence and underuse self-similar structures. MuCAN aggregates multiple temporal correspondences and cross-scale nonlocal correspondences, achieving state-of-the-art performance on multiple benchmark datasets.
Problem
Existing VSR methods often model alignment explicitly or implicitly with one-to-one pixel correspondence, while similar structures within and across frames are underused.
Method
MuCAN is an end-to-end network that uses temporal multi-correspondence aggregation, cross-scale nonlocal-correspondence aggregation, and reconstruction to restore high-resolution frames.
Results
MuCAN achieves state-of-the-art performance on multiple benchmark datasets, while TM-CAM and CN-CAM improve PSNR by 1.15dB and 0.12dB, respectively.
Takeaways & Limitations
Highly correlated patches support robust alignment-based frame recovery, while multi-scale information and edge-aware loss improve details and refined edges.
Abstract
from arXiv · showhide
Video super-resolution (VSR) aims to utilize multiple low-resolution frames to generate a high-resolution prediction for each frame. In this process, inter- and intra-frames are the key sources for exploiting temporal and spatial information. However, there are a couple of limitations for existing VSR methods. First, optical flow is often used to establish temporal correspondence. But flow estimation itself is error-prone and affects recovery results. Second, similar patterns existing in natural images are rarely exploited for the VSR task. Motivated by these findings, we propose a temporal multi-correspondence aggregation strategy to leverage similar patches across frames, and a cross-scale nonlocal-correspondence aggregation scheme to explore self-similarity of images across scales. Based on these two new modules, we build an effective multi-correspondence aggregation network (MuCAN) for VSR. Our method achieves state-of-the-art results on multiple benchmark datasets. Extensive experiments justify the effectiveness of our method.
1 Introduction
MuCAN reframes VSR as aggregating multiple inter- and intra-frame correspondences rather than relying on one-to-one alignment. It combines temporal patch matching, cross-scale self-similarity, and edge-aware refinement in an end-to-end network.
- Motivation: VSR reconstructs high-resolution frames by exploiting temporal structure across multiple neighboring low-resolution frames.This extends single-image super-resolution, which primarily uses spatial information.
- Motivation: MuCAN treats VSR as inter- and intra-frame correspondence aggregation instead of separate alignment and regression stages.The approach aggregates similar content from multiple corresponding patches.
- Temporal Correspondence: Optical-flow alignment can make erroneous one-to-one pixel mappings that cannot recover the correct pixels.MuCAN addresses this by considering multiple correspondence candidates for each pixel.
- Temporal Correspondence: TM-CAM uses top-K similar feature patches with pixel-adaptive aggregation to provide robust temporal alignment.A patch-matching experiment found better flow estimates when more correspondence candidates were considered.
- Intra-frame Correspondence: CN-CAM aggregates similar features across spatial scales to exploit nonlocal self-similarity and recover image details.The module targets repeated patterns that may lie outside neighboring positions.
- Network: MuCAN combines TM-CAM, CN-CAM, and an edge-aware loss in an end-to-end network, achieving state-of-the-art performance on multiple benchmark datasets.The edge-aware loss emphasizes ground-truth edge regions during training.
2 Related Work
Prior super-resolution work includes deep single-image methods, while VSR methods aggregate multiple frames through alignment-free, optical-flow, or deformable-convolution approaches. These strategies trade simplicity, computational cost, alignment reliability, and artifact sensitivity.
- Single-Image Super-Resolution: Deep-learning SISR methods include convolutional, residual, sub-pixel, back-projection, channel-attention, and residual-dense architectures.These methods progressively improve reconstruction performance using deeper networks, efficient upsampling, and large image datasets.
- Video Super-Resolution: VSR methods use multiple frames and can be grouped by how they aggregate temporal information.The three broad categories are alignment-free processing, optical-flow compensation, and deformable convolution.
- Video Super-Resolution: Alignment-free methods use 3D convolutions or dynamic upsampling, but may incur high computation or ignore informative neighboring-frame details.Noise from misaligned regions can also harm reconstruction.
- Video Super-Resolution: Optical-flow methods compensate motion before or within reconstruction, but estimated flow can critically affect quality, especially in large-motion scenes.These methods also handle motion estimation and high-resolution reconstruction as relatively separated tasks.
- Video Super-Resolution: Deformable-convolution methods align features with learnable offsets, yet unreasonable offsets may produce noticeable reconstruction artifacts.EDVR is cited as extracting and aligning features at multiple levels.
3 Our Method
MuCAN is an end-to-end VSR network that aggregates multiple temporal correspondences and cross-scale self-similar features, while using edge-aware training to recover detailed high-resolution frames.
- MuCAN predicts a high-resolution central frame from consecutive low-resolution inputs using TM-CAM, CN-CAM, and reconstruction modules.The architecture is end-to-end and combines temporal correspondence aggregation with cross-scale aggregation.
- Temporal Multi-Correspondence Aggregation Module: TM-CAM aggregates multiple feature-patch correspondences to handle large motion and subtle sub-pixel shifts across neighboring frames.It performs hierarchical aggregation from low-resolution feature levels to high-resolution levels and operates in feature space.
- Temporal Multi-Correspondence Aggregation Module: TM-CAM selects top-K similar patches locally and fuses them with pixel-adaptive weights rather than relying on one-to-one correspondence.The aggregation weights vary by location and are produced from feature maps through convolution layers.
- Cross-Scale Nonlocal-Correspondence Aggregation Module: CN-CAM searches for nonlocal correspondences across feature scales and uses self-attention before merging the retrieved features.The module is designed to exploit multi-scale self-similarity to recover more details.
- Edge-Aware Loss: An edge-aware loss weights detected ground-truth edge regions more heavily during training to refine reconstructed image edges.Edges are extracted with a Laplacian filter, and the final loss balances the reconstruction and edge-aware terms.
4 Experiments
Experiments evaluate MuCAN’s modules, correspondence settings, edge-aware loss, benchmark performance, and generalization across REDS, Vimeo-90K, and real-world video frames.
- Ablation Study: The ablation study evaluates TM-CAM, CN-CAM, and EAL against a baseline on the REDS testing dataset.The baseline excludes the proposed modules and loss.
- Temporal Multi-Correspondence Aggregation: 1.15dB PSNR improvement comes from the temporal multi-correspondence alignment module over the 28.98dB baseline.The baseline also achieves 0.8280 SSIM.
- Temporal Multi-Correspondence Aggregation: More than 0.1dB PSNR gain is obtained with four temporal correspondences versus one, while additional weak correspondences can introduce noise.The implementation therefore uses K=4.
- Cross-Scale Nonlocal-Correspondence Aggregation: 0.12dB PSNR improvement is obtained with CN-CAM, which recovers more details in repeated patterns such as windows and buildings.The module aggregates information across spatial scales.
- Edge-Aware Loss: Edge-aware loss produces more refined edges, with clearer and sharper wall textures and light edges.These effects are illustrated in the REDS visual results.
- Comparison and Generalization: MuCAN outperforms prior methods by at least 0.17dB on REDS and improves over DUF by nearly 1.2dB on Vimeo-90K RGB channels.It also improves over RBPN by 0.25dB on the Vimeo-90K Y channel and shows decent generality on real-world video frames.
5 Conclusion
MuCAN combines temporal and cross-scale correspondence aggregation with edge-aware loss for video super-resolution, and experiments support its effectiveness and generality.
- MuCAN uses TM-CAM to exploit highly correlated patches for high-quality alignment-based frame recovery.
- CN-CAM utilizes multi-scale information to further improve MuCAN's performance.
- An edge-aware loss encourages more refined edges in the high-resolution output.
- Extensive experiments demonstrate the effectiveness and generality of the proposed method.