Source-linked AI summary
Video Super-resolution with Temporal Group Attention
Takashi Isobe, Songjiang Li, Xu Jia, Shanxin Yuan, Gregory Slabaugh, Chunjing Xu, Ya-Li Li, Shengjin Wang, Qi Tian
TL;DR
Video super-resolution must recover high-resolution frames while exploiting temporal information despite motion and alignment challenges. The paper hierarchically groups frames by frame rate, fuses complementary information with attention, and adds fast spatial alignment for large motion; it reports state-of-the-art performance on two benchmarks.
Problem
Video super-resolution methods face inaccurate or computationally expensive motion alignment and inefficient fusion of information from distant neighboring frames.
Method
The method hierarchically integrates frame-rate-aware groups with intra-group and inter-group fusion, attention, and fast homography-based spatial alignment.
Results
The proposed method achieves state-of-the-art performance on two popular video super-resolution benchmarks.
Takeaways & Limitations
The method reconstructs high-quality high-resolution frames while adaptively using complementary information from groups with different frame rates.
Takeaways & Limitations
The grouping design assumes that distant groups have larger motion and near groups have smaller motion.
Abstract
from arXiv · showhide
Video super-resolution, which aims at producing a high-resolution video from its corresponding low-resolution version, has recently drawn increasing attention. In this work, we propose a novel method that can effectively incorporate temporal information in a hierarchical way. The input sequence is divided into several groups, with each one corresponding to a kind of frame rate. These groups provide complementary information to recover missing details in the reference frame, which is further integrated with an attention module and a deep intra-group fusion module. In addition, a fast spatial alignment is proposed to handle videos with large motion. Extensive results demonstrate the capability of the proposed model in handling videos with various motion. It achieves favorable performance against state-of-the-art methods on several benchmark datasets.
1. Introduction
The paper addresses limitations of video super-resolution methods that depend on motion estimation or inefficient temporal fusion. It proposes hierarchical frame-rate grouping, attention-based fusion, and fast spatial alignment, achieving state-of-the-art performance on two benchmarks.
- Video super-resolution combines spatial information across positions with temporal information across frames to recover missing details in low-resolution video.
- Explicit motion-compensation methods can suffer distortion from inaccurate alignment, especially with occlusion or complex motion, while optical flow is computationally expensive.
- The proposed network divides input sequences into frame-rate-aware groups, integrates information within and across groups hierarchically, and uses attention to borrow complementary details.
- Fast spatial alignment uses homography-based rough motion compensation to address large motion with less distortion than optical-flow-based alignment.
- The method achieves state-of-the-art performance on two popular video super-resolution benchmarks.
2. Related Work
Related video super-resolution methods use explicit or implicit temporal alignment to exploit neighboring frames. The proposed method differs from prior grouped-frame approaches by organizing groups by frame rate and hierarchically integrating their information.
- Video super-resolution methods use temporal alignment, either explicit or implicit, to exploit complementary information from neighboring low-resolution frames.
- Explicit approaches estimate motion or optical flow for inter-frame alignment, while implicit approaches use learned spatio-temporal representations or dynamic filters.
- The method most closely related to this work groups input frames but produces a super-resolution result for each group before combining them with attention.
- Unlike that prior approach, the proposed method divides frames by frame rate and integrates temporal information hierarchically.
3. Methodology
The method hierarchically integrates temporal information by grouping frames according to temporal distance, fusing features within groups, and then applying attention-based fusion across groups. A fast homography-based alignment module further addresses large motion while reducing distortion relative to pixel-level optical-flow warping.
- Motivation: The method uses hierarchical fusion because distant neighboring frames are not explicitly guided by the reference frame when directly processed with 3D convolutions.This can lead to insufficient information fusion and limit the reference frame’s ability to borrow information from distant frames.
- Temporal Grouping: The input sequence is divided into groups based on temporal distance from the reference frame, with the reference frame appearing in every group.This grouping supports integration of neighboring frames at different temporal distances and can generalize to arbitrary input lengths.
- Intra-group Fusion: Intra-group fusion extracts and combines spatio-temporal features within each group using dilation rates associated with the group’s frame rate.The module uses spatial feature extraction, a 3D convolution for spatio-temporal fusion, and 2D dense units for deeper integration.
- Inter-group Fusion: Temporal attention assigns position-wise weights across groups so complementary group features can be integrated according to their temporal relevance.Group-wise features produce one-channel attention maps, which are concatenated and normalized with a softmax along the temporal axis.
- Inter-group Fusion: The inter-group fusion module concatenates attention-weighted group features, processes them with 3D and 2D dense blocks, and produces a high-resolution residual map.The final reconstruction adds this residual map to a bicubic-upsampled reference image.
- Fast Spatial Alignment: Fast spatial alignment estimates homographies between consecutive frames and composes them to warp neighboring frames toward the reference frame.Unlike independently estimated pixel-level optical flow, global homography preserves structure better and introduces fewer artifacts, while reducing the alignment burden for large motion.
4. Experiments
Experiments evaluate the method on Vid4 and Vimeo-90K-T against state-of-the-art VSR approaches, including qualitative, quantitative, temporal-consistency, and ablation analyses. Results show strong reconstruction quality, temporal consistency, benefits from hierarchical grouping and attention, and improved performance from fast spatial alignment.
- Comparison with State-of-the-arts: The proposed method produces sharper edges and finer texture while exhibiting less flickering than other approaches.Temporal profiles are used to assess consistency across consecutive frames.
- Comparison with State-of-the-arts: On Vimeo-90K-T, the method outperforms TOFlow, DUF, and RBPN by a large margin in PSNR and SSIM.It remains comparable in PSNR and slightly better in SSIM than the substantially larger EDVR-L model.
- Ablation Study: Hierarchical temporal grouping outperforms direct 3D-convolutional stacking, while including the reference frame in each group improves results.The proposed grouping strategy adds a further 0.05dB improvement attributed to motion-based grouping.
- Ablation Study: Removing group attention slightly worsens performance, and additional input frames can provide useful information for reconstruction.The proposed method outperforms DUF even with two fewer input frames.
- Ablation Study: Fast spatial alignment raises TGA performance from 37.32dB to 37.59dB and is more efficient than PyFlow.The alignment reduces appearance differences without requiring perfect frame alignment.
5. Conclusion
The paper concludes that hierarchical implicit temporal integration, group fusion, attention, and fast spatial alignment enable high-quality video super-resolution across varied motion. Experiments also demonstrate temporal consistency.
- 5. Conclusion: The proposed network hierarchically integrates temporal information implicitly through groups with different frame rates.It uses intra-group fusion followed by adaptive inter-group fusion to leverage complementary information across frames.
- 5. Conclusion: Fast spatial alignment is proposed to handle videos with large motion.The method reconstructs high-quality HR frames while maintaining temporal consistency.
- 5. Conclusion: Extensive experiments on several benchmark datasets demonstrate the effectiveness of the proposed method.