Source-linked AI summary
Video Super-Resolution with Recurrent Structure-Detail Network
Takashi Isobe, Xu Jia, Shuhang Gu, Songjiang Li, Shengjin Wang, Qi Tian
TL;DR
Video super-resolution needs efficient use of temporal information while recurrent processing can accumulate appearance-related errors. The paper decomposes frames into structure and detail components for recurrent processing and selectively adapts hidden-state information. Experiments and ablations report favorable performance and speed, with the method's component separation relying on distinct reconstruction difficulties for the two components.
Problem
Video super-resolution methods need to exploit temporal information efficiently, while recurrent hidden states may contain information that is not equally useful for every current frame.
Method
The method recurrently processes separately decomposed structure and detail components using two-stream blocks and adapts hidden-state information for each current frame.
Results
Experiments on several benchmark datasets report favorable super-resolution performance and speed against state-of-the-art methods, while ablations support the proposed modules.
Takeaways & Limitations
Separating structure and detail, together with selective hidden-state use, provides the paper's recurrent route to robust video super-resolution.
Takeaways & Limitations
The approach assumes structure and detail components have different reconstruction difficulties and benefit differently from information in other frames.
Abstract
from arXiv · showhide
Most video super-resolution methods super-resolve a single reference frame with the help of neighboring frames in a temporal sliding window. They are less efficient compared to the recurrent-based methods. In this work, we propose a novel recurrent video super-resolution method which is both effective and efficient in exploiting previous frames to super-resolve the current frame. It divides the input into structure and detail components which are fed to a recurrent unit composed of several proposed two-stream structure-detail blocks. In addition, a hidden state adaptation module that allows the current frame to selectively use information from hidden state is introduced to enhance its robustness to appearance change and error accumulation. Extensive ablation study validate the effectiveness of the proposed modules. Experiments on several benchmark datasets demonstrate the superior performance of the proposed method compared to state-of-the-art methods on video super-resolution.
1 Introduction
Video super-resolution must exploit temporal information efficiently while avoiding the computational costs and artifacts associated with motion compensation. The proposed recurrent network separates structure and detail, adapts hidden-state use, and performs favorably in both speed and super-resolution performance.
- VSR belongs to multi-frame super-resolution, where exploiting complementary temporal information across frames is pivotal.
- Explicit motion-compensation methods estimate and align frames but incur heavy computation and can produce artifacts from inaccurate optical flow.
- Recurrent VSR methods stream frames by propagating previous outputs or hidden states, using many past frames with less processing time.
- The proposed recurrent network decomposes each frame into structure and detail, then aggregates current and previous components to reconstruct each frame.
- Hidden-state adaptation correlates the reference frame with hidden-state channels to emphasize helpful information and suppress outdated information.
- Ablations support the proposed modules, while benchmark experiments report favorable performance against state-of-the-art methods in both quality and speed.
2 Related Work
Related VSR work uses explicit motion compensation, implicit fusion, or recurrent processing to exploit information across frames. The proposed approach builds on recurrent processing while separately propagating structure and detail components.
- SISR methods rely on image priors and self-similarity, whereas VSR must exploit complementary information across frames.
- Explicit motion compensation: Explicit VSR methods estimate motion, compensate or align frames, fuse information, and upsample, but motion processing introduces heavy computational load.
- Implicit motion compensation: Implicit motion-compensation methods avoid explicit frame alignment by designing fusion modules that exploit spatial-temporal information.
- Recurrent networks: Recurrent networks integrate sequential information in hidden states and have been used for efficient video processing tasks.
3 Method
The method recurrently super-resolves video by separating each frame into structure and detail components, processing them through interacting two-stream blocks, and adapting hidden-state information to current appearance.
- Overview: Each low-resolution frame is decomposed into structure and detail components before recurrent processing.Structure models low-frequency information and motion, while detail captures fine high-frequency information and slight appearance changes.
- Recurrent Unit: The recurrent unit combines current and previous structure-detail information to sharpen structures and recover missing details over time.The D-branch extracts complementary details from past frames, whereas the S-branch enhances existing edges and textures.
- Structure-Detail block: The proposed SD block specializes structure and detail processing while exchanging information between the two components through addition.This design addresses the limitation of separate component-specific branches, where each branch cannot use all input information.
- Hidden State Adaptation: The HSA module computes current-frame-to-hidden-state channel correlations to highlight useful historical information and suppress outdated information.Spatially variant filters generated from the current frame are applied to hidden-state neighborhoods, producing similarity values used for adaptation.
- Hidden State Adaptation: The adapted hidden state helps make recurrent fusion more robust to appearance changes, occlusions, large deformation, and accumulated errors.The hidden state is treated as a historical dictionary whose channels may represent different scene appearances over time.
- Loss functions: Training uses separate structure, detail, and whole-frame loss terms balanced by hyper-parameters.The three terms address the trade-off between sharpened frames, recovered details, and weak edges or textures.
4 Experiments
Experiments evaluate the proposed recurrent VSR method through dataset choices, ablations, and comparisons with state-of-the-art methods, emphasizing reconstruction quality and efficiency.
- Datasets: The method is trained on Vimeo-90K and evaluated on Vimeo-90K-T, Vid4, and UDM10, with the latter two providing long sequences for recurrent information accumulation.Vimeo-90K contains about 90K seven-frame clips; Vid4 and UDM10 contain long sequences with diverse scenes.
- Ablation Study: Information exchange between structure and detail branches improves both PSNR and SSIM over separately processed branches.The comparison indicates that decomposition alone is insufficient because independent branches become blind to complementary information.
- Ablation Study: The HSA module adds about 0.04 dB PSNR and 0.002 SSIM on average across tested architectures.It also selectively uses historical hidden-state information, improving robustness to appearance change and accumulated errors to some extent.
- Ablation Study: Equal supervision weights for structure, detail, and the whole frame achieve 27.79/0.8474 in PSNR/SSIM.The (1, 1, 0) setting is 0.06 dB lower than (1, 1, 1), indicating that supervision on the combined image contributes to training.
- Comparison with State-of-the-arts: 27.61dB PSNR in Y channel and 26.13dB PSNR in RGB channels are achieved on Vid4 with only five SD blocks.Increasing to seven and nine SD blocks adds 0.18dB and 0.31dB Y-channel PSNR, respectively, with little FLOPs increase.
- Comparison with State-of-the-arts: On UDM10, RSDN 9-128 achieves new state-of-the-art performance, is about 15 and 37 times faster than DUF and RBPN, and is 0.61dB better than PFNL while three times faster.On Vimeo-90K-T, the method lags only EDVR-L and is six times faster.
5 Conclusion
The paper presents an effective and efficient recurrent network for streaming video super-resolution. It decomposes frames into structure and detail components and adapts hidden-state use to improve robustness, with experiments demonstrating effectiveness and efficiency.
- 5 Conclusion: The proposed recurrent network decomposes inputs into structure and detail components processed by two interleaved branches for high-resolution reconstruction.The design supports flexible supervision for the separate components.
- 5 Conclusion: Selective hidden-state use improves robustness to appearance change and error accumulation in recurrent video super-resolution.The hidden state is treated as temporal information whose useful content is selected for the current frame.
- 5 Conclusion: Experiments on several benchmark datasets demonstrate superiority in both effectiveness and efficiency.